Kim's Final: Feature-specific evaluation of surface front predictions


       The assessment of forecast quality is an essential part of the forecast process and in the development of new forecasting systems and techniques to improve future forecasts. Traditional verification methods typically evaluate forecasts on a point-by-point basis (assuming both forecasts and observations are on a grid) and can provide useful information. However, in a "feature-specific" approach which considers a specific meteorological phenomena as an entity, traditional verification methods are not usually appropriate since they do not take into account the spatial structure of phenomena (e.g. fronts, precipitation systems) , such as displacement. Incidentally, spatial verification methods are being actively developed within the research community, including automated methods of evaluating “feature-specific” predictions. This study will present the initial steps toward creating an automated, objective feature-based method of evaluating numerical predictions of frontal systems. These steps will primarily focus on the process of identifying frontal "objects" and obtaining attributes associated with them, such as size, shape, and intensity measurements.

  • Case Selection
    • 42 total cases, from 2007-2012

    • 12 cases manually selected 
    • The remaining case dates were selected using a random number generator, though the dates were quality controlled to ensure that a front existed (i.e. one was analyzed by the Hydrometeorological Prediction Center (HPC)) within the eastern CONUS domain at that valid time
  • Criteria for Front Identification in this study
    • Magnitude of the equivalent potential temperature gradient (lowest 30 mb) must be greater than or equal to 4 K/100km
    • Represents a contiguous zone that is at least 500 km in length and 100 km in width
    • Is located east of -105ºW and mainly over the CONUS.
  • Data sets
    • "Long Range"
      • GFS 84 hour forecast
    • "Short Range"
      • NAM 36 hour forecast
    • "Observations"
      • NAM 00 hour analysis
    • All datasets are on the 211 grid (~81 km grid spacing)
    • Each data file converted from gempak format into grib
  • Identifying Frontal "Objects"
    • Utilizes "connected component labeling"
    • A common process in the image processing field
    • Contiguous area with common characteristics (in this case, theta-e gradient >= 4 K/100 km) are given one unique "label" to identify the object as one entity
      • This requires a structuring element to define connectivity
      • 8-point connectivity utilized here
    • Uses scipy.ndimage (a Python image processing module) to identify objects
  • Obtaining Object Characteristics
    • Used external image processing python package (scikits image processing- "skimage"), specifically using "regionprops"
      • Centroid location (used to deduce centroid latitude and longitude)
      • Area
      • Major and Minor axis length
      • Maximum, minimum, and mean gradient intensity
      • Eccentricity
      • Orientation
    • Applied a frontal "strength" label based upon the following arbitrary classification
      • Weak:          4 K/100 km<= gradient < 10 K/100 km
      • Moderate:    10 K/100 km <= gradient < 15 K/100 km
      • Strong:        >=15 K/100 km 
      • Weak = 1, Moderate = 2,  Strong = 3
  • The Python code will first identify objects based upon the gradient threshold criteria. Objects will then be removed if they do not satisfy the length and width criteria (used major and minor axis length to determine this). This requires that remaining objects to be relabeled to maintain order

Example Case: 00Z January 30, 2008

As analyzed by HPC (see surface analysis figure below left), there were two significant frontal boundaries present at 00Z January 30, 2008 within our domain of the eastern CONUS. A cold front extended from Canada southward to the Gulf of Mexico while further to the west, a stationary front was located from western Missouri northwest into British Columbia in Canada. Also depicted in the western U.S. is a cold front extending south from Washington State into California. Before the passage of the front, temperatures during the late afternoon and early evening hours were in the mid 50’s in northern Indiana. In the span of about 20 minutes, from approximately 2325Z  to 2345Z on the 29th, the temperature and dew point in West Lafayette, Indiana dropped nearly 22ºF! There was a corresponding drop in the surface θ, θv, and θe values (12.93 K, 15.56 K, and 62.11 K, respectively), and a rise in the mean sea-level pressure of nearly 4 mb, as observed from the weather station atop the Civil Engineering Building at Purdue University (see meteogram figure).
      HPC 00Z January 30, 2008 Surface Analysis                                         Meteogram for the HAMP(CIVL) Roof Observations

Below are the frontal "objects" identified for each of the datasets, with tables of corresponding object attributes.

GFS 84 hr Forecast
(20080126 12Z run)
(click figure to enlarge)

NAM 36 hr Forecast
(20080128 12Z run)
(click figure to enlarge)

Verifying NAM 00hr Analysis ("Obs")
(20080130 00Z)
(click figure to enlarge)


GFS object attributes                                                        
 Object NumberArea
(# Grid Boxes) 
Centroid Lat/LonMajor Axis Length (km)Minor Axis Length (km)Max Intensity (K/100km) Mean Intensity (K/100km)Strength Category OrientationEccentricity
228 37.1129ºN,-91.4184ºW3857.1871096.56913.691127.024389 2-1.0779 0.9587 
34 30.2755ºN,-84.0339ºW1159.413  422.83765.571619 4.524262 -1.45811 0.9308 
312 33.4446ºN,-78.5025ºW 551.3998 174.9581 6.120784 5.019843 -0.91570.9483
17 39.3786ºN,-68.3040ºW 574.856 240.5946 5.107 4.609603 -1.37681 0.9082 
78 44.6116ºN,-104.6430ºW 1733.483 418.5537 11.30933 6.507114 0.55585 0.9704 

NAM object attributes                                                        
 Object NumberArea 
(# Grid Boxes) 
Centroid Lat/LonMajor Axis Length (km)Minor Axis Length (km)Max Intensity (K/100km) Mean Intensity (K/100km)Strength Category OrientationEccentricity
263 35.7096ºN,-93.2323ºW4018.8531271.2414.395577.5574832-1.11230.948653
2633.4446ºN,-78.5025ºW739.5835 334.12246.227486 4.926971 -0.814680.892134
310043.9721ºN,-103.6200ºW 2083.723 422.6715 10.97265 6.7791750.685390.979211

Obs object attributes                                                        
 Object NumberArea 
(# Grid Boxes) 
Centroid Lat/LonMajor Axis Length (km)Minor Axis Length (km)Max Intensity (K/100km) Mean Intensity (K/100km)Strength Category OrientationEccentricity
282 35.7166ºN,-94.1162ºW4127.3841182.1815.813787.717159 3-1.0779 0.9587 
22 29.6578ºN,-85.7726ºW592.8219 336.7815.8875044.725968 -1.45811 0.9308 
350 34.6901ºN,-76.5511ºW 1520.5740 349.6413 6.7404555.216867 -0.91570.9483
16 35.3240ºN,-83.5357ºW 547.9631278.31365.3797244.510112 -1.37681 0.9082 
112 43.9259ºN,-104.5750ºW2056.345 509.504210.0282 6.2566010.55585 0.9704 

  • Results without object matching
    • This is "useful to assess the climatology of features produced by a model." (Davis et al. 2006). While a climatology should encompass more than 42 cases over 5 year period, these fields still are useful to get a sense of the typical characteristics associated with objects identified in each dataset
    • Objects identified

Total # of Objects Identified   131  142 

    • Object frequency by month
    • Centroid frequency distribution
      • Latitude

      • Longitude

    • Size frequency distribution

    • Strength frequency distribution

    • Length frequency distribution (major axis length)
    • Width frequency distribution (minor axis length)

    • Orientation frequency distribution
    • Eccentricity frequency distribution
    • Spatial frequency distribution

Future Work
  • We would like to explore these ideas:
    1. Are short term forecasts of fronts are more accurate than long-term forecasts?
    2. Are forecasts of "strong" fronts are more accurate than forecasts of "weak" fronts?
    3. Compare our objective results with the subjective analyses produced by students in EAS226, a sophomore-level course to introduce students to atmospheric research
  • In order to provide objective verification statistics, we need to develop code that will match forecast objects to observed objects
    • Euclidean Distance
    • Will likely use scikits machine learning package (sklearn)
  • Type fronts
    • Use sign of temperature advection?
  • Use of other thermodynamic variables
    • potential temperature
    • virtual potential temperature
  • More case studies

Python Code: see attached file (example for identifying objects from NAM)

Davis, C., B. Brown, R. Bullock, 2006: Object-Based Verification of Precipitation Forecasts. Part I: Methodology and Application to Mesoscale Rain Areas. Mon. Wea. Rev., 134, 1772-1784.
Michael Baldwin,
Dec 13, 2012, 2:36 PM