Background
The assessment of forecast quality is an essential part of the forecast process and in the development of new forecasting systems and techniques to improve future forecasts. Traditional verification methods typically evaluate forecasts on a point-by-point basis (assuming both forecasts and observations are on a grid) and can provide useful information. However, in a "feature-specific" approach which considers a specific meteorological phenomena as an entity, traditional verification methods are not usually appropriate since they do not take into account the spatial structure of phenomena (e.g. fronts, precipitation systems) , such as displacement. Incidentally, spatial verification methods are being actively developed within the research community, including automated methods of evaluating “feature-specific” predictions. This study will present the initial steps toward creating an automated, objective feature-based method of evaluating numerical predictions of frontal systems. These steps will primarily focus on the process of identifying frontal "objects" and obtaining attributes associated with them, such as size, shape, and intensity measurements.
Methods - Case Selection
- 42 total cases, from 2007-2012
- 12 cases manually selected
- The remaining case dates were selected using a random number generator, though the dates were quality controlled to ensure that a front existed (i.e. one was analyzed by the Hydrometeorological Prediction Center (HPC)) within the eastern CONUS domain at that valid time
- Criteria for Front Identification in this study
- Magnitude of the equivalent potential temperature gradient (lowest 30 mb) must be greater than or equal to 4 K/100km
- Represents a contiguous zone that is at least 500 km in length and 100 km in width
- Is located east of -105ºW and mainly over the CONUS.
- Data sets
- "Long Range"
- "Short Range"
- "Observations"
- All datasets are on the 211 grid (~81 km grid spacing)
- Each data file converted from gempak format into grib
- Identifying Frontal "Objects"
- Utilizes "connected component labeling"
- A common process in the image processing field
- Contiguous area with common characteristics (in this case, theta-e gradient >= 4 K/100 km) are given one unique "label" to identify the object as one entity
- This requires a structuring element to define connectivity
- 8-point connectivity utilized here
- Uses scipy.ndimage (a Python image processing module) to identify objects
- Obtaining Object Characteristics
- Used external image processing python package (scikits image processing- "skimage"), specifically using "regionprops"
- Centroid location (used to deduce centroid latitude and longitude)
- Area
- Major and Minor axis length
- Maximum, minimum, and mean gradient intensity
- Eccentricity
- Orientation
- Applied a frontal "strength" label based upon the following arbitrary classification
- Weak: 4 K/100 km<= gradient < 10 K/100 km
- Moderate: 10 K/100 km <= gradient < 15 K/100 km
- Strong: >=15 K/100 km
- Weak = 1, Moderate = 2, Strong = 3
- The Python code will first identify objects based upon the gradient threshold criteria. Objects will then be removed if they do not satisfy the length and width criteria (used major and minor axis length to determine this). This requires that remaining objects to be relabeled to maintain order
Example Case: 00Z January 30, 2008
As analyzed by HPC (see surface analysis figure below left), there were two significant frontal boundaries present at 00Z January 30, 2008 within our domain of the eastern CONUS. A cold front extended from Canada southward to the Gulf of Mexico while further to the west, a stationary front was located from western Missouri northwest into British Columbia in Canada. Also depicted in the western U.S. is a cold front extending south from Washington State into California. Before the passage of the front, temperatures during the late afternoon and early evening hours were in the mid 50’s in northern Indiana. In the span of about 20 minutes, from approximately 2325Z to 2345Z on the 29th, the temperature and dew point in West Lafayette, Indiana dropped nearly 22ºF! There was a corresponding drop in the surface θ, θv, and θe values (12.93 K, 15.56 K, and 62.11 K, respectively), and a rise in the mean sea-level pressure of nearly 4 mb, as observed from the weather station atop the Civil Engineering Building at Purdue University (see meteogram figure). HPC 00Z January 30, 2008 Surface Analysis Meteogram for the HAMP(CIVL) Roof Observations
Below are the frontal "objects" identified for each of the datasets, with tables of corresponding object attributes.
GFS 84 hr Forecast (20080126 12Z run) (click figure to enlarge)
| NAM 36 hr Forecast (20080128 12Z run) (click figure to enlarge)
| Verifying NAM 00hr Analysis ("Obs") (20080130 00Z) (click figure to enlarge) |
GFS object attributes | Object Number | Area
(# Grid Boxes) | Centroid Lat/Lon | Major Axis Length (km) | Minor Axis Length (km) | Max Intensity (K/100km) | Mean Intensity (K/100km) | Strength Category | Orientation | Eccentricity | | 1 | 228 | 37.1129ºN,-91.4184ºW | 3857.187 | 1096.569 | 13.69112 | 7.024389 | 2 | -1.0779 | 0.9587 | | 2 | 34 | 30.2755ºN,-84.0339ºW | 1159.413 | 422.8376 | 5.571619 | 4.524262 | 1 | -1.45811 | 0.9308 | | 3 | 12 | 33.4446ºN,-78.5025ºW | 551.3998 | 174.9581 | 6.120784 | 5.019843 | 1 | -0.9157 | 0.9483 | | 4 | 17 | 39.3786ºN,-68.3040ºW | 574.856 | 240.5946 | 5.107 | 4.609603 | 1 | -1.37681 | 0.9082 | | 5 | 78 | 44.6116ºN,-104.6430ºW | 1733.483 | 418.5537 | 11.30933 | 6.507114 | 2 | 0.55585 | 0.9704 |
NAM object attributes | Object Number | Area
(# Grid Boxes) | Centroid Lat/Lon | Major Axis Length (km) | Minor Axis Length (km) | Max Intensity (K/100km) | Mean Intensity (K/100km) | Strength Category | Orientation | Eccentricity | | 1 | 263 | 35.7096ºN,-93.2323ºW | 4018.853 | 1271.24 | 14.39557 | 7.557483 | 2 | -1.1123 | 0.948653 | | 2 | 26 | 33.4446ºN,-78.5025ºW | 739.5835 | 334.1224 | 6.227486 | 4.926971 | 1 | -0.81468 | 0.892134 | | 3 | 100 | 43.9721ºN,-103.6200ºW | 2083.723 | 422.6715 | 10.97265 | 6.779175 | 2 | 0.68539 | 0.979211 |
Obs object attributes | Object Number | Area
(# Grid Boxes) | Centroid Lat/Lon | Major Axis Length (km) | Minor Axis Length (km) | Max Intensity (K/100km) | Mean Intensity (K/100km) | Strength Category | Orientation | Eccentricity | | 1 | 282 | 35.7166ºN,-94.1162ºW | 4127.384 | 1182.18 | 15.81378 | 7.717159 | 3 | -1.0779 | 0.9587 | | 2 | 22 | 29.6578ºN,-85.7726ºW | 592.8219 | 336.781 | 5.887504 | 4.725968 | 1 | -1.45811 | 0.9308 | | 3 | 50 | 34.6901ºN,-76.5511ºW | 1520.5740 | 349.6413 | 6.740455 | 5.216867 | 1 | -0.9157 | 0.9483 | | 4 | 16 | 35.3240ºN,-83.5357ºW | 547.9631 | 278.3136 | 5.379724 | 4.510112 | 1 | -1.37681 | 0.9082 | | 5 | 112 | 43.9259ºN,-104.5750ºW | 2056.345 | 509.5042 | 10.0282 | 6.256601 | 2 | 0.55585 | 0.9704
|
Results - Results without object matching
- This is "useful to assess the climatology of features produced by a model." (Davis et al. 2006). While a climatology should encompass more than 42 cases over 5 year period, these fields still are useful to get a sense of the typical characteristics associated with objects identified in each dataset
| GFS | NAM
| Obs
| | Total # of Objects Identified | 131 | 142
| 143
|
- Object frequency by month
- Centroid frequency distribution
- Size frequency distribution
- Strength frequency distribution
- Length frequency distribution (major axis length)
- Width frequency distribution (minor axis length)
- Orientation frequency distribution
- Eccentricity frequency distribution
- Spatial frequency distribution
Future Work - We would like to explore these ideas:
- Are short term forecasts of fronts are more accurate than long-term forecasts?
- Are forecasts of "strong" fronts are more accurate than forecasts of "weak" fronts?
- Compare our objective results with the subjective analyses produced by students in EAS226, a sophomore-level course to introduce students to atmospheric research
- In order to provide objective verification statistics, we need to develop code that will match forecast objects to observed objects
- Euclidean Distance
- Will likely use scikits machine learning package (sklearn)
- Type fronts
- Use sign of temperature advection?
- Use of other thermodynamic variables
- potential temperature
- virtual potential temperature
Python Code: see attached file (example for identifying objects from NAM)
References Davis, C., B. Brown, R. Bullock, 2006: Object-Based Verification of Precipitation Forecasts. Part I: Methodology and Application to Mesoscale Rain Areas. Mon. Wea. Rev., 134, 1772-1784. |
 Updating... Michael Baldwin, Dec 13, 2012, 2:36 PM
|
