evaluation results

this page contains a summary of the results from the analysis of the front data that were obtained by EAS 226 in Fall 2012

data:

I have collected all of the groups individual spreadsheets into a single spreadsheet, following some edits that were made to fix problems in the individual spreadsheets.  The modifications that were made are detailed in the following Google doc:


note that most of these modifications were to dates and times.  Changes were made to ensure that each group had a consistent number of cases in the VER, NAM, and GFS data sets.  The number of dates varies from group to group.  Several dates were analyzed by each group, several dates are unique to individual groups.  The total number of analyzed VER front events that could be compared to both NAM and GFS front events was 294.

The data set can be found in the following Google spreadsheet:

Each row corresponds to a different observed (VER) analyzed front.  The information collected for each front consists of a feature number, date/time information, group #, lat/lon info (mid, end #1, end #2), type of front, strength of front, and the lengths and curvature value that was computed for each front.  Following these attributes are the normalized attributes that were used in the calculation of the Euclidean distances.  There are five normalized attributes that correspond to information related to the location, strength, size, and curvature of each observed front.  These are slightly different from what we discussed in class, the formulas are as follows:

normalized latitude = (average of (lat mid, lat end #1, lat end #2) - 30) / 16
normalized longitude = (average of (lon mid, lon end #1, lon end #2) + 100) / 20
normalized strength = (strength rating - 1) / 2
normalized length = (length from end #1 to end #2 - 700km) / 1500km
normalized curvature = (curvature - 1) / 1.15

for each of these, what we are subtracting is approximately the 10th percentile value for each attribute
what we are dividing by is the difference between the 90th and 10th percentile value for each attribute
the normalized attributes should vary between roughly 0 and 1 with: 
0 indicating the: southern-most, western-most, smallest, shortest, straightest
1 indicated the: northern-most, eastern-most, largest, longest, most curved

The next set of numbers in the spreadsheet are for the closest forecast front in the NAM in terms of Euclidean distance (square root of the sum of the squared differences between all five normalized attributes).  The Euclidean distance is the next number, followed by the five normalized attributes associated with the nearest NAM front.

A similar set of numbers follows in the spreadsheet for the closest forecast front in the GFS.

=== RESULTS ===

hypothesis #1: short-term forecasts are better than long-term forecasts

here, the short-term forecasts were from the NAM (36h forecasts) and the long-term forecasts were from the GFS (84h forecasts)

the results show that this hypothesis cannot be rejected, although we do not have a large enough sample size to come to any definite conclusion

the median Euclidean distance between all observed fronts and the nearest NAM front was = 0.503
the median Euclidean distance between all observed fronts and the nearest GFS front was = 0.578

62% of fronts had NAM distances that were lower than GFS distances

the figure below shows a scatter plot with the NAM distance on the x-axis and the GFS distance on the y-axis
the location of the median distance for the NAM and GFS is indicated by red lines
the majority (62%) of the "+" points are on the GFS side of the y=x line, which indicates that most of the NAM forecast fronts were closer to the observed attributes than the GFS fronts 

(click on the image to get a full-sized png file)


these data can also be summarized with a histogram showing the frequency of the differences between GFS and NAM Euclidean distances (GFS-NAM)
along the x-axis: "zero" indicates that the GFS distance = the NAM distance, a negative number indicates that GFS was closer to the observed frontal attributes than NAM, a positive number indicates that NAM was closer than GFS
along the y-axis: the bars indicate the number of observed fronts (labelled as "events") that fell into each bin of (GFS-NAM) distance difference, there are more observed fronts to the right of the zero line (indicating that NAM was closer) than to the left

click on image for a full-sized png

hypothesis #2: forecasts of strong fronts are better than forecasts of weak fronts

roughly half (52.7%) of the VER (observed) analyzed fronts were classified as "weak", 35.7% were classified as "moderate", and 11.6% were classified as "strong"

the results vary depending on whether we look at the GFS by itself, the NAM by itself, or both models together

for the GFS forecasts, the results are consistent with our hypothesis

median Euclidean distances for GFS forecasts (distance from observed front to nearest GFS forecast front) by observed strength classification:
weak: 0.622
moderate: 0.597
strong: 0.406

the figure below shows the distribution of Euclidean distance values for all of the weak, moderate, and strong observed fronts.  This is a "box-whisker" plot which summarizes several characteristics of the distribution of values in one plot.  For this plot, I have taken the logarithm of the Euclidean distance, this helps to "spread out" the distribution when values are clustered (skewed) towards lower values. The "box" outlines the middle 50% of the values (between Q1 [= 25th percentile] and Q3 [= 75th percentile]), with the red line in the middle of the box indicating the median value (Q2 = 50th percentile).  The "whiskers" are the lines that go above and below each box, this indicates a reasonable range of values for a maximum and minimum.  The size of the "whiskers" are 1.5 time the "IQR" or Inter-Quartile Range = Q3 - Q1.  The top whisker goes up to Q3+1.5*IQR, the bottom whisker goes to Q1-1.5 * IQR.  Values that are above or below the whiskers are considered "outliers" and are denoted by red symbols.

for the GFS forecasts, the distribution of Euclidean distance values for weak fronts are slightly larger values than for moderate fronts, which are also slightly larger values than for strong fronts.  This is consistent with our hypothesis that forecasts are better for strong fronts than for weak fronts.

click on the image to get the full-sized image


however, for the NAM forecasts, the results are inconsistent with our hypothesis

median Euclidean distances for NAM forecasts (distance from observed front to nearest NAM forecast front) by observed strength classification:
weak: 0.458
moderate: 0.487
strong: 0.528

the figure below shows the box-whisker diagram for the log(Euclidean distance) value distributions, classified by the observed frontal strength. the distribution of Euclidean distance values for weak fronts are very similar to those for moderate fronts, and those distance values for strong fronts appear to be larger than those for moderate fronts.  This is inconsistent with our hypothesis that forecasts are better for strong fronts than for weak fronts.

click on the image to get the full-sized image


if we combine the NAM and GFS datasets into a single distribution...

median Euclidean distances for NAM&GFS forecasts (distance from observed front to nearest forecast front) by observed strength classification:
weak: 0.540
moderate: 0.550
strong: 0.507

the figure below shows the box-whisker diagram for the log(Euclidean distance) value distributions, classified by the observed frontal strength. the distribution of Euclidean distance values are quite similar among the three strength classifications.  looking at the entire dataset as a whole, there does not appear to be a strong relationship between Euclidean distance and strength of the observed front


Comments