Three months into a pilot with a new wind forecast provider, a balancing service operator on the east coast North Sea corridor checked 24-hour-ahead wind speed forecasts against metered actuals over 90 days. Averages and correlations looked acceptable. But the 20 worst days held most imbalance costs. The provider's quoted average accuracy revealed nothing about this tail.
Informal validation hides what matters operationally by compressing results into summary statistics. Proper validation selects metrics that reflect your cost structure, rather than simply those on a provider's benchmarking slide.
What Forecast Metrics Measure
Professional forecast evaluation repeatedly uses three verification metrics: Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Continuous Ranked Probability Score (CRPS). They capture different properties, so treating them as interchangeable can mislead procurement.
Mean Absolute Error
MAE averages the absolute forecast errors, in the forecast variable's units. For wind speed, 1.4 m/s means the mean error is 1.4 m/s. Its penalty is linear: a 4 m/s error adds exactly four times the score contribution of a 1 m/s error.
MAE is the clearest metric and a sound starting point. It answers: how far off is this provider, on average, in usable units? For logistics and supply chain teams comparing precipitation or wind for routing, MAE is usually enough as the primary metric.
MAE weights errors only by size, not context. A 2-degree error on a mild autumn day may barely affect demand forecasts, while the same error on a winter peak day can separate committed generation from demand. MAE cannot distinguish them.
Root Mean Square Error
RMSE is the square root of mean squared error. Squaring errors gives disproportionate weight to large misses. A wind provider with RMSE 1.8 m/s has more large errors than one with MAE 1.4 m/s and RMSE 1.5 m/s, even with similar mean deviation. The MAE to RMSE gap describes error shape: large means frequent large misses, small means a more even spread.
In energy trading and grid operations, balancing mechanism penalties make cost non-linear, so RMSE often says more than MAE. If large misses drive exposure, use RMSE. If provider A has lower MAE but higher RMSE than B, A is more consistent while B has fewer catastrophic days. The relevant property depends on your exposure.
CRPS for Probability Forecasts
Many providers now issue probabilistic forecasts, such as an ensemble or parametric distribution instead of one temperature value for tomorrow at 14:00. CRPS is the standard verification metric for this format.
CRPS measures distance between the forecast cumulative distribution function and the observed outcome, represented as a step function. Lower is better. It rewards probability near the outcome and penalises overconfident, incorrectly concentrated distributions. A point forecast is a degenerate distribution, so CRPS becomes MAE.
For probabilistic forecasts used in reserve sizing, options pricing, or uncertain route planning, include CRPS in evaluation. Comparing the ensemble mean with MAE or RMSE is a valid first step, but discards the distributional information being purchased.
Skill Scores and Raw Error
An MAE of 1.4 m/s depends on what a region and season can achieve. Without a reference, raw error is not skill. Formal verification therefore uses skill scores normalised against a naive baseline.
Two common baselines are climatological persistence, using yesterday's observed value for today, and climatological average, the long-term mean for that day and location. A skill score of 0 matches the baseline. A score of 0.6 means 60% less error than the baseline.
Skill scores improve provider comparisons across regions because they reflect forecast difficulty. A provider with RMSE 2.1 m/s in the Scottish Highlands may show more genuine skill than one with RMSE 1.2 m/s across the English Midlands. Wind in complex coastal terrain is structurally harder to forecast. Raw errors without baseline normalisation cannot establish this.
Building a Validation Dataset
Self-managed validation begins by aligning issued forecast values with observed actuals at the same location, for the same variable and time step. Correct alignment is harder than it sounds.
The commonest preparation error is using the wrong initialisation time. Checking T+48 forecasts when operations use T+24 creates overly positive skill scores. Mixing spatial resolutions also misleads: a 1 km grid interpolated to a point differs from a 10 km aggregate, and combining them inflates apparent error.
For a weather-station actual, record its exact location, elevation, and instrument type. Metered generation used as a wind-speed proxy includes curtailment, turbine availability, and wake noise absent from the meteorological signal. Keep the window as long as practical: 12 months covers a winter peak and summer shoulder. Under 6 months cannot reveal seasonal bias, a major operational failure mode.
What Metrics Leave Unstated
Climingo's benchmarking output has a defined limit. Skill metrics measure central tendency and distributional fit against historical actuals, not direct cost savings. The link between a 15% MAE improvement and lower imbalance charges depends on how operations respond to the signal. Point-forecast position sizing and ensemble-spread hedging can produce different cost responses.
Metrics describe past performance over a defined window. A provider with strong scores over 18 months may have recently changed its model configuration, and updates can occur without announcement. Refresh validation at least annually, rather than treating procurement as a permanent choice.
This validation framework underpins Climingo's provider scoring. We archive forecasts at issuance, pair them with verified actuals, and calculate MAE, RMSE, CRPS, and skill scores against a climatological baseline for each provider, region, variable, and horizon covered. We cannot set metric weights for your cost function. That translation requires your team's operational knowledge.