The default trap in weather procurement
Among companies onboarded during Climingo's early-access pilot programme, the usual answer to "how did you choose your current weather data provider" was "it came with something else" or "we have always used them." Some had run a trial, but seldom against a systematic set of their own operational actuals.
This default is understandable. Weather data is rarely the main operations budget item, and switching feels costly when a feed sits inside a forecasting stack or trading system. Yet without deliberate procurement, you pay for accuracy you cannot verify, while provider skill varies by region, variable, and horizon.
The five steps below are the procurement process we recommend to new customers and pilot participants. The same logic applies whether Climingo runs the benchmark or you build it independently.
Step 1: Specify your operational exposure
Before comparing providers, define what your operations actually need from a forecast. Three dimensions matter.
Variable specificity. Temperature, hub-height wind speed, precipitation rate and onset, solar irradiance, and significant wave height are different forecasting tasks. A provider leading on 2-meter temperature in the English Midlands may be mediocre on precipitation onset, which involves different physical processes and modelling investment. Identify the two or three variables driving your decisions before comparing providers.
Geographic specificity. Informal comparisons often fail here. National and regional averages hide variation across terrain, coastal exposure, and local circulation. Teams in the East Anglian fenlands and Scottish Highlands face different forecasting problems. Strong performance in one location does not ensure strong performance in the other. Base procurement on your geography, not a continental average accuracy headline.
Horizon specificity. A day-ahead energy desk needs accuracy at 12 to 36 hours. An intraday grid balancing team needs 1 to 6 hours. A weekly logistics planner has different needs. Skill scores do not transfer across horizons: providers strong in ensemble initialisation may excel at medium range yet add nothing over persistence at short range. Define your relevant horizon before benchmarking.
Step 2: Assemble historical actuals
A credible procurement benchmark starts with your historical observations: what weather actually occurred at your locations over a period covering seasonal variation.
For energy operations, this usually means metered output or demand actuals linked to the weather variables identified. For logistics, it may mean shipment records, dock disruption logs, or route delays tied to weather events. Format matters less than independence from provider forecasts.
Twelve months is a practical first-benchmark target. Six months is the minimum, since shorter windows can capture an anomalous season and distort rankings. We have seen a calm-summer benchmark reverse when tested over a full annual cycle with winter storms. Your actuals' seasonal mix is part of the test.
Step 3: Align forecasts with your actuals
Once actuals are available, retrieve forecast files issued before the outcome was known, at the issuance time relevant to your decision horizon.
The distinction matters. A forecast retrieved at verification time can look better than the version available at issuance, after model updates during the intervening hours. Fair comparison requires the forecast you would have acted on. Providers issue on regular schedules, so use archived versions of those issuances.
Climingo holds archived files from over 40 providers and matches them automatically when customers upload actuals. Building independently? Start archiving candidate-provider forecasts now, before you have enough actuals for a full benchmark. A longer archive produces a richer comparison.
Step 4: Calculate skill scores in context
With matched forecasts and actuals, the core metrics are simple. For deterministic points, Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) compare providers for the same variable, region, and horizon. RMSE weights large errors more heavily, useful when tail errors matter: a severe temperature miss on a critical trading day can cost more than a typical 1-degree bias. MAE shows average error linearly.
For ensemble or percentile products, Continuous Ranked Probability Score (CRPS) measures distributional accuracy. That matters when probabilistic forecasts size positions or set reserve margins instead of supporting binary decisions.
Interpret the numbers operationally. An RMSE of 1.6 degrees Celsius on 48-hour temperature forecasts matters to a team that can estimate the cost of a 1-degree bias in demand error or imbalance charges. Without that mapping, skill scores are only numbers. Link accuracy to operational cost, even approximately, so procurement reflects the provider performance gap.
Step 5: Repeat the evaluation regularly
A one-off benchmark gives a current ranking. It becomes strategy only with a planned re-evaluation cadence.
Provider models change: data assimilation, spatial resolution, and post-processing calibration are revised. A provider ranked first today may look very different within 18 months. Annual re-benchmarking is a sensible baseline. Run quarterly updates when forecast exposure is high, such as high-renewable periods when intraday imbalance costs are most sensitive to error.
Document the methodology so a colleague unfamiliar with the first run can reproduce it. Reproducibility lets you track provider performance instead of treating each evaluation as isolated.
What outcome benchmarking cannot show
This approach has limits. Historical skill scores show how a provider performed across the weather distribution in your evaluation window. They do not show performance during future extremes outside that distribution.
Tail events are rare by definition. A provider may score well on routine temperature and wind yet struggle during a once-in-fifteen-years cold snap or rapid cyclonic deepening. If exposure is material, such as for a coastal energy asset or disruption-sensitive supply chain, historical scores are necessary but incomplete. Add provider information on ensemble spread, extreme-event handling, and model uncertainty for a fuller purchasing view.
This limitation is structural to empirical benchmarking. You can assess what happened, not performance in conditions absent from your dataset.
Where to begin
Start by assembling 12 months of primary actuals for the two or three variables and geography most important to operations. This is the foundation. Provider evaluation, skill scoring, and cadence planning follow.
If clean, benchmark-ready actuals are unavailable, that gap is useful evidence: your procurement decision lacks verification. Building that mechanism is worthwhile, whichever provider you ultimately assess with it.