Which rainfall product should we trust where?

The answer

Checked against independent rain gauges

The only real measurement on this page. AMMA-CATCH runs dense research networks of rain gauges in West Africa, with 120 gauges in the cells used here. They are not used by any of the twelve products: GPCC used 0–1 gauges a month in these cells, and the 8 AMMA gauges that also appear in CHIRPS v3's station lists were left out. Each cell's monthly rainfall is the average of at least three complete gauge-months, which is close to the true area average rather than one point.

10 cells of 0.5° (Benin (Ouémé), Mali (Gourma), Niger (Niamey)), 1,159 cell-months in rain-season months, 2001–2020; a median of 3–14 gauges per cell-month. West Africa only: a strong local test, not a global verdict.

How it was done, step by step

Every picture below is drawn from the same run that produced the scores, so each step can be checked.

0. What gauge data exists, and which years are used

The gauges come from ten AMMA-CATCH daily datasets, each with its own sites and years: Niamey from 1990, Benin from 1999, most of Gourma only 2005–2011. The scores use 2001–2020, because that is when every product has data. GPCC Full, one of the products and one of the four yardsticks, ends in 2020. The Benin and Niamey gauges continue to 2025, so the check could be extended by five years with GPCC Monitoring as the gauge yardstick instead. Darker cells mean more complete station-months that year; the outlined band is the scoring window.

"Used" = stations that contributed at least one scored month. The 1990–2004 Niamey dataset has 166 stations but only 2001–2004 falls in the window; the Gourma transect, Cotonou and the other Niger towns have too few gauges in any one 0.5° cell.

1. Where the gauges are, and which were used

Every gauge record in the ten daily AMMA-CATCH datasets, coloured by what happened to it. Squares are the 0.5° grid cells that were scored; the products estimate rain for these squares. Each square is lettered by site (Ouémé A–E, Niamey A–D, Gourma A), and the charts below use the same names. Hover a dot for the station and the reason, or a square for its coordinates; click a square to look at it in step 5.

2. From gauge records to scored months

A month counts for a station only if every day has a value the network marked as quality-checked. Days the network filled from a backup bucket after a gauge failed are treated as missing. A cell-month needs at least 3 such stations, so it is an area average, not a single point. "Rain-season months" are months with at least 10 mm in GPCC's long-term average for that cell.

3. Do the products already use these gauges?

If a product was built from these gauges, comparing them would be marking it against its own answers. For each scored cell: how many AMMA gauges report in a typical rainy-season month, against how many gauges GPCC and CHIRPS v3 used in that cell. GPCC uses 0–1, so it is not built from them. AMMA gauges within 2 km of a CHIRPS v3 station were removed (red dots in step 1).

4. When each cell has data

Complete station-months per cell and year (darker = more). The scores use 2001–2020 (outlined), the years every product covers.

5. What is compared: one cell, month by month

The gauge average (black) against products for one cell, as monthly totals in mm. Dots on the gauge line mark the months that were scored; the gaps are dry-season months or months without three complete gauges.

6. Turning months into a score

Each month is expressed as how unusual it was for that cell and calendar month (0 = normal, +1 = one standard deviation wetter). That removes whether a product runs wet or dry overall. Each dot is one scored cell-month: the gauges' value across, the product's value up. Tight along the diagonal = the product gets wet and dry months right. r² is the share of the gauges' ups and downs the product reproduces.

7. The scores

r² = share of the gauges' month-to-month ups and downs (standardised anomalies, rain-season months) that the product reproduces. Spearman = the same with ranks, so a few extreme months cannot dominate. Dry-month hit rate = when the gauges put a month in its driest third, how often the product does too; 33% would be chance. False alarms = the share of the product's driest-third months that the gauges did not call dry. Ranges are 90% bootstrap intervals over years. Bias in totals does not count here; see the original report for that.

3-month totals: the scale drought triggers use

The same comparison on 3-month totals (each 3-month window compared with the same window in other years). A window counts only if all three months have gauge data; windows overlap, which the year-resampled ranges account for.

By month and site

The same comparison for each calendar month, so a product that is good in the core rains but poor at the start or end of the season shows up. Dry-season months in the Sahel have too little rain to score. Each cell holds about 60–80 values, so its 90% range is wide: treat differences under about 0.15 as noise. The outlined value in each column is the best product available in time for a trigger.

By period

How well did the estimates on this page do?

The estimates further down use no gauges. At these same cells and months they can be compared with the gauge result directly. They got CHIRPS v3 right and IMERG Final, GPCC and ERA5 roughly right, but overrated CHIRPS v2 and underrated IMERG Late. The likely reason is that both kinds of shared mistake exist, satellites sharing infrared and GPCC with ERA5 sharing smoothing, while the estimates have to assume one or the other. So outside gauged areas, read the estimated ranks of IMERG Late, ERA5 and CHIRPS v2 as uncertain by about ±0.15.

Gauge-based ρ² = r² ÷ the gauges' own reliability (0.78, estimated from product pairs whose mistakes are independent either way). "If satellites share mistakes" is this page's main estimate; "if GPCC and ERA5 share mistakes" is the alternative.

Checked against TAHMO stations across Africa

A second, independent gauge network: TAHMO's curated weather stations, 538 stations in 192 grid cells across East, West, Southern and Central Africa, 2015–2025. GPCC Full ends in 2020, so here GPCC Monitoring takes its place. Stations within 2 km of a station CHIRPS v3 uses were removed (32 of them). Only regional summaries are shown: TAHMO's research licence does not allow sharing its data or station-level results.

From stations to scored months

A month counts for a station only if every day is complete, and a day only if at least 95% of its 5–15-minute readings are present. A cell is scored only if it has at least 24 rain-season months with every product present, which is why many stations with short records drop out at the last step. TAHMO is young: most stations started between 2015 and 2019, so records are short (median 6 years with data per station).

The scores

Cells per region: East Africa 79; West Africa 84; Southern Africa 22; Central Africa 7. Independence from TAHMO is established for CHIRPS (station-list match); it is not established for CPC, GPCC Monitoring, PREC/L and IMERG Final, which take gauge data through national weather services and may include TAHMO stations. CRU scores near zero because it has almost no African stations after 2015 and falls back to its long-term average. Data: TAHMO (Van de Giesen, Hut and Selker 2014, WIREs Water 1(4), 341–348, doi:10.1002/wat2.1034).

What is this compared against?

Main findings

1. The best product depends on the region. In West Africa CHIRPS v3 leads against both gauge networks and the gap to IMERG Late widens at 3 months (AMMA r² 0.53 against 0.37). In East Africa IMERG Late and CHIRPS v3 are level; in Southern Africa ERA5 and IMERG Late track 3-month anomalies better than CHIRPS v3, with dry-period detection about equal.
2. ERA5 is unusable for rainfall triggers in the Sahel (AMMA 3-month r² 0.24, TAHMO West Africa 0.06, dry hit rates 54% and 46%) but mid-pack or better in East and Southern Africa.
3. IMERG Late does not improve with longer windows. Against AMMA its r² falls from 0.45 monthly to 0.37 for 3-month totals while CHIRPS v3 holds (0.55 to 0.53): its mistakes persist from month to month.
4. One gauge per cell ranks products fairly. On the dense AMMA cells, scoring against a single gauge instead of the cell average lowered every product's r² by the same factor (0.61–0.63) and left the ranking unchanged, so TAHMO's mostly single-station cells give valid rankings at lower levels.
5. Where there are no gauges, the gauge analyses are not the reference. Against AMMA, GPCC, CPC, PREC/L and CRU score 0.28–0.42 monthly, behind CHIRPS v3 and IMERG; CRU scores near zero against TAHMO (2015–2025) because it has almost no African stations left.
6. Twelve products, far fewer independent opinions. After removing the common signal, CHIRP v2 (no stations) and v3 still move together at +0.60, the gauge analyses at +0.06 to +0.25, and IMERG Final at +0.20 with GPCC, to which it is calibrated.
7. The estimates without gauges are useful but imperfect. At the AMMA cells they got CHIRPS v3 right (0.72 estimated, 0.71 measured) but overrated CHIRPS v2 (0.77 against 0.56) and underrated IMERG Late (0.46 against 0.57).

The twelve products in one picture

Yardstick: the average of all twelve products (agreement, not accuracy).

A principal component analysis of how the products' monthly rain anomalies move together, pooled over every cell and month in the region. Each dot is a product, placed by its loadings: products that move together sit together, and a product far from the rest is an outlier. The axes give the share of variance each component explains. This shows agreement, which is related to accuracy but not the same: a product built from different inputs than the rest can sit apart and still be good (ERA5 in Africa), and a product made from the same ingredients as the rest sits in the middle regardless (CHIRPS, IMERG Final).

When do they disagree?

Spread of the twelve products' standardised anomalies in each month (standard deviation across products, median over the region's cells). Higher means the products tell more different stories about that month.

Scoring products without a true rainfall dataset

Why not just compare with rain gauges?

Where independent gauges exist, that is the best check. Three things get in the way here. A gauge measures rain in a funnel about 20 cm wide, while each grid cell is about 55 × 55 km: a storm can drop 50 mm on the gauge and nothing 10 km away, so even a perfect gauge differs from the cell's true average. Two-thirds of these cells never had a GPCC gauge, so "GPCC" there is interpolated from stations that can be hundreds of kilometres away. And the gauges that are available are already inside CHIRPS and IMERG Final, so scoring those products against them partly scores them against themselves.

What is done instead

Inputs: G = the shared gauge network (GPCC / GHCN / CAMS stations); C = CHIRP satellite infrared and climatology; I = IMERG satellite constellation; E = ERA5 reanalysis. ● marks the four yardstick products. CHIRPS v3 fills gaps with ERA5; that is 0.25% of this domain, and the cells with more than 1% fill are left unscored.

Which products share mistakes? The data lean one way, the gauges say both

The four products' disagreement could in principle come from a different pair sharing mistakes, GPCC with ERA5 (the what-if below). The two explanations predict opposite things where gauges are dense. If the satellites share mistakes, the disagreement should grow there, because GPCC gets better while the satellites' shared mistake stays. If GPCC and ERA5 did, it should shrink, because GPCC's mistakes shrink with more gauges. It grows, everywhere and within Africa, so the satellites' shared mistake is the larger one, and it is what the report's triple collocation assumed. But the independent gauges show GPCC and ERA5 share mistakes too: at the gauged cells, ERA5's skill relative to IMERG Late matches the GPCC/ERA5 explanation, not the satellite one. Both exist; the estimates below use the satellite one.

Where agreement is shared error

Left: how strongly each pair of products' anomalies move together (median over cells). Right: what is left after removing the estimated common signal, with CHIRP v3 (no stations) and IMERG Late treated as sharing mistakes. A residual near zero means the pair agrees only through the truth. A positive residual means the pair shares an error, so their agreement is not independent evidence. Pairs a product was scored against are zero by construction.

How closely each product tracks real rainfall, by region (estimated)

Yardstick: what the independent products have in common (see above).

Median estimated ρ² over cells. These are estimates from the method above, not comparisons with measured rainfall. Blue rows are available in near-real time, which is what a trigger can use.

Map

Click a cell to see every product's ρ² there. The regime layer is an exploratory k-means clustering of each cell's relative skill profile. Its structure is weak (silhouette ), so treat it as a sketch, not as zones.

Loading the map (about 2 MB)…

Which near-real-time product, by region (estimated)

Yardstick: what the independent products have in common.

For each region, the best of the products a trigger can use today (CHIRPS v2 and v3, IMERG Late, ERA5, CPC, CHIRP v3 (no stations)), the runner-up, and how far ahead the best is. A lead under 0.02 is a tie at this sample size.

Robustness

Alternative: GPCC and ERA5 share mistakes. The independent gauges show this shared mistake is real, alongside the satellites' one. The possible reason: both capture the broad weather pattern of a month but not where individual storms fell, so both could miss the same local wet or dry spell. If that were so, GPCC and ERA5 would be over-credited, the satellites under-credited, and IMERG Late would rank first (column 2). The truth is likely between the two columns.

Two more checks. Swapping GPCC for CRU tests the choice of gauge analysis. The naive column uses every triplet and puts IMERG Final in the core, ignoring that Final is calibrated to GPCC. Shared error is then counted as signal, and Final, CHIRPS and the gauges all inflate. That inflation is what this analysis is built to avoid. "–" means the product cannot be scored with that core.

Method and limits

• Data: ds-precip-intercomparison 0.5° monthly cubes, land, 2001–2020, cells every product covers (50°S–50°N). Months where GPCC's climatology is under 10 mm (30 mm for 3-month sums) are left out. Values are standardised anomalies per calendar month, with the same months used for every product in a cell. UDel (ends 2017) and the ASAP blend (= CHIRPS v2 here) are left out. Implausible months are masked as in the report.
• Model: a one-factor model. Each core product's loading comes from the triple-collocation estimates whose three products are independent once CHIRP v3 (no stations) and IMERG Late are treated as sharing mistakes: two estimates each for GPCC and ERA5, one each for CHIRP v3 (no stations) and IMERG Late. Every other product's covariance with the truth is averaged over the core products independent of it.
• Assumptions that can fail: errors independent between families, a linear relation to the truth, stationarity over 20 years. The residual matrix checks the first. Where CHIRP and IMERG share infrared inputs, or ERA5 and gauges share circulation errors, scores are biased. The direction is toward over-crediting the products involved.
• ρ² is about timing, not amount. It says whether a product gets the anomalies right (wet or dry relative to its own normal), which is what most triggers use. It says nothing about bias in totals; see the report's bias section.
• Per-cell estimates are noisy. About 200 months per cell. Read regions, not single pixels: the per-pixel best product is the same in 2001–2010 and 2011–2020 in only of cells.
• The satellite assumption: four products cannot prove which pair shares mistakes; the gauge-density test supports the satellite pair. A fifth independent source would settle it, for example gauge networks not used by GPCC or CHIRPS, or soil-moisture-based rainfall (SM2RAIN).
• Gauge data: AMMA-CATCH, CC-BY 4.0. Galle et al. (2018), AMMA-CATCH a Critical Zone Observatory in West Africa Monitoring a Region in Transition, Vadose Zone Journal 17(1), 18006; AMMA-CATCH (1990), doi:10.17178/AMMA-CATCH.all. Daily, quality-checked values only; data from the backup-bucket gap estimates are excluded. The network's data policy asks that the data providers be offered co-authorship on publications. TAHMO: Van de Giesen, Hut and Selker (2014), WIREs Water 1(4), 341–348, doi:10.1002/wat2.1034; used under TAHMO's research licence, so this page shows regional scores and selection counts only, no station data or locations.
• Code: scripts/multivariate.py, multivariate_scales.py, multivariate_outliers.py, validate_gauges.py, validate_3month.py, tahmo_daily.py and validate_tahmo.py compute the results; which_product_data.py writes this page's data. All in OCHA-DAP/ds-precip-intercomparison; gauge sources and licences in docs/gauges/README.md.