CHIRPS-GEFS is the Climate Hazards Center’s 16-day daily precipitation forecast, published as global Cloud-Optimized GeoTIFFs (COGs). The forecast part of the trigger takes its rainfall from it. This chapter describes src/datasources/chirps_gefs.py, which downloads the forecasts for Cuba, stores them in blob storage and summarises them over the country. The next chapter goes through the processing for a single issue date.
The rasters are clipped in two stages. On download, each global COG (about 7,200 × 3,600 pixels) is opened over HTTP and cut to Cuba’s bounding box (about 218 × 70 pixels) before it is written to blob storage. Because COGs can be read by window, only that box is transferred:
# ChirpsGefsDownloader.download_single_file
with rxr.open_rasterio(source_url) as da_global:
da_aoi = da_global.rio.clip_box(*self.config.geometry.total_bounds)
stratus.upload_cog_to_blob(da_aoi, blob_path)When the stored rasters are processed, they are clipped again to Cuba’s actual boundary, which removes sea pixels inside the box:
# ChirpsGefsProcessor.process_spatial_mean
da_clipped = da_loaded.rio.clip(self.config.geometry.geometry, all_touched=True)
spatial_mean = da_clipped.mean(dim=["x", "y"]).valuesThe module has five classes:
ChirpsGefsConfig: the region geometry, date range, number of lead times and blob location.ChirpsGefsDownloader: builds source URLs and blob paths, and downloads, clips and uploads files.ChirpsGefsLoader: loads stored rasters, one at a time or as a lazily loaded (dask) time series.ChirpsGefsProcessor: clips to the region geometry and computes spatial means.ChirpsGefsManager: runs downloading and processing together.import pandas as pd
import geopandas as gpd
import datetime
import xarray as xr
import rioxarray as rxr
from pathlib import Path
import ocha_stratus as stratus
# Import our CHIRPS GEFS module components
from src.datasources.chirps_gefs import (
ChirpsGefsManager,
ChirpsGefsConfig,
ChirpsGefsDownloader,
ChirpsGefsLoader,
ChirpsGefsProcessor,
)
from src.datasources import codab
from src.constants import *The national boundary from the CODAB, and its bounding box, which sets the download window:
# Load Cuba administrative boundaries for demonstration
cuba_gdf = codab.load_codab_from_blob()
cuba_gdf = cuba_gdf[cuba_gdf["ADM0_PCODE"] == "CU"]
cuba_gdf = cuba_gdf[["ADM0_ES", "ADM0_PCODE", "geometry"]].copy()
cuba_gdf.head()| ADM0_ES | ADM0_PCODE | geometry | |
|---|---|---|---|
| 0 | Cuba | CU | MULTIPOLYGON (((-83.06403 21.55069, -83.06403 ... |
# Calculate Cuba's spatial footprint
cuba_bounds = cuba_gdf.total_bounds
print(f"Cuba bounding box: {cuba_bounds}")
print(f"Longitude: {cuba_bounds[0]:.2f}° to {cuba_bounds[2]:.2f}°")
print(f"Latitude: {cuba_bounds[1]:.2f}° to {cuba_bounds[3]:.2f}°")Cuba bounding box: [-84.95236206 19.8259716 -74.13118744 23.27763939]
Longitude: -84.95° to -74.13°
Latitude: 19.83° to 23.28°A short configuration: five issue dates with five lead times each.
# Create a CHIRPS GEFS configuration for demonstration
config = ChirpsGefsConfig(
geometry=cuba_gdf,
region_name="cuba_cog_demo",
start_date="2024-12-01",
end_date="2024-12-05",
leadtime_days=5, # Short demo period
verbose=True
)
print(f"Region: {config.region_name}")
print(f"Date range: {config.start_date} to {config.end_date}")
print(f"Leadtime: {config.leadtime_days} days")
# Calculate total number of files: days × leadtimes
date_range = pd.date_range(config.start_date, config.end_date, freq='D')
total_files = len(date_range) * config.leadtime_days
print(f"Total forecasts: {total_files} files")Region: cuba_cog_demo
Date range: 2024-12-01 to 2024-12-05
Leadtime: 5 days
Total forecasts: 25 files# Initialize each component to show their specific functions
downloader = ChirpsGefsDownloader(config)
loader = ChirpsGefsLoader(config)
processor = ChirpsGefsProcessor(config)Each file is one issue date and one valid date. The source URL and blob path follow from the pair:
# Example: Forecast issued on Dec 10th for Dec 15th (5-day leadtime)
issue_date = pd.Timestamp("2024-12-10")
valid_date = pd.Timestamp("2024-12-15")
url = downloader.generate_url(issue_date, valid_date)
filename = downloader.generate_filename(issue_date, valid_date)
blob_path = downloader.generate_blob_path(filename)
print(f"Issue date: {issue_date.date()}")
print(f"Valid date: {valid_date.date()} (5-day leadtime)")
print(f"Source URL: {url}")
print()
print("Local Storage Structure:")
print(f"Filename: {filename}")
print(f"Blob path: {blob_path}")
print()
print("Date Pair Examples:")
# Generate a few examples to show the pattern
date_range = pd.date_range(config.start_date, config.end_date, freq='D')
examples_shown = 0
for issue_date in date_range[:2]: # Show first 2 issue dates
for leadtime in range(min(3, config.leadtime_days)): # Show first 3 leadtimes
valid_date = issue_date + pd.Timedelta(days=leadtime)
examples_shown += 1
print(f" {examples_shown}. Issue: {issue_date.date()} → Valid: {valid_date.date()}")
if total_files > examples_shown:
print(f" ... ({total_files - examples_shown} more combinations)")Issue date: 2024-12-10
Valid date: 2024-12-15 (5-day leadtime)
Source URL: https://data.chc.ucsb.edu/products/EWX/data/forecasts/CHIRPS-GEFS_precip_v12/daily_16day/2024/12/10/data.2024.1215.tif
Local Storage Structure:
Filename: chirps-gefs-cuba_cog_demo_issued-2024-12-10_valid-2024-12-15.tif
Blob path: ds-aa-cub-hurricanes/raw/chirps_gefs/cuba_cog_demo/chirps-gefs-cuba_cog_demo_issued-2024-12-10_valid-2024-12-15.tif
Date Pair Examples:
1. Issue: 2024-12-01 → Valid: 2024-12-01
2. Issue: 2024-12-01 → Valid: 2024-12-02
3. Issue: 2024-12-01 → Valid: 2024-12-03
4. Issue: 2024-12-02 → Valid: 2024-12-02
5. Issue: 2024-12-02 → Valid: 2024-12-03
6. Issue: 2024-12-02 → Valid: 2024-12-04
... (19 more combinations)The cells below are reference only and are not run when the book renders. They download from the CHC server and write to blob storage.
# Initialize downloader
downloader = ChirpsGefsDownloader(config)
download_stats = downloader.download_date_range(
config.start_date,
config.end_date
)
print("Download Results:")
print(f"Successful: {download_stats['success']}")
print(f"Failed: {download_stats['failed']}")
print(f"Total attempted: {download_stats['total']}")# Load and examine one of the downloaded rasters
loader = ChirpsGefsLoader(config)
# Option 1: Load a single specific forecast
da_single = loader.load_raster(
issue_date=pd.Timestamp("2024-12-01"),
valid_date=pd.Timestamp("2024-12-03"),
)
print("Single COG Properties:")
print(f"Shape: {da_single.shape}")
print(f"CRS: {da_single.rio.crs}")
print(f"Resolution: {da_single.rio.resolution()}")
print(f"Bounds: {da_single.rio.bounds()}")
print(f"Data range: {da_single.min().values:.2f} to {da_single.max().values:.2f} mm/day")
print(f"Data size: {da_single.nbytes / 1024:.1f} KB")# Load a time series lazily (dask): data is read from blob only when accessed
ds_timeseries = loader.load_raster_time_series(
start_date = "2024-12-01",
end_date = "2024-12-02",
chunks = {"x": -1, "y": -1}, # one chunk per raster; fine for a small region
lazy = True
)
print("\nTime Series Dataset Properties (Lazy Loading):")
print(f"Dataset dimensions: {dict(ds_timeseries.dims)}")
print(f"Issue dates: {len(ds_timeseries.issue_date)} dates")
print(f"Valid dates: {len(ds_timeseries.valid_date)} dates")
print(f"Leadtimes: {len(ds_timeseries.leadtime)} leadtimes")
if "precipitation" in ds_timeseries.data_vars:
precip = ds_timeseries.precipitation
print(f"Precipitation range: {precip.min().values:.2f} to {precip.max().values:.2f} mm/day")
print(f"Processing mode: {ds_timeseries.attrs.get('processing_mode', 'unknown')}")
# Show some sample data structure
print(f"\nSample coordinate values:")
print(f" Issue dates: {[str(d.date()) for d in ds_timeseries.issue_date.values[:3]]}")
print(f" Valid dates: {[str(d.date()) for d in ds_timeseries.valid_date.values[:3]]}")
print(f" Leadtimes: {ds_timeseries.leadtime.values[:3]}")ChirpsGefsManager wraps the same steps:
# Create manager with custom configuration
manager = ChirpsGefsManager(
geometry=cuba_gdf,
region_name="cuba_production",
start_date="2024-12-09",
end_date="2024-12-10",
leadtime_days=16, # Full 16-day forecasts
verbose=True,
)
# Run the steps separately
print("Downloading data...")
download_stats = manager.download_only(recent_only=True, days_back=30)
print(f" Downloaded {download_stats['success']}/{download_stats['total']} files")
print("Processing data...")
processed_data = manager.process_only(recent_only=True)
print(f" Processed {len(processed_data)} forecast records")
# Or both in one call
print("Running complete pipeline...")
pipeline_results = manager.run_full_pipeline(download_recent_only=True, days_back=30)projects/ds-aa-cub-hurricanes/
├── raw/chirps_gefs/cuba_test/
│ ├── chirps-gefs-cuba_test_issued-2024-12-10_valid-2024-12-10.tif
│ ├── chirps-gefs-cuba_test_issued-2024-12-10_valid-2024-12-11.tif
│ └── chirps-gefs-cuba_test_issued-2024-12-10_valid-2024-12-12.tif
└── processed/chirps_gefs/cuba_test/
└── cuba_test_chirps_gefs_mean_daily_2024_2024.parquet