Data Fetchers

The Data Fetchers module contains implementations for communicating with external data providers. We use tiered fallback logic and cache everything locally to minimize bandwidth usage and guarantee reproducible simulations.

Available Sources

Atmospheric Forcing: - HRRR: High-Resolution Rapid Refresh (AWS S3, noaa-hrrr-bdp-pds) - 3 km surface fields, served by /api/v1/atmosphere on a regular 0.03 degree grid. Archive spans 2014-07-30 to present. No authentication required. See Atmospheric Forcing Datasets; earlier dates use ERA5 through NumericalEarth in the model.

Parent Ocean (/api/v1/obc): - NYOFS (Primary for NY/NJ Harbor): NOAA New York/New Jersey Operational Forecast System (70–150m resolution, Princeton Ocean Model/POM). Historical data >= 2024 sourced from noaa-nos-ofs-pds AWS S3 bucket. - DBOFS (Primary for Mid-Atlantic Bight): NOAA Delaware Bay Operational Forecast System (ROMS, ~100m resolution, domain [-76.5, 37.5, -73.0, 40.0]). Historical data >= 2024 sourced from noaa-nos-ofs-pds AWS S3 bucket. - NECOFS: New England Coastal and Ocean Forecasting System (FVCOM, 200m). - HYCOM: Global 9km ocean state. v2.0 adds support for continuous 60-day historical hindcasts by stitching experiment series (e.g., expt_53.X and expt_93.0). - TPXO10: Global Tidal Harmonics (OTIS format) via pyTMD v3. Reserved as a future roadmap item given usage rights; not yet integrated.

Note

NOAA Big Data Program (NODD) migrated the historic OFS archives from NCEI THREDDS to AWS S3 starting in 2024. The fetchers for NYOFS and DBOFS automatically retrieve pre-2024 data via NCEI OPeNDAP and 2024+ data via AWS S3 utilizing h5netcdf.

Known Data Gap: There is a known missing data gap for December 2023 (12/2023). During the infrastructure migration, NCEI stopped archiving at the end of November 2023, while the new AWS S3 pipeline did not begin archiving until January 2024. Hindcasts attempting to span this window through NYOFS or DBOFS may fail.

For details on the AWS S3 structure, see the NOAA NODD OFS documentation.

V2.0 Processing Features

  • Temporal Alignment: Linear interpolation of disparate sources (where applicable) to a strictly hourly shared temporal index, simplifying timestep integration for the end client.