Data Fetchers

A fetcher (forcingkit.fetchers) talks to one external provider: it resolves where the data for a time lives, reads only the subset a request needs, and returns it in a common layout. The dispatcher (forcingkit.dispatcher) chooses among fetchers, falls back when one cannot deliver, and writes the result to the local Zarr cache, so a request is fetched from the provider once.

Atmospheric forcing (/api/v1/atmosphere)

Source

Notes

HRRR

NOAA High-Resolution Rapid Refresh (developed by the NOAA Global Systems Laboratory, run operationally by NCEP), 3 km, from the anonymous AWS S3 bucket noaa-hrrr-bdp-pds; the archive starts 2014-07-30. Surface fields are interpolated to a regular 0.03 degree grid and streamed hour by hour (schema hrrr-atm-v1). Earlier dates use ERA5 through NumericalEarth in the model, outside this service. See Atmospheric Forcing Datasets.

HRRR levels

/api/v1/atmosphere/levels reads the lowest hybrid levels of the same cycles’ native-level file (wrfnatf01) by byte range: U and V (rotated to east and north), vertical velocity (converted from omega with the hydrostatic relation) and temperature every hour, and the levels’ heights and pressures once, from the first hour, since they move only a few metres with the surface pressure. Over open water level 1 is about 10 m above the surface (its wind agrees with the surface delivery’s 10 m wind), level 6 about 430 m and level 14 about 2.8 km. Each CONUS message is fetched whole (1 to 2.4 MB), so 14 levels read about 63 MB an hour: a 48 h window took 3 minutes and made a 7 MB store (schema hrrr-levels-v1).

Surface waves (/api/v1/waves)

Source

Notes

NWPS

NOAA’s Nearshore Wave Prediction System: SWAN run by each coastal Weather Forecast Office on its own computational grids, at 00 and 12 UTC, published on NOMADS as one GRIB2 file per grid and cycle (the analysis and 144 hourly steps; about 26 MB for OKX CG1). One office and grid per request, kept on that grid and cropped to the box (schema nwps-waves-v1). NOMADS keeps about five days; a directory or gs:// archive named by FORCINGKIT_NWPS_ARCHIVE is read first. See Surface Waves (NWPS).

Parent ocean (/api/v1/obc)

Source

Notes

NECOFS

Northeast Coastal Ocean Forecast System, FVCOM GOM7 (UMass Dartmouth SMAST and WHOI), unstructured mesh with 45 sigma layers. Daily history files from 2025-01-01, else a monthly file (the hindcast from 1978, GOM3 mesh to 2016; the monthly GOM7 archive for 2025), then the rolling forecast. The only donor that streams the z-v3 parent store. See NECOFS: Northeast Coastal Ocean Forecast System (FVCOM).

NYOFS

NOAA New York/New Jersey Operational Forecast System (NOAA CO-OPS), POM on a curvilinear grid with 7 sigma levels, NY/NJ Harbor. Ranked first inside its domain. 7-day CO-OPS aggregation, then the nowcast archive (AWS S3 from 2024-11-19, NCEI from 2014). Legacy output layout rather than z-v3. See NYOFS: NOAA New York/New Jersey Operational Forecast System.

DBOFS

NOAA Delaware Bay Operational Forecast System (NOAA CO-OPS), ROMS, about 100 m, domain [-75.875, 37.810, -73.264, 40.206]. 7-day CO-OPS aggregation, then the hourly nowcast archive (AWS S3 from 2024-11-19, NCEI from 2014). Legacy output layout.

HYCOM

HYCOM (1/12 degree, about 9 km, 40 z levels), global, from the HYCOM consortium data server, 1994 to the present. The experiment follows the date: GLBv0.08 expt_53.X (reanalysis) to 2015, a chain of GLBv0.08 analysis experiments to 2018-12-04, GLBy0.08 expt_93.0 to 2024-09-05, then ESPC-D-V02. A window that spans a switch is stitched; if any piece fails, the whole window fails. Tidal only from 2024-09-05, recorded in the store’s tides attribute ("none", "included" or "mixed"). Last-resort fallback; legacy output layout.

Doppio is not a donor yet: forcingkit.fetchers.doppio.volume returns its full-depth subsets on the model’s own grid for comparison with NECOFS, and the dispatcher does not rank it.

TPXO10 tidal harmonics are a roadmap item, not integrated; the pyTMD dependency was removed on 2026-10-05. The include_tides and tidal_model request fields are still accepted and form part of the cache key, but no tide is added to the parent store. A parent that already contains the tide (a HYCOM store with tides = "included") should not have tides added again downstream; see HYCOM: global ocean analyses and reanalysis.

Donor selection

Each parent-ocean fetcher declares a domain box and an approximate resolution. A donor is a candidate when it accepts the request box (NYOFS and DBOFS require the box to lie inside their domain; NECOFS accepts any overlap; HYCOM accepts everything). Candidates are ranked by domain area first, smallest first, then by resolution, on the reasoning that the most specialised system is the best local one:

Donor

Declared domain (lon/lat)

Area (degree squared)

Declared resolution

NYOFS

[-74.475, 40.389, -73.743, 40.940]

0.40

100 m

DBOFS

[-75.875, 37.810, -73.264, 40.206]

6.3

100 m

NECOFS

[-77.0, 35.0, -65.0, 46.0]

132

200 m

HYCOM

global

64800

9 km

The dispatcher tries the candidates in order. A donor that returns nothing or raises is logged and skipped unless the request sets allow_donor_fallback: false, in which case the first failure is an error. A donor that streams (NECOFS) either completes its store or fails; it never publishes a shortened one.

Observations

These serve validation rather than forcing:

  • CO-OPS water level (/api/v1/tide), from the NOAA CO-OPS data API, with a datum fallback to MSL where a station has no NAVD88 datum. Requests go through insitukit, which keeps them at least 1 s apart, splits windows longer than the API’s 31-day limit, and retries server errors and throttling; end_time is inclusive, as in the API. Each reading carries CO-OPS’s quality level (q: p preliminary, v verified) and metadata.quality sums up the window. Since 0.17.0 a verified window is cached for good and one still holding preliminary readings for a day, because CO-OPS replaces preliminary readings once it has reviewed them. Current predictions (/api/v1/currents) and the current-station list (/api/v1/currents/stations) use the same client; cached predictions are fetched again after 30 days (since 0.17.1), so harmonic constants NOAA re-publishes reach them.

  • NDBC buoy meteorology and ADCP profiles (/api/v1/ndbc), from the NOAA National Data Buoy Center, through insitukit’s NDBC client: each month from the monthly, yearly or real-time file, each file downloaded once, and each column’s own missing-value sentinel read as missing (since 0.16.0; earlier releases also dropped real readings equal to another column’s sentinel, such as a 999 hPa low or a 99-degree wind). A window is cached only when none of it came from the real-time files, whose readings have not yet been through NDBC’s quality control (since 0.17.0).

  • UConn ERDDAP water-column profiles (/api/v1/telemetry/station and /bbox), from the University of Connecticut Department of Marine Sciences ERDDAP server. The station list holds four Long Island Sound moorings, so a box elsewhere contains none and /bbox returns empty data. Setting "allow_proxy_stations": true substitutes EXRX and WLIS instead, for testing; every station carries within_bbox and distance_km (0 inside the box), and the response’s fallback is true when any lies outside it. Each dataset read has a 10 s connect and 60 s read timeout (FORCINGKIT_ERDDAP_CONNECT_TIMEOUT, FORCINGKIT_ERDDAP_READ_TIMEOUT). A host that fails to connect, or a dataset that times out or errors, is skipped for 300 s (FORCINGKIT_ERDDAP_FAILURE_TTL, 0 to turn off), and the response’s skipped lists each dataset left out, with the reason. A station’s three datasets are read at once, through insitukit’s ERDDAP client (one attempt each, starting at least 0.2 s apart).

  • HF radar surface currents (library only: forcingkit.fetchers.hfradar), the hourly total vectors of the NOAA IOOS HF Radar National Network (HFRNet) for the US East and Gulf Coast at 1 km and 2 km, from NDBC’s THREDDS server over OPeNDAP (since 0.19.0). They represent roughly the upper 0.5 m, for scoring a model’s surface layer. fetch returns a box and window as stored on the server (16-bit integers with scale 0.01, so nothing is rounded twice). The server keeps only about 92 days, so archive keeps a box one NetCDF-4 file per UTC day under FORCINGKIT_CACHE_DIR/hfradar/usegc_<grid>/<box>/<YYYY>/<MM>/, written once the server has published past the day’s end and six hours have passed, and never rewritten; it takes any store with has and put (a create-only bucket, for example). load reads a box and window back in m/s from any archived box that contains it, and the other days from the server while it holds them. The hfradar directory is an archive, not a cache: do not purge it with the Zarr stores. FORCINGKIT_HFRADAR_ARCHIVE puts it elsewhere (since 0.20.0). Coverage is uneven: in western Long Island Sound about 2 % of the 1 km cells in the box 40.85-41.3 N, 73.8-71.8 W have a solution in a covered hour, and whole weeks have none.

    Older days come from NOAA NCEI, which archives the same hourly totals (the same variables, stored the same way, on the same grids) as one file per hour in monthly accessions, from about two months back to the network’s start. archive_ncei finds a window’s accessions through NCEI’s metadata search, reads each hour’s file (about 0.8 MB at 1 km for the whole coast), cuts the box out and writes the same daily files, so load reads days from either source alike; source_dataset names the accession. Days already kept are skipped, and a day is written with the hours NCEI holds (since 0.20.0). The month list and folders found are kept in ncei_accessions.json beside the archive, so a rerun skips NCEI’s slow search (since 0.20.1). From the command line:

    python -m forcingkit.fetchers.hfradar ncei --box -73.8 40.85 -71.8 41.3 \
        --start 2025-01-01 --end 2026-09-01 [--grids 1km 2km] [--root DIR] [--workers 4]
    python -m forcingkit.fetchers.hfradar ndbc --box -73.8 40.85 -71.8 41.3 --start all [--root DIR]
    

NOAA OFS archive locations

As checked on 2026-10-06, NYOFS and DBOFS output is available from three places:

Location

Span

Layout

CO-OPS THREDDS FMRC aggregation

Rolling 7 days

One virtual OPeNDAP dataset per system (opendap.co-ops.nos.noaa.gov/thredds/dodsC/<OFS>/fmrc/Aggregated_7_day_<OFS>_Fields_Forecast_best.ncd).

NOAA NCEI THREDDS (model-nyofs-files, model-dbofs-files)

2014 to the present month, with occasional missing days (NYOFS 2024-06 has 25 of 30)

One directory per month, {yyyy}/{mm}/, with every file of the month in it. File names changed with the 2024-09-09 cycles (from nos.<ofs>.fields.<n|f><hhh>.<yyyymmdd>.t<cc>z.nc and nos.nyofs.fields.<nowcast|forecast>.<yyyymmdd>.t<cc>z.nc to <ofs>.t<cc>z.<yyyymmdd>.fields.<...>.nc).

AWS S3 noaa-nos-ofs-pds (NOAA Open Data Dissemination)

2024 to the present, with gaps before 2024-11-19

From 2024-11-19, one directory per day, <ofs>/netcdf/{yyyy}/{mm}/{dd}/; the first day holds only its last cycle. Before that, one flat directory per month, <ofs>/netcdf/{yyyymm}/, with a mix of file names (NYOFS 2024-07 uses the older names, DBOFS 2024-07 the newer ones); several of those months are partial and some are absent (NYOFS lacks 2024-01 to 2024-03, 2024-05 and 2024-06; DBOFS lacks 2024-02 and starts 2024-01 on the 29th).

Note

An earlier version of this page said that NCEI stopped archiving at the end of November 2023, that AWS began in January 2024, and that December 2023 was therefore missing. None of that holds: NCEI has every day of December 2023 for both NYOFS and DBOFS (124 NYOFS and 744 DBOFS field files) and has continued archiving through 2026, while the AWS archive is the one with gaps.

The NYOFS and DBOFS fetchers read nowcast files only, so a window is a chain of analyses rather than forecasts. For each file they try AWS (per-day layout) first when the date is 2024-11-19 or later, then NCEI, which covers every date; the flat AWS month directories are not used. A window with a file missing from both is not served by that donor, and the dispatcher falls back. DBOFS publishes one file per hour: file n001 to n006 of cycle t<cc>z (00, 06, 12, 18 UTC) hold hours cc - 5 to cc. NYOFS publishes one file per cycle (05, 11, 17, 23 UTC) holding the same six hours.

Both fetchers were corrected on 2026-10-06. Before that, every NYOFS archive date failed (see NYOFS: NOAA New York/New Jersey Operational Forecast System), and DBOFS sent 2024 dates before 2024-11-19 to an AWS layout that does not hold them, took some hours from the wrong file, joined NCEI files along a time dimension they do not have (theirs is ocean_time), and left the ROMS fill value (1e37) in land cells read through NCEI.

For the AWS layout, see the NOAA NODD OFS documentation and the Registry of Open Data entry.

Common processing

  • Hourly time axis. Non-streaming donors are resampled to a strict hourly index by linear interpolation. Streaming donors deliver hourly records directly. HYCOM steps are 3 hours apart with occasional gaps of up to 51 hours, which this bridges with a straight line; a store does not yet record the longest gap it filled (HYCOM: global ocean analyses and reanalysis).

  • Float32 fields, full-precision coordinates. Data variables are cast to Float32 for the model; lat, lon, z and z_face keep full precision, since Float32 longitudes near -74 resolve only about 8e-6 degrees.

  • Fill values. Missing data is written as -9999 (integers) or -9999.0 (floats) in the Zarr encoding, and as NaN in memory.

  • Time in seconds. z-v3 stores encode time as seconds since their first record, which the model’s clock uses directly.