Data Fetchers¶
A fetcher (forcingkit.fetchers) talks to one external provider: it resolves where the data
for a time lives, reads only the subset a request needs, and returns it in a common layout. The
dispatcher (forcingkit.dispatcher) chooses among fetchers, falls back when one cannot deliver,
and writes the result to the local Zarr cache, so a request is fetched from the provider once.
Atmospheric forcing (/api/v1/atmosphere)¶
Source |
Notes |
|---|---|
HRRR |
NOAA High-Resolution Rapid Refresh (developed by
the NOAA Global Systems Laboratory, run operationally by
NCEP), 3 km, from the anonymous AWS S3 bucket
noaa-hrrr-bdp-pds; the archive starts
2014-07-30. Surface fields are interpolated to a regular 0.03 degree grid and streamed hour
by hour (schema |
HRRR levels |
|
Surface waves (/api/v1/waves)¶
Source |
Notes |
|---|---|
NWPS |
NOAA’s Nearshore Wave Prediction System: SWAN run by each coastal Weather Forecast Office on
its own computational grids, at 00 and 12 UTC, published on NOMADS as one GRIB2 file per grid
and cycle (the analysis and 144 hourly steps; about 26 MB for OKX CG1). One office and grid
per request, kept on that grid and cropped to the box (schema |
Parent ocean (/api/v1/obc)¶
Source |
Notes |
|---|---|
NECOFS |
Northeast Coastal Ocean Forecast System, FVCOM GOM7 (UMass Dartmouth SMAST and WHOI), unstructured mesh
with 45 sigma layers. Daily history files from 2025-01-01, else a monthly file (the hindcast
from 1978, GOM3 mesh to 2016; the monthly GOM7 archive for 2025), then the rolling forecast.
The only donor that streams the |
NYOFS |
NOAA New York/New Jersey Operational Forecast System (NOAA CO-OPS), POM on a curvilinear grid with 7
sigma levels, NY/NJ Harbor. Ranked first inside its domain. 7-day CO-OPS aggregation,
then the nowcast archive (AWS S3 from 2024-11-19, NCEI from 2014). Legacy output layout
rather than |
DBOFS |
NOAA Delaware Bay Operational Forecast System (NOAA CO-OPS), ROMS, about 100 m, domain [-75.875, 37.810, -73.264, 40.206]. 7-day CO-OPS aggregation, then the hourly nowcast archive (AWS S3 from 2024-11-19, NCEI from 2014). Legacy output layout. |
HYCOM (1/12 degree, about 9 km, 40 z levels), global, from
the HYCOM consortium data server, 1994 to
the present. The experiment follows the date: GLBv0.08 |
Doppio is not a donor yet: forcingkit.fetchers.doppio.volume returns its
full-depth subsets on the model’s own grid for comparison with NECOFS, and the dispatcher does
not rank it.
TPXO10 tidal harmonics are a roadmap item, not integrated; the
pyTMD dependency was removed on
2026-10-05. The include_tides and tidal_model request fields
are still accepted and form part of the cache key, but no tide is added to the parent store.
A parent that already contains the tide (a HYCOM store with tides = "included") should not
have tides added again downstream; see HYCOM: global ocean analyses and reanalysis.
Donor selection¶
Each parent-ocean fetcher declares a domain box and an approximate resolution. A donor is a candidate when it accepts the request box (NYOFS and DBOFS require the box to lie inside their domain; NECOFS accepts any overlap; HYCOM accepts everything). Candidates are ranked by domain area first, smallest first, then by resolution, on the reasoning that the most specialised system is the best local one:
Donor |
Declared domain (lon/lat) |
Area (degree squared) |
Declared resolution |
|---|---|---|---|
NYOFS |
[-74.475, 40.389, -73.743, 40.940] |
0.40 |
100 m |
DBOFS |
[-75.875, 37.810, -73.264, 40.206] |
6.3 |
100 m |
NECOFS |
[-77.0, 35.0, -65.0, 46.0] |
132 |
200 m |
HYCOM |
global |
64800 |
9 km |
The dispatcher tries the candidates in order. A donor that returns nothing or raises is logged
and skipped unless the request sets allow_donor_fallback: false, in which case the first
failure is an error. A donor that streams (NECOFS) either completes its store or fails; it never
publishes a shortened one.
Observations¶
These serve validation rather than forcing:
CO-OPS water level (
/api/v1/tide), from the NOAA CO-OPS data API, with a datum fallback to MSL where a station has no NAVD88 datum. Requests go through insitukit, which keeps them at least 1 s apart, splits windows longer than the API’s 31-day limit, and retries server errors and throttling;end_timeis inclusive, as in the API. Each reading carries CO-OPS’s quality level (q:ppreliminary,vverified) andmetadata.qualitysums up the window. Since 0.17.0 a verified window is cached for good and one still holding preliminary readings for a day, because CO-OPS replaces preliminary readings once it has reviewed them. Current predictions (/api/v1/currents) and the current-station list (/api/v1/currents/stations) use the same client; cached predictions are fetched again after 30 days (since 0.17.1), so harmonic constants NOAA re-publishes reach them.NDBC buoy meteorology and ADCP profiles (
/api/v1/ndbc), from the NOAA National Data Buoy Center, through insitukit’s NDBC client: each month from the monthly, yearly or real-time file, each file downloaded once, and each column’s own missing-value sentinel read as missing (since 0.16.0; earlier releases also dropped real readings equal to another column’s sentinel, such as a 999 hPa low or a 99-degree wind). A window is cached only when none of it came from the real-time files, whose readings have not yet been through NDBC’s quality control (since 0.17.0).UConn ERDDAP water-column profiles (
/api/v1/telemetry/stationand/bbox), from the University of Connecticut Department of Marine Sciences ERDDAP server. The station list holds four Long Island Sound moorings, so a box elsewhere contains none and/bboxreturns emptydata. Setting"allow_proxy_stations": truesubstitutes EXRX and WLIS instead, for testing; every station carrieswithin_bboxanddistance_km(0 inside the box), and the response’sfallbackis true when any lies outside it. Each dataset read has a 10 s connect and 60 s read timeout (FORCINGKIT_ERDDAP_CONNECT_TIMEOUT,FORCINGKIT_ERDDAP_READ_TIMEOUT). A host that fails to connect, or a dataset that times out or errors, is skipped for 300 s (FORCINGKIT_ERDDAP_FAILURE_TTL, 0 to turn off), and the response’sskippedlists each dataset left out, with the reason. A station’s three datasets are read at once, through insitukit’s ERDDAP client (one attempt each, starting at least 0.2 s apart).HF radar surface currents (library only:
forcingkit.fetchers.hfradar), the hourly total vectors of the NOAA IOOS HF Radar National Network (HFRNet) for the US East and Gulf Coast at 1 km and 2 km, from NDBC’s THREDDS server over OPeNDAP (since 0.19.0). They represent roughly the upper 0.5 m, for scoring a model’s surface layer.fetchreturns a box and window as stored on the server (16-bit integers with scale 0.01, so nothing is rounded twice). The server keeps only about 92 days, soarchivekeeps a box one NetCDF-4 file per UTC day underFORCINGKIT_CACHE_DIR/hfradar/usegc_<grid>/<box>/<YYYY>/<MM>/, written once the server has published past the day’s end and six hours have passed, and never rewritten; it takes any store withhasandput(a create-only bucket, for example).loadreads a box and window back in m/s from any archived box that contains it, and the other days from the server while it holds them. Thehfradardirectory is an archive, not a cache: do not purge it with the Zarr stores.FORCINGKIT_HFRADAR_ARCHIVEputs it elsewhere (since 0.20.0). Coverage is uneven: in western Long Island Sound about 2 % of the 1 km cells in the box 40.85-41.3 N, 73.8-71.8 W have a solution in a covered hour, and whole weeks have none.Older days come from NOAA NCEI, which archives the same hourly totals (the same variables, stored the same way, on the same grids) as one file per hour in monthly accessions, from about two months back to the network’s start.
archive_nceifinds a window’s accessions through NCEI’s metadata search, reads each hour’s file (about 0.8 MB at 1 km for the whole coast), cuts the box out and writes the same daily files, soloadreads days from either source alike;source_datasetnames the accession. Days already kept are skipped, and a day is written with the hours NCEI holds (since 0.20.0). The month list and folders found are kept inncei_accessions.jsonbeside the archive, so a rerun skips NCEI’s slow search (since 0.20.1). From the command line:python -m forcingkit.fetchers.hfradar ncei --box -73.8 40.85 -71.8 41.3 \ --start 2025-01-01 --end 2026-09-01 [--grids 1km 2km] [--root DIR] [--workers 4] python -m forcingkit.fetchers.hfradar ndbc --box -73.8 40.85 -71.8 41.3 --start all [--root DIR]
NOAA OFS archive locations¶
As checked on 2026-10-06, NYOFS and DBOFS output is available from three places:
Location |
Span |
Layout |
|---|---|---|
CO-OPS THREDDS FMRC aggregation |
Rolling 7 days |
One virtual OPeNDAP dataset per system
( |
NOAA NCEI THREDDS (model-nyofs-files,
|
2014 to the present month, with occasional missing days (NYOFS 2024-06 has 25 of 30) |
One directory per month, |
2024 to the present, with gaps before 2024-11-19 |
From 2024-11-19, one directory per day, |
Note
An earlier version of this page said that NCEI stopped archiving at the end of November 2023, that AWS began in January 2024, and that December 2023 was therefore missing. None of that holds: NCEI has every day of December 2023 for both NYOFS and DBOFS (124 NYOFS and 744 DBOFS field files) and has continued archiving through 2026, while the AWS archive is the one with gaps.
The NYOFS and DBOFS fetchers read nowcast files only, so a window is a chain of analyses
rather than forecasts. For each file they try AWS (per-day layout) first when the date is
2024-11-19 or later, then NCEI, which covers every date; the flat AWS month directories are
not used. A window with a file missing from both is not served by that donor, and the
dispatcher falls back. DBOFS publishes one file per hour: file n001 to n006 of cycle
t<cc>z (00, 06, 12, 18 UTC) hold hours cc - 5 to cc. NYOFS publishes one file per
cycle (05, 11, 17, 23 UTC) holding the same six hours.
Both fetchers were corrected on 2026-10-06. Before that, every NYOFS archive date failed (see
NYOFS: NOAA New York/New Jersey Operational Forecast System), and DBOFS sent 2024 dates before 2024-11-19 to an AWS layout that does not
hold them, took some hours from the wrong file, joined NCEI files along a time dimension
they do not have (theirs is ocean_time), and left the ROMS fill value (1e37) in land
cells read through NCEI.
For the AWS layout, see the NOAA NODD OFS documentation and the Registry of Open Data entry.
Common processing¶
Hourly time axis. Non-streaming donors are resampled to a strict hourly index by linear interpolation. Streaming donors deliver hourly records directly. HYCOM steps are 3 hours apart with occasional gaps of up to 51 hours, which this bridges with a straight line; a store does not yet record the longest gap it filled (HYCOM: global ocean analyses and reanalysis).
Float32 fields, full-precision coordinates. Data variables are cast to Float32 for the model;
lat,lon,zandz_facekeep full precision, since Float32 longitudes near -74 resolve only about 8e-6 degrees.Fill values. Missing data is written as
-9999(integers) or-9999.0(floats) in the Zarr encoding, and as NaN in memory.Time in seconds.
z-v3stores encode time as seconds since their first record, which the model’s clock uses directly.