Reading only what you need: bounded scientific data reads and 134-million-voxel volumes on an 8 GB GPU

The question

Simulation data is large and awkward. One CFD case can hold hundreds of millions of cells over thousands of time steps, and a campaign holds hundreds of cases. Training reads short time windows. An engineer looking at a slice wants one plane. A dashboard wants twelve rows of a catalog. Most tooling serves all of these requests by reading whole files, which is why a team's first physics-AI project often stalls on I/O and memory rather than on modelling.

Frontier's answer is a data layer built around bounded requests: you ask for a field, a region and a time, you get back that much, and the cost is known before the read starts. The open part of it is ArrayRoute, an Apache-2.0 library that plans selections over chunked scientific arrays and hands the actual reads to native engines (Zarr-Python, TensorStore, Zarrista, HDF5). This page collects measured results from October 2026.

The data

• ERA5 reanalysis: 192 regional frames of temperature, wind, humidity and geopotential, shape 192 × 13 × 33 × 81 × 5, 133 MB of float32. Every acquired frame was hash-checked.
• Synthetic fields with known content, used where an exact byte-for-byte reference matters: a 6 GiB sharded Zarr store, and a 512³ grid with a scalar and a vector field (2.15 GB of float32).
• A one-million-sample catalog fixture for the app's dataset views.

What Frontier measured

1. Layout decides most of the read cost

On the ERA5 tensor, one weather station's temperature history over 192 frames is only 9,984 bytes. In a frame-chunked layout, answering it still means decoding all 133,436,160 bytes. ArrayRoute materialized a station-chunked layout from the same data and verified it byte for byte. The read then touched 12 chunks instead of 192, and its decoded-byte bound fell to exactly the 9,984 bytes requested.

The second row is the point: no single layout wins. A training window is fastest on frame chunks, and a station read is 85× faster on station chunks. Frontier therefore treats layout as a workload decision with measured trade-offs. It does not switch layouts automatically, and the production default stayed unchanged.

On a synthetic 3D vector field, bricked and sharded storage read through TensorStore took a small region of interest from 23.4 ms to 0.51 ms, and a four-step training window from 25.4 ms to 0.59 ms. All 378 timed outputs were hashed and compared with a reference reader. This speed-up comes from the layout and the native reader, not from a new algorithm in ArrayRoute. ArrayRoute's job is to keep selections consistent, buffers bounded and costs measurable.

2. Planning without reading

Planning is cheap and bounded. ArrayRoute planned 10,000 selections over a 1 PiB logical array in 0.048 seconds, and no payload was read. This measures planning only, not petabyte throughput.

3. Million-sample datasets without loading the manifest

The dataset card in the Frontier app used to fetch the complete manifest to show at most 12 rows. On a one-million-sample fixture, each of 15 fresh-process trials checked the exact IDs returned:

4. One fill, not four

When four processes ask for the same uncached tile, for example several viewers or validation jobs on one shared sample, ArrayRoute's disk cache now lets one of them fill it while the others wait. In every trial, field bytes transferred over HTTP fell by exactly 75% (693.8 MB to 173.5 MB), and origin fills fell from 32 to 8. All 32 returned arrays matched the fixture hashes.

5. 10⁸-node volumes in Omniverse on a laptop GPU

For 3D visualization, Frontier exports grid fields as NanoVDB files that NVIDIA Omniverse Kit-CAE reads directly. A 512³ grid (134 million nodes) was benchmarked headless on an RTX 3080 Laptop GPU with 8 GB:

The 8-bit scalar file is 27% of its float size: 150 MB for 134 M voxels, decoded on the GPU when an operator first asks for it. NVIDIA IndeX volume-rendered the same grid using 3.1 GB of GPU memory for the whole app. A separate change to the tile pager's visibility test cut its working memory at one million tiles from 240 MB to 4.7 MB, and made it 2.4× faster.

Limits and caveats

These are engineering measurements, not marketing benchmarks, and the source documents state their scope:

• Most timings come from a shared developer Mac with uncontrolled OS caches. They show relative access-pattern costs, not production latency guarantees.
• Nothing here claims TB- or PB-scale throughput, or superiority over other products. Full-TiB verification and multi-node scaling are still open work.
• The 75% byte reduction applies to overlapping readers only. For disjoint training partitions the controls showed no benefit, as expected.
• The NanoVDB vector field (Vec3f) was not quantized and dominates the file totals.

ERA5 data: Copernicus Climate Change Service, ECMWF. Reanalysis, used here as input data only.

Try it on your data

Is your simulation data too large to move or to load whole? Join the waitlist or talk to us about profiling your own access patterns.