How the RaQuet Viewer Works

The RaQuet viewer is a fully client-side application that renders RaQuet files directly in the browser using HTTP range requests. No server-side processing is required.

Technology Stack

Architecture Overview

Loading Process

1. Parquet Metadata Fetch

When you load a RaQuet file, the viewer first fetches the Parquet footer (last 8 bytes + metadata section) using HTTP range requests:

GET /file.parquet
Range: bytes=-8              # Get footer size
Range: bytes=X-Y             # Get metadata based on footer size

The metadata contains:

• Schema (column names, types)
• Row group information (offsets, sizes, min/max statistics)
• Total row count

2. RaQuet Metadata Extraction

RaQuet stores its metadata in the last row of the file (where block=0). The viewer reads this row to get:

{
  "version": "0.1.0",
  "bounds": [-122.5, 37.5, -122.0, 38.0],
  "minresolution": 10,
  "maxresolution": 14,
  "block_width": 256,
  "block_height": 256,
  "bands": [
    {"name": "band_1", "type": "uint8"},
    {"name": "band_2", "type": "uint8"},
    {"name": "band_3", "type": "uint8"}
  ]
}

3. Row Group Index Building

The viewer builds an in-memory index of row groups with their QUADBIN block ID ranges. This enables efficient tile lookups:

Row Group 0: blocks 5270201491262341119 - 5270201491262406655
Row Group 1: blocks 5270201491262472191 - 5270201491262537727
...

Tile Loading Flow

1. Map Requests Tiles

When you pan/zoom the map, deck.gl's TileLayer calculates which tiles (z/x/y) are visible and requests them.

2. Convert to QUADBIN

Each tile coordinate is converted to a QUADBIN cell ID:

function tileToQuadbin(x, y, z) {
    // Mode 9 (tile mode) header
    const mode = 9n << 59n;
    const resolution = BigInt(z) << 52n;
    // Interleave x,y bits to create spatial index
    let index = 0n;
    for (let i = 0; i < z; i++) {
        index |= ((BigInt(x) >> BigInt(i)) & 1n) << BigInt(2 * i);
        index |= ((BigInt(y) >> BigInt(i)) & 1n) << BigInt(2 * i + 1);
    }
    return mode | resolution | index;
}

3. Find Matching Row Groups

Using the row group index, the viewer identifies which row groups might contain the requested block. This is fast because QUADBIN IDs are sorted, so only row groups with matching min/max ranges need to be checked.

4. Two-Phase Data Fetch

For efficiency, the viewer uses a two-phase approach:

5. Decode and Render

The band data is decoded:

1. Decompress gzip-compressed bytes using pako
2. Convert to appropriate pixel format (uint8, uint16, etc.)
3. Combine bands into RGB(A) canvas image
4. Pass to deck.gl BitmapLayer for WebGL rendering

Row Group Size Impact

Row group size significantly affects HTTP request efficiency. Our testing shows:

Recommendation: Use --row-group-size 8 when creating RaQuet files optimized for web viewing.

Key Optimizations

Request Batching

Tile requests are batched with a 50ms delay to combine multiple visible tiles into fewer operations.

Tile Caching

Loaded tiles are cached in memory to avoid re-fetching when panning back to previously viewed areas.

Row Group Pruning

Using min/max statistics on the block column, the viewer skips row groups that can't contain the requested tile.

Column Projection

Only the columns needed (block ID + band data) are read, not the entire row.

Requirements

Server Requirements

• CORS headers: Access-Control-Allow-Origin: *
• Range requests: Server must support HTTP 206 Partial Content
• Expose headers: Access-Control-Expose-Headers: Content-Length, Content-Range

File Requirements

• Valid RaQuet file with sorted blocks (QUADBIN order)
• Metadata row present (block=0 in last row)
• Standard Parquet compression (Snappy, Gzip, or Zstd)

Limitations

• Memory: Large tiles or many visible tiles increase browser memory usage
• Latency: Multiple HTTP round-trips per tile (metadata + data phases)
• Compression: Some Parquet compression codecs may not be supported by hyparquet-compressors

Source Code

The viewer source code is available in the RaQuet repository. It's a single HTML file with inline JavaScript that can be easily customized or integrated into other applications.