Cloudini · wire format version 6 · default · PR #150
V6 wire format: changes from V5
V6 changes how quantized FLOAT32 fields are coded inside each 32,768-point chunk. x, y and z are coded as three residual streams against a predictor selected per chunk. Points without a return are excluded by a validity bitmap. Every other FLOAT32 field with a resolution gets one residual stream. The V5 adaptive integer sections are unchanged. Encoders write V6 only when EncodingInfo::version = 6.
Median compressed size (ZSTD), % of raw, 13 datasets
Size vs V5 (ZSTD)−7.8 to −38.6%smaller on 13 of 13 datasets; median −13.1%
Instructions vs V5, steady-state encode0.59–1.35×first encode 0.77–1.76×; decode 0.64–1.08×, below 1.0 on 10 of 13
1. Motivation: prediction from the previous point
V4 and V5 code each quantized coordinate as a delta from the preceding point in the buffer. How good that prediction is depends on the point order the driver uses.
In firing order, all lasers of one firing are stored, then all lasers of the next firing. The preceding point therefore belongs to a different beam, and the previous return of the same beam is K = number of lasers points back.
In row order, used by organized clouds (height > 1), the preceding point is the adjacent return on the same beam, and K = width reaches the adjacent beam.
buffer orderpreceding point (V4/V5 reference)point K back (V6 LagK reference)
Ranges in metres, illustrative. Residuals are zig-zag varints of the quantized difference. At 1 mm resolution, 7.47 m is 7,470 steps (2 bytes) and 0.03 m is 30 steps (1 byte). A jump to the 39 m beam is 3 bytes.
Firing order. The LagK or Median predictor references the same beam. Measured effect of the predictor selection: nuScenes +33.1%, Hesai +17.5% (size increase with it disabled).
Row order. The previous-point predictor is already close to optimal. The gain comes mostly from the validity mask and the stream layout. The Median predictor combines the left and upper neighbours.
2. Chunk layout
Both versions split the cloud into chunks of 32,768 points. Stage 1 codes each chunk and stage 2 compresses it with LZ4 or ZSTD.
V5 stage 1 writes the FLOAT32 fields interleaved per point (FloatN varints), then one adaptive section per integer field.
V6 stage 1 writes, in order: the geometry section, one section per other quantized FLOAT32 field, and the V5 integer sections.
0 = raw: x, y, z as FLOAT32 columns, n × 4 bytes each; the other geometry fields are absent. 1 = predicted.
predictor
u8
0 Previous, 1 LagK, 2 Median (LOCO-I), 3 SecondOrder
K
uvarint
lag in points
mask_kind
u8
0 none, 1 NaN, 2 zero
valid
⌈n/8⌉ bytes
1 bit per point, LSB first; present when mask_kind ≠ 0
size_x, size_y
uvarint ×2
byte sizes of the x and y streams; the z stream ends where it ends
x, y, z
streams
one encodeVarint64 per valid point (zig-zag + 1; 0 = NaN): q − prediction
q = round(v / resolution). The encoder quantizes in float precision when |q| < 222 and in double precision otherwise. The decoder reconstructs float(double(q) × resolution). An invalid point decodes as NaN or 0.0f on all three axes, and the predictors see the previous value repeated. Each other quantized FLOAT32 field is a section of its own: a u8 mode, then either n raw FLOAT32 values or a residual stream with the Previous predictor.
3. Changes and ablation
Each effect is the increase in ZSTD output size when only that feature is disabled, across the 13 datasets.
Per-chunk predictor
Rationale: the best reference depends on point order and scene.
Selection: the encoder estimates the stage-1 size of each candidate on the first 4,096 points and keeps the smallest. LagK and Median are candidates only when 1 < K and K + 1 < 4,096.
K: the width when height > 1. Otherwise, the lag in [2, 2100] that minimizes Σ|qi − qi−K| over 64 sampled points of the first chunk.
median +2.6%nuScenes +33.1%, Hesai +17.5%
Validity mask
Rationale: no-return points (all three axes NaN, or all three 0.0f) break the prediction twice, once on entry and once on exit.
Coding: the mask kind is the majority kind in the chunk. Invalid points are left out of the streams.
median +4.7%max +10.2% (Ouster OS0-128)
Per-field streams
Rationale: x, y and z residuals have different distributions, and separate streams give stage 2 homogeneous input. Other quantized FLOAT32 fields are coded the same way, one stream each.
V5 +3.2% to +22.4%vs V6 with predictor and mask off
Stage-2 block boundaries
Rationale: ZSTD keeps one set of literal and sequence statistics per block.
Coding: stage 1 passes its section offsets to stage 2, which ends a block (ZSTD_e_flush) at each one. The output is a single standard frame.
median +5.3%range +1.6% to +8.0%
Encoder state cache
Rationale: lag detection and predictor probing need a separate pass over the chunk.
Behaviour: the encoder keeps K, the predictor and the mask kind of each chunk for the next clouds of the stream, whatever their size, and probes again every 16 encodes. x, y, z are quantized with SSE and coded in one pass. On the first encode and on re-probes, only the first 4,096 points (8,192 for lag detection) are quantized for probing. Output bytes are identical either way.
Callers: keep one encoder per stream (PointcloudEncoderCache). A new encoder per cloud always pays the first-encode cost.
−3% to −27%encode instructions; KITTI 1.83 → 1.35× V5
Output size increase with one feature disabled (ZSTD)
Resolution 1 mm. The last column compares V5 with V6 with predictor and mask disabled, so it is the stream layout and block boundaries only. 0.0% on the mask means the dataset has no no-return points.
4. Compressed size
V5V6
ZSTD, % of raw size. Datasets are listed in the same order in every chart and table: largest reduction vs V5 first. Hover for V4, V5 and V6.
5. Encode and decode cost
Instructions counted with callgrind, with ZSTD compression included, relative to V5: the first encode with a new encoder, a steady-state encode with a reused encoder, and one decode of each cloud.
first encodeencode (reused encoder)decode
The first encode also probes the predictors on a prefix of each chunk: 0.77–1.76× V5, highest on KITTI (1.76×) and the PCD sample (1.74×). Those two datasets have no integer fields, so V5 takes its V4 FloatN path on them, which is the fastest V5 case.
6. Compatibility and validation
Version selection
Encoders write V6 by default (kEncodingVersion = 6). Decoders from 1.3.1 and earlier reject version 6: set EncodingInfo::version = 5 (--encoding-version 5 in the rosbag converter) to write for them.
Fallbacks
A chunk with an infinite value, or with |q| ≥ 250, is written in raw mode and is bit-exact. The V5 layout is used for version-6 clouds unless x, y, z are the first three FLOAT32 fields with a resolution in (0, 1018); the decoder applies the same test to the header.
Determinism
The float-or-double quantization rule is applied per value. The first (probing) encode and later cached encodes of the same cloud therefore produce identical bytes, which a test checks.
Format freeze and fuzzing
The experiment switches were removed from encoding_config. A mutation test ran 81,000 corrupted V4/V5/V6 messages under ASan and UBSan with no failures. It found signed-overflow bugs, now fixed, and a heap overflow in the V5 RLE decoder, fixed in #149.
7. Rollout
1.3.1
Writes V5
Reads versions 2–5; rejects version 6.
Next minor · #150
Writes V6 by default
Reads versions 2–6. Resolutions refined to the data by default.
Readers on 1.3.1
Write V5 for them
--encoding-version 5 until they upgrade.
Appendix A. Datasets
All sources are public. Each was converted to a raw PointCloud2 buffer with the layout listed; the charts use the names in the first column. Check each dataset's license on its home page before redistributing.
all frames of the test pcaps OS-0-32-U1_v2.2.0_1024x10, OS-0-128-U1_v2.3.0_1024x10, OS-1-32-G_v2.1.1_1024x10, OS-1-64_1024x10_fw20, OS-1-128_v2.3.0_1024x10_lb_n3, OS-2-32-U0_v2.0.0_1024x10, OS-2-128-U1_v2.3.0_1024x10 (sdk/tests/pcaps/), destaggered, no return = (0, 0, 0)
ouster-ros: x, y, z, intensity f32; t u32; reflectivity, ring, ambient u16; range u32 (48 B)
sweeps 315969904359876000, 315969904460072000, 315969904559605000 of sensor val log 02678d04-cc9f-3148-9f95-1ba66347dff9; xyz are float16 in the source
ROS 2 velodyne: x, y, z, intensity f32; ring u16; time f32 (22 B)