tiyuvta
1 byte store, dropped at -O3 7 min

The compiler ate my byte

How a bit-identity gate caught an nvcc miscompile

On 2026-08-04 I was running the first end-to-end battery for the official Qwen3.6-27B-FP8 checkpoint — 29 GB, 407 block-128 scale grids — on a rented 2x RTX 5090 box. The very first gate in the sequence, a bit-parity check on the FP8 block-dequant kernel, failed. The same commit had passed that gate, green, on my laptop the day before.

The kernel source had not changed. The compiler had. The laptop runs nvcc 13.1; the rented box runs nvcc 13.0.88. At -O3 targeting sm_120a, nvcc 13.0.88 drops the first of two adjacent byte stores issued under a lane predicate — which in this kernel zeroes the low byte of every block scale it writes. That is the whole bug. This post is the receipt trail: how a gate that compares bytes, not vibes, caught it on the first run, what it would have cost had it shipped, and the operational rule I extracted.

Everything below links to raw files in the memra repo; the finding write-up is research/fp8ship-20260804/official/RESULTS.md and the fix landed as commit 93e1b6a1.

The setup

The kernel under test dequantizes FP8 block-128 weights on the GPU and re-encodes them as Q8_0 — 34-byte blocks, each an f16 scale d followed by 32 int8 quants. Its contract in this engine is bit-identity: the GPU path must produce byte-for-byte the same Q8_0 buffer as the CPU reference path. Not “close”, not “within tolerance” — identical. That contract is enforced by a kernel-check arm called [fp8-blk-gpu], which runs the dequant on three tensor shapes and compares every output byte against the CPU reference.

Box conditions, since every number should state them: vast.ai 2x RTX 5090 32GB, CUDA 13.0.1 driver stack, nvcc 13.0.88, 256-thread EPYC host. Comparison baseline: the identical commit, green on a laptop with nvcc 13.1.

First run, first failure

The [fp8-blk-gpu] arm failed immediately: 3968, 680, and 8 bad bytes on the three shapes. Bad-byte counts alone tell you something is wrong; they do not tell you what. The step that localized it was a byte-in-block histogram — bucket every mismatched byte by its offset within its 34-byte Q8_0 block.

Every single bad byte, across all three shapes, sat at offset 0. Offset 0 is the low byte of the f16 scale d. The 32 quant bytes were untouched; the high byte of the scale was untouched. Something was zeroing exactly one byte, in exactly one position, in every affected block.

The kernel writes that scale like this: lane 0 of each warp converts the float scale to f16, splits the 16 bits into two bytes, and stores them at dst[0] and dst[1], while every lane stores its own quant at dst[2 + lane]. Two adjacent byte stores under an if (lane == 0) predicate. Legal CUDA C++, and the pattern nvcc 13.1 compiles correctly.

Forty lines to isolate it

A miscompile claim needs a repro small enough that anyone can check it, so I extracted the exact store pattern into a standalone kernel — halfstore_repro.cu. The core is the suspect pattern, verbatim:

uint8_t *dst = out + (size_t)qb * 34;
if (lane == 0) {
    const uint16_t bits = __half_as_ushort(__float2half_rn(d));
    dst[0] = (uint8_t)(bits & 0xffu);
    dst[1] = (uint8_t)(bits >> 8);
}
dst[2 + lane] = (uint8_t)(int8_t)qi;

The harness poisons the output buffer with 0xAA before launch, so a store that never executes reads back as 0xAA, distinguishable from a store of zero. Then it recomputes the same math on the host and compares. On nvcc 13.0.88 at -O3 -arch=sm_120a, all eight blocks fail (halfstore-repro-out.log):

blk 0: ref 0x250e got 0x2500 (got==ref&0xff00: YES; low byte 0x00)
blk 1: ref 0x2522 got 0x2500 (got==ref&0xff00: YES; low byte 0x00)
...
blk 7: ref 0x259c got 0x2500 (got==ref&0xff00: YES; low byte 0x00)
REPRO: 8/8 d-half stores wrong in the ISOLATED pattern

Two details worth reading closely. First, got == ref & 0xff00 on every block: the high byte lands, the low byte does not. Second, the low byte reads 0x00, not the 0xAA poison — the compiled code actively writes a zero there. That is consistent with the two byte stores being merged into one wider store that lost its low half, though I bounded my claim at the observable behavior: the first of two adjacent predicated byte stores does not deliver its value.

The companion file, halfstore_fix.cu, is the same kernel with one change — the two byte stores become a single aligned 16-bit store:

if (lane == 0) {
    *(uint16_t *)dst = __half_as_ushort(__float2half_rn(d));
}

Same compiler, same flags: u16-store variant CORRECT (0/8 bad). The miscompile is specifically the byte-split form. The alignment for the cast is guaranteed in the real kernel because dst walks in 34-byte strides from a cudaMalloc’d base, so it is always even.

The fix, and the gate going green

The shipped fix in crates/memra-engine/cu/fp8_blk_dequant.cu is that one store, plus a comment that names the toolchain so the next person who “simplifies” it back to byte stores knows what they are undoing:

if (lane == 0) {
    // ONE aligned u16 store, NOT two byte stores. nvcc 13.0.88 (CUDA 13.0, sm_120a)
    // miscompiles the byte-split form: dst[0] lands as 0x00 (isolated 40-line repro,
    // 8/8 blocks; nvcc 13.1 compiles the same source correctly).
    *(uint16_t *)dst = __half_as_ushort(__float2half_rn(d));
}

Post-fix, the [fp8-blk-gpu] arm is green on all three shapes on the affected box (kernel-check-postfix-gpu1.log), and a byte-level probe over the real checkpoint’s tensors (fp8_blk_probe.rs) reports zero bad bytes (parity-probe-postfix-gpu1.log). The battery then ran to completion: bit-identity between the GPU and CPU dequant paths held on the official 27B artifact across four interleaved run pairs, 993,280 of 993,280 prefill-logit bytes clean on every comparison.

What would have shipped

Here is the part that justifies the gate. Before the fix, I measured the miscompiled kernel against the CPU path on the real checkpoint: first-token logits differed by max_abs 3.04, with rms(diff)/rms(ref) at 0.164, and the greedy token stream diverged at token 14. One honest limit, stated in the same breath: the raw logs for those three numbers were lost to a mid-battery rsync --delete; they are transcript-preserved in RESULTS.md and reproducible on demand by reverting the one-line fix. The isolated-repro logs and post-fix logs survive in full.

Sit with what “diverged at token 14” means operationally. A model whose every block scale has its low mantissa byte zeroed does not print garbage. It loads. It answers questions. It produces fluent text that is subtly, pervasively wrong-by-a-little — thirteen tokens of agreement with the correct model, then a different path. A unit test that checks for coherent output passes. A smoke test passes. An eval run would score it, and you would have no idea the number you got belongs to a model nobody intended to ship. The only test class that catches this on day one is the one that refuses to tolerate any difference at all: bit parity against an independent reference. The gate did exactly its job, on its first execution on the new toolchain.

The law: gates re-run per toolchain, not per commit

The reflex that almost let this through is a reasonable-sounding one: “this commit already passed kernel-check.” It did — compiled by nvcc 13.1. CI typically keys correctness to commits, because source is what changes. But a kernel binary is a function of source and toolchain and target architecture, and this failure only exists at one point in that product: nvcc 13.0.88, -O3, sm_120a. We had already learned a version of this on another architecture — Hopper-side work in this repo found ptxas outcomes sensitive to source form per toolchain revision — and this incident generalizes it into an operational rule: parity gates re-run per toolchain, on every box, not per commit. A green gate is a statement about one compiled artifact, not about the source that produced it.

The vendor-neutral reading matters too. This is not an nvcc dunk — nvcc 13.1 compiles the same source correctly, the affected pattern is narrow, and every optimizing compiler has shipped a wrong-code bug at some point. If you are on nvcc 13.0.88 targeting sm_120a and you write adjacent narrow stores under a lane predicate, the 40-line repro above is yours to run; that is the useful artifact here. The lesson is not “compilers are bad.” It is that correctness claims need to be re-earned whenever anything under them moves.

Why you should get to see this class of receipt

tiyuvta serves inference on the engine this gate protects, and the reason this post exists is not the bug — it is the paper trail. Every claim above links to a raw log or a source file in a public repo: the failing counts, the repro, the fix, the post-fix green. That is the standard I think inference customers should hold providers to, because this bug is exactly the kind that provider-side stacks accrue silently: no crash, no error rate, plausible output, wrong model. If your provider rebuilds kernels on a new CUDA image and their acceptance test is “the server came up and the output looks fine,” you inherit whatever the compiler did that day. Ask what their equivalent of the [fp8-blk-gpu] arm is, when it last ran, and on which toolchain. If the answer is a link to a log, you are in good hands — whoever you buy from.