MatX

About

MatX designs optimized chips and systems for training and inference of large language models, with a focus on maximizing low-precision compute performance for frontier AI labs.

Featured work

Future Leakage in Block-Quantized Attention

Block quantization is essential for high-throughput LLM inference — grouping matrix elements into blocks that share a scale factor enables efficient low-precision compute on modern accelerators. But when those blocks span multiple token positions in the attention mechanism, a subtle problem emerges: later tokens can leak information back to earlier ones during training, creating a mismatch between how the model learns and how it runs at inference.

MatX researchers identified this future leakage problem in MXFP4-quantized attention, showed it causes measurable divergence between training and inference loss, and proposed a targeted fix: keep block-diagonal tiles unquantized where leakage occurs while quantizing everywhere else. Validating on 1B-parameter models trained on C4, the corrected approach maintained consistent performance — a key step toward quantized attention that matches float baselines while delivering real end-to-end speedups.

Message from Grant

If you haven't seen it already, I highly recommend looking at the blackboard lecture on chip design from the CEO of MatX, Reiner Pope. Some of you may also know him as an author on the Efficiently Scaling Transformer Inference paper, and while learning more I also really enjoyed this conversation he had with John Collison. The takeaway from all of these is not just that they're making chips designed for modern LLMs, but that the way they build is rooted in deep understanding and first-principles thinking. One apt quote Reiner has here is that "a few days of thinking can be worth more than weeks of simulation."

If you're the kind of person who thrives on understanding the full stack, from logic gates up to attention heads, there are not a lot of places where you can engage with that under one roof. This is especially true if you also want to be part of a lean and highly leveraged team. I know many viewers here cherish a deep understanding and getting to the bottom of what's really going on. What makes this not just pleasing but rewarding is when you work on something that requires that level of depth.

The featured post on future leakage in block-quantized attention offers a good example here, because the relevant bug is actually very subtle. A later token's magnitude changes how an earlier token gets rounded, and information quietly flows backward through a causal mask. It's the kind of thing you only notice if you understand not just the model, but the underlying numerics. More generally, a lot of hardware performance improvement can come from low-precision arithmetic, but this immediately raises the question of when throwing away bits can work, and when it bites you.

The open roles span silicon, rack design, compilers, kernels, and ML models. If any of that interests you, do check it out.