GitHub - SRKRZ23/vramkit: Lossless-verified VRAM compression for LLM inference: a weight codec (-27% mean / -33% best vs 4-bit on real Qwen weights, beats zstd, lossless) + drop-in PyTorch KV-cache compression (int8 ~2x near-lossless, outlier-aware int4 at cos-sim 0.999). ROCm/CUDA/MPS/CPU.

Why it matters

The article introduces vramkit, a toolkit targeting the two main consumers of GPU memory during LLM inference: model weights and the KV-cache. Its author reports two techniques, both checked with bit-exact roundtrip assertions. A custom range coder using a two-dimensional adaptive context model compresses quantized weights losslessly, outperforming general-purpose compressors like zstd and xz on most tensor types of Qwen2.5-0.5B, suggesting the gains reflect a general property of quantized weights rather than one lucky tensor. For the KV-cache, per-token int8 quantization roughly halves memory with a negligible change in perplexity, which the author calls the result ready to ship. An outlier-aware int4 scheme, which keeps a small share of channels at higher precision, still regresses on the end task and is labeled work-in-progress. Everything was built and benchmarked on a MacBook, with GPU validation still pending.