A pure Rust Transformer inference engine of roughly 2,000 lines of code runs Llama-3.2 1B at about 40 tokens per second on an Nvidia Tesla T4 GPU, according to the project's announcement. Built on the enkiruntime platform, the engine supports live migration of execution and KV-cache between GPU VRAM and CPU RAM mid-generation, and the same binary runs on Nvidia GPUs, Intel integrated graphics, and multi-core CPUs. Source code is available on GitHub.
No score is assigned. Sources and their independence are shown in the citation chain below.