TokenGate learns which prompt tokens to drop, cutting long-context prefill time by up to 11.3x
A team including Ion Stoica and Matei Zaharia bolts a small learned gate and LoRA onto a pretrained LLM; the gate scores document tokens and forwards only the top-K to upper layers. They report 1.4-11.3x faster time-to-first-token and up to 3.4x less memory, including on hybrid models, with code and checkpoints released.
Why it matters
A drop-in way to speed up RAG and long-prompt prefill on existing models, including hybrid-attention ones that KV-pruning methods can't handle.
Source: Read the full story on arXiv