Cheetah.ai
A KV cache control plane for multi-agent LLM pipelines. Reads the workflow DAG to predictively pre-warm GPU memory between agent calls - eliminating the amnesia tax where each agent cold-reads the same documents from scratch. Built in 24 hours at Uncommon Hacks 2026. Won Best Use of Snowflake.
Architecture
How it works
The pipeline is declared in manifest.yaml - a YAML DAG specifying which agent runs in which order, which documents each needs, and what task each is assigned. This declaration is the key input that vanilla serving engines never see. Cheetah.ai's entire scheduling advantage comes from having this information ahead of time rather than reacting to each request in isolation.
The Bridge restructures every agent prompt into a canonical [SYS] + [DOC] + [TASK] format and SHA-256 fingerprints the heavy block (system prompt + document). This ensures byte-identity across agents even when agent-prepended metadata diverges, making the KV prefix stably cacheable across agents that read the same document in different roles.
While Agent N is still streaming its response, the Orchestrator reads the DAG, identifies what Agent N+1 needs, and fires keep_resident warmup requests to vllm-mlx using max_tokens=1. This fills idle GPU time between agent calls. The cold prefill does not disappear - it just moves to where the user cannot feel it: behind the previous agent's streaming output.
64-bit SimHash on 3-word shingles detects near-duplicate documents (Hamming distance <= 10/64 bits), so near-duplicate pairs reuse the same cache entry instead of triggering a redundant warmup. The budget-aware LRU eviction policy is forward-looking - it preserves documents the next pipeline step needs rather than evicting purely by recency.
Agent 2 and Agent 3 query vllm-mlx and find their required document already resident in the KV cache. First-token latency drops from 35 seconds (cold prefill of a 14k-token document) to under 0.5 seconds - a 70x reduction per agent. The vllm-mlx engine itself is completely unchanged; Cheetah.ai is a pure scheduling layer above it with no modifications to the cache internals.
Every orchestrator decision streams asynchronously to Snowflake via a non-blocking queue - zero impact on inference latency. Snowflake Cortex AI (llama3.1-70b) generates natural-language summaries of each pipeline run. Dynamic Tables compute a real-time leaderboard of pipeline configurations ranked by throughput. The Streamlit dashboard renders live latency charts, a decision log, and a hot-document timeline showing which docs were pre-warmed and when.