Docs, runbooks and code
A shallow clone of any https Git remote. Pick a branch or a subfolder. Private repositories use a token from an environment variable.
kvpack build https://github.com/acme/runbooks
Open source · Apache 2.0 · runs on your hardware
Pack your knowledge into the model's memory.
kvpack turns a Git repo, a docs site or a folder into a cartridge: a small trained KV cache your model reads instead of a giant prompt. Build it once, keep it in sync, and serve it through an OpenAI-compatible API.
pip install git+https://github.com/Numoy/kvpack
manual.md2,336 tokens
kestrel.safetensors256 tokens · 28 MiB
The cartridge is a trained key/value cache that sits where the system prompt would be. The model reads it exactly as it would read a document.
Connect your knowledge
Point it at where your knowledge already lives.
A cartridge remembers its sources. Run kvpack sync from cron, or let kvpack Studio do it on a schedule: it checks the sources and rebuilds only when something changed, while the current version keeps answering.
A docs site or wiki
The start page and every page linked below its path. kvpack uses the sitemap and respects robots.txt.
kvpack build https://docs.acme.dev/api/
Files you already have
Markdown, HTML, PDFs, code and plain text, skipping hidden and dependency folders.
kvpack build ./policies ./handbook.pdf
$ kvpack sync runbook.safetensors Fetched 1 document runbook.safetensors is up to date. $ echo "The Gondola now carries 8 people." >> runbook/manual.md $ kvpack sync runbook.safetensors Fetched 1 document Sources changed. Rebuilding with the original settings... Corpus: 2,360 tokens -> cartridge of 128 tokens Self-study ━━━━━━━━━━━━━━━━━━━━ 16/16 Training ━━━━━━━━━━━━━━━━━━━━ 2/2 Saved cartridge to runbook.safetensors
A real run, with a small test build. A running kvpack serve picks up the new file without a restart. More connectors (Notion, Google Drive, Confluence) are one class each: write one.
Three ways to give a model your documents
Three trade-offs between memory, speed and recall.
Paste everything into the context
The model sees it all, but every conversation re-reads and stores the whole corpus.
- Corpus seen
- all
- Memory
- grows with corpus
- First token
- after full prefill
- Exact recall
- best
Retrieve a few chunks
Cheap per question, but the model only sees what the search step happened to find.
- Corpus seen
- top-k chunks
- Memory
- small
- First token
- search + prefill
- Exact recall
- good, if found
Load a trained cache
Trained once on the whole corpus into a compressed memory, then loaded in front of every conversation.
- Corpus seen
- all, via training
- Memory
- small, fixed
- First token
- nothing to prefill
- Exact recall
- lossy, grows with scale
The memory math
How much GPU memory does one conversation need?
Every token in the context costs 128 KiB of KV cache on Llama-3.1-8B (32 layers × 8 KV heads × 128 dimensions × keys and values × 2 bytes). Drag the slider to change the size of your corpus.
Each conversation also holds 2,000 tokens of chat. "Conversations per GPU" assumes an 80 GB GPU with 16 GB taken by the model weights. The cartridge's corpus coverage comes from training on all of it, not from holding it in the context.
Why build something other than RAG?
Retrieval can only answer with what it finds.
These are the problems cartridges are designed for. Whether they solve them better than RAG depends on the model, the budget and the data, so measure it on yours with kvpack eval.
Questions that span the corpus
Ask which procedures changed between revision 5 and 7? and there's no single chunk to retrieve. A cartridge is trained on every section, so nothing depends on a search step picking the right one.
Chunk boundaries cut meaning
A definition on page 3 and the rule that depends on it on page 40 rarely land in the same retrieved chunk. The cartridge never splits the corpus at query time.
Search misses paraphrases
Embedding search matches wording. When a question uses different words from the document, the right passage can rank below the cutoff and the model never sees it.
Less infrastructure at query time
No vector database and no ranking step. A cartridge is one .safetensors file that you load next to the model.
When RAG is the better choice today
For exact fact lookup with a small model, RAG wins in our benchmark below, even with the same memory budget. It also wins if your data changes every minute or you need a citation to the source passage. A cartridge costs minutes to hours of GPU time to train, so it suits documents that change occasionally, get asked about often, and are served where memory per conversation or time to first token matters.
How it works
The model studies your documents, then the cache learns what it learned.
kvpack implements self-study with context distillation from the Cartridges paper (ICLR 2026). The model's weights never change. Only the cartridge is trained.
1 · Self-study
Practice conversations
kvpack fetches your sources, shows the model a random chunk and has it write a question: a fact to recall, a section to summarize, data to extract. The model then answers with the chunk in view, and kvpack records its next-token probabilities.
kvpack build https://github.com/acme/docs
2 · Distill
Train the cache
The cartridge starts as the real KV cache of the corpus's opening. For each practice question, the model sees only the cartridge, and gradient descent updates the cached keys and values until its answer probabilities match the recorded ones.
kvpack sync docs.safetensors
3 · Load
Use it anywhere
The cartridge replaces the system prompt. Chat with it in the terminal, compare it against the full document, or serve it behind an OpenAI-compatible API where each cartridge is its own model.
kvpack serve docs.safetensors
Results
The research says it works at scale. Here's our own benchmark, including where it loses.
38.6×less memory than putting the corpus in context, at matching quality (Cartridges paper)
26.4×higher serving throughput, because each conversation needs so much less KV cache (Cartridges paper)
484ktokens of effective context on MTOB, up from the model's native 128k (Cartridges paper)
A small model, a document it has never seen, 28 questions
Qwen3-0.6B on an M1 Max. We packed a 2,341-token fictional manual into 256- and 1,024-token cartridges trained on 384 practice conversations, then asked 28 questions, each graded by the facts a correct answer must contain. RAG uses BM25 retrieval; "cartridge + 1 chunk" gives the model both.
| Setup | Correct | Context held | KV cache | Time to first token |
|---|---|---|---|---|
| Full document | 25/28 | 2,341 tokens | 256 MiB | 1,129 ms |
| RAG, 4 chunks | 24/28 | 1,006 tokens | 110 MiB | 448 ms |
| RAG, 2 chunks | 25/28 | 507 tokens | 55 MiB | 273 ms |
| RAG, 1 chunk | 24/28 | 255 tokens | 28 MiB | 162 ms |
| Cartridge (256) + 1 chunk | 25/28 | 506 tokens | 55 MiB | 207 ms |
| Cartridge, 1,024 tokens | 16/28 | 1,024 tokens | 112 MiB | 81 ms |
| Cartridge, 256 tokens | 12/28 | 256 tokens | 28 MiB | 85 ms |
| No context | 0/28 | 0 | 0 | 41 ms |
On its own, a cartridge is the fastest setup that knows the document, but at this tiny scale it recalls fewer facts than RAG. Together with one retrieved chunk it matches the full document and the best RAG setup (25/28) with a fifth of the full document's memory. The paper's results come from 4B–8B models and thousands of practice conversations; we'll publish larger runs here. Every command is in the README.
kvpack Studio · self-hosted
A workspace for your team, on your own machines.
Studio is the same engine with a web app on top. Connect sources, watch builds, try cartridges in a playground, and hand out API keys. Every cartridge is a model name in an OpenAI-compatible API.
- Keeps itself current. Hourly, daily or weekly syncs rebuild only on change, with zero downtime.
- Safe for many users. Private network addresses, stray server folders and unlisted secrets are refused by default.
- Runs where you do. One GPU box, or your own Modal account with GPUs that scale to zero.
- No lock-in. Download any cartridge as a file and serve it with plain
kvpack serve.
# one GPU machine
cd studio && uv sync
uv run kvpack-studio serve --host 0.0.0.0
Quickstart
Build your first cartridge.
- Works with open-weight models you run yourself: Llama 3.x, Qwen3 and others built like them. Hosted APIs like ChatGPT can't load a KV cache, so they're not supported.
- Runs on NVIDIA GPUs, Apple Silicon and CPU.
- On a GPU server, point it at vLLM to generate practice data much faster.
# install pip install git+https://github.com/Numoy/kvpack # a repo, a docs site or a folder in, a cartridge out kvpack build https://github.com/your-org/handbook --model Qwen/Qwen3-4B # how does it do against RAG on your questions? kvpack eval handbook.safetensors --docs https://github.com/your-org/handbook -q questions.jsonl # serve it behind an OpenAI-compatible API, and keep it fresh kvpack serve handbook.safetensors --port 8000 kvpack sync handbook.safetensors