https://arxiv.org/pdf/2609.02737 This paper has a few interesting ideas. Details of the tag trapping to KV block restriction mechanics lead to a better understanding of vLLM and its pluggable attention and KV-cache mechanisms. The overall control loop was helped by the following questions and answers. A simulation is added at https://ruchirtewari.github.io/html/da_vllm_diagrams.html
How tags get generated
The prompt explains the available modes, their syntax, and when to use them; it also presents the context as numbered chunks so the model can refer to specific regions.arxiv
The model then generates a tag such as <focus magic_chunks="7"> as ordinary output text, using its existing instruction-following abilities—not a separate routing head or tag-generation module.arxiv
The control loop is:
- Model generates text: It chooses a mode and, for focus, a chunk identifier through its normal output generation.arxiv
- Runtime parses the declaration: Once the opening tag’s final
>appears, the parser changes the visible KV-cache blocks.arxiv - Model continues under that restriction: Subsequent tokens are generated with the selected context access until a closing tag restores global mode.arxiv
How does the masking step determine what to mask out based on the tags in the output .
The input context is untagged. only the output is tagged. so the output has an answer that the later part draws attention to ? are the KV cache blocks tagged and the tags used to correlate the tags in the output with the blocks ?
The KV blocks are not tagged with text, and the model is not looking at its own final answer. Instead, the system uses a shared positional map (created before the model starts talking) that lets the metadata layer translate the model’s words into block numbers.
Here is the step-by-step linkage, using the exact “who knows what” logic :
1. The input is “pre-tagged” by position, not by content
Before generation begins, vLLM splits the long context into magic chunks (~2,048 tokens each). The model sees them as a simulated tool transcript, but the metadata layer builds a lookup table:arxiv
| Chunk ID | Token range (logical) | Physical KV blocks |
|---|---|---|
| 1 | tokens 0–2047 | blocks 0–63 |
| 2 | tokens 2048–4095 | blocks 64–127 |
| … | … | … |
| 12 | tokens 22528–24575 | blocks 704–767 |
This map is stored in the DA state machine (the “metadata code”). The text inside chunk 7 is never scanned; the system just knows “chunk 7 = blocks 448–511.”
2. The model generates a “pointer” (the tag), not the answer
When the model writes <focus magic_chunks="7">, it is not quoting the final answer from chunk 7. It is generating a reference string based on what it saw during the previous <global> phase. During <global>, the model attended to all chunks and learned (from the special tool-format tokens) that chunk 7 contains the IPO information.
The tag itself is just text: "<focus magic_chunks=\"7\">". The model has no idea what the final answer is yet—that comes later inside the <focus> block.
3. The state machine traps the pointer and converts it to block numbers
The DA state machine (the middleman) is parsing the output stream token-by-token. It maintains a small buffer of recently generated text. When it sees the closing > of a tag, it runs a regex:x+1
- Sees
<focus magic_chunks="7">→ looks up chunk 7 in the map from Step 1 → gets block range 448–511. - Sees
<local>→ gets empty set (no context blocks). - Sees
</focus>→ reverts to all blocks.
This happens before the next forward pass. The state machine then edits the block table (the numpy staging buffer) to keep only:
- The scaffold blocks (system + question)
- The response blocks (the tokens the model has written so far)
- The specific chunk blocks (448–511)
4. The correlation is positional, not semantic
To answer your specific question: No, the KV blocks are not “tagged” with text. The correlation works like this:
- Output tag says: “I want chunk 7”
- Metadata map says: “Chunk 7 lives at token offset 12,288–14,335”
- vLLM’s block table says: “Tokens 12,288–14,335 are stored in physical blocks 448–511”
The metadata layer is the translator between the model’s semantic language (“chunk 7”) and vLLM’s physical language (“blocks 448–511”).
5. Why this is “surgical”
The model code (the transformer weights) doesn’t know what chunk 7 is. The kernel (FlashAttention) doesn’t know what a “chunk” is—it just reads whatever blocks are listed in the block table. The DA hook lives in the metadata layer, which is the only place that knows both:
- The prompt structure (where chunk 7 is)
- The current output text (that the model just asked for chunk 7)
It edits the block table—the one data structure that both the model code and the kernel treat as read-only configuration—and thereby changes what the kernel reads without either side knowing the difference.github+1
Summary: The model generates a textual pointer (chunk 7), the metadata layer resolves that pointer to physical block numbers using a pre-computed map, and the kernel simply skips the unlisted blocks. No semantic tagging of KV cache, no retrospective analysis of the answer—just a lookup table and a parser.
