Zhenyu Zhang, Zhichao Cao
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
LLM serving systems cache prompt KV state, yet most front ends still re-tokenize the full request text on every call. The mismatch is most costly for coding agents, whose sessions repeatedly submit a long transcript after appending a small tool result. However, reusing prior tokenization results is hard because even a short append can change token boundaries near the end of the previous sequence. Across 153,951 calls from two agent ecosystems, the median call appends only about 1.4 K characters, and only 1.0-3.6% of calls start or rebuild a session. Those calls, however, carry full contexts that reach millions of characters. At the fleet level, the aggregate prompt-cache hit rate is 94.1%, and as it approaches 0.99, tokenization grows from 10% to 64% of time to first token (TTFT) in our component measurements. We present TokTier, a stateful tokenization service for this two-mode workload with CPU and GPU integration. TokTier enforces one contract: emitted token IDs from Tok-Tier are always identical to full reference tokenization of the request text. For a session continuation, TokTier keeps the session's previous token sequence, re-tokenizes a small window around the append, and accepts the splice only when a per-request check finds a stable pre-tokenization boundary. A failed check triggers a wider window or full reference tokenization. For a call without a reusable prefix, TokTier decomposes GPT-family regex pre-tokenization into run-local rules and executes exact pre-tokenization and BPE on a GPU for low latency and high throughput. A sampled shadow verifier re-checks live traffic against the reference. Across 17 production tokenizer families, our differential campaigns include 1.50 × 10 10 split checks, full sweeps of a 12.4 TB real-text corpus, and 93,000+ replayed agent steps, all with zero divergence. Incremental repair takes 0.5-1.1 ms from 100 K to 3 M characters, up to 437× faster than HF tokenization and 2.1× faster at 1 M characters than the strongest cache-based baseline (i.e., Gigatoken) in its most favorable, fully prewarmed mode. GPU full tokenization encodes a 1 M-character request in 0.87 ms, up to 491× below HF tokenization and 23.4× below the fastest previously published CPU method on the same texts and protocol. With vLLM in the loop, TokTier lowers median TTFT by 16-34% in loaded regimes and P99 TTFT by 23% under recorded burst arrivals. Under a 50 ms P99 objective, a four-core CPU-based repair pool plus one GPU sustains 1,821 requests/s, where a 16-core stateless CPU front end saturates at only 40 requests/s.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2607.29678")
get_code_for_paper("2607.29678")
have("2607.29678")
Connect an agent — have() is free.