SHIRITAS

論文 / arXiv:2611.26619

TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming

PLAIN SUMMARY / やさしい要約

大きな文章を扱うAIは、途中結果を一時的にためる「置き場」をいくつも使います。TideCacheは、その置き場をGPUとCPUの間でうまくやりくりし、必要なものを先回りして運ぶ仕組みを提案します。たとえば、台所の作業台(GPU)に全部を置きっぱなしにせず、使う直前に必要な道具だけ運ぶイメージです。その結果、同時に処理できる量や待ち時間の改善が報告されています。

AIが専門用語を使わずに書いた解説です。内容の正確さは、下の原文の要旨で確認してください。

ABSTRACT / 要旨(原文)

Hybrid architectures have become mainstream among open-weight language models. Their global-attention and bounded-state layers maintain heterogeneous caches whose demands vary with context length and concurrency. We present TideCache, a serving system on CPU--GPU superchips with three components for allocating GPU memory among caches and managing their residency across GPU and host memory. The Transparent Cache Pool redistributes physical pages among cache components behind fixed virtual addresses, preserving cache layouts and backend interfaces. The Cache Streamer uses computation intervals provided by bounded-state layers to prefetch the next global-attention layer's prefix into GPU slots shared across global-attention layers, reducing GPU residency while overlapping layer-wise transfers with computation. Generated KV remains resident. The Request Scheduler selects resident or streamed execution and triggers capacity adjustments according to request size and queue state. Together, they let HBM released by streaming accommodate other cache components and additional requests. Implemented in SGLang and evaluated on GB300 and GH200 across seven model/platform configurations, TideCache completes up to 2.04x as many agent turns, reduces overall P50 per-token latency by 20--59%, and improves agent P50 TTFT by 1.40--3.05x compared with SGLang with HiCache.

ここに表示しているのは原論文の要旨です。AIによる要約や解釈は含みません。

この論文を取り上げた記事