Concepts on inference-cache

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

One operator, two binaries

inference-cache is a single operator split across two control-plane binaries, plus a CLI and an in-cluster sidecar. The split follows the responsibilities: the controller owns the Kubernetes reconcile loop and admission; the server owns the routing decision and the cache-state index.

CacheBackend

Mon, 01 Jan 0001 00:00:00 +0000

What is a CacheBackend?

A namespaced CacheBackend selects an inference runtime, its engine-side cache integration, and an optional remote L3. The inference system owns the engine Deployment and image. CacheBackend injects the selected connector and the cache components it owns into matching Pods.

CachePolicy

Mon, 01 Jan 0001 00:00:00 +0000

What is a CachePolicy?

A CachePolicy tunes how the server filters and ranks lookups, and how it evicts, for a single namespace. It is entirely opt-in: the server ships sane defaults, so most namespaces need no CachePolicy at all. It is purely declarative — the controller flattens all policies and pushes the resolved values to the server; the reconciler never writes CachePolicy.status.

CacheTenant

Mon, 01 Jan 0001 00:00:00 +0000

What is a CacheTenant?

A CacheTenant gives an external tenant two things:

A stable identity (spec.tenantID) that the cache-state index isolates on, so one tenant’s cache hints can never collide with another’s.
An optional entry-count quota that caps how many distinct prefix keys the tenant may occupy in the index.

CacheTenant is namespaced, short name ct.

CacheIndex

Mon, 01 Jan 0001 00:00:00 +0000

What is a CacheIndex?

CacheIndex is a cluster-scoped, status-only singleton named cluster-default. It mirrors the server’s in-memory cache-state aggregate into the Kubernetes API so you can observe cluster-wide cache state with kubectl get cacheindex. It is for observability, not routing — the live routing substrate is the server’s in-memory index, and LookupRoute never reads the CR.

PromptTemplate

Mon, 01 Jan 0001 00:00:00 +0000

What is a PromptTemplate?

A PromptTemplate declares a prompt template and marks which of its slots are stable (part of the cacheable prefix) versus mutable (vary per request). This backs the mutable-slot render pipeline — the “wedge” that lets a mostly-fixed prompt keep a stable, cacheable prefix even as small parts of it change.

PDTopology

Mon, 01 Jan 0001 00:00:00 +0000

What is a PDTopology?

A PDTopology declares the prefill and decode pools for phase-disaggregated (P/D) serving — a topology where the prefill phase (prompt processing) and the decode phase (token generation) run on separate pools of replicas, so the KV cache produced by prefill is transferred to a decode replica.

LookupRoute & ranking

Mon, 01 Jan 0001 00:00:00 +0000

The routing hint

LookupRoute is the hot-path RPC a gateway calls for every request. Given a prompt-prefix key plus model_id and tenant_id, it returns the replicas that hold that prefix warm, ranked best-first, plus a reason_code. The gateway routes to the top replica for a prefix cache hit — reusing KV / skipping prefill, lowering TTFT and cost — or round-robins when there is no hint.

The gRPC contract

Mon, 01 Jan 0001 00:00:00 +0000

The InferenceCache service

The InferenceCache gRPC service is the data-plane API that gateways and engines integrate against. It is defined in proto/inferencecache/v1alpha1/inferencecache.proto, package inferencecache.v1alpha1, and served on :9090 (plaintext by default; TLS is an opt-in overlay — see gRPC TLS).

Eight RPCs, three roles

Consumer side (gateways) — fail-open; only delivered LFU hits update eviction bookkeeping: