inference-cache

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

One operator, two binaries

inference-cache is a single operator split across two control-plane binaries, plus a CLI and an in-cluster sidecar. The split follows the responsibilities: the controller owns the Kubernetes reconcile loop and admission; the server owns the routing decision and the cache-state index.

Contributing

Mon, 01 Jan 0001 00:00:00 +0000

This page summarizes the contributor workflow. The authoritative version is CONTRIBUTING.md in the repository.

Local setup

Requirements: Go 1.26.6+, Make, Docker, kind, and protoc.

Install the git hooks once per clone — they enforce the project rules below:

make install-hooks

Run the baseline checks before sending a PR:

CRD API

Mon, 01 Jan 0001 00:00:00 +0000

All CRDs are in the API group inferencecache.io, version v1alpha1.

Kind	Scope	Short name	Reconciled?	Purpose
`CacheBackend`	Namespaced	`cb`	Yes	Bind engine Pods to typed MP; optionally provision a remote Redis provider.
`CachePolicy`	Namespaced	`cpol`	Declarative (pushed to server)	Per-namespace lookup and eviction tuning.
`CacheTenant`	Namespaced	`ct`	Declarative (pushed to server)	Tenant identity + entry-count quota.
`CacheIndex`	Cluster	`ci`	Yes (status-only)	Cluster-wide mirror of the server aggregate.
`PromptTemplate`	Namespaced	`pt`	Declarative	Cache-aware prompt template + stable/mutable slots.
`PDTopology`	Namespaced	`pdt`	Declarative	Prefill/decode topology for disaggregated serving.

v1alpha1 stability

v1alpha1 is not compatibility-frozen before the first production deployment. The CRD may make breaking schema corrections while the API shape is being finalized. The gRPC/proto contract follows its own compatibility policy for external consumers.

CacheBackend

Key spec fields:

Deploy a CacheBackend

Mon, 01 Jan 0001 00:00:00 +0000

Get a cache-aware inference setup running in about five minutes. This assumes the inference-cache operator (controller + policy server + CRDs) is already installed.

1. Write the minimum CacheBackend

A CacheBackend binds to your inference-engine pods by label and makes their KV cache reusable across requests. Here is the minimum-viable spec:

Observability & Alerts

Mon, 01 Jan 0001 00:00:00 +0000

The bundle

A Prometheus alert bundle for the operational silent-failure patterns this system has hit in production ships under config/observability/. It is not part of config/default — the alerts are opt-in so that installs without prometheus-operator CRDs are not affected by an unknown apiVersion.

Bind an engine

Mon, 01 Jan 0001 00:00:00 +0000

Binding is how a CacheBackend claims inference-engine Pods in its namespace. The engine Deployment and image remain inference-system owned.

Lifecycle

Apply a typed CacheBackend with spec.lmCache.topology: PodLocal.
Create engine Pods whose labels include every spec.engineSelector.matchLabels entry.
At Pod CREATE, the webhook atomically injects the LMCache MP native sidecar, shared memory, and the runtime-specific connector wire.
When configured, the kvevent-subscriber sidecar reports KV events to the routing index.

The match is evaluated once, at Pod CREATE

Relabeling an existing Pod does not re-run admission. Recreate the Pod after changing labels or CacheBackend configuration.

Typed vLLM MP wire

The vLLM adapter injects LMCacheMPConnector with the module path lmcache.integration.vllm.lmcache_mp_connector, loopback host/port, --disable-hybrid-kv-cache-manager, PYTHONHASHSEED=0, and the fail-open value. Reserved entries are:

CacheBackend

Mon, 01 Jan 0001 00:00:00 +0000

What is a CacheBackend?

A namespaced CacheBackend selects an inference runtime, its engine-side cache integration, and an optional remote L3. The inference system owns the engine Deployment and image. CacheBackend injects the selected connector and the cache components it owns into matching Pods.

gRPC API

Mon, 01 Jan 0001 00:00:00 +0000

Service InferenceCache, package inferencecache.v1alpha1, proto proto/inferencecache/v1alpha1/inferencecache.proto. Served on :9090 (plaintext by default; TLS overlay available). See The gRPC contract for the guarantees; this page is the per-RPC reference.

Consumer RPCs

`LookupRoute(LookupRouteRequest) → LookupRouteResponse`

The core cache-aware routing hint. Unary and fail-open. It does not mutate cache membership or response-visible routing state. It emits metrics, and a delivered prefix hit under an LFU policy credits the matched entries’ internal eviction access counters.

Index sizing

Mon, 01 Jan 0001 00:00:00 +0000

The server’s cache-state index lives in memory. This page helps you size the server pod and choose eviction settings so the index never approaches the pod’s memory limit (an OOM-kill is the one failure the soft-state design cannot hide).

Pod budget at a glance

Storage entries (distinct prefix keys × replicas reporting each)	Peak RSS	Recommended pod memory
100K storage entries	~110 MiB	256 MiB
500K storage entries	~300 MiB	512 MiB
1M storage entries (the default cap)	~540 MiB	1 GiB (recommended floor)
1.5M storage entries	~700 MiB	1.5 GiB

These include roughly 20% headroom over measured heap. The default entry cap is 1,000,000.

CachePolicy

Mon, 01 Jan 0001 00:00:00 +0000

What is a CachePolicy?

A CachePolicy tunes how the server filters and ranks lookups, and how it evicts, for a single namespace. It is entirely opt-in: the server ships sane defaults, so most namespaces need no CachePolicy at all. It is purely declarative — the controller flattens all policies and pushes the resolved values to the server; the reconciler never writes CachePolicy.status.

gRPC TLS

Mon, 01 Jan 0001 00:00:00 +0000

The decision

The gRPC API on :9090 uses one-sided Service TLS via cert-manager, terminated in-process — no sidecar, Envoy, or Ingress. Mutual TLS (client-certificate auth) is a Phase-2 feature flag, not yet implemented.

The server binary supports TLS via --tls-cert-file / --tls-key-file (both-or-neither — supplying exactly one fails startup). The certificate is served through a reloading GetCertificate hook, so a cert-manager-rotated Secret is picked up without a restart. The posture is exposed as the gauge inferencecache_server_grpc_tls_enabled (0/1).

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

Both inference-cache binaries expose Prometheus metrics on their pod’s :8080/metrics, all prefixed inferencecache_*. The two binaries use separate registries — the server’s series cover the index and gRPC handlers; the controller’s cover the reconcilers. Standard go_* and process_* collectors are also present but are not part of this schema. Successfully injected PodLocal LMCache sidecars expose their upstream lmcache_mp_* metrics separately; the observability overlay includes a cross-namespace PodMonitor for them.

Server metrics (`inference-cache-server`)

Gauges

Metric	Labels	Meaning
`inferencecache_server_up`	—	`1` when the server is serving.
`inferencecache_server_grpc_tls_enabled`	—	`1` when gRPC TLS is configured.
`inferencecache_index_entries`	`model`	Distinct prefix keys in the index (excludes the reserved probe tenant). Drained models are zeroed, not left stale.

Counters

Metric	Labels	Meaning
`inferencecache_lookup_route_calls_total`	`model`, `reason_code`, `hint_used`	Lookups by outcome. `hint_used="true"` ⇔ a non-empty score list (`PREFIX_MATCH` / `TENANT_HOT` / `AFFINITY_HINT`).
`inferencecache_index_evictions_total`	`algorithm` (`lru`/`lfu`), `reason` (`cap`/`ttl`)	Index evictions. `reason="cap"` is the authoritative over-budget signal.
`inferencecache_tenant_evictions_total`	`tenant_id`, `reason` (`over_entries`)	Per-tenant quota evictions. A bounded non-zero rate is normal Fairness.
`inferencecache_snapshot_auth_total`	`result` (`ok`/`unauth`/`forbidden`/`error`)	`/snapshot` auth outcomes. Wrong audience lands in `unauth`.
`inferencecache_policy_auth_total`	`result`	`/policy` auth outcomes.
`inferencecache_probe_auth_total`	`result`	`/probe` auth outcomes.

Histogram

Metric	Labels	Buckets
`inferencecache_lookup_route_latency_seconds`	`model`	100µs, 250µs, 500µs, 1ms, 2.5ms, 5ms, 10ms, 25ms, 50ms, 100ms

Controller metrics (`inferencecache-controller`)

The controller pod has no Service in front of it — scrape it with a PodMonitor (the shipped observability bundle includes one) or pod-IP service discovery, or the controller-side alerts have no series to evaluate.

Tune lookup and eviction

Mon, 01 Jan 0001 00:00:00 +0000

Most namespaces need no CachePolicy — the server ships sane defaults. Reach for one when you want to change how lookups are filtered/ranked, or how the index evicts, in a specific namespace. There is at most one CachePolicy per namespace.

See CachePolicy for the field-by-field reference; this page is worked examples.

A production-shaped policy

apiVersion: inferencecache.io/v1alpha1
kind: CachePolicy
metadata:
 name: default
 namespace: serving
spec:
 eviction: LRU
 evictionTTL: 30m
 minimumPrefixTokens: 16 # skip lookups for very short prompts
 minimumMatchedTokens: 64 # a replica needs 4 KV blocks of overlap to be offered
 routingFloorScore: "0.1" # the top replica must clear this score
 lookupTimeoutMs: 5 # server-side time budget per lookup (fail-open TIMEOUT)
 strategy:
 enableChainMatching: true
 enableTenantHot: true
 affinityRouting: Enabled

Recipes

Reduce noisy hints from shared system prompts

Chat templates give every request the same long system-prompt prefix, which would otherwise match every replica. The default floors already handle this, but you can tighten them:

CacheTenant

Mon, 01 Jan 0001 00:00:00 +0000

What is a CacheTenant?

A CacheTenant gives an external tenant two things:

A stable identity (spec.tenantID) that the cache-state index isolates on, so one tenant’s cache hints can never collide with another’s.
An optional entry-count quota that caps how many distinct prefix keys the tenant may occupy in the index.

CacheTenant is namespaced, short name ct.

Isolate tenants

Mon, 01 Jan 0001 00:00:00 +0000

Multi-tenancy in inference-cache comes in two layers: structural identity isolation (always on) and an optional entry-count quota. For byte-level isolation you use Kubernetes, not a cache field — this page explains why and how.

See CacheTenant for the field reference.

Identity isolation is automatic

The index is keyed by (tenant, model, hash_scheme, adapter_id, prefix_hash). Because tenant is part of the key, one tenant’s hints can never collide with another’s, and LookupRoute is tenant-scoped. You get this the moment your producers report a tenant_id — you do not need a CacheTenant for isolation to hold.

Reason codes

Mon, 01 Jan 0001 00:00:00 +0000

reason_code is a string, not a proto enum — new codes can be added without breaking old clients. Clients must treat any unrecognized code as the no-hint default (fail open, round-robin).

LookupRoute / LookupPDRoute

Code	When emitted
`PREFIX_MATCH`	A replica holds the exact prefix or a leading block-run; the ranker returns a non-empty list; at least one replica clears `minimumMatchedTokens` (default 64) and the top score clears `routingFloorScore` (default 0.1).
`NO_HINT`	The fail-open default: a novel prefix with no affinity fallback, an empty/unspecified key, cold start (globally empty index), a request gated below `minimumPrefixTokens`, a sub-floor matched-tokens/score result (with affinity disabled), or a disabled index.
`TENANT_HOT`	Prefix miss, `strategy.enableTenantHot ≠ false`, and a warm replica exists (recent stats, `hit_rate` above the floor, serving the requested scheme). `matched_tokens = 0`.
`AFFINITY_HINT`	Would-be `NO_HINT`, `affinityRouting: Enabled`, a usable fingerprint, and ≥1 serving replica. A single stable replica; `score` / `matched_tokens` / `estimated_cache_hit_prob` are all `0`.
`POLICY_REQUIRES_CHAIN`	`strategy.requireChain: true` and the request has no valid block-hash chain. Returned before touching the index.
`TIMEOUT`	The lookup deadline expired (context deadline, or `lookupTimeoutMs` elapsed). Fail-open. Clients may also synthesize this locally.
`UNKNOWN_TENANT`	Miss, and the (non-empty) `tenant_id` has zero entries anywhere (index not globally empty). A contract-key mismatch — likely a misconfigured client.
`UNKNOWN_MODEL`	Miss, tenant known, but `(tenant, model)` has zero entries.
`UNKNOWN_HASH_SCHEME`	Miss, `(tenant, model)` has entries, but none under the request’s `hash_scheme`.

LookupPDRoute is a stub today and always returns no hint.

Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Reading CacheBackend readiness

A managed backend’s Ready is the composition of three gates (managed-readiness → KV-event gate → functional-probe gate), so the READY column of kubectl get cachebackend only tells half the story. The reason lives in .status.conditions[], and the condition .reason is the actionable string:

CacheIndex

Mon, 01 Jan 0001 00:00:00 +0000

What is a CacheIndex?

CacheIndex is a cluster-scoped, status-only singleton named cluster-default. It mirrors the server’s in-memory cache-state aggregate into the Kubernetes API so you can observe cluster-wide cache state with kubectl get cacheindex. It is for observability, not routing — the live routing substrate is the server’s in-memory index, and LookupRoute never reads the CR.

CLI: inferencecache doctor

Mon, 01 Jan 0001 00:00:00 +0000

inferencecache doctor is a read-only pre-flight diagnostic — the cache-plane analogue of istioctl analyze. It runs against a live install, prints stable finding codes, and returns CI-friendly exit codes. The binary is bin/inferencecache (from make build).

inferencecache doctor -n serving

Checks and finding codes

After the Kubernetes API prerequisite (check 0), nine diagnostic checks run in a fixed order. Finding codes are stable and greppable; severity is shown in parentheses.

Monitor the cache plane

Mon, 01 Jan 0001 00:00:00 +0000

There are three complementary ways to watch a running install: the Kubernetes status surfaces, the Prometheus metrics, and the inferencecache doctor pre-flight diagnostic.

kubectl surfaces

# Per-backend health, matched pods, and live prefix/event evidence
kubectl get cachebackend -A

# The cluster-wide cache "world map"
kubectl get cacheindex cluster-default -o yaml

# Per-tenant entry counts
kubectl get cachetenant -A

When a backend is not Ready, the reason is in .status.conditions[]:

PromptTemplate

Mon, 01 Jan 0001 00:00:00 +0000

What is a PromptTemplate?

A PromptTemplate declares a prompt template and marks which of its slots are stable (part of the cacheable prefix) versus mutable (vary per request). This backs the mutable-slot render pipeline — the “wedge” that lets a mostly-fixed prompt keep a stable, cacheable prefix even as small parts of it change.

PDTopology

Mon, 01 Jan 0001 00:00:00 +0000

What is a PDTopology?

A PDTopology declares the prefill and decode pools for phase-disaggregated (P/D) serving — a topology where the prefill phase (prompt processing) and the decode phase (token generation) run on separate pools of replicas, so the KV cache produced by prefill is transferred to a decode replica.

LookupRoute & ranking

Mon, 01 Jan 0001 00:00:00 +0000

The routing hint

LookupRoute is the hot-path RPC a gateway calls for every request. Given a prompt-prefix key plus model_id and tenant_id, it returns the replicas that hold that prefix warm, ranked best-first, plus a reason_code. The gateway routes to the top replica for a prefix cache hit — reusing KV / skipping prefill, lowering TTFT and cost — or round-robins when there is no hint.

The gRPC contract

Mon, 01 Jan 0001 00:00:00 +0000

The InferenceCache service

The InferenceCache gRPC service is the data-plane API that gateways and engines integrate against. It is defined in proto/inferencecache/v1alpha1/inferencecache.proto, package inferencecache.v1alpha1, and served on :9090 (plaintext by default; TLS is an opt-in overlay — see gRPC TLS).

Eight RPCs, three roles

Consumer side (gateways) — fail-open; only delivered LFU hits update eviction bookkeeping:

Search Results

Mon, 01 Jan 0001 00:00:00 +0000

inference-cache

Architecture

One operator, two binaries

Contributing

Local setup

CRD API

v1alpha1 stability

CacheBackend

Deploy a CacheBackend

1. Write the minimum CacheBackend

Observability & Alerts

The bundle

Bind an engine

Lifecycle

The match is evaluated once, at Pod CREATE

Typed vLLM MP wire

CacheBackend

What is a CacheBackend?

gRPC API

Consumer RPCs

LookupRoute(LookupRouteRequest) → LookupRouteResponse

Index sizing

Pod budget at a glance

CachePolicy

What is a CachePolicy?

gRPC TLS

The decision

Metrics

Server metrics (inference-cache-server)

Gauges

Counters

Histogram

Controller metrics (inferencecache-controller)

Tune lookup and eviction

A production-shaped policy

Recipes

Reduce noisy hints from shared system prompts

CacheTenant

What is a CacheTenant?

Isolate tenants

Identity isolation is automatic

Reason codes

LookupRoute / LookupPDRoute

Troubleshooting

Reading CacheBackend readiness

CacheIndex

What is a CacheIndex?

CLI: inferencecache doctor

Checks and finding codes

Monitor the cache plane

kubectl surfaces

PromptTemplate

What is a PromptTemplate?

PDTopology

What is a PDTopology?

LookupRoute & ranking

The routing hint

The gRPC contract

The InferenceCache service

Eight RPCs, three roles

Search Results

`LookupRoute(LookupRouteRequest) → LookupRouteResponse`

Server metrics (`inference-cache-server`)

Controller metrics (`inferencecache-controller`)