Tasks on inference-cache

Deploy a CacheBackend

Mon, 01 Jan 0001 00:00:00 +0000

Get a cache-aware inference setup running in about five minutes. This assumes the inference-cache operator (controller + policy server + CRDs) is already installed.

1. Write the minimum CacheBackend

A CacheBackend binds to your inference-engine pods by label and makes their KV cache reusable across requests. Here is the minimum-viable spec:

Bind an engine

Mon, 01 Jan 0001 00:00:00 +0000

Binding is how a CacheBackend claims inference-engine Pods in its namespace. The engine Deployment and image remain inference-system owned.

Lifecycle

Apply a typed CacheBackend with spec.lmCache.topology: PodLocal.
Create engine Pods whose labels include every spec.engineSelector.matchLabels entry.
At Pod CREATE, the webhook atomically injects the LMCache MP native sidecar, shared memory, and the runtime-specific connector wire.
When configured, the kvevent-subscriber sidecar reports KV events to the routing index.

The match is evaluated once, at Pod CREATE

Relabeling an existing Pod does not re-run admission. Recreate the Pod after changing labels or CacheBackend configuration.

Typed vLLM MP wire

The vLLM adapter injects LMCacheMPConnector with the module path lmcache.integration.vllm.lmcache_mp_connector, loopback host/port, --disable-hybrid-kv-cache-manager, PYTHONHASHSEED=0, and the fail-open value. Reserved entries are:

Tune lookup and eviction

Mon, 01 Jan 0001 00:00:00 +0000

Most namespaces need no CachePolicy — the server ships sane defaults. Reach for one when you want to change how lookups are filtered/ranked, or how the index evicts, in a specific namespace. There is at most one CachePolicy per namespace.

See CachePolicy for the field-by-field reference; this page is worked examples.

A production-shaped policy

apiVersion: inferencecache.io/v1alpha1
kind: CachePolicy
metadata:
 name: default
 namespace: serving
spec:
 eviction: LRU
 evictionTTL: 30m
 minimumPrefixTokens: 16 # skip lookups for very short prompts
 minimumMatchedTokens: 64 # a replica needs 4 KV blocks of overlap to be offered
 routingFloorScore: "0.1" # the top replica must clear this score
 lookupTimeoutMs: 5 # server-side time budget per lookup (fail-open TIMEOUT)
 strategy:
 enableChainMatching: true
 enableTenantHot: true
 affinityRouting: Enabled

Recipes

Reduce noisy hints from shared system prompts

Chat templates give every request the same long system-prompt prefix, which would otherwise match every replica. The default floors already handle this, but you can tighten them:

Isolate tenants

Mon, 01 Jan 0001 00:00:00 +0000

Multi-tenancy in inference-cache comes in two layers: structural identity isolation (always on) and an optional entry-count quota. For byte-level isolation you use Kubernetes, not a cache field — this page explains why and how.

See CacheTenant for the field reference.

Identity isolation is automatic

The index is keyed by (tenant, model, hash_scheme, adapter_id, prefix_hash). Because tenant is part of the key, one tenant’s hints can never collide with another’s, and LookupRoute is tenant-scoped. You get this the moment your producers report a tenant_id — you do not need a CacheTenant for isolation to hold.

Monitor the cache plane

Mon, 01 Jan 0001 00:00:00 +0000

There are three complementary ways to watch a running install: the Kubernetes status surfaces, the Prometheus metrics, and the inferencecache doctor pre-flight diagnostic.

kubectl surfaces

# Per-backend health, matched pods, and live prefix/event evidence
kubectl get cachebackend -A

# The cluster-wide cache "world map"
kubectl get cacheindex cluster-default -o yaml

# Per-tenant entry counts
kubectl get cachetenant -A

When a backend is not Ready, the reason is in .status.conditions[]: