Reference on inference-cache

CRD API

Mon, 01 Jan 0001 00:00:00 +0000

All CRDs are in the API group inferencecache.io, version v1alpha1.

Kind	Scope	Short name	Reconciled?	Purpose
`CacheBackend`	Namespaced	`cb`	Yes	Bind engine Pods to typed MP; optionally provision a remote Redis provider.
`CachePolicy`	Namespaced	`cpol`	Declarative (pushed to server)	Per-namespace lookup and eviction tuning.
`CacheTenant`	Namespaced	`ct`	Declarative (pushed to server)	Tenant identity + entry-count quota.
`CacheIndex`	Cluster	`ci`	Yes (status-only)	Cluster-wide mirror of the server aggregate.
`PromptTemplate`	Namespaced	`pt`	Declarative	Cache-aware prompt template + stable/mutable slots.
`PDTopology`	Namespaced	`pdt`	Declarative	Prefill/decode topology for disaggregated serving.

v1alpha1 stability

v1alpha1 is not compatibility-frozen before the first production deployment. The CRD may make breaking schema corrections while the API shape is being finalized. The gRPC/proto contract follows its own compatibility policy for external consumers.

CacheBackend

Key spec fields:

gRPC API

Mon, 01 Jan 0001 00:00:00 +0000

Service InferenceCache, package inferencecache.v1alpha1, proto proto/inferencecache/v1alpha1/inferencecache.proto. Served on :9090 (plaintext by default; TLS overlay available). See The gRPC contract for the guarantees; this page is the per-RPC reference.

Consumer RPCs

`LookupRoute(LookupRouteRequest) → LookupRouteResponse`

The core cache-aware routing hint. Unary and fail-open. It does not mutate cache membership or response-visible routing state. It emits metrics, and a delivered prefix hit under an LFU policy credits the matched entries’ internal eviction access counters.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

Both inference-cache binaries expose Prometheus metrics on their pod’s :8080/metrics, all prefixed inferencecache_*. The two binaries use separate registries — the server’s series cover the index and gRPC handlers; the controller’s cover the reconcilers. Standard go_* and process_* collectors are also present but are not part of this schema. Successfully injected PodLocal LMCache sidecars expose their upstream lmcache_mp_* metrics separately; the observability overlay includes a cross-namespace PodMonitor for them.

Server metrics (`inference-cache-server`)

Gauges

Metric	Labels	Meaning
`inferencecache_server_up`	—	`1` when the server is serving.
`inferencecache_server_grpc_tls_enabled`	—	`1` when gRPC TLS is configured.
`inferencecache_index_entries`	`model`	Distinct prefix keys in the index (excludes the reserved probe tenant). Drained models are zeroed, not left stale.

Counters

Metric	Labels	Meaning
`inferencecache_lookup_route_calls_total`	`model`, `reason_code`, `hint_used`	Lookups by outcome. `hint_used="true"` ⇔ a non-empty score list (`PREFIX_MATCH` / `TENANT_HOT` / `AFFINITY_HINT`).
`inferencecache_index_evictions_total`	`algorithm` (`lru`/`lfu`), `reason` (`cap`/`ttl`)	Index evictions. `reason="cap"` is the authoritative over-budget signal.
`inferencecache_tenant_evictions_total`	`tenant_id`, `reason` (`over_entries`)	Per-tenant quota evictions. A bounded non-zero rate is normal Fairness.
`inferencecache_snapshot_auth_total`	`result` (`ok`/`unauth`/`forbidden`/`error`)	`/snapshot` auth outcomes. Wrong audience lands in `unauth`.
`inferencecache_policy_auth_total`	`result`	`/policy` auth outcomes.
`inferencecache_probe_auth_total`	`result`	`/probe` auth outcomes.

Histogram

Metric	Labels	Buckets
`inferencecache_lookup_route_latency_seconds`	`model`	100µs, 250µs, 500µs, 1ms, 2.5ms, 5ms, 10ms, 25ms, 50ms, 100ms

Controller metrics (`inferencecache-controller`)

The controller pod has no Service in front of it — scrape it with a PodMonitor (the shipped observability bundle includes one) or pod-IP service discovery, or the controller-side alerts have no series to evaluate.

Reason codes

Mon, 01 Jan 0001 00:00:00 +0000

reason_code is a string, not a proto enum — new codes can be added without breaking old clients. Clients must treat any unrecognized code as the no-hint default (fail open, round-robin).

LookupRoute / LookupPDRoute

Code	When emitted
`PREFIX_MATCH`	A replica holds the exact prefix or a leading block-run; the ranker returns a non-empty list; at least one replica clears `minimumMatchedTokens` (default 64) and the top score clears `routingFloorScore` (default 0.1).
`NO_HINT`	The fail-open default: a novel prefix with no affinity fallback, an empty/unspecified key, cold start (globally empty index), a request gated below `minimumPrefixTokens`, a sub-floor matched-tokens/score result (with affinity disabled), or a disabled index.
`TENANT_HOT`	Prefix miss, `strategy.enableTenantHot ≠ false`, and a warm replica exists (recent stats, `hit_rate` above the floor, serving the requested scheme). `matched_tokens = 0`.
`AFFINITY_HINT`	Would-be `NO_HINT`, `affinityRouting: Enabled`, a usable fingerprint, and ≥1 serving replica. A single stable replica; `score` / `matched_tokens` / `estimated_cache_hit_prob` are all `0`.
`POLICY_REQUIRES_CHAIN`	`strategy.requireChain: true` and the request has no valid block-hash chain. Returned before touching the index.
`TIMEOUT`	The lookup deadline expired (context deadline, or `lookupTimeoutMs` elapsed). Fail-open. Clients may also synthesize this locally.
`UNKNOWN_TENANT`	Miss, and the (non-empty) `tenant_id` has zero entries anywhere (index not globally empty). A contract-key mismatch — likely a misconfigured client.
`UNKNOWN_MODEL`	Miss, tenant known, but `(tenant, model)` has zero entries.
`UNKNOWN_HASH_SCHEME`	Miss, `(tenant, model)` has entries, but none under the request’s `hash_scheme`.

LookupPDRoute is a stub today and always returns no hint.

CLI: inferencecache doctor

Mon, 01 Jan 0001 00:00:00 +0000

inferencecache doctor is a read-only pre-flight diagnostic — the cache-plane analogue of istioctl analyze. It runs against a live install, prints stable finding codes, and returns CI-friendly exit codes. The binary is bin/inferencecache (from make build).

inferencecache doctor -n serving

Checks and finding codes

After the Kubernetes API prerequisite (check 0), nine diagnostic checks run in a fixed order. Finding codes are stable and greppable; severity is shown in parentheses.

Reference on inference-cache

CRD API

v1alpha1 stability

CacheBackend

gRPC API

Consumer RPCs

LookupRoute(LookupRouteRequest) → LookupRouteResponse

Metrics

Server metrics (inference-cache-server)

Gauges

Counters

Histogram

Controller metrics (inferencecache-controller)

Reason codes

LookupRoute / LookupPDRoute

CLI: inferencecache doctor

Checks and finding codes

`LookupRoute(LookupRouteRequest) → LookupRouteResponse`

Server metrics (`inference-cache-server`)

Controller metrics (`inferencecache-controller`)