Administration on inference-cache

Observability & Alerts

Mon, 01 Jan 0001 00:00:00 +0000

The bundle

A Prometheus alert bundle for the operational silent-failure patterns this system has hit in production ships under config/observability/. It is not part of config/default — the alerts are opt-in so that installs without prometheus-operator CRDs are not affected by an unknown apiVersion.

Index sizing

Mon, 01 Jan 0001 00:00:00 +0000

The server’s cache-state index lives in memory. This page helps you size the server pod and choose eviction settings so the index never approaches the pod’s memory limit (an OOM-kill is the one failure the soft-state design cannot hide).

Pod budget at a glance

Storage entries (distinct prefix keys × replicas reporting each)	Peak RSS	Recommended pod memory
100K storage entries	~110 MiB	256 MiB
500K storage entries	~300 MiB	512 MiB
1M storage entries (the default cap)	~540 MiB	1 GiB (recommended floor)
1.5M storage entries	~700 MiB	1.5 GiB

These include roughly 20% headroom over measured heap. The default entry cap is 1,000,000.

gRPC TLS

Mon, 01 Jan 0001 00:00:00 +0000

The decision

The gRPC API on :9090 uses one-sided Service TLS via cert-manager, terminated in-process — no sidecar, Envoy, or Ingress. Mutual TLS (client-certificate auth) is a Phase-2 feature flag, not yet implemented.

The server binary supports TLS via --tls-cert-file / --tls-key-file (both-or-neither — supplying exactly one fails startup). The certificate is served through a reloading GetCertificate hook, so a cert-manager-rotated Secret is picked up without a restart. The posture is exposed as the gauge inferencecache_server_grpc_tls_enabled (0/1).

Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Reading CacheBackend readiness

A managed backend’s Ready is the composition of three gates (managed-readiness → KV-event gate → functional-probe gate), so the READY column of kubectl get cachebackend only tells half the story. The reason lives in .status.conditions[], and the condition .reason is the actionable string: