The gRPC contract

The InferenceCache service

The InferenceCache gRPC service is the data-plane API that gateways and engines integrate against. It is defined in proto/inferencecache/v1alpha1/inferencecache.proto, package inferencecache.v1alpha1, and served on :9090 (plaintext by default; TLS is an opt-in overlay — see gRPC TLS).

Eight RPCs, three roles

Consumer side (gateways) — fail-open; only delivered LFU hits update eviction bookkeeping:

Producer side (engine adapters):

Observer side (dashboards / debug) — server-stream stubs:

Key messages

message LookupRouteRequest {
  string model_id             = 1;
  string tenant_id            = 2;
  bytes  prefix_hash          = 3;
  int32  prefix_token_count   = 4;
  string hash_scheme          = 5;
  SLO    slo                  = 6;
  repeated bytes block_hashes = 7;   // ordered block-hash chain
  repeated int32 block_token_counts = 8;
  repeated uint32 token_ids   = 9;   // dual-input: server fingerprints
  string prompt_text          = 10;  // dual-input: server tokenizes + fingerprints
  string adapter_id           = 11;  // LoRA index partition
}

message LookupRouteResponse {
  repeated ReplicaScore replica_scores = 1;
  string reason_code                   = 2;
  int64  lookup_latency_us             = 3;
  repeated uint32 token_ids            = 4;  // set only when prompt_text was tokenized
  string adapter_id                    = 5;  // partition consulted
}

enum CacheTier {
  CACHE_TIER_UNSPECIFIED = 0;
  CACHE_TIER_T1          = 1;
  CACHE_TIER_T2          = 2;
  CACHE_TIER_T3          = 3;
}

message ReplicaScore {
  string replica_id               = 1;
  float  score                    = 2;
  int32  matched_tokens           = 3;
  float  estimated_cache_hit_prob = 4;
  CacheTier tier                  = 5;
}

message PrefixEntry {              // metadata only — never KV tensors or prompt text
  bytes  prefix_hash        = 1;
  int32  token_count        = 2;
  repeated bytes block_hashes       = 3;
  repeated int32 block_token_counts = 4;
  CacheTier tier                    = 5;
  string adapter_id                 = 6;
}

message ReplicaStats {
  string replica_id        = 1;
  int64  cache_memory_bytes = 2;  // engine total across all tenants
  float  hit_rate          = 3;
  float  pressure          = 4;
  string client_version    = 5;
  int64  t2_hit_tokens     = 6;   // from vLLM external_prefix_cache_hits_total
  int64  t2_query_tokens   = 7;   // from vLLM external_prefix_cache_queries_total
}

Ack { bool accepted; string reason_code; } and SLO { int32 ttft_ms; int32 tbt_ms; } round out the core shapes.

The guarantees

These are the contract properties clients are allowed to depend on:

• reason_code is a string, not a proto enum. New codes can be added without breaking old clients — a client that does not recognize a code must treat it as the no-hint default.
• Fail-open. An empty replica_scores list is always valid. TIMEOUT is a fail-open outcome, not an error. The hot path never returns a gRPC error for a cache miss.
• No cache-membership mutation on lookups. A lookup does not add, remove, or rewrite indexed prefixes. Metrics are emitted, and an LFU namespace credits internal eviction access counters on a delivered prefix-match hit.
• Engine-opaque hashes, scheme-scoped. prefix_hash / block_hashes are matched only within a matching hash_scheme; an empty scheme is not a valid domain.
• Metadata-only index updates. CacheStateUpdate and PrefixEntry carry hashes and statistics — never KV tensors or prompt text. LookupRoute may carry optional prompt_text input, which strengthens the case for enabling TLS on :9090.
• CacheStateUpdate is an additive delta, not a snapshot. A prefix’s absence from a later update does not remove it. Removals come only via CacheEvent (PREFIX_EVICTED / ALL_CLEARED) or TTL. This matches the engine’s own KV-event model, and it is why a stale entry degrades to a cache miss, never a wrong answer.
• Reserved tenant. tenant_id = "inferencecache.io/probe" is reserved for the server’s functional self-test — excluded from /snapshot and from public reads and writes.

The producer path

Engine adapters feed the index through the observation sidecar:

• ReportCacheState is the authoritative, idempotent ingest keyed on (replica, hash_scheme, adapter_id, prefix_hash). It carries PrefixEntry metadata plus ReplicaStats; an entry-level adapter_id falls back to the update-level value.
• PublishEvent carries scheme-safe deltas. PREFIX_ADDED is a no-op (events carry no hash_scheme, and ReportCacheState is authoritative for adds); REPLICA_UPDATED refreshes replica stats; PREFIX_EVICTED / ALL_CLEARED remove.

Keeping the contract honest

Because there are external consumers of this contract (gateway clients in other languages), the proto stays backward-compatible — fields are not renumbered or removed. Any change to proto/ is required to update the contract design doc in the same change, and generated code is verified drift-free in CI.

Related pages

• Reason codes — the full string-code vocabulary.
• gRPC API reference — RPC-by-RPC detail.
• LookupRoute & ranking — how the hint is computed.