Multi-Tenant AI Infrastructure: The 5 Isolation Layers That Determine Whether Your Customers’ Data Stays Separate
A production architecture guide covering vector store isolation, knowledge graph boundaries, KV-cache side channels, event stream partitioning, and session state scoping with code.
The question sounds operational. “How do you serve 200 customers from one AI system?”
The dangerous assumption is that it’s the same problem you’ve already solved in your SaaS product. Separate databases. Tenant ID in every query. Auth middleware that validates the JWT. Done.
In AI infrastructure, that assumption is wrong in a specific and consequential way.
The isolation surface in AI systems extends into layers that traditional software doesn’t have: a vector similarity index, a knowledge graph traversal engine, an inference server managing shared GPU memory, an event log that doubles as your compliance record, and an in-memory state cache that spans multiple agent turns. Each of these layers can fail independently. Most of them fail silently.
This post covers each layer, the failure mode, and the pattern I’m using in NEXUS to address it.
Why AI Multi-Tenancy Is Harder
In a relational database, tenant isolation is a solved problem. Foreign keys, row-level security policies, separate schemas. The database enforces boundaries. Violations throw exceptions.
AI infrastructure doesn’t have the same enforcement mechanisms.
When a vector similarity search returns results, it doesn’t tell you if those results crossed a tenant boundary. The cosine similarity score doesn’t include a provenance check. Your KV cache server doesn’t know which tenant “owns” a cached key. Your inference server doesn’t know that two requests from different tenants shouldn’t share memory state.
All of that has to be engineered by you. Nothing enforces it by default.
There are five distinct isolation surfaces. Let’s work through each.
Layer 1: Vector Store Isolation (Qdrant)
The naive implementation: one collection, tenant_id in the payload, filter on every query.
from qdrant_client import QdrantClient
from qdrant_client.http.models import Filter, FieldCondition, MatchValue
client = QdrantClient(host="localhost", port=6333)
# Naive: single collection, filter per query
results = client.search(
collection_name="documents",
query_vector=query_embedding,
query_filter=Filter(
must=[
FieldCondition(
key="tenant_id",
match=MatchValue(value=tenant_id)
)
]
),
limit=10
)This works until it doesn’t. The problem is that the filter is caller-enforced, not schema-enforced. If any code path forgets to pass the filter, or passes the wrong tenant_id, the query runs against the full collection. Qdrant returns results. No exception is raised. No log entry marks the violation.
The more robust pattern is separate collections per tenant.
# Robust: per-tenant collection
collection_name = f"tenant_{tenant_id}_documents"
results = client.search(
collection_name=collection_name,
query_vector=query_embedding,
limit=10
)The collection name becomes the access control boundary. A misconfigured query for tenant_abc literally cannot reach tenant_xyz's collection because it's a different object in the system. The cost is overhead in collection management. For NEXUS, I'm maintaining a tenant registry in Dragonfly that maps tenant IDs to their collection names and validates the mapping on every request.
import dragonfly # same API as redis-py
registry_client = dragonfly.StrictRedis(host="localhost", port=6379)
def get_tenant_collection(tenant_id: str) -> str:
key = f"tenant:{tenant_id}:qdrant_collection"
collection = registry_client.get(key)
if not collection:
raise ValueError(f"No collection registered for tenant {tenant_id}")
return collection.decode("utf-8")The payload filter still exists as a secondary guard, but the primary boundary is the collection name.
Tradeoff to understand: Separate collections multiply your Qdrant resource usage proportionally. At 200 tenants, that’s 200 HNSW graphs in memory. Evaluate whether your vector dimensions and collection sizes justify this. For tenants with small datasets, a tiered approach (shared collection for small tenants, dedicated collection past a size threshold) is a practical middle ground.
Layer 2: Knowledge Graph Isolation (Neo4j)
Neo4j has two isolation options at the database level:
Option A: Separate database per tenant Clean. Hard boundaries. Expensive at scale. Spinning up a Neo4j database for each of 200 tenants is operationally heavy and resource-intensive. Best reserved for enterprise customers with strict data residency requirements.
Option B: Label-based isolation in a shared database More practical. Every node gets a tenant_id property. Every relationship traversal includes a WHERE clause.
// Safe: explicit tenant scoping
MATCH (e:Entity {tenant_id: $tenant_id})-[:RELATES_TO]->(related:Entity {tenant_id: $tenant_id})
WHERE e.name = $entity_name
RETURN relatedThe failure mode is the missing WHERE clause on join conditions.
// DANGEROUS: tenant_id on source node but not on traversal target
MATCH (e:Entity {tenant_id: $tenant_id})-[:RELATES_TO]->(related:Entity)
RETURN related
// 'related' nodes could belong to any tenantThis query returns related entities across all tenants. Neo4j executes it without error. Your application receives cross-tenant data with no indication that anything went wrong.
The discipline required: every Cypher query that traverses relationships must include tenant_id constraints on both the source and all traversal targets. For NEXUS, I’m enforcing this with a query builder wrapper that injects tenant_id into every WHERE clause automatically.
from neo4j import GraphDatabase
class TenantScopedNeo4j:
def __init__(self, uri: str, auth: tuple, tenant_id: str):
self.driver = GraphDatabase.driver(uri, auth=auth)
self.tenant_id = tenant_id
def run_scoped_query(self, query: str, params: dict) -> list:
# Inject tenant_id into every query automatically
params["_tenant_id"] = self.tenant_id
with self.driver.session() as session:
result = session.run(query, params)
return [record.data() for record in result]Combined with a graph schema convention where every node creation requires tenant_id as a mandatory property, this gives you enforcement without the overhead of separate databases.
Layer 3: Inference Layer Isolation (the KV-Cache Problem)
This is the layer that catches people off guard, because the failure mode isn’t a misconfigured query. It’s a performance optimization that becomes a side channel attack.
How KV-cache sharing works:
vLLM and SGLang share KV cache across requests that start with identical token sequences. If Tenant A sends a request beginning with “Analyze this contract and identify risk clauses:”, and Tenant B sends the same prefix, the inference server reuses Tenant A’s computed KV cache for the shared prefix. This reduces compute and GPU memory pressure. It’s a meaningful optimization.
The security risk, documented by Wu et al. at NDSS 2025, is that the Longest Prefix Match scheduling policy that enables KV-cache sharing also creates a timing side channel. The time-to-first-token for a request is measurably faster when there’s a KV cache hit. An adversary can exploit this timing difference to craft requests that probe whether specific token sequences exist in another tenant’s cached prompt, then iteratively reconstruct that prompt token by token.
Their PROMPTPEEK attack achieved:
- 99% accuracy when the adversary knows the prompt template
- 98% accuracy reversing the template itself
- 95% accuracy with zero background knowledge
These aren’t theoretical results. They ran this against a Llama2–13B model on an A100 80G GPU, across four real datasets.
The mitigation:
Per-tenant KV cache isolation. In vLLM, this means disabling prefix caching globally or scoping cache keys by tenant.
# vLLM engine initialization with per-tenant cache isolation
from vllm import AsyncLLMEngine, AsyncEngineArgs
engine_args = AsyncEngineArgs(
model="meta-llama/Llama-2-13b-chat-hf",
enable_prefix_caching=False, # Disable cross-tenant cache sharing
# OR: implement tenant-scoped cache key hashing in a custom scheduler
)Disabling prefix caching entirely is the safe option. The cost is real: TTFT increases, especially for requests with long system prompts. The more surgical fix is implementing a tenant-aware cache key that includes the tenant ID in the hash, so cache hits only occur within the same tenant’s request history.
# Custom cache key hashing (pseudocode for vLLM extension)
def compute_cache_key(tokens: list[int], tenant_id: str) -> str:
import hashlib
token_str = str(tokens)
return hashlib.sha256(f"{tenant_id}:{token_str}".encode()).hexdigest()The tradeoff is explicit: security vs. latency. For NEXUS, I’m using tenant-scoped cache keys rather than disabling caching entirely, which preserves within-tenant cache reuse while eliminating cross-tenant exposure.
Layer 4: Event Stream Isolation (Redpanda)
The event sourcing architecture from Day 10 introduces another isolation surface: your audit log.
Two approaches for Redpanda multi-tenancy:
Option A: Per-tenant topics
agent-events-tenant-abc
agent-events-tenant-xyzPros: per-tenant retention policies, clean ACL boundaries, simple compliance reporting per customer. Cons: topic proliferation at 200 tenants (200 topics minimum, likely 600+ with staging/prod/DLQ variants).
Option B: Shared topic with tenant partition key
from redpanda import KafkaProducer
import json
producer = KafkaProducer(bootstrap_servers=["localhost:9092"])
def publish_agent_event(tenant_id: str, event: dict):
event["tenant_id"] = tenant_id # Always in payload
producer.send(
topic="agent-events",
key=tenant_id.encode("utf-8"), # Partition key ensures ordering per tenant
value=json.dumps(event).encode("utf-8"),
headers=[("tenant_id", tenant_id.encode("utf-8"))] # Also in header
)The partition key ensures all events for a given tenant land on the same partition, which gives you ordered event replay per tenant without per-tenant topics. The consumer group pattern then isolates which service consumes which tenant’s partition stream.
For NEXUS, I’m using Option B (shared topic, tenant partition key) during development and planning to migrate high-value tenants to dedicated topics for compliance isolation.
Critical rule: tenant_id must be in the event header, not just the payload. Header-level filtering is available to Redpanda’s broker-side filtering and ACL policies. Payload-level filtering requires your consumer to read and parse the message first.
Layer 5: Session State Isolation (Dragonfly)
The simplest layer to implement. Also the most frequently skipped.
import dragonfly
class TenantScopedCache:
def __init__(self, tenant_id: str, session_id: str):
self.client = dragonfly.StrictRedis(host="localhost", port=6379)
self.tenant_id = tenant_id
self.session_id = session_id
self.key_prefix = f"tenant:{tenant_id}:session:{session_id}"
def set_agent_state(self, agent_id: str, state: dict) -> None:
key = f"{self.key_prefix}:agent:{agent_id}:state"
self.client.setex(
key,
3600, # 1 hour TTL
json.dumps(state)
)
def get_agent_state(self, agent_id: str) -> dict | None:
key = f"{self.key_prefix}:agent:{agent_id}:state"
data = self.client.get(key)
return json.loads(data) if data else NoneThe key namespace is the isolation boundary. Without the tenant prefix, one agent’s state write can overwrite another tenant’s active session. This is especially dangerous in LangGraph where checkpointing writes state between nodes. Every checkpoint key in NEXUS includes tenant_id.
The discipline check: search your codebase for client.set( or client.get( calls without tenant:{id} in the key string. Any bare key is a potential cross-tenant state collision.
Noisy Neighbor Prevention
Isolation prevents data leakage. It doesn’t prevent resource contention.
Tenant A triggers a batch re-indexing operation. 50,000 vectors, Qdrant upsert, Neo4j write-heavy traversal, vLLM warm-up with a long system prompt. All at 2am. Your inference pool saturates. Tenant B’s interactive query arrives at 2:01am. Latency spikes from 480ms to 4,200ms. Tenant B calls support. You have no explanation.
The controls:
1. Per-tenant rate limiting at the API gateway
from fastapi import FastAPI, HTTPException, Request
from fastapi_limiter import FastAPILimiter
from fastapi_limiter.depends import RateLimiter
app = FastAPI()
# Per-tenant rate limit: 100 requests/minute
async def tenant_rate_limiter(request: Request):
tenant_id = request.headers.get("X-Tenant-ID")
return f"tenant:{tenant_id}"
@app.post("/query")
async def query_endpoint(
request: Request,
_: None = Depends(RateLimiter(times=100, seconds=60, identifier=tenant_rate_limiter))
):
...2. Per-tenant token budget at the inference router
Before routing a query to vLLM, check whether the tenant has remaining token budget for the current billing period. Reject or queue requests that exceed the budget. This is enforced at the router layer, not the inference layer, so saturated tenants don’t consume GPU time.
3. Per-tenant observability lanes
Day 7’s OpenTelemetry setup pays off here. If tenant_id is a span attribute on every trace, you can build a per-tenant latency dashboard. When Tenant B spikes, you can answer: “Was this caused by Tenant A’s batch job consuming the inference pool?” in 30 seconds by filtering spans by tenant_id and looking at queue depths.
NEXUS Implementation Status
At Day 11 of the build, isolation is now wired end-to-end in NEXUS:
- Per-tenant Qdrant collections with a registry in Dragonfly
- Neo4j label-based isolation with a TenantScopedNeo4j query wrapper that injects tenant_id into every parameterized query
- Tenant-scoped KV cache keys in the vLLM configuration
- Redpanda partitioning by tenant_id with tenant_id in every event header
- Dragonfly key format:
tenant:{id}:session:{id}:agent:{id}:* - LangGraph checkpoint keys namespaced by tenant_id
- Per-tenant rate limiting middleware in FastAPI
- tenant_id as a span attribute on every OpenTelemetry trace
The next challenge: as tenant count grows, the Dragonfly tenant registry becomes a single point of failure. Day 12 covers self-healing pipelines, including what happens when the registry goes down and agents can’t resolve their collection names.
What to Read
📎 Paper: “I Know What You Asked: Prompt Leakage via KV-Cache Sharing in Multi-Tenant LLM Serving”-Wu, Zhang, Zhang, Wang, Niu, Wu, Zhang (NDSS 2025)
Read in order:
- Figure 1 (page 1): The KV cache reuse mechanism between two users
- Figure 3 (page 3): The full multi-tenant system model and where the side channel lives
- Section VI (pages 7–9): Evaluation across three adversary knowledge scenarios
📚 Book: “Building Microservices” by Sam Newman (O’Reilly, 2nd ed., 2021)
- Chapter 11 (Security): Auth boundaries, data in transit, encapsulating services -the principles map directly to agent isolation boundaries
- Chapter 13 (Scaling): Horizontal scaling, partitioning strategies, and noisy neighbor patterns at service scale
90 minutes total. The PROMPTPEEK paper is short and the Wu et al. evaluation section alone is worth the read.
This is Day 11 of a 20-day series on production AI infrastructure. Yesterday I covered event sourcing and audit trails with Redpanda. Tomorrow: self-healing pipelines and why retrying everything is a system design mistake.
