Building Secure Sandboxes for AI Agents

TL;DR

At Deductive, we are building an AI SRE agent that investigates production issues by reasoning across telemetry, code, and operational context. The moment we gave the agent the ability to execute generated code, it became dramatically more useful; it could inspect files, transform data, run diagnostic tools, and adapt its approach as an investigation unfolded. At the same time, it also became dramatically more dangerous. The same flexibility that lets Deductive solve an unfamiliar problem can theoretically give compromised or incorrectly generated code a potential path to the underlying production environment.

For us, this created a deceptively simple requirement: let agents execute arbitrary code without allowing that code to trust, reach, or affect anything beyond its explicitly authorized context. And because execution sits directly inside the agent’s reasoning loop, the isolation layer could not add seconds of latency to every step. It had to be secure enough for untrusted code, fast enough to feel invisible, and portable across cloud environments.

This is the story of how our first, straightforward architecture exposed a fundamental bottleneck, and how that failure led us from a centralized container proxy to a modular control loop, a direct mTLS data plane, and a one-way initialization mechanism we call sFuse.

Threat Model

It was tempting to begin by comparing sandbox technologies. But without first defining what we trusted, what we did not, and what an attacker could plausibly do, we would have been optimizing against an ambiguous security model. So before choosing a runtime, we made the trust boundary explicit.

Deductive’s architecture comprises (1) frontends such as the UI and webhooks, (2) backends responsible for inference and orchestration, and (3) the underlying infrastructure and platform layers. Across these layers, we drew a strict boundary between two classes of code:

1. Trusted code: code that has been reviewed and audited before execution.
2. Untrusted code: all code that doesn’t meet the trusted code criteria.

This distinction mattered because generated code is not necessarily malicious, but it must be treated as though it could be. From that boundary, three concrete attack vectors emerged:

1. Host and kernel escape: Compromised code exploits a shared Linux kernel vulnerability (e.g., Dirty Pipe) to escape the container boundary and gain host-level capabilities.
2. Lateral movement: The execution environment becomes a pivot point for reaching restricted internal networks.
3. Data Exfiltration: Generated code establishes an outbound connection to an attacker-controlled endpoint and transmits sensitive data outside the customer’s environment.

Evaluating Isolation Boundaries

With the threat model in place, we could evaluate isolation techniques against concrete failure modes rather than broad claims about “sandboxing”. The obvious starting point was a restricted Linux container. It was familiar, operationally simple, and easy to control with network policies. But it failed the most important part of our threat model: it still shares the host kernel. Once we treated kernel escape as a first-class risk rather than an edge case, several otherwise reasonable approaches fell away.  We evaluated 4 broad options:

1. Restricted Linux containers with deny-by-default network policies (e.g. Alpine Linux): Simple and configurable, but still dependent on the shared host kernel. This doesn’t protect against Vector 1 (Kernel Escape).
2. Nested Containers (Container-in-Container / Docker-in-Docker): CINC still shares the underlying host kernel. Furthermore, DinD typically requires privileged mode in both the Kubernetes cluster and the local container environment.  It may also share or bridge the inference layer’s network. Thus, it doesn’t adequately address Vectors 1 or 2 and introduces additional filesystem overhead for agent workloads.
3. Namespace wrappers (e.g., Bubblewrap, Flatpak): Highly performant and lightweight, but structurally similar to standard containers. They rely on the host kernel's namespace isolation, and therefore do not protect against Vector 1.
4. Third-party sandbox services: Offloading execution to external managed sandboxes introduces customer auditability and data governance concerns. Their underlying infrastructure is also opaque to us and our customers, making zero-day vulnerabilities harder to audit, patch, and fix.

The Implementation Choice: gVisor + Cilium

The evaluation revealed an important design insight: compute isolation and network isolation are separate problems, and no single primitive solved both. For compute, our threat model narrowed the decision to two credible boundaries: userspace kernels such as gVisor, and microVMs such as Kata Containers or Firecracker). Both addressed our central concern by placing a stronger boundary between untrusted code and the host kernel.

We selected gVisor. It provides a lightweight, memory-safe userspace kernel, implemented in Go, that intercepts and handles application system calls, substantially reducing exposure to the host kernel. From an infrastructure perspective, it requires fewer changes to the Kubernetes layer than custom microVM infrastructure and preserves portability across cloud providers (EKS, GKE, AKS).

gVisor addressed the execution boundary, but it did not address where a sandbox could connect. To address Vector 2 and 3 (lateral movement and data exfiltration), we leverage Cilium. By applying eBPF-based network policies, we block all egress traffic from the sandboxes by default, allowing only the specific FQDNs and IP addresses required by the agent's current context.

The Orchestration Challenge 

At this point, we had selected the right isolation primitives, but we did not yet have a usable system. A sandbox that takes a minute to appear, or adds noticeable latency to every tool call, breaks the feedback loop that makes an AI agent effective. Security determined the boundary; the agent workload determined the architecture around it.

The hard part only appeared after we tried to put those primitives into the execution path of a real agent. The actual engineering challenge and the focus of the remainder of this post lies in the architecture required to orchestrate these ephemeral sandboxes at scale. We needed a system that could provision and connect these isolated environments to AI agents without introducing prohibitive latency to the inference loop.

Version 1: Centralized Proxy

We deliberately began with the simplest architecture that could enforce the boundary end-to-end. A centralized container manager (CM) handled both the lifecycle of the gVisor sandboxes and the routing of execution commands. The core interface was straightforward: the inference layer invoked containermanager::run_command(cmd, env). This abstraction allowed the rest of the Deductive backend to execute generated code without reasoning about the underlying isolation model or command contention. It also gave us a fast path to production without prematurely overengineering the system.

Cold-Start Mitigation

The first problem surfaced immediately. Deductive investigations are interactive: an agent forms a hypothesis, executes a tool, observes the result, and decides what to do next. Waiting 60–120 seconds for Kubernetes to provision a node and pod at any step would destroy that loop. To bypass this, the CM maintained a pre-warmed pool of disposable sandboxes using a FIFO queue, backed by Karpenter for dynamic node provisioning.

This solved the first visible problem. Agents no longer waited for Kubernetes to create capacity before every execution. But as traffic increased, the system began exhibiting a more confusing failure mode: requests stalled even when more than a third of the warm pool was idle. Capacity was available, yet agents could not reach it efficiently.

Evolution I: The Modular Control Loop

The warm pool solved startup latency, but it also turned the CM into a stateful scheduler whose responsibilities were beginning to blur. Before attacking the next latency problem, we wanted to separate it from the growing complexity of the CM itself. We needed to support different runtimes, change allocation strategies without invasive rewrites, and track pods efficiently without lock contention. So we stepped back and decomposed the CM into three essential components:

1. The Container Allocator(CA): an abstract interface that handles the lifecycle of a container unit, regardless of whether it is a gVisor container, a microVM, a docker-in-docker environment, or a process. This allows the CM to swap the underlying runtime without changing its orchestration logic.
2. The Container State Operator(CSO): A dedicated component that manages state transitions for execution units.
3. The Estimator (Predictive Logic): a pluggable control-policy interface. Today, it can use simple heuristics, but it is designed to support PID, MPC, or AI policies without changing the Allocator or State Operator.

By modularizing these components, we transformed the CM from a monolithic block into a control-loop system. This provided the flexibility to iterate on allocation strategies and runtime backends independently. However, while this improved maintainability, our profiling still showed a fundamental performance bottleneck: the proxy path.

That distinction mattered. The modular control loop made the system easier to evolve, but it did not shorten the critical path. We had improved the shape of the control plane without questioning why execution traffic passed through it at all.

Evolution II: Decoupling Control Panel and Data Plane

The production symptoms were initially counterintuitive:

• Agent’s commands experienced high latency.
• Command execution latency increased under load.
• Sandbox allocation stalled periodically.
• System telemetry showed these stalls occurred even when >37% of the pre-warmed container pool remained idle.

Together, these signals pointed away from insufficient capacity and toward the centralized proxy itself. The architecture inherently introduced five problems:

1. Security Blast Radius: Because CM handles sandbox provisioning and command forwarding, a compromised CM instance could expose execution context across active agents.
2. Double-hop Latency: Every execution required two network hops: inference -> CM -> sandbox.
3. Lock Contention: The CM had to maintain stateful locks mapping SandboxKey(agent_id, other_unique_id) to specific sandboxes. Under load, these locks stalled the CM and bottlenecked agent execution.
4. Kubernetes API rate limits: The CM polled the K8S API to verify sandbox state and route commands. Even with a nearly idle CM and only one active sandbox, this polling could trigger API throttling.
5. Single point of failure: if the CM is overloaded (due to (2)-(4)), it could enter a crash loop, interrupting every active agent.

With a modular control loop in place, the system was maintainable, but it still relied on the CM as a proxy. Every request flowed through the control path, introducing latency and lock contention. The lesson was subtle: separating responsibilities in code was not enough; we also had to separate them on the network path. To reach our performance and security goals, we decoupled the Control Plane from the Data Plane.

The new rule was simple: the CM could allocate a sandbox, but it could not remain in the execution path. Once allocation was completed, the agent needed to communicate directly with its sandbox.

Implementing the Data Plane: gRPC, mTLS, and sFuse

Decoupling removed the CM bottleneck, but it also removed the component that authenticated every request. The sandbox could no longer be a passive process wrapper. It had to become a dedicated execution server with its own protocol, identity model, and connection lifecycle. Four design decisions made that possible.

1. The Protocol: Strict Contracts via gRPC. We skipped REST and JSON entirely. This path would carry commands, files, and streaming results on nearly every step of a Deductive investigation, so the protocol itself needed to be efficient and difficult to misuse. We chose a strict, schema-driven contract that minimized serialization overhead and prevented schema drift. gRPC provides a strongly typed, portable interface over HTTP/2 and supports efficient streaming directly into the execution environment. For our workload, this substantially outperformed discrete HTTP payloads or intermediate object storage, such as S3, for transferring code and data.

2. The Trust Anchor: mTLS. A direct data path removed latency, but it also removed the CM as the gatekeeper for every request. That tradeoff was acceptable only if trust moved with the connection rather than disappearing from it. Without a central proxy to validate requests, we needed to prevent man-in-the-middle (MITM) attacks and unauthorized command execution. The core question was bidirectional: how could a Deductive agent verify the sandbox’s identity, and how could the sandbox reject commands from any other client?

We leveraged mutual TLS (mTLS). Because both Python (the inference side) and Go (the sandbox side) natively support mTLS, we can enforce a strict one-to-one trust relationship between a specific agent session and a specific sandbox.

3. The sFuse Handshake (Soft-Fuse). The critical security challenge in decoupled systems is the key exchange. If the Container Manager holds the keys, it remains a high-value target. If the sandbox keeps its provisioning port open, it can be hijacked. We resolved this using a mechanism we call sFuse (Soft-Fuse), inspired by Samsung’s Knox - a chip-level fuse that protects against sensitive data leaks, but designed for ephemeral lifecycles.

This was the subtle part of the design. Direct communication improved performance, but a conventional long-lived provisioning endpoint would have created a new attack surface. We needed the CM to bootstrap trust exactly once and then permanently lose the ability to reconfigure that sandbox.

The sFuse initialization follows a strict sequence:

1. Request: The agent generates an in-memory keypair and passes its public key to the Container Manager (CM) when requesting a sandbox.
2. Allocation: The CM pulls a sandbox from the Allocator pool and calls the sandbox's initializeTLS endpoint. (Note: Cilium eBPF network policies restrict this endpoint so it is only reachable by the CM).
3. Exchange: The sandbox generates its own keypair, receives the agent's public key, and returns its own public key to the CM. The CM passes this back to the agent without retaining the keys in memory.
4. The Fuse Blows: The moment the initializeTLS gRPC response is sent, the sandbox permanently shuts down its control panel listener.

By "blowing the fuse," the initialization channel is permanently closed. The sandbox becomes an immutable execution environment that can only communicate over the established mTLS channel, contained by the eBPF policy fabric.

The CM remains responsible for allocation, but it is no longer trusted with ongoing execution traffic or persistent session credentials. Compromising it after initialization does not provide a path to commandeer an already-bound sandbox through the closed control-plane endpoint.

4. The Immutability Principle (Zero Reuse). Once we treated generated code as potentially adversarial, it became difficult to justify sandbox reuse. A common industry optimization is to "clean" and reuse sandboxes to save compute time. We explicitly reject this pattern for Deductive’s agent workloads.

If a container is compromised, reusing it silently expands the blast radius to subsequent agents. Furthermore, implementing reliable "cleanup" logic introduces complexity into the State Operator and Estimator modules. By treating sandboxes as strictly single-use, disposable entities, we trade a modest compute overhead for strong workload isolation, no cross-session state reuse, and a vastly simplified control loop.

Performance Impact: The decoupled architecture yielded immediate results. For a pre-warmed pool, control plane allocation latency dropped to P99 < 21ms. Once handed over, the direct mTLS data plane reduced per-command execution latency and overhead to <10ms.

Optimizing Agent Performance Inside the Sandbox

By now, Deductive could acquire a strongly isolated sandbox and communicate with it directly. Yet end-to-end investigations were still slower than the per-command benchmarks suggested. The discrepancy stemmed from the workload shape: agents rarely perform a single large operation. They perform thousands of small reads, writes, searches, and commands whose fixed costs compound. This forced us to optimize the agent-sandbox interface as a workload, not merely a collection of APIs.

1. The I/O Tax: Overcoming the 3,000-File Bottleneck

Agentic workflows can generate thousands of small files. For example, an agent might invoke sandbox.run_command(“mkdir -p /workspace && cat <<syscl> /workspace/immediate.txt”) repeatedly for small filesystem operations. With N-files to read or write, this produces O(N) discrete operations and accumulates substantial filesystem and network latency.

This architecture performs poorly. Invoking a shell requires starting a comparatively heavyweight, less-restricted bash process inside the sandbox, executing the command, and serializing stdout/stderr back over the network. This adds process-creation, system-call, and serialization overhead that can accumulate into seconds of latency.

From a security perspective, raw shell commands also introduce command-injection risk.

We replaced raw shell execution with native, restricted filesystem primitives built directly into our gRPC data plane: Read, Write, List, and Grep/Find, and Symlink. Moving these operations to native Go implementations inside the sandbox server accomplished two things: it tightly restricted the permitted filesystem operations and paths, thereby shrinking the attack surface, and it eliminated shell process initialization overhead.

The first optimization had merely moved the bottleneck: from shell startup to network RTT. The primitives were safer and individually faster, but thousands of sequential calls still added up to an unacceptable delay. The solution was to make batching part of the filesystem protocol itself:

• Bin-Packing Writes: Because gRPC enforces a default 4MB payload limit, we implemented a client-side algorithm to bin-pack small files. We pack workloads into a single async batch up to 3.5MB (leaving 500KB as a buffer for Protobuf framing and metadata). Files exceeding 3.5MB are offloaded to a native streaming queue.
• Path Pruning: Batching directory creation (mkdir) can still trigger redundant, slow-path system calls. We introduced a client-side path-merge operator that prunes redundant paths before transmission. For example, a request for [a/b/c, a/b/d, a/b] is optimized to just [a/b/c, a/b/d].
• Two-Step Plan Reads: Dynamically reading unknown file sizes from the server without overwhelming the channel requires an explicit strategy. The client first invokes GetReadFilesPlan. The sandbox server evaluates the file targets, determines their sizes, and generates an execution blueprint. The sandbox_client::ReadFiles runner then executes the plan blindly, streaming large files and batching small ones.

The Result: We consolidated thousands of sequential blocking network operations into 8-9 asynchronous gRPC batches. Workspace materialization dropped from 30+ seconds to under 4 seconds.

That addressed throughput, but concurrency introduced a different class of problem. Faster parallel execution is useful only if the resulting state remains predictable.

2. The Concurrency Tax: Deterministic State via Lazy Commits

Batching made I/O fast enough, but Deductive also executes independent investigative steps concurrently. That surfaced a different correctness problem: parallel commands shared a logical environment, while the operating system completed them in an inherently nondeterministic order. Without an explicit state model, identical investigation plans could produce different environments.

To maximize execution velocity, our platform allows agents to run sandbox commands concurrently. However, concurrent execution in a shared runtime introduces severe race conditions regarding the shell environment (e.g., two parallel tasks attempting to export to same environment variables simultaneously). Locking the entire execution runtime to synchronize state defeats the purpose of concurrency.

We resolved this tension by decoupling command execution from state application using a lazy-commit state machine keyed by unique task_group_id values.

The key insight was that physical completion order should not determine logical state. A slow command issued first should not overwrite the state of a faster command that was issued later merely because it happened to finish last.

Instead of mutating the base environment synchronously, the sandbox manages concurrency through a strict logical lifecycle:

1. Monotonic Ordering: Every command issued within an active execution turn is assigned a monotonically increasing cmdExecLogicalOrder sequence counter.
2. Buffered Deltas: While parallel commands run, they leave the base environment completely untouched. Instead, their mutations (upserts and removals) are captured as independent EnvDelta structs and buffered in memory.
3. Lazy Evaluation: The runtime delays the state merge until a new task_group_id hits the server, signaling the transition to a new logical turn. Only then does the system acquire a lock to flush the state.
4. Deterministic Resolution: During the flush, the buffered deltas are sorted strictly by their logical sequence numbers, rather than their wall-clock completion times. The state is applied sequentially, guaranteeing that the logically last command wins any conflict.

By buffering execution deltas and evaluating them lazily, the data plane supports highly parallel agent execution while preserving deterministic environment state for subsequent steps.

If there are N commands issued in the same task group N, with a maximum M environment variable changes, it is very obvious that it will take O(NlogN) to sort and O(N*M) memory space for merging and generating the BaseEnv for the next task group. Notice that only the last logical order environment variable matters (last-write-win), and we can easily improve as follows:

1. BaseEnv is now Copy-On-Write, no mutation needed, thus no lock is needed
2. Added an overlay that replaces the EnvDelta; when a command finishes, update the environment variable key by its logical order
3. When a new task_group_id comes in, apply the overlay on the BaseEnv to get the new BaseEnv

With this change, no sorting is needed, and no logs are generated for each environment change.

Conclusion

At Deductive, we began with what looked like a container-isolation problem. In production, it became clear that the real problem crossed three boundaries at once: kernel isolation, network trust, and orchestration latency. Solving only one of them produced a system that was either unsafe, operationally heavy, or too slow for an agent’s execution loop.

Securing runtime environments for AI agent execution requires moving beyond standard container boundaries. By combining core zero-trust design principles with powerful open-source technologies, this Container Manager architecture demonstrates how agents can run in a strictly isolated environment without introducing unnecessary infrastructure complexity.

The most important engineering result lies in how the architecture evolved to match real production workload patterns. Transitioning from a stateful, centralized proxy to a modular control loop and a decoupled sFuse data plane brought our allocation latency down to P99 < 21ms and command execution overhead to < 10ms. Coupled with batched filesystem primitives and lazy-commit state tracking, this system shows that strong isolation and high-performance concurrency need not be mutually exclusive, even when handling arbitrary, potentially adversarial workloads in production.

The broader lesson is that secure agent execution cannot be bolted onto an existing request path as another infrastructure layer. The isolation boundary, trust bootstrap, data plane, and state model must be designed together. Once we treated them as one system, both the security model and the performance characteristics became substantially simpler.