In this episode of Alexa’s Input (AI), I sit down with Yan Avlasov, Staff Software Engineer at Google and a senior maintainer of Envoy, to talk about building the network for AI agents.
We discussed the complications of agentic networking, including the need for the infrastructure to manage agents with their own permissions and state and to account for bursty traffic patterns at scale. Yan also shared how AI is helping identify security vulnerabilities and the work involved in fixing them in systems people already depend on. A great, practical conversation on an important topic for engineers building AI platforms.
Key takeaways
These decisions affect whether agents can keep working reliably, within their permissions and budget:
Scaling agents means managing more than traffic. Each agent can have its own permissions and state, changing how the infrastructure loads policies and routes requests.
Keeping requests moving between AI providers takes more than a retry. Switching models can require a different request format, while spending limits constrain which models an agent can use.
Pausing agents saves resources but makes routing harder. The network may need to wake an agent and locate it on another machine before delivering a request.
A security check is only useful if it protects the request that gets executed. Transformations after the check can create a policy bypass.
AI can uncover vulnerabilities faster than teams can safely fix them. Finding combinations of bugs is a major advance; resolving them still requires careful changes to systems already in production.
Alexa’s Input (AI) is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Watch the full episode: YouTube | Spotify
Load policies when they are needed, and plan for a failed lookup
A policy that once covered an entire application may now be different for every agent. At Google’s scale, Yan says, keeping all of those policies in every proxy was not feasible. His team redesigned the system to load policies as requests arrive and remove them when they are no longer needed. That lets the proxies serve more agents without holding every agent’s configuration in memory.
Those memory savings introduce a dependency. When a proxy already has a policy, it can use it immediately. When it doesn’t, it needs to retrieve the policy from the control plane before applying it to the request. If that lookup fails, the proxy cannot simply proceed as though the agent has permission. Yan describes the difficulty of making dynamic policy loading work with the availability protections Google had already built. The challenge is to support more individual policies while keeping requests moving through disruptions.
Pausing agents saves resources but changes request delivery
Policies are only part of what the infrastructure must manage. Agents also carry their own working state, even while they wait. Yan gives the example of a coding agent: it produces code, then sits idle while the user reviews it. Keeping all those agents running would consume resources during periods when they have no work to do.
Pausing them saves resources, but the next request becomes more complicated. The network must wake the agent, wait for it to resume, and find it—even if it is now running on a different machine. Yan describes Project Substrate as a way to handle this at scale, alongside isolation and networking policies. Saving compute depends on being able to resume the right agent and deliver its next request.
Keep fallback requests within the agent’s budget
Once a request reaches an agent, the infrastructure may also need to choose which model will handle it. Yan contrasts this with retrying a request against another identical server. If one AI provider is unavailable or its token allowance is exhausted, the replacement may be a different provider with a different request format. The proxy must understand and translate the request for the retry to work.
Choosing a model also needs to respect spending limits. Yan says customers want budgets for individual agents or users, but the final cost of an inference request is known only after it finishes. The gateway must decide whether to allow another request while earlier requests are still accumulating costs. That makes enforcing a budget harder than counting requests; estimating those unfinished costs is part of the problem, and Yan says there is no perfect solution yet.
Semantic routing adds another decision, which model suits the task? Yan describes using a small language model to distinguish requests such as coding and history questions. That can help select an appropriate model, but the classifier itself requires inference. Any benefit from better model selection has to account for the cost of making that selection.
Check the request after transformations, not just before them
As proxies do more work routing requests, the order of that work matters. A permission check may examine one value, then a later transformation may change that value before it reaches the receiving system. The original check no longer protects the request that gets processed.
Yan gives a concrete example of a numeric identifier with a small decimal fraction that doesn’t match the value a security policy checks for. A later transformation removes the fraction, leaving the very identifier the policy was meant to restrict. This creates a bypass. The lesson is to follow the request through the whole system and check that parsing or transformations cannot undo the policy’s protection.
AI scanning is helping uncover these failures by combining bugs that may look harmless on their own. Yan says models can find combinations in days that could take a human researcher months to assemble. But finding the vulnerability does not finish the job. Fixing it may change behavior that existing deployments rely on, so maintainers must work out how to introduce the correction without creating new problems for operators.
That work continues as new features are added. Yan describes feeding discoveries back into the scanning system so it can look for further weaknesses, while trying to keep the cost of continuous scanning manageable. AI makes more vulnerabilities discoverable; turning those findings into safer production systems still depends on investigation, maintenance, and rollout work.
Thank you, Yan, for giving a practical view into what it looks like to build a system like this at scale.
Watch the full episode: YouTube | Spotify
Until next time!
xx Alexa
Alexa’s Input (AI) is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.