An AI agent can connect to a model or tool in minutes. The harder questions arrive later: Which identity made the call? Which tools was it allowed to discover? Would a failed model route recover? Could an operator trace the request without adding custom instrumentation to every client?
I wanted answers I could demonstrate, not assumptions inferred from configuration. I therefore built an independent, fully local reference platform around the AAIF-hosted open source agentgateway project. The result runs on a laptop and brings model routing, MCP tool federation, identity, authorization, observability, and a Kubernetes promotion path into one reproducible environment.
This is not a production deployment or an official agentgateway solution. It is a community-built test bed for one practical question: what does it take to move from connected agents to governed connections?
Why I built the platform
Direct connections are reasonable at the prototype stage. One application calls one model, adds an MCP server, and keeps the required credentials nearby. As the number of agents, tools, and providers grows, those direct links create an ownership problem. Authentication, retries, authorization, and telemetry become scattered across clients, and no single layer can explain the complete request path.
A platform team's visibility and policy gaps before and after introducing one governed data plane.
The gateway pattern interested me because it creates a shared enforcement point. Instead of asking each agent implementation to solve the same platform concerns, clients connect through one data plane that understands LLM, MCP, and ordinary service traffic. The value is not simply an extra proxy hop. The value is a consistent place to verify identity, apply policy, observe behavior, and handle backend failure.
A laptop-sized but complete test environment
I used agentgateway v1.4.0 as the data plane and kept every supporting component local. Ollama provides an OpenAI-compatible model endpoint, while Keycloak issues JWTs containing role and tenant claims. Three local tool sources sit behind one MCP endpoint: a SQLite-style server, a streamable HTTP server, and an OpenAPI application exposed as MCP tools. OpenTelemetry, Prometheus, Grafana, and Jaeger provide the telemetry path. A kind cluster, Gateway API resources, and Helm provide the Kubernetes path.
The default model is deliberately modest: llama3.2:3b. Optional larger models are documented separately, but a reference environment is more useful when another engineer can reproduce its primary flow without a GPU or paid model key.
I divided the work into independently testable milestones. A model call without the configured bearer credential returned HTTP 401. The authenticated request returned HTTP 200 and a real Ollama completion through agentgateway. The MCP path then completed initialize, tools/list, and tools/call through the gateway before identity policy was added.
Authorization at the tool-call boundary
For MCP, server-level access is often too broad. An incident responder might need to read an incident without being able to modify it. A support agent might read tickets but not change their status. For these cases, authorization needs to reach the tool rather than stop at the server connection.
The demo federates three target types into tenant-prefixed tool menus. A reader sees only its three read-only tools in its own tenant. An operator sees all six tools for its own tenant. If a reader invokes a write tool or either identity names another tenant's tool directly, the gateway rejects the call. The automated security test exercises 13 outcomes, including unauthenticated rejection, filtered discovery, allowed reads with matching backend tenant data, denied writes, direct cross-tenant rejection for two readers and an operator, and an operator write in its own tenant.
Virtual MCP design with Keycloak identity, backend-level authorization, reader and operator roles, and six federated tools.
Filtering tools/list is useful because it keeps unavailable capabilities out of the client context, but discovery filtering is not the authorization boundary. For this design, enforcement happens on tools/call. Both behaviors must agree.
The token carries a tenant claim, but the claim is not treated as proof by itself. The gateway selects a matching tenant-prefixed target, and the target calls a tenant-scoped backend that accepts no caller-controlled tenant argument. The security suite verifies both that allowed responses contain the expected tenant and that direct cross-tenant tool calls are rejected.
Five lessons that only appeared at runtime
Static review helped with structure, but the most useful lessons appeared only when a real gateway loaded the configuration and handled requests.
Five runtime lessons covering the distroless image, health-driven failover, tenant-bound backends, Kubernetes CRDs, and identity-provider readiness.
First, the agentgateway container is distroless. My initial stdio MCP design assumed that Node.js would exist inside the gateway container. It did not. I moved that sample to an HTTP MCP target running in a sibling container, which better respected the runtime boundary.
Second, authorization policy placement and naming were exact. In this standalone configuration, MCP authorization had to be attached at the MCP backend level. The CEL expression receives the bare tool name, not the target-prefixed name returned after federation. A plausible-looking policy in the wrong place is still an unenforced policy.
Third, the original OpenAPI tenant query parameter was the wrong isolation boundary. It allowed a caller to name another tenant even when the token carried a tenant claim. I replaced it with tenant-scoped targets and backends, so tenant selection is no longer a tool argument. Direct cross-tenant calls are now regression-tested negative cases.
Fourth, failover required health-based eviction. Declaring an ordered failover route did not retry the failed request against another model. Adding health.eviction allowed the first failure to trip the breaker, after which subsequent requests moved to the backup. The behavior is cross-request failover, not a transparent retry of the original request.
Fifth, the secure MCP service depends on identity-provider readiness. The gateway fetches Keycloak signing keys at startup. A Compose health check now waits for Keycloak's actual /health/ready response before the gateway starts, eliminating the local JWKS startup race.
Each discovery became either a test or an explicit limitation. That practice matters more to me than the number of features listed in a configuration file. Configured, accepted, and behaviorally proven are different states.
Closing the observability loop
Telemetry was treated the same way. A provisioned dashboard was not enough. After authenticated traffic passed through the gateway, Prometheus reported the agentgateway target as healthy and its request counter increased. Grafana loaded the provisioned Prometheus datasource and dashboard, while Jaeger received agentgateway traces.
Grafana dashboard displaying locally collected agentgateway metrics.
This gave the platform a useful operational loop: make an authenticated model or tool request, observe whether policy allowed it, and follow the resulting metrics and trace data through the local stack.
Promoting the working route to Kubernetes
The last milestone moved the model route into a kind cluster. I installed Gateway API v1.5.0 and agentgateway v1.4.0 through Helm. The Gateway reported Programmed, the backend reported Accepted, and the policy reported Accepted and Attached. A real llama3.2:3b completion then flowed through the in-cluster gateway to Ollama running on the host.
The exercise exposed version-sensitive API details. The custom resources use the agentgateway.dev API group, and both the AI backend and traffic policy must match the live CRD schemas. I used server-side dry runs and status conditions before the behavioral test.
The Kubernetes milestone currently proves the model route and CORS policy. The complete standalone JWT and MCP tool-authorization policy has not yet been promoted to the Kubernetes resources. That is the next meaningful security milestone, not something the current cluster result should be interpreted as proving.
What I would change before production
The reference platform demonstrates the control points, but a production design needs additional engineering. Demo credentials should move to managed secrets, all relevant connections should use TLS or mTLS, and the gateway should run with multiple replicas. The local tenant boundary now has explicit negative tests. A production design would still need durable tenant data boundaries, distributed rate limits with a shared enforcement backend where exact limits matter, and authorization expressions reviewed, versioned, and tested like application security code.
I would also make identity-provider readiness resilient, promote the verified standalone policies to Kubernetes, and define service-level objectives for both model and tool traffic. Project versions and CRD schemas should remain pinned and be revalidated during every upgrade.
The broader lesson is simple: MCP gives agents a standard way to reach tools, but interoperability is not governance. Identity, least privilege, failure handling, and observability still need a shared operational home. For this build, agentgateway became useful when every important claim could be tied to a request that succeeded, failed, or produced telemetry for the expected reason.
About the Author
Ganesh Gurudu is a Principal Platform and SRE Architect and IEEE Senior Member with more than 15 years of experience building large-scale, multi-tenant infrastructure across healthcare, telecommunications, and financial services. His work focuses on Kubernetes platforms, DevSecOps, identity and authorization, observability, reliability engineering, MCP, and agentic AI infrastructure.
Share
Author
