Securing the LLM API Layer: Why Your AI Gateway Matters

On a Tuesday morning in Q2 2025, a cybersecurity firm's production infrastructure began minting OpenAI API calls at a rate that didn't match any known workload. By the time the on-call engineer noticed the billing alert, $140,000 in API costs had accrued in 11 hours. The cause: a developer had committed an OpenAI API key to a public GitHub repository in an .env file, where automated scanners found it within four minutes and began harvesting it for cryptocurrency mining operations that use LLMs as computational resources.

This incident is not unusual. It's not even particularly notable anymore. GitGuardian's 2026 State of Secrets Sprawl report found over 12 million API keys and secrets exposed in public repositories in the prior 12 months — a 28% increase year over year — with AI provider API keys among the fastest-growing category. The problem is structural: developers need API keys to build things, storing secrets in code is the path of least resistance, and the consequences of exposure in the AI era are both immediate (cost) and lasting (data).

The API Key Sprawl Problem

The typical enterprise with a meaningful AI development practice has API keys from some combination of OpenAI, Anthropic, Google, AWS Bedrock, Azure OpenAI, and Cohere — and potentially dozens of wrapper services built on top of those providers. Each team that builds with AI tends to obtain its own keys. Those keys are stored in .env files, CI/CD secret stores, developer laptops, hardcoded in scripts, and periodically in places that are genuinely alarming (Slack messages, Notion pages, GitHub issues).

The core security problem with this model isn't just the risk of accidental exposure. It's the complete absence of visibility and control:

• No attribution — when a request hits OpenAI's API using your key, you know the cost, but not which application sent it, which user triggered it, or what data it contained.
• No blast radius control — a single compromised key can consume your entire spending limit and potentially expose data from any application that shares it.
• No policy enforcement — you cannot apply DLP scanning, content policies, or model restrictions to traffic from a key that lives directly in an application.
• No audit trail — there's no session-level record of what was sent and received, which is a compliance problem in any regulated industry.

What a Leaked Key Actually Costs

The financial exposure from a leaked AI provider key is worse than most developers appreciate. Modern LLM APIs are priced per token, and automated abuse is designed to maximise token consumption. An attacker who obtains an API key will typically use it to run bulk text generation jobs, model-as-a-service operations, or embedding generation at scale — workloads that can generate six-figure API bills within hours at current pricing.

Beyond cost, the data exposure dimension is more serious and less recoverable. An attacker who has access to your API key can observe the data your applications send to the model — customer data, proprietary code, internal documents, anything that your AI applications process. This isn't hypothetical: sophisticated key-theft operations don't just abuse keys for compute, they use the access to exfiltrate data from the API traffic stream.

The Gateway as Security Boundary

The architectural solution to API key sprawl is a centralised AI gateway — a proxy that sits between all internal applications and external AI providers, centralising key management, policy enforcement, and audit logging in a single controlled layer.

In this model, applications never hold AI provider API keys. They authenticate to the gateway using application-scoped credentials, and the gateway forwards requests to the appropriate provider using centrally-managed keys. The provider keys never leave the gateway's secure credential store.

Per-Application Scoped Keys

Centralising key management through a gateway enables per-application scoped credentials — a significant improvement over shared key models for blast radius control and attribution.

In AIGuard's gateway model, each application or team that needs AI access receives a scoped token with configurable limits: which models it can access (model allowlist), maximum monthly spend, rate limits per minute and hour, and which data classification levels it's permitted to process. If any token is compromised, only that application's access is affected — the blast radius is bounded to what that application was permitted to do.

This also enables per-application attribution. Every request through the gateway is tagged with the application identity, making it possible to produce spend reports, usage analytics, and audit logs at the application level rather than just the org level.

Model Allowlists and Shadow Model Prevention

As the AI model landscape has proliferated — there are now hundreds of production-grade open and closed models available via API — the risk of employees or applications using unapproved models has grown substantially. Shadow model use is a variant of shadow AI: a developer wires their application to a low-cost, uncertified model to reduce costs, bypassing the organisation's vetted model set.

A gateway-based model allowlist prevents this by restricting which models can be called through the gateway. Requests to non-allowlisted models are blocked at the proxy layer, with a clear error message explaining the policy. This ensures that model risk assessment — evaluating which models are appropriate for which data classifications — is actually enforced, not just documented.

Rate Limiting as Cost and Abuse Control

Rate limiting at the gateway layer serves two distinct functions. First, it's cost governance: per-application and per-user rate limits prevent runaway spend from buggy applications, misconfigured retry logic, or compromised credentials. Second, it's abuse detection: a sudden spike in request rate from a normally-quiet application is a strong signal that something unusual is happening — potentially a compromised credential or an injection attack that has triggered unexpected behaviour.

AIGuard's gateway applies three levels of rate limiting: per-token (application credential), per-user (the end user making the request through the application), and per-model. The per-model limits are particularly useful for managing access to frontier models like GPT-4o and Claude Opus, which carry significantly higher per-token costs than smaller models.

The Response DLP Problem

Most AI security discussions focus on what goes into the model — prompt inspection, DLP scanning of inputs. The response side is equally important and substantially more neglected.

AI models trained on large internet corpora can and do reproduce information from their training data in responses. This is most commonly seen as copyright infringement (models reproducing copyrighted text verbatim), but it extends to any information type: contact details of real people, fragments of leaked source code that made it into training data, financial figures from documents that shouldn't have been publicly accessible.

Response DLP at the gateway layer scans model outputs before they're delivered to the calling application, detecting and blocking responses that contain high-confidence sensitive entity matches. For organisations in regulated industries, this isn't optional: reproducing PHI, PII, or material non-public information in an AI response carries the same regulatory exposure as transmitting it in any other channel.