Your Agent Demo Is Lying to You: A Map of the Production Gap

Your demo worked because you were the reliability layer. You picked a clean input, the APIs happened to be up, and you stopped the run before it got strange. Production removes you from that loop.

The numbers say most teams never make the crossing. MIT's NANDA initiative found in 2025 that 95% of enterprise generative AI pilots failed to deliver measurable financial return, against an estimated $30 to 40 billion of enterprise investment. The report's own framing matters: the problem was not model quality, it was pilots that never got the operational discipline to become reliable. Gartner now predicts over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls as the leading causes. Read those causes again. None of them is a model problem. Every one of them is an operations problem.

So here is the map I wish someone had handed me earlier. Five systems: secrets and identity, evals, versioning, monitoring, rollback. Plus one afternoon of container hygiene that too many guides mistake for the whole job.

The failure modes your demo never showed you

Before the systems, the failures they exist to catch. These are the ones that actually happen:

• The runaway loop. An agent stuck in a tool-call loop over a weekend, burning the month's token budget on a task that failed on Friday. A max-iterations cap is not optional.
• The silent model update. The provider ships a new checkpoint and your agent's tool-call formatting, refusal behavior, and tone drift overnight. Nothing in your code changed. Everything in your traces changed.
• The injection in the data. A prompt injection payload sitting in a retrieved document or a customer ticket, turning your helpful agent into an exfiltration path with tool access.
• The integration that passed CI. Unit tests green, then the MCP server is unreachable from the production network, or the vector index is stale and retrieval feeds the model confident garbage.
• The half rollback. Someone reverts the model version but keeps the new system prompt. Now you are debugging a hybrid that never existed in any environment, at 2 a.m.

That last one is why versioning gets its own section.

The boring infrastructure, in one afternoon

Most deployment guides spend 80% of their length here, so let me be blunt: this part is a day of work, and it is the easy part.

FROM python:3-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
RUN useradd -m agent && chown -R agent /app
USER agent
HEALTHCHECK CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8080/healthz')" || exit 1
CMD ["python", "agent.py"]

Non-root user, slim base image, dependency layer cached above the code copy, and a health endpoint so the orchestrator knows the agent is alive. Then resource limits, because a runaway agent without them can consume an entire server:

# docker-compose.yml (excerpt)
services:
  agent:
    build: .
    mem_limit: 512m
    cpus: 0.5
    restart: unless-stopped

On platforms: under roughly 10,000 requests a day, Google Cloud Run ($3 to 10 a month, scales to zero, fits event-driven agents) or Railway/Render ($5 to 15) will carry you. ECS/Fargate runs $20 to 50. Managed Kubernetes starts around $50 to 200 a month with very high operational complexity, and choosing it for a single agent is resume-driven development. Whatever you pick, autoscale on active agent sessions, queue depth, and response latency rather than CPU. These workloads are bursty, and CPU looks fine right up until the queue backs up.

Secrets, identity, and the audit trail

This is the section most agent guides bolt on at the end. It deserves better.

The agent gets its own identity

Not your personal API key. Not the shared team key. Give the agent a distinct identity per environment with least-privilege permissions per tool: read-only database credentials if it only reads, scoped tokens per API, nothing it does not need. When the agent eventually gets prompt-injected into doing something dumb, the blast radius is whatever its credentials allow.

Credentials live in a vault, and they rotate

No API keys in code, obviously. Also no long-lived keys sitting in environment variables on shared servers; use a secrets manager and short-lived credentials where the platform supports them, and rotate on a schedule. Agents leak secrets in ways web apps usually do not: tokens end up in traces, prompt logs, debugging screenshots, pasted into tickets. Assume any long-lived credential the agent touches will eventually be copied somewhere it should not be, and make sure it expires before that matters.

Log everything, then protect the logs

Audit-log every tool call, decision, and output, plus every configuration change. When an agent does something weird three months from now, the trace is the only witness. Two rules that bite teams constantly: PII never lands in plain-text logs, and conversation data stays encrypted at rest and in transit. Put RBAC around who can modify agent configuration and who can approve a deployment.

Here is the part people miss: the system prompt is a privileged control plane. Whoever can edit the agent's instructions effectively controls everything the agent can reach. Guard config changes with the same review and audit discipline you would apply to IAM policy changes, because functionally that is what they are. And sanitize inputs against prompt injection, treating retrieved content as untrusted, because it is.

Evals are the only promotion gate you have

Code has tests. Agents have evals, and skipping them is how you end up in the 95%.

The target is a test set of 500 to 2000 cases built from real historical traffic, including edge cases, adversarial inputs, and out-of-distribution examples. Run them automatically in CI with tools like RAGAS, DeepEval, or an LLM-as-judge pipeline, and benchmark every candidate against the current production baseline so a regression is a number, not a vibe. Encode compliance boundaries as eval rules too: HIPAA limits for a healthcare agent, no unauthorized advice for a financial one, GDPR data minimization for EU user data. Guardrails that are not tested are decoration.

If your agent uses RAG, test the retrieval pipeline on its own (precision, recall, MRR). The most common RAG failure in production is retrieval returning irrelevant documents with the model hallucinating on top of them. Testing only the full pipeline end to end will not tell you which half broke.

Two measurements to take before launch, not after:

• Latency. Measure end to end under realistic load and record P50, P95, and P99. Synchronous use cases like chatbots need to stay under about 2 seconds.
• Token spend. Profile usage per request (Tiktoken works for OpenAI models). An agent averaging 5,000 tokens per request at 100,000 requests a month consumes 500M tokens, which can run to thousands of dollars. Know that number before finance does.

Then write down the operational baselines: normal latency, normal cost, acceptable failure thresholds. Without them you cannot separate a regression from ordinary variance, and every alert becomes an argument.

The promotion rule, stated plainly: no agent version ships unless eval scores clear the threshold, projected cost is within budget, and there are zero guardrail violations. Changes affecting customer-facing behavior also get human sign-off. Automated checks plus human judgment, in that order.

Version the whole agent or version nothing

An agent's behavior is not in its code. It lives in the combination of instructions, model selection, tool bindings, guardrail policies, and memory settings. Change any one of them and you have a different agent. So version all of them as one atomic unit, and deploy and roll back that unit, never the pieces:

agent: support-agent
version: 1.4.2
model: <exact provider version string>
instructions_sha: 9f31c2
tools: [ticket_search, refund_status, knowledge_base]
guardrails: [pii_redaction, no_refunds_over_500]
memory: { window: 20_turns, store: redis }

Partial rollbacks are worse than no rollback. Reverting the model while keeping the new instructions gives you a combination that was never evaluated and never existed before. You will debug it longer than the original bug would have taken.

Two more rules. Pin the exact model version your provider exposes, and where providers update models under you (several do), run your eval suite on a schedule against the live endpoint so drift pages you instead of your users. Treat every config change like a code change: pull request, review, audit log entry.

Monitoring and the reliability floor

Evals catch what you have seen before. Monitoring catches everything else.

The reliability floor, before anything ships: retries with exponential backoff, a hard timeout on every external call, graceful degradation to a fallback model, and idempotent tool handlers so a retried action does not execute twice. Next to them, the cost controls: token budgets per run, model routing so cheap models handle easy steps, caching, and the max-iterations cap on tool loops. For most task agents that cap should be low. If your agent needs 40 tool calls to answer a question, something else is wrong.

Pre-flight every integration, meaning MCP servers, databases, vector stores, and third-party APIs. Deployments pass unit tests and then fail on an unreachable MCP server or a stale vector index, and both look like "the agent got dumber" from the outside.

In production, watch traces, quality metrics, policy violations, and spend. Alert on spend deltas, not just absolute spend. A 3x jump in tokens per conversation is usually a loop or a retrieval failure before it shows up as a bill.

Rollback is a drill, not a button

For a deterministic service, rollback means redeploying the old artifact. For an agent that is necessary and nowhere near sufficient. Redeploying last week's config does not restore last week's behavior if the provider shipped a model update on Tuesday, and nothing you roll back will un-send the emails, un-close the tickets, or un-issue the refunds the bad version already produced.

Which leads to my actual position: prevention beats rollback speed. Test against real historical cases, then roll out gradually behind shadow mode and human approval gates. Keep irreversible tool actions behind human confirmation until a version has earned trust.

Then rehearse the rollback you hope to skip. Quarterly drills, where the team reverts to a previous agent version under simulated time pressure, sound theatrical until the first time you do it for real and the whole thing takes four minutes because everyone knows their part. For the first 24 to 72 hours after any release, assign a named on-call owner, define escalation thresholds, and watch traces, quality metrics, policy violations, and spend deltas closely. Most agent incidents surface in that window.

The small-team objection

The pushback I hear: "We are three people. Fifteen hundred eval cases and quarterly drills are enterprise cosplay."

Scale the gates, not the discipline. A hundred cases pulled from real historical tickets beats zero by an enormous margin, and the harness you write for a hundred grows on its own once you add every incident as a regression case. A rollback drill can be 45 minutes in a conference room. The expensive version of all this is the one you improvise during the incident, and eventually you will do one or the other.

And yes, LLM-as-judge scores are noisy. Treat them as regression signals against a pinned baseline rather than as truth, and they remain the best early warning you have.

Key takeaways

• The demo-to-production gap is operational. The 95% pilot failure rate and the 40%-plus cancellation forecast are about cost, risk, and discipline, not model quality.
• Give the agent its own least-privilege identity, keep credentials in a vault with rotation, and audit-log every tool call and config change. The system prompt is a control plane, so guard it like one.
• Version instructions, model, tools, guardrails, and memory as one atomic unit. Partial rollbacks create agents that never existed in testing.
• Nothing reaches users without clearing the eval gate: scores above threshold, cost in budget, zero guardrail violations, and human sign-off for customer-facing changes.
• Rehearse rollback quarterly and staff the first 24 to 72 hours after release with a named owner. The incident is the wrong classroom.