AI proofs of concept are intentionally optimized for speed. A small team chooses a model, obtains credentials, writes an application, demonstrates a useful result, and proves that the idea can work.
Production has a different standard. The model output is only one part of the service. Everything around the model has to become repeatable, supportable, secure, observable, and owned.
The following controls are commonly left until late in an AI project because they are not required to make the first demo work.
1. Identity and workload identity
A pilot may use one API key shared by a team or stored in a developer environment. Production needs to answer:
- Who is the user?
- Which application is calling the model?
- Which agent is acting?
- Which service identity should be used?
- How is access revoked?
- Which users or workloads can reach which models?
For agentic systems, identity becomes even more important because the AI may call tools or downstream systems on someone’s behalf.
2. Credential and secret handling
API keys and service credentials should not live in source code, chat transcripts, configuration files checked into a repository, or unmanaged environment files.
Production architecture needs a clear secret-storage and rotation pattern, plus a way to revoke credentials without redesigning the application.
3. Sensitive-data handling
A proof of concept often uses synthetic or limited data. Production may introduce customer data, internal documents, source code, regulated information, credentials, or other sensitive content.
The production design should define:
- What data categories are allowed?
- Which destinations can receive them?
- Which controls inspect them?
- What happens when policy is triggered?
- What is logged?
- What is retained?
4. Model and provider approval
During a pilot, a team may select whichever model produces the best result. Production should define which models and providers are approved for which workloads.
Approval criteria can include:
- Capability
- Data handling
- Contractual terms
- Availability
- Residency
- Cost
- Latency
- Security review
- Supportability
A secure AI gateway or model-routing layer can help centralize these decisions when multiple applications need them.
5. Failure and fallback behavior
What happens when the preferred model is unavailable?
A production system may need to decide whether to:
- Retry
- Fail closed
- Route to another approved model
- Degrade functionality
- Queue work
- Notify an operator
The correct behavior depends on the workload. Silent fallback to an unapproved provider is rarely a good default.
6. Prompt and response logging
Logging is necessary for troubleshooting and audit, but storing every prompt and response can create a new sensitive-data repository.
Decide explicitly:
- Which metadata is required?
- Is content logging necessary?
- Can sensitive content be excluded or redacted?
- Who can access logs?
- How long are logs retained?
“Log everything” is not a substitute for an evidence strategy.
7. Rate limits, budgets, and cost controls
Pilot usage is often small. Production usage can grow quickly, especially when applications or agents create repeated requests.
Define:
- Per-user or per-application limits
- Provider budgets
- Alert thresholds
- Model selection rules
- Concurrency limits
- Protection against accidental loops
For agentic workflows, tool and model loops deserve explicit testing.
8. Tool access for agents
An AI system that can call tools has a different risk profile from a chat interface.
Review:
- Which tools the agent can access
- Which credentials it uses
- Whether actions require approval
- What data can be returned to the model
- Which operations are reversible
- How actions are logged
- How the agent is stopped
Tool permissions should be designed with the same care as application permissions.
9. Operational monitoring
A production service needs more than application logs.
Useful signals can include:
- Request volume
- Latency
- Provider errors
- Policy blocks
- Fallback events
- Cost trends
- Model availability
- Exception use
- Tool failures
The operations team should know what normal looks like before the first incident.
10. Support ownership
A pilot belongs to the project team. Production belongs to an operating model.
Define:
- Service owner
- Technical owner
- Security owner
- Support route
- Escalation path
- Vendor contacts
- Change approval
- On-call expectations where required
Without ownership, production incidents become organizational discovery exercises.
11. Change management for models
Models change. Providers add versions, retire versions, change behavior, and introduce new capabilities.
Production systems should be able to test and approve model changes rather than silently inheriting them.
Where possible, separate the application’s business logic from provider-specific model configuration so models can be evaluated without redesigning the application.
12. Rollback and recovery
A production launch needs a way back.
Define:
- Previous configuration
- Previous model route
- Previous prompt/template version where applicable
- Backup of gateway/configuration state
- Recovery owner
- Post-change validation
This is particularly important when policy and model-routing changes affect many applications at once.
A production-readiness pattern
A practical sequence is:
- Confirm the business use case.
- Map the data and trust boundaries.
- Define workload identity and access.
- Approve model/provider destinations.
- Add data and policy controls.
- Add observability and evidence.
- Test failure paths.
- Define support and ownership.
- Document rollback.
- Launch in a controlled increment.
Production readiness is an architecture problem
The difference between a successful pilot and a reliable production service is often not the model. It is the identity, data handling, routing, resilience, observability, support, and governance around the model.
Deadlights helps organizations design and implement those controls as a connected architecture.