1Customer support + RAG
Amazon Bedrock behind a LiteLLM gatewayVoice and chat front door, RAG over product docs, an agent that reads the CRM and issues refunds, human review on escalation.
Voice Agent StackVoice / chat front door
→RAG Pipeline CostRAG retrieval
→Vector DB CostVector database
→Agent Loop CostModel inference and agent turns
→AI Agent Integration CostCRM and ticketing tool calls
→Retry Cost CalculatorRetries
→Guardrail / Security CostGuardrail screening
→LLM Observability / Tracing CostObservability
→Human-in-the-Loop Review CostHuman review
→Policy envelope$16,400/mo, $711/day, 12-turn cap
→LiteLLMenforces in your network
→Amazon Bedrockyour cloud tenant
→Claude Sonnet 5.5economy route Claude Haiku 5.5
What AICost received
Numbers and names only. The shape every chain call takes.
{
"workload": "Customer support + RAG",
"outcomes_per_month": 44000,
"unit": "resolved case",
"model": "claude-sonnet-5-5",
"economy_model": "claude-haiku-5-5",
"data_class": "Customer PII and order history",
"engines": [
"voice-agent-stack",
"rag-pipeline",
"vector-db-cost",
"agent-loop-cost",
"integration-cost",
"retry-cost",
"guardrail-security-cost",
"observability-cost",
"hitl-review-cost"
]
}Cost envelope (cost-firewall engine)
$16,400budget / month
$711daily hard cap
$1.27per-run ceiling
12turn cap
Budgets cover model spend only. Reviewer time and integrations are real costs, but they are not spent through the gateway. Model spend in this example: $14,827/mo of $54,006/mo total.
Model allowlist for this data class
Claude Sonnet 5.5, Claude Haiku 5.5 · green for Regulated PII (GDPR/CCPA)
Who enforces it
LiteLLM in your VPC holds the budget, rate limits and model list; Bedrock cross-region inference and your AWS Budgets sit underneath.
Where the data lives
Customer chats, order history and help-center articles stay in your AWS account and your vector store.
Gateway config: LiteLLM (proxy config fragment)
{
"router_settings": {
"rpm": 420
},
"litellm_settings": {
"max_budget": 710.67,
"budget_duration": "1d"
},
"x_aicost_policy": {
"firewall_policy": {
"workload_id": "support",
"scope": "workload",
"monthly_budget_usd": 16400,
"daily_warning_usd": 497.47,
"daily_hard_limit_usd": 710.67,
"max_cost_per_run_usd": 1.27,
"max_model_calls_per_run": 12,
"max_retries": 2,
"max_output_tokens": 1200,
"requests_per_min": 420,
"anomaly": {
"rolling_window": "7d",
"spike_multiplier": 3,
"rule": "flag any hour/day exceeding 3x the rolling average, or any single day above the daily hard limit"
},
"actions": {
"warning_threshold": "notify",
"soft_limit": "route_to:claude-haiku-5-5",
"hard_limit": "block_noncritical_requests",
"anomaly": "throttle_and_alert",
"emergency": "block_all_and_require_approval"
},
"emergency_stop": {
"enabled": true,
"requires_approval_to_resume": true
}
}
}
}Cloud-native policy (cloud-gateway-policy engine)
- Bedrock cross-region inference across 3 regions for resilience + capacity.
{
"policy_type": "cloud_gateway_policy",
"cloud": "bedrock",
"cross_region_policy": {
"model": "Claude Sonnet 5.5",
"regions": [
"us-east-1",
"us-west-2",
"eu-central-1"
],
"note": "Cross-region inference reflects the source region rate; no extra fee."
}
}