AWS Bedrock
The right model path for each request.
ColdAI helps teams build inference applications on Amazon Bedrock with a deliberate model strategy. We evaluate candidate models on your own requests, design how each type of request is routed by quality, latency and cost, and connect retrieval, logging and release checks to your AWS environment, so the application can change models without being rebuilt each time.
The opportunity
Start with a decision
worth improving.
One application handles short classification tasks and complex document questions. The workflow needs a deliberate model strategy rather than using the same expensive path for everything.
From possibility to a working system
Follow the flow.
Three connected decisions. One considered architecture.
Conceptual flow, not a live system or measured result.
Understand the request
Classify the tasks, data sensitivity and response requirements. Build a representative evaluation set so model selection starts with your workload rather than a generic benchmark.
Route with context
Design the model and retrieval paths available to the application. Compare quality, latency and usage cost, with explicit handling for unavailable models and incomplete source material.
Operate the application
Connect logging, access controls and deployment checks to the AWS environment. Monitor actual task outcomes and revisit the routing policy as requirements or model behavior change.
The work, made concrete
What we can build
with your team.
We agree scope, dependencies and acceptance criteria before delivery. Your project can start with an assessment, a pilot or an integration into an existing system.
Workload-based model evaluation
Clarify the system boundary, the responsible owners and the decisions the architecture needs to support.
Retrieval and inference integration
Turn the agreed design into reviewable work, evaluated against representative inputs and explicit success criteria.
Access, observability and release design
Make the next step operable: document responsibilities, known limitations and the path from pilot to ongoing use.
In depth
Go deeper into Bedrock architecture
Two reference guides for the parts of a Bedrock application that most often decide quality and cost: retrieval, and how inference capacity is bought.
- 01ArchitectureAmazon Bedrock Knowledge Bases: a RAG architecture guideDesign RAG on Amazon Bedrock Knowledge Bases: source sync, chunking, embeddings and vector stores, retrieval tuning, citations, grounding checks and evaluation.
- 02ComparisonBedrock on-demand, batch or provisioned throughput?Compare Amazon Bedrock on-demand tiers, batch inference, Provisioned Throughput, Reserved capacity and cross-Region profiles, and match each to a traffic shape.
Why a single model rarely serves every request well
Most inference applications handle more than one kind of work. A support tool might classify incoming messages, extract order details and answer detailed questions from policy documents. Sending everything to the most capable model makes simple tasks slow and expensive; sending everything to the cheapest model fails the hard ones. A model strategy assigns each kind of request to the path that meets its quality bar at an acceptable latency and cost.
Amazon Bedrock makes that strategy practical because it offers models from several providers behind one service. Its Converse API gives a consistent request format across the models that support messages, with model-specific parameters passed separately, so changing the model behind a route does not mean rewriting the application1. Which models are available still depends on the Region and account, which is why availability is checked before any route is designed.
This page covers model evaluation and routing. Agent runtimes and tool orchestration on AWS belong to our Bedrock and AgentCore practice.
Building an evaluation set from your own traffic
Model selection starts from your requests, not from public benchmark tables.
Sample real requests
Pull a representative sample from logs or from the people who handle requests today, including messy, ambiguous and out-of-scope examples. Remove or mask personal data before it enters the set.
Group requests into task classes
Label each request by the kind of work it needs, such as classification, extraction, grounded question answering or drafting. Routing decisions are later made per class.
Write acceptance criteria per class
Define what correct, acceptable and unacceptable responses look like for each class, with reference answers where possible, and agree latency targets for the classes users wait on.
Shortlist candidate models
Check availability in your Regions, data terms, context limits and lifecycle status, and drop any model that fails a hard requirement before testing begins.
Run the comparison
Score each candidate on each class. Bedrock evaluations support automatic jobs on your own prompt datasets, jobs graded by human reviewers and jobs that use a second model as judge2; judge scores are calibrated against a human-graded subset.
Decide routes and fallbacks
Give each class a primary model, a fallback for throttling or unavailability and a rule for requests that match no class. Record the evidence behind each choice so it can be revisited.
How a routed request moves through the application
- Classify the request
A rule set or small model assigns the task class and flags sensitive data.
- Apply data policy
Blocks or redacts content that a route is not approved to receive.
- Select the route
Chooses the model for the class, with a fallback when the primary is throttled or unavailable.
- Retrieve context
Adds grounding passages for classes that answer from documents.
- Generate and check
Calls the model and applies output checks such as schema validation or a guardrail.
- Respond and trace
Returns the answer and logs class, route, model version, latency and outcome.
Three ways to route requests between Bedrock models
Routing can be written as rules, delegated to Bedrock's managed router or built as your own classifier, and the approaches can be combined.
| Consideration | Rules by task class | Bedrock intelligent prompt routing | Your own classifier or cascade |
|---|---|---|---|
| How a model is chosen | A fixed mapping from task class to model | Predicts response quality per request and picks between two models in one family3 | A trained classifier, or a try-small-then-escalate cascade |
| Uses your quality data | Yes, through the evaluation behind the mapping | No; AWS notes it cannot adapt to application-specific performance data3 | Yes, if trained and recalibrated on your labeled outcomes |
| Language coverage | Whatever the chosen models support | Optimized for English prompts3 | Depends on your training data |
| Transparency | Easy to explain and audit | The response reports which model was used | Needs logging of scores and thresholds |
| Effort to run | Low, but needs re-evaluation when models change | Low: choose the models, a fallback and a quality-difference criterion | Highest, because it is a model of its own to maintain |
We usually start with rules by task class, because the evaluation set already produces them, and test managed or learned routing only where one class varies widely in difficulty.
Account checks before committing to a model
How Bedrock model strategies drift after launch
Retired model versions
Early signalA route still points at a model that has entered its Legacy period.
MitigationTrack lifecycle state for every model in the routing policy and schedule re-evaluation as soon as a retirement date appears.
A stale evaluation set
Early signalUsers ask new kinds of question that the set does not contain.
MitigationRefresh the set from recent traffic on a regular cycle and re-score the current routes against it.
Averages hiding failures
Early signalOverall quality looks stable while one task class quietly degrades.
MitigationReport quality per task class and alert on drops in any class, not only in the total.
Untracked prompt changes
Early signalBehavior shifts without any model change because someone edited a prompt.
MitigationVersion prompts alongside routing rules and run the evaluation set before any change ships.
A hypothetical claims-intake application with three routes
Frequently asked questions
Which foundation models can we use on Amazon Bedrock?
It depends on your AWS Regions and account. Bedrock offers models from Amazon and several third-party providers, but availability, features such as prompt caching or service tiers, and lifecycle dates vary by model and Region. We confirm what is available in your deployment accounts during shortlisting, before any routing design depends on it.
How is this different from your Bedrock and AgentCore practice?
This practice covers the inference application: which model handles which request, how retrieval feeds it and how quality, latency and cost are measured. The Bedrock and AgentCore practice covers the runtime around autonomous agents, including tool access, identity, task state and observability. Many applications need only the first; agent workloads usually need both.
Should we fine-tune a model on Bedrock or rely on retrieval?
Usually retrieval first. If the problem is missing or changing knowledge, retrieval from your documents is easier to keep current. Fine-tuning helps when the model needs a consistent behavior or format that prompting cannot achieve, and it brings its own hosting and capacity choices. Our LLM fine-tuning practice tests whether adaptation beats a prompting and retrieval baseline.
Do model providers see the prompts we send through Bedrock?
AWS documents that each model runs in a deployment account owned and operated by the Bedrock service team, that model providers have no access to those accounts, and therefore no access to Bedrock logs or to customer prompts and completions5. You still own the configuration on your side, including invocation logging, encryption and who can read stored outputs.
What happens when a Bedrock model is throttled or unavailable?
Each route in the policy has a named fallback, and the application retries throttled requests with backoff before switching. Fallbacks are evaluated on the same task set as the primary, so a switch lowers quality predictably rather than unexpectedly. Persistent throttling is a capacity question, handled through quotas, cross-Region inference or a different consumption mode.
What do you need from us to start a Bedrock engagement?
An AWS environment with a named owner for access and security, a sample of real requests with personal data removed or approved for use, and the quality, latency and budget constraints the application must meet. With those in place, the evaluation set and first routing policy are built against your workload instead of assumptions.
Sources
- Inference using Converse API (Amazon Bedrock User Guide) — Amazon Web Services
- Evaluate the performance of Amazon Bedrock resources — Amazon Web Services
- Understanding intelligent prompt routing in Amazon Bedrock — Amazon Web Services
- Model lifecycle (Amazon Bedrock User Guide) — Amazon Web Services
- Data protection (Amazon Bedrock User Guide) — Amazon Web Services
Connect the architecture
The next connection.
Explore the complementary capabilities that can turn an individual technology into a complete workflow.
Your next move
Bring us the
real problem.
A workflow that takes too long. A system that cannot connect. An idea that needs a technical path. Start there, and we can define what to investigate, build and measure.
- An AWS environment and access owner
- Sample requests and approved data
- Quality, latency and budget constraints