Eval methods for Tools, Skills, and Prompts, and how to ensure correctness

1. Evaluating Tools (MCP / Agentic Tools)

Tools have the most structured evaluation surface because they expose defined inputs and outputs.

Metrics to Measure

Methods

a) Deterministic comparison (fastest, most reliable)

Compare tools_called against expected_tools. Check three things:

• name match
• argument match
• order match

from deepeval.test_case import LLMTestCase, ToolCall
from deepeval.metrics import ToolCorrectnessMetric

test_case = LLMTestCase(
    input="What's the return policy?",
    actual_output="We offer a 30-day refund.",
    tools_called=[ToolCall(name="WebSearch"), ToolCall(name="ToolQuery")],
    expected_tools=[ToolCall(name="WebSearch")],
)

metric = ToolCorrectnessMetric(threshold=0.7)

# Score = Correctly Used Tools / Total Tools Called

b) Trajectory-level evaluation

Do not just check the final output. Evaluate the full sequence of tool calls to detect missing tools, extra tools, and parameter mismatches.

c) LLM-as-judge fallback

When tool usage is correct but not obvious, use a judge model. Ask it whether the chosen tools were optimal, given the available tools context.

2. Evaluating Skills

Evaluate skills on two things: activation (did the correct skill load?) and output quality (did it improve results?).

a) Skill Activation / Routing Evals

Grading is a deterministic pass or fail. Did the expected skill activate?

b) Skill Output Quality Evals

• LLM-as-judge with rubric scoring (1 to 5 scale)
• Exact or string match for structured sections
• A/B comparison (with skill vs without skill)

c) Progressive Disclosure Check

Measure token usage when multiple skills are available to ensure context does not grow unnecessarily.

3. Evaluating Prompts

a) Code-based grading (preferred)

# Exact match
def eval_exact(output, expected):
    return output.strip().lower() == expected.lower()

# String containment
def eval_contains(output, key_phrase):
    return key_phrase in output

b) LLM-as-judge (nuanced assessment)

def evaluate_likert(model_output, rubric):
    prompt = f"""Rate this response on a scale of 1-5:
    <rubric>{rubric}</rubric>
    <response>{model_output}</response>
    Think step-by-step, then output only the number."""
    return call_judge_model(prompt)

c) Embedding similarity

Use cosine similarity to ensure paraphrased inputs produce semantically consistent outputs.

d) ROUGE-L for summarization

ROUGE-L measures overlap between generated and reference summaries.

Universal Best Practices

1. Choose volume over perfection. Automate at scale.
2. Include edge cases such as typos, ambiguity, long inputs, and topic shifts.
3. Use a different model as the judge, not the model that you evaluate.
4. Ask the judge to reason before scoring.
5. Automate and version evaluations like tests.
6. Combine deterministic checks with LLM-based scoring.

Quick Reference

• TOOLS: deterministic tool and argument match plus trajectory validation
• SKILLS: activation tests plus rubric-based output quality
• PROMPTS: exact match where possible plus LLM-judge for qualitative tasks

1) Prompt (Runtime System Prompt)

What it is: Instructions that you pass in the API call for a single request.

• One-time instruction (per request)
• Not enforced. The LLM may skip or reorder steps
• Good for quick control (tone, format, role)

Use when: prototyping, low-risk tasks, or temporary behavior changes.

Avoid when: you need guaranteed step execution or strict sequencing at scale.

2) Skill (Reusable Structured Prompt Module)

What it is: A reusable, structured reasoning template/module that improves consistency across repeated tasks.

• Reusable and standardized
• More consistent than ad-hoc prompts
• Still LLM-driven (probabilistic), not a hard execution engine

Use when: the task repeats often and you want consistent analysis structure, formatting, or output schema.

Avoid when: the workflow must never skip steps or must follow an exact sequence every time.

3) Tool (Deterministic Capability)

What it is: An executable function that performs a real action (API call, database query, file write, etc.).

• Deterministic execution (given correct code and inputs)
• Interacts with real systems or data
• Auditable and testable

Use when: you need real data, guaranteed operations, and repeatable correctness.

Important: Orchestrator (Code) for Strict Multi-Step Workflows

If your process requires a strict sequence of steps that must always execute in order, the most reliable design is:

1. Use code (an orchestrator or state machine) to enforce the required steps deterministically.
2. Then pass the final combined results to the LLM for reasoning. Optionally, use a Skill for consistent formatting.

Quick Decision Rule

• Need guaranteed execution? Use Tools + Orchestrator.
• Need consistent repeated reasoning/output? Use a Skill.
• Need a one-off behavior tweak? Use a Prompt.

What I Learned Building a Streaming Agent on AWS Bedrock AgentCore Runtime

I spent a week building a conversational agent on AWS Bedrock AgentCore Runtime. It supports:

• hybrid memory (fast in-session + persistent cross-session)
• real-time token streaming
• multi-user isolation
• a 12-test automated suite

Here is everything I wish someone told me before I started.

The full source code is available on GitHub: github.com/avparkhi/AWS-BedrockAgentCore-Testing

1. The MicroVM Mental Model

AgentCore doesn’t run your agent in a container you manage. It runs inside microVMs. These are tiny, isolated virtual machines that start per session. The first request to a new session triggers a cold start (~1.5–10s depending on what you initialize). Subsequent requests to the same session hit a warm microVM where all your Python globals are still in memory (~180ms overhead).

This means you can cache expensive things as module-level globals, and they persist across warm invocations. Examples: model clients, config, database connections. But the moment a user creates a new session, a new microVM starts and everything resets.

Think of it like AWS Lambda, but with a session-sticky routing layer on top.

2. Memory is a Two-Tier Problem

The session-sticky microVM model creates an interesting memory challenge:

• Within a session (warm starts): You want fast, in-process memory. A simple Python dictionary works. It has zero latency and makes no API calls. This is your RAM layer.
• Across sessions (cold starts, redeploys, crashes): You need persistent storage. AgentCore provides a Memory API for this. It is durable and survives everything, but it adds ~300ms per read.

The hybrid approach: always save to durable memory (background, on every message). On warm starts, use RAM. On cold starts, load from durable memory. The user never notices the difference.

3. Streaming Changes the User Experience Dramatically

Without streaming, a typical warm chat response takes ~5.4 seconds. The user sees nothing until the entire response is ready. With streaming, the first token appears in ~1–1.5 seconds. The total time is the same, but the perceived latency drops by 75%.

If your agent takes more than 2 seconds to respond, streaming isn’t a nice-to-have. It’s table stakes for user experience.

4. The SDK Has Opinions About Streaming (And They’re Inconvenient)

The AgentCore SDK supports streaming on the server side well. If your entrypoint returns a Python generator, the SDK automatically detects it and wraps the response as Server-Sent Events (SSE). You need no configuration.

The problem is on the client side. The SDK’s built-in invoke method handles streaming by printing chunks directly to the console. It then returns an empty dictionary. There is no programmatic access to the streamed content.

If you want to parse streaming responses in your own client, you need to bypass the SDK and call the API directly via boto3. This applies if you want to:

• show metadata
• measure time-to-first-token
• show formatted output

It’s not hard, but it’s undocumented and unexpected.

5. The Five Gotchas

These are the things that cost me the most time and are not documented anywhere I could find.

Gotcha #1: The Payload Isn’t a Real Dict

The payload your entrypoint receives looks like a Python dictionary, but it’s actually a JSONSerializableDict. It behaves differently in subtle ways:

• The two-argument form of .get(key, default) throws a TypeError. Only the single-argument form works.
• Bracket access (payload["key"]) throws a TypeError. There is no subscript support.
• Assignment (payload["key"] = value) also fails.

The workaround is to use single-argument .get() with an or fallback for defaults. It’s ugly, but it works reliably.

The same limitation applies to agent state objects in the Strands SDK. Don’t try to use them as dicts. Store your state elsewhere.

Gotcha #2: Return Values Get Double-Encoded

The SDK JSON-encodes whatever your entrypoint returns. If you return a JSON string, the client receives a JSON-encoded JSON string. That means escaped quotes, escaped newlines, the works. You then need to decode the response iteratively (sometimes 3–4 rounds) to get the original data.

In the end, I used a custom tag format for metadata instead of JSON, just to avoid the encoding nightmare.

Gotcha #3: Streaming Generators Have the Same Encoding Trap

This was the most frustrating bug. When your streaming generator yields values, the SDK JSON-encodes each one for the SSE wire format. If you pre-encode your dicts to JSON strings before yielding, the SDK encodes them again. The client receives a quoted string instead of a parseable dict.

The symptom: metadata objects appear as raw text inline with the chat response, and all metadata fields show as “unknown.”

The fix: yield raw Python objects (dicts, strings) and let the SDK handle serialization. Never call json.dumps() on values you’re about to yield from a streaming generator.

Gotcha #4: Non-Streaming Responses Silently Truncate at 1024 Bytes

There is no error. There is no warning. If your non-streaming response exceeds 1024 bytes, it is truncated mid-JSON. You spend an hour debugging your JSON parser before you realize the data was simply incomplete.

Keep non-streaming responses compact. For my memory query mode, I had to truncate each conversation turn to ~80 characters to stay under the limit.

Gotcha #5: A Generator Function Can’t Conditionally Stream

In Python, a function that contains yield anywhere in its body always returns a generator. You can’t conditionally choose between returning a string and yielding chunks from the same function.

The solution is a dispatch pattern. Your entrypoint is a regular function that returns either a generator object (for streaming) or a string (for non-streaming). The SDK inspects the return value’s type, not the function itself. You write two separate handler functions and one thin dispatcher.

6. Testing Observations

Test Through the Non-Streaming Path

All 12 of my automated tests use non-streaming mode. Streaming responses are harder to assert against (you’d need to reconstruct the full text from chunks). Non-streaming gives you a clean, complete response string to validate.

Streaming is a transport concern. If your logic works non-streaming, it works streaming. The same code runs either way, just yielded differently.

Memory Persistence Is the Key Test

The most important test has two steps. Send a math problem on session A. Then ask about the result on a completely different session B. Session B has empty RAM (new microVM). If the agent remembers the answer, durable memory works. If the response attributes the memory source as “durable,” the hybrid architecture works correctly.

Isolation Tests Are Easy to Get Wrong

Testing that User 2 doesn’t see User 1’s history is tricky because LLMs can coincidentally mention the same numbers. Testing that the response doesn’t contain both “50” and “48” (from User 1’s specific calculation chain) is more reliable than testing for either alone. Even so, these tests can be flaky.

Observed Latencies

The wide range on cold-start chat (8–17s) comes from two things: microVM provisioning and the first-time initialization of the model and memory clients. Subsequent cold starts are faster (~8s) because ECR image layers are cached.

7. Observability Setup

AgentCore supports three observability pillars, all routed through the runtime role’s IAM permissions:

The runtime role needs permissions for all three:

• create and write to log groups and log streams
• submit X-Ray segments and telemetry
• publish CloudWatch metrics, scoped to the bedrock-agentcore namespace

The GenAI Observability Dashboard in the CloudWatch console provides a unified view across all your AgentCore agents. It is worth bookmarking.

8. Deployment Notes

• CodeBuild is the default and easiest path. It builds ARM64 containers in the cloud. You do not need local Docker. Build time is ~40 seconds.
• Use your venv Python. On macOS, the system Python is typically 3.9.x. The SDK needs 3.11+. I wasted time on cryptic import errors before I realized I had the wrong interpreter.
• Memory resources are idempotent-ish. Creating a memory that already exists throws a ValidationException. You need to catch it and find the existing one via list. It is not a big deal, but it is not obvious from the API.
• The agent ARN lives in the YAML config (.bedrock_agentcore.yaml), not in your deploy info JSON. You need it for direct boto3 calls. The SDK handles this internally when you use its invoke method.
• Redeployment is simple. Update your code and run deploy. The agent updates in place. Session IDs reset, but durable memory persists. Warm microVMs from the old version drain naturally.

9. What I’d Do Differently Next Time

1. Build non-streaming first, add streaming last. Streaming is purely a transport layer concern. Get your agent logic, memory, and tools working with simple request/response. Then add streaming on top. It is miserable to debug streaming issues while your core logic is also broken.
2. Assume everything the SDK touches will be JSON-encoded. This includes return values, streaming yields, and headers. If the SDK handles it, expect encoding. Design your data formats around this from day one.
3. Keep non-streaming responses tiny. Because of the 1024-byte truncation limit, you should design response formats to be compact from the start. Do not try to shrink them later.
4. Write the ping test first. A no-LLM ping mode that returns cache status and invocation count is invaluable for debugging microVM behavior. It isolates platform issues from agent logic issues. Every new agent I build will start with this.
5. Use tags, not JSON, for non-streaming metadata. A simple delimited format like <<META:key=val,key=val>> survives the SDK’s encoding gauntlet intact. JSON metadata in non-streaming responses is a losing battle.