01
Agent Evaluation
Definition
The process of testing an agent's reasoning, tool calls, final answers, safety, cost, and latency before or during production use.
Back to Agentic AI Glossary
Agentic AI Glossary
Evaluation, Guardrails & Safety
Evaluation, Guardrails & Safety terms and explanations from the Agentic AI Glossary.
61 terms in this chapter
02
Benchmarking
Definition
Comparing an AI system against baselines, alternative models, datasets, or performance targets.
03
Confidence Scoring
Definition
Estimating how reliable an output, classification, tool call, or decision is likely to be.
04
Cost per Task
Definition
The average spend required for one completed task, including model tokens, tool calls, infrastructure, retries, and human review when needed.
05
Escalation Rate
Definition
The percentage of tasks or conversations that must be handed to a human, specialist agent, or higher-trust workflow.
06
Eval Dataset
Definition
A curated set of prompts, scenarios, expected behaviors, and edge cases used to test an AI system repeatedly.
07
Evaluation (Eval)
Definition
A structured process for measuring quality, safety, correctness, and business value of AI behavior.
08
Failure Rate
Definition
The share of tasks where the agent gives a wrong answer, calls the wrong tool, violates policy, or fails to finish.
09
False Negative
Definition
A missed detection, such as failing to flag a risky output, bad retrieval result, policy violation, or defect.
10
False Positive
Definition
An incorrect alert or block, such as flagging safe content, valid tool use, or correct output as unsafe.
11
Final Answer Evaluation
Definition
Reviewing the final response for correctness, groundedness, completeness, tone, policy compliance, and usefulness to the user.
12
Goal Completion Rate
Definition
The percentage of tasks where the agent reaches the requested outcome without unnecessary failure, escalation, or user rework.
13
Golden Dataset
Definition
A trusted set of examples, expected outputs, or human-labeled judgments used for evaluation.
14
Groundedness
Definition
How strongly an answer is supported by retrieved documents, verified data, tool results, or other trusted evidence.
15
Helpfulness
Definition
How well the response solves the user's real problem with clear, relevant, and actionable information.
16
Human Evaluation
Definition
Quality review performed by people who judge usefulness, correctness, safety, tone, and real-world task success.
17
Latency per Task
Definition
The total time required to finish one task, including model calls, retrieval, tool execution, retries, and final response generation.
18
LLM-as-Judge
Definition
Using a language model to assess outputs, often with rubrics, references, or pairwise comparisons.
19
Multi-Turn Evaluation
Definition
Testing whether an agent stays accurate, safe, and context-aware across a conversation or long-running workflow.
20
Pairwise Evaluation
Definition
Comparing two outputs side by side so reviewers or judge models can select the better answer or behavior.
21
Plan Accuracy
Definition
How well the generated plan matches the task requirements, dependencies, constraints, and expected order of execution.
22
Plan Quality
Definition
The usefulness, feasibility, and ordering of an agent's proposed steps.
23
RAG Evaluation
Definition
Testing retrieval and answer quality together, including context relevance, citation accuracy, faithfulness, and answer completeness.
24
Regression Test
Definition
A repeatable test that catches quality drops after model, prompt, retrieval, or tool changes.
25
Relevance
Definition
The degree to which retrieved context, tool output, or generated text directly answers the user's request.
26
Safety Score
Definition
A metric summarizing whether outputs and actions comply with safety and policy expectations.
27
Scenario Test
Definition
An evaluation case built around a realistic user situation, including inputs, constraints, expected behavior, and pass criteria.
28
Simulation
Definition
Controlled testing of agent behavior in representative scenarios before production exposure.
29
Simulation Test
Definition
Running agents inside controlled mock environments to test behavior before exposing them to live users or real systems.
30
Step-Level Evaluation
Definition
Checking each intermediate plan step, tool call, observation, and decision instead of only judging the final answer.
31
Task Success Rate
Definition
The percentage of tasks an agent completes according to predefined success criteria.
32
Token Usage
Definition
The number of input and output tokens consumed by a request, conversation, or task, used for cost and latency control.
33
Tool-Call Accuracy
Definition
How often an agent chooses the correct tool and passes the correct arguments.
34
Tool Selection Accuracy
Definition
How often the agent chooses the correct tool, with the correct arguments, for the user's intent and system constraints.
35
Trajectory Evaluation
Definition
Assessing the full sequence of agent thoughts, tool calls, observations, and revisions.
36
Approval Gate
Definition
A required human or policy checkpoint before the agent performs a risky, expensive, or irreversible action.
37
Auditability
Definition
The ability to reconstruct what the agent saw, decided, called, and produced for review or compliance.
38
Compliance Check
Definition
A validation step that verifies output or action meets legal, regulatory, contractual, or internal policy requirements.
39
Content Filter
Definition
A rule or model that blocks, labels, or redirects content that violates safety, quality, or policy standards.
40
Data Loss Prevention
Definition
Controls that detect and prevent sensitive data from being exposed, copied, logged, or sent to unsafe destinations.
41
Escalation
Definition
Routing a case to a human, specialist agent, or safer workflow when confidence, risk, or complexity requires it.
42
Escalation Path
Definition
A route from the agent to a human, specialist, or safer workflow when automation should not continue alone.
43
Fallback Response
Definition
A safe alternative answer used when the model is uncertain, retrieval fails, tools are unavailable, or policy blocks completion.
44
Fallback Strategy
Definition
A predefined alternative path when an agent has low confidence, fails, times out, or reaches a safety boundary.
45
Guardrail
Definition
A rule, check, model, or workflow constraint that keeps AI behavior safe, compliant, and aligned with expectations.
46
Guardrails
Definition
Rules, checks, filters, permissions, and approval gates that keep agent behavior safe and compliant.
47
Human Review
Definition
Manual inspection of an AI output, decision, or planned action before it is approved, revised, or rejected.
48
Input Guardrail
Definition
A check applied to user input before model processing, often detecting harmful requests, prompt injection, or sensitive data.
49
Jailbreak Detection
Definition
The ability to identify jailbreak signals in inputs, outputs, logs, retrieved content, or system behavior.
50
Moderation
Definition
Classifying content for safety categories so the system can allow, block, transform, or escalate it appropriately.
51
Output Guardrail
Definition
A check applied after generation to catch unsafe, incorrect, private, or non-compliant output before delivery.
52
PII Detection
Definition
The ability to identify pii signals in inputs, outputs, logs, retrieved content, or system behavior.
53
PII Redaction
Definition
Removing or masking personally identifiable information so it is not exposed to users, logs, models, or downstream tools.
54
Policy-as-Code
Definition
Representing rules and compliance logic in executable configuration so checks are consistent and auditable.
55
Policy Check
Definition
A validation step that compares a request, plan, tool call, or answer against approved rules.
56
Prompt Injection Detection
Definition
The ability to identify prompt injection signals in inputs, outputs, logs, retrieved content, or system behavior.
57
Refusal
Definition
A safe response that declines to help with disallowed or harmful requests while keeping the tone professional.
58
Risk Score
Definition
A numeric or labeled measure that estimates risk for an output, action, user experience, or workflow result.
59
Safe Completion
Definition
A response that answers within allowed boundaries while avoiding unsafe instructions, private data, or unsupported claims.
60
Safety Check
Definition
A pre- or post-processing validation that looks for harm, misuse, policy violation, or high-risk behavior.
61
Tool Permission Check
Definition
A verification that the agent is allowed to use a specific tool, data source, action, or permission scope.
Explore more chapters or test your knowledge with quizzes.