Pillar 03 · measure
Evaluation Lifecycle
The evaluation lifecycle is what separates demos from production. Agents are non-deterministic, so quality, safety, cost, latency and drift must be measured continuously — offline against fixed datasets in CI and pre-promotion, and online against real production traffic via tracing, shadow runs and user feedback. Every result feeds back into Development as new test cases and into Deployment as promotion gates.
Modules involved
plan hub →kaif-platformThe technology substrate every other module runs on.kaif-designBusiness use case → agentic design proposal, evaluated offline and online.kaif-evalOne evaluation spine, four surfaces — design, build, deploy, runtime.kaif-guardAll guardrails and policies. Publishes bundles kaif-deploy enforces.kaif-valueValue reporting and KPIs. Business-legible dashboards from raw signals.
What good looks like
Outcomes
- Every agent version has reproducible offline scores before promotion.
- Every production request is captured as an OpenTelemetry GenAI span.
- Online quality, cost and latency regressions raise alerts within the same hour.
- Safety violations are blocked at runtime and reappear as fixtures in the next eval run.
- Datasets, judges and rubrics are versioned — yesterday's scores are still reproducible today.
The loop
How it actually runs
- step 01Capture spans (OTel)
- step 02Score online (judges · feedback)
- step 03Run offline suites (Ragas · DeepEval)
- step 04Aggregate (Langfuse · Phoenix)
- step 05Gate promotion · alert
- step 06Feed failing cases back to dev
Cloud-native enablers
Technology on Kubernetes
Every capability maps to a CNCF or Kubernetes-native project. No parallel stack — the same cluster runs your agents and your platform.
Offline evaluation
- RagasRAG faithfulness, answer relevancy, context precision.
- DeepEvalPytest-style assertions for LLM outputs.
- promptfooMatrix eval across prompts, models and providers.
- Inspect AIStructured eval harness for agent tasks.
- Argo WorkflowsRun eval suites on schedule, on PR, on new model.
- Tekton ChainsAttach eval results as artifact attestations.
Online evaluation
- Istio mirroringShadow real traffic to candidate agents.
- Argo Rollouts AnalysisRunPromote / abort canaries on live metrics.
- LLM-as-judge serviceScore live responses against rubrics.
- Feedback serviceCapture thumbs / edits / accept-rate from users.
- A/B routing (Envoy AI GW)Split live traffic across agent variants.
Observability fabric
- OpenTelemetry (GenAI semconv)One standard for LLM, tool and agent spans.
- OpenLLMetryDrop-in OTel instrumentation for LLM SDKs.
- PrometheusMetrics: tokens, latency, cost, error rate.
- TempoDistributed traces across agent and tool calls.
- LokiStructured logs correlated with traces.
- GrafanaUnified dashboards and alerting.
LLM-specific tracing & analytics
- Langfuse (self-hosted)Per-prompt analytics, dataset capture, judge runs.
- Arize PhoenixTrace inspection, embeddings drift, evals UI.
- HeliconeCost and latency observability per request.
Storage · data plane for eval
- CloudNativePGEval results, feedback, judge scores.
- MinIO / S3Eval datasets, trace exports, golden outputs.
- ClickHouseHigh-cardinality span analytics at scale.
- Weaviate / QdrantDrift detection on embeddings of prod inputs.
Safety · policy
- OPAPolicy as code over agent decisions — appears in eval too.
- NeMo GuardrailsConversational safety rails, scored as eval signal.
- Llama GuardInput/output classification sidecar.
- GarakRed-team / probe attacks as part of the eval suite.
Cross-cut
How the Harness plugs in
The Harness exposes a single Eval Gateway. Every span — from dev laptops, CI runs and production — lands in the same store, scored against the same versioned rubrics. Deployment promotion checks read from it. Development pulls failing examples back into datasets. The eval loop is the feedback mechanism that makes the other two pillars trustworthy.