Production failuresbecome regression tests.
LLM observability that closes the loop: every agent trace graded as it lands, failures clustered into issues, bad runs frozen into tests that block the PR.
production trace→failure detection→regression test→CI gateCluster #12 — refund tool timeout ignored
23 traces this week · prod
Frozen as case rc_118
fixtures bundled · fail → pass contract
GATE ✗ BLOCKED
PR #214 reintroduces cluster #12
instruments any agent stack
From production incident to CI gate.
Four moves, no hand-authored datasets. Each one is derived from the last, and the trace starts it all.
Trace
Every production run streams in over OTLP — durable blob first, one indexed row per span. Agent, conversation and tool semantics are first-class columns, not strings.
Detect
Online evaluators grade each trace as it lands — LLM-as-judge at conversation, run or span level. One FAIL on a blocking evaluator flips the verdict.
Freeze
One click promotes a failing trace into a hermetic case: recorded input, tool and LLM fixtures bundled, a fail-to-pass contract attached.
Gate
tracely replay re-runs the suite on every PR — offline, deterministic, $0 in model spend — and exits non-zero on the merge that regresses it.
POST /v1/traces — 3 spans · agent support_bot
one trace through the write path — ingest, grade, cluster, gate
Your agents, acted out.
Every conversation replays as a scene. Delegations walk over and talk, knowledge is read at the library, tools run at the wall — and the failure raises its hand where it happened.
order_lookup runs search_orders
a sample turn · every conversation in the dashboard replays like this — Sessions → Fleet
You never author a test set. Production already wrote the perfect failing example — Tracely freezes it and guards it forever.
Everything downstream of one trace.
Scores, clusters, cases, gates and trends are all computed from the trace — so they never disagree with each other, or with production.
Evaluators as columns
Judges live where you look: every evaluator is a column on the trace table, and scores stream into the grid live over SSE as they run.
Failure clustering
Structural + semantic clustering groups failing traces into named issues — with a description, a proposed fix and a suggested evaluator to catch it next time.
Hermetic replay
Cases replay against recorded tool and LLM fixtures — no API keys, no model spend, no flakes. Add --live when you want real calls.
The PR gate
tracely gate exits non-zero, posts the commit status and upserts a PR comment. Two lines of YAML in the workflow you already have.
Trends & meta-analysis
Daily failure and gate pass-rates, plus per-agent cross-metric analysis: Spearman correlations, z-score outliers, LLM-synthesized findings.
Judge calibration
Label judge verdicts against human review, get per-evaluator agreement, and catch an over-flagging judge before you let it gate a release.
The PR that re-breaks prod never merges.
The gate replays your promoted cases on every pull request, exits non-zero on failure, and posts the verdict as a commit status and PR comment.
# .github/workflows/tracely.ymlon: pull_requestjobs: gate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 # → run your agent here; traces stream in with env=ci - uses: Jwuthri/Tracely/.github/actions/tracely-gate@master with: agent: support-agent api: https://tracely.your-co.dev key: ${{ secrets.TRACELY_KEY }}
fix: retry refund_api on timeout
#214 · mira wants to merge 3 commits
Tracely gate / gate (pull_request)
Required · failing after 42s
8f31c2a — await refund_api(order)
12/12 passed · gate green, merge unblocked
Three lines in.
Every span out.
tracely.init() switches on auto-instrumentation for whatever your agent already uses — OpenAI, Anthropic, LangChain / LangGraph, LiteLLM — and stamps agent, conversation and turn onto every span. Zero span code in your codebase; manual context managers remain the escape hatch.
import tracely_sdk as tracelyfrom openai import OpenAI tracely.init(endpoint="https://tracely.your-co.dev", api_key=os.environ["TRACELY_KEY"], service_name="support-agent", env="prod") with tracely.trace(agent="support-agent", conversation="conv-42"): OpenAI().chat.completions.create(...) # traced — zero span code
Ask your editor
what broke.
Every Tracely deployment is also an MCP server. Point Claude Code or Cursor at it and your coding agent reads the failing traces, opens the cluster behind them, and writes the evaluator that catches it next time — authenticated by an ordinary ingest key, scoped to that one workspace. Nothing to install, nothing extra to run.
# one line — the endpoint ships with the APIclaude mcp add --transport http tracely \ https://api.tracely-studio.xyz/mcp \ --header "Authorization: Bearer $TRACELY_KEY" › what failed in the last 20 traces, and add a column that catches it next time ✓ get_trace, list_clusters, create_evaluator …
MCP gives it your data.
The skill gives it the know-how.
One command and your coding agent knows how Tracely actually works — zero-span-code instrumentation, the manual span API, evaluator design, the PR gate, and the handful of conventions that fail silently when you get them wrong.
# plain Markdown — Claude Code, Cursor, Copilot …npx skills add https://github.com/Jwuthri/Tracely \ --skill tracely ✓ auto + manual tracing · evaluators · CI gate
Free to self-host.
Free to start hosted.
The whole product is MIT-licensed — API, worker, UI, evaluators, the CI gate. Run it yourself and pay nobody. The hosted plan exists so you don't have to run ClickHouse.
Self-host
The entire product, MIT-licensed, on your own infrastructure.
- Every feature — no paywalled internals
- Your traces never leave your network
- One-click deploy to Railway, or docker compose
- Unlimited traces, agents and seats
- Community support on GitHub
Free
Hosted, for trying it on a real agent without running ClickHouse.
- 20k traces / month
- 7-day trace retention
- All evaluators + failure clustering
- CI gate on one agent
- 3 workspaces, 3 seats
Team
For teams gating real releases on real production failures.
- 1M traces / month
- 90-day retention
- Unlimited agents + CI gates
- 10 workspaces and seats, one subscription
- Judge calibration + adversarial scenarios
- Email support
Bring your own model key. LLM judges run on your OpenRouter key, scoped to your workspace and encrypted at rest — we never bill you a markup on inference, and we never use a shared key. No key configured? The structural evaluators still grade every run; the LLM ones switch off cleanly.
Ship agents that don't regress.
Instrument an agent in two lines, send one trace, and the loop starts on its own — graded on arrival, clustered when it fails, frozen into a case that guards the next PR.
Built and maintained by Julien Wuthrich
11 passed · 1 failed — case rc_118 (cluster #12 · refund tool timeout) regressed at step 4/6. Trajectory diff attached. Merge blocked.