eval-watch — discipline layer over the agent stack

The problem

The trilogy ships its own eval discipline per repo — ARA has a 20-account golden runner, Signal Monitor has offline replay tests with hallucination-bait fixtures, Meeting Prep Agent has six Tier-1 goldens with code-enforced clamps. But each repo’s eval is its own island. There’s no unified answer to how do you know your agents — all of them — are right? That’s the question ~85% of GTM Engineer JDs ask and almost no candidate has a concrete answer to.

The naive answers fail the question:

• “I run pytest locally” — table stakes; no discipline visible
• “My CI runs evals” — better, but per-repo CI badges don’t tell a unified story
• “I have a dashboard” — sounds good but is usually unmaintained

eval-watch answers it with one auditable artifact: a STATUS.md committed back to a public repo every month, showing each sibling’s pass/fail + cost + wall-clock + regressions-vs-baseline + per-dimension histograms over time.

What it does

On a monthly GitHub Actions cron (or any manual workflow_dispatch trigger), eval-watch:

1. Checks out eval-watch + all four sibling repos
2. Subprocess-invokes ARA’s evals/eval_runner.py against an 8-golden subset
3. Subprocess-invokes OCC’s evals/eval_runner.py (4 governance goldens — 2 refusals at $0, 2 live sequences)
4. Subprocess-invokes SM’s pytest evals/replay/
5. Subprocess-invokes MPA’s pytest evals/
6. Aggregates per-sibling pass/fail + cost + wall-clock into SQLite
7. Compares to an anchored baseline run — flags any previously-passing golden that now fails
8. Renders unicode-block histograms of per-dimension scores over the last 6 runs
9. Writes STATUS.md at repo root + commits + pushes via the workflow’s bot identity

Hiring managers landing on the repo see the latest STATUS.md at the front door. GH Actions runs are visible at /actions — the discipline is auditable from outside.

Architectural decisions worth defending

1. Adapter pattern, not upstream --eval-json PRs. The cleanest move would be to add a stable --eval-json flag to each sibling’s eval entry (same shape as the --json flag that powers Meeting Prep Agent’s subprocess composition). For V1, the adapter pattern in eval-watch keeps the project self-contained — no upstream PRs across three repos. Adapters are ~100 lines each; if a sibling’s eval format ever drifts, you fix one adapter. The “modular not coupled” thesis still holds: eval-watch composes the trilogy via CLI subprocess, not Python imports.

2. Sequential composition (ARA → SM → MPA), not parallel. Same Anthropic Tier 1 rate-limit lesson Meeting Prep Agent learned. ARA’s internal parallel super-step + SM’s six parallel producers + MPA’s pipeline would, if fired concurrently, burst past 50k Haiku tokens/min and trigger 429s. Sequential composition raises wall-clock from ~12 to ~15 minutes per run, runs deterministically, and lets save_run(partial) after each adapter survive mid-run crashes.

3. Monthly cadence + 8-golden ARA subset = ~$30/year ops cost. ARA’s full 20-golden eval costs ~$5-8/run. The 8-golden subset (clean buyer-fit, hybrid disqualifier-override, peer-vendor edge cases, hard disqualifier — four classes × 2 goldens each for redundancy) preserves diagnostic coverage at ~60% cost reduction. Monthly cadence catches drift well within Anthropic’s ~2-month model-bump cycle without burning per-week cost.

4. STATUS.md + GH Actions, not email + dashboard. The audience is the hiring manager landing on the repo, not the operator reading email. STATUS.md is rendered by GitHub at the repo root, badged from the README, and committed back by the workflow’s bot identity. Anyone who clicks into the repo sees the latest status at the front door.

5. The regression gate is golden-level, not run-level. Computes three sets per sibling per run: regressions (baseline passing → now failing — red), newly fixed (baseline failing → now passing — informational), still-failing (failing in both — pre-existing, not counted as regression). A run with simultaneous regressions and newly-fixed goldens shows both. Pre-existing failures don’t trigger the gate.

6. Sixth instance of the code-over-prompt rule pattern. Disqualifier clamp in ARA → length compliance in the blog autopilot → three clamps in Signal Monitor → concurrency clamp in SM runtime → four clamps + sequential composition in MPA → eval discipline enforced in CI → Flow gates + task guardrails in Outbound Campaign Crew (the seventh). Not a one-shot. A way of thinking that shows up across surfaces.

7. The adapter contract is framework-agnostic — proven, not claimed. V1.1 added Outbound Campaign Crew as the fourth sibling: the first CrewAI repo under the same discipline layer as three LangGraph ones. Nothing in the adapter contract cares what orchestrates the agent — anything with a subprocess-runnable eval that exits non-zero on failure plugs in. One framework’s discipline layer would be a framework habit; two is an architecture.

See a real STATUS

From the live 2026-06-11 CI run (the first with all four siblings) — note this is an honest snapshot, ARA’s pre-existing vendor-vs-buyer mislabeling included:

# eval-watch status

> ✗ 1/4 siblings failing · $2.900 total · 479.5s wall-clock
> Last updated: 2026-06-11 16:24 UTC · Triggered by: ci · Run: 2026-06-11T16-24-29Z

## Per-sibling

| Sibling | Goldens | Cost | Wall-clock | Notes |
|---|---|---|---|---|
| ARA | 5/8 ✗ | $2.600 | 6.7m | failing: anthropic.com, clay.com, apollo.io |
| SM  | 5/5 ✓ | $0    | 1.0s | pytest evals/replay/ |
| MPA | 22/22 ✓ | $0  | 870ms | pytest evals/ |
| OCC | 4/4 ✓ | $0.300 | 1.3m | python evals/eval_runner.py |

## Regression gate

Baseline: 2026-06-09T23-04-31Z
✓ 1 newly fixed: ARA retool.com — was failing in baseline, now passing
(3 goldens failing in both baseline and current — pre-existing, not regressions.)

## Score histograms (last 6 runs, oldest → newest)

### ARA

| Dimension | Median trend | Current | Baseline | Drift |
|---|---|---|---|---|
| fit_score              | ▁▃▅▅█▇▇ | 4.10 | 4.00 | ↑2.5% |
| icp_confidence         | ▆▇▇█▇▇▇ | 0.84 | 0.82 | ↑2.4% |
| revops_maturity        | ▄▄▅▅▅▅▅ | 3.50 | 3.50 | —    |
| critic_score           | ▆▆▇▆▇▇▇ | 4.20 | 4.10 | ↑2.4% |

The agent stack — and the discipline layer above it

Repo	Answers	Framework
Account Research Agent	Who fits	LangGraph
Signal Monitor	When to act	LangGraph
Meeting Prep Agent	What to say	LangGraph
Outbound Campaign Crew	What to send	CrewAI
eval-watch (this tool)	How do you know your agents — all of them — are right?	—

eval-watch sits above the siblings as the discipline layer. It doesn’t compose with them at the level Meeting Prep Agent composes ARA + SM; it watches them. The trilogy stays at three pinned repos on the GitHub profile — eval-watch is reached via the sibling READMEs and standalone URL.

Use it

git clone https://github.com/aehirota/eval-watch
cd eval-watch
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env  # fill in ARA_REPO_PATH, SM_REPO_PATH, MPA_REPO_PATH, OCC_REPO_PATH
                      # (ANTHROPIC_API_KEY for ARA + OCC; FIRECRAWL_API_KEY for ARA)

python run.py --sibling sm        # smoke test (offline, $0)
python run.py                     # full run (ARA + OCC + SM + MPA, ~$2.90)
python run.py --set-baseline RUN_ID  # anchor regression gate
python run.py --list-runs         # historical runs
cat STATUS.md                     # generated report

GitHub Actions config in .github/workflows/eval-watch.yml. Architecture deep-dive in docs/architecture.md. Honest limitations in KNOWN_ISSUES.md — V1 doesn’t include cross-version model drift detection, an offline ARA fixture path, or a web dashboard. Trade-offs documented with concrete reasoning.