Build notes · AI Architecture & Evaluation

Engineering Klaros AI WhatsApp Agent with Diligent Evaluation

1 September 2026 · Founder, Klaros

Nuts and Bolts of Building a Production Grade Voice and Text Product Orchestrator.

1. The Human Gap: Priya’s Voice Note at 11:42 PM

At 11:42 PM on a Tuesday, Priya sends a 14-second WhatsApp voice note to a boutique jeweler: “Bhaiya, mera order #ORD-9821 cancel kardo aur ₹4,500 account mein refund kar do, main abhi travel kar rahi hoon.”

A 3-star chatbot powered by basic prompt engineering does one of two bad things. Either it responds with a polite promise (“Your order is canceled!”) while failing to call the refund backend, or it misidentifies the noisy order ID and cancels someone else's booking. We saw early forms of this in the auto-reply that answered a hello with a price list and the assistant that could explain booking but not book.

Digital distance is the gap between what a customer believes your business remembers about them and what your software actually executes. Closing that distance in an AI voice and text agent requires more than telling an LLM to “be careful.” It requires code-level boundaries that refuse to act when evidence is uncertain, and an evaluation rig that proves those boundaries hold under production pressure.

2. The Four-Layer Truth Architecture

Most AI platforms evaluate their models solely on **natural language output**—asking whether the generated text sounds fluent and polite. In our earlier As-Is AI Architecture note, we outlined a rules-first decision tree. But in a transactional CRM handling real money and order fulfillment (see software that asks for money), text fluency is dangerously deceptive.

Klaros evaluates every conversational turn across **Four Layers of Truth**:

If an agent generates a customer response saying “I have updated your address to Indiranagar,” but Layer 3 reveals the D1 database write failed or wrote to the wrong column, the test fails immediately. Fluency without state fidelity is marked as a critical defect.

3. Multi-Modal Voice Safeguards & Self-Governing Agentic Boundaries

Real-world AI Governance is not a static policy PDF sitting on a server—it is a deterministic execution harness. An AI system is genuinely self-governing when it measures its own uncertainty in real-time and voluntarily surrenders control (failing closed to a human colleague) rather than guessing when context is ambiguous.

WhatsApp voice notes (.ogg / Opus audio) present severe challenges: background traffic noise, regional accents, mixed Hinglish vocabulary, and rapid speech. When we outlined our To-Be Autonomous AI Roadmap, native voice processing and self-governing boundaries were top priority.

Instead of feeding raw Automatic Speech Recognition (ASR) output directly into the reasoning engine, Klaros implements a **Critical-Word Micro-Confidence Safeguard** inside src/shared/voice-adapter.mjs:

Code Snippet: Micro-Confidence Thresholding in voice-adapter.mjs

// Evaluate overall transcription AND individual critical-entity micro-confidences
const overallConf = transcription.confidence;
const criticalEntities = extractCriticalTokens(transcription.words); // Order IDs, Currency, Action Verbs

const minCriticalConf = Math.min(...criticalEntities.map(w => w.confidence));

if (overallConf < 0.70 || minCriticalConf < 0.70) {
  // FAIL-CLOSED: Hand off to human inbox with zero autonomous writes
  return triggerHumanHandoff({
    reason: "LOW_VOICE_CONFIDENCE",
    audioUrl: message.mediaUrl,
    transcript: transcription.text,
    overallConf,
    minCriticalConf
  });
}

If overall confidence or critical entity confidence (e.g. order numbers or currency figures) drops below **0.70**, the self-governing boundary triggers. The system transcribes the note, flags the conversation, and routes it to the human team inbox with **zero autonomous state mutations**.

4. The agent-eval v1.0 Testing & Mutation Rig

To ensure our agent never regresses, we constructed tools/agent-eval/, an automated testing framework comprising four specialized evaluation tracks. Much like our investigation into the contacts we silently stopped scoring, we require every test to prove it can fail before trusting it:

In Track B2 (Synthetic Defect Mutation), we proved our evaluation harness actually catches bugs by deliberately breaking our own code. The harness detected **100% of injected mutations (20/20)**, proving that passing tests reflect true system integrity rather than overly permissive assertions.

5. The Infrastructure Fidelity Budget Ladder (F0 to F5)

A test suite that passes in local Node.js but fails on Cloudflare Workers edge infrastructure is useless. In our work on the inbox that re-downloaded itself every visit and polling DOM rebuilds, we saw how infrastructure assumptions break under real traffic. We established the **Infrastructure Fidelity Budget Ladder** to guarantee parity across runtimes:

F5: Production Observation ────── Live customer traffic observation & trace harvesting
 ▲
F4: Live Provider Sandbox ─────── Deployed Staging Worker + Live Deepgram / LLM / Meta sandboxes
 ▲
F3: Deployed Staging ──────────── Cloudflare Staging Worker (createTestHarness) + Remote D1/DO/Queue
 ▲
F2: Local Production Runtime ──── Real Cloudflare workerd engine + Local D1 SQLite + DO + Queue
 ▲
F1: Real Code In-Process ──────── Node.js + Real Klaros Modules + Controlled Mocks (CI Baseline)
 ▲
F0: Pure Unit Tests ───────────── Isolated function assertions

By running Track C0 (local workerd) and Track C1 (deployed Cloudflare staging), we certified **100% outcome parity between F1, F2, and F3 runtimes**, eliminating edge-runtime surprises before touching production code.

6. Zero-Downtime Controlled Production Rollout Protocol

We reject "push and pray" deployments. Promoting candidate build 4144d7d to production followed an 8-step machine-gated sequence:

7. Empirical Receipts & Auditable Machine Gates

At each rollout stage (25% and 50%), our decision engine consumed raw event counts and derived metrics directly (following the same verification rigor we applied in the billing code we fixed before charging anyone), eliminating percentage transcription ambiguity:

Every gate evaluation produced a canonical RFC 8785 JSON Canonicalization Scheme (JCS) SHA-256 digest string, independently verified by the deployment controller before advancing traffic:

{
  "hash_verification": {
    "stored_hash": "4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f",
    "recomputed_hash": "4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f",
    "match": true,
    "canonicalization_version": "1.0-JCS"
  }
}

8. Unflattering Lessons & What Remains Open

A build note that only reports triumphs is marketing, not engineering. Here is what was hard, what cost us time, and what remains open:

1. **Synthetic Hash Placeholders**: In early test iterations, our logger emitted dummy SHA-256 hashes (e.g. e3b0c442... or a1b2c3d4...). Caught during audit, we replaced report strings with strict machine verification over canonical RFC 8785 JCS byte strings.
2. **Percentage Ambiguity**: Our early gate definitions checked error_rate < 0.01%, creating confusion when candidate error rate reached 0.010%. We updated the decision engine to operate on explicit raw integer ratios (errors / eligible ≤ 0.0001).
3. **What Remains Open**: Track B4 Harvester is currently operating in **Observe-Only Quarantine Mode**. Expanding the automated quarantine approval pipeline for non-PII edge cases into auto-generated benchmark PRs is scheduled for release `v1.1`.

Related: Klaros Chatbot As-Is AI Architecture · To-Be Autonomous Agent Architecture