Maintenance intake triage
checking mode

The model proposes. Code decides.

Every place a wrong answer could cost money or trust has a deterministic check in front of it. Here is the whole thing, then what each piece is for, then what changes for a production client.

Pipeline architectureA tenant message passes an intake scan, then Claude produces one structured tool call, then the draft reply is scanned, then a code-owned gate routes either to an auto-routed ServiceNow incident or to a needs-review incident. A model outage goes straight to needs review. The golden eval set gates whether autopilot is allowed.Tenant messagesms, email, web formIntake scanlength cap, injection scanClaudesubmit_triage(…)single tool call, cached prefixReply guardrailsmoney, legal, links, datesGatepure code, no modelconf ≥ 0.80, category knownautohumanServiceNow incident, routedcategorized, prioritized, reply readyServiceNow incident, needs reviewunassigned, reasons in work notesmodel unavailable: intake still lands, flagged for a personone incident per message, keyed by correlation_id; a replay returns the existing oneGolden set, 26 casesunder 95% keeps the gate in draft mode
Boxes are the modules in src/lib; arrows are the order the code runs in.

The pipeline

  1. Intake checks. Length cap and a pattern scan for instruction-like text. The message is treated as data from here on; nothing in it can call a tool.
  2. Claude, one tool. The only tool is submit_triage, whose input schema is generated from the same Zod schema the server validates against. The model cannot invent fields, and if it ever answers in prose instead of calling the tool, the request fails open to a person. The system prompt carries a cache breakpoint, so repeat calls read the stable prefix from cache.
  3. Reply guardrails. The drafted tenant reply is scanned for money, legal language, links, phone numbers, and repair-date promises. Any hit holds the reply as a draft.
  4. The gate. Pure code, no model. Auto-route only when confidence clears the threshold, the category is known, and no guardrail fired. Otherwise the incident is created unassigned with a needs-review flag. Emergencies are auto-routed and flagged for a phone call.
  5. ServiceNow. One incident per message, keyed by correlation_id. A replay returns the existing incident instead of creating a second one. Transient failures retry with backoff; auth failures do not.
  6. Fail open to a person. If the model is down, the incident is still created, marked for human triage. Intake is never blocked by the AI step.

What the evals are for

Twenty-six cases with answers a property manager would sign off on. The pass rate is the release gate: below 95%, the system stays in draft mode and a person approves every route. The set is small on purpose so a miss is readable, and it grows every time a reviewer corrects the model, which is how the threshold earns its way up.

What changes for a production client

  • Intake becomes a webhook into a queue with an idempotent worker, not a synchronous request. The gate logic does not change.
  • Basic auth becomes an OAuth client-credentials grant scoped to the incident table; the mock adapter stays for CI.
  • Incident creation moves behind an outbox with a unique correlation key, which closes the small window where two identical requests could both pass the lookup.
  • Categories map to the client's own taxonomy and assignment groups through configuration, not prompt edits.
  • Every human correction is logged against the request and feeds the golden set; repeated corrections become a prompt or rule change.
  • Traces ship to the client's observability stack as the same JSON lines this app already writes.

Run it yourself

cp .env.example .env.local   # add ANTHROPIC_API_KEY and, optionally, SERVICENOW_*
npm install
npm run dev                   # http://localhost:3000
npm test                      # unit tests, no network
npm run eval                  # golden set from the command line