Per-attempt timeouts for every LLM step; the card says when a step didn't finish, adds up its checks and says what a check cost #261

Open
kroshtan wants to merge 1 commit from llm-step-timeouts into main
Owner

Follow-up to #255, plus two result-card fixes.

1. A per-attempt timeout for every LLM step

#255 bounded the content read's attempts (20 s, one retry). The other LLM calls still had only the client's 60 s timeout, so a runaway attempt could push the analysis to its 90 s limit.

  • get_structured_llm now applies UNFRET_LLM_ATTEMPT_TIMEOUT_S (15 s) unless the caller sets its own. That covers classification (one or two calls), triage's question, the headline and the LLM judge (used only without a scorer). The review embeddings use it too. The content read keeps UNFRET_CONTENT_ATTEMPT_TIMEOUT_S (20 s).
  • Why 15 s: each of these asks for a short answer (a page type, yes/no, one sentence). On dev the headline took 4-8 s; that's the gap between the triage and analysis_llm_usage log lines across the 5 analyses in the dev logs. The content read, which writes the longest answer, takes 2-14 s. So 15 s is about 2x the slowest headline we saw.
  • The budget, tested on the shipped defaults (test_two_attempts_of_every_llm_step_fit_within_its_timeout):
    • 2 x 15 + 1 s backoff = 31 s, which is under the 45 s skill timeout (embeddings).
    • 31 s + a 45 s skill round = 76 s, which is under the 90 s analysis timeout (graph steps).
    • The content read: 2 x 20 + 1 = 41 s, under 45 s.
  • Model settings and prompts are unchanged, so the cache keys stay the same.

2. The card says when part of the check didn't finish

verdict.incomplete = {"steps": ["content"], "text": ...}, under the same condition as assessment.degraded (a key skill unavailable). It is shown right under the headline on /check (and in the extension and Android, separate PRs):

Part of this check didn't finish (the AI read of the page), so this answer may be milder than it should be. It still counted as a check. Checking again after an hour may help.

How charging works today (unchanged here): a degraded result is charged like any other. Only a score of -1, a failed or timed-out analysis, or a client that hung up is refunded. The result cache serves the same input for an hour, so re-checking within the hour returns the same result: free for the same account, a check for anyone else. That's why the line says "after an hour". A shared result shows only the first sentence.

3. "What we checked" adds up

checklist[].checks gives each area's count. A check that serves two areas (dark patterns) counts once, in the first. checks_run is now the sum of the listed areas, so an area the page doesn't have no longer adds hidden checks. /check shows "What we checked: 16 checks in 5 areas" and "Who's behind the shop · 4 checks" on each area; a clean area is read out as "nothing to note". Older results without counts keep the old heading.

4. What a check cost, as a sentence

charge_note is new on /v1/analyze responses and the stream result:

  • "This check used one of your free checks; 4 are left."
  • "This check used one of your checks; 23 are left."
  • "Quick checks are free."
  • "You checked this page recently, so no new check was used."
  • "This page couldn't be assessed, so no check was used."

The count is left out when it isn't known. /check shows it in place of the lone "One of your free checks.".

Tests: attempt timeouts (graph steps, embeddings, default budget), the incomplete line (card, shared result, catalogue), counts adding up on every page kind, charge sentences (unit + integration + the JS renderer). OpenAPI snapshot and api-contract updated.

🤖 Generated with Claude Code

https://claude.ai/code/session_014HgVKRoNZC5gcKdzyYgCWJ

Follow-up to #255, plus two result-card fixes. ## 1. A per-attempt timeout for every LLM step #255 bounded the content read's attempts (20 s, one retry). The other LLM calls still had only the client's 60 s timeout, so a runaway attempt could push the analysis to its 90 s limit. - `get_structured_llm` now applies `UNFRET_LLM_ATTEMPT_TIMEOUT_S` (**15 s**) unless the caller sets its own. That covers classification (one or two calls), triage's question, the headline and the LLM judge (used only without a scorer). The review embeddings use it too. The content read keeps `UNFRET_CONTENT_ATTEMPT_TIMEOUT_S` (20 s). - Why 15 s: each of these asks for a short answer (a page type, yes/no, one sentence). On dev the headline took 4-8 s; that's the gap between the `triage` and `analysis_llm_usage` log lines across the 5 analyses in the dev logs. The content read, which writes the longest answer, takes 2-14 s. So 15 s is about 2x the slowest headline we saw. - The budget, tested on the shipped defaults (`test_two_attempts_of_every_llm_step_fit_within_its_timeout`): - 2 x 15 + 1 s backoff = 31 s, which is under the 45 s skill timeout (embeddings). - 31 s + a 45 s skill round = 76 s, which is under the 90 s analysis timeout (graph steps). - The content read: 2 x 20 + 1 = 41 s, under 45 s. - Model settings and prompts are unchanged, so the cache keys stay the same. ## 2. The card says when part of the check didn't finish `verdict.incomplete` = `{"steps": ["content"], "text": ...}`, under the same condition as `assessment.degraded` (a key skill unavailable). It is shown right under the headline on `/check` (and in the extension and Android, separate PRs): > Part of this check didn't finish (the AI read of the page), so this answer may be milder than it should be. It still counted as a check. Checking again after an hour may help. How charging works today (unchanged here): a degraded result is charged like any other. Only a score of -1, a failed or timed-out analysis, or a client that hung up is refunded. The result cache serves the same input for an hour, so re-checking within the hour returns the same result: free for the same account, a check for anyone else. That's why the line says "after an hour". A shared result shows only the first sentence. ## 3. "What we checked" adds up `checklist[].checks` gives each area's count. A check that serves two areas (dark patterns) counts once, in the first. `checks_run` is now the sum of the listed areas, so an area the page doesn't have no longer adds hidden checks. `/check` shows "What we checked: 16 checks in 5 areas" and "Who's behind the shop · 4 checks" on each area; a clean area is read out as "nothing to note". Older results without counts keep the old heading. ## 4. What a check cost, as a sentence `charge_note` is new on `/v1/analyze` responses and the stream result: - "This check used one of your free checks; 4 are left." - "This check used one of your checks; 23 are left." - "Quick checks are free." - "You checked this page recently, so no new check was used." - "This page couldn't be assessed, so no check was used." The count is left out when it isn't known. `/check` shows it in place of the lone "One of your free checks.". Tests: attempt timeouts (graph steps, embeddings, default budget), the incomplete line (card, shared result, catalogue), counts adding up on every page kind, charge sentences (unit + integration + the JS renderer). OpenAPI snapshot and api-contract updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_014HgVKRoNZC5gcKdzyYgCWJ
Per-attempt LLM timeouts for every step; the card says when a step didn't finish, adds up its checks and says what a check cost
All checks were successful
CI / check (pull_request) Successful in 8m14s
CI / deploy-dev (pull_request) Has been skipped
f7a07bf138
Every LLM step gets a time limit per attempt. Classification, triage, headline, the LLM judge and the review
embeddings had only the client's 60 s timeout, so one runaway call could push the analysis to its 90 s limit.
get_structured_llm now applies UNFRET_LLM_ATTEMPT_TIMEOUT_S (15 s) unless the caller sets its own (the content read
keeps 20 s), and the embeddings use it too. These ask for a short answer; the headline takes 4-8 s on dev. Two attempts
and the backoff fit within the skill timeout, and within the analysis timeout next to a skill round that runs to its
limit (tested on the shipped defaults). Prompts and model settings are unchanged, so the recorded responses replay.

The result card:
- verdict.incomplete: when the AI read of the page failed or timed out (assessment.degraded), one line under the
  headline names the step, says the answer may be milder, that it still counted as a check, and that checking again
  after an hour may help (the result cache serves the same input for an hour). Shared results get the first sentence.
- checklist[].checks: each area's number of checks; a check serving two areas counts in the first, and checks_run is
  their sum (an area the page doesn't have no longer adds to it). /check shows "What we checked: 16 checks in 5
  areas" and "· N checks" per area; a clean area reads "nothing to note".
- charge_note: what the check cost, as a sentence ("This check used one of your free checks; 4 are left.", "Quick
  checks are free.", ...), shown on /check instead of the lone "One of your free checks.". Charging is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014HgVKRoNZC5gcKdzyYgCWJ
All checks were successful
CI / check (pull_request) Successful in 8m14s
Required
Details
CI / deploy-dev (pull_request) Has been skipped
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin llm-step-timeouts:llm-step-timeouts
git switch llm-step-timeouts
Sign in to join this conversation.
No description provided.