Per-attempt timeouts for every LLM step; the card says when a step didn't finish, adds up its checks and says what a check cost #261
No reviewers
Labels
No labels
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
kroshtan/unfret_backend!261
Loading…
Reference in a new issue
No description provided.
Delete branch "llm-step-timeouts"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Follow-up to #255, plus two result-card fixes.
1. A per-attempt timeout for every LLM step
#255 bounded the content read's attempts (20 s, one retry). The other LLM calls still had only the client's 60 s timeout, so a runaway attempt could push the analysis to its 90 s limit.
get_structured_llmnow appliesUNFRET_LLM_ATTEMPT_TIMEOUT_S(15 s) unless the caller sets its own. That covers classification (one or two calls), triage's question, the headline and the LLM judge (used only without a scorer). The review embeddings use it too. The content read keepsUNFRET_CONTENT_ATTEMPT_TIMEOUT_S(20 s).triageandanalysis_llm_usagelog lines across the 5 analyses in the dev logs. The content read, which writes the longest answer, takes 2-14 s. So 15 s is about 2x the slowest headline we saw.test_two_attempts_of_every_llm_step_fit_within_its_timeout):2. The card says when part of the check didn't finish
verdict.incomplete={"steps": ["content"], "text": ...}, under the same condition asassessment.degraded(a key skill unavailable). It is shown right under the headline on/check(and in the extension and Android, separate PRs):How charging works today (unchanged here): a degraded result is charged like any other. Only a score of -1, a failed or timed-out analysis, or a client that hung up is refunded. The result cache serves the same input for an hour, so re-checking within the hour returns the same result: free for the same account, a check for anyone else. That's why the line says "after an hour". A shared result shows only the first sentence.
3. "What we checked" adds up
checklist[].checksgives each area's count. A check that serves two areas (dark patterns) counts once, in the first.checks_runis now the sum of the listed areas, so an area the page doesn't have no longer adds hidden checks./checkshows "What we checked: 16 checks in 5 areas" and "Who's behind the shop · 4 checks" on each area; a clean area is read out as "nothing to note". Older results without counts keep the old heading.4. What a check cost, as a sentence
charge_noteis new on/v1/analyzeresponses and the stream result:The count is left out when it isn't known.
/checkshows it in place of the lone "One of your free checks.".Tests: attempt timeouts (graph steps, embeddings, default budget), the incomplete line (card, shared result, catalogue), counts adding up on every page kind, charge sentences (unit + integration + the JS renderer). OpenAPI snapshot and api-contract updated.
🤖 Generated with Claude Code
https://claude.ai/code/session_014HgVKRoNZC5gcKdzyYgCWJ
Every LLM step gets a time limit per attempt. Classification, triage, headline, the LLM judge and the review embeddings had only the client's 60 s timeout, so one runaway call could push the analysis to its 90 s limit. get_structured_llm now applies UNFRET_LLM_ATTEMPT_TIMEOUT_S (15 s) unless the caller sets its own (the content read keeps 20 s), and the embeddings use it too. These ask for a short answer; the headline takes 4-8 s on dev. Two attempts and the backoff fit within the skill timeout, and within the analysis timeout next to a skill round that runs to its limit (tested on the shipped defaults). Prompts and model settings are unchanged, so the recorded responses replay. The result card: - verdict.incomplete: when the AI read of the page failed or timed out (assessment.degraded), one line under the headline names the step, says the answer may be milder, that it still counted as a check, and that checking again after an hour may help (the result cache serves the same input for an hour). Shared results get the first sentence. - checklist[].checks: each area's number of checks; a check serving two areas counts in the first, and checks_run is their sum (an area the page doesn't have no longer adds to it). /check shows "What we checked: 16 checks in 5 areas" and "· N checks" per area; a clean area reads "nothing to note". - charge_note: what the check cost, as a sentence ("This check used one of your free checks; 4 are left.", "Quick checks are free.", ...), shown on /check instead of the lone "One of your free checks.". Charging is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014HgVKRoNZC5gcKdzyYgCWJView command line instructions
Checkout
From your project repository, check out a new branch and test the changes.