Content read: abandon a runaway attempt after 20 s and try once more #255
No reviewers
Labels
No labels
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
kroshtan/unfret_backend!255
Loading…
Reference in a new issue
No description provided.
Delete branch "content-timeout"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why the test shop's deep check came back degraded
Dev logged
skill_timed_outforcontentat 20:16:37Z (45 s after the check began at 20:15:52Z). A clean shop checked straight after took 16 s.It isn't the input size. The test shop's cleaned text is 3,721 characters (≈2,150 prompt tokens); the clean shop's is 2,279 (≈1,600). The content input is already capped at 20,000 characters.
It's the model running away now and then. Against dev's provider and model (Scaleway,
mistral-small-3.2-24b-instruct-2506), using the content skill's exact prompt:In a runaway, the model writes the JSON correctly as far as
"implausible_claim_examples": [...], "and then emits spaces, tabs or```jsonwithout end: 2,500–7,000 tokens and 5–290 kB in 40–75 s+, still going when cut off. The reads quote the page's text verbatim, including its line breaks and quotes, so the arguments fill up with\\nand\\"; the test shop has many such quotes, and the clean shop has few. No single call was ever bounded: the client timeout is 60 s, longer than the skill's 45 s. So one runaway used up the whole step and the assessment was degraded.The fix
throttled()takes an attempt timeout. An attempt that runs longer is abandoned and retried like a rate limit, with the same backoff and the samellm_max_retries. Waiting for a free concurrency slot doesn't count against it. The content skill uses 20 s (UNFRET_CONTENT_ATTEMPT_TIMEOUT_S). That is 6 s more than the slowest normal read measured, and two attempts plus the backoff (≤ 41 s) fit within the 45 s skill timeout. A test checks that the shipped defaults keep fitting.Why this and not the alternatives:
max_tokens, or a client-sidetimeout=, change the model's serialized settings. Those are part of the key of the evaluation's recorded LLM responses (evals/llm_cache.py), so the golden-set replay would miss every content read. The attempt timeout is enforced around the call, so prompts and model settings stay the same.Latency: normal reads are unchanged. A runaway now costs about 30 s (20 s, then a fresh read) instead of 45 s and a degraded verdict.
Measured with the fix: the skill run as the graph runs it (
_isolated, 45 s) on the test shop, 2 × 64 calls, 8 at a time. 5 attempts timed out and all 5 retries succeeded (27–32 s), with no content timeout. The medians were 9.1 and 10.8 s. Separately, one read in the first batch failed after 10 s with a non-timeout error (skill_failed; the error type wasn't captured and it didn't happen again in the second batch). That is a different failure and isn't touched here.Each abandoned attempt now logs
llm_attempt_timed_out.What a degraded card tells the shopper (unchanged here)
Nothing says that a step was missing.
assessment.degradedis in the API response, but neither the web check page, the extension nor the Android app shows it. On a shop page, "The product and the brand" still reads checked, becauselistingran. Only when no other skill in an area ran does that area say "Couldn't be checked this time." The shopper just sees a milder band ("Think twice") and fewer checks.Tests
test_llm.py: a runaway attempt is abandoned and tried again; the last attempt raisesTimeoutError; without an attempt timeout, aTimeoutErrorisn't retried; queueing for a slot isn't counted;get_structured_llmpasses the timeout on.test_skills.py: a runaway first read recovers within the skill timeout and leaves nocontent.unavailable; the defaults fit two attempts in the skill timeout.make lintandmake test(2242 passed) pass locally.🤖 Generated with Claude Code
https://claude.ai/code/session_014HgVKRoNZC5gcKdzyYgCWJ