Content read: abandon a runaway attempt after 20 s and try once more #255

Merged
kroshtan merged 1 commit from content-timeout into main 2026-11-18 20:56:38 +00:00
Owner

Why the test shop's deep check came back degraded

Dev logged skill_timed_out for content at 20:16:37Z (45 s after the check began at 20:15:52Z). A clean shop checked straight after took 16 s.

It isn't the input size. The test shop's cleaned text is 3,721 characters (≈2,150 prompt tokens); the clean shop's is 2,279 (≈1,600). The content input is already capped at 20,000 characters.

It's the model running away now and then. Against dev's provider and model (Scaleway, mistral-small-3.2-24b-instruct-2506), using the content skill's exact prompt:

calls normal time runaways
test shop, one at a time 24 5.6–14.0 s 0
test shop, 8 at a time (plain and streamed) 192 2.2–13.3 s (median ≈ 9.5 s) 10
clean shop 32 7.3–12.6 s 0

In a runaway, the model writes the JSON correctly as far as "implausible_claim_examples": [...], " and then emits spaces, tabs or ```json without end: 2,500–7,000 tokens and 5–290 kB in 40–75 s+, still going when cut off. The reads quote the page's text verbatim, including its line breaks and quotes, so the arguments fill up with \\n and \\"; the test shop has many such quotes, and the clean shop has few. No single call was ever bounded: the client timeout is 60 s, longer than the skill's 45 s. So one runaway used up the whole step and the assessment was degraded.

The fix

throttled() takes an attempt timeout. An attempt that runs longer is abandoned and retried like a rate limit, with the same backoff and the same llm_max_retries. Waiting for a free concurrency slot doesn't count against it. The content skill uses 20 s (UNFRET_CONTENT_ATTEMPT_TIMEOUT_S). That is 6 s more than the slowest normal read measured, and two attempts plus the backoff (≤ 41 s) fit within the 45 s skill timeout. A test checks that the shipped defaults keep fitting.

Why this and not the alternatives:

  • Raising the skill timeout wouldn't help: a runaway doesn't finish, and normal reads are well inside 45 s.
  • max_tokens, or a client-side timeout=, change the model's serialized settings. Those are part of the key of the evaluation's recorded LLM responses (evals/llm_cache.py), so the golden-set replay would miss every content read. The attempt timeout is enforced around the call, so prompts and model settings stay the same.
  • Collapsing whitespace in the page text might remove the trigger. But it changes every content prompt, which means re-recording the corpus and possibly retraining the scorer, and the extension's text-cleaning parity has to match. That's a separate piece of work.
  • A smaller fallback model: the scorer was trained on this model's reads, so a different model's scores wouldn't mean the same.

Latency: normal reads are unchanged. A runaway now costs about 30 s (20 s, then a fresh read) instead of 45 s and a degraded verdict.

Measured with the fix: the skill run as the graph runs it (_isolated, 45 s) on the test shop, 2 × 64 calls, 8 at a time. 5 attempts timed out and all 5 retries succeeded (27–32 s), with no content timeout. The medians were 9.1 and 10.8 s. Separately, one read in the first batch failed after 10 s with a non-timeout error (skill_failed; the error type wasn't captured and it didn't happen again in the second batch). That is a different failure and isn't touched here.

Each abandoned attempt now logs llm_attempt_timed_out.

What a degraded card tells the shopper (unchanged here)

Nothing says that a step was missing. assessment.degraded is in the API response, but neither the web check page, the extension nor the Android app shows it. On a shop page, "The product and the brand" still reads checked, because listing ran. Only when no other skill in an area ran does that area say "Couldn't be checked this time." The shopper just sees a milder band ("Think twice") and fewer checks.

Tests

  • test_llm.py: a runaway attempt is abandoned and tried again; the last attempt raises TimeoutError; without an attempt timeout, a TimeoutError isn't retried; queueing for a slot isn't counted; get_structured_llm passes the timeout on.
  • test_skills.py: a runaway first read recovers within the skill timeout and leaves no content.unavailable; the defaults fit two attempts in the skill timeout.

make lint and make test (2242 passed) pass locally.

🤖 Generated with Claude Code

https://claude.ai/code/session_014HgVKRoNZC5gcKdzyYgCWJ

## Why the test shop's deep check came back degraded Dev logged `skill_timed_out` for `content` at 20:16:37Z (45 s after the check began at 20:15:52Z). A clean shop checked straight after took 16 s. **It isn't the input size.** The test shop's cleaned text is 3,721 characters (≈2,150 prompt tokens); the clean shop's is 2,279 (≈1,600). The content input is already capped at 20,000 characters. **It's the model running away now and then.** Against dev's provider and model (Scaleway, `mistral-small-3.2-24b-instruct-2506`), using the content skill's exact prompt: | | calls | normal time | runaways | |---|---|---|---| | test shop, one at a time | 24 | 5.6–14.0 s | 0 | | test shop, 8 at a time (plain and streamed) | 192 | 2.2–13.3 s (median ≈ 9.5 s) | 10 | | clean shop | 32 | 7.3–12.6 s | 0 | In a runaway, the model writes the JSON correctly as far as `"implausible_claim_examples": [...], "` and then emits spaces, tabs or `` ```json `` without end: 2,500–7,000 tokens and 5–290 kB in 40–75 s+, still going when cut off. The reads quote the page's text verbatim, including its line breaks and quotes, so the arguments fill up with `\\n` and `\\"`; the test shop has many such quotes, and the clean shop has few. No single call was ever bounded: the client timeout is 60 s, longer than the skill's 45 s. So one runaway used up the whole step and the assessment was degraded. ## The fix `throttled()` takes an attempt timeout. An attempt that runs longer is abandoned and retried like a rate limit, with the same backoff and the same `llm_max_retries`. Waiting for a free concurrency slot doesn't count against it. The content skill uses **20 s** (`UNFRET_CONTENT_ATTEMPT_TIMEOUT_S`). That is 6 s more than the slowest normal read measured, and two attempts plus the backoff (≤ 41 s) fit within the 45 s skill timeout. A test checks that the shipped defaults keep fitting. Why this and not the alternatives: - **Raising the skill timeout** wouldn't help: a runaway doesn't finish, and normal reads are well inside 45 s. - **`max_tokens`, or a client-side `timeout=`**, change the model's serialized settings. Those are part of the key of the evaluation's recorded LLM responses (`evals/llm_cache.py`), so the golden-set replay would miss every content read. The attempt timeout is enforced around the call, so prompts and model settings stay the same. - **Collapsing whitespace in the page text** might remove the trigger. But it changes every content prompt, which means re-recording the corpus and possibly retraining the scorer, and the extension's text-cleaning parity has to match. That's a separate piece of work. - **A smaller fallback model**: the scorer was trained on this model's reads, so a different model's scores wouldn't mean the same. Latency: normal reads are unchanged. A runaway now costs about 30 s (20 s, then a fresh read) instead of 45 s and a degraded verdict. **Measured with the fix**: the skill run as the graph runs it (`_isolated`, 45 s) on the test shop, 2 × 64 calls, 8 at a time. 5 attempts timed out and all 5 retries succeeded (27–32 s), with no content timeout. The medians were 9.1 and 10.8 s. Separately, one read in the first batch failed after 10 s with a non-timeout error (`skill_failed`; the error type wasn't captured and it didn't happen again in the second batch). That is a different failure and isn't touched here. Each abandoned attempt now logs `llm_attempt_timed_out`. ## What a degraded card tells the shopper (unchanged here) Nothing says that a step was missing. `assessment.degraded` is in the API response, but neither the web check page, the extension nor the Android app shows it. On a shop page, "The product and the brand" still reads *checked*, because `listing` ran. Only when no other skill in an area ran does that area say "Couldn't be checked this time." The shopper just sees a milder band ("Think twice") and fewer checks. ## Tests - `test_llm.py`: a runaway attempt is abandoned and tried again; the last attempt raises `TimeoutError`; without an attempt timeout, a `TimeoutError` isn't retried; queueing for a slot isn't counted; `get_structured_llm` passes the timeout on. - `test_skills.py`: a runaway first read recovers within the skill timeout and leaves no `content.unavailable`; the defaults fit two attempts in the skill timeout. `make lint` and `make test` (2242 passed) pass locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_014HgVKRoNZC5gcKdzyYgCWJ
Content read: abandon a runaway attempt after 20 s and try once more
All checks were successful
CI / deploy-dev (pull_request) Has been skipped
CI / check (pull_request) Successful in 2s
b00fe34f23
Now and then (about 1 in 20 reads of the test shop, under load) the model quotes the page's multi-line text, loses
track of its JSON and writes whitespace until it is cut off. With no bound on one attempt, that used up the content
skill's whole 45 s (the client timeout is 60 s), so the check came back degraded: no content findings, confidence
capped at 0.3, "Think twice" instead of "Serious concerns".

throttled() now takes an attempt timeout: an attempt that runs past it is abandoned and retried like a rate limit
(waiting for a free slot doesn't count against it). The content skill uses 20 s (UNFRET_CONTENT_ATTEMPT_TIMEOUT_S;
normal reads take 2-14 s), so two attempts plus the backoff fit within the skill timeout. The timeout is enforced
around the call, not set on the client, so the model settings, and the evaluation's recorded responses keyed by
them, stay the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014HgVKRoNZC5gcKdzyYgCWJ
kroshtan deleted branch content-timeout 2026-11-18 20:56:38 +00:00
Sign in to join this conversation.
No description provided.