Wrapping a synchronous LLM client in asyncio.wait_for does not guarantee that its background work stops after a timeout. A caller can receive TimeoutError and the asyncio task can report cancelled while the function inside its thread continues running. Retrying immediately or returning a concurrency permit at that point can leave more work in flight than the application intended.

A local experiment using threads and events makes the distinction visible without contacting a model or sending network requests. An asyncio semaphore with capacity one can coexist with two active worker threads. Separate the caller stopping its wait, cancellation of an awaitable and actual worker termination before defining an interface’s timeout guarantee. 中文版.

A timeout first changes the waiting relationship

The Python wait_for documentation specifies cancellation of the awaited object and TimeoutError on timeout. Cancellation itself is awaited, so elapsed waiting may exceed the configured interval. A timeout is consequently neither a forced-stop switch for arbitrary synchronous functions nor a precise millisecond-level total-duration guarantee.

For a cooperative coroutine, cancellation injects CancelledError at an appropriate suspension point, and cleanup may still run. The boundary here is different: the coroutine is awaiting a synchronous function that has already started in a thread. Cancelling that waiting relationship does not transform the ordinary function into a coroutine that can be interrupted at any point. Before a queued thread task starts, cancellation may prevent execution. That is why the experiment confirms started first and limits its conclusion to work already running.

The CPython v3.13.0 to_thread implementation submits a function through run_in_executor and awaits its result. The executor Future cancellation contract separately explains that a running call cannot be stopped by cancel. An asyncio.Task and a concurrent.futures.Future are different objects. The first object’s cancelled state does not establish that the second object’s underlying work exited.

How capacity one can coexist with two workers

Consider an async with semaphore block containing wait_for(to_thread(…)). The permit follows the coroutine’s scope. Once the wait times out, that scope can exit even though the thread has not finished. The next caller obtains the released permit and starts another worker while the previous one still occupies resources.

That semaphore limits coroutines within its scope. It does not automatically count the real work that remains alive after those scopes close. An actual SDK may add connection pools, retries and remote queues. The example establishes a mismatch between two local lifecycles, not a prediction of any particular provider’s connection count or billing behavior.

The test first waits for an explicit started signal before beginning its short timeout. This avoids confusing cancellation before thread execution with cancellation after execution has begun. A second call repeats the sequence after the first caller times out. A closed gate keeps both workers alive until both request scopes have exited, eliminating accidental completion as an explanation for the observations.

Thirteen checks observe work after cancellation

Each worker increments active, waits at gate, appends its own name to effects once the gate opens, then decrements active and signals finished. These are local teaching states, not file writes, external API calls or actual business side effects.

import asyncio,threading,sys
from concurrent.futures import ThreadPoolExecutor
checks=0
def check(ok):
    global checks
    assert ok
    checks+=1
async def main():
    asyncio.get_running_loop().set_default_executor(ThreadPoolExecutor(max_workers=4))
    gate=threading.Event();lock=threading.Lock()
    state={'active':0,'peak':0};effects=[];jobs=[]
    sem=asyncio.Semaphore(1)
    def worker(name,started,finished):
        with lock:
            state['active']+=1
            state['peak']=max(state['peak'],state['active'])
        started.set()
        try:
            if not gate.wait(3):
                raise RuntimeError('test cleanup gate timed out')
            with lock:effects.append(name)
            return name
        finally:
            with lock:state['active']-=1
            finished.set()
    async def call(name):
        started=threading.Event();finished=threading.Event()
        async with sem:
            task=asyncio.create_task(asyncio.to_thread(worker,name,started,finished))
            jobs.append((task,finished))
            check(await asyncio.to_thread(started.wait,1))
            try:await asyncio.wait_for(task,.02)
            except TimeoutError:check(True)
            else:raise AssertionError('expected timeout')
            check(task.cancelled())
            check(not finished.is_set())
    try:
        await call('one')
        await call('two')
        with lock:
            check(state['active']==2)
            check(state['peak']==2)
            check(effects==[])
    finally:
        gate.set()
        for task,finished in jobs:
            if not await asyncio.to_thread(finished.wait,1):
                raise RuntimeError('worker did not finish')
    with lock:
        check(state['active']==0)
        check(set(effects)=={'one','two'})
    print(checks,'thread-lifecycle checks passed; Python',sys.version.split()[0])
    print('async semaphore capacity1; observed worker peak',state['peak'])
    print('late effects after caller timeouts',sorted(effects))
asyncio.run(main())

The experiment ran with the Python 3.15.0 standard library. Thirteen checks passed: both workers started; both callers timed out; both asyncio tasks became cancelled; neither finished event was set at its caller’s timeout; active and peak reached two; opening the gate produced both late effects; and active ultimately returned to zero. The reference implementation is pinned to CPython v3.13.0. That is not the runtime version used for the experiment.

The twenty-millisecond interval triggers a timeout in this small demonstration; it is not a machine-latency measurement. Event waits also have one-second and three-second failure bounds. A finally block opens the gate and waits for workers to exit rather than leaving indefinitely blocked threads behind. These teaching intervals are not proposed production settings.

A timed-out caller does not establish a failed operation

Before the gate opens, the caller knows only that it did not obtain a result in time. It cannot establish that the worker succeeded, nor that the worker will do nothing later. Both names appearing in effects after the gate opens provide a local counterexample to that second assumption.

For a real tool call, distinguish at least three states: whether the caller timed out, whether the local worker remains active and whether the remote operation completed. Even a closed local connection does not by itself establish that a remote operation was revoked. That depends on the service’s protocol and execution state rather than the local exception class.

Before retrying, determine whether repeating the operation is safe, and resolve ambiguous outcomes using operation identifiers or status lookup where the service supports them. The earlier idempotent agent-tool retry article addresses duplicate-submission semantics. This article supplies earlier lifecycle evidence rather than inventing another idempotency protocol.

Attach permits to the lifetime actually being limited

If the goal is to bound real in-flight synchronous work, a work-management layer should retain its handles and release corresponding resource permits after actual completion or confirmed termination. Releasing a permit merely because a request coroutine timed out is a different policy. A caller may return a timed-out, outcome-unknown response while a controlled executor continues supervising the work.

Adding shield everywhere is not a complete repair. Shield changes cancellation propagation; it cannot forcibly terminate a running thread. Retained background tasks still require references, eventual result or exception collection, bounded queues and shutdown handling. No shield variant was executed here, so it is not presented as a verified complete solution.

A thread pool’s max_workers limits simultaneously executing pool threads. It does not automatically add backpressure to unlimited submissions or terminate existing work. This example explicitly uses four workers so that two blocked workers and short event-checking helper calls can run together. Setting the pool to one changes the experiment. Its observed peak must not be confused with a default pool configuration.

The suitable mechanism depends on the operation: a cooperative loop can inspect a stop event; a cancellable or deadline-aware SDK should use its documented protocol; work requiring stronger isolation may need a separate process or external worker. Requesting a stop and confirming completion of that stop remain distinct states. Those alternative mechanisms were not tested in this article.

Retain separate timelines

When diagnosing an LLM gateway timeout, record request start, caller timeout, local work start and finish, and any available remote-operation state. Count active requests separately from active work to expose a quiet frontend with accumulating background jobs. Avoid placing prompts, credentials or private user data in those logs.

The connection to backpressure and queues is the identity of the resource being admitted: queued requests, awaiting coroutines or started operations. Naming an object Semaphore does not make that resource definition for the application.

If this executable cancellation counterexample helps locate threads that remain active after a timeout, voluntary support through the section below is welcome. The reusable result is a distinction between started, timeout and finished evidence, plus a check that permit release follows the work lifetime actually being limited.

Sources and backfill note

Python wait_for · Executor Future.cancel · CPython v3.13.0 to_thread

Checked against Python’s cancellation contracts and the pinned CPython v3.13.0 implementation. The original experiment ran the local Python 3.15.0 standard library, without models, networking or remote cancellation protocols. Article date: October 22, 2026 at 19:00 Beijing; actual backfill time is recorded separately and does not imply earlier publication.