When LLM traffic grows, do not create a task for every request and assume a semaphore solves admission. A semaphore can limit simultaneous calls without limiting tasks already waiting in memory. A bounded queue with a fixed worker pool makes waiting capacity and execution capacity separate, explicit controls.

A slow dependency can hide an infinite queue

Suppose every arriving request immediately creates a task waiting to call a model. When the dependency slows, active calls remain modest while waiting tasks accumulate. They may retain prompts, session state, callbacks and timeout contexts. Not having sent a network request does not mean consuming no resources.

Active-connection counts alone can miss this. The service may stay alive while waits grow and cancelled user work continues in the background. Increasing coroutine counts does not create downstream throughput; it can retain more work that is already too late to finish usefully.

Backpressure communicates that more work cannot currently be accepted. A producer may wait, attempt admission for a bounded period, or receive a clear rejection. Waiting must itself have limits rather than moving pressure from a visible list into an invisible task collection.

A small capacity model makes the limit visible. Assume a sustained arrival rate of twelve equal-work requests per second and a stable processing capacity of eight. Ignoring startup boundaries, backlog grows by about four requests per second, or forty over ten seconds. Expanding the queue to a thousand slots does not remove that gap. It delays saturation while allowing more work to wait. A bounded queue makes the capacity mismatch visible at admission instead.

Those rates are constructed assumptions, not model-service measurements. Real requests vary in length, arrivals fluctuate and quotas may be allocated per caller. Examine sustained load and bursts separately. One short peak does not prove that permanent expansion is needed, while an acceptable average does not establish acceptable tail waits. Record queue age as well as count so that a few very old items are not hidden by a small backlog.

Do not collapse three limits into one concurrency setting

Queue capacity counts admitted items not yet taken by workers. Execution concurrency counts work currently calling the dependency. The service also has its own limits, such as request rate or token budgets. Concurrency two does not mean two requests per second; short calls can turn over many times.

The asyncio.Queue documentation explains that positive maxsize makes put wait when full, whereas default zero does not supply that capacity bound. This example uses two fixed workers, so at most two mock calls run concurrently. It does not create a separate waiting task for every input.

A queue limit of two still does not mean only two requests exist in the process. There are active workers, an item held by the producer and any work buffered by the entry framework. An HTTP service allowing every connection to wait indefinitely for admission can still collect many ingress tasks. Bound admission waits, connections, request-body size and total duration as well.

Queue waiting consumes the user’s budget too

A request may wait for admission, retrieve data, call a model and postprocess. Giving each stage a fresh full 30 seconds can exceed the user’s acceptable total. A clearer design establishes an overall deadline at entry and lets later stages use the remaining budget.

Expired queued work should not consume a costly call. After taking an item, a worker needs to check whether it remains useful. For clarity, the program below bounds only the mock-call wait; it does not implement end-to-end deadlines and is not a complete gateway.

Rejection should distinguish temporary capacity, a caller’s quota and invalid input. Labeling everything a model error can induce pointless retries. Timeouts on tools with effects also need the idempotency contract before replay.

Removing an item is not completing it

Queue.get reduces the waiting length, but the work may only be starting. Using qsize as total in-flight load understates activity when busy. Observe waiting and active counts separately.

Every successfully retrieved item eventually needs exactly one task_done so join can account for admitted work. Completion is queue bookkeeping, not business success. Record failures and finish their bookkeeping too, or join can wait forever for a counter that will never decrease.

A finally block corresponding to get covers normal returns, handled failures and cancellation cleanup. A task cancelled before get succeeds must not invent a task_done call. The example also accounts for its stop markers so the exit path does not leak unfinished-work counts.

A local bounded worker pool

import asyncio

async def check_boundaries():
    q = asyncio.Queue(maxsize=1)
    q.put_nowait("first")
    blocked = asyncio.create_task(q.put("second"))
    await asyncio.sleep(0)
    assert not blocked.done()
    assert q.get_nowait() == "first"
    q.task_done()
    await blocked
    assert q.get_nowait() == "second"
    q.task_done()
    await q.join()

    cleanup = []
    async def never_finishes():
        try:
            await asyncio.Event().wait()
        finally:
            cleanup.append(True)
    try:
        await asyncio.wait_for(never_finishes(), timeout=0.01)
    except asyncio.TimeoutError:
        pass
    else:
        raise AssertionError("Expected timeout")
    assert cleanup == [True]

async def main():
    await check_boundaries()
    q = asyncio.Queue(maxsize=2)
    active = peak = max_queued = 0
    outcomes = []

    async def mock_call(item):
        nonlocal active, peak
        active += 1
        peak = max(peak, active)
        try:
            await asyncio.sleep(0)
            if item == 3:
                raise ValueError("Simulated invalid request")
            return f"result-{item}"
        finally:
            active -= 1

    async def worker():
        while True:
            item = await q.get()
            try:
                if item is None:
                    return
                try:
                    await asyncio.wait_for(mock_call(item), timeout=1)
                except (asyncio.TimeoutError, ValueError) as error:
                    outcomes.append((item, type(error).__name__))
                else:
                    outcomes.append((item, "ok"))
            finally:
                q.task_done()

    workers = [asyncio.create_task(worker()) for _ in range(2)]
    for item in range(6):
        await q.put(item)
        max_queued = max(max_queued, q.qsize())
    for _ in workers:
        await q.put(None)
    await q.join()
    await asyncio.gather(*workers)
    assert 1 <= peak <= 2 and max_queued <= 2
    assert active == 0 and len(outcomes) == 6
    assert sum(status == "ok" for _, status in outcomes) == 5
    print("peak active:", peak, "max queued:", max_queued)
    print(sorted(outcomes))

asyncio.run(main())

The original Python program was executed. Boundary checks show a full queue blocks its producer until an item is removed. A separate local check shows wait_for timeout triggers the cancelled coroutine’s finally block. The worker pool handles six mock tasks: one deliberate failure and five successes, with active and waiting counts never above two.

No model service was called and no throughput benchmark was run. Simulated failure is not a model-quality result. The outcomes list retains results for demonstration; an indefinitely running service needs a sink or bounded result storage rather than creating another unbounded queue. None is reserved as a stop marker and cannot also be a valid business item.

Cancelling locally does not prove the server stopped

The asyncio task documentation explains cleanup and propagation of cancellation. For network work, ending a local wait differs from revoking remote computation. Connection closure, continued generation and charging depend on the client and service contract.

After a timeout, do not immediately claim resources were released or execution never happened. Streaming consumers should close streams according to the client’s contract and check that background tasks exit. Write tools need outcome lookup or replay of the original operation, not a new intent because waiting ended.

The example turns a local ValueError into a failed result and continues processing. It does not swallow cancellation of the entire worker. Production code also needs supervision for unexpected worker exits, a policy for outstanding items and a shutdown procedure that avoids an undrainable queue.

Request counts hide differences between long and short work

Two short prompts and two requests near the context limit impose different memory, token and completion-time costs. A count-only limit hides that variation. Limit input size before admission, then assess whether grouping, priorities or weighted budgets fit the workload.

Those policies have costs too. Always favoring short requests can starve long ones, while strict arrival order can leave short requests behind long work. No scheduling policy is universally optimal. Observe request distribution and service goals before comparing waits and completion across important categories.

An application queue also differs from the engine’s continuous batching. The former controls admission and call timing; the latter schedules generation inside inference. Changing client-worker counts does not replace engine memory and batch management.

Observe more than aggregate throughput

Record waiting length, active calls, admission-to-start duration, and counts of completion, failure, expiry and rejection. A queue that never drains and waits that keep growing indicate an arrival-versus-capacity mismatch even when successful call durations look normal.

Exercise bursts, slow work, failures, cancellation and normal shutdown. Every admitted item should reach a final category. Reporting only successful completions can make delay look deceptively low. Use the evaluation denominator and recording rules to explain load and outcomes before changing limits or adding capacity.

A usable backpressure design does not promise to accept everything. It admits work that can be completed, gives excess work an explicit outcome and puts every wait somewhere observable and bounded. A slow dependency then has fewer opportunities to become a silent application-memory problem.

阅读中文版

Sources

• Python3.13 — asyncio queues
• Python3.13 — Semaphores
• Python3.13 — Task cancellation

Backfilled on the evening of October 14, 2026, retaining the planned 19:00 Beijing article date. Code uses local mock work, not measured model-service traffic.