Thread safety in lru_cache keeps its internal state coherent during concurrent updates. It does not guarantee one computation for a cold key. Two requests can arrive before the first result is cached and each execute the wrapped function. Avoiding that duplicate work requires coordinating in-progress computation, then checking the cache again after acquiring coordination rights, with explicit eviction and failure policies.

阅读中文版. This distinction matters for model-configuration loading, expensive feature calculations, or cached embedding requests. A healthy-looking hit rate does not prove the backend performed only one cold computation. The example uses synthetic string results, with no model, network, or GPU calls and no cost or performance estimate.

A coherent cache can still allow duplicate work

A result cache holds completed values that can be reused. Same-key request coalescing coordinates work that has not finished. One asks whether a value exists; the other asks who is producing it and what other requests should do. Before a result is available, an ordinary cache need not contain an in-progress operation for waiters to subscribe to.

The Python functools documentation explicitly separates those guarantees: the cache supports multithreaded use, but the wrapped function may be called again before the initial call finishes and is cached. A lock somewhere inside a decorator does not establish that the whole business calculation runs under it.

The Python implementation in CPython v3.13.0 illustrates the mechanism. Its bounded-cache branch looks up and maintains cache state under a lock, executes a missed user call outside that lock, then returns to the lock to check and update state. The file also imports a C accelerator when available. We did not force the local runtime to execute the Python fallback: source reading explains the design, while the experiment below exercises the public API.

Putting every expensive operation under one global lock could suppress overlapping work while also making unrelated keys wait for each other. Identify the state that needs protection separately from the business work that needs coalescing. “Add a lock” is not a sufficient design description.

Reproduce the ordering with a barrier

The first experiment calls duplicate(“same”) from two worker threads. Each invocation increments a protected counter inside the function body, then waits at a Barrier. The barrier releases only after both calls have entered. A fast first call cannot finish and turn the second into a cache hit before the intended overlap occurs.

After both Futures return the same string, the body counter is two. cache_info reports two misses and one current entry. A subsequent sequential call hits the cache and leaves the body counter at two. These observations are compatible: concurrent misses performed separate work, while later calls can still reuse the cached result.

Barrier and Event objects control ordering; bounded waits prevent a failed test from hanging indefinitely. They are not sleeps used to guess thread progress. The experiment does not measure elapsed time or throughput for different keys. Rewriting “fewer duplicate executions” as “twice as fast” would exceed its evidence.

Acquire the key lock, then check the cache

The second demonstration briefly holds a registry lock to obtain the Lock for a key. It then leaves the registry lock and calls the cached function inside that key’s lock. While A holds the key lock and executes the body, B can obtain a reference to the same lock but must wait before calling cached. Once A returns and caches its value, B can check that cache.

The order is the point: acquire key-level coordination before deciding whether the cached result exists. If both requests conclude outside the lock that a value is missing, then take turns executing already-decided work, waiting merely serializes the duplicate computations. The decision to produce a value must be reconsidered after acquiring key-level coordination.

The demonstration uses entered to confirm that the first body started and attempted to confirm that the second request obtained its key-lock reference. Before release is set, it checks that the body executed only once and neither outer Future completed. After release, both calls return the same value; statistics show one miss, one hit, and one entry. No other keys evict that value and nobody clears the cache during this experiment.

from functools import lru_cache
from concurrent.futures import ThreadPoolExecutor
from threading import Barrier, Event, Lock
import sys

checks = 0
def check(ok):
    global checks
    assert ok
    checks += 1

counter_lock = Lock()
calls = 0
both = Barrier(2, timeout=3)
@lru_cache(maxsize=8)
def duplicate(key):
    global calls
    with counter_lock:
        calls += 1
    both.wait()  # Both cold calls entered before either can return.
    return 'value:' + key

with ThreadPoolExecutor(max_workers=2) as pool:
    a = pool.submit(duplicate, 'same')
    b = pool.submit(duplicate, 'same')
    check(a.result(timeout=4) == b.result(timeout=4) == 'value:same')
check(calls == 2)
i = duplicate.cache_info()
check(i.misses == 2 and i.currsize == 1)
check(duplicate('same') == 'value:same')
check(calls == 2 and duplicate.cache_info().hits == 1)

# Teaching registry: no eviction, recursion, cancellation or failure policy.
registry = {}
registry_lock = Lock()
attempts = 0
attempted = Event()
entered = Event()
release = Event()
serial_calls = 0
@lru_cache(maxsize=8)
def cached(key):
    global serial_calls
    with counter_lock:
        serial_calls += 1
    entered.set()
    if not release.wait(3):
        raise RuntimeError('cleanup gate timed out')
    return 'value:' + key

def merged(key):
    global attempts
    with registry_lock:
        key_lock = registry.setdefault(key, Lock())
        attempts += 1
        if attempts == 2:
            attempted.set()
    with key_lock:
        return cached(key)  # Recheck cache after acquiring the key lock.

with ThreadPoolExecutor(max_workers=2) as pool:
    a = pool.submit(merged, 'same')
    b = pool.submit(merged, 'same')
    try:
        check(entered.wait(1) and attempted.wait(1))
        with counter_lock:
            check(serial_calls == 1)
        check(not a.done() and not b.done())
    finally:
        release.set()
    check(a.result(timeout=2) == b.result(timeout=2) == 'value:same')
check(serial_calls == 1)
i = cached.cache_info()
check(i.misses == 1 and i.hits == 1 and i.currsize == 1)
print(checks, 'checks passed; Python', sys.version.split()[0])
print('same-key cold body calls: lru_cache=2; key-lock plus cache=1')

The program ran on Python 3.15.0 and passed 11 checks. Its return values are immutable strings. It does not establish that cached mutable arrays, model instances, or connection objects are safe to share. What object is reused, and whether callers mutate it, requires a separate review of return-value semantics.

maxsize does not bound the lock registry

The teaching code deliberately leaves registry entries in place forever. lru_cache(maxsize=8) limits result-cache entries, not locks in an external dictionary. A service receiving an unbounded stream of new keys grows that dictionary. This is not a bounded-memory production implementation.

Simply popping the key when a computation ends introduces a race. Suppose B already holds a reference to old lock L1 and is waiting. A finishes and removes K from the registry; C subsequently creates lock L2 for K. B and C now use different locks, so the same-key exclusion rule has been lost. Reclamation has to account for every holder and waiter, not just the completion of one calculation.

Reference counting or explicit in-progress entries are possible management approaches, but their creation, waiter registration, completion notification, and removal must be coordinated. This article does not implement a complete single-flight component. The key-lock/cache demonstration teaches ordering. If other keys evict the result while a request waits, or someone clears the cache, a later caller can compute again. Do not promise that the entire waiting queue necessarily receives one shared result.

The example uses ordinary non-reentrant Locks. It is unsuitable for same-key recursive calls or cyclic dependencies between keys, which can wait on each other. Passing the successful-path checks does not make the wrapper suitable for arbitrary recursive computation.

Define failures, timeouts, and replica scope separately

A real component needs answers to several questions. When a producer fails, do waiters receive the same exception or compete to retry? Does one waiter’s timeout affect work still running? If the component retains failure state, for how long, and when may the next request become producer? These are interface policies, not decisions a lock makes for you. Failure propagation and cancellation were not tested here.

A caller stopping its wait also does not establish that background work stopped. Threaded model computation or remote requests need a lifecycle distinct from request timeout; see asyncio timeouts and worker threads. Waiter capacity needs its own limit too. Reducing duplicate work can still leave a large queue of pending requests.

Both registry and cache live within one process. Different service processes or replicas have separate state, so this example establishes no cross-replica coalescing. If the operation includes a remote write, caching and key locks also provide no end-to-end exactly-once guarantee. The receiving system still needs explicit deduplication and retry and idempotency rules.

For AI services, first define what a key means

For an embedding-style calculation, the key must represent conditions that change the result’s meaning, such as model revision, preprocessing rules, and output configuration. Reusing by text alone while changing the model can produce an excellent hit rate with semantically stale values. If results depend on access scope or tenant-specific data, those contexts need correct isolation as well.

This example accepts one string argument, so the registry and cache use the same key. For multiple arguments, keyword arguments, or custom objects, align both key rules first. Calls that the cache considers equivalent must not acquire unrelated locks.

Normalize keys according to application semantics. Two similar-looking strings are not necessarily equivalent to a model or its preprocessing. Removing spaces or punctuation merely to increase hits can change the computation being requested. Conversely, equivalent requests mapped to different keys bypass same-key coordination. These are cache-correctness considerations, not claims about a particular model SDK’s defaults.

For deployment acceptance, distinguish incoming request count, actual function-body count, cache hits and misses, waiter count, and failures. Begin with fixed cold requests for one key, then verify different keys, eviction, failure, and cancellation against explicit policies. Only then is there evidence for which path the component supports. Final cache size cannot substitute for a count of computations actually performed.

This is a mechanism explanation using CPython v3.13.0 source that predates the intended article date. The documentation version is not the locally executed version. The article was backfilled on October 25, 2026, retaining its intended October 23, 2026, 09:00 Beijing article date.