Corrupted Chinese in an LLM stream can originate in a client that treats every network read as complete UTF-8 text. Transport chunks may split a character, and complete characters may still form only part of an SSE event. Those boundaries require separate state.

The executable Python example below tests byte, character, line, and event boundaries independently. 阅读中文版.

A read is not a token

A network read returns available bytes, not a guaranteed model token, JSON object, or event. Server writes and client reads need not correspond one to one. Updating business state directly from each received chunk turns transport scheduling into an accidental protocol delimiter.

Identify the actual response format first. SSE needs event parsing; other formats need their own framing rules. An interface named stream does not establish shared completion markers or error-message structure across providers.

Replacement decoding does not repair fragmentation

A Chinese character can span several UTF-8 bytes. A chunk containing only its beginning may fail standalone decoding. Using errors=replace inserts replacement characters; independently decoding the next chunk does not restore the original character. A visible exception has become silent corruption.

Python’s codecs documentation provides stateful incremental decoders. They retain incomplete trailing bytes until another chunk arrives. The final=True call at EOF distinguishes temporary incompleteness from a genuinely truncated input.

A complete character is not a complete event

The WHATWG SSE specification defines UTF-8 streams, blank-line event processing, and joining multiple data fields with newlines. One data line does not establish that the event has ended. Assemble the event before interpreting its application payload.

If a service defines data as JSON, parse JSON after assembling the relevant event data, not independently for each transport chunk. JSON payloads and business completion states are application contracts; SSE does not define a universal model-service completion string.

Use a small boundary reference first

This example handles LF/CRLF, data fields, and blank-line dispatch, using strict UTF-8 decoding for diagnostics. It is not a full EventSource client: lone CR, BOM, event types, id, retry, and reconnection are outside its scope. Production code should use a component matching the actual protocol.

# Diagnostic subset: UTF-8, LF/CRLF, data fields, blank-line dispatch.
# Not a complete EventSource client (no lone CR/BOM/id/retry handling).
import codecs
from itertools import chain

def data_events(chunks):
    decoder = codecs.getincrementaldecoder("utf-8")("strict")
    pending, data, events = "", [], []
    for chunk in chain(chunks,[None]):
        pending += decoder.decode(b"" if chunk is None else chunk,
                                  final=chunk is None)
        while "\n" in pending:
            line,pending = pending.split("\n",1)
            if line.endswith("\r"):
                line = line[:-1]
            if line == "":
                if data:
                    events.append("\n".join(data))
                data = []
            elif not line.startswith(":"):
                field,sep,value = line.partition(":")
                if sep and value.startswith(" "):
                    value = value[1:]  # Remove at most one space.
                if field == "data":
                    data.append(value)
    # Incomplete trailing event is intentionally not dispatched.
    return events

wire = ": keepalive\r\ndata: 你好🙂\r\ndata: 第二行\r\n\r\ndata: 完成\n\n".encode()
expected = ["你好🙂\n第二行", "完成"]
for cut in range(1,len(wire)):
    assert data_events([wire[:cut],wire[cut:]]) == expected
assert data_events([bytes([b]) for b in wire]) == expected
assert data_events([b"data: unfinished\n"]) == []
assert data_events([b"data:  leading\n\n"]) == [" leading"]
wrong = "".join(bytes([b]).decode("utf-8",errors="replace") for b in wire)
assert wrong != wire.decode("utf-8")
try:
    data_events([b"data: \xe4"])
except UnicodeDecodeError:
    pass
else:
    raise AssertionError("truncated UTF-8 accepted in strict mode")
print("every two-piece split, one-byte chunks, framing and EOF passed")

Executed checks cover every single cut into two chunks and one-byte chunks, producing the same two events. They also cover multiline data, preservation of an extra leading space, incomplete-event discard, and truncated UTF-8 failure in strict mode. No model service or real conversation was accessed.

EOF is not proof of business completion

The standard discards an incomplete event lacking its final blank line; the example does too. Transport closure, complete framing, and successful business completion remain separate conditions. A closed connection may be expected or interrupted, so EOF alone should not set the interface to completed.

Use the actual service contract to recognize completion or error payloads and retain the last confirmed business stage. Do not store half a structured object as a complete result. Displayed provisional text and final deliverables need an explicit distinction too.

A heartbeat is connection activity, not task progress

SSE comments can keep a connection active; keepalive in the example produces no data event. Treating every received byte as task progress misclassifies a connection that sends only heartbeats. Record transport activity, complete events, and business progress separately.

Timeout policy also has layers: no bytes arriving differs from continuing heartbeats without a business result. An interface can report waiting accurately without promising completion merely because a socket remains open. Thresholds depend on service and product requirements; no universal number is invented here.

The revised reference consumes the chunk iterator incrementally rather than collecting all chunks first. It still retains output events in a list for testing. A long-running client should deliver events as appropriate and use backpressure and limits; this small test is not a constant-memory client.

Test deliberately inconvenient boundaries

English-only tests with large chunks may never expose the defect. Include Chinese, four-byte characters, split CRLF, multiple events per chunk, split field names, and interruption before event termination. Compare chunked results with an unfragmented reference before adding live networking.

Changing chunk size should not change valid-data meaning. Every single cut covers all two-piece splits, and one-byte input covers another extreme; neither proves every arbitrary multi-piece partition. Add randomized partitions, size limits, and other protocol fields as needed rather than presenting finite testing as an absolute guarantee.

Buffers need limits and cancellation

Incremental processing avoids waiting for the complete response, but it does not justify unlimited accumulation of unfinished lines or events. Set bounds based on the service contract. On overflow, follow an explicit failure policy rather than dropping bytes and continuing to claim completeness.

Handle cancellation, disconnection, and service failure separately, releasing resources. Before retrying, consider whether actions could repeat. The site’s idempotent agent retries covers duplication. Correct byte decoding does not replace a retry protocol.

Debugging need not log every character

Chunk lengths, decoder state, event counts, and error positions can assist diagnosis without default collection of all user content. The text and events here are fictional. Capturing real payloads requires appropriate authorization, minimization, and retention decisions.

Narrow failures by layer: reconstruct bytes, assemble lines, frame events, then inspect application JSON and completion state. If these checks help repair streaming corruption or truncation, voluntary support below is welcome.

Sources and backfill note

The Python incremental codec interfaces and WHATWG SSE standard. Diagrams and tests are original teaching content, not a complete SSE or vendor client. Backfilled on October 16 with the planned 14:00 Beijing article date.