A GPU workload can remain Pending even while the cluster has free memory. Aggregate capacity does not guarantee enough space on an individual device, and starting one worker does not establish that an entire distributed job can be placed. Separate entitlement, placement, and group constraints before interpreting a utilization chart.

Run:ai’s open-source release makes orchestration components easier to inspect and combine. It does not remove capacity or topology constraints. A two-device packing example below turns “resources are available” into several independently testable statements. 中文版.

The announcement and the four layers

Google News surfaced the October 15 Konsulteer report. Its underlying event is the October 13 NVIDIA announcement, rather than a new release today. The full Run:ai platform source is available, with four independently adoptable components: KAI Scheduler, GPU Fractioning Operator, KAI Resource Management, and Workload Map.

Their names describe different responsibilities. Scheduling decides placement; organizational resource management expresses team shares; the fractioning operator integrates GPU sharing; Workload Map describes workload structures. The capacity exercise here is an original teaching model, not a reconstruction of Run:ai internals or a benchmark of any physical GPU.

Why adding up free memory can mislead

Consider two fictional devices, each with 10 GiB available to the example workloads. Four tasks require 4, 4, 6, and 6 GiB. Demand totals 20 GiB, exactly matching capacity. A simple first-fit rule places each task on the first device with enough free space: the two 4s fill 8 GiB on A, and the first 6 goes to B. The final 6 has nowhere to go.

A has 2 GiB left and B has 4 GiB left. Their total is the required 6 GiB, but a task with a single-device memory requirement cannot split that requirement across the two gaps. Pairing 4+6 on each device gives a feasible arrangement instead. This compares layouts before launch. Already-running tasks cannot be assumed movable at zero cost: migration or restart adds state-save and recovery costs.

A failed greedy attempt therefore does not prove the batch is impossible, while adequate aggregate capacity does not prove the current layout can accept it. Compute capacity, runtime overhead, driver reservations, and topology are deliberately omitted to isolate memory fragmentation. Measure real peak demands rather than treating these fictional numbers as hardware guidance.

Six assertions distinguish failure from infeasibility

The first function implements first-fit. The second searches placements exhaustively for this tiny example without mutating the caller’s capacity list or allocating a GPU. Exhaustive search is not a production-cluster recommendation. Jobs are positive and capacities nonnegative in the stated example. With n jobs and m candidate devices, this direct search can explore on the order of m to the n branches in the worst case, which is why it is confined to a small construction.

# Fictional memory capacities in GiB; no GPU or scheduler is run.
def first_fit(jobs, capacity):
    free = list(capacity)
    for need in jobs:
        for i in range(len(free)):
            if free[i] >= need:
                free[i] -= need
                break
        else:
            return False, free
    return True, free

def feasible(jobs, capacity):
    # Exhaustive search for this tiny example; not a production scheduler.
    free = list(capacity)
    def place(k):
        if k == len(jobs):
            return True
        for i in range(len(free)):
            if free[i] >= jobs[k]:
                free[i] -= jobs[k]
                if place(k+1):
                    return True
                free[i] += jobs[k]
        return False
    return place(0)

jobs, capacity = [4,4,6,6], [10,10]
ok, left = first_fit(jobs,capacity)
assert not ok and left == [2,4]
assert sum(left) == 6  # enough in total for the remaining 6, but fragmented
assert feasible(jobs,capacity)
assert first_fit([6,6,4,4],capacity)[0]
assert not feasible([6,6,6,4],capacity)
assert capacity == [10,10]  # planning did not mutate external capacity
print("6 placement checks passed")

All six checks passed locally: the original order fails, the failure leaves six units free in aggregate, another placement exists, a changed order succeeds, an over-capacity batch is infeasible, and the external capacity list is unchanged. Both functions model one dimension only. They establish nothing about fairness, performance isolation, or actual throughput.

Why starting half a group may not help

A distributed workload may need multiple workers to participate together. Giving a two-worker job resources for only one can leave that worker waiting while occupying capacity another workload could use. “Some Pods are running” then hides the fact that the job cannot make progress.

The KAI Scheduler project lists batch scheduling, packing, queues, and fairness policies; the official announcement describes gang scheduling for multipod jobs. The important concept is group-level placement under the workload’s constraints. It is not a promise that every Pod begins executing at the same physical instant.

Suppose the four tasks in the example form one group that must be fully placed. The partial first-fit result cannot simply be committed. Validate a whole-group plan on a planning copy, then reserve actual resources with appropriate consistency protection. Concurrent capacity changes require retrying or undoing reservations. The function above provides only a small feasibility calculation, not a distributed reservation transaction.

Quota answers a different question from placement

Quota asks how much a team may consume. Placement asks which concrete devices can carry the workload now. A 20 GiB entitlement does not guarantee two suitable 10 GiB device slots at any arbitrary moment. Existing workloads, node-selection rules, GPU type, and communication topology can all narrow the candidate set.

Do not enlarge quota merely to clear a Pending status. Determine whether the blocker is queue policy, per-device memory, incomplete group capacity, or incompatible node and device constraints. A quota change directly addresses only an entitlement-related blocker. Other cases may call for waiting, repacking, reducing demand, or changing the permissible placement scope.

Fairness can also conflict with immediate throughput. Filling gaps with small jobs may improve utilization while repeatedly delaying a larger group. Keep waiting-time distributions, completed-job counts, preemptions, and restart costs alongside utilization. An average utilization percentage cannot show which team or workload keeps being postponed.

A GPU fraction is not a throughput fraction

Memory capacity does not translate directly into service performance. Memory limits, compute time, bandwidth, model loading, and request lengths affect different parts of execution. Half a device does not guarantee half its throughput or proportional latency. Verify the sharing mechanism’s isolation and supported hardware/software combinations against its component documentation.

For online inference, examine time to first token, subsequent token intervals, tail latency, concurrency, and errors. For offline training, also retain useful training progress and restart loss. Compare exclusive and shared execution using the same workload and observation window. No GPU experiment was run for this article, so it reports no percentage improvement.

The earlier KV cache memory article covers a request-dependent portion of inference memory. Including such peaks in resource estimates is more useful than requesting a fraction based only on model weights. Packing cannot compensate for an understated requirement; a compact placement of inaccurate requests can still fail at runtime.

Test a narrow capability before adopting a platform

Open source allows inspection of interfaces, constraints, and implementation, with modular adoption where appropriate. It does not supply production operating expertise automatically. NVIDIA states that the Community Product requires no paid Run:ai subscription but excludes enterprise support; source and distributed products have applicable terms. Evaluate component versions, support boundaries, and upgrade paths before relying on a community deployment.

Begin in a non-production environment with workloads whose peaks are understood and retain the existing scheduler as a baseline. Deliberately create aggregate shortage, fragmented capacity, incomplete groups, and quota exhaustion. Check whether scheduling events and waiting reasons explain each outcome. Establish which capability addresses which delay before considering a platform-wide replacement.

If an existing system lacks only GPU-aware placement, a full control plane may be unnecessary. If it lacks organizational quota semantics, tighter packing alone will not fill that gap. Selecting components from a concrete operational problem keeps failure attribution and rollback understandable.

Sources and backfill note

Discovery followed the Google News listing of the October 15 report, then the October 13 original announcement. Relevant source repositories are KAI Scheduler and the Run:ai platform. Both diagrams, the capacity construction, and code are original. Backfilled on October 16, retaining the planned 11:30 Beijing article date; the news sources predate that slot.

If the packing example and diagnostic order help distinguish a quota wait from a placement wait, voluntary support through the section below is welcome.