Scheduler

PandaStack has a tiny global scheduler — a handful of SQL queries plus a deterministic scoring function — that picks the right agent host for every create. It's not Kubernetes; it doesn't need to be. The whole thing fits in one file.

The decision

score(agent) =
    0.6 × free_cpu(agent)
  + 0.3 × free_mem_gb(agent)
  + 5.0 × (stream_restore_enabled ? 1 : 0)

Whichever agent scores highest wins the create. Agents that fail the resource floor (the request doesn't fit in free capacity) are excluded before scoring — with one asymmetry: memory is a hard gate, while vCPUs are burstable capacity arbitrated at runtime by cgroup weights (CPU admission is off by default; every template bakes 8 burstable vCPUs).

What "fits" means for memory depends on the agent's admission mode, which it reports on every heartbeat:

• working-set — the agent measures each sandbox's resident memory (RSS + hugetlb) every 15 s and advertises memory_mb_admittable: its budget minus measured residency, minus reserves promised to creates that have not been measured yet, minus the headroom the host's pressure ladder keeps free. A create is charged max(512 MB, memory_mb × 0.25) at placement — not its baked size — because a 4 GiB sandbox that is merely alive holds a few hundred MB. Managed databases are the exception: they are the guaranteed class and are always charged their full size.
• committed — the sum of every live sandbox's baked memory_mb must fit. This is the pre-T2.1 behaviour and the code default; it caps a 32 GiB host at roughly seven 4 GiB sandboxes regardless of how idle they are.

The agent computes the admittable figure with the same arithmetic its own host-side gate refuses on, so the scheduler and the host can never disagree about how full a host is.

Every create takes the same snapshot-restore fast path on any agent that has the template's seed, so placement is purely a load-spreading decision: prefer the host with the most free CPU (memory as a secondary term). The +5.0 streaming bonus is a tiebreaker — an agent with UFFD streaming restore boots a template it has never seen without first downloading the whole vm.mem, so among otherwise-equal hosts it wins.

Inputs (all from Postgres)

An in-memory cache sits in front of the agents query. At 1000 RPS we don't want to slam Postgres for the same answer.

Leases (sandbox ownership)

Once a sandbox is created, a row in the leases table records which agent owns it. Subsequent requests for that sandbox bypass scoring entirely: the edge looks up the lease (in-memory cache first, then Postgres) and proxies straight to the owning agent. Expired leases are swept by the agent's lease sweeper, which also flips orphaned sandbox rows to failed.

This is the mechanism for "an agent went away (host died, network partition, OOM kill)" — stale agents drop out of the candidate list in ≤30 s, and their dead sandboxes are reconciled by the sweeper.

NATID partitioning

NATID slots are a per-agent resource (16,384 per agent, see networking). The scheduler doesn't move sandboxes between agents to balance NATID — the binding constraint in practice is host memory/CPU, which the score already spreads.

Zombie reaping

A reconciliation loop in each agent compares its DB rows to its live Firecracker processes (see snapshot & restore: recovery), and a startup sweep removes leases pointing at sandboxes the agent no longer has. The scheduler trusts these — it never tries to second-guess what's actually running on a host.

Affinity

Two affinities exist:

1. Volume affinity. If create references a volume by name, the scheduler restricts candidates to agents that already have that volume file on disk. (Volumes are host-local in v1.)
2. Fork affinity. POST /fork defaults to placing the child on the parent's agent (memory + rootfs are already there). Pass ?cross_host=1 to opt into the cross-host path; the scheduler picks the best other agent.

No labels, no taints, no node selectors. Either the volume is here, or it isn't.

Walked example

A code-interpreter create (8 vCPU / 2 GiB) lands on the API gateway. The scheduler lists fresh agents (free_cpu is against the 4× burst envelope):

Scores:

• pz20: 0.6×10 + 0.3×6 + 5 = 6.0 + 1.8 + 5 = 12.8
• n1v2: 0.6×40 + 0.3×24 + 5 = 24.0 + 7.2 + 5 = 36.2

Pick n1v2. POST /v1/sandboxes is forwarded to its admin endpoint. Done — ~1 ms of scheduling, then the snapshot restore runs inside the agent.

What can go wrong

Why so simple

Two reasons:

1. The hard part isn't picking the right host — it's making every create fast. When every agent restores the same seed through the same fast path, scheduling is a 1 ms load-spreading preference, not a contract.
2. State is the bottleneck, not compute. A million-line scheduler with thousands of node features doesn't help when the actual constraints are "does the request fit?" and "is the seed here?".

Files

• api/internal/scheduler/scheduler.go — score function + candidate query + lease cache.
• api/cmd/api/multinode.go — retry logic, agent dispatch.
• infra/terraform/modules/gcp-agent-mig/main.tf — google_compute_region_autoscaler.agent (the layer above the scheduler — adds/removes hosts).

Known limits

• Single global scheduler shard (it's a stateless function over a Postgres view; sharding is trivial when needed but not yet useful).
• 10 s heartbeat means the capacity view is up to 10 s stale. Acceptable: a mis-pick costs nothing because every host restores at the same speed.
• No bin-packing of large (cpu, mem) requests. Edge cases visible only at >70 % cluster fullness; autoscaler keeps us well under.