PandaStack

Database operations: clone, restore, failover, and credentials

Run day-two operations on a managed PostgreSQL database — clone it, restore it to a point in time, resize it, rotate its credentials, wake it, and fail it over to a healthy host.

This guide covers the operational endpoints on a managed PostgreSQL database: cloning and point-in-time restore, resizing, failover, credential rotation, and waking a suspended database. Every operation here works against the same database id you got from POST /v1/databases.

All of these run against the continuously archived backup stream (a base backup plus archived WAL in object storage). The first base backup lands roughly two minutes after a database first reports running — until then, clone and failover both refuse with 412.

Before you start

shell
export PANDASTACK_API_KEY=pds_...
export DB_ID=3f95041e-0ba0-4068-8302-512e811edd60

Two things to know before you send anything:

  • Send a body on every POST. The edge rejects a bodyless POST with 411 Length Required before it reaches the API. Send -d '{}' and Content-Type: application/json even when the endpoint takes no parameters.
  • Clone, failover, and credential reset need a multi-node deployment. On a single-node self-hosted install they return 501.

Read the current state first — GET /v1/databases/{id} returns status, size, always_on, cloned_from, and (while running) the live connection_url, password, and broker_token.

Clone a database

A clone is provisioned into a brand-new database id from the source's archive. The source is never touched, so cloning is safe against a database that is running, suspended, or failed. The clone gets its own host, its own credentials, and its own backup stream.

1. Send the clone request

shell
curl -X POST https://api.pandastack.ai/v1/databases/$DB_ID/clone \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"label":"staging copy"}'

The API answers 202 immediately with the new database id:

{
  "id": "9c1d7f22-77a1-4f0c-9f0e-2a3b6d5e8c10",
  "status": "provisioning",
  "template": "postgres-16",
  "size": "1g",
  "label": "staging copy",
  "cloned_from": "3f95041e-0ba0-4068-8302-512e811edd60",
  "host": "9c1d7f22-77a1-4f0c-9f0e-2a3b6d5e8c10.db.pandastack.ai",
  "port": 5432,
  "broker_url": "https://api.pandastack.ai/v1/databases/9c1d7f22-.../proxy"
}

Omit label and the clone inherits "<source label> (clone)".

2. Poll the new id until it is running

The clone downloads the base backup and replays WAL before PostgreSQL accepts connections — minutes for a large database.

shell
CLONE_ID=9c1d7f22-77a1-4f0c-9f0e-2a3b6d5e8c10
curl -s https://api.pandastack.ai/v1/databases/$CLONE_ID \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" | jq '.status, .connection_url'

Poll until status is running and connection_url is present. A running status with no credentials means PostgreSQL is still coming up; the API reports that back as provisioning with an error hint.

SDKs

The SDKs wait for you by default and return the ready clone.

clone.py
import pandastack

client = pandastack.Client(api_key="pds_...")

clone = client.databases.clone(db_id, label="staging copy")
print(clone["id"], clone["connection_url"])   # a NEW database

# Return the 202 body immediately instead of waiting:
pending = client.databases.clone(db_id, wait=False)
clone.ts
import { Client } from "@pandastack/sdk";

const client = new Client({ apiKey: "pds_..." });

const clone = await client.databases.clone(dbId, { label: "staging copy" });
console.log(clone.id, clone.connection_url);

// Don't wait for WAL replay:
const pending = await client.databases.clone(dbId, {}, { wait: false });
shell
pandastack db clone $DB_ID --label "staging copy"
pandastack db clone $DB_ID --no-wait

Deleting the source while one of its clones is still provisioning is refused with 409 — the clone is replaying that archive segment by segment, and the delete purges it. Wait until the clone reports running, then delete.

Restore to a point in time

Add target_time (RFC3339, UTC) to stop WAL replay at that instant. This is how you undo a destructive change: clone to a moment just before it, then point your application at the clone or copy the rows back.

shell
curl -X POST https://api.pandastack.ai/v1/databases/$DB_ID/clone \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"label":"before the bad migration","target_time":"2026-08-06T09:30:00Z"}'
pitr.py
clone = client.databases.clone(
    db_id,
    label="before the bad migration",
    target_time="2026-08-06T09:30:00Z",
)
shell
pandastack db clone $DB_ID --target-time 2026-08-06T09:30:00Z

Rules the API enforces on target_time:

RuleResponse when violated
Must parse as RFC3339400 invalid target_time (want RFC3339, …)
Must be at least 2 minutes in the past — WAL reaches the archive up to ~60s behind live writes400 target_time must be at least 2 minutes in the past …
Must not be later than the source's newest archived WAL segment400 target_time is beyond the last archived WAL (<timestamp>) …

The last rule bites on idle databases: no writes means no new WAL, so the newest archived segment can be hours old. Omit target_time to clone the latest archived state instead.

Resize a database

RAM is baked into the database's microVM snapshot and cannot be changed in place. Cloning into a different tier is the supported resize path:

shell
curl -X POST https://api.pandastack.ai/v1/databases/$DB_ID/clone \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"size":"4g","label":"prod (4g)"}'
resize.py
bigger = client.databases.clone(db_id, size="4g", label="prod (4g)")

size accepts 1g, 4g, or 16g; anything else is a 400. Omit it and the clone keeps the source's tier. Once the clone is running, repoint your application at the new connection_url and delete the old database when you are done with it.

You can combine size with target_time — a resize and a point-in-time restore in one call.

Rotate credentials

POST /v1/databases/{id}/reset-credentials rotates both the PostgreSQL password and the per-database broker_token. It is synchronous: a 200 carries the new, verified credentials.

shell
curl -X POST https://api.pandastack.ai/v1/databases/$DB_ID/reset-credentials \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  -H "Content-Type: application/json" -d '{}'
rotate.py
fresh = client.databases.reset_credentials(db_id)
print(fresh["connection_url"], fresh["broker_token"])
rotate.ts
const fresh = await client.databases.resetCredentials(dbId);
shell
pandastack db reset-credentials $DB_ID

What to expect:

  • The database must be running. Any other status returns 409 with the current status in the message.
  • Clients holding the old password are cut off. Update your secret store and redeploy anything that connects before you rotate, or accept a short window of connection failures.
  • Two overlapping rotations are refused with 409 — one runs at a time per database.
  • An error means the rotation did not complete, so retrying is safe.
  • If the response comes back without connection_url (a rare read-back race), the values are still rotated — fetch them with GET /v1/databases/{id}. The SDKs do that follow-up read for you.

Wake a suspended database

An idle database suspends its compute and wakes on the next connection, so you rarely need this. Use it when you want the database warm before traffic arrives, or when a bare client won't survive the resume.

shell
curl -X POST https://api.pandastack.ai/v1/databases/$DB_ID/wake \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  -H "Content-Type: application/json" -d '{}'
{
  "id": "3f95041e-0ba0-4068-8302-512e811edd60",
  "status": "waking",
  "detail": "database is waking; poll GET /v1/databases/{id} until running"
}
wake.py
client.databases.wake(db_id)
ready = client.databases.wait_until_ready(db_id)
shell
pandastack db wake $DB_ID

curl -X POST with no -d is the most common failure here: the edge returns 411 Length Required with no JSON body before the API sees the request. Always send -d '{}'.

Opt out of auto-suspend

Set always_on to keep a database running continuously — at creation, or later with PATCH /v1/databases/{id}:

shell
curl -X PATCH https://api.pandastack.ai/v1/databases/$DB_ID \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"always_on": true}'
always_on.py
client.databases.create(label="prod", always_on=True)
client.databases.update(db_id, always_on=True)      # or always_on=False to re-enable suspend
shell
pandastack db update $DB_ID --always-on
pandastack db update $DB_ID --no-always-on

always_on is the only field PATCH accepts; a body without it returns 400. The response is the refreshed database, so you can confirm the flag took effect. Storage is billed either way — always_on only changes whether compute keeps running while nothing is connected. See billing.

Fail over to a healthy host

POST /v1/databases/{id}/failover rebuilds the database on a different host from its archive, under the same database id and the same hostname. Use it when the host of a database has died and GET reports status: "failed".

1. Check that failover is possible

While a database is failed, GET /v1/databases/{id} populates three extra fields:

{
  "status": "failed",
  "failover_available": true,
  "failover_reason": "database can be restored on one of 3 healthy agents",
  "failover_eta_seconds": 180
}

If failover_available is false, failover_reason says why — no healthy host, no archive yet, or a single-node deployment.

2. Start the failover

shell
curl -X POST https://api.pandastack.ai/v1/databases/$DB_ID/failover \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  -H "Content-Type: application/json" -d '{}'

The endpoint is preflight-then-act: it refuses up front if the database is not eligible, if there is no healthy host to move to, or if there is no restorable archive — and in every one of those cases the existing database is left untouched. Only once all three checks pass does it return 202 and run the drain-and-restore in the background:

{
  "id": "3f95041e-0ba0-4068-8302-512e811edd60",
  "status": "restoring",
  "error": "failover started (target agent …); poll GET /v1/databases/{id} — the database may briefly report not-found while the restore provisions"
}

3. Poll until it is running again

Expect roughly three minutes (failover_eta_seconds) for the base-backup download plus WAL replay. Credentials are republished during the restore, so re-read GET /v1/databases/{id} afterwards and use the connection_url from that response — the hostname is unchanged, the password is not.

Planned migration with force

Without force, failover only applies to a failed database. Everything else is refused, deliberately:

Current statusResponse
failedProceeds
running409 — "failover is for failed databases"; pass force to migrate anyway
hibernated409 — use wake instead
anything else409 — refuses rather than guessing

To move a healthy database to another host on purpose:

shell
curl -X POST https://api.pandastack.ai/v1/databases/$DB_ID/failover \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"force":true}'
failover.py
client.databases.failover(db_id)

A forced failover is not free: the running database is stopped and rebuilt from its archive, which trails live writes by up to about 60 seconds. Writes committed in that final window are lost. Do it during a maintenance window, or clone instead if you only need a copy.

A second POST while a failover is in flight returns 409.

What can go wrong

SymptomCauseFix
411 Length Required, no JSON bodyBodyless POST rejected at the edgeSend -d '{}' and Content-Type: application/json
412 on clone or failover: "no restorable archive"The first base backup has not landed yet, or archiving is not configuredWait ~2 minutes after the database first reports running, then retry
400 "target_time must be at least 2 minutes in the past"The archive trails live writes by up to ~60sPick an older instant, or omit target_time
400 "target_time is beyond the last archived WAL"The source has had no writes since that timestampUse the reported timestamp or earlier, or omit target_time
409 on delete: "a clone of this database is still provisioning"The clone is replaying the source's archiveWait for the clone to report running, then delete
409 on reset-credentialsThe database is not running, or a rotation is already in flightWake it and poll to running, then retry
409 on failoverThe database is running or hibernatedWake it, or pass {"force":true} for a planned migration
503 "no healthy agent available"No host currently has capacity or a fresh heartbeatRetry shortly; capacity is bounded by fleet memory
501 on clone / failover / reset-credentialsSingle-node deploymentThese operations need a multi-node deployment
GET briefly returns not-found mid-failoverThe row is recreated during the restoreKeep polling; it reappears as provisioning

More symptoms — connection refused, missing connection_url, TLS errors — are covered in Troubleshooting.

On this page