Rebuild the control plane

Use this procedure when the machine running Flui is gone, and what you have left is a platform backup in object storage and the key you generated when you set it up, either in your vault or in its recovery copy.

The rebuild is a series of manual steps rather than one command. It writes an installation’s identity onto a new machine, and you should understand each step before it runs.

Before you start

You need three things. If any is missing, stop here, because the rest will not work.

The key the backup was sealed to: either the vault on the machine where you ran flui backup platform init, unlocked, or the recovery copy it wrote (by default ~/flui-<profile>-platform-recovery.age) and your vault passphrase, which seals it. Flui does not hold a copy. Without one of them the backup can never be read.
The two objects from your backup destination, from the same run:
- platform/<env-id>/<timestamp>/keys/keybundle.age
- platform/<env-id>/<timestamp>/db/flui-pg.dump.gz.enc
They are a pair. A bundle from one run cannot open a dump from another.
A machine to rebuild onto, and the CLI. This can be a completely different workstation from the one that built the original.

1. Open the backup

# on the machine whose vault holds the key
flui backup platform restore \
  --bundle ./keybundle.age \
  --dump ./flui-pg.dump.gz.enc \
  --out ./flui-rebuild

# anywhere else, from the recovery copy
flui backup platform restore \
  --bundle ./keybundle.age \
  --dump ./flui-pg.dump.gz.enc \
  --identity ./flui-<profile>-platform-recovery.age \
  --out ./flui-rebuild

This runs entirely on your machine. It takes the key from your vault (or asks for your vault passphrase to open the recovery copy), uses it to open the sealed bundle, and uses the per-run key inside the bundle to decrypt the dump. Keys replaced with --rotate are tried too, so an older backup still opens.

You get a directory containing:

File	What it is
`flui-control-plane.sql`	the control-plane database, in the clear
`install-keys.env`	the keys the new installation must boot with
`retire-old-control-row.sql`	retires the dead master’s cluster row
`cluster-secrets.json`	secrets that live only in the cluster (present when the backup captured them)
`ssh-ca`, `ssh-ca.pub`	the SSH certificate authority the nodes trust
`secure-keys/`	when the backup carries them, the files of the API’s key directory

Everything in it is plaintext key material. Delete the directory when you are done.

2. Create the new installation with the original key

This is the step people get wrong, and the failure is quiet: the database restores, the API starts, and every provider token, cluster credential and application secret is unreadable, because they were sealed with a key the new installation does not have.

export $(grep FLUI_RESTORE_ENCRYPTION_KEY ./flui-rebuild/install-keys.env)
flui env create --host <new-server-ip>

flui env create picks the key up from FLUI_RESTORE_ENCRYPTION_KEY instead of generating a fresh one. Rebuilding from the original workstation also works without this, because the key is sealed in that workstation’s vault, but do not rely on that.

3. Load the database

The two .sql files were written on your machine, so copy them to the new master first. Then open a shell there and load the dump into its Postgres:

flui ssh master
kubectl exec -i -n flui-system postgres-0 -- \
  psql -U fluicloud -d postgres < flui-control-plane.sql

The dump is taken with --no-role-passwords. The new installation generated its own database passwords, and a dump carrying the old hashes would overwrite them and lock the API out of the database it just restored.

Then retire the old master’s cluster row:

kubectl exec -i -n flui-system postgres-0 -- \
  psql -U fluicloud -d fluicloud < retire-old-control-row.sql

The dump carries the control-cluster row of the installation it came from, which names a machine that no longer exists. The new installation seeded its own row when it booted. This script soft-deletes the old one so nothing follows a dead address.

4. Put back the secrets that live only in the cluster

Two pieces of key material are created inside the cluster and are not in the database:

The Zitadel masterkey. Without it the restored identity database cannot be decrypted. It is in cluster-secrets.json under zitadel-secrets.
SSH_KEY_ENCRYPTION_KEY. The installer creates a new one on every install, and the restored rows were sealed with the old one. It is in install-keys.env.

Apply both to the new cluster, then restart the API so it picks them up:

kubectl -n flui-system patch secret flui-secrets \
  --type merge -p "{\"stringData\":{\"SSH_KEY_ENCRYPTION_KEY\":\"<from install-keys.env>\"}}"

kubectl -n flui-system patch secret zitadel-secrets \
  --type merge -p "{\"stringData\":{\"masterkey\":\"<from cluster-secrets.json>\"}}"

kubectl -n flui-system rollout restart deploy/flui-api
kubectl -n flui-system rollout restart deploy/zitadel

If the directory has a secure-keys/ folder, put its files into the API’s key directory, /secure/keys, keeping their permissions.

5. Check that it is actually back

Restored rows do not prove that the installation works. Run these three checks; each one fails for a different reason:

flui app list               # the key is right: cluster credentials decrypt and clusters answer
flui ssh <cluster>/<node>   # the SSH CA was restored and nodes still trust it
flui app redeploy <app>     # the installation can act, not only read

Then log in to the dashboard as the original administrator. If your installation uses OIDC and that login fails, the masterkey did not make it; see step 4.

6. Bring back the applications that ran beside Flui

On an installation with a single cluster, your applications ran on the same cluster as Flui and were lost with it. The restored records still describe them, and their data is in their own backups. This step puts the two together on the new control cluster.

Look at the plan first. It reads everything and changes nothing:

flui cluster rebuild-control --plan

It finds the earlier control cluster (the one retire-old-control-row.sql retired in step 3) and lists each application it held, what will be restored into it, and in which order: databases first, then what depends on them. Flui’s own components are not in the list, because the new installation runs its own.

Read the warnings. The most important one is about DNS: the new control cluster does not publish the zones the old one did. Assign each zone it names before going on, or the applications come back with their names and no zone to publish them:

flui dns zone assign <zone> --cluster <new-control-cluster>

Then run it:

flui cluster rebuild-control

It asks for confirmation and follows the work, one application at a time. Each application is re-created from the records, its databases are loaded from their continuous backup or their last dump, and its volumes from their most recent snapshot.

--include-stopped also brings back applications that were not running when the old control cluster was lost.
--from <cluster> names the earlier control cluster, when more than one still has applications recorded on it.
--yes skips the confirmation; --no-wait returns once the work is queued.

If it is interrupted, running it again continues where it stopped. An application whose data was never backed up comes back empty, and the plan says so before you start.

What this does not bring back

Data that was never backed up. This procedure restores the platform’s own records: users, tokens, clusters, application definitions. The data inside your databases and volumes comes from their own backups, in step 6. See How Flui protects data.
Workload clusters that were also lost. If a workload cluster died with the master, the restored control plane will know it existed and find it gone. Bring its applications back with flui cluster rebuild.
Sessions. Everyone logs in again.