High availability
High availability has a separate answer for each thing that can fail, and in Flui those answers differ. This page takes them one at a time: what happens, what Flui does about it and where the current limits are.
What can fail, and what happens
| What fails | Applications on that cluster | Your data |
|---|---|---|
| A worker | Those with two or more copies keep answering from the other nodes | Unaffected, unless a database or a volume tied to that node lived on it |
| The cluster’s master | Keep answering from the other nodes; nothing new can be deployed until it returns | Shared storage waits for the master; nothing is lost |
| The node a database lives on | The database is down until it is restored | At most the last few minutes of changes |
| The control cluster | Keep answering | Unaffected; the dashboard, the API and central logs are gone until it returns |
A worker
A cluster with at least two nodes keeps serving when a worker goes away. Flui spreads an application’s copies across nodes wherever there is room, and the application’s address points at every ready node that takes traffic. An application with two or more copies keeps answering from the ones that are left.
For about the first minute, until the cluster marks the worker as gone, some requests still go to the copy that was on it and fail. After that, the remaining copies answer every request. The copies that were on the lost worker start again elsewhere once the cluster stops waiting for it.
Once Flui has checked that every node of the cluster sees the shared storage, new volumes are not tied to the node that created them, so an application that keeps its files there starts again elsewhere with its data. Until every node sees it, new volumes stay on their node, and an application with one of them waits for that node to come back.
flui app status tells you, for each application, whether it
survives losing a worker. That needs:
- at least two copies, running on at least two nodes
- no dedicated placement, which keeps it on one node
- no volume tied to one node
- a real domain
- more than one node taking traffic
When an application falls short, the command lists each reason
with what to change. Agents get the same answer from the
app_availability tool.
The cluster’s master
Every cluster has one master. It does not sit between your users and your applications: an application’s domain points at every ready node of its cluster that takes traffic. Flui measures that set every minute and moves the name when it changes, with a one-minute lifetime on a name that points at several nodes. Clients that looked the address up before a node dropped out can take a minute or two to move on.
What still depends on the master is the cluster’s ability to change. While it is down, nothing new is deployed, lost copies are not started again, and shared storage, which is served through the master, waits for it. The data on it is kept: where the provider sells volumes, it lives on one, separate from the machine’s own disk.
The automatic address an application gets when it has no domain of its own encodes the master’s IP, so it cannot point at more than one node. An application that needs to stay up needs a real domain.
Flui runs one master per cluster. Why K3s explains how more could be added without changing how a cluster stores its state.
The node a database lives on
Databases from the catalog run as a single instance on the disk of the node they are placed on, because a local disk is the fastest thing a database can write to. This also makes that node a single point of failure for the database. If the node is lost, the database is down until it is restored on another one.
PostgreSQL and MariaDB ship every change off the cluster as it happens, so a restore loses at most the last few minutes of changes. How Flui protects data explains how this works and how to check it is on before you need it.
In progress: databases that survive the loss of a node
The design is settled and being built, PostgreSQL first.
A database marked for high availability runs as two copies on two different nodes, each on its own local disk, so it keeps the fast storage. The copies are kept in step synchronously: a change is confirmed only once both have it, so the copy that takes over has everything.
A vote decides which copy takes over. Each of the two nodes has one vote, and the control cluster has the third. A copy becomes the primary only with two votes out of three, so a node cut off from the others can never become a second primary. Losing any one of the three, the control cluster included, still leaves a majority to decide. The smallest setup that covers this is a control cluster plus a workload cluster of two machines.
MariaDB and Redis follow, on the same pattern.
The control cluster
The control cluster runs Flui itself. Losing it takes away the dashboard, the API, sign-in and the central logs and metrics. Every workload cluster keeps serving its applications, because a cluster needs the control plane only to make changes. See Failure boundaries and, for the worst case, Rebuild the control plane.
A cluster across several locations
Where a provider network spans several locations, a cluster’s nodes can sit in more than one of them, as long as its scaling group is allowed to buy there. Every node accepts traffic and copies are spread across nodes, so losing a location takes only what ran there.
Making an application highly available
- Run at least two copies. Set
deploy.scaling.min: 2in the application’sflui.yaml. - Give the cluster at least two nodes, so the copies have somewhere else to be.
- Give it a real domain, in a zone whose DNS Flui can drive (see which providers), so its address can point at every node.
- Protect the cluster, so its databases are backed up continuously and any loss is counted in minutes.
- Keep the control cluster separate from the clusters your applications run on, so a problem with one does not reach the other.
What is not here yet
- Databases that survive the loss of their node (in progress, see above).
- Shared storage that does not depend on the master.