Clustering

Overview

A Flo cluster is a group of nodes holding the same log. One member leads at a time. Every write goes through the leader, reaches a majority of members before it is acknowledged, and is then applied on every member. Clients connect to any node: a write sent to a node that does not lead is carried to the leader over the peer link and answered on the node the client wrote to, once that node has applied it, so a client always reads its own writes. Reads are served locally by every node.

Membership is recorded in the log itself. A configuration entry names the members, every member applies the same entry, and a restarted member learns its group from its own data directory.

:::note Two limits of the current release. A cluster replicates one shard: leave shards at its default (it resolves to one on a member) or set it to 1; more is refused at start, and so is a data directory that was written with more than one shard — a member starts from an empty one. And there is no remove yet: a member that leaves for good keeps counting toward the majority until it is removed — see issue #95. :::

Starting a Cluster

Every member proves the same shared secret at the peer port. Put it in flo.toml or in the FLO_CLUSTER_SECRET environment variable; generate one with openssl rand -base64 32. A node refuses to bind the peer port without one.

[cluster]
secret = "the same value on every node"

The first member

flo server start --cluster

It leads a group of one until others join. [cluster] enabled = true in the config file means the same as --cluster.

Joining

flo server start --join 10.0.1.10:9500

A seed is any member's peer port (listen_port + 500), as host:port; the host may be an IPv4 address or a DNS name (a Compose service name works). Several seeds are comma-separated. In the config file the same thing is seeds = ["10.0.1.10:9500"] with enabled left off.

The joiner starts with no vote and asks the members it can reach, once a second, to be added. The leader writes a configuration entry naming it, one change at a time, and from then on it is a member. flo cluster status shows joining until that happens; a node still asking after thirty seconds logs what to check (the seed addresses, the secret, and that the group has room — a group holds at most 8 members).

Restarting a member

Start it the same way it was started before, with --cluster or --join. Its membership comes from its log, not from the flags: a member restarted with --join needs no seed to be up to know its group, and a first member restarted with --cluster follows whatever configuration its log holds.

A data directory that belonged to a group refuses to run without either flag: started plainly it would lead alone and take writes the group never sees.

What is refused at start

Ports

The peer port is bound only by a node started with --cluster, --join, or seeds; a single node leaves it unbound, so there is no undeclared port to account for. It carries membership and log contents and must not be reachable from outside the cluster's private network; the secret authenticates peers, it does not encrypt the link.

Node ids are derived from hostname:port unless --node-id is given, and are stored in the data directory on first boot; to change one, start from an empty data directory.

Writes and Reads

client → any member
  ├─ the member leads: append to its log, replicate, a majority acks,
  │  apply, answer
  └─ it does not: carry the request to the leader over the peer link,
     hold the client until this member has applied the result, answer

A client never needs to know which node leads. What it can see:

The two "may still apply" answers are honest: the entry is in a log and a later leader holding it will commit it. A KV write can be sent again freely. A stream append or a queue push is not idempotent — read back before re-sending.

Reads are local. On the node a client wrote to, the write is visible when the response arrives; on other members within about a heartbeat.

Failure Handling

Failover

One setting governs timing:

[cluster]
failover_timeout_ms = 1500   # default; minimum 100

A leader unheard for between half and all of it is replaced; heartbeats go out at a sixth of it. A leader that cannot reach a majority for that long steps down and answers every write it was holding. Elections are a poll first, so a member that was cut off does not depose a live leader when it returns. Typical failover is one to three seconds at the default.

The default is deliberately generous: a busy disk's fsync must not look like a dead leader. Lower it only on fast, quiet disks.

Durability and the diverged state

With durability = "sync" an acknowledgement means the write is on disk on a majority. A leader whose log contradicts history a member has committed is a bug, and the member refuses it.

With async_flush (the default) an acknowledgement comes from memory and is flushed within a second. A majority that crashes inside that window can lose acknowledged writes, and the survivors' logs follow the new leader.

Either way, a member whose applied state no longer matches the group marks itself diverged: it stops taking part (writes are refused with unavailable: this node's data diverged from the group and it takes no writes; use another node, reads still serve what it has), and flo cluster status and its log both say what to do: stop it, delete its data directory, and start it again with --join naming a live member — not --cluster.

A member that lost its data

Start it with --join, never with --cluster: with --cluster an empty data directory leads a group of one and takes writes until the old group speaks, then ends up diverged with those writes lost.

Monitoring

Cluster status

flo cluster status

Cluster Status
──────────────
Node ID:    flo-24e3b2
Address:    127.0.0.1:9000
Role:       leader
Leader:     flo-24e3b2
Term:       3
Members:    3

--output json gives the same as one object, with "leader_id": null while none is known. A single node reports itself as the leader of a one-member group.

flo cluster members and flo cluster transfer-leader are accepted by the CLI but not yet implemented by the server (issue #95).

Prometheus metrics

The node serves the Prometheus exposition format on the metrics port (listen_port + 1 by default, so 9001 for a node on 9000):

curl localhost:9001/metrics

GET /health on the same port returns a small JSON liveness document with the shard count and uptime. It does not yet carry the node's role, so a load balancer cannot use it to route around a diverged member (issue #96).

:::note The metrics port binds to 127.0.0.1 by default. /metrics and /health are unauthenticated — unlike the dashboard, which is API-key gated — so the endpoint is loopback-only unless you opt in. To scrape from another host set:

[metrics]
bind = "0.0.0.0"

Prefer a node-local collector or a sidecar over exposing the port directly. :::

Metric families include server counters (flo_commands_total, flo_bytes_received_total, flo_connections_current, flo_uptime_seconds), per-shard gauges (flo_shard_*), per-stream / per-queue / per-KV-namespace counters, workflow and processing aggregates, and the peer link counters:

The same counters are available as JSON from the dashboard at GET /api/v1/metrics, under replication. A peers_linked that stays below the member count minus one names a link that is down.

:::note There are no flo_raft_* families yet (role, term, commit index, per-member lag); flo cluster status on each node is the signal until they land (issue #96). A counter such as flo_tiered_log_cold_hits_total is always 0: the read path consults the hot ring and the warm store only. :::

Configuration

[cluster]
enabled = false              # true starts the first member (same as --cluster)
# secret = "..."             # required whenever the peer port is bound
# node_id = 1                # default: derived from hostname:port, then stored
# raft_port = 0              # 0 = listen_port + 500
# seeds = ["10.0.1.10:9500"] # peer ports of members to join
# failover_timeout_ms = 1500 # minimum 100

These are the only keys [cluster] reads; any other key is refused at start, not ignored, and the line names the ones that exist. There are no per-timer settings (everything derives from failover_timeout_ms), no replication factor (every member holds a full copy, so it is the cluster size), and no gossip (members talk over the peer port).

Recommended Topologies

Development. One node; nothing to configure.

Production. Three members (tolerates one failure) or five (tolerates two), the peer port reachable only inside the private network, and durability = "sync" where an acknowledged write must survive a majority crash.

[storage]
durability = "sync"
 
[cluster]
secret = "the same value on every node"
seeds = ["10.0.1.10:9500"]