Storage Internals

Philosophy: The Log Is the Database

Most databases maintain two copies of every write: a write-ahead log (WAL) for durability and a primary data structure (B-tree, LSM tree, heap file) for queries. Flo doesn't do this. The log is the primary data structure.

The Unified Append Log captures every mutation — KV puts, queue enqueues, stream appends, time-series writes — as a sequenced, typed entry. The Raft consensus log and the storage log are the same thing. There is no separate WAL.

Everything else — the KV hash table, the queue's priority heap, the time-series columnar buffers — is a projection: a derived view rebuilt deterministically from the log. If you deleted every projection and replayed the log from the beginning, you'd get back to exactly the same state.

Unified Append Log (UAL)

The UAL is a sequence of entries, each identified by a monotonically increasing 64-bit index.

Entry Format

Every UAL entry has a 40-byte header followed by a variable-length payload:

┌──────────┬──────────┬───────┬───────────┬────────────┬─────────────┬───────┐
│ CRC32C   │ entry_   │ flags │ raft_term │ raft_index │ timestamp   │ pay-  │
│ (4B)     │ type(2B) │ (2B)  │ (8B)      │ (8B)       │ _ns (8B)    │ load  │
└──────────┴──────────┴───────┴───────────┴────────────┴─────────────┴───────┘

The CRC covers the header and payload together. Payload layout varies by entry type — each projection knows how to parse its own payloads.

Three-Tier Storage

When entries age out of the hot ring, they're flushed to warm segments. A warm store hash map keeps payload copies in memory (default 32 MB) for the stream read path, which fetches record payloads by index, so those reads don't have to go to disk immediately. KV entries are never copied there: the KV projection holds the value, and nothing reads a KV payload from the log.

Segment Format

Each sealed segment is a .flseg file:

[SegmentHeader]   magic "FLO_SEG\0", version, partition id, index range, entry count, commit index at seal
[Entry 0..N]      Sequential entries
[SparseIndex]     Sampled every ~256 entries: index → file offset
[SegmentFooter]   index_offset, index_count, crc32c

The sparse index enables binary search by UAL index without scanning every entry. The footer CRC covers the entire segment — corruption is detected on read.

The durable log

The Raft log of a shard lives in three places: the hot ring, the segment writer's not-yet-flushed buffer, and the sealed segments. The buffer and the segments together hold every entry and are the durable log; the hot ring holds the most recent ones as well. The Raft layer uses the durable log for three things the ring cannot do:

• Restart applies only what was committed. Each segment records the Raft commit index at the moment it was sealed. On boot, entries at or below the highest such watermark are replayed into the projections; entries above it are loaded into the Raft log only and applied once commit is re-established (immediately on a single node, where commit is re-established at bootstrap). A follower that flushed an uncommitted tail its next leader then truncates never applies that tail. Nothing committed is lost.
• Truncation reaches the files. When a follower's log conflicts with its leader and a suffix is dropped, the same suffix is dropped from the writer's buffer and from any sealed segment that holds it: the segment is rewritten under its own name with only the entries below the cut, or deleted if nothing remains. The cut is recorded on disk before any file is touched and the record removed after the last one, so a cut interrupted by a crash is finished by the next boot; until it is finished nothing is flushed, and two segments holding the same index refuse to replay rather than picking one. Without this a later flush would put a second entry at the truncated index into a newer file, and replay would apply the wrong one.
• Catching a peer up from below the ring. A follower that restarted, or was partitioned for longer than the ring holds, needs entries the ring has evicted. The log reads them from the writer's buffer or from the segment that covers the index, so as long as segments exist from index 1 any follower can be repaired from any point without a snapshot.

Projections

Projections are specialized data structures that consume UAL entries and maintain queryable state.

KV Projection

• Hash table mapping keys to values, plus MVCC version chains (ring buffer, default 8 versions per key)
• TTL tracking for automatic expiry
• Reads are O(1) hash table lookups — no Raft round-trip needed
• Historical lookups walk the version chain

Queue Projection

The old system stored each message as 8 separate KV entries (~1.5 KB overhead per message). The new design uses native structures:

• Ready heap — min-heap ordered by priority, 16 bytes per node
• Lease tracker — maps sequences to lease expiry timestamps
• DLQ state — tracks retry counts, moves failures to dead-letter queue

Per-message overhead: ~64 bytes (23× reduction from the old design).

Stream Projection

Streams have no traditional projection. Stream records are UAL entries with entry_type = stream_append. Reading a stream is reading a range of UAL entries — the log index is the stream offset. Zero copy, no derived state.

Consumer group offsets are stored as KV entries (prefixed with cg:), so the KV projection handles that state.

Time-Series Projection

TS data has no dedicated disk files. All ts_write entries are appended to the shared UAL (same .flseg segments as KV, queue, and stream data).

• Write buffers — per-series, per-field in-memory buffers (1024 points capacity)
• Block index — in-memory metadata: min/max timestamp, point count, UAL index range
• Series metadata lives in KV projection under _ts:meta:* keys

When a write buffer fills, flushBuffer() records a block metadata entry (timestamps + UAL index range) and discards the raw points. Reconstructing historical block data requires replaying the UAL in [ual_index_start..ual_index_end]. This means hot reads (from the write buffer) are fast, while cold reads (from flushed blocks) require UAL replay.

One apply path

Every committed entry is applied by exactly one function, whether it was just written on this node, replicated from a peer, or replayed from a segment at boot: the entry goes into the partition (the projection router, plus the hot ring for the types that read back from the log) and then to the applier registered for its type. A live write proposes the entry, applies it through that path, and only then reads its response from projection state — the same state a restart rebuilds.

The projection router owns the three in-memory projections:

Every other type — stream appends, trims and deletes, consumer-group commits, namespaces, actions, workflows, processing jobs — has one registered applier in its subsystem, and an entry type may have one or the other, never both; the server refuses to start otherwise. Raft's own control entries (raft_noop, raft_config, raft_snapshot) belong to the consensus layer and have no applier. Blocking reads wake on the apply, not on the request that caused it.

Snapshots

Projections live in RAM. Without snapshots, recovery requires replaying the entire UAL from the beginning.

Snapshot Format (.fsnap)

┌─────────────────────────────────┐
│ Header (64 bytes)               │
│   magic: "FLO_SNP\0"           │
│   ual_index (snapshot point)    │
│   raft_term, section_count      │
├─────────────────────────────────┤
│ Section: KV (type=0x01)         │
│ Section: Queue (type=0x02)      │
│ Section: TS (type=0x03)         │
│ Section: Stream (type=0x04)     │
├─────────────────────────────────┤
│ Footer (16 bytes)               │
│   crc32c, magic: "FLO_SNE"     │
└─────────────────────────────────┘

Lifecycle

1. Serialize all four projections at the current applied_index
2. Write to .fsnap.tmp
3. fdatasync the temp file
4. Atomic rename .fsnap.tmp → .fsnap
5. Update MANIFEST
6. Old UAL segments with indices ≤ snapshot_index can now be compacted

The atomic rename guarantees crash safety — if the process crashes before the rename, the previous snapshot remains valid.

Crash Safety

Recovery

When a node restarts, each partition recovers:

1. Load snapshot — read MANIFEST, validate CRC, deserialize all projections
2. Open UAL — discover warm segments on disk
3. Replay — feed UAL entries from snapshot_index + 1 through the one apply path (the same code a live write and a replicated entry take)
4. Load cold manifest — metadata only, no data fetched
5. Ready for traffic

If no snapshot exists, recovery replays the entire UAL from the first segment … slow, but correct.

Memory Controller

Each shard gets a fixed memory budget. Default split for a 2 GB shard:

Backpressure Levels

1. Eviction — ask the component to free memory (drop old MVCC versions, spill to disk)
2. Reserve borrow — borrow from the reserve pool, tracked and repaid
3. Client backpressure — return ShardMemoryPressure as a retriable error
4. Hard reject — write rejected immediately to prevent OOM

Directory Layout

{data_dir}/
├── SYSTEM                              # Topology lock (shards, partitions, version)
├── 00000/                              # Shard 0 (zero-padded 5 digits)
│   ├── MANIFEST                        # Latest snapshot ref + cold segment index
│   ├── HARDSTATE                       # Raft node id, current term, vote (28 bytes)
│   ├── segs/
│   │   ├── 0000000001.flseg            # UAL segment (10-digit zero-padded first index)
│   │   ├── 0000000257.flseg
│   │   └── *.flseg.tmp                 # Transient (in-flight writes only)
│   ├── snaps/
│   │   ├── 0000001000-1709234567.fsnap # Snapshot at UAL index 1000
│   │   └── *.fsnap.tmp                 # Transient (in-flight writes only)
│   └── cold.fcold                      # (optional) Cold tier manifest
├── 00001/
│   ├── MANIFEST
│   ├── HARDSTATE
│   ├── segs/
│   └── snaps/
└── ...{shard_count - 1}/

SYSTEM

Written once on first boot. If shards or partitions don't match on restart, the node refuses to start with TopologyMismatch. Format:

{
  "shards": 8,
  "partitions": 256,
  "created_at": 1772033477,
  "version": "1.0.0"
}

MANIFEST

One per shard — a JSON file tracking the latest snapshot pointer and cold segment inventory. No per-directory manifests to coordinate.

HARDSTATE

One per shard — a fixed 28-byte, CRC-protected record of the node's identity and its Raft hard state: node id, current term, and the vote cast in that term. It is rewritten (.tmp → fsync → rename) before the node grants a vote or adopts a new term, never on the write path. On restart the node resumes one term past the stored one, so a node can neither vote twice in a term nor re-enter a term it already left. The stored node id also fixes the node's cluster identity: once written, it survives restarts and port changes, and a configured [cluster] node_id that disagrees with it is refused at startup with NodeIdMismatch.

The Raft log itself is the segment files — index 1 (the first leadership no-op) onward. On boot the node rebuilds its in-memory Raft log (last index, term index, and the hot-ring tail) from the same pass over segs/ that replays projections.

Atomic Writes

Segments, snapshots and HARDSTATE are written atomically via .tmp → fdatasync → rename. If the process crashes before the rename, the previous file remains valid.