Cluster Scaling: Development to Production

Ferrosa scales from a single development node to a production cluster without downtime. Most databases force you to choose your deployment topology up front — a single-node install that you later throw away and replace with a cluster, or a heavyweight multi-node setup that’s overkill for development. Ferrosa eliminates that tradeoff: start with one node on your laptop, add a hot standby when you’re ready for production, then grow into a full Raft consensus cluster when your traffic demands it.

This tutorial walks through three stages of that journey: standalone (development), pair mode (low-volume production with a hot standby), and Raft cluster (full production with consensus). At each transition, writes and schema changes continue uninterrupted. The data you inserted on a single node is still there after you’ve scaled to three — no migrations, no re-bootstrapping, no downtime windows.

Prerequisites

Docker and Docker Compose — see the Cluster Setup Guide if you need to install Docker
cqlsh — pip install cqlsh
About 4 GB of RAM available for the three nodes plus the object store

Act 1: Standalone — Development

A single Ferrosa node is all you need for development. It starts fast, uses minimal resources, and gives you a fully functional CQL endpoint.

Start node1

docker compose up -d

This launches node1 in standalone mode along with a local S3-compatible object store (RustFS). Wait for node1 to report healthy:

docker compose ps

You should see node1 with status "healthy" and rustfs running.

Create the schema

Connect to node1 and bootstrap the event analytics schema:

cqlsh localhost 9042

-- Act 1: Development / Standalone Mode
-- Create the application keyspace and core table.
-- RF=1 is appropriate for single-node development.

CREATE KEYSPACE IF NOT EXISTS app
    WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};

CREATE TABLE IF NOT EXISTS app.events (
    tenant_id  text,
    event_date text,
    event_id   int,
    event_type text,
    payload    text,
    PRIMARY KEY ((tenant_id, event_date), event_id)
) WITH CLUSTERING ORDER BY (event_id ASC);

This creates the app keyspace with replication_factor: 1 — appropriate for a single development node — and an events table partitioned by tenant and date.

Insert seed data

-- Act 1: Standalone — seed initial event data
-- 5 events for acme, 5 for globex (10 total)

USE app;

-- acme events (IDs 1-5)
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('acme', '2026-03-18', 1, 'page_view', '{"url": "/home"}');
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('acme', '2026-03-18', 2, 'click', '{"button": "signup"}');
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('acme', '2026-03-18', 3, 'signup', '{"plan": "free"}');
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('acme', '2026-03-18', 4, 'page_view', '{"url": "/dashboard"}');
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('acme', '2026-03-18', 5, 'purchase', '{"item": "pro_plan", "amount": 29.99}');

-- globex events (IDs 1-5)
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('globex', '2026-03-18', 1, 'page_view', '{"url": "/products"}');
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('globex', '2026-03-18', 2, 'click', '{"button": "add_to_cart"}');
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('globex', '2026-03-18', 3, 'purchase', '{"item": "widget", "amount": 9.99}');
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('globex', '2026-03-18', 4, 'page_view', '{"url": "/checkout"}');
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('globex', '2026-03-18', 5, 'signup', '{"plan": "enterprise"}');

-- Verify counts
SELECT COUNT(*) FROM events WHERE tenant_id = 'acme' AND event_date = '2026-03-18';
-- Expected: 5
SELECT COUNT(*) FROM events WHERE tenant_id = 'globex' AND event_date = '2026-03-18';
-- Expected: 5

replication_factor: 1 means every write lands on the only node that exists. No coordination overhead, no quorum delays — ideal for rapid iteration during development.

Act 2: Pair Mode — Low-Volume Production

Pair mode adds a hot standby that receives replicated writes in real time. If the primary fails, the secondary can serve reads immediately. Think of it like PostgreSQL streaming replication, but it also replicates DDL — schema changes propagate automatically.

Add node2

Scale from standalone to pair mode by layering the pair override file:

docker compose -f docker-compose.yml -f docker-compose.pair.yml up -d

This does two things: it restarts node1 with FERROSA_CLUSTER_MODE=pair, and it launches node2 as the secondary. Node2 connects to node1, receives the current schema and data, and begins replicating writes.

Writes continue uninterrupted during this transition. Node1 accepts CQL commands throughout the mode change. If you have a background writer running (like run-demo.sh provides), it will not see any errors.

Watch the pair form:

docker compose logs -f node1 node2

You should see node2 complete its handshake and report "Pair mode established with node1". Press Ctrl+C to stop following logs.

Write more data

With both nodes running, insert additional events. These writes replicate to node2 in real time:

-- Act 2: Pair mode — writes replicated to second node
-- 3 more for acme (total 8), 2 more for globex (total 7)

USE app;

-- acme events (IDs 6-8)
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('acme', '2026-03-18', 6, 'click', '{"button": "export_csv"}');
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('acme', '2026-03-18', 7, 'page_view', '{"url": "/reports"}');
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('acme', '2026-03-18', 8, 'purchase', '{"item": "addon_analytics", "amount": 14.99}');

-- globex events (IDs 6-7)
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('globex', '2026-03-18', 6, 'page_view', '{"url": "/settings"}');
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('globex', '2026-03-18', 7, 'click', '{"button": "invite_team"}');

-- Verify counts after pair writes
SELECT COUNT(*) FROM events WHERE tenant_id = 'acme' AND event_date = '2026-03-18';
-- Expected: 8
SELECT COUNT(*) FROM events WHERE tenant_id = 'globex' AND event_date = '2026-03-18';
-- Expected: 7

Schema changes replicate too

One of pair mode’s key features is DDL replication. Add a new column and a secondary index — the schema change propagates to node2 automatically:

-- Phase 2: Pair mode — DDL replication test
-- Create a counter table and populate it with daily summaries

USE app;

CREATE TABLE IF NOT EXISTS daily_summary (
    tenant_id  text,
    event_date date,
    event_type text,
    event_count counter,
    PRIMARY KEY ((tenant_id, event_date), event_type)
);

-- acme summaries for 2026-03-18
UPDATE daily_summary SET event_count = event_count + 3
    WHERE tenant_id = 'acme' AND event_date = '2026-03-18' AND event_type = 'page_view';
UPDATE daily_summary SET event_count = event_count + 2
    WHERE tenant_id = 'acme' AND event_date = '2026-03-18' AND event_type = 'click';
UPDATE daily_summary SET event_count = event_count + 1
    WHERE tenant_id = 'acme' AND event_date = '2026-03-18' AND event_type = 'signup';
UPDATE daily_summary SET event_count = event_count + 2
    WHERE tenant_id = 'acme' AND event_date = '2026-03-18' AND event_type = 'purchase';

-- globex summaries for 2026-03-18
UPDATE daily_summary SET event_count = event_count + 3
    WHERE tenant_id = 'globex' AND event_date = '2026-03-18' AND event_type = 'page_view';
UPDATE daily_summary SET event_count = event_count + 2
    WHERE tenant_id = 'globex' AND event_date = '2026-03-18' AND event_type = 'click';
UPDATE daily_summary SET event_count = event_count + 1
    WHERE tenant_id = 'globex' AND event_date = '2026-03-18' AND event_type = 'signup';
UPDATE daily_summary SET event_count = event_count + 1
    WHERE tenant_id = 'globex' AND event_date = '2026-03-18' AND event_type = 'purchase';

-- Verify counter table
SELECT * FROM daily_summary WHERE tenant_id = 'acme' AND event_date = '2026-03-18';
SELECT * FROM daily_summary WHERE tenant_id = 'globex' AND event_date = '2026-03-18';

Verify the schema arrived on node2 by connecting to port 9043:

cqlsh localhost 9043 -e "DESCRIBE TABLE app.events;"

You should see the new user_id column and events_by_user index on both nodes.

In traditional Cassandra, schema changes propagate via gossip and can temporarily disagree across nodes. In Ferrosa pair mode, DDL is replicated synchronously — the secondary’s schema is always consistent with the primary’s.

Act 3: Raft Cluster — Full Production

When your traffic outgrows a two-node pair, add a third node to form a Raft consensus cluster. Three nodes give you:

Fault tolerance — the cluster continues operating if any single node fails
Tunable consistency — choose between ONE, QUORUM, or ALL per query
Consensus-backed metadata — schema changes and topology updates go through Raft, guaranteeing agreement

Add node3

Layer all three compose files to bring up the full cluster:

docker compose -f docker-compose.yml -f docker-compose.pair.yml -f docker-compose.cluster.yml up -d

This transitions all three nodes to FERROSA_CLUSTER_MODE=cluster. Node3 joins, a Raft leader election completes, and the cluster stabilizes.

docker compose logs -f node1 node2 node3

Watch for these log messages:

Node3 connects to node1 (the seed)
Raft leader election completes — one of the three nodes becomes leader
All three nodes report "Cluster mode active, 3 nodes in ring"

Write through the cluster

Now writes are coordinated through Raft consensus. Insert more events — these are acknowledged after a quorum (2 of 3 nodes) confirms:

-- Act 3: Raft cluster — writes across a 3-node cluster
-- 2 more for acme (total 10), 3 for new tenant initech (total 3)

USE app;

-- acme events (IDs 9-10)
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('acme', '2026-03-18', 9, 'page_view', '{"url": "/billing"}');
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('acme', '2026-03-18', 10, 'click', '{"button": "upgrade_plan"}');

-- initech events (IDs 1-3, new tenant)
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('initech', '2026-03-18', 1, 'page_view', '{"url": "/home"}');
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('initech', '2026-03-18', 2, 'signup', '{"plan": "trial"}');
INSERT INTO events (tenant_id, event_date, event_id, event_type, payload)
    VALUES ('initech', '2026-03-18', 3, 'page_view', '{"url": "/onboarding"}');

-- Verify counts across all tenants
SELECT COUNT(*) FROM events WHERE tenant_id = 'acme' AND event_date = '2026-03-18';
-- Expected: 10
SELECT COUNT(*) FROM events WHERE tenant_id = 'globex' AND event_date = '2026-03-18';
-- Expected: 7
SELECT COUNT(*) FROM events WHERE tenant_id = 'initech' AND event_date = '2026-03-18';
-- Expected: 3

DDL through Raft

Schema changes now go through Raft consensus, guaranteeing that all nodes agree on the schema before the DDL statement returns:

-- Phase 3: Raft cluster — DDL through consensus
-- Create a tenants table and register all known tenants

USE app;

CREATE TABLE IF NOT EXISTS tenants (
    tenant_id   text PRIMARY KEY,
    name        text,
    plan        text,
    created_at  timestamp,
    contact     text
);

INSERT INTO tenants (tenant_id, name, plan, created_at, contact)
    VALUES ('acme', 'Acme Corporation', 'pro', '2026-01-15T09:00:00Z', 'admin@acme.example');
INSERT INTO tenants (tenant_id, name, plan, created_at, contact)
    VALUES ('globex', 'Globex Corporation', 'enterprise', '2026-02-01T14:30:00Z', 'ops@globex.example');
INSERT INTO tenants (tenant_id, name, plan, created_at, contact)
    VALUES ('initech', 'Initech', 'trial', '2026-03-18T10:00:00Z', 'peter@initech.example');

-- Verify tenant records
SELECT * FROM tenants;

In cluster mode, DDL statements are linearizable. A CREATE TABLE that succeeds is guaranteed to be visible on every node immediately — no schema agreement delays.

Verification

Confirm that all data survived every transition and that the cluster is healthy.

-- Final verification: confirm all data across the scaled cluster

USE app;

-- Event counts per tenant
SELECT COUNT(*) FROM events WHERE tenant_id = 'acme' AND event_date = '2026-03-18';
-- Expected: 10
SELECT COUNT(*) FROM events WHERE tenant_id = 'globex' AND event_date = '2026-03-18';
-- Expected: 7
SELECT COUNT(*) FROM events WHERE tenant_id = 'initech' AND event_date = '2026-03-18';
-- Expected: 3

-- Daily summaries
SELECT * FROM daily_summary WHERE tenant_id = 'acme' AND event_date = '2026-03-18';
SELECT * FROM daily_summary WHERE tenant_id = 'globex' AND event_date = '2026-03-18';

-- Tenant registry
SELECT * FROM tenants;

-- Cluster health
SELECT cluster_name, listen_address, data_center, rack FROM system.local;
SELECT peer, data_center FROM system.peers;

Connect to each node and check:

Row counts — the app.events table should contain all rows inserted across all three phases
Schema agreement — DESCRIBE TABLE app.events should show identical output on all three nodes, including the columns and indexes added during pair mode and cluster mode
Cluster health — query the web API on any node:

curl -s http://localhost:9090/api/cluster/status | python3 -m json.tool

All three nodes should show "status": "UP".

Running the Demo

The run-demo.sh script automates the entire sequence above — starting each phase, running CQL scripts, maintaining a background writer to prove zero-downtime transitions, and asserting correctness at each stage.

bash run-demo.sh

The script prints progress for each act and exits with a non-zero status if any assertion fails.

Teardown

Remove all containers and volumes:

docker compose -f docker-compose.yml -f docker-compose.pair.yml -f docker-compose.cluster.yml down -v

This deletes all data. The S3 bucket, node volumes, and container state are all removed.