Several instances

The instances of a deployment find and talk to each other on a port of their own, over TLS 1.3 in both directions, admitted by the certificate they share. The Kubernetes chart sets all of this up as soon as replicaCount exceeds 1.

[cluster]
listen = "0.0.0.0:8083"
peers = "dns:craft-file-gate-cluster:8083"
cert_file = "/var/lib/craft-file-gate/cluster/tls.crt"
key_file = "/var/lib/craft-file-gate/cluster/tls.key"

All keys: Reference [cluster].

The shared certificate

Its SHA-256 fingerprint is the identity of the cluster: neither name nor date is checked, expiry is reported by craftfilegate_cluster_cert_not_after_seconds. To change it, remove the pair and restart every instance; during the rollout, a peer with the old certificate is certificate_mismatch.

kubectl patch secret sftp-cluster --type=json \
  -p '[{"op":"remove","path":"/data/tls.crt"},{"op":"remove","path":"/data/tls.key"}]'
kubectl rollout restart deployment/sftp

Detectors and withdrawal from service

After each round, each instance evaluates three detectors: storage_alone (a backend down here that a reachable peer sees up: an outage that everyone sees does not count), isolated (fewer than min_peers reachable peers for isolated_after_secs) and isolated_and_storage_down (isolated, and every non-local backend down here; a local backend, the pod’s disk, does not count: without a non-local backend, it never fires). By default they only alert: WARN cluster detector active, craftfilegate_unready_detector{detector}, craftfilegate_cluster_isolated, checks.cluster of /admin/health.

Named in unready_when, they withdraw the instance from service: /readyz not ready after unready_after_secs of an active detector, ready again after ready_after_secs with none; /livez does not change. storage_alone only withdraws a pod if another pod, in service and with no backend down, sees each of its failed backends available, and only one pod at a time: if every pod has an outage somewhere, none leaves.

[cluster]
min_peers = 1                    # the chart: floor(replicaCount / 2)
unready_when = ["storage_alone", "isolated_and_storage_down"]  # the chart: auto

isolated alone can withdraw every pod: cut of the cluster port, loss of a majority of nodes, split into two equal halves, manual reduction of the replica count. Only name it knowingly, and never go below 2 × min_peers + 1 replicas without helm upgrade.

With the defaults, a storage outage is seen in 35 s at worst, the withdrawal follows 60 s later, and Kubernetes takes the pod out of the Service after 3 readiness probes at 10 s: about 2 min.

What you will see

At startup, INFO cluster channel listening (TLS 1.3, pinned on the shared certificate); for each peer that answers, INFO cluster peer reachable; per status: craftfilegate_cluster_peers{status}. The Instances, Sessions and Bans tabs of the console read each peer with your credential, which the peer judges itself (scope=cluster); a kick goes only to the peer that holds the session, a lift to every peer, except on a ConfigMap list; per call: craftfilegate_cluster_relay_total{route,result}.

A ban there is a guard against attempts, not a withdrawal of access: an address banned on a peer still reads it through another instance with a valid credential (a revoked session stays refused).