Uploads and atomic writes

Atomic writes (hidden stores)

By default, an upload writes into the destination file, like HiddenStores off in ProFTPD: a reader can see a partial file while the transfer lasts.

With enabled = true, the upload writes into an in-flight file in the same directory, then renames it to the destination at the end. The destination appears, or changes, only once the file is complete.

[server.hidden_stores]
enabled = true
prefix = ".in."
extension = "."

All the keys: Reference [server.hidden_stores].

The in-flight file is named <prefix><name>.<token><extension>: .in.rapport.csv.3f9a1c04b7e25d68. with the defaults. The token, 16 hexadecimal characters, is unique to each upload. A .in.* pattern matches them all.

In-flight files are listed and read like the others. They concern the local and sftp (proxy) backends; an S3 object appears only once its upload is finished, with no file next to it.

Per backend, per mount

The same table can be set at three levels. Each key resolves on its own: the mount wins over the backend, which wins over [server.hidden_stores], which wins over the defaults. SFTP and REST resolve the same value.

[server.hidden_stores]            # the whole server
enabled = true

[[backends]]
name = "disque"
type = "local"
root = "/srv/sftp"
hidden_stores = { enabled = false }   # this backend writes in place

[[roles]]
name = "depot"

[[roles.mounts]]
backend = "disque"
mount_path = "/depot"
home_dir = "/depot"
hidden_stores = { enabled = true }    # except this mount
acl = [{ path = "/", rights = ["write", "list"], recursive = true }]

Leftovers of an interrupted transfer

A process killed in the middle of an upload leaves its in-flight file behind. The server deletes it later, when an upload goes through the same directory, once the file is older than grace_secs.

[uploads.stale_partials]
grace_secs = 900
age_check = true

All the keys: Reference [uploads.stale_partials].

The same table can be set per backend (stale_partials = { grace_secs = 1800 } in its [[backends]]), key by key on top of [uploads.stale_partials]. On an S3 backend, it sets the cleanup of abandoned multipart uploads (see S3).

A leftover is deleted only if its name has the exact form the server writes, if nobody holds its lock, and if its age exceeds grace_secs on the storage clock. At the slightest doubt, it stays. Each deletion leaves an INFO line.

The instances that share a storage have the same uploads.idle_timeout_secs: the threshold is computed from that of the instance that sweeps.

Two uploads to the same destination

The first upload to open keeps the destination until it ends. A second one, from the same account or another, through the same door or another, is refused as soon as it opens, before sending a byte.

A stuck upload (client suspended by Ctrl+Z) is taken over by a retry from the same account on the same instance, after uploads.takeover_idle_secs without data.

Between several instances, a lock set on the storage, next to the destination, carries the same rule. It is refreshed every 20 s; the lock of a process that is gone is taken over after 2 x idle_timeout_secs + 60 s (120 s by default). cross_instance_reservation and lock_prefix: Reference [[backends]].

What the lock is on each type: Local, S3, SFTP proxy. Names that start with .craftfilegate-upload or with the lock_prefix belong to the server: you can list and read them, not write to them. With a single instance, cross_instance_reservation = false is enough.

Capping the size of a file

max_file_mb, on a mount, caps the size of each uploaded file (1 MB = 1,048,576 bytes). It is not a space counter.

[[roles.mounts]]
backend = "disque"
home_dir = "/{username}"
max_file_mb = 500

What you will see