S3 and compatibles
The s3 backend stores files as objects in an AWS S3 bucket or in a
compatible storage (MinIO, Garage, Scaleway…).
[[backends]]
name = "archives"
type = "s3"
bucket = "archives"
region = "eu-west-3"
prefix = "sftp/"
endpoint_url = "https://minio.interne.example:9000" # without this key: AWS
[backends.credentials]
type = "static"
access_key_id = "AKIAEXEMPLE"
secret_access_key = "changez-moi"
[[roles]]
name = "archivistes"
[[roles.mounts]]
backend = "archives"
home_dir = "/{username}"
acl = [{ path = "/", rights = ["read", "write", "list"], recursive = true }]
alice uploads /in/x.txt: the object is the key sftp/alice/in/x.txt.
The keys
All keys: Reference [[backends]] type = “s3”.
iam_role: AWS_* variables, profile, instance or pod role. Common
keys: Backends.
Files and directories
| What the client sees | On the storage |
|---|---|
| a file | a key |
| a directory | a marker <key>/, or any key under <key>/ |
a mkdir | writes the marker |
| missing parents of an upload | one marker per level, each judged by the ACL like a mkdir |
| a directory without a marker | size 0, no date |
A key and a directory with the same name can coexist, except with
refuse_upload_over_directory = true.
| Operation | S3 requests |
|---|---|
| SFTP read | one HeadObject at open, then one GetObject with Range per read |
| REST download | one GetObject read as a stream |
| upload under 8 MiB | one PutObject with If-None-Match: *, at the end |
| upload of 8 MiB or more | a multipart upload, one 8 MiB part at a time |
| file rename | CopyObject with If-None-Match: *, then DeleteObject of the source |
| listing | ListObjectsV2 with the / delimiter, page after page |
| tree deletion | ListObjectsV2 by 1,000 keys, then DeleteObjects in batches |
An object is written whole: no resume (reput), no append. Only a file
can be renamed.
Bucket permissions
On the bucket and <prefix>/*:
| Action | For |
|---|---|
s3:ListBucket | list, tell a directory apart, find missing parents, mkdir, rename, delete a tree |
s3:GetObject | download, stat, and the check that precedes a deletion or a rename |
s3:PutObject | upload, mkdir, rename, the conditional write self-test |
s3:DeleteObject | delete, rename, clean up the self-test key |
s3:AbortMultipartUpload | abort a multipart upload that will not be completed |
s3:ListBucketMultipartUploads | sweep abandoned multipart uploads |
s3:ListMultipartUploadParts | date the last part of a sweep candidate |
A read-only backend only needs s3:ListBucket and s3:GetObject.
Multipart uploads
A multipart upload that is never completed is billed until it is aborted:
| When | Abort |
|---|---|
| an upload ends without completing | immediately, whatever the cause |
| clean shutdown | those still in flight, 5 s at most |
| a killed process | the sweep under <prefix>/: at startup, every grace_secs and at each upload, when the last part is older than grace_secs on the storage clock |
- Give each backend a
prefix: the sweep only works under it. - A slow upload sends an empty part every 15 s, which dates it for the
other instances; they have the same
uploads.idle_timeout_secs. - A lifecycle rule is still recommended:
{"Rules": [{"ID": "abort-incomplete-multipart", "Status": "Enabled", "Filter": {"Prefix": ""},
"AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 2}}]}
Its delay, counted from initiation, exceeds the longest upload. MinIO
expires incomplete uploads on its own (stale_uploads_expiry).
Conditional writes
replaced in the audit and the reservation marker rely on
If-None-Match: * and If-Match of PutObject. Behind an endpoint_url (and
on AWS with cross_instance_reservation), the server measures them at
startup, on the key <prefix>/.craftfilegate-upload-precondition.<16 hex>,
deleted afterwards; the verdict holds until the next restart. MinIO: honoured.
Common keys on S3
| Key | On S3 |
|---|---|
home_dir (mount) | a prefix: the key is <prefix>/<home_dir>/<path>, empty parts omitted |
create_home (mount) | no effect: a prefix exists as soon as a key is under it |
refuse_upload_over_directory | true: an upload to a name that is also a directory is refused; cost: a one-key listing when each upload opens |
cross_instance_reservation | a marker object <lock_prefix><name>, put with If-None-Match: * |
lock_prefix | the name of these markers |
stale_partials | the sweep of abandoned multipart uploads |
case_insensitive | false by default: keys are case-sensitive |
hidden_stores | not applicable: an object only appears once complete |
Performance and limits
- Each upload in progress holds an 8 MiB buffer (Memory); its writes are sequential.
- An upload outside the root pays one listing per parent level.
- The client addresses the bucket in the URL path (path-style), AWS included.
- 10,000 parts at most per object, about 78 GiB at 8 MiB per part.
- On a versioned bucket, the self-test leaves two empty versions and one delete marker per startup.
What you will see
| When | Line |
|---|---|
| startup | INFO S3 backend initialized, fields bucket, prefix, endpoint |
| startup, self-test | INFO S3 upload precondition self-test, fields backend, verdict = honoured, probe_key |
| sweep | INFO aborted an abandoned multipart upload: ..., fields bucket, key, upload_id, age_secs |
| shutdown | INFO uncompleted S3 multipart uploads aborted before the shutdown completes, field aborted |