Troubleshooting
An index of errors: the exact word you see (startup message,
reason of an audit line, SFTP status, REST code) leads to the guide page
that defines the option at fault, then, in parentheses, to its table in the
reference, and to the identifier of the behaviour rule, to quote to support.
<...> marks a variable part. The full vocabulary of audit lines is in the
reference.
Startup and configuration
A broken setting refuses to start; a setting with no effect starts with a
WARN that names the key.
| Message | Where it is set | Rule |
|---|---|---|
--config is required when not using a subcommand, Configuration error: failed to read config file: <cause> | The configuration file | R-CONFIG-001 |
Configuration error: failed to parse config TOML: unknown field <key>, expected one of ... (at line <l>, column <c>) | the guide page of the named section (reference): the misspelled key | R-CONFIG-005 |
... `sftp.<key>` is now `server.<key>`: move it to the [server] table: shutdown_grace_period_secs, max_sessions_per_user or hidden_stores written under [sftp] | The keys of [sftp] ([server]) | R-CONFIG-015 |
[sftp]: this binary is built without the SFTP door (Cargo feature door-sftp) (or [api], REST, door-rest), no door is configured, nothing would be served: ..., this binary is built without any door ... | reference: [sftp], or [api] with [admin]; The binary’s features: a binary built without that door | R-CONFIG-016 |
server.probes_listen = <address> is also <key> — the probes need a port of their own | One door or two ([server]) | R-ADMIN-020 |
server.backend_probe.<key> = <n> is out of bounds: between <min> and <max>, server.backend_probe.timeout_secs = <n> is not below interval_secs = <m>: ... | Availability ([server.backend_probe]) | R-AVAIL-001 |
Configuration error: <section>: <cause>, <key> = <n> is out of range: accepted values are <min> to <max> | the guide page of the named section or key (reference) | R-CONFIG-010 |
Configuration error: failed to load local users: <cause> | Local accounts ([[users]]) | R-AUTH-004 |
duplicate username: <name> — each [[users]] entry needs its own username | Local accounts ([[users]]) | R-CONFIG-011 |
user <name> has a sha512-crypt password hash, which is not accepted — set [auth.methods] local.allow_sha512_crypt = true ... (or bcrypt); WARN local user's password_hash is the example published in the README or the example files ..., local user's argon2 hash asks for more memory per verification than auth.methods.local.max_argon2_memory_kib allows: ..., local users whose password hash has another format or other parameters than most of the users file: ..., local users whose password hash cannot be verified (...): ..., accounts still depend on a legacy password hash flag: ..., legacy password hash flag enabled but no account uses this format ..., password hash is not argon2id — ... | Hashes ([auth.methods]) | R-AUTH-004, R-AUTH-005, R-AUTH-007, R-AUTH-008, R-AUTH-010, R-CONFIG-012 |
Configuration error: failed to load roles/backends: <cause>; duplicate role name, role without [[roles.mounts]], mount to an unknown backend | The keys of a role and a mount, Keys of every backend ([[roles]], [[roles.mounts]], [[backends]]) | R-AUTH-028 |
unknown field backend (or home_dir, acl) under [[roles]]: a mount key written at the role level | The keys of a role and a mount ([[roles.mounts]]) | R-CONFIG-005 |
role has no ACL entry: access is deny-by-default, ... (WARN) | Writing an ACL ([[roles.mounts.acl]]) | R-ACL-003 |
role "<name>" holds ACL paths that name one path on this backend, whose ACL compares paths folded (case and Unicode normalization), with different rights: [...] | Name case, Case of ACL paths ([[roles.mounts.acl]], [[backends]]) | R-ACL-006 |
role missing required fields: role '<name>': mount '<p>': home_dir = "<h>" climbs out of the backend's root | Where the files go ([[roles.mounts]]) | R-AUTH-032 |
user '<name>': roles '<a>' and '<b>' both claim '<mount_path>' ... (at startup, or failed to reload roles file): conflicting mounts of a local user | Combining roles, Mount recipes ([[roles.mounts]]) | R-AUTH-030 |
role '<name>': user_key_algorithms: "<algorithm>" is not an algorithm this server supports; ... | Signature algorithms of a key ([[roles]]) | R-AUTH-019 |
no authentication methods enabled | Choosing the methods ([auth.methods]) | R-AUTH-001 |
no JWT key source, jwks_url with another source, unknown algorithm; first JWKS load failed, the first fetch from auth.jwt.jwks_url failed, ...; WARN auth.jwt.public_key_file is never read: ..., every JWT will be refused: auth.jwt.secret and auth.jwt.public_key_file are both set ..., [auth.jwt] is configured but auth.methods.jwt.enabled is false: ... | JWT, Connecting an identity provider ([auth.jwt]) | R-AUTH-020, R-AUTH-026, R-AUTH-025 |
auth: auth.hash_workers = 0 is refused: it must be between 1 and 1024 ... | Checking passwords ([auth]) | R-AUTH-014 |
unknown backend type "<type>": this binary knows <types> | Types, The binary’s features ([[backends]]) | R-CONFIG-013 |
sftp.host_keys: <path> does not exist. Create it (ssh-keygen ...) or set sftp.generate_host_key ..., sftp.host_keys: cannot read <path>: <error> (the key must be readable by the image’s user) | Host keys, Docker, Secrets ([sftp]) | R-SFTP-003 |
| a refused algorithm list (unknown name, mixed forms, empty list) | Algorithms ([sftp.algorithms]) | R-SFTP-006 |
[cluster] has no listen ..., ... has no peers ..., ... has no certificate source ..., ... sets both cert_file/key_file and secret_name ..., cluster.peers entry "<entry>" cannot be read: ..., cluster.secret_name is empty: ..., cluster.secret_name is set, but this binary was built without the k8s feature: ..., cannot listen on cluster.listen = <address>: <cause>, the cluster channel cannot start, refusing to boot: ..., cluster.<key> is set without cluster.<other> ..., cluster.peer_timeout_ms = <n> is out of bounds ..., cluster.listen = <address> is also <key> ..., cluster certificate (cert_file=<c>, key_file=<k>): <reason>; WARN [cluster] has nothing to add across instances: ... | Cluster ([cluster]) | R-CLUSTER-001, R-CLUSTER-002 |
cluster.<key> = <n> is out of bounds ..., cluster.unready_when names <detector>, but cluster.min_peers = 0 turns isolation off ..., ... but server.backend_probe.enabled = false ..., cluster.<key> = <n> is under <min> s (...): one storage verdict could withdraw or restore the instance, CRAFT_FILE_GATE_CLUSTER_MIN_PEERS = ... cannot be read ..., CRAFT_FILE_GATE_CLUSTER_UNREADY_WHEN = "..." cannot be read: ...; WARN a [cluster] detector key is set where it has no effect | Detectors and withdrawal from service ([cluster]) | R-CLUSTER-016 |
WARN cluster detector active (detector, reason), cluster detector inactive; withdrawn from service: ..., back in service: ...; /readyz 503 withdrawn_by; storage_alone active without withdrawal | Detectors and withdrawal from service: storage_alone, the storage of this pod alone (mount, node network); active without withdrawal: no pod in service and healthy (with no backend down) sees all its down backends available, or a pod with a smaller identifier goes first; isolated, the network between pods; isolated_and_storage_down, isolated and every non-local backend down: the node network; the reason is in /admin/health (checks.cluster) | R-CLUSTER-015, R-CLUSTER-016 |
<anchor>: [the directory ]<path> is <reason>: whoever can write there decides who gets in. ...; a file this server trusts can be rewritten: ... (WARN) | Trusted files: chmod go-w, or mount read-only | R-TRUST-008 |
<anchor>: <path> is writable by the server's own group (<gid>): ... | [security] ([security]) | R-TRUST-004 |
<anchor>: <path> cannot be resolved: <error> | Setting them | R-TRUST-007 |
private key readable by others (WARN) | Trusted files | R-TRUST-011 |
<X> and <X>_FILE are both set: one secret, two sources..., <X>_FILE is set but empty, ... is empty: an empty secret is no secret, ... is not UTF-8 text | A file rather than a variable (Environment variables) | R-CONFIG-009 |
<VARIABLE>="<value>" is refused: it must be a whole number...; env override ignored, the configured value stands, env override is empty (WARN) | The environment (Environment variables) | R-CONFIG-008 |
server.shutdown_grace_period_secs must be > 0 | Shutdown ([server]) | R-SHUTDOWN-004 |
admin requires a bearer_token or at least one role under [[admin.roles]] | Authenticating ([admin], [[admin.roles]]) | R-ADMIN-001 |
admin.listen and sftp.listen are the same address — they cannot share a port, admin.control_listen = <address> is also admin.listen — the control door needs a port of its own, the admin door cannot start, refusing to boot: ... cannot listen on <address> (port already taken) | One door or two ([admin]) | R-ADMIN-001, R-ADMIN-002 |
tls: cert_file and key_file must both be set, tls: enabled but no cert source ..., tls: cert_file and auto_generate are mutually exclusive; pair refused at startup: <cause> (cert_file=<path>, key_file=<path>) | Deployment recommendations ([admin.tls]) | R-ADMIN-001, R-ADMIN-003 |
[[admin.roles]] has an entry with a blank name, [[admin.roles]] names the role <name> twice, [[admin.roles]] <name> grants no permission: ..., unknown variant <permission>; [[admin.roles]] is set but no credential can open the admin door: ... | Authenticating ([[admin.roles]], [auth.methods], [auth.jwt]) | R-ADMIN-006 |
role <name> is also the name of an [[admin.roles]] entry: admin roles and file roles need different names (at startup, or failed to reload roles file, keeping old config); an admin role and a file role have names that differ only by case or spaces: ... (WARN) | Authenticating ([[admin.roles]], [[roles]]): rename one of the two | R-ADMIN-006 |
this user's keys open no door: its authorities name no role with mounts, only admin roles, ... (WARN) | Authenticating ([[admin.roles]]): remove authorized_keys, or give a file role | R-ADMIN-006 |
the admin door signs local accounts in with their passwords ... and has no [admin.ban]: ... | Console sessions ([admin.ban]) | R-ADMIN-022 |
admin.session.key_file <path> holds <n> bytes: a session key needs 32 at least ..., admin.session.key_file: ... writable by ..., admin.session.key_file <path>: <cause>; admin.session.ttl_secs = <n> is out of range ..., admin.session.max_age_secs = <n> is under admin.session.ttl_secs ..., admin.session.key_file and admin.session.secret_name are both set ..., admin.session.secret_name is empty ..., admin.session.secret_name is set, but this binary was built without the k8s feature ... | The session key ([admin.session]): head -c 32 /dev/urandom, chmod 400; The binary’s features | R-ADMIN-022 |
auth.jwt.issuer = "craft-file-gate" is the issuer of the console's own session tokens ... | [auth.jwt]: another issuer | R-ADMIN-005 |
admin session key generated for this process: ... (WARN), tokens refused by another pod or after a restart | The session key ([admin.session]): key_file or secret_name | R-ADMIN-022 |
admin.allow_static_token = false, and a static admin token is set by <source>: ...; the static admin token is enabled beside admin roles: ..., static admin token used: it is meant for break-glass only (WARN) | Authenticating ([admin]): remove the token from <source>, or allow_static_token = false | R-ADMIN-024 |
api.enabled requires [admin] section to be configured | REST file API ([api], [admin]) | R-REST-001 |
[api.ui] enabled = true requires [api] enabled = true: ..., [api.ui] path "<path>" is not usable, ... collides with ... | File explorer ([api.ui], [api]) | R-EXPLORER-001, R-EXPLORER-002 |
[api] openapi = true requires [api] enabled = true: ... | REST file API ([api]) | R-REST-012 |
admin.metrics_thresholds.jwks_age_secs = <age> is not above auth.jwt.jwks_refresh_interval_secs ...; WARN a metrics threshold is set where it has no effect: ... | Console thresholds ([admin.metrics_thresholds], [auth.jwt]) | R-METRICS-017 |
invalid telemetry otlp_endpoint, invalid telemetry protocol, invalid telemetry metrics_interval_secs; cannot build the OTLP <signal> exporter: ...; refusing to start | The endpoint, [telemetry] ([telemetry]): on_exporter_error | R-TELEMETRY-001, R-TELEMETRY-003 |
unknown ban backend ...: expected "file" or "configmap", sftp.ban: ..., api.ban: ..., admin.ban: ...; the file API has no ban list: set [api.ban], ..., the admin routes have no ban list: set [admin.ban], ..., [api.ban] is set where it has no effect: ... (WARN) | The keys of a ban, One list per door ([sftp.ban], [api.ban], [admin.ban]) | R-BAN-018 |
[api.ban] trusted_proxies [...] and [admin.ban] trusted_proxies [...] differ, while the file API and the admin routes share [admin] listen: ..., [admin.ban] trusted_proxies [...] is set and the file API has no [api.ban]: ..., [api.ban] trusted_proxies [...] is set and there is no [admin.ban], while ... share [admin] listen: ... | The address of an HTTP client ([api.ban], [admin.ban]): both lists with the same trusted_proxies, or [admin] control_listen | R-BAN-021 |
hidden_stores: prefix shaped like a lock, no affix, grace_secs out of bounds (... is too short: ...) | Atomic writes ([server.hidden_stores], [[backends]], [[roles.mounts]]) | R-HIDDEN-004, R-HIDDEN-005, R-HIDDEN-014 |
uploads.takeover_idle_secs is not below uploads.idle_timeout_secs: ... | Two uploads to the same destination ([uploads]) | R-RESERVE-017, R-CONFIG-010 |
WARN create_home is set on a mount whose backend is not local: ..., home_dir holds a marker that is not {username}: ..., follow_symlinks = true on this local backend: ... | Common keys on a local backend, The root and symbolic links, {username} | R-LOCAL-002, R-LOCAL-005, R-AUTH-032 |
lock_prefix invalid; cross_instance_reservation = true is not supported on a local backend on Windows | Reservation across instances ([[backends]]) | R-RESERVE-013, R-RESERVE-016 |
reload.poll_interval_secs = <n> is out of range: accepted values are 1 to 60; refusing to start with reload.watch = "inotify" and the inotify instance cannot be created (or a directory cannot be watched) | Hot reload ([reload]): watch = "auto" or "poll" | R-RELOAD-010 |
a [telemetry] metrics key is set where it has no effect, ... is set where it has no effect (WARN) | the guide page of the named key (reference): remove it | R-CONFIG-010 |
this secret is the example published in the README or the example files (WARN) | Secrets: change the secret | R-CONFIG-012 |
outbound TLS is configured but there are no CA certificates to verify it with, ... (WARN), cause: SSL_CERT_FILE points at <path>, which does not exist, so it is ignored | TLS trust roots (Environment variables) | R-TRUST-014 |
Hot reload
A file refused at reload keeps what is in force. See Hot reload.
| Message | Where it is set | Rule |
|---|---|---|
configuration edited but not applied until restart (keys=...) | What a reload applies, key by key (reference): restart | R-RELOAD-003 |
failed to reload users file, keeping old config, failed to reload roles file, keeping old config, failed to build backends registry from reloaded roles, keeping old config, failed to re-read the config for its log level, keeping the current one, [log] level is not a log level (accepted values: ...); keeping the log filter currently in force | Local accounts, The keys of a role and a mount, Keys of every backend, [log] ([[users]], [[roles]], [[backends]], [log]): the field e gives the cause | R-RELOAD-005 |
password hashing pool size edit not applied: the pool is sized once, at startup | Checking passwords ([auth]) | R-AUTH-014 |
[log] format edit ignored: ... | [log] ([log]): restart | R-AUDIT-029 |
[log] level edit ignored: <variable> is in force ... | Who decides the level ([log], Environment variables): the variable wins | R-CONFIG-007 |
admin TLS certificate reload failed, still serving the previous certificate; admin TLS certificate reload refused, still serving the previous certificate (ERROR) | What a reload applies, key by key, Trusted files ([admin.tls]): pair unreadable, mismatched or refused | R-RELOAD-008, R-TRUST-010 |
file hot reload cannot use inotify, falling back to re-reading the files every <n> s; the file watcher lost events: its event queue overflowed, ..., the file watcher failed and may have lost events ... | Hot reload ([reload]) | R-RELOAD-010, R-RELOAD-012 |
reload panicked, previous configuration kept; hot reload still armed (ERROR) | to report | R-RELOAD-013 |
Connection and authentication
reason of the connection_rejected lines, and what the client sees. The HTTP
doors answer an error in application/problem+json, field detail.
reason, code or detail | Where it is set | Rule |
|---|---|---|
absent, 401 invalid credentials (the file explorer: Invalid credentials; 429: Too many attempts); at DEBUG, local password authentication rejected (or public key), reason no such user in the users file, password does not match the stored hash, the stored hash could not be parsed, no local user store configured, user has no authorities | Local accounts ([[users]]): wrong password or unknown name | R-AUDIT-022, R-AUTH-006, R-EXPLORER-006 |
no authorized key offered | The public key ([[users]]): authorized_keys | R-AUTH-016 |
signature algorithm not allowed | Signature algorithms of a key ([[roles]], [sftp.algorithms]) | R-AUTH-018 |
missing credential, empty credential, malformed credential; 401 missing or invalid authorization header, invalid Basic auth, JWT not configured, local auth not configured | REST file API, Choosing the methods ([auth.methods]): send Basic or Bearer | R-REST-003 |
invalid token, expired token, wrong issuer, wrong audience, 401 invalid JWT | JWT ([auth.jwt]): issuer, audience, key | R-AUTH-022 |
verifier unavailable; ERROR JWKS background refresh failed ... (the cache keeps its keys) | JWT, Connecting an identity provider ([auth.jwt]) | R-AUTH-026, R-AUTH-027 |
no username claim, 401 token carries no username claim | JWT ([auth.jwt]): username_path | R-AUTH-024 |
method disabled, 401 authentication method disabled | Choosing the methods ([auth.methods]) | R-AUTH-003 |
no matching roles, 403 no roles resolved; role resolution, 503 roles could not be resolved, retry later; WARN authz service returned non-200, authz service call failed | Where roles come from, The authorization service contract ([[users]], [auth.jwt], [auth]): authorities, authorities_path, authz_base_url | R-AUTH-029 |
mount conflict, 503; backend initialization failed, 500; SFTP disconnection server configuration error: <cause>; see the server log | Combining roles, Types ([[roles.mounts]], [[backends]]): the application log gives the cause | R-AUTH-030, R-AUTH-031 |
username not usable as home directory | {username} ([[roles.mounts]]) | R-AUTH-032 |
session limit, session rejected: session limit exceeded for user <name> | The keys of [sftp] ([server]): max_sessions_per_user | R-SFTP-011 |
banned, 403 IP temporarily banned; SFTP: disconnection address banned, WARN address banned while this login was in flight — ... | The life of a ban, Lifting a ban ([sftp.ban], [api.ban], [admin.ban]) | R-BAN-008, R-BAN-008 |
rate limit, 429 rate limit exceeded | Rate limits ([sftp.rate_limit], [api.rate_limit]) | R-BAN-022 |
password checks saturated, 503 password checks saturated, retry later | Checking passwords ([auth]) | R-AUTH-011 |
password checks saturated for address | Checking passwords ([auth], [api.ban]): NAT, trusted_proxies | R-AUTH-012 |
shutting down | Shutdown | R-AUTH-015 |
invalid ticket, expired ticket, revoked ticket, 403 invalid download ticket, download ticket expired, download ticket revoked; 400 a download ticket is only redeemed by GET, ... is asked for with POST ?ticket, ... and an Authorization header are exclusive, ... only downloads a file, ?ticket and ?rename are exclusive | Downloading: the ticket, REST file API: ask for a ticket again | R-REST-009, R-REST-010 |
429 too many live download tickets | REST file API: 32 live tickets per user, wait for the oldest to expire | R-REST-009 |
range not satisfiable, 416 | REST file API: the Range starts after the end of the file, the client already has all of it | R-REST-006 |
login grace time exceeded | SSH: [sftp] ([sftp]): login_grace_secs | R-TIMEOUT-001 |
WARN failed to accept SFTP connections; retrying with backoff (descriptors exhausted, ulimit -n) | SFTP door | R-SFTP-002 |
SSH session ended: the peer offered no algorithm in common | An old client ([sftp.algorithms]) | R-SFTP-010 |
SSH request refused: ... (shell, exec, port forwarding); SSH_MSG_CHANNEL_FAILURE on a second sftp subsystem | only the sftp subsystem is served, once per connection: Doors | R-SFTP-012, R-SFTP-013 |
SSH_FX_FAILURE internal server error: this SFTP session is closed; ERROR a thread panicked: a defect in the server, ... (panic_payload, location, thread, backtrace with RUST_BACKTRACE=1) | a verb panicked; session_end reason=internal_error: to report | R-SFTP-019 |
File operations
reason of the operation lines, SFTP status and REST code.
reason | SFTP | REST | Where it is set | Rule |
|---|---|---|---|---|
acl | SSH_FX_PERMISSION_DENIED | 403 | Which entry decides ([[roles.mounts.acl]]) | R-ACL-003 |
acl subtree | SSH_FX_PERMISSION_DENIED | 403 | Rights ([[roles.mounts.acl]]): delete on the whole tree | R-DELETE-005 |
synthetic path | SSH_FX_PERMISSION_DENIED | 403 | Synthetic directories | R-ACL-005 |
rename across mounts | SSH_FX_OP_UNSUPPORTED | 422 | One mount or several | R-RENAME-010 |
invalid path | SSH_FX_PERMISSION_DENIED | 400 | control character, \, :, trailing dot, 8.3 short name: Client paths | R-UPLOAD-001 |
reserved name | SSH_FX_PERMISSION_DENIED | 403 | Two uploads to the same destination | R-RESERVE-010 |
rename into restricted | SSH_FX_PERMISSION_DENIED | 403 | Rights ([[roles.mounts.acl]]): write at the destination | R-RENAME-008 |
exists | SSH_FX_FAILURE | 409, 412 | existing destination | R-RENAME-002 |
is a directory, not a directory | SSH_FX_FAILURE | 409 | rm of a directory, rmdir of a file | R-DELETE-002 |
upload in progress; recursive delete refused: an upload is in progress beneath (WARN) | SSH_FX_PERMISSION_DENIED (curl: Permission denied (3)) | 409 | Two uploads to the same destination | R-RESERVE-001, R-RESERVE-012 |
quota exceeded | SSH_FX_FAILURE | 507 | Capping the size of a file ([[roles.mounts]]): max_file_mb | R-UPLOAD-007 |
session killed | SSH_FX_CONNECTION_LOST | - | session cut: Sessions | R-SFTP-020 |
unsupported | SSH_FX_OP_UNSUPPORTED | 501 | Atomic writes, Files and directories on S3 ([server.hidden_stores]): resume or append, send the whole file again | R-UPLOAD-009 |
upload in progress at CLOSE, client: upload not published: another upload took this file: <path> | SSH_FX_PERMISSION_DENIED | 409 | this upload’s lock was taken over by another instance before publication (refresh blocked): nothing is published, send the file again; Reservation across instances | R-RESERVE-008 |
taken over | SSH_FX_FAILURE on the old handle | 409 to the old upload | a stuck upload (suspended client) taken over by a retry from the same account: Two uploads to the same destination ([uploads]): takeover_idle_secs | R-RESERVE-017 |
upload idle timeout, upload below minimum rate | SSH_FX_FAILURE | 408 | Uploads: [uploads] ([uploads]) | R-UPLOAD-013 |
session ended: admin_kick | - | 503 the upload was cut by an administrator: nothing was written | a REST transfer cut from the console: Sessions | R-REST-013 |
session ended: <cause>, session ended | - | - | the client left; <cause> is the reason of the session_end | R-AUDIT-018 |
commit interrupted (result=unknown) | - | - | client left during publication: check the file | R-AUDIT-017 |
no roles | - | 403 | see no matching roles above | R-AUDIT-020 |
| - | SSH_FX_OP_UNSUPPORTED | - | READLINK, SYMLINK, posix-rename@openssh.com: not served | R-LIST-012 |
| - | - | 400 missing ?rename= query param; 405 (verb not served) | REST file API | R-REST-002 |
| - | SSH_FX_FAILURE | - | unknown handle or one of another kind: client bug | R-SFTP-017 |
Storage
A storage error carries its kind in reason (result=error); its
text is in the application log, at the same time.
reason, message | SFTP | REST | Where it is set | Rule |
|---|---|---|---|---|
not found | SSH_FX_NO_SUCH_FILE | 404 | Common keys on a local backend ([[roles.mounts]]): missing path; home_dir missing without create_home; on S3, a directory cannot be renamed | R-UPLOAD-015, R-S3-004 |
permission denied | SSH_FX_PERMISSION_DENIED | 403 | storage permissions; S3 without s3:ListBucket: Bucket permissions; WebHDFS (AuthorizationException, doAs not allowed): Prerequisites | R-UPLOAD-015, R-WEBHDFS-005 |
symlink escape | SSH_FX_PERMISSION_DENIED | 403 | The root and symbolic links ([[backends]] local): metric craftfilegate_local_symlink_refusals_total | R-LOCAL-004 |
already exists, directory not empty | SSH_FX_FAILURE | 409 | state of the storage | R-UPLOAD-015 |
storage error | SSH_FX_FAILURE backend error | 500 | the application log gives the cause | R-UPLOAD-015 |
not implemented | SSH_FX_OP_UNSUPPORTED | 501 | an operation this backend does not have: Types | R-UPLOAD-015 |
root <path> cannot be canonicalized and opened as a directory: <error> | - | - | The root and symbolic links ([[backends]] local) | R-LOCAL-003 |
no host_key_fingerprint: the upstream's host key would not be checked, ...; SFTP proxy: the upstream's host key does not match host_key_fingerprint (ERROR) | - | - | The upstream’s host key ([[backends]] sftp): ssh-keyscan -p <port> <host> | ssh-keygen -lf - | R-PROXY-002, R-PROXY-004 |
SFTP proxy: authentication failed: the upstream accepts only ssh-rsa (SHA-1) ... | - | - | Keys ([[backends]] sftp): ed25519 or ECDSA key | R-PROXY-005 |
WARN this store refused the conditional CopyObject with a 400 this code does not treat as "precondition not implemented": ... (every rename fails); could not list the multipart uploads under this backend's prefix, ..., listing the parts of a multipart upload was refused, ..., could not read the S3 service's clock (no usable Date header on its response) and ..., this S3 backend has no prefix, so its abandoned multipart uploads are never swept: ... | - | - | Bucket permissions, Multipart uploads, Conditional writes | R-S3-010, R-S3-013 |
SFTP proxy connect: ...; WARN accept_any_host_key = true on this SFTP proxy backend: ..., sftp proxy: the upstream cannot replace a file atomically (posix-rename@openssh.com), ..., sftp proxy: the second SFTP channel used for posix-rename@openssh.com could not be opened (...), ... | - | - | The upstream’s host key, Uploads | R-PROXY-001, R-PROXY-003, R-PROXY-007 |
ERROR sftp proxy: the destination was removed to publish an upload and the rename ... which is kept, sftp proxy: an upload was interrupted after the removal of its destination ... which is kept (fields temp, destination) | - | - | Uploads: the in-flight file is the only copy, rename it by hand | R-PROXY-008 |
the conditional writes are not proven on this S3 store: <why> (WARN), refusal with cross_instance_reservation; INFO S3 upload precondition self-test verdict = ignored, not implemented, if-match ignored, delete refused, inconclusive | - | - | Conditional writes ([[backends]]): cross_instance_reservation | R-S3-007, R-RESERVE-015 |
the SFTP upstream refuses the name of the upload lock files, ..., could not refresh the lock file of an upload in progress: ..., this upload's lock was taken over by another instance: it is not published (WARN) | - | - | Reservation across instances ([[backends]]): lock_prefix | R-RESERVE-005, R-RESERVE-007, R-RESERVE-008 |
backend "<name>" (webhdfs): Knox refused the service account on GETFILESTATUS of the root ...; session refused: a username that cannot be a doAs; WARN this WebHDFS backend's url is plain http ..., this WebHDFS backend's gateway could not be checked at startup ..., this WebHDFS answers no LISTSTATUS_BATCH ... | - | - | Prerequisites, The mount on WebHDFS ([[backends]] webhdfs): auth, url, ca_bundle | R-WEBHDFS-004, R-WEBHDFS-005, R-WEBHDFS-001, R-WEBHDFS-010 |
case_insensitive = true on a WebHDFS backend: ... | - | - | The mount on WebHDFS ([[backends]]) | R-WEBHDFS-014 |
the storage's clock differs from this server's by more than 30 s (WARN); stale in-flight files on this SFTP upstream are never collected (age_check = false): ..., stale in-flight files of this backend are aged on this server's clock (age_check = false): ..., abandoned multipart uploads of this S3 backend are aged on this server's clock (age_check = false): ..., could not read the storage's clock with a probe file in this directory (age_check is on), ..., this storage offers no lock to tell an in-flight upload from the leftovers of one killed with the server ... | - | - | Leftovers of an interrupted transfer ([uploads.stale_partials]): NTP of the storage | R-HIDDEN-016, R-HIDDEN-011, R-HIDDEN-013 |
this filesystem does not support RENAME_NOREPLACE ... (WARN) | - | - | Performance and limits: NFS, FUSE | R-LOCAL-011 |
backend unavailable: its storage did not answer the availability probe (WARN, backend, backend_type, failures, reason); backend available again: ... (WARN, down_secs) | - | - | Availability ([server.backend_probe]): reason says what the visit found (missing root, bucket refused, upstream unreachable, different host key, timeout) | R-AVAIL-002 |
reason SFTP proxy: authentication failed — probe paused until reload | - | - | SFTP proxy: the service account or its password; the probe reconnects only at the next reload of the roles, so the upstream does not ban the gateway | R-AVAIL-001 |
Admin console and API
Code and detail | Where it is set | Rule |
|---|---|---|
401 missing or invalid authorization header, invalid token | Authenticating ([admin], [auth.jwt]) | R-ADMIN-005 |
401 invalid credentials on POST /admin/login (counts for [admin.ban]; an account without an admin role: admin sign-in refused: the password is right, ... at WARN), 503 password checks saturated, retry later, 400 expected a JSON body {"username": ..., "password": ...}, 415 expected Content-Type: application/json (a client that does not send Content-Type: application/json), 403 sign-in from another site refused (Sec-Fetch-Site other than same-origin or none: a page from another site; neither counts for the ban), 400 username longer than 256 bytes; 401 revoked token (account removed, password changed; or revoked, see below), 401 session too old: sign in again, 400 only a session token is renewed: ... | Signing in ([admin.session], users_file): sign in again | R-ADMIN-022 |
404 password sign-in is not available on this door on POST /admin/login; [admin.session] sets a session key or a revocation store, but no account signs in to the console: ... (WARN); console: only the token field, no password form (GET /admin/login says {"password": false}) | Console sessions ([auth.methods], [[admin.roles]]): local passwords and admin roles | R-ADMIN-022, R-ADMIN-018 |
401 revoked token after POST /admin/revocations or POST /admin/logout: sign in again; lift local_only and WARN the session tokens were revoked on this instance only: ...; admin session revocations are kept in memory beside a shared session key: ..., [admin.session] sets revocation keys this store does not read, shared admin session revocation records refused ..., ... ahead of this clock ... (WARN); ERROR the shared admin session revocations cannot be read: ...; refusal to start ... the revocations cannot be read ..., admin.session.backend = ..., admin.session.reread_interval_secs must be between 1 and 30 ..., admin.session.revocation_configmap_name is empty ..., admin.session.persist_file <path> is also the persist_file of a ban list ...; 400 expected a JSON body {"username": ...}, username blank or longer than 256 bytes, 404 this door issues no session token: there is nothing to revoke | Revoking, signing out, Sharing revocations ([admin.session]): persist_file or backend, NTP | R-ADMIN-023 |
POST /admin/grants: 400 unknown role, until in the past, invalid period, duration over max; 403 role not grantable; 409 role already held, mount conflict, already granted, too many grants; 403 the grant permission is required; 404 this door grants no temporary access (listener without [admin]); DELETE: 404 grant not found, 409 grant ended, grant ended: it is already <state>; lift local_only; refusal to start admin.grants.* ..., [cluster] is set and the temporary access grants are kept in memory: ..., ... the grants cannot be read ...; WARN [admin.grants] is set, but no credential holds the grant permission: ...; SFTP session cut temporary access expired / temporary access revoked, REST upload 503 the upload was cut: the temporary access it used ended; nothing was written; WARN a temporary access grant names a role this server does not define: ignored, ... names an admin role: ignored; ERROR the shared temporary access grants cannot be read: ... | Reference [admin.grants]: max_duration_secs, persist_file, backend | R-GRANT-002, R-GRANT-006, R-GRANT-003, R-GRANT-005, R-GRANT-009, R-GRANT-010 |
console: Wrong username or password. (a wrong password, an unknown name or an account without an admin role: a single message); Your session was revoked: sign in again., Your session expired: sign in again., Your session reached its maximum duration: sign in again., Token rejected: sign in again.; the page asks to sign in again after a reload or in a new tab | In the console ([admin.session]): sign in again; the token lives only in the page | R-ADMIN-018 |
blank or unstyled console behind a reverse proxy, Refused to ... / violates the following Content Security Policy directive in the browser console | The console: the proxy’s CSP allows at least the page’s | R-ADMIN-018 |
403 no matching admin roles; 403 the <permission> permission is required | Authenticating ([[admin.roles]]) | R-ADMIN-006, R-ADMIN-007 |
404 session not found: <id>, 400 invalid session ID format | Sessions | R-ADMIN-010 |
404 no log files: [log] dir is not set, 404 unknown log source: expected app or audit, 500 the log file could not be read, 403 the log file is a symbolic link, which the log viewer does not follow; 400 lines: not a number, level: unknown level, q: longer than 256 bytes, field: ...; 429 too many log reads at once: retry in a moment | Log files, Logs and streams ([log]) | R-ADMIN-013 |
429 too many admin streams open: close one or retry later | Logs and streams | R-ADMIN-014 |
unban 409 lift=overruled (a ban later than the request applies), 200 lift=local_only (shared state not written), 404 not banned, no ban manager, 400 unknown protocol (sftp, api or admin) | Lifting a ban, Sharing bans between instances ([sftp.ban], [api.ban], [admin.ban]) | R-BAN-011 |
401 (404 with the file explorer, [api.ui]) on /metrics, /admin, /health, /livez or /readyz called on admin.listen; 404 on <prefix>: [api] enabled missing, or request on control_listen | One door or two ([admin], [api]): with control_listen or probes_listen, each route has its port | R-METRICS-001, R-REST-001 |
404 or 401 on /api/docs, /api/openapi.json | REST file API ([api]): openapi = true | R-REST-012 |
port unreachable from outside a container, connection refused; container unhealthy (craft-file-gate healthcheck failing) | Docker, The binary’s tools ([admin]): listen = "0.0.0.0:...", not 127.0.0.1; the address that healthcheck queries | R-ADMIN-001, R-ADMIN-015 |
the admin door shares its listener with the file API ... (WARN) | One door or two ([admin]): set control_listen | R-ADMIN-002 |
admin TLS certificate expires soon, has expired, admin TLS certificate expiry could not be read | What you will see ([admin.tls]): renew; an expired certificate is served anyway | R-ADMIN-004 |
Kubernetes
| Symptom or message | Where it is set | Rule |
|---|---|---|
probes failing, connection refused | Kubernetes ([sftp], [admin]): listen = "0.0.0.0:..."; probes blocked by the NetworkPolicy depending on the CNI: the node CIDR in networkPolicy.controlFrom | R-ADMIN-001 |
/readyz 503, data_runtime=unresponsive (runtime saturated) or sftp=not_accepting (SFTP port closed, shutdown in progress) | Probes | R-K8S-002 |
chart rendering fails: networkPolicy.controlFrom is empty ..., config.existingConfigMap and config.inline are both set, no configuration: ..., adminRevocations.backend = ...: expected memory, file, configmap or empty, replicaCount > 1 with clusterSecret.enabled = false ..., replicaCount > 1 with a shared console session key and adminRevocations kept in memory ..., cluster is on (replicaCount > 1, or cluster.enabled) and clusterSecret.enabled is false ..., cluster.unreadyWhen names <value>: ... | The chart, The pods’ shared Secret | R-K8S-005 |
backend = "configmap" refused: ConfigMap unreadable for 30 s (message naming <namespace>/<name> and the Role); WARN at each re-read of the bans, propagation in 30 s (watch right missing) | Sharing through a ConfigMap ([sftp.ban], [api.ban], [admin.ban]): Role and RoleBinding | R-BAN-017 |
cannot read or fill the entry admin-session.key of Secret <ns>/<name> ..., shared Secret not readable or writable yet; ... (WARN) | The pods’ shared Secret ([admin.session]): the Secret created by the chart, the Role (get, update on that name) | R-ADMIN-022 |
cannot read the admin session revocations in ConfigMap <ns>/<name> ..., admin session revocations not readable yet; ... (WARN) | Sharing revocations: the chart’s ConfigMap and Role (adminRevocations.backend: configmap, get, update, patch on that name) | R-ADMIN-023 |
cannot read or fill the entry tls.key of Secret <ns>/<name> ..., the entries tls.crt and tls.key of Secret <name> do not go together ..., ... the key does not belong to the certificate; WARN cluster peers name does not resolve: its last addresses are kept; WARN cluster peer unreachable (only from a peer that has already answered; before that, at DEBUG), cluster peer presents another certificate than this instance ..., cluster state cut to fit: ..., cluster peer state holds absurd records ..., cluster peer's failure series are all past their window ... (clocks to synchronize, NTP) | The shared certificate: the chart’s Secret and Role, a rotation in progress; the cluster port in the NetworkPolicy | R-CLUSTER-002, R-CLUSTER-006, R-CLUSTER-005, R-CLUSTER-003 |
Instances tab, peers of a scope=cluster list, relayed kick or lift: unauthorized on <instance>, forbidden on <instance>; WARN audit: relayed credential refused ... on the peer; 400 unknown scope: ... (scope other than local or cluster) | Cluster: the same static token, the same session key, the same [auth.jwt] and the same [[admin.roles]] everywhere | R-CLUSTER-009, R-CLUSTER-010, R-CLUSTER-008 |
backend = "configmap" refused by a binary without the k8s feature | The binary’s features | R-BAN-018 |
failed to create the file watcher, falling back to re-reading ... | inotify on a shared node ([reload]) | R-RELOAD-010 |
pod OOMKilled during a burst of connections | Memory, Checking passwords ([auth]) | R-AUTH-014 |
“host key changed” from one pod to another; WARN host key not found, generated a new one (sftp.generate_host_key) | SSH host key ([sftp]) | R-SFTP-004 |
Logs, metrics and telemetry
| Message | Where it is set | Rule |
|---|---|---|
craft-file-gate: <n> log line(s) could not be written ... (stderr) | An incomplete trail | R-AUDIT-031 |
audit target disabled by RUST_LOG, audit successes disabled by RUST_LOG | RUST_LOG (Environment variables): RUST_LOG=warn,audit=info | R-AUDIT-002 |
process resource sampling failed; the process_* series are absent from /metrics | Metrics | R-METRICS-010 |
OTLP collector unreachable — telemetry spans will be dropped until recovery | Exports, retries and shutdown ([telemetry]) | R-TELEMETRY-010 |
failed to create the OTLP <signal> exporter, ... ([telemetry] on_exporter_error = "warn") (ERROR) | [telemetry] ([telemetry]) | R-TELEMETRY-003 |
sessions were still ending when the grace period ran out: ... | Tuning the orchestrator ([server]): shutdown_grace_period_secs | R-SHUTDOWN-007 |
ban file was still being written when the shutdown stopped waiting; the shutdown stopped waiting for the removal of upload lock files; ... | Shutdown | R-BAN-019, R-SHUTDOWN-008 |