Metrics

curl http://serveur:8081/metrics          # Prometheus text, no credential

# prometheus.yml
scrape_configs:
  - job_name: craftfilegate
    static_configs:
      - targets: ['serveur:8081']   # probes_listen, else control_listen, else admin.listen

Output	Format	Access
`GET /metrics`	Prometheus text 0.0.4	public; on `[server] probes_listen` or `control_listen` only, when they are set
`GET /admin/resources`	the same snapshot in JSON, plus sessions, active bans, backends, thresholds	permission `overview`
OTLP export	the same numbers, one snapshot per interval	`[telemetry] metrics = true`, see Telemetry

Values are cumulative since startup, per pod. Compute a rate between two reads (rate()).
An unknown value is absent (null in JSON, no data point in OTLP), never 0: the “Absent when” column says when.
The OTLP name is the Prometheus name without _total.

Catalog

Prometheus series	OTLP instrument	Type	Labels	Absent when	Meaning
`craftfilegate_connections_total`	`craftfilegate_connections`	counter			SSH sessions opened after a successful authentication
`craftfilegate_connections_rejected_total`	`craftfilegate_connections_rejected`	counter			refusals at the SFTP door, one per attempt (`connection_rejected` lines with `source=sftp`, summarized ones included)
`craftfilegate_sftp_operations_total`	`craftfilegate_sftp_operations`	counter	`op` (21 values)		SFTP packets received, refusals included
`craftfilegate_sftp_bytes_read_total`	`craftfilegate_sftp_bytes_read`	counter			bytes served over SFTP
`craftfilegate_sftp_bytes_written_total`	`craftfilegate_sftp_bytes_written`	counter			bytes accepted over SFTP
`craftfilegate_api_requests_total`	`craftfilegate_api_requests`	counter			requests received by the file API, `429` and refusals included
`craftfilegate_banned_ips_total`	`craftfilegate_banned_ips`	counter			ban verdicts of this pod, all doors
`craftfilegate_sftp_accept_errors_total`	`craftfilegate_sftp_accept_errors`	counter			failed `accept()` on the SFTP port
`craftfilegate_log_write_errors_total`	`craftfilegate_log_write_errors`	counter			log or audit lines not written
`craftfilegate_audit_refusals_suppressed_total`	`craftfilegate_audit_refusals_suppressed`	counter	`source` (`sftp`, `api`, `admin`)		anonymous refusals summarized instead of written
`craftfilegate_audit_refusal_summary_overflow_total`	`craftfilegate_audit_refusal_summary_overflow`	counter			refusals that arrived with the summary table full
`craftfilegate_otel_spans_ended_total`	`craftfilegate_otel_spans_ended`	counter		telemetry off	spans ended
`craftfilegate_otel_spans_exported_total`	`craftfilegate_otel_spans_exported`	counter		telemetry off	spans accepted by the collector
`craftfilegate_otel_spans_export_failed_total`	`craftfilegate_otel_spans_export_failed`	counter		telemetry off	spans lost
`craftfilegate_admin_tls_cert_not_after_seconds`	`craftfilegate_admin_tls_cert_not_after_seconds`	gauge		no TLS, unreadable date	expiry of the served certificate, Unix seconds
`craftfilegate_admin_tls_cert_expiry_unreadable`	`craftfilegate_admin_tls_cert_expiry_unreadable`	gauge		no TLS	`1` unreadable date, `0` read
`craftfilegate_sftp_rate_limit_tracked_addresses`	`craftfilegate_sftp_rate_limit_tracked_addresses`	gauge		no `[sftp.rate_limit]`, or before its first sweep	addresses tracked by the limiter
`craftfilegate_api_rate_limit_tracked_addresses`	`craftfilegate_api_rate_limit_tracked_addresses`	gauge		no `[api.rate_limit]`, or before its first sweep	same for the API
`craftfilegate_local_users`	`craftfilegate_local_users`	gauge	`hash_format`	no local store	local accounts per hash format
`craftfilegate_reload_watch_mode`	`craftfilegate_reload_watch_mode`	gauge	`mode` (`inotify`, `poll`, `sighup`)	before arming	`1` for the active mode
`craftfilegate_jwks_cache_age_seconds`	`craftfilegate_jwks_cache_age_seconds`	gauge		no JWKS, no successful load	age of the JWKS cache
`craftfilegate_jwks_refresh_failures_total`	`craftfilegate_jwks_refresh_failures`	counter		no JWKS	failed refreshes
`craftfilegate_jwks_cached_keys`	`craftfilegate_jwks_cached_keys`	gauge		no JWKS	keys in the cache
`craftfilegate_grants_active`	`craftfilegate_grants_active`	gauge		no `[admin]`	temporary accesses that grant their role now
`craftfilegate_grant_sessions_cut_total`	`craftfilegate_grant_sessions_cut`	counter	`reason` (`expired`, `revoked`)	no `[admin]`	SFTP sessions and REST transfers cut at the end of a temporary access they used
`craftfilegate_grants_store_local_only`	`craftfilegate_grants_store_local_only`	gauge		no `[admin]`	`1`: temporary accesses held in memory while a cluster peer answers
`craftfilegate_cluster_peers`	`craftfilegate_cluster_peers`	gauge	`status`	no `[cluster]`	peers of the channel between instances by status: `reachable`, `unreachable`, `certificate_mismatch`
`craftfilegate_cluster_relay_total`	`craftfilegate_cluster_relay`	counter	`route` (`health`, `sessions`, `bans`, `kick`, `unban`), `result` (`answered`, `refused`, `failed`)	no `[cluster]`	operator requests relayed to a peer
`craftfilegate_cluster_cert_not_after_seconds`	`craftfilegate_cluster_cert_not_after_seconds`	gauge		no `[cluster]`, unreadable date	expiry of the shared certificate, Unix seconds; never checked
`craftfilegate_cluster_reachable_peers`	`craftfilegate_cluster_reachable_peers`	gauge		no `[cluster]`	peers reachable in the last round
`craftfilegate_cluster_isolated`	`craftfilegate_cluster_isolated`	gauge		no `[cluster]`	`1`: fewer than `min_peers` peers reachable for `isolated_after_secs` (detectors)
`craftfilegate_unready_detector`	`craftfilegate_unready_detector`	gauge	`detector` (`storage_alone`, `isolated`, `isolated_and_storage_down`)	no `[cluster]`	`1` while the detector is active
`craftfilegate_withdrawn`	`craftfilegate_withdrawn`	gauge		no `[cluster]`	`1`: `/readyz` not ready because of a detector in `unready_when`
`craftfilegate_stale_partials_clock_skew_seconds`	`craftfilegate_stale_partials_clock_skew_seconds`	gauge	`backend`	nothing measured	storage clock minus server clock
`craftfilegate_password_hash_pool_places`	`craftfilegate_password_hash_pool_places`	gauge		no local store	threads plus queue slots of the hashing pool
`craftfilegate_password_hash_pool_in_use`	`craftfilegate_password_hash_pool_in_use`	gauge		no local store	slots taken
`craftfilegate_password_hash_pool_full_total`	`craftfilegate_password_hash_pool_full`	counter		no local store	verifications refused because the pool was full
`craftfilegate_jwt_refused_total`	`craftfilegate_jwt_refused`	counter		no JWT	tokens refused by the verifier
`craftfilegate_backend_operations_total`	`craftfilegate_backend_operations`	counter	`backend`	backend unused	calls to the storage
`craftfilegate_backend_errors_total`	`craftfilegate_backend_errors`	counter	`backend`	backend unused	storage failures (I/O, unreachable, credentials refused); not client refusals
`craftfilegate_backend_bytes_read_total`	`craftfilegate_backend_bytes_read`	counter	`backend`	backend unused	bytes read from the storage
`craftfilegate_backend_bytes_written_total`	`craftfilegate_backend_bytes_written`	counter	`backend`	backend unused	bytes written to the storage
`craftfilegate_backend_up`	`craftfilegate_backend_up`	gauge	`backend`	before the first visit, probe off	`1` the storage answers the probe, `0` it no longer answers
`craftfilegate_backend_probe_seconds`	`craftfilegate_backend_probe_seconds`	gauge	`backend`	same	duration of the last visit
`craftfilegate_local_symlink_refusals_total`	`craftfilegate_local_symlink_refusals`	counter	`backend`	backend not local or unused	paths refused with `symlink escape`
`process_resident_memory_bytes`	`process_resident_memory_bytes`	gauge		outside Linux, `/proc` silent	resident memory
`process_peak_resident_memory_bytes`	`process_peak_resident_memory_bytes`	gauge		same	peak resident memory
`process_cpu_seconds_total`	`process_cpu_seconds`	counter		same	CPU time, user + system
`process_threads`	`process_threads`	gauge		same	threads

craftfilegate_connections_rejected_total counts per attempt, several per connection: its ratio to craftfilegate_connections_total is not a failure rate per connection.

Console thresholds

[admin.metrics_thresholds] only decides where the Metrics tab flags a tile; the server does not alert. The keys and their defaults: reference.

Alerts

groups:
  - name: craftfilegate
    rules:
      - alert: CraftFileGateAdminCertExpiringSoon
        expr: craftfilegate_admin_tls_cert_not_after_seconds - time() < 30 * 86400
        for: 1h
      - alert: CraftFileGateAdminCertExpiryUnknown
        expr: craftfilegate_admin_tls_cert_expiry_unreadable == 1
      - alert: CraftFileGateBackendErrors
        expr: increase(craftfilegate_backend_errors_total[5m]) > 0
      - alert: CraftFileGateBackendDown
        expr: craftfilegate_backend_up == 0
      - alert: CraftFileGateStorageAlone
        expr: craftfilegate_unready_detector{detector="storage_alone"} == 1
      - alert: CraftFileGateIsolated
        expr: craftfilegate_cluster_isolated == 1

Alerting on a stale JWKS cache

      - alert: CraftFileGateJwksStale
        expr: craftfilegate_jwks_cache_age_seconds > 2 * 3600   # 2 x jwks_refresh_interval_secs
        for: 5m
      - alert: CraftFileGateJwksNeverLoaded
        expr: absent(craftfilegate_jwks_cache_age_seconds) and on() craftfilegate_jwks_cached_keys >= 0
        for: 5m

A stale cache keeps its keys: tokens signed by a key rotated since then are refused. jwks_age_secs makes the same judgment for /admin/status.

Keyboard shortcuts

CraftFileGate

Metrics

Catalog

Console thresholds

Alerts

Alerting on a stale JWKS cache