Telemetry

An example

[telemetry]
enabled = true
otlp_endpoint = "http://otel-collector:4318"
protocol = "http"
service_name = "craft-file-gate"
metrics = true

The endpoint

`protocol`	`otlp_endpoint`	Spans sent to
`http`	`http://collector:4318`	`http://collector:4318/v1/traces` (and `/v1/metrics`)
`http`	`http://gw/otel/v1/traces`	as is
`grpc`	`http://collector:4317`	as is

https:// is encrypted in both protocols; the collector’s certificate is checked against the system roots (see TLS trust roots).
protocol is authoritative: OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, which some operators inject, do not change it.
The exporter reads OTEL_EXPORTER_OTLP_HEADERS (authentication headers) and OTEL_EXPORTER_OTLP_TIMEOUT (10 s by default).

What is exported

The export receives the spans and events of CraftFileGate and of the audit trail from info up, whatever the log level: a server set to warn still exports its spans.

Span	Attributes	Parent
`ssh_connection`	`peer`, `username`	-
`auth_password`, `auth_publickey`	`username`	`ssh_connection`
`sftp_session`	`session_id`, `username`	the connection
`sftp_operation`	`operation`, `session_id`, `username`, `path`, `bytes`	`sftp_session`
`api_request`	`method`, `path` (as received, percent-encoded), `username`	-
`api_operation`	`operation`, `path` (decoded), `username`	`api_request`
`authz_service_call`	`http.method`, `http.url`, `http.status_code`	the caller

The audit line of an operation is an event of its span; a failure sets the span to status=error.

For a Tempo or Jaeger panel:

a transfer that reaches close has two sftp_operation spans with the same operation (upload, download): the opening, then the commit, which alone carries bytes. Count the spans that carry bytes;
an rmdir of a non-empty directory has an rmdir span then an rmdir_recursive span, for a single rmdir_recursive audit line;
reads and writes have no per-packet span.

A path or a username that contains a control character or a Unicode separator (U+2028, bidirectional controls) arrives escaped, in a visible form (\u{2028}). The exact value is in the audit line.

Metrics over OTLP

With metrics = true, the server pushes its metrics every metrics_interval_secs, in addition to GET /metrics, which is still served; the name mapping: Metrics.

Sends, retries and shutdown

Moment	Behavior
transient refusal (`http`: `429`, `502`, `503`, `504`, no response; `grpc`: `UNAVAILABLE`, `DEADLINE_EXCEEDED`, …)	up to 4 attempts within the 10 s timeout; `Retry-After` honored
other refusal	a single attempt
collector unreachable	the batch is lost and counted, nothing stops; one line at the outage, one at the recovery
shutdown	last send of metrics and spans, 5 s at most

Lost spans are counted on /metrics:

Metric	Counts
`craftfilegate_otel_spans_ended_total`	spans handed to the export
`craftfilegate_otel_spans_exported_total`	spans accepted by the collector
`craftfilegate_otel_spans_export_failed_total`	spans of a batch lost after its last attempt

ended - exported - export_failed is what is still in flight (up to 2560 spans: the queue and the current batch), plus what a full queue rejected. A gap that grows beyond that means lost spans.

increase(craftfilegate_otel_spans_ended_total[5m])
  - increase(craftfilegate_otel_spans_exported_total[5m])

What you will see

When	Line
startup, export off	`INFO` `OpenTelemetry tracing disabled ([telemetry] absent or enabled = false); no spans are exported`
startup, export on	`INFO` `OpenTelemetry tracing enabled`, fields `endpoint` (completed for `http`), `protocol`, `service_name`
startup, metrics	`INFO` `OpenTelemetry metrics export enabled`, fields `endpoint`, `interval_secs`
collector back after an outage	`INFO` `OTLP collector recovered — telemetry export resumed`
shutdown	`INFO` `flushing OpenTelemetry spans`, then `OpenTelemetry shutdown complete`

Keyboard shortcuts

CraftFileGate