WebHDFS (Knox)

The webhdfs backend reads HDFS, read-only, through the WebHDFS of Apache Knox: HTTPS, a service account over Basic, on behalf of the user (doAs). Knox carries Kerberos to the cluster. It also serves an HttpFS or a WebHDFS without Kerberos; Hadoop 2.8 and later.

[[backends]]
name = "datalake"
type = "webhdfs"
url = "https://knox.example.com:8443/gateway/default"
auth = { type = "basic", username = "svc-sftp", password_file = "/run/secrets/knox-password" }
ca_bundle = "/etc/craft-file-gate/knox-ca.pem"   # the private CA of Knox

[[roles]]
name = "analystes"

[[roles.mounts]]
backend = "datalake"
mount_path = "/datalake"
home_dir = "/data/projets"
acl = [{ path = "/", rights = ["read", "list"], recursive = true }]

alice reads /datalake/2026/x.csv: the request targets <url>/webhdfs/v1/data/projets/2026/x.csv?op=OPEN&doAs=alice.

The keys

All keys: Reference [[backends]] type = “webhdfs”.

The connection to Knox opens in 5 s at most; HTTPS_PROXY is not followed. Common keys: Backends.

The mount on WebHDFS

Key	On WebHDFS
`home_dir` (mount)	an HDFS directory; `.` and `..` are never resolved
`create_home` (mount)	no effect
`acl` (mount)	`read` and `list`: the rest has no effect

With doAs, the user name is made of A-Z a-z 0-9 . _ -, 64 characters at most.

hidden_stores, refuse_upload_over_directory, stale_partials, cross_instance_reservation and lock_prefix do not apply: nothing is written. case_insensitive: leave it out, HDFS compares byte by byte.

Prerequisites

The right to name a user (doAs), with impersonate = true. Who grants it depends on what is behind url:

Behind `url`	The setting
Knox (the intended case)	on the Knox side: in the topology, the identity assertion with impersonation enabled and `hadoop.proxyuser.svc-sftp.users`, `.groups`, `.hosts`; the topology exposes `WEBHDFS` and authenticates `svc-sftp` over Basic (`ShiroProvider` on LDAP/AD). On the cluster side, `hadoop.proxyuser.knox.*`, usually set by the Knox installation
Direct WebHDFS	`svc-sftp` declared as a proxy user in `core-site.xml` (below)
HttpFS	the same rights, under `httpfs.proxyuser.svc-sftp.` in `httpfs-site.xml`

<property>
  <name>hadoop.proxyuser.svc-sftp.hosts</name>
  <value>sftp-gateway.example.com</value>  <!-- where it connects from -->
</property>
<property>
  <name>hadoop.proxyuser.svc-sftp.groups</name>
  <value>sftp-users</value>                <!-- on whose behalf it acts -->
</property>

At startup, the server runs a GETFILESTATUS of the HDFS root as the service account, without doAs, 5 s at most.

What the backend does

Operation	WebHDFS request
`stat`, parent existence	`GETFILESTATUS`, one request per path
listing	`LISTSTATUS_BATCH`, page by page (`startAfter`); `LISTSTATUS` in one response on a cluster older than 2.8
download, resume, read at an offset	`OPEN?offset=&length=`, by ranges
upload, `mkdir`, rename, deletion	none: read-only

Read-ahead: 10 MiB read in 32 KiB SFTP packets cost 3 OPEN with the default; a read that skips only asks for its size.
Listings: read to the end, capped at 10,000 pages, 1,000,000 entries and 64 MiB per response.
Redirects: a 307 from Knox is followed once, only to the same origin (scheme, host, port) as url.
A SYMLINK entry is shown as a link, and cannot be read; HDFS compares names byte by byte, and so does the ACL.

Performance and limits

Knox is the bottleneck: each stat, page and range is an HTTPS request it relays to the cluster. The sessions of a backend share one HTTP client.
timeout_secs covers a whole range: on a slow link, raise it rather than lowering read_ahead_bytes.
Small files each cost one GETFILESTATUS and one OPEN: this backend is for browsing HDFS, not for everyday storage.
The body of a Knox error is never copied into a log: only the exception name and the code.