# Quick Start - [Usage](#usage) - [Configure access permissions](#configure-access-permissions) - [Mount as read-only](#mount-as-read-only) - [Enable logging](#enable-logging) - [Update](#update) - [Help](#help) - [Config file ↔ CLI parameter mapping](#config-file--cli-parameter-mapping) - [Option list](#option-list) - [Examples](#examples) - [Scenario-based tuning](#scenario-based-tuning) - [Performance-related parameters](#performance-related-parameters) - [Small file scenario](#small-file-scenario) - [High read throughput scenario](#high-read-throughput-scenario) - [Shared read cache](#shared-read-cache) - [Auto mount](#auto-mount) ## Usage - Mount ``` us3fs [global options] ``` > `` and `` must be the last two arguments in order; otherwise, other options may not take effect. - Unmount ``` umount ``` > On Windows, press `Ctrl+C` in `cmd`. ----- ### Configure access permissions By default, US3FS uses the current user’s permissions for the mount. If you want other users/groups to access the mountpoint, you can use: - `-o allow_other`: allow any user to access the filesystem - `--uid=xxx`: set default owner UID - `--gid=xxx`: set default owner GID - `--mp_mask`: mountpoint permission mask. Takes effect only when `allow_other` is enabled. Default `0000`. Usage is similar to `umask`. For example, to set mountpoint permissions to 770, use `-o allow_other --mp_mask 0007`. **Note: root ignores `mp_mask` set by non-root mounts.** You can get UID/GID via `id`. Example: ```bash // Mount as ubuntu user, set default user/group to www ubuntu:~$ id www uid=1001(www) gid=1001(www) groups=1001(www) ubuntu:~$ us3fs --uid=1001 --gid=1001 -o allow_other ``` `mp_mask` examples: | Mountpoint permission | mp_mask | | - | - | | rwxrwxrwx | 0000 | | rwxrwx--- | 0007 | | rwx------ | 0077 | | rwxr--r-- | 0033 | | rwxrwxr-- | 0003 | | rwxr-xr-x | 0022 | If you see: ```bash stderr: /bin/fusermount: option allow_other only allowed if 'user_allow_other' is set in /etc/fuse.conf ``` Add `user_allow_other` to `/etc/fuse.conf`. > On Windows, only `--uid` and `--gid` are supported. ### Mount as read-only Mount with `-o ro`. > Not supported on Windows. ### Enable logging - `--level=info/debug/error`: enable US3FS logs at a specific level - `--debug_fuse`: enable user-space fuse logs - CentOS: `/var/log/messages` - Ubuntu: `/var/log/syslog` - Mount with `-f` to run in the foreground and print logs to the screen. ### Update Run: ``` us3fs --update ``` The updated binary is placed under `/bin/`. > Not available on Windows. ----- ## Help Use `us3fs -h` to view supported options. #### Linux ```bash us3fs - a single posix file system based on us3 USAGE us3fs [global options] bucket mountpoint Version US3FS Version: v2.0.3 Commit ID: 96bd92a Build: 2024-02-27:17:02:18 Go Version: go1.17.13 linux/amd64 FUSE -o value Specify fuse/winfsp option --entry_timeout value How long to cache dentry for inode for fuse. (default: 5m0s) --attr_timeout value How long to cache inode attr for fuse (default: 5m0s) --disable_async_read Disable all read (even read-ahead) operations asynchronously --wb Enable writeback mode, which is turned off by default --max_background value Specify the max_background parameter of fuse kernel(>=7.13), currently fuse usespace supports up to 1024 (default: 64) --congestion_threshold value Specify the congestion_threshold parameter of fuse kernel(>=7.13), currently fuse usespace supports up to 768 (default: 48) --async_dio Enable the async_dio parameter of fuse kernel, async_dio is disabled by default --keep_pagecache Turn on pagecache, when the file is opened, it will be decided whether to update according to the modification time of the inode, so please pay attention to the attr_timeout and dcache_timeout parameters will have a certain impact on this OS --dcache_timeout value How long to cache dentry for us3fs (default: 5m0s) --retry value Number of times to retry a failed I/O (default: 5) --parallel value Number of parallel I/O thread (default: 32) --disable_remove Disable remove op, such as unlink, rmdir, rename --debug Set debug level for fuse/winfsp --level value Set log level: error/warn/info/debug (default: "info") --log_dir value Set log dir --log_max_age value Set log max age (default: 72h0m0s) --log_rotation_time value Set log rotation time (default: 1h0m0s) --enable_load_dentries enable auto init dentries in memory --cache_db value specify cache db path, e.g.: dbtype:dbpath --local_write write file to local and upload async --max_local_file_size value specify local file max size (default: "32m") --read_after_write_finish read operation will wait all write operation done --finish_write_when_release all written data will be uploaded when release --readahead value Readahead size. e.g.: 1m/1k/1 (default: "0") --max_cache_per_file value Max cache per file when enable readahead. e.g.: 32m/64m/128m (default: "1024m") --etag value Check etag for part. value is percent(0~100) (default: 50) --passwd value specify access file (default: "/etc/us3fs/us3fs.yaml") --enable_md5 Enalbe md5 in http header --uid value Specify default uid (default: 0) --gid value Specify default gid (default: 0) --mp_mask value Specify mountpoint mask (default: 0) --disable_check_vdir disable detection of virtual directories --update Update us3fs to /bin/us3fs -n Doesn't check access when mount us3fs -l Enable local cache for small file -p value Specify local cache location (default: "/tmp/us3fs/") --prefix value Specify bucket prefix path --open_rename Use rename api rather than copy and delete --gfl Enable get_file_list --direct_read Enable cache bypass read --perf_dump value How long to output the performance dump (default: 1h0m0s) --skip_ne_dir_lookup Skip non-essential directory checking, such as files ending in ".log",".png",".jpg", etc. --storage_class value Storage type, including "STANDARD", "IA" (default: "STANDARD") --enable_remote_cache --cache_dirs value specify cache db path, e.g.: /tmp/read-cache1,/tmp/read-cache2 --cache_size_limit value specify cache size limit, unit is GB (default: 4) --master_addr value master server addr --data_port value if data_port is specified, then other clients will connect it to get chunk data (default: 0) --page_size value (default: 1048576) --fuse_session_cnt value (default: 0) MISC --help, -h show help -f foreground ``` #### Windows ```shell us3fs - a single posix file system based on us3 USAGE us3fs [global options] bucket mountpoint Version US3FS Version: v1.6.8 Commit ID: c87ec9c Build: 2022-08-17:11:01:05 Go Version: go1.16.3 linux/amd64 WinFSP -o value Specify fuse/winfsp option --dir_info_timeout value The expiration time of the directory information, in seconds (default: 5) --file_info_timeout value File information expiration time, in seconds (default: 5) --volume_info_timeout value Volume information expiration time, in seconds (default: 5) --case_insensitive Is case sensitive --keep_filecache keep filecache OS --dcache_timeout value How long to cache dentry for us3fs (default: 5m0s) --retry value Number of times to retry a failed I/O (default: 5) --parallel value Number of parallel I/O thread (default: 32) --debug Set debug level for fuse/winfsp --level value Set log level: error/warn/info/debug (default: "info") --readahead value Readahead size. e.g.: 1m/1k/1 (default: "0") --etag value Check etag for part. value is percent(0~100) (default: 50) --passwd value specify access file (default: "/etc/us3fs/us3fs.conf") --enable_md5 Enalbe md5 in http header --uid value Specify default uid (default: -1) --gid value Specify default gid (default: -1) --disable_check_vdir disable detection of virtual directories --update Update us3fs to /bin/us3fs -n Doesn't check access when mount us3fs -l Enable local cache for small file -p value Specify local cache location (default: "/tmp/us3fs/") --prefix value Specify bucket prefix path --gfl Enable get_file_list --direct_read Enable cache bypass read --perf_dump value How long to output the performance dump (default: 1h0m0s) --skip_ne_dir_lookup Skip non-essential directory checking, such as files ending in ".log",".png",".jpg", etc. --storage_class value Storage type, including "STANDARD", "IA" (default: "STANDARD") MISC --help, -h show help -f foreground ``` > The differences between Linux and Windows options are mainly in `FUSE` vs `WinFsp`. ## Config file ↔ CLI parameter mapping Some mount parameters can be configured in the config file. Mapping between CLI parameters and config file fields: | CLI parameter | Config field | Example in config | | --- | --- | --- | | gfl | get_file_list | get_file_list: true | | keep_pagecache | keep_pagecache | keep_pagecache: true | | n | no_check | no_check: true | | disable_check_vdir | disable_check_vdir | disable_check_vdir: true | | async_dio | async_dio | async_dio: true | | skip_ne_dir_lookup | skip_ne_dir_look | skip_ne_dir_look: true | | l | enable_local | enable_local: true | | wb | writeback | writeback: true | | direct_read | direct_read | direct_read: true | | enable_md5 | enable_md5 | enable_md5: true | | debug | debug | debug: true | | -o allow_other | allow_other | allow_other: true | | enable_load_dentries | enable_load_dentries | enable_load_dentries: true | | disable_async_read | disable_async_read | disable_async_read: true | | retry | retry | retry: 66 | | parallel | parallel | parallel: 77 | | disable_remove | disable_remove | disable_remove: true | | congestion_threshold | congestion_threshold | congestion_threshold: 88 | | max_background | max_background | max_background: 99 | | uid | uid | uid: 100 | | gid | gid | gid: 100 | | etag | etag | etag: 100 | | dcache_timeout | dcache_timeout | dcache_timeout: 2h | | entry_timeout | entry_timeout | entry_timeout: 3h | | attr_timeout | attr_timeout | attr_timeout: 4h | | perf_dump | perf_dump | perf_dump: 5h | | log_max_age | log_max_age | log_max_age: 6h | | log_dir | log_dir | log_dir: /a/b/c | | prefix | prefix | prefix: /a/b/c | | level | level | level: debug | | storage_class | storage_class | storage_class: STANDARD | | p | local_path | local_path: /a/b/c | | readahead | readahead | readahead: 8m | | max_cache_per_file | max_cache_per_file | max_cache_per_file: 1024m | | cache_db | cache_db | cache_db: leveldb:/data/us3fs_cachedb | | local_write | local_write | local_write: true | | max_local_file_size | max_local_file_size | max_local_file_size: 32M | | finish_write_when_release | finish_write_when_release | finish_write_when_release: true | | read_after_write_finish | read_after_write_finish | read_after_write_finish: true | | enable_remote_cache | enable_remote_cache | enable_remote_cache: true | | cache_dirs | cache_dirs | cache_dirs: /mnt/nvme01,/mnt/nvme02 | | cache_size_limit | cache_size_limit | cache_size_limit: 100 | | master_addr | master_addr | master_addr: : | | data_port | data_port | data_port: 3333 | | page_size | page_size | page_size: 8388608 | | fuse_session_cnt | fuse_session_cnt | fuse_session_cnt: 4 | Example: configure mount parameters in config file Edit `/etc/us3fs/us3fs.yaml` (create directory if needed) and add parameters as required to simplify mount commands: ``` access_key: ************************************ secret_key: ************************************ endpoint: ufile.cn-north-02.ucloud.cn hosts: [] ``` ## Option list #### WinFsp | Option | Description | | --- | --- | | o | WinFsp `-o` option value | | dir_info_timeout | Directory info cache timeout (default 5s) | | file_info_timeout | File info cache timeout (default 5s) | | volume_info_timeout | Volume info cache timeout (default 5s) | | keep_filecache | Keep file cache | #### FUSE | Option | Description | | --- | --- | | o | FUSE `-o` option value | | entry_timeout | How long to cache looked-up names in FUSE
Default 5min | | attr_timeout | How long to cache file/dir attributes in FUSE
Default 5min | | disable_async_read | Disable async read-ahead in FUSE kernel (async by default) | | wb | Enable writeback mode. Overwrite/append are not supported | | max_background | FUSE kernel `max_background` (kernel>=7.13). fuse userspace supports up to 1024 (default 64). Improves direct I/O parallelism | | congestion_threshold | FUSE kernel `congestion_threshold` (kernel>=7.13). fuse userspace supports up to 768 (default 48). Triggers congestion control for parallel I/O | | async_dio | Enable async direct I/O in FUSE kernel (disabled by default) | | keep_pagecache | Keep VFS pagecache until inode modtime/size changes. Note: `entry_timeout` and `dcache_timeout` may delay observing changes | Common FUSE options (use with `-o`): | Option | Description | | --- | --- | | allow_other | Allow all users to access the filesystem (default off) | | ro | Mount as read-only | **Usage** ```bash -o option=value ``` #### OS (Object Storage) | Option | Description | | --- | --- | | dcache_timeout | Dentry cache TTL in US3FS
Default 5min | | retry | Retry count after request failure (default 5) | | parallel | Number of parallel I/O threads (default 20) | | debug | Set user-space fuse log level (default off) | | level | US3FS log level (default Info) | | readahead | Readahead size, e.g. 1m/1k/1 (default "0") | | etag | Percentage (0~100) of parts to check ETag for upload integrity (default 50) | | passwd | Credential file path (default `/etc/us3fs/us3fs.yaml`; Windows: custom) | | enable_md5 | Add MD5 check to HTTP headers (default off) | | uid | Default file owner user (default current user) | | gid | Default file owner group (default current group) | | disable_check_vdir | Disable virtual directory detection | | update | Update US3FS; new binary path `/bin/us3fs` | | n | Do not check bucket access on mount (useful if region has no list service) | | l | Enable local cache for small files and upload asynchronously (see [Small file scenario](#small-file-scenario)) | | p | Local cache directory for small-file async upload | | prefix | Bucket prefix to mount (default empty) | | gfl | For endpoints without ListObjects API, use PrefixFileList API to simulate listing | | direct_read | Bypass internal cache and read from US3 directly (may reduce performance for hot files; can reduce latency otherwise) | | perf_dump | Output latency statistics periodically (default 1 hour) | | skip_ne_dir_lookup | Skip non-essential directory checking for specific suffixes; ensure you don’t have directories ending with those suffixes | | storage_class | Storage class for objects written to US3: `STANDARD`, `IA` (default `STANDARD`) | | cache_db | Cache metadata on disk, e.g. `leveldb:/data/us3fs_cachedb` | | finish_write_when_release | Upload written data when releasing FD (supports multiple flushes per FD) | | read_after_write_finish | Reads wait until in-flight writes finish | | local_write | Write to local filesystem then upload asynchronously | | max_local_file_size | With `local_write`, max file size written locally (default 32M) | | enable_remote_cache | Enable remote cache | | cache_dirs | Local cache directories, comma-separated | | cache_size_limit | Cache size limit (GB) | | master_addr | Master server address | | data_port | Port for chunk data | | page_size | Page size | | fuse_session_cnt | FUSE session count | #### MISC | Option | Description | | --- | --- | | help, h | Show help | | f | Run in foreground and print output to stdout | > On Windows, `-f` has no effect. ## Examples - **entry_timeout**, **attr_timeout**, **dcache_timeout** Setting `dcache_timeout` increases how long file/dir attributes stay valid in memory, improving UX. We recommend setting `entry_timeout` and `attr_timeout` to be smaller than `dcache_timeout`. Note: After enabling caching, directory content may be temporarily inconsistent with the actual bucket state. This is enabled by default; set timeouts to `0s` to disable. Example: time to `ls` a directory containing 10,000 files ``` [root@10-9-120-211 ~]# us3fs --dcache_timeout=60s --entry_timeout=60s --attr_timeout=60s [root@10-9-120-211 ~]# time ls -la | wc -l 10003 real 0m5.964s user 0m0.033s sys 0m0.232s [root@10-9-120-211 ~]# [root@10-9-120-211 ~]# [root@10-9-120-211 ~]# time ls -la | wc -l 10003 real 0m0.872s user 0m0.029s sys 0m0.133s ``` - **disable_async_read** By default reads are asynchronous; synchronous reads perform worse. Example: ``` [root@10-9-120-211 ~]# us3fs --disable_async_read [root@10-9-120-211 ~]# dd if= of=/dev/null bs=4M count=10 10+0 records in 10+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 10.2345 s, 4.1 MB/s [root@10-9-120-211 ~]# us3fs [root@10-9-120-211 ~]# dd if= of=/dev/null bs=4M count=10 10+0 records in 10+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.685801 s, 61.2 MB/s ``` - **parallel** Increasing concurrency can improve read/write performance, but also increases system resource usage. Example: ``` // Default parallelism is 20 [root@10-9-120-211 ~]# us3fs [root@10-9-120-211 ~]# dd if=/dev/zero of= bs=4M count=1024 1024+0 records in 1024+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 25.5351 s, 168 MB/s // Set parallelism to 32 [root@10-9-120-211 ~]# us3fs --parallel=32 [root@10-9-120-211 ~]# dd if=/dev/zero of= bs=4M count=1024 1024+0 records in 1024+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 18.3614 s, 234 MB/s ``` - **readahead** Adjusting readahead has significant impact on sequential reads of large files. We recommend `16m~32m`, but it increases memory usage; reduce if needed. Example: ```shell // Default readahead size is 16MB [root@10-9-120-211 ~]# dd if= of=/dev/null bs=4M count=1024 1024+0 records in 1024+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 60.0498 s, 71.5 MB/s // Set readahead size to 32MB [root@10-9-120-211 ~]# us3fs --readahead=32m [root@10-9-120-211 ~]# dd if= of=/dev/null bs=4M count=1024 1024+0 records in 1024+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 37.6013 s, 114 MB/s ``` ## Scenario-based tuning ### Performance-related parameters - `parallel`: parallel threads; affects CPU load. Recommended 20~40. - `critical`: enable local ETag validation on write; can increase CPU usage by ~50%. - `readahead`: readahead window. Recommended 16m~32m; increases memory usage. - `keep_pagecache`: keep data in VFS pagecache until file modtime/size changes. - `wb`: merges writes in pagecache and sends larger I/O (default 128K). - `fuse_session_cnt`: number of connections to the kernel FUSE module; can improve performance but increases CPU/memory usage. ### Small file scenario For large volumes of small files, you can enable local cache with `-l`. When enabled, US3FS will first upload existing files <=4MB under the cache directory to the bucket, preserving paths. Writes <=4MB are written to local cache first and uploaded asynchronously; if local write fails, it falls back to synchronous writes to US3. Note: async uploads may fail on the backend; US3FS retries until success. ### High read throughput scenario - `cache_dirs`: local disks for read cache. Prefer local NVMe. You can set multiple disks separated by `,`. Avoid using the system disk. - `cache_size_limit`: cache size upper bound; LRU eviction triggers when reaching the limit. - `page_size`: when enabling local read cache, set `page_size` from `1048576` to `8388608`. ### Shared read cache To share read cache between multiple mountpoints, start a master node: ``` us3fs run-master --listen_addr 192.168.0.10:6667 ``` Then set other mountpoints’ `master_addr` to `192.168.0.10:6667`. The mountpoints will heartbeat to the master and form a small read-cache cluster. - `enable_remote_cache`: when enabled on a mountpoint, other mountpoints may connect to it to read cached data when available. - `data_port`: the port exposed for remote access when `enable_remote_cache` is enabled. ## Auto mount The recommended best practice is to use `systemctl` so US3FS remounts after abnormal exits or machine reboots. Ensure your Linux distribution supports `systemctl` (Windows is not supported currently): - Ubuntu >= 15.04 - CentOS/RHEL >= 7 After the system requirements are met, follow these steps: 1. Create the service unit Create `/etc/systemd/system/us3fs.service` with the following content: ``` [Unit] Description=US3FS (User Space FileSystem for US3) Requires=network-online.target After=network-online.target [Service] Type=forking User= Group= Restart=always RestartSec=10 ExecStart=/bin/us3fs --passwd=/etc/us3fs/us3fs.conf --keep_pagecache ExecStop=/bin/umount [Install] WantedBy=multi-user.target ``` - `User`: the user that needs to access the mountpoint (e.g., root) - `Group`: the group that needs to access the mountpoint (e.g., root) - `ExecStart`: the mount command; customize as needed Run `systemctl daemon-reload` to load the new unit. 2. Start/stop the service - Start: `systemctl start us3fs.service` - Stop: `systemctl stop us3fs.service` - Restart: `systemctl restart us3fs.service` - Status: `systemctl status us3fs.service` 3. Enable auto-start on boot Run `systemctl enable us3fs.service`.