This is the 10th version of the series. Please find the changelog at the
bottom of this cover letter.
Add support for file system passthrough read/write of files when enabled in
userspace through the option FUSE_PASSTHROUGH.
There are file systems based on FUSE that are intended to enforce special
policies or trigger complicated decision makings at the file operations
level. Android, for example, uses FUSE to enforce fine-grained access
policies that also depend on the file contents.
Sometimes it happens that at open or create time a file is identified as
not requiring additional checks for consequent reads/writes, thus FUSE
would simply act as a passive bridge between the process accessing the FUSE
file system and the lower file system. Splicing and caching help reduce the
FUSE overhead, but there are still read/write operations forwarded to the
userspace FUSE daemon that could be avoided.
This series has been inspired by the original patches from Nikhilesh Reddy,
the idea and code of which has been elaborated and improved thanks to the
community support.
When the FUSE_PASSTHROUGH capability is enabled, the FUSE daemon may decide
while handling the open/create operations, if the given file can be
accessed in passthrough mode. This means that all the further read and
write operations would be forwarded by the kernel directly to the lower
file system using the VFS layer rather than to the FUSE daemon.
All the requests other than reads or writes are still handled by the
userspace FUSE daemon.
This allows for improved performance on reads and writes, especially in the
case of reads at random offsets, for which no (readahead) caching mechanism
would help.
Benchmarks show improved performance that is close to native file system
access when doing massive manipulations on a single opened file, especially
in the case of random reads, for which the bandwidth increased by almost 2X
or sequential writes for which the improvement is close to 3X.
The creation of this direct connection (passthrough) between FUSE file
objects and file objects in the lower file system happens in a way that
reminds of passing file descriptors via sockets:
- a process requests the opening of a file handled by FUSE, so the kernel
forwards the request to the FUSE daemon;
- the FUSE daemon opens the target file in the lower file system, getting
its file descriptor;
- the FUSE daemon also decides according to its internal policies if
passthrough can be enabled for that file, and, if so, can perform a
FUSE_DEV_IOC_PASSTHROUGH_OPEN ioctl() on /dev/fuse, passing the file
descriptor obtained at the previous step and the fuse_req unique
identifier;
- the kernel translates the file descriptor to the file pointer navigating
through the opened files of the "current" process and temporarily stores
it in the associated open/create fuse_req's passthrough_filp;
- when the FUSE daemon has done with the request and it's time for the
kernel to close it, it checks if the passthrough_filp is available and in
case updates the additional field in the fuse_file owned by the process
accessing the FUSE file system.
From now on, all the read/write operations performed by that process will
be redirected to the corresponding lower file system file by creating new
VFS requests.
Since the read/write operation to the lower file system is executed with
the current process's credentials, it might happen that it does not have
enough privileges to succeed. For this reason, the process temporarily
receives the same credentials as the FUSE daemon, that are reverted as soon
as the read/write operation completes, emulating the behavior of the
request to be performed by the FUSE daemon itself. This solution has been
inspired by the way overlayfs handles read/write operations.
Asynchronous IO is supported as well, handled by creating separate AIO
requests for the lower file system that will be internally tracked by FUSE,
that intercepts and propagates their completion through an internal
ki_completed callback similar to the current implementation of overlayfs.
The ioctl() has been designed taking as a reference and trying to converge
to the fuse2 implementation. For example, the fuse_passthrough_out data
structure has extra fields that will allow for further extensions of the
feature.
Performance on SSD
What follows has been performed with this change [V6] rebased on top of
vanilla v5.8 Linux kernel, using a custom passthrough_hp FUSE daemon that
enables pass-through for each file that is opened during both "open" and
"create". Tests were run on an Intel Xeon E5-2678V3, 32GiB of RAM, with an
ext4-formatted SSD as the lower file system, with no special tuning, e.g.,
all the involved processes are SCHED_OTHER, ondemand is the frequency
governor with no frequency restrictions, and turbo-boost, as well as
p-state, are active. This is because I noticed that, for such high-level
benchmarks, results consistency was minimally affected by these features.
The source code of the updated libfuse library and passthrough_hp is shared
at the following repository:
https://github.com/balsini/libfuse/tree/fuse-passthrough-stable-v.3.9.4
Two different kinds of benchmarks were done for this change, the first set
of tests evaluates the bandwidth improvements when manipulating a huge
single file, the second set of tests verify that no performance regressions
were introduced when handling many small files.
The first benchmarks were done by running FIO (fio-3.21) with:
- bs=4Ki;
- file size: 50Gi;
- ioengine: sync;
- fsync_on_close: true.
The target file has been chosen large enough to avoid it to be entirely
loaded into the page cache.
Results are presented in the following table:
+-----------+--------+-------------+--------+
| Bandwidth | FUSE | FUSE | Bind |
| (KiB/s) | | passthrough | mount |
+-----------+--------+-------------+--------+
| read | 468897 | 502085 | 516830 |
+-----------+--------+-------------+--------+
| randread | 15773 | 26632 | 21386 |
+-----------+--------+-------------+--------+
| write | 58185 | 141272 | 141671 |
+-----------+--------+-------------+--------+
| randwrite | 59892 | 75236 | 76486 |
+-----------+--------+-------------+--------+
The higher FUSE passthrough performance compared to bind mount in the case
of randread has been identified as the result or SSD performance
fluctuations when dealing with random offsets. Updated results are reported
below, with the measurements performed on a lower file system created on
top of a RAM block device.
As long as this patch has the primary objective of improving bandwidth,
another set of tests has been performed to see how this behaves on a
totally different scenario that involves accessing many small files. For
this purpose, measuring the build time of the Linux kernel has been chosen
as a well-known workload. The kernel has been built with as many processes
as the number of logical CPUs (-j $(nproc)), that besides being a
reasonable number, is also enough to saturate the processor’s utilization
thanks to the additional FUSE daemon’s threads, making it even harder to
get closer to the native file system performance.
The following table shows the total build times in the different
configurations:
+------------------+--------------+-----------+
| | AVG duration | Standard |
| | (sec) | deviation |
+------------------+--------------+-----------+
| FUSE | 144.566 | 0.697 |
+------------------+--------------+-----------+
| FUSE passthrough | 133.820 | 0.341 |
+------------------+--------------+-----------+
| Raw | 109.423 | 0.724 |
+------------------+--------------+-----------+
Similar performance measurements were performed with the current version of
the patch, the results of which are comparable with what is shown above.
Performance on RAM block device
Getting rid of the discrete storage device removes a huge component of
slowness, highlighting the performance difference of the software parts
(and probably goodness of CPU cache and its coherence/invalidation
mechanisms).
What follows has been performed with this change [V10] rebased on top of
vanilla v5.8 Linux kernel.
More specifically, out of my system's 32 GiB of RAM, I reserved 24 for
/dev/ram0, which has been formatted as ext4. That file system has been
completely filled and then cleaned up before running the benchmarks to make
sure all the memory addresses were marked as used and removed from the page
cache.
The following tests were ran using fio-3.23 with the following
configuration:
- bs=4Ki
- size=20Gi
- ioengine=sync
- fsync_on_close=1
- randseed=0
- create_only=0 (set to 1 during a first dry run to create the test
file)
As for the tool configuration, the following benchmarks would perform a
single open operation each, focusing on just the read/write performance.
The file size of 20 GiB has been chosen to not completely fit the page
cache.
As mentioned in my previous email, all the caches were dropped before
running every benchmark with
echo 3 > /proc/sys/vm/drop_caches
All the benchmarks were run 10 times, with 1 minute cool down between each
run.
Here the updated results for this patch set:
+-----------+-------------+-------------+-------------+
| | | FUSE | |
| MiB/s | FUSE | passthrough | native |
+-----------+-------------+-------------+-------------+
| read | 1341(±4.2%) | 1485(±1.1%) | 1634(±.5%) |
+-----------+-------------+-------------+-------------+
| write | 49(±2.1%) | 1304(±2.6%) | 1363(±3.0%) |
+-----------+-------------+-------------+-------------+
| randread | 43(±1.3%) | 643(±11.1%) | 715(±1.1%) |
+-----------+-------------+-------------+-------------+
| randwrite | 27(±39.9%) | 763(±1.1%) | 790(±1.0%) |
+-----------+-------------+-------------+-------------+
This table shows that FUSE, except for the sequential reads, is left behind
FUSE passthrough and native performance. The extremely good FUSE
performance for sequential reads is the result of a great read-ahead
mechanism, that has been easy to prove by showing that performance dropped
after setting read_ahead_kb to 0.
Except for FUSE randwrite and passthrough randread with respectively ~40%
and ~11% standard deviations, all the other results are relatively stable.
Nevertheless, these two standard deviation exceptions are not sufficient to
invalidate the results, which are still showing clear performance benefits.
I'm also kind of happy to see that passthrough, that for each read/write
operation traverses the VFS layer twice, now maintains consistent slightly
lower performance than native.
Further testing and performance evaluations are welcome.
Description of the series
Patch 1 introduces the data structures and function signatures required
both for the communication with userspace and for the internal kernel use.
Patch 2 introduces the ioctl() and initialization and release functions for
FUSE passthrough.
Patch 3 enables the synchronous read and write operations for those FUSE
files for which the passthrough functionality is enabled.
Patch 4 extends the read and write operations to also support asynchronous
IO.
Patch 5 allows FUSE passthrough to target files for which the requesting
process would not have direct access to, by temporarily performing a
credentials switch to the credentials of the FUSE daemon that issued the
FUSE passthrough ioctl().
Changelog
Changes in v10:
* UAPI updated: ioctl() now returns an ID that will be used at
open/create response time to reference the passthrough file
* Synchronous read/write_iter functions does not return silly errors (fixed
in aio patch)
* FUSE daemon credentials updated at ioctl() time instead of mount time
* Updated benchmark results with RAM block device
[Requested by Miklos Szeredi]
Changes in v9:
* Switched to using VFS instead of direct lower FS file ops
[Attempt to address a request from Jens Axboe, Jann Horn, Amir
Goldstein]
* Removal of useless included aio.h header
[Proposed by Jens Axboe]
Changes in v8:
* aio requests now use kmalloc/kfree, instead of kmem_cache
* Switched to call_{read,write}_iter in AIO
* Revisited attributes copy
* Passthrough can only be enabled via ioctl(), fixing the security issue
spotted by Jann
* Use an extensible fuse_passthrough_out data structure
[Attempt to address a request from Nikolaus Rath, Amir Goldstein and
Miklos Szeredi]
Changes in v7:
* Full handling of aio requests as done in overlayfs (update commit
* message).
* s/fget_raw/fget.
* Open fails in case of passthrough errors, emitting warning messages.
[Proposed by Jann Horn]
* Create new local kiocb, getting rid of the previously proposed ki_filp
* swapping.
[Proposed by Jann Horn and Jens Axboe]
* Code polishing.
Changes in v6:
* Port to kernel v5.8:
* fuse_file_{read,write}_iter() changed since the v5 of this patch was
* proposed.
* Simplify fuse_simple_request().
* Merge fuse_passthrough.h into fuse_i.h
* Refactor of passthrough.c:
* Remove BUG_ON()s.
* Simplified error checking and request arguments indexing.
* Use call_{read,write}_iter() utility functions.
* Remove get_file() and fputs() during read/write: handle the extra
* FUSE references to the lower file object when the fuse_file is
* created/deleted.
[Proposed by Jann Horn]
Changes in v5:
* Fix the check when setting the passthrough file.
[Found when testing by Mike Shal]
Changes in v3 and v4:
* Use the fs_stack_depth to prevent further stacking and a minor fix.
[Proposed by Jann Horn]
Changes in v2:
* Changed the feature name to passthrough from stacked_io.
[Proposed by Linus Torvalds]
Alessio Balsini (5):
fuse: Definitions and ioctl() for passthrough
fuse: Passthrough initialization and release
fuse: Introduce synchronous read and write for passthrough
fuse: Handle asynchronous read and write in passthrough
fuse: Use daemon creds in passthrough mode
fs/fuse/Makefile | 1 +
fs/fuse/dev.c | 40 +++++--
fs/fuse/dir.c | 1 +
fs/fuse/file.c | 12 +-
fs/fuse/fuse_i.h | 31 +++++
fs/fuse/inode.c | 23 +++-
fs/fuse/passthrough.c | 245 ++++++++++++++++++++++++++++++++++++++
include/uapi/linux/fuse.h | 13 +-
8 files changed, 349 insertions(+), 17 deletions(-)
create mode 100644 fs/fuse/passthrough.c
--
2.29.0.rc1.297.gfa9743e501-goog
All the read and write operations performed on fuse_files which have the
passthrough feature enabled are forwarded to the associated lower file
system file via VFS.
Sending the request directly to the lower file system avoids the userspace
round-trip that, because of possible context switches and additional
operations might reduce the overall performance, especially in those cases
where caching doesn't help, for example in reads at random offsets.
Verifying if a fuse_file has a lower file system file associated with can
be done by checking the validity of its passthrough_filp pointer. This
pointer is not NULL only if passthrough has been successfully enabled via
the appropriate ioctl().
When a read/write operation is requested for a FUSE file with passthrough
enabled, a new equivalent VFS request is generated, which instead targets
the lower file system file.
The VFS layer performs additional checks that allow for safer operations
but may cause the operation to fail if the process accessing the FUSE file
system does not have access to the lower file system.
This change only implements synchronous requests in passthrough, returning
an error in the case of asynchronous operations, yet covering the majority
of the use cases.
Signed-off-by: Alessio Balsini <[email protected]>
---
fs/fuse/file.c | 8 +++--
fs/fuse/fuse_i.h | 2 ++
fs/fuse/passthrough.c | 70 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 78 insertions(+), 2 deletions(-)
diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index 84daaf084197..f7a12489c0ef 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -1545,7 +1545,9 @@ static ssize_t fuse_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
if (is_bad_inode(file_inode(file)))
return -EIO;
- if (!(ff->open_flags & FOPEN_DIRECT_IO))
+ if (ff->passthrough.filp)
+ return fuse_passthrough_read_iter(iocb, to);
+ else if (!(ff->open_flags & FOPEN_DIRECT_IO))
return fuse_cache_read_iter(iocb, to);
else
return fuse_direct_read_iter(iocb, to);
@@ -1559,7 +1561,9 @@ static ssize_t fuse_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
if (is_bad_inode(file_inode(file)))
return -EIO;
- if (!(ff->open_flags & FOPEN_DIRECT_IO))
+ if (ff->passthrough.filp)
+ return fuse_passthrough_write_iter(iocb, from);
+ else if (!(ff->open_flags & FOPEN_DIRECT_IO))
return fuse_cache_write_iter(iocb, from);
else
return fuse_direct_write_iter(iocb, from);
diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h
index 32da45ce86e0..a888d3df5877 100644
--- a/fs/fuse/fuse_i.h
+++ b/fs/fuse/fuse_i.h
@@ -1118,5 +1118,7 @@ int fuse_passthrough_open(struct fuse_dev *fud,
int fuse_passthrough_setup(struct fuse_conn *fc, struct fuse_file *ff,
struct fuse_open_out *openarg);
void fuse_passthrough_release(struct fuse_passthrough *passthrough);
+ssize_t fuse_passthrough_read_iter(struct kiocb *iocb, struct iov_iter *to);
+ssize_t fuse_passthrough_write_iter(struct kiocb *iocb, struct iov_iter *from);
#endif /* _FS_FUSE_I_H */
diff --git a/fs/fuse/passthrough.c b/fs/fuse/passthrough.c
index a135c955cc33..5a78cb336db4 100644
--- a/fs/fuse/passthrough.c
+++ b/fs/fuse/passthrough.c
@@ -4,6 +4,76 @@
#include <linux/fuse.h>
#include <linux/idr.h>
+#include <linux/uio.h>
+
+static void fuse_copyattr(struct file *dst_file, struct file *src_file)
+{
+ struct inode *dst = file_inode(dst_file);
+ struct inode *src = file_inode(src_file);
+
+ i_size_write(dst, i_size_read(src));
+}
+
+static inline rwf_t iocb_to_rw_flags(int ifl)
+{
+ rwf_t flags = 0;
+
+ if (ifl & IOCB_APPEND)
+ flags |= RWF_APPEND;
+ if (ifl & IOCB_DSYNC)
+ flags |= RWF_DSYNC;
+ if (ifl & IOCB_HIPRI)
+ flags |= RWF_HIPRI;
+ if (ifl & IOCB_NOWAIT)
+ flags |= RWF_NOWAIT;
+ if (ifl & IOCB_SYNC)
+ flags |= RWF_SYNC;
+
+ return flags;
+}
+
+ssize_t fuse_passthrough_read_iter(struct kiocb *iocb_fuse,
+ struct iov_iter *iter)
+{
+ ssize_t ret;
+ struct file *fuse_filp = iocb_fuse->ki_filp;
+ struct fuse_file *ff = fuse_filp->private_data;
+ struct file *passthrough_filp = ff->passthrough.filp;
+
+ if (!iov_iter_count(iter))
+ return 0;
+
+ ret = vfs_iter_read(passthrough_filp, iter, &iocb_fuse->ki_pos,
+ iocb_to_rw_flags(iocb_fuse->ki_flags));
+
+ return ret;
+}
+
+ssize_t fuse_passthrough_write_iter(struct kiocb *iocb_fuse,
+ struct iov_iter *iter)
+{
+ ssize_t ret;
+ struct file *fuse_filp = iocb_fuse->ki_filp;
+ struct fuse_file *ff = fuse_filp->private_data;
+ struct inode *fuse_inode = file_inode(fuse_filp);
+ struct file *passthrough_filp = ff->passthrough.filp;
+
+ if (!iov_iter_count(iter))
+ return 0;
+
+ inode_lock(fuse_inode);
+
+ file_start_write(passthrough_filp);
+ ret = vfs_iter_write(passthrough_filp, iter, &iocb_fuse->ki_pos,
+ iocb_to_rw_flags(iocb_fuse->ki_flags));
+ file_end_write(passthrough_filp);
+ if (ret > 0)
+ fuse_copyattr(fuse_filp, passthrough_filp);
+
+ inode_unlock(fuse_inode);
+
+ return ret;
+}
int fuse_passthrough_open(struct fuse_dev *fud,
struct fuse_passthrough_out *pto)
--
2.29.0.rc1.297.gfa9743e501-goog
When using FUSE passthrough, read/write operations are directly forwarded
to the lower file system file through VFS, but there is no guarantee that
the process that is triggering the request has the right permissions to
access the lower file system. This would cause the read/write access to
fail.
In passthrough file systems, where the FUSE daemon is responsible for the
enforcement of the lower file system access policies, often happens that
the process dealing with the FUSE file system doesn't have access to the
lower file system.
Being the FUSE daemon in charge of implementing the FUSE file operations,
that in the case of read/write operations usually simply results in the
copy of memory buffers from/to the lower file system respectively, these
operations are executed with the FUSE daemon privileges.
This patch adds a reference to the FUSE daemon credentials, referenced at
FUSE_DEV_IOC_PASSTHROUGH_OPEN ioctl() time so that they can be used to
temporarily raise the user credentials when accessing lower file system
files in passthrough.
The process accessing the FUSE file with passthrough enabled temporarily
receives the privileges of the FUSE daemon while performing read/write
operations. Similar behavior is implemented in overlayfs.
These privileges will be reverted as soon as the IO operation completes.
This feature does not provide any higher security privileges to those
processes accessing the FUSE file system with passthrough enabled. This is
because it is still the FUSE daemon responsible for enabling or not the
passthrough feature at file open time, and should enable the feature only
after appropriate access policy checks.
Signed-off-by: Alessio Balsini <[email protected]>
---
fs/fuse/fuse_i.h | 5 ++++-
fs/fuse/passthrough.c | 11 +++++++++++
2 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h
index a888d3df5877..59e033a59551 100644
--- a/fs/fuse/fuse_i.h
+++ b/fs/fuse/fuse_i.h
@@ -165,10 +165,13 @@ struct fuse_release_args;
/**
* Reference to lower filesystem file for read/write operations handled in
- * passthrough mode
+ * passthrough mode.
+ * This struct also tracks the credentials to be used for handling read/write
+ * operations.
*/
struct fuse_passthrough {
struct file *filp;
+ struct cred *cred;
};
/** FUSE specific file data */
diff --git a/fs/fuse/passthrough.c b/fs/fuse/passthrough.c
index 10b6872cdaa7..ab81dd8f010b 100644
--- a/fs/fuse/passthrough.c
+++ b/fs/fuse/passthrough.c
@@ -67,6 +67,7 @@ ssize_t fuse_passthrough_read_iter(struct kiocb *iocb_fuse,
struct iov_iter *iter)
{
ssize_t ret;
+ const struct cred *old_cred;
struct file *fuse_filp = iocb_fuse->ki_filp;
struct fuse_file *ff = fuse_filp->private_data;
struct file *passthrough_filp = ff->passthrough.filp;
@@ -74,6 +75,7 @@ ssize_t fuse_passthrough_read_iter(struct kiocb *iocb_fuse,
if (!iov_iter_count(iter))
return 0;
+ old_cred = override_creds(ff->passthrough.cred);
if (is_sync_kiocb(iocb_fuse)) {
ret = vfs_iter_read(passthrough_filp, iter, &iocb_fuse->ki_pos,
iocb_to_rw_flags(iocb_fuse->ki_flags));
@@ -91,6 +93,7 @@ ssize_t fuse_passthrough_read_iter(struct kiocb *iocb_fuse,
if (ret != -EIOCBQUEUED)
fuse_aio_cleanup_handler(aio_req);
}
+ revert_creds(old_cred);
return ret;
}
@@ -99,6 +102,7 @@ ssize_t fuse_passthrough_write_iter(struct kiocb *iocb_fuse,
struct iov_iter *iter)
{
ssize_t ret;
+ const struct cred *old_cred;
struct file *fuse_filp = iocb_fuse->ki_filp;
struct fuse_file *ff = fuse_filp->private_data;
struct inode *fuse_inode = file_inode(fuse_filp);
@@ -110,6 +114,7 @@ ssize_t fuse_passthrough_write_iter(struct kiocb *iocb_fuse,
inode_lock(fuse_inode);
+ old_cred = override_creds(ff->passthrough.cred);
if (is_sync_kiocb(iocb_fuse)) {
file_start_write(passthrough_filp);
ret = vfs_iter_write(passthrough_filp, iter, &iocb_fuse->ki_pos,
@@ -137,6 +142,7 @@ ssize_t fuse_passthrough_write_iter(struct kiocb *iocb_fuse,
fuse_aio_cleanup_handler(aio_req);
}
out:
+ revert_creds(old_cred);
inode_unlock(fuse_inode);
return ret;
@@ -174,6 +180,7 @@ int fuse_passthrough_open(struct fuse_dev *fud,
return -ENOMEM;
passthrough->filp = passthrough_filp;
+ passthrough->cred = prepare_creds();
idr_preload(GFP_KERNEL);
spin_lock(&fc->passthrough_req_lock);
@@ -231,4 +238,8 @@ void fuse_passthrough_release(struct fuse_passthrough *passthrough)
fput(passthrough->filp);
passthrough->filp = NULL;
}
+ if (passthrough->cred) {
+ put_cred(passthrough->cred);
+ passthrough->cred = NULL;
+ }
}
--
2.29.0.rc1.297.gfa9743e501-goog
On Tue, Oct 27, 2020 at 1:00 AM Alessio Balsini <[email protected]> wrote:
>
> This is the 10th version of the series. Please find the changelog at the
> bottom of this cover letter.
>
> Add support for file system passthrough read/write of files when enabled in
> userspace through the option FUSE_PASSTHROUGH.
>
> There are file systems based on FUSE that are intended to enforce special
> policies or trigger complicated decision makings at the file operations
> level. Android, for example, uses FUSE to enforce fine-grained access
> policies that also depend on the file contents.
> Sometimes it happens that at open or create time a file is identified as
> not requiring additional checks for consequent reads/writes, thus FUSE
> would simply act as a passive bridge between the process accessing the FUSE
> file system and the lower file system. Splicing and caching help reduce the
> FUSE overhead, but there are still read/write operations forwarded to the
> userspace FUSE daemon that could be avoided.
>
> This series has been inspired by the original patches from Nikhilesh Reddy,
> the idea and code of which has been elaborated and improved thanks to the
> community support.
>
> When the FUSE_PASSTHROUGH capability is enabled, the FUSE daemon may decide
> while handling the open/create operations, if the given file can be
> accessed in passthrough mode. This means that all the further read and
> write operations would be forwarded by the kernel directly to the lower
> file system using the VFS layer rather than to the FUSE daemon.
> All the requests other than reads or writes are still handled by the
> userspace FUSE daemon.
> This allows for improved performance on reads and writes, especially in the
> case of reads at random offsets, for which no (readahead) caching mechanism
> would help.
> Benchmarks show improved performance that is close to native file system
> access when doing massive manipulations on a single opened file, especially
> in the case of random reads, for which the bandwidth increased by almost 2X
> or sequential writes for which the improvement is close to 3X.
>
> The creation of this direct connection (passthrough) between FUSE file
> objects and file objects in the lower file system happens in a way that
> reminds of passing file descriptors via sockets:
> - a process requests the opening of a file handled by FUSE, so the kernel
> forwards the request to the FUSE daemon;
> - the FUSE daemon opens the target file in the lower file system, getting
> its file descriptor;
> - the FUSE daemon also decides according to its internal policies if
> passthrough can be enabled for that file, and, if so, can perform a
> FUSE_DEV_IOC_PASSTHROUGH_OPEN ioctl() on /dev/fuse, passing the file
> descriptor obtained at the previous step and the fuse_req unique
> identifier;
> - the kernel translates the file descriptor to the file pointer navigating
> through the opened files of the "current" process and temporarily stores
> it in the associated open/create fuse_req's passthrough_filp;
> - when the FUSE daemon has done with the request and it's time for the
> kernel to close it, it checks if the passthrough_filp is available and in
> case updates the additional field in the fuse_file owned by the process
> accessing the FUSE file system.
> From now on, all the read/write operations performed by that process will
> be redirected to the corresponding lower file system file by creating new
> VFS requests.
> Since the read/write operation to the lower file system is executed with
> the current process's credentials, it might happen that it does not have
> enough privileges to succeed. For this reason, the process temporarily
> receives the same credentials as the FUSE daemon, that are reverted as soon
> as the read/write operation completes, emulating the behavior of the
> request to be performed by the FUSE daemon itself. This solution has been
> inspired by the way overlayfs handles read/write operations.
> Asynchronous IO is supported as well, handled by creating separate AIO
> requests for the lower file system that will be internally tracked by FUSE,
> that intercepts and propagates their completion through an internal
> ki_completed callback similar to the current implementation of overlayfs.
> The ioctl() has been designed taking as a reference and trying to converge
> to the fuse2 implementation. For example, the fuse_passthrough_out data
> structure has extra fields that will allow for further extensions of the
> feature.
>
>
> Performance on SSD
>
> What follows has been performed with this change [V6] rebased on top of
> vanilla v5.8 Linux kernel, using a custom passthrough_hp FUSE daemon that
> enables pass-through for each file that is opened during both "open" and
> "create". Tests were run on an Intel Xeon E5-2678V3, 32GiB of RAM, with an
> ext4-formatted SSD as the lower file system, with no special tuning, e.g.,
> all the involved processes are SCHED_OTHER, ondemand is the frequency
> governor with no frequency restrictions, and turbo-boost, as well as
> p-state, are active. This is because I noticed that, for such high-level
> benchmarks, results consistency was minimally affected by these features.
> The source code of the updated libfuse library and passthrough_hp is shared
> at the following repository:
>
> https://github.com/balsini/libfuse/tree/fuse-passthrough-stable-v.3.9.4
The libfuse changes are not updated with the latest ioctl UAPI change yet.
> * UAPI updated: ioctl() now returns an ID that will be used at
> open/create response time to reference the passthrough file
Cheers,
Tao
--
Into Sth. Rich & Strange
On Sat, Nov 28, 2020 at 10:10:37AM +0800, Peng Tao wrote:
> On Tue, Oct 27, 2020 at 1:00 AM Alessio Balsini <[email protected]> wrote:
> >
> > This is the 10th version of the series. Please find the changelog at the
> > bottom of this cover letter.
> >
> > Add support for file system passthrough read/write of files when enabled in
> > userspace through the option FUSE_PASSTHROUGH.
> >
> > There are file systems based on FUSE that are intended to enforce special
> > policies or trigger complicated decision makings at the file operations
> > level. Android, for example, uses FUSE to enforce fine-grained access
> > policies that also depend on the file contents.
> > Sometimes it happens that at open or create time a file is identified as
> > not requiring additional checks for consequent reads/writes, thus FUSE
> > would simply act as a passive bridge between the process accessing the FUSE
> > file system and the lower file system. Splicing and caching help reduce the
> > FUSE overhead, but there are still read/write operations forwarded to the
> > userspace FUSE daemon that could be avoided.
> >
> > This series has been inspired by the original patches from Nikhilesh Reddy,
> > the idea and code of which has been elaborated and improved thanks to the
> > community support.
> >
> > When the FUSE_PASSTHROUGH capability is enabled, the FUSE daemon may decide
> > while handling the open/create operations, if the given file can be
> > accessed in passthrough mode. This means that all the further read and
> > write operations would be forwarded by the kernel directly to the lower
> > file system using the VFS layer rather than to the FUSE daemon.
> > All the requests other than reads or writes are still handled by the
> > userspace FUSE daemon.
> > This allows for improved performance on reads and writes, especially in the
> > case of reads at random offsets, for which no (readahead) caching mechanism
> > would help.
> > Benchmarks show improved performance that is close to native file system
> > access when doing massive manipulations on a single opened file, especially
> > in the case of random reads, for which the bandwidth increased by almost 2X
> > or sequential writes for which the improvement is close to 3X.
> >
> > The creation of this direct connection (passthrough) between FUSE file
> > objects and file objects in the lower file system happens in a way that
> > reminds of passing file descriptors via sockets:
> > - a process requests the opening of a file handled by FUSE, so the kernel
> > forwards the request to the FUSE daemon;
> > - the FUSE daemon opens the target file in the lower file system, getting
> > its file descriptor;
> > - the FUSE daemon also decides according to its internal policies if
> > passthrough can be enabled for that file, and, if so, can perform a
> > FUSE_DEV_IOC_PASSTHROUGH_OPEN ioctl() on /dev/fuse, passing the file
> > descriptor obtained at the previous step and the fuse_req unique
> > identifier;
> > - the kernel translates the file descriptor to the file pointer navigating
> > through the opened files of the "current" process and temporarily stores
> > it in the associated open/create fuse_req's passthrough_filp;
> > - when the FUSE daemon has done with the request and it's time for the
> > kernel to close it, it checks if the passthrough_filp is available and in
> > case updates the additional field in the fuse_file owned by the process
> > accessing the FUSE file system.
> > From now on, all the read/write operations performed by that process will
> > be redirected to the corresponding lower file system file by creating new
> > VFS requests.
> > Since the read/write operation to the lower file system is executed with
> > the current process's credentials, it might happen that it does not have
> > enough privileges to succeed. For this reason, the process temporarily
> > receives the same credentials as the FUSE daemon, that are reverted as soon
> > as the read/write operation completes, emulating the behavior of the
> > request to be performed by the FUSE daemon itself. This solution has been
> > inspired by the way overlayfs handles read/write operations.
> > Asynchronous IO is supported as well, handled by creating separate AIO
> > requests for the lower file system that will be internally tracked by FUSE,
> > that intercepts and propagates their completion through an internal
> > ki_completed callback similar to the current implementation of overlayfs.
> > The ioctl() has been designed taking as a reference and trying to converge
> > to the fuse2 implementation. For example, the fuse_passthrough_out data
> > structure has extra fields that will allow for further extensions of the
> > feature.
> >
> >
> > Performance on SSD
> >
> > What follows has been performed with this change [V6] rebased on top of
> > vanilla v5.8 Linux kernel, using a custom passthrough_hp FUSE daemon that
> > enables pass-through for each file that is opened during both "open" and
> > "create". Tests were run on an Intel Xeon E5-2678V3, 32GiB of RAM, with an
> > ext4-formatted SSD as the lower file system, with no special tuning, e.g.,
> > all the involved processes are SCHED_OTHER, ondemand is the frequency
> > governor with no frequency restrictions, and turbo-boost, as well as
> > p-state, are active. This is because I noticed that, for such high-level
> > benchmarks, results consistency was minimally affected by these features.
> > The source code of the updated libfuse library and passthrough_hp is shared
> > at the following repository:
> >
> > https://github.com/balsini/libfuse/tree/fuse-passthrough-stable-v.3.9.4
> The libfuse changes are not updated with the latest ioctl UAPI change yet.
>
> > * UAPI updated: ioctl() now returns an ID that will be used at
> > open/create response time to reference the passthrough file
>
> Cheers,
> Tao
> --
> Into Sth. Rich & Strange
Hi Tao,
You are right, sorry, this is the correct branch that uses the
FUSE_DEV_IOC_PASSTHROUGH_OPEN ioctl with the current patch set:
https://github.com/balsini/libfuse/tree/fuse-passthrough-v10-linux-5.8-v.3.9.4
I think I'll stick to this libfuse branch naming pattern from now on. :)
Thanks,
Alessio