Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Mon, 30 Nov 2020 11:08:39 +0000
From:   Alessio Balsini <balsini@android.com>
To:     Peng Tao <bergwolf@gmail.com>
Cc:     Alessio Balsini <balsini@android.com>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Akilesh Kailash <akailash@google.com>,
        Amir Goldstein <amir73il@gmail.com>,
        Antonio SJ Musumeci <trapexit@spawn.link>,
        David Anderson <dvander@google.com>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Jann Horn <jannh@google.com>, Jens Axboe <axboe@kernel.dk>,
        Martijn Coenen <maco@android.com>,
        Palmer Dabbelt <palmer@dabbelt.com>,
        Paul Lawrence <paullawrence@google.com>,
        Stefano Duo <duostefano93@gmail.com>,
        Zimuzo Ezeozue <zezeozue@google.com>,
        fuse-devel@lists.sourceforge.net, kernel-team@android.com,
        "linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH V10 0/5] fuse: Add support for passthrough read/write
Message-ID: <20201130110839.GA1980518@google.com>
References: <20201026125016.1905945-1-balsini@android.com>
 <CA+a=Yy76W4Xob6UVXsPLA_FKF_8+QFQSEF98yALjRmuOhnVOdw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CA+a=Yy76W4Xob6UVXsPLA_FKF_8+QFQSEF98yALjRmuOhnVOdw@mail.gmail.com>
Precedence: bulk

On Sat, Nov 28, 2020 at 10:10:37AM +0800, Peng Tao wrote:
> On Tue, Oct 27, 2020 at 1:00 AM Alessio Balsini <balsini@android.com> wrote:
> >
> > This is the 10th version of the series. Please find the changelog at the
> > bottom of this cover letter.
> >
> > Add support for file system passthrough read/write of files when enabled in
> > userspace through the option FUSE_PASSTHROUGH.
> >
> > There are file systems based on FUSE that are intended to enforce special
> > policies or trigger complicated decision makings at the file operations
> > level. Android, for example, uses FUSE to enforce fine-grained access
> > policies that also depend on the file contents.
> > Sometimes it happens that at open or create time a file is identified as
> > not requiring additional checks for consequent reads/writes, thus FUSE
> > would simply act as a passive bridge between the process accessing the FUSE
> > file system and the lower file system. Splicing and caching help reduce the
> > FUSE overhead, but there are still read/write operations forwarded to the
> > userspace FUSE daemon that could be avoided.
> >
> > This series has been inspired by the original patches from Nikhilesh Reddy,
> > the idea and code of which has been elaborated and improved thanks to the
> > community support.
> >
> > When the FUSE_PASSTHROUGH capability is enabled, the FUSE daemon may decide
> > while handling the open/create operations, if the given file can be
> > accessed in passthrough mode. This means that all the further read and
> > write operations would be forwarded by the kernel directly to the lower
> > file system using the VFS layer rather than to the FUSE daemon.
> > All the requests other than reads or writes are still handled by the
> > userspace FUSE daemon.
> > This allows for improved performance on reads and writes, especially in the
> > case of reads at random offsets, for which no (readahead) caching mechanism
> > would help.
> > Benchmarks show improved performance that is close to native file system
> > access when doing massive manipulations on a single opened file, especially
> > in the case of random reads, for which the bandwidth increased by almost 2X
> > or sequential writes for which the improvement is close to 3X.
> >
> > The creation of this direct connection (passthrough) between FUSE file
> > objects and file objects in the lower file system happens in a way that
> > reminds of passing file descriptors via sockets:
> > - a process requests the opening of a file handled by FUSE, so the kernel
> >   forwards the request to the FUSE daemon;
> > - the FUSE daemon opens the target file in the lower file system, getting
> >   its file descriptor;
> > - the FUSE daemon also decides according to its internal policies if
> >   passthrough can be enabled for that file, and, if so, can perform a
> >   FUSE_DEV_IOC_PASSTHROUGH_OPEN ioctl() on /dev/fuse, passing the file
> >   descriptor obtained at the previous step and the fuse_req unique
> >   identifier;
> > - the kernel translates the file descriptor to the file pointer navigating
> >   through the opened files of the "current" process and temporarily stores
> >   it in the associated open/create fuse_req's passthrough_filp;
> > - when the FUSE daemon has done with the request and it's time for the
> >   kernel to close it, it checks if the passthrough_filp is available and in
> >   case updates the additional field in the fuse_file owned by the process
> >   accessing the FUSE file system.
> > From now on, all the read/write operations performed by that process will
> > be redirected to the corresponding lower file system file by creating new
> > VFS requests.
> > Since the read/write operation to the lower file system is executed with
> > the current process's credentials, it might happen that it does not have
> > enough privileges to succeed. For this reason, the process temporarily
> > receives the same credentials as the FUSE daemon, that are reverted as soon
> > as the read/write operation completes, emulating the behavior of the
> > request to be performed by the FUSE daemon itself. This solution has been
> > inspired by the way overlayfs handles read/write operations.
> > Asynchronous IO is supported as well, handled by creating separate AIO
> > requests for the lower file system that will be internally tracked by FUSE,
> > that intercepts and propagates their completion through an internal
> > ki_completed callback similar to the current implementation of overlayfs.
> > The ioctl() has been designed taking as a reference and trying to converge
> > to the fuse2 implementation. For example, the fuse_passthrough_out data
> > structure has extra fields that will allow for further extensions of the
> > feature.
> >
> >
> >     Performance on SSD
> >
> > What follows has been performed with this change [V6] rebased on top of
> > vanilla v5.8 Linux kernel, using a custom passthrough_hp FUSE daemon that
> > enables pass-through for each file that is opened during both "open" and
> > "create". Tests were run on an Intel Xeon E5-2678V3, 32GiB of RAM, with an
> > ext4-formatted SSD as the lower file system, with no special tuning, e.g.,
> > all the involved processes are SCHED_OTHER, ondemand is the frequency
> > governor with no frequency restrictions, and turbo-boost, as well as
> > p-state, are active. This is because I noticed that, for such high-level
> > benchmarks, results consistency was minimally affected by these features.
> > The source code of the updated libfuse library and passthrough_hp is shared
> > at the following repository:
> >
> >     https://github.com/balsini/libfuse/tree/fuse-passthrough-stable-v.3.9.4
> The libfuse changes are not updated with the latest ioctl UAPI change yet.
> 
> > * UAPI updated: ioctl() now returns an ID that will be used at
> >   open/create response time to reference the passthrough file
> 
> Cheers,
> Tao
> -- 
> Into Sth. Rich & Strange

Hi Tao,

You are right, sorry, this is the correct branch that uses the
FUSE_DEV_IOC_PASSTHROUGH_OPEN ioctl with the current patch set:

  https://github.com/balsini/libfuse/tree/fuse-passthrough-v10-linux-5.8-v.3.9.4

I think I'll stick to this libfuse branch naming pattern from now on. :)

Thanks,
Alessio