Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 28 Oct 2021 09:01:32 -0400
From:   Vivek Goyal <vgoyal@redhat.com>
To:     JeffleXu <jefflexu@linux.alibaba.com>
Cc:     Ira Weiny <ira.weiny@intel.com>,
        "Darrick J. Wong" <djwong@kernel.org>,
        Theodore Ts'o <tytso@mit.edu>, adilger.kernel@dilger.ca,
        linux-xfs@vger.kernel.org,
        "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
        linux-fsdevel@vger.kernel.org, dan.j.williams@intel.com,
        Christoph Hellwig <hch@lst.de>,
        Dave Chinner <dchinner@redhat.com>
Subject: Re: [Question] ext4/xfs: Default behavior changed after per-file DAX
Message-ID: <YXqfLDLogZlk3m9e@redhat.com>
References: <26ddaf6d-fea7-ed20-cafb-decd63b2652a@linux.alibaba.com>
 <20211026154834.GB24307@magnolia>
 <YXhWP/FCkgHG/+ou@redhat.com>
 <20211026205730.GI3465596@iweiny-DESK2.sc.intel.com>
 <YXlj6GhxkFBQRJYk@redhat.com>
 <665787d0-f227-a95b-37a3-20f2ea3e09aa@linux.alibaba.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <665787d0-f227-a95b-37a3-20f2ea3e09aa@linux.alibaba.com>
Precedence: bulk

On Thu, Oct 28, 2021 at 01:52:27PM +0800, JeffleXu wrote:
> 
> 
> On 10/27/21 10:36 PM, Vivek Goyal wrote:
> > [snip]
> > 
> >>
> >> Is the biggest issue the lack of visibility to see if the device supports DAX?
> > 
> > Not necessarily. I think for me two biggest issues are.
> > 
> > - Should dax be enabled by default in server as well. If we do that,
> >   server will have to make extra ioctl() call on every LOOKUP and GETATTR
> >   fuse request. Local filesystems probably can easily query FS_XFLAGS_DAX
> >   state but doing extra syscall all the time will probably be some cost
> >   (No idea how much).
> 
> I tested the time cost from virtiofsd's perspective (time cost of
> passthrough_ll.c:lo_do_lookup()):
> - before per inode DAX feature: 2~4 us
> - after per inode DAX feature: 7~8 us
> 
> It is within expectation, as the introduction of per inode DAX feature,
> one extra ioctl() system call is introduced.
> 
> Also the time cost from client's perspective (time cost of
> fs/fuse/dir.c:fuse_lookup_name())
> - before per inode DAX feature: 25~30 us
> - after per inode DAX feature: 30~35 us
> 
> That is, ~15%~20% performance loss.

Hi Jeffle,

Thanks for measuring the performance impact of enabling dax=inode by
default in server.

> 
> Currently we do ioctl() to query the persitent inode flags every time
> FUSE_LOOKUP request is received, maybe we could cache the result of
> ioctl() on virtiofsd side, but I have no idea how to intercept the
> runtime modification to these persistent indoe flags from other
> processes on host, e.g. sysadmin on host, to maintain the cache consistency.
> 
> So if the default behavior of client side is 'dax=inode', and virtiofsd
> disables per inode DAX by default (neither '-o dax=server|attr' is
> specified for virtiofsd) for the sake of performance, then guest won't
> see DAX enabled and thus won't be surprised. This can reduce the
> behavior change to the minimum.

Agreed. Lets not enable any dax by default in server and let admin/user
enable dax explicitly in server. From fuse client perspective, we can
assume dax=inode by default. That way kernel side behavior will be
similar to ext4/xfs (as long as server has been started with per
inode dax policy).

Vivek
> 
> 
> > 
> > - So far if virtiofs is mounted without any of the dax options, just
> >   by looking at mount option, I could tell, DAX is not enabled on any
> >   of the files. But that will not be true anymore. Because dax=inode
> >   be default, it is possible that server upgrade enabled dax on some
> >   or all the files.
> > 
> >   I guess I will have to stick to same reason given by ext4/xfs. That is
> >   to determine whether DAX is enabled on a file or not, you need to
> >   query STATX_ATTR_DAX flag. That's the only way to conclude if DAX is
> >   being used on a file or not. Don't look at filesystem mount options
> >   and reach a conclusion (except the case of dax=never).
> 
> 
> -- 
> Thanks,
> Jeffle
>