Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Sat, 19 Jun 2021 14:47:05 +0800
From:   Wang Yugui <wangyugui@e16-tech.com>
To:     "NeilBrown" <neilb@suse.de>, linux-nfs@vger.kernel.org
Subject: Re: any idea about auto export multiple btrfs snapshots?
In-Reply-To: <20210618213417.2314.409509F4@e16-tech.com>
References: <20210618152631.F3DE.409509F4@e16-tech.com> <20210618213417.2314.409509F4@e16-tech.com>
Message-Id: <20210619144703.D583.409509F4@e16-tech.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="US-ASCII"
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi,

> > > On Thu, 17 Jun 2021, Wang Yugui wrote:
> > > > > Can we go back to the beginning.  What, exactly, is the problem you are
> > > > > trying to solve?  How can you demonstrate the problem?
> > > > > 
> > > > > NeilBrown
> > > > 
> > > > I nfs/exported a btrfs with 2 subvols and 2 snapshot(subvol).
> > > > 
> > > > # btrfs subvolume list /mnt/test
> > > > ID 256 gen 53 top level 5 path sub1
> > > > ID 260 gen 56 top level 5 path sub2
> > > > ID 261 gen 57 top level 5 path .snapshot/sub1-s1
> > > > ID 262 gen 57 top level 5 path .snapshot/sub2-s1
> > > > 
> > > > and then mount.nfs4 it to /nfs/test.
> > > > 
> > > > # /bin/find /nfs/test/
> > > > /nfs/test/
> > > > find: File system loop detected; '/nfs/test/sub1' is part of the same file system loop as '/nfs/test/'.
> > > > /nfs/test/.snapshot
> > > > find: File system loop detected; '/nfs/test/.snapshot/sub1-s1' is part of the same file system loop as '/nfs/test/'.
> > > > find: File system loop detected; '/nfs/test/.snapshot/sub2-s1' is part of the same file system loop as '/nfs/test/'.
> > > > /nfs/test/dir1
> > > > /nfs/test/dir1/a.txt
> > > > find: File system loop detected; '/nfs/test/sub2' is part of the same file system loop as '/nfs/test/'
> > > > 
> > > > /bin/find report 'File system loop detected'. so I though there is
> > > > something wrong.
> > > 
> > > Certainly something is wrong.  The error message implies that some
> > > directory is reporting the same dev an ino as an ancestor directory.
> > > Presumably /nfs/test and /nfs/test/sub1.
> > > Can you confirm that please. e.g. run the command
> > > 
> > >    stat /nfs/test /nfs/test/sub1
> > > and examine the output.
> > 
> > # stat /nfs/test /nfs/test/sub1
> >   File: /nfs/test
> >   Size: 42              Blocks: 32         IO Block: 32768  directory
> > Device: 36h/54d Inode: 256         Links: 1
> > Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
> > Access: 2021-06-18 13:50:55.409457648 +0800
> > Modify: 2021-06-13 10:05:10.830825901 +0800
> > Change: 2021-06-13 10:05:10.830825901 +0800
> >  Birth: -
> >   File: /nfs/test/sub1
> >   Size: 8               Blocks: 0          IO Block: 32768  directory
> > Device: 36h/54d Inode: 256         Links: 1
> > Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
> > Access: 2021-06-18 13:51:14.463621411 +0800
> > Modify: 2021-06-12 21:59:10.598089917 +0800
> > Change: 2021-06-12 21:59:10.598089917 +0800
> >  Birth: -
> > 
> > same 'Device/Inode' are reported.
> > 
> > 
> > but the local btrfs mount,
> > # stat /mnt/test/ /mnt/test/sub1
> >   File: /mnt/test/
> >   Size: 42              Blocks: 32         IO Block: 4096   directory
> > Device: 33h/51d Inode: 256         Links: 1
> > Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
> > Access: 2021-06-18 13:50:55.409457648 +0800
> > Modify: 2021-06-13 10:05:10.830825901 +0800
> > Change: 2021-06-13 10:05:10.830825901 +0800
> >  Birth: -
> >   File: /mnt/test/sub1
> >   Size: 8               Blocks: 0          IO Block: 4096   directory
> > Device: 34h/52d Inode: 256         Links: 1
> > Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
> > Access: 2021-06-18 13:51:14.463621411 +0800
> > Modify: 2021-06-12 21:59:10.598089917 +0800
> > Change: 2021-06-12 21:59:10.598089917 +0800
> >  Birth: -
> > 
> > 'stat' command should cause nfs/crossmnt to happen auto, and then return
> > the 'stat' result?
> > 
> > 
> > > As sub1 is considered a different file system, it should have a
> > > different dev number.  NFS will assign a different device number only
> > > when the server reports a different fsid.  The Linux NFS server will
> > > report a different fsid if d_mountpoint() is 'true' for the dentry, and
> > > follow_down() results in no change the the vfsmnt,dentry in a 'struct
> > > path'.
> > > 
> > > You have already said that d_mountpoint doesn't work for btrfs, so that
> > > is part of the problem.  NFSD doesn't trust d_mountpoint completely as
> > > it only reports that the dentry is a mountpoint in some namespace, not
> > > necessarily in this namespace.  So you really need to fix
> > > nfsd_mountpoint.
> > > 
> > > I suggest you try adding your "dirty fix" to nfsd_mountpoint() so that
> > > it reports the root of a btrfs subvol as a mountpoint, and see if that
> > > fixes the problem.  It should change the problem at least.  You would
> > > need to get nfsd_mountpoint() to return '1' in this case, not '2'.
> > > 
> > > NeilBrown
> > 
> > I changed the return value from 2 to 1.
> >         if (nfsd4_is_junction(dentry))
> >                 return 1;
> > +       if (is_btrfs_subvol_d(dentry))
> > +               return 1;
> >         if (d_mountpoint(dentry))
> > 
> > but the crossmnt still does not happen auto.
> > 
> > I tried to mount the subvol manual, 
> > # mount.nfs4 T7610:/mnt/test/sub1 /nfs/test/sub1
> > mount.nfs4: Stale file handle
> > 
> > we add trace to is_btrfs_subvol_d(), it works as expected.
> > +static inline bool is_btrfs_subvol_d(const struct dentry *dentry)
> > +{
> > +    bool ret= dentry->d_inode && dentry->d_inode->i_ino == 256ULL &&
> > +		dentry->d_sb && dentry->d_sb->s_magic == 0x9123683E;
> > +	printk(KERN_INFO "is_btrfs_subvol_d(%s)=%d\n", dentry->d_name.name, ret);
> > +	return ret;
> > +}
> > 
> > It seems more fixes are needed.
> 
> for a normal crossmnt,
> 	/mnt/test			btrfs
> 	/mn/test/xfs1		xfs
> this xfs1 have 2 inode,
> 1) inode in xfs /mn/test/xfs, as the root.
> 2) inode in btrfs /mnt/test, as a directory.
> when /mn/test/xfs1 is mounted, nfs client with nocrossmnt option will
> show 2).
> 
> but for a btrfs subvol,
> 	/mnt/test		btrfs
> 	/mnt/test/sub1	 btrfs subvol
> this sub1 have just 1 inode
> 1) inode in /mnt/test/sub1, as the root
> 
> This difference break the nfs support of btrfs multiple subvol?

When os shutdown, btrfs subvol in nfs client is firstly unmounted,
then the btrfs subvol entry in nfs client will have a unmounted status.

this btrfs subvol unmounted status in nfs client does NOT exist in btrfs,
we could use a dummy inode value(BTRFS_LAST_FREE_OBJECTID/-255ULL).

btrfs subvol entry mounted status
	st_dev	: from subvol
	st_ino	 : form subvol (255ULL) 

btrfs subvol entry mounted status
	st_dev	: from parent
	st_ino	 : dummy inode ( -255ULL) 

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2021/06/19