by Jonathan Katz

[permalink] [raw]

Subject: Re: [PATCH] [REGRESSION] ovl: Handle ENOSYS when fileattr support is missing in lower/upper fs

2023-03-22 18:44:23

by Jonathan Katz

[permalink] [raw]

Subject: Re: [PATCH] [REGRESSION] ovl: Handle ENOSYS when fileattr support is missing in lower/upper fs

Confirmed bindfs interaction:

Based on your bindfs comment I redid my configuration as follows:
ORIGINAL (FAILS):
FS1 - exports "/Data" (nfs)
FS2 - Mounts "/Data", does a bindfs, does an overlay

TEST (SUCCEEDS):
FS1 - does a bindfs and exports a series of directories:
# bindfs -u 5007, -g 5007 /Data /Data-jiajun
/etc/exports:
/Data machine.org(ro,sync,no_subtree_check)
/Data-jiajun machine.org(ro,fsid=12,sync,no_subtree_check)
FS2 - used to do bindfs to make the lowers, but, now mounts
"/Data-jiajun" as the lower
FS2 then does the overlay and samba share.
It would not let me do the 2nd export if I did not
include the fsid entry....

WOOT WOOT.

Not an ideal solution as I have to make changes to 2 servers in order
to accomplish my goal :/.

On Tue, Mar 14, 2023 at 7:43 PM Jonathan Katz <[email protected]> wrote:
>
> On Thu, Mar 9, 2023 at 7:31 AM Miklos Szeredi <[email protected]> wrote:
> >
> > On Tue, 7 Mar 2023 at 18:14, Jonathan Katz <[email protected]> wrote:
> > >
> > > On Tue, Mar 7, 2023 at 12:38 AM Miklos Szeredi <[email protected]> wrote:
> > > >
> > > > On Tue, 7 Mar 2023 at 02:12, Jonathan Katz <[email protected]> wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > In pursuing this issue, I downloaded the kernel source to see if I
> > > > > could debug it further. In so doing, it looks like Christian's patch
> > > > > was never committed to the main source tree (sorry if my terminology
> > > > > is wrong). This is up to and including the 6.3-rc1. I could also
> > > > > find no mention of the fix in the log.
> > > > >
> > > > > I am trying to manually apply this patch now, but, I am wondering if
> > > > > there was some reason that it was not applied (e.g. it introduces some
> > > > > instability?)?
> > > >
> > > > It's fixing the bug in the wrong place, i.e. it's checking for an
> > > > -ENOSYS return from vfs_fileattr_get(), but that return value is not
> > > > valid at that point.
> > > >
> > > > The right way to fix this bug is to prevent -ENOSYS from being
> > > > returned in the first place.
> > > >
> > > > Commit 02c0cab8e734 ("fuse: ioctl: translate ENOSYS") fixes one of
> > > > those bugs, but of course it's possible that I missed something in
> > > > that fix.
> > > >
> > > > Can you please first verify that an upstream kernel (>v6.0) can also
> > > > reproduce this issue?
> > >
> > > Got ya. that makes a lot of sense, thank you.
> > >
> > > I have confirmed that I continue to get the error with 6.2 .
> > > quick summary of the lowerdir:
> > > server ---- NFS(ro) ---- > client "/nfs"
> > > client "/nfs" --- bindfs(uidmap) --- > client "/lower"
> >
> > Can you please run bindfs in debugging mode (-d) and send the
> > resulting log after reproducing the issue?
> >
> > Thanks,
> > Miklos
>
> OUCH -- MY LAST EMAIL WAS REJECTED FOR BEING TOO BIG
> I HOPE THAT I AM SUMMARIZING THE RELEVANT INFORMATION HERE:
>
>
> Hi Miklos, thank you.... I am sorry for the delay.
>
> The log is somewhat long and was sent in a separate email.
>
> I broke up the log into entries to try to match the chronology of actions:
> * ENTRY 1 nfs mount the external drive
> * ENTRY 2 perform the bind fs
> * ENTRY 3 perform the overlay
> * ENTRY 4 restart smb
> * ENTRY 5 mount the filesystem on a windows box
> * ENTRY 6 performing some navigation on the windows file explorer
> * ENTRY 7 attempt to open a data file with the windows application.
>
> The only place that generated a kernel error in dmesg was at ENTRY 7.
>
> Because the logs are so big, I tried to parse them, I may have made a
> mistake or omitted information -- if you think so, as mentioned, the
> full bindfs logs were sent separately
>
>
> Here is my attempt to parse out the errors associated with this dmesg entry:
>
> [ 1925.705908] overlayfs: failed to retrieve lower fileattr (8020
> MeOHH2O RecoverySample1-20221216-A-JJL-WebinarHilic10C-TOF-TT54-Neg-1632.d/chromatography-data.sqlite,
> err=-38)
>
> --
> unique: 1550, opcode: GETXATTR (22), nodeid: 71, insize: 73, pid: 3458
> getxattr /eimstims1/deleteme2/8020 MeOHH2O
> RecoverySample1-20221216-A-JJL-WebinarHilic10C-TOF-TT54-Neg-1632.d/chromatography-data-pre.sqlite
> trusted.overlay.metacopy 0
> unique: 1550, error: -95 (Operation not supported), outsize: 16
> --
> unique: 3922, opcode: GETXATTR (22), nodeid: 71, insize: 72, pid: 3458
> getxattr /eimstims1/deleteme2/8020 MeOHH2O
> RecoverySample1-20221216-A-JJL-WebinarHilic10C-TOF-TT54-Neg-1632.d/chromatography-data-pre.sqlite
> system.posix_acl_access 132
> unique: 3922, error: -95 (Operation not supported), outsize: 16
> --
> unique: 3954, opcode: GETXATTR (22), nodeid: 71, insize: 72, pid: 3458
> getxattr /eimstims1/deleteme2/8020 MeOHH2O
> RecoverySample1-20221216-A-JJL-WebinarHilic10C-TOF-TT54-Neg-1632.d/chromatography-data-pre.sqlite
> system.posix_acl_access 132
> unique: 3954, error: -95 (Operation not supported), outsize: 16
> --
> unique: 3960, opcode: GETXATTR (22), nodeid: 71, insize: 72, pid: 3458
> getxattr /eimstims1/deleteme2/8020 MeOHH2O
> RecoverySample1-20221216-A-JJL-WebinarHilic10C-TOF-TT54-Neg-1632.d/chromatography-data-pre.sqlite
> system.posix_acl_access 132
> unique: 3960, error: -95 (Operation not supported), outsize: 16
>
>
> Thank you again!
>
> -Jonathan

2023-04-21 14:29:52

by Miklos Szeredi

[permalink] [raw]

Subject: Re: [PATCH] [REGRESSION] ovl: Handle ENOSYS when fileattr support is missing in lower/upper fs

Hi Jonathan,

Can you please try out the attached patch?

Thanks,
Miklos

On Wed, 22 Mar 2023 at 19:42, Jonathan Katz <[email protected]> wrote:
>
> Confirmed bindfs interaction:
>
> Based on your bindfs comment I redid my configuration as follows:
> ORIGINAL (FAILS):
> FS1 - exports "/Data" (nfs)
> FS2 - Mounts "/Data", does a bindfs, does an overlay
>
> TEST (SUCCEEDS):
> FS1 - does a bindfs and exports a series of directories:
> # bindfs -u 5007, -g 5007 /Data /Data-jiajun
> /etc/exports:
> /Data machine.org(ro,sync,no_subtree_check)
> /Data-jiajun machine.org(ro,fsid=12,sync,no_subtree_check)
> FS2 - used to do bindfs to make the lowers, but, now mounts
> "/Data-jiajun" as the lower
> FS2 then does the overlay and samba share.
> It would not let me do the 2nd export if I did not
> include the fsid entry....
>
> WOOT WOOT.
>
>
> Not an ideal solution as I have to make changes to 2 servers in order
> to accomplish my goal :/.
>
>
>
>
>
>
>
>
> On Tue, Mar 14, 2023 at 7:43 PM Jonathan Katz <[email protected]> wrote:
> >
> > On Thu, Mar 9, 2023 at 7:31 AM Miklos Szeredi <[email protected]> wrote:
> > >
> > > On Tue, 7 Mar 2023 at 18:14, Jonathan Katz <[email protected]> wrote:
> > > >
> > > > On Tue, Mar 7, 2023 at 12:38 AM Miklos Szeredi <[email protected]> wrote:
> > > > >
> > > > > On Tue, 7 Mar 2023 at 02:12, Jonathan Katz <[email protected]> wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > In pursuing this issue, I downloaded the kernel source to see if I
> > > > > > could debug it further. In so doing, it looks like Christian's patch
> > > > > > was never committed to the main source tree (sorry if my terminology
> > > > > > is wrong). This is up to and including the 6.3-rc1. I could also
> > > > > > find no mention of the fix in the log.
> > > > > >
> > > > > > I am trying to manually apply this patch now, but, I am wondering if
> > > > > > there was some reason that it was not applied (e.g. it introduces some
> > > > > > instability?)?
> > > > >
> > > > > It's fixing the bug in the wrong place, i.e. it's checking for an
> > > > > -ENOSYS return from vfs_fileattr_get(), but that return value is not
> > > > > valid at that point.
> > > > >
> > > > > The right way to fix this bug is to prevent -ENOSYS from being
> > > > > returned in the first place.
> > > > >
> > > > > Commit 02c0cab8e734 ("fuse: ioctl: translate ENOSYS") fixes one of
> > > > > those bugs, but of course it's possible that I missed something in
> > > > > that fix.
> > > > >
> > > > > Can you please first verify that an upstream kernel (>v6.0) can also
> > > > > reproduce this issue?
> > > >
> > > > Got ya. that makes a lot of sense, thank you.
> > > >
> > > > I have confirmed that I continue to get the error with 6.2 .
> > > > quick summary of the lowerdir:
> > > > server ---- NFS(ro) ---- > client "/nfs"
> > > > client "/nfs" --- bindfs(uidmap) --- > client "/lower"
> > >
> > > Can you please run bindfs in debugging mode (-d) and send the
> > > resulting log after reproducing the issue?
> > >
> > > Thanks,
> > > Miklos
> >
> > OUCH -- MY LAST EMAIL WAS REJECTED FOR BEING TOO BIG
> > I HOPE THAT I AM SUMMARIZING THE RELEVANT INFORMATION HERE:
> >
> >
> > Hi Miklos, thank you.... I am sorry for the delay.
> >
> > The log is somewhat long and was sent in a separate email.
> >
> > I broke up the log into entries to try to match the chronology of actions:
> > * ENTRY 1 nfs mount the external drive
> > * ENTRY 2 perform the bind fs
> > * ENTRY 3 perform the overlay
> > * ENTRY 4 restart smb
> > * ENTRY 5 mount the filesystem on a windows box
> > * ENTRY 6 performing some navigation on the windows file explorer
> > * ENTRY 7 attempt to open a data file with the windows application.
> >
> > The only place that generated a kernel error in dmesg was at ENTRY 7.
> >
> > Because the logs are so big, I tried to parse them, I may have made a
> > mistake or omitted information -- if you think so, as mentioned, the
> > full bindfs logs were sent separately
> >
> >
> > Here is my attempt to parse out the errors associated with this dmesg entry:
> >
> > [ 1925.705908] overlayfs: failed to retrieve lower fileattr (8020
> > MeOHH2O RecoverySample1-20221216-A-JJL-WebinarHilic10C-TOF-TT54-Neg-1632.d/chromatography-data.sqlite,
> > err=-38)
> >
> > --
> > unique: 1550, opcode: GETXATTR (22), nodeid: 71, insize: 73, pid: 3458
> > getxattr /eimstims1/deleteme2/8020 MeOHH2O
> > RecoverySample1-20221216-A-JJL-WebinarHilic10C-TOF-TT54-Neg-1632.d/chromatography-data-pre.sqlite
> > trusted.overlay.metacopy 0
> > unique: 1550, error: -95 (Operation not supported), outsize: 16
> > --
> > unique: 3922, opcode: GETXATTR (22), nodeid: 71, insize: 72, pid: 3458
> > getxattr /eimstims1/deleteme2/8020 MeOHH2O
> > RecoverySample1-20221216-A-JJL-WebinarHilic10C-TOF-TT54-Neg-1632.d/chromatography-data-pre.sqlite
> > system.posix_acl_access 132
> > unique: 3922, error: -95 (Operation not supported), outsize: 16
> > --
> > unique: 3954, opcode: GETXATTR (22), nodeid: 71, insize: 72, pid: 3458
> > getxattr /eimstims1/deleteme2/8020 MeOHH2O
> > RecoverySample1-20221216-A-JJL-WebinarHilic10C-TOF-TT54-Neg-1632.d/chromatography-data-pre.sqlite
> > system.posix_acl_access 132
> > unique: 3954, error: -95 (Operation not supported), outsize: 16
> > --
> > unique: 3960, opcode: GETXATTR (22), nodeid: 71, insize: 72, pid: 3458
> > getxattr /eimstims1/deleteme2/8020 MeOHH2O
> > RecoverySample1-20221216-A-JJL-WebinarHilic10C-TOF-TT54-Neg-1632.d/chromatography-data-pre.sqlite
> > system.posix_acl_access 132
> > unique: 3960, error: -95 (Operation not supported), outsize: 16
> >
> >
> > Thank you again!
> >
> > -Jonathan

Attachments:

fuse-ioctl-translate-enosys-in-outarg.patch (2.33 kB)