Hello Miklos, Amir
We are having some problems with inotify + overlayfs.
If we start to monitor a directory on an overlayfs,
and get into a low memory situation, or if
/proc/sys/vm/drop_caches is called explicitly,
our inotify stops reporting events.
This appears to be caused by non-persistent inodes.
Looking at the commits, it looked like inode numbers
were persistent since commit
b7a807dc2010 ("ovl: persistent inode number for directories").
However, this doesn't seem to take different file systems
into account:
# cat /proc/version
Linux version 4.15.0-rc7-next-20180109-00003-gd1acc32c9ee5 (niklass@lnxartpec1)
(gcc version 6.3.0 (crosstool-NG crosstool-ng-1.22.0-610-g21cde94)) #74
SMP Fri Jan 12 13:23:57 CET 2018
# mkdir -p /test/dir /mnt/tmp
# mount -t tmpfs tmpfs /mnt/tmp
# mkdir /mnt/tmp/.work-test /mnt/tmp/test
# mount -t overlay overlay /test -o lowerdir=/test,upperdir=/mnt/tmp/test,
workdir=/mnt/tmp/.work-test
# ls -lid /test/dir
6313 drwxr-xr-x 2 root root 40 Jan 1 00:01 /test/dir
# echo 3 > /proc/sys/vm/drop_caches
[ 229.762380] sh (118): drop_caches: 3
# ls -lid /test/dir
6318 drwxr-xr-x 2 root root 40 Jan 1 00:01 /test/dir
There are probably a few user space applications that
rely on inotify. If inotify suddenly and silently stops
reporting events after some memory pressure, all kinds
of bugs could be introduced in user space.
I'm guessing that it will not be so easy to figure out
that overlayfs might be the root-cause of their
seemingly random bug.
Regards,
Niklas
On Sat, Jan 13, 2018 at 2:19 AM, Niklas Cassel <[email protected]> wrote:
> Hello Miklos, Amir
>
> We are having some problems with inotify + overlayfs.
>
> If we start to monitor a directory on an overlayfs,
> and get into a low memory situation, or if
> /proc/sys/vm/drop_caches is called explicitly,
> our inotify stops reporting events.
>
> This appears to be caused by non-persistent inodes.
Not "non-persistent" inodes, but "non-unique" inodes.
inotify pins overlay inode in memory, but because overlay
directory inodes are not hashed in inode cache, after drop
caches, if nothing holds a reference to overlay dentry,
overlay lookup will allocate a new dentry with a new inode
instead of re-using the inotify pinned inode.
>
> Looking at the commits, it looked like inode numbers
> were persistent since commit
> b7a807dc2010 ("ovl: persistent inode number for directories").
>
Persistent inode *number*, meaning that all non-unique overlay
directory inodes have the same inode number - doesn't help
inotify.
> However, this doesn't seem to take different file systems
> into account:
>
Are you saying that inotify issue is not happening when layers
are on same fs? I doubt that.
Please try attached patch.
Thanks,
Amir.