Hi Linus,
Please pull from:
git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git tags/ovl-update-4.18
This contains two new features:
1) Stack file operations: this allows removal of several hacks from the
VFS, proper interaction of read-only open files with copy-up,
possibility to implement fs modifying ioctls properly, and others.
2) Metadata only copy-up: when file is on lower layer and only metadata is
modified (except size) then only copy up the metadata and continue to
use the data from the lower file.
The series starts with a cleanup of the internal dedupe API. There's some
late discussion on details (should vfs limit the size of a dedepe request,
and if yes, how much). I've ignored it for this pull request, it can
easily be fixed later.
Other pain point: overlay doesn't want to double account open files (due to
stacking) for fear of breaking existing setups. So added infrastruture
that allows to skip accounting an open file in nr_files. I don't much like
this, but can't see any other way of keeping backward compatibility.
There are two conflicts when merging, attaching my resolution.
Thanks,
Miklos
---
Miklos Szeredi (37):
vfs: dedupe: return loff_t
vfs: dedupe: rationalize args
vfs: dedupe: extract helper for a single dedup
vfs: add path_open()
vfs: optionally don't account file in nr_files
vfs: export vfs_ioctl() to modules
vfs: export vfs_dedupe_file_range_one() to modules
ovl: copy up times
ovl: copy up inode flags
Revert "Revert "ovl: get_write_access() in truncate""
ovl: copy up file size as well
ovl: deal with overlay files in ovl_d_real()
ovl: stack file ops
ovl: add helper to return real file
ovl: add ovl_read_iter()
ovl: add ovl_write_iter()
ovl: add ovl_fsync()
ovl: add ovl_mmap()
ovl: add ovl_fallocate()
ovl: add lsattr/chattr support
ovl: add ovl_fiemap()
ovl: add O_DIRECT support
ovl: add reflink/copyfile/dedup support
vfs: don't open real
ovl: obsolete "check_copy_up" module option
ovl: fix documentation of non-standard behavior
vfs: simplify dentry_open()
Revert "ovl: fix may_write_real() for overlayfs directories"
Revert "ovl: don't allow writing ioctl on lower layer"
vfs: fix freeze protection in mnt_want_write_file() for overlayfs
Revert "ovl: fix relatime for directories"
Revert "vfs: update ovl inode before relatime check"
Revert "vfs: add flags to d_real()"
Revert "vfs: do get_write_access() on upper layer of overlayfs"
Partially revert "locks: fix file locking on overlayfs"
Revert "fsnotify: support overlayfs"
vfs: remove open_flags from d_real()
Vivek Goyal (28):
ovl: Initialize ovl_inode->redirect in ovl_get_inode()
ovl: Move the copy up helpers to copy_up.c
ovl: Provide a mount option metacopy=on/off for metadata copyup
ovl: During copy up, first copy up metadata and then data
ovl: Copy up only metadata during copy up where it makes sense
ovl: Add helper ovl_already_copied_up()
ovl: A new xattr OVL_XATTR_METACOPY for file on upper
ovl: Use out_err instead of out_nomem
ovl: Modify ovl_lookup() and friends to lookup metacopy dentry
ovl: Copy up meta inode data from lowest data inode
ovl: Add helper ovl_dentry_lowerdata() to get lower data dentry
ovl: Fix ovl_getattr() to get number of blocks from lower
ovl: Store lower data inode in ovl_inode
ovl: Add helper ovl_inode_realdata()
ovl: Open file with data except for the case of fsync
ovl: Do not expose metacopy only dentry from d_real()
ovl: Move some dir related ovl_lookup_single() code in else block
ovl: Check redirects for metacopy files
ovl: Treat metacopy dentries as type OVL_PATH_MERGE
ovl: Add an inode flag OVL_CONST_INO
ovl: Do not set dentry type ORIGIN for broken hardlinks
ovl: Set redirect on metacopy files upon rename
ovl: Set redirect on upper inode when it is linked
ovl: Check redirect on index as well
ovl: add helper to force data copy-up
ovl: Do not do metadata only copy-up for truncate operation
ovl: Do not do metacopy only for ioctl modifying file attr
ovl: Enable metadata only feature
---
Documentation/filesystems/Locking | 3 +-
Documentation/filesystems/overlayfs.txt | 90 ++++--
Documentation/filesystems/vfs.txt | 16 +-
fs/btrfs/ctree.h | 5 +-
fs/btrfs/ioctl.c | 7 +-
fs/file_table.c | 13 +-
fs/inode.c | 46 +--
fs/internal.h | 17 +-
fs/ioctl.c | 1 +
fs/locks.c | 20 +-
fs/namei.c | 2 +-
fs/namespace.c | 69 +----
fs/ocfs2/file.c | 10 +-
fs/open.c | 87 +++---
fs/overlayfs/Kconfig | 19 ++
fs/overlayfs/Makefile | 4 +-
fs/overlayfs/copy_up.c | 190 ++++++++----
fs/overlayfs/dir.c | 105 +++++--
fs/overlayfs/export.c | 3 +
fs/overlayfs/file.c | 508 ++++++++++++++++++++++++++++++++
fs/overlayfs/inode.c | 175 +++++++----
fs/overlayfs/namei.c | 195 +++++++-----
fs/overlayfs/overlayfs.h | 47 ++-
fs/overlayfs/ovl_entry.h | 6 +-
fs/overlayfs/super.c | 103 ++++---
fs/overlayfs/util.c | 252 +++++++++++++++-
fs/read_write.c | 91 +++---
fs/xattr.c | 9 +-
fs/xfs/xfs_file.c | 8 +-
include/linux/dcache.h | 15 +-
include/linux/fs.h | 31 +-
include/linux/fsnotify.h | 14 +-
include/uapi/linux/fs.h | 1 -
33 files changed, 1590 insertions(+), 572 deletions(-)
create mode 100644 fs/overlayfs/file.c
On Fri, Jun 08, 2018 at 02:13:30PM +0200, Miklos Szeredi wrote:
> Hi Linus,
>
> Please pull from:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git tags/ovl-update-4.18
>
> This contains two new features:
>
> 1) Stack file operations: this allows removal of several hacks from the
> VFS, proper interaction of read-only open files with copy-up,
> possibility to implement fs modifying ioctls properly, and others.
Which includews all kinds of NAKed or at least non-acked VFS changes.
Please get these through Als tree after proper review first.
Hmm.
So I had held off on pulling this in the hope that it would get more comments.
I like most of the vfs-level stuff - it gets rid of some of the
hackery we had for overlayfs.
It does add some new hackery to replace it (like the file accounting), though.
And Christoph's copmmentary isn't really helping the situation.
Christoph, I haven't seen the NAK history, can you elaborate?
And Al, can you please take a look?
Linus
On Fri, Jun 8, 2018 at 11:52 PM Christoph Hellwig <[email protected]> wrote:
>
> > This contains two new features:
> >
> > 1) Stack file operations: this allows removal of several hacks from the
> > VFS, proper interaction of read-only open files with copy-up,
> > possibility to implement fs modifying ioctls properly, and others.
>
> Which includews all kinds of NAKed or at least non-acked VFS changes.
>
> Please get these through Als tree after proper review first.
On Sat, Jun 09, 2018 at 02:42:20PM -0700, Linus Torvalds wrote:
> Hmm.
>
> So I had held off on pulling this in the hope that it would get more comments.
>
> I like most of the vfs-level stuff - it gets rid of some of the
> hackery we had for overlayfs.
>
> It does add some new hackery to replace it (like the file accounting), though.
>
> And Christoph's copmmentary isn't really helping the situation.
> Christoph, I haven't seen the NAK history, can you elaborate?
>
> And Al, can you please take a look?
Will post tonight...
On Fri, Jun 08, 2018 at 11:52:08PM -0700, Christoph Hellwig wrote:
> On Fri, Jun 08, 2018 at 02:13:30PM +0200, Miklos Szeredi wrote:
> > Hi Linus,
> >
> > Please pull from:
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git tags/ovl-update-4.18
> >
> > This contains two new features:
> >
> > 1) Stack file operations: this allows removal of several hacks from the
> > VFS, proper interaction of read-only open files with copy-up,
> > possibility to implement fs modifying ioctls properly, and others.
>
> Which includews all kinds of NAKed or at least non-acked VFS changes.
Umm... The worst of yours had been ->pre_mmap(), right? He *did* drop that...
> Please get these through Als tree after proper review first.
OK, summary of sort (see fsdevel thread for details):
* path_open() is dubious; why not simply use vfsmount/dentry from the
right layer when opening an underlying file? Then it would be vfs_open()...
* ovl_mmap() is broken, plain and simple. Failure ends up leaking
a layer struct file *and* doing double fput() on overlayfs one.
* ovl_mmap() is also trivially DoSable - you can trigger tons and tons
of reopens, each sticking a new (writable layer) struct file into a vma.
We *do* want some scheme avoiding once-per-operations reopens in the
copied-up-after-r/o-open case. See possible kinda-sorta solution on fsdevel;
I'm not sure I like it, though.
The rest is pretty minor.
On Sat, Jun 09, 2018 at 02:42:20PM -0700, Linus Torvalds wrote:
> And Christoph's copmmentary isn't really helping the situation.
> Christoph, I haven't seen the NAK history, can you elaborate?
Most of the bits just need a bit of refinement I think, instead of
being forced through the overlayfs tree and are generally fine.
The pre_mmap hook I think is a clear no-go. We've had this tried
multiple times and always rejected it. Unlike previous uses the
overlayfs use isn't outright broken, but still questionalable
as it will still lead to a copyup that "leaks" if the actual mmap
wasn't successfull. The whole discussion of how mmap happens,
how we deal with mmap_sem and failures needs a broader discussion
with all MM and VFS folks first.
On Sun, Jun 10, 2018 at 06:54:38AM +0100, Al Viro wrote:
> Umm... The worst of yours had been ->pre_mmap(), right? He *did* drop that...
Oh, hadn't notied that. Still odd to change a huge pull request after
the end merge window.
On Sun, Jun 10, 2018 at 7:54 AM, Al Viro <[email protected]> wrote:
> On Fri, Jun 08, 2018 at 11:52:08PM -0700, Christoph Hellwig wrote:
>> On Fri, Jun 08, 2018 at 02:13:30PM +0200, Miklos Szeredi wrote:
>> > Hi Linus,
>> >
>> > Please pull from:
>> >
>> > git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git tags/ovl-update-4.18
>> >
>> > This contains two new features:
>> >
>> > 1) Stack file operations: this allows removal of several hacks from the
>> > VFS, proper interaction of read-only open files with copy-up,
>> > possibility to implement fs modifying ioctls properly, and others.
>>
>> Which includews all kinds of NAKed or at least non-acked VFS changes.
>
> Umm... The worst of yours had been ->pre_mmap(), right? He *did* drop that...
>
>> Please get these through Als tree after proper review first.
>
> OK, summary of sort (see fsdevel thread for details):
> * path_open() is dubious; why not simply use vfsmount/dentry from the
> right layer when opening an underlying file? Then it would be vfs_open()...
> * ovl_mmap() is broken, plain and simple. Failure ends up leaking
> a layer struct file *and* doing double fput() on overlayfs one.
> * ovl_mmap() is also trivially DoSable - you can trigger tons and tons
> of reopens, each sticking a new (writable layer) struct file into a vma.
> We *do* want some scheme avoiding once-per-operations reopens in the
> copied-up-after-r/o-open case. See possible kinda-sorta solution on fsdevel;
> I'm not sure I like it, though.
Al, thanks for the review.
Posted incremental for the ovl_mmap() issues to -fsdevel. I'm pretty
confident that that addresses your comments.
Linus, would you still pull if Al's satisfied with that resolution?
I can post the fixes (just few liners) after the merge window.
I'm definitely not going to prepare another pull request this cycle if
the old one cannot be pulled.
Thanks,
Miklos
On Mon, Jun 11, 2018 at 10:41:29AM +0200, Miklos Szeredi wrote:
> Linus, would you still pull if Al's satisfied with that resolution?
> I can post the fixes (just few liners) after the merge window.
Please repost all changes outside overlayfs itself to -fsdevel so
that we can improve them for merging in the next merge window instead
of hasting them in now.