Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32;
Date:   Fri, 13 Oct 2023 12:14:24 +0200
From:   Jan Kara <jack@suse.cz>
To:     Dave Chinner <david@fromorbit.com>
Cc:     Jan Kara <jack@suse.cz>, Ted Tso <tytso@mit.edu>,
        linux-ext4@vger.kernel.org,
        "Ritesh Harjani (IBM)" <ritesh.list@gmail.com>,
        Gao Xiang <hsiangkao@linux.alibaba.com>
Subject: Re: [PATCH] ext4: Properly sync file size update after O_SYNC direct
 IO
Message-ID: <20231013101424.ctn5zfpntu4kjyl4@quack3>
References: <20231011142155.19328-1-jack@suse.cz>
 <ZSc9J9zFChyxl1U2@dread.disaster.area>
 <20231012085937.pzfsttumi6q4g3tm@quack3>
 <ZSiCNPAFuKZbmFPg@dread.disaster.area>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZSiCNPAFuKZbmFPg@dread.disaster.area>
Precedence: bulk

On Fri 13-10-23 10:33:08, Dave Chinner wrote:
> On Thu, Oct 12, 2023 at 10:59:37AM +0200, Jan Kara wrote:
> > On Thu 12-10-23 11:26:15, Dave Chinner wrote:
> > > On Wed, Oct 11, 2023 at 04:21:55PM +0200, Jan Kara wrote:
> > > > Gao Xiang has reported that on ext4 O_SYNC direct IO does not properly
> > > > sync file size update and thus if we crash at unfortunate moment, the
> > > > file can have smaller size although O_SYNC IO has reported successful
> > > > completion. The problem happens because update of on-disk inode size is
> > > > handled in ext4_dio_write_iter() *after* iomap_dio_rw() (and thus
> > > > dio_complete() in particular) has returned and generic_file_sync() gets
> > > > called by dio_complete(). Fix the problem by handling on-disk inode size
> > > > update directly in our ->end_io completion handler.
> > > > 
> > > > References: https://lore.kernel.org/all/02d18236-26ef-09b0-90ad-030c4fe3ee20@linux.alibaba.com
> > > > Reported-by: Gao Xiang <hsiangkao@linux.alibaba.com>
> > > > Signed-off-by: Jan Kara <jack@suse.cz>
> > > > ---
> > > >  fs/ext4/file.c | 139 ++++++++++++++++++-------------------------------
> > > >  1 file changed, 52 insertions(+), 87 deletions(-)
> > > .....
> > > > @@ -388,9 +342,28 @@ static ssize_t ext4_handle_inode_extension(struct inode *inode, loff_t offset,
> > > >  		 */
> > > >  		if (inode->i_nlink)
> > > >  			ext4_orphan_del(NULL, inode);
> > > > +		return;
> > > >  	}
> > > > +	/*
> > > > +	 * If i_disksize got extended due to writeback of delalloc blocks while
> > > > +	 * the DIO was running we could fail to cleanup the orphan list in
> > > > +	 * ext4_handle_inode_extension(). Do it now.
> > > > +	 */
> > > > +	if (!list_empty(&EXT4_I(inode)->i_orphan) && inode->i_nlink) {
> > > > +		handle_t *handle = ext4_journal_start(inode, EXT4_HT_INODE, 2);
> > > 
> > > So this has to be called after the DIO write completes and calls
> > > ext4_handle_inode_extension()?
> > 
> > Yes, if the write was setup as extending one ('extend' is set to true in
> > ext4_dio_write_iter()).
> 
> Then that is worth a comment to document the constraint for anyone
> that is trying to understand how ext4 is using the iomap DIO code.

Fair enough, comment added.

> > > > @@ -606,9 +570,8 @@ static ssize_t ext4_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
> > > >  			   dio_flags, NULL, 0);
> > > >  	if (ret == -ENOTBLK)
> > > >  		ret = 0;
> > > > -
> > > >  	if (extend)
> > > > -		ret = ext4_handle_inode_extension(inode, offset, ret, count);
> > > > +		ext4_inode_extension_cleanup(inode, ret);
> > > 
> > > Because this doesn't wait for AIO DIO to complete and actually
> > > extend the file before running the cleanup code...
> > 
> > As Gao wrote, ext4 sets IOMAP_DIO_FORCE_WAIT if 'extend' is set (see
> > ext4_dio_write_checks()) so if we get to calling
> > ext4_inode_extension_cleanup() we are guaranteed the IO has already
> > completed.
> 
> Ugh. That's a pretty nasty undocumented landmine. It definitely
> needs a comment (or better, a WARN_ON_ONCE()) to document that this
> code -only- works if AIO is disabled. This isn't for ext4
> developers, it's for people working on the iomap code to understand
> that ext4 has some really non-obvious constraints in it's DIO code
> paths and that's why the landmine is not being stepped on....

OK. Ext4 has always been this way so I never felt the need to document it
but you're right. I've added:

	if (extend) {
                /*
                 * We always perform extending DIO write synchronously so by
                 * now the IO is completed and ext4_handle_inode_extension()
                 * was called. Cleanup the inode in case of error or race with
                 * writeback of delalloc blocks.
                 */
                WARN_ON_ONCE(ret == -EIOCBQUEUED);
                ext4_inode_extension_cleanup(inode, ret);
        }

Thanks for the suggestions!

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR