Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 17 Jun 2022 18:12:59 -0400
From:   "Theodore Ts'o" <tytso@mit.edu>
To:     Santosh S <santosh.letterz@gmail.com>
Cc:     linux-ext4@vger.kernel.org
Subject: Re: Overwrite faster than fallocate
Message-ID: <Yqz8a0ggTjIU3h7T@mit.edu>
References: <CAGQ4T_Jne-bxdP9rMNBzqXw16a4kD4FM=F5VuGgUbczj5WgCLA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAGQ4T_Jne-bxdP9rMNBzqXw16a4kD4FM=F5VuGgUbczj5WgCLA@mail.gmail.com>
Precedence: bulk

On Fri, Jun 17, 2022 at 12:38:20PM -0400, Santosh S wrote:
> Dear ext4 developers,
> 
> This is my test - preallocate a large file (2G) and then do sequential
> 4K direct-io writes to that file, with fdatasync after every write.
> I am preallocating using fallocate mode 0. I noticed that if the 2G
> file is pre-written rather than fallocate'd I get more than twice the
> throughput. I could reproduce this with fio. The storage is nvme.
> Kernel version is 5.3.18 on Suse.
>
> Am I doing something wrong or is this difference expected? Any
> suggestion to get a better throughput without actually pre-writing the
> file.

This is, alas, expected.  The reason for this is because when you use
fallocate, the extent is marked as uninitialized, so that when you
read from the those newly allocated blocks, you don't see previously
written data belonging to deleted files.  These files could contain
someone else's e-mail, or medical information, etc.  So if we didn't
do this, it would be a walking, talking HIPPA or PCI violation.

So when you write to an fallocated region, and then call fdatasync(2),
we need to update the metadata blocks to clear the uninitialized bit
so that when you read from the file after a crash, you actually get
the data that was written.  So the fdatasync(2) operation is quite the
heavyweight operation, since it requries journal commit because of the
required metadata update.  When you do an overwrite, there is no need
to force a metadata update and journal update, which is why write(2)
plus fdatasync(2) is much lighter weight when you do an overwrite.

What enterprise databases (e.g., Oracle Enterprise Database and IBM's
Informix DB) tend to do is to use fallocate a chunk of space (say,
16MB or 32MB), because for Legacy Unix OS's, this tends enable some
file system's block allocators to be more likely to allocate a
contiguous block range, and then immediate write zero's on that 16 or
32MB, plus a fdatasync(2).  This fdatasync(2) would update the extent
tree once to make that 16MB or 32MB to be marked initialized to the
database's tablespace file, so you only pay the metadata update once,
instead of every few dozen kilobytes as you write each database commit
into the tablespace file.

There is also an old, out of tree patch which enables an fallocate
mode called "no hide stale", which marks the extent tree blcoks which
are allocated using fallocate(2) as initialized.  This substantially
speeds things up, but it is potentially a walking, talking, HIPPA or
PCI violation in that revealing previously written data is considered
a horrible security violation by most file system developers.

If you know, say, that a cluster file system is the only user of the
file system, and all data is written encrypted at rest using a
per-user key, such that exposing stale data is not a security
disaster, the "no hide stale" flag could be "safe" in that highly
specialized user case.

But that assumes that file system authors can trust application
writers not to do something stupid and insecure, and historically,
file system authors (possibly with good reason, given bitter past
experience) don't trust application writesr to do something which is
very easy, and gooses performance, even if it has terrible side
effects on either data robustness or data security.

Effectively, the no hide stale flag could be considered an "Attractive
Nuisance"[1] and so support for this feature has never been accepted
into the mainline kernel, and never to any distro kernels, since the
distribution companies don't want to be held liable for making an
"acctive nuisance" that might enable application authors from shooting
themselves in the foot.

[1] https://en.wikipedia.org/wiki/Attractive_nuisance_doctrine

In any case, the technique of fallocatE(2) plus zero-fill-write plus
fdatasync(2) isn't *that* slow, and is only needed when you are first
extending the tablespace file.  In the steady state, most database
applications tend to be overwriting space, so this isn't an issue.

In any case, if you need to get that last 5% or so of performance ---
say, if you are are an enterprise database company interested in
taking a full page advertisement on the back cover of Business Week
Magazine touting how your enterprise database benchmarks are better
than the competition --- the simple solution is to use a raw block
device.  Of course, most end users want the convenience of the file
system, but that's not the point if you are engaging in
benchmarketing.   :-)

Cheers,

						- Ted