Received-SPF: pass (google.com: best guess record for domain of linux-ext4-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Andreas Dilger <adilger@dilger.ca>
Message-Id: <E02E44A9-6206-4B73-B52F-C3A1BC4C7D1E@dilger.ca>
Content-Type: multipart/signed;
 boundary="Apple-Mail=_8B0EEE79-E2B0-48DD-B6D5-D659ABF991D0";
 protocol="application/pgp-signature"; micalg=pgp-sha256
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Re: [RFC] improve malloc for large filesystems
Date:   Mon, 25 Nov 2019 14:39:59 -0700
In-Reply-To: <BCFC8274-0A4E-42E7-9D11-647D47316BD2@whamcloud.com>
Cc:     "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>
To:     Alex Zhuravlev <azhuravlev@whamcloud.com>,
        =?utf-8?B?0JHQu9Cw0LPQvtC00LDRgNC10L3QutC+INCQ0YDRgtGR0Lw=?= 
        <artem.blagodarenko@gmail.com>
References: <8738E8FF-820F-48A5-9150-7FF64219ED42@whamcloud.com>
 <20191120181353.GG4262@mit.edu>
 <9E04C147-D878-45CE-8473-EF8C67FE4E86@whamcloud.com>
 <4EB2303A-01A3-49AC-B713-195126DB621B@gmail.com>
 <9114E776-B44E-4CA5-BD49-C432A688C24E@whamcloud.com>
 <43DA6456-AAF9-4225-A79F-CF632AC5241B@gmail.com>
 <BCFC8274-0A4E-42E7-9D11-647D47316BD2@whamcloud.com>
Sender: linux-ext4-owner@vger.kernel.org
Precedence: bulk


--Apple-Mail=_8B0EEE79-E2B0-48DD-B6D5-D659ABF991D0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

On Nov 21, 2019, at 7:41 AM, Alex Zhuravlev <azhuravlev@whamcloud.com> =
wrote:
>=20
> On 21 Nov 2019, at 12:18, Artem Blagodarenko =
<artem.blagodarenko@gmail.com> wrote:
>> Assume we have one fragmented part of disk and all other parts are =
quite free.
>> Allocator will spend a lot of time to go through this fragmented =
part, because
>> will brake cr0 and cr1 and get range that satisfy c3.
>=20
> Even at cr=3D3 we still search for the goal size.
>=20
> Thus we shouldn=E2=80=99t really allocate bad chunks because we break =
cr=3D0 and cr=3D1,
> we just stop to look for nicely looking groups and fallback to regular =
(more
> expensive) search for free extents.

I think it is important to understand what the actual goal size is at =
this
point.  The filesystems where we are seeing problems are _huge_ (650TiB =
and
larger) and are relatively full (70% or more) but take tens of minutes =
to
finish mounting.  Lustre does some small writes at mount time, but it =
shouldn't
take so long to find some small allocations for the config log update.

The filesystems are automatically getting "s_stripe_size =3D 512" from =
mke2fs
(presumably from the underlying RAID), and I _think_ this is causing =
mballoc
to inflate the IO request to 8-16MB prealloc chunks, which would be much
harder to find, and unnecessary for a small allocation.

>> c3 requirement is quite simple =E2=80=9Cget first group that have =
enough free
>> blocks to allocate requested range=E2=80=9D.
>=20
> This is only group selection, then we try to find that extent within =
that
> group, can fail and move to the next group.
> EXT4_MB_HINT_FIRST is set outside of the main cr=3D0..3 loop.
>=20
>> With hight probability allocator find such group at the start of c3 =
loop,
>> so goal (allocator starts its searching from goal) will not =
significantly
>> changed. Thus allocator go through this fragmented range using small =
steps.
>>=20
>> Without suggested optimisation, allocator skips this fragmented range =
at
>> moment and continue to allocate blocks.
>=20
> 1000 groups * 5ms avg.time =3D 5 seconds to skip 1000 bad =
uninitialized groups. This is the real problem. You mentioned 4M =
groups...

Yes, these filesystems have 5M or more groups, which is a real problem.
Alex is working on a patch to do prefetch of the bitmaps, and to read =
them
in chunks of flex_bg size (256 blocks =3D 1MB) to cut down on the number =
of
seeks needed to fetch them from disk.

Using bigalloc would also help, and getting the number of block groups =
lower
will avoid the need for meta_bg (which puts each group descriptor into a
separate group, rather than packed contiguously)  but we've had to fix a =
few
performance issues with bigalloc as well, and have not deployed it yet =
in
production.

Cheers, Andreas


--Apple-Mail=_8B0EEE79-E2B0-48DD-B6D5-D659ABF991D0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIzBAEBCAAdFiEEDb73u6ZejP5ZMprvcqXauRfMH+AFAl3cSi8ACgkQcqXauRfM
H+A0fA//a4LOcDF3Nva6VDbKoBv0X0tSp9c3rrfR6TW4IiT3AxVivcwXnrKeYnc1
LuP3glPgkGRLJnthfuzhO7WuuVXdor27gdMqQqix3RhFOajWwR8Vtx5OEiHBMIfX
uf4jx9DWOHskCFdbBcL6p3q/yXGc0YcMhAVs0lLs05dGIQxnJjNfSTAHoDts3TOm
4G8pQVeipL3u4JK+v+omgzVCDEkZsstb2OydMa81nEqnx5etb7rkPBxcVB59yQNh
MVowNVuFk3OhK7DY9WgcpwmtyvXScaylqcl5RBVUnW3Euqe/RArhy5+H0iIhtd8v
BSchWvrHmOXBITRDTJOv1s+RyRVPL2JDdM2J47vS0gpsB0URwZ7ATXDeCD5sZoUr
yfgMv/BhGXz0o6EO5BYdZpPPbNt79xUAPgxIwAhnmGly6ACyLXPCXqFCEx27+77O
nikfAwEIJzjVmtX9YOlqUKSabTlRyDD2BFI/iV8owgKtossgeEZD3ydsgAQjoV7m
7pMuySO6y8Rfj/VI1JWtPsX3JDskB+m/gJ33yQslRTTkntRpaRwbooIKGGgQOn8d
b6nINYvnfWYwMEbJi9LMHtRtkSvfkunCzqXooZj2L2i3xhPCRP68nIIkKb6xVaI0
mmSeYjmQlkGTB22se6cGk1lpUbmJSUMFSI/916SuJ/XoExSr7cY=
=1Q9d
-----END PGP SIGNATURE-----

--Apple-Mail=_8B0EEE79-E2B0-48DD-B6D5-D659ABF991D0--