Received-SPF: pass (google.com: domain of linux-kernel+bounces-147814-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <8360ec33-65f1-497f-8230-665b4328e1c0@valvesoftware.com>
Date: Tue, 16 Apr 2024 18:04:30 -0700
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Increase Default vm_max_map_count to Improve Compatibility with
 Modern Games
To: "Liam R. Howlett" <Liam.Howlett@oracle.com>, David Hildenbrand
	<david@redhat.com>, Oleksandr Natalenko <oleksandr@natalenko.name>,
	<linux-kernel@vger.kernel.org>, <vincentdelor@free.fr>, Andrew Morton
	<akpm@linux-foundation.org>, <linux-mm@kvack.org>
References: <566168554.272637693.1710968734203.JavaMail.root@zimbra54-e10.priv.proxad.net>
 <13499186.uLZWGnKmhe@natalenko.name>
 <1a91e772-4150-4d28-9c67-cb6d0478af79@redhat.com>
 <ugjnhfiwuoslupkbmrrbbxxdosotzpqjfljebzj4dkuibmf6sr@kuew2qemsen4>
 <8f6e2d69-b4df-45f3-aed4-5190966e2dea@valvesoftware.com>
 <ht35j3ig4reppggvpcjpjc7gotg6ndrjwvwfiqvxrwrtr4iauh@kw46hnsliy5l>
From: "Pierre-Loup A. Griffais" <pgriffais@valvesoftware.com>
In-Reply-To: <ht35j3ig4reppggvpcjpjc7gotg6ndrjwvwfiqvxrwrtr4iauh@kw46hnsliy5l>
Content-Language: en-US
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable


On 4/15/24 12:57 PM, Liam R. Howlett wrote:
> * Pierre-Loup A. Griffais <pgriffais@valvesoftware.com> [240414 20:22]:
>>
>>
> ...
>=20
>>>
>>> To be clear, what you are doing here is akin to adding more memory to
>>> your system when there is a memory leak.  This is not the solution you
>>> should be pushing.  Ironically, this is using more memory and performin=
g
>>> worse than it should.  At best, the limit increase is a workaround for
>>> buggy programs.
>>>
>>> At worst, you are enabling bad things to keep happening and normalising
>>> poor programming choices.  Please put pressure on the applications that
>>> clearly have issues.
>>
>> We don't get to prescribe what those applications do. The fact of the ma=
tter
>> is that there are several high-performance memory allocators in wide use=
 by
>> game applications that make heavy internal use of mmap(), and that using
>> hundreds of thousands of different memory mappings is well supported on =
the
>> platform those applications were written for. (or mapping regions with
>> different permissions, which results in different regions after platform
>> translation to Linux happens within Wine)
>=20
> Thank you for the information on the situation that causes the kernel to
> use such a large number of vmas.
>=20
> The mmap operations will run faster if there are significantly less
> vmas.  Having such a large number of objects will cause the faulting of
> information into the memory to be slower, and that would hold true for
> all platforms.
>=20
> If this is for high-performance, then it would be unlikely that it was
> designed to run with 65,530 objects to search.  It is also odd that
> there are several allocators running into the same issue. If I were to
> guess, the allocators are trying to bypass the operating systems use of
> memory and implement another way of tracking it specific to your usecase
> for speed.  It sounds like it is being translated incorrectly and
> causing a monster data structure to track it on the kernel side.
>=20
> If it's a translation layer in wine making a decision on how to
> translate a particular set of calls then it could be fixed, or at least
> examined for inefficiencies.

I mentioned translation because it can play a role if the original=20
mappings contain regions with different permissions, as it would need to=20
translate those into several different mappings on Linux, but I wouldn't=20
expect it's really having a meaningful effect. By and large, I think=20
those mappings are coming as-is through the app.

>=20
> Either way, the performance will be sub-optimal on the page fault path
> (probably the most common) and any other path that uses such a large
> number of vmas.
>=20
>>
>> Pointing out that there exists one game that doesn't happen to do that i=
s
>> not terribly useful for the purpose of this discussion.
>=20
> I provided the data I could collect reasonably quickly, but the scale of
> the difference was the important part of my statement.
>=20
>>
>> The problem statement seems pretty simple - distributions that want to
>> support those usecases out of the box can make that change, like we've d=
one
>> for years on SteamOS. On those that don't, users of those applications w=
ill
>> have to discover and learn to apply the change by hand after having a li=
kely
>> sub-par experience trying to get their game up and running.
>=20
> This number of vmas is indicating an issue with the utilisation of the
> virtual memeory areas.  Increasing the limit is allowing the game to
> run, but it will not be performant. It is unfortunate that the solution
> was to increase the value.

Games don't necessarily care if mmap() (and ensuing faults) is a bit=20
slower than the fastest case. Doing such an operation is already=20
considered a relatively slow path and would likely happen on a resource=20
loading thread instead of the hot main loop.

>=20
>>
>> I've yet to hear a specific downside of making the change other than a r=
eal
>> concern about DoS of kernel memory in another discussion - it seems to m=
e
>> like there is much lower hanging fruit for DoSing a Linux system you hav=
e
>> shell access to, at the moment.
>=20
> Poor performance is the downside. The specific downside is the overly
> large data structure that the kernel has to navigate on every page fault
> or any other vma operation.  This isn't specific to changing the number,
> but to the fact that it needed to be changed in the first place.
>=20
> Is there an upper limit of vmas that you have seen?  Can you provide a
> copy of the mappings when you see this for testing?  This works out to a
> 5 level maple tree.

I don't really know of an upper limit. I can provide a contrasting=20
anecdote that seems to use a fair amount of mappings - running the title=20
`Hogwarts Legacy` after having loaded into interactive gameplay in the=20
initial area:

plagman@redcore:~$ cat /proc/2009007/maps | wc -l
27217

Here's a copy of /proc/maps if you're curious:

https://www.dropbox.com/scl/fi/rf970vdxoexsx8u1otufl/hogwarts_maps?rlkey=3D=
ws8uwz9ivjo6rh0y9h15nsbna&dl=3D0

I'm guessing there is a guard page after all of those mmap()ed=20
mini-arenas the allocator creates, effectively doubling the mapping count.

Thanks,
  - Pierre-Loup

>=20
> Thanks,
> Liam
>=20