Date:   Wed, 22 Feb 2023 21:53:56 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Alistair Popple <apopple@nvidia.com>
Cc:     Tejun Heo <tj@kernel.org>, Michal Hocko <mhocko@suse.com>,
        Yosry Ahmed <yosryahmed@google.com>, linux-mm@kvack.org,
        cgroups@vger.kernel.org, linux-kernel@vger.kernel.org,
        jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org,
        surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch,
        "Daniel P . Berrange" <berrange@redhat.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        Zefan Li <lizefan.x@bytedance.com>,
        Andrew Morton <akpm@linux-foundation.org>
Subject: Re: [PATCH 14/19] mm: Introduce a cgroup for pinned memory
Message-ID: <Y/bHNO7A8T3QQ5T+@nvidia.com>
References: <Y/T+pw25oGmKqz1k@nvidia.com>
 <Y/T/bkcYc9Krw4rE@slm.duckdns.org>
 <Y/UEkNn0O65Pfi4e@nvidia.com>
 <Y/UIURDjR9pv+gzx@slm.duckdns.org>
 <Y/Ua6VcNe/DFh7X4@nvidia.com>
 <Y/UfS8TDIXhUlJ/I@slm.duckdns.org>
 <Y/UiQmuVwh2eqrfA@nvidia.com>
 <87o7pmnd0p.fsf@nvidia.com>
 <Y/YRJNwwvqp7nKKt@nvidia.com>
 <87k009nvnr.fsf@nvidia.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87k009nvnr.fsf@nvidia.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?58vWHRxx/J5K5EnV6Au/F9xn9ZHc3dy7B/ZEw6UaWYjdkd9IUNuai0qvkcOC?=
 =?us-ascii?Q?+BOfJWRryF1Iw16MEbQX6mjP65cep51p8zsWL8TMuvRahXt8h/vT7GLK/a+K?=
 =?us-ascii?Q?ew/xWb2W8JNnt8B62m+9baa74P90wlZe3F/c27IjteELTtjW/zmcV9JEDqjq?=
 =?us-ascii?Q?Gr3sTYpzMZLXpUCgo1VxIdYzOHx2M2Zrb3XtS+DK6M92q2VdP7sJCY0xJwF2?=
 =?us-ascii?Q?hh84Fg9mqdsKurwT6SMG1Ww8Uwvmu7vUQI2L6XT3k7uRx/0z5XxyBXv/sk/W?=
 =?us-ascii?Q?qT7BxsE4SHThQZZu7/oGRjiV8kXrrFUmOWpNJQvv8wzAnLrSTTAiB9yrSqM9?=
 =?us-ascii?Q?UhQjUbC9svYt08nJ8H2iN2abvPFdJyEhnXwm02VrOVfyqqUc0aT+4UeppGMn?=
 =?us-ascii?Q?w1dMsAUwRPLjZuAoNxKaMmg5ITHd8LWUWAsMdwHmEyKLSnho0f+h4wCvZ8Du?=
 =?us-ascii?Q?rV7nLsGXMIvQI6srzoObmA0P3S0FcD57I2J6IRXlMuju7y/JDzj5q/D8W/Ep?=
 =?us-ascii?Q?7SAkz8/CN4TjSU2iCEjPshyZhho/Q4IDD9aCH8qqa3k/HrXIic6K4iYzl0Jq?=
 =?us-ascii?Q?KV5k5nrUf6ynZfgk5IK9ZFVGKNtx+O7Uw6n7XCIZYUXBHbAuRenJJVEr/s6x?=
 =?us-ascii?Q?X7atA4ycY4qnF49GQ7bcfMe6W7w+cuQvQ8M7yFK6UZXPuLAFVu4ERJGFrfqq?=
 =?us-ascii?Q?WioY4IYUQlOQxFldndei/QxdPC3Ry8jRH/7d/y/ObjKRJmA7a4fb1jPrMegt?=
 =?us-ascii?Q?IcfglwFO09tatAzhzSkomu0QnI3uXGh8Yt3DAiQ/M3jW5oT/IPq1wSVGGAWg?=
 =?us-ascii?Q?ZH7RItbP3JombDx7G4Vwrq2jt+mh9l1pxggXdZ6CXFkk6McsB+o28Gnj7gIF?=
 =?us-ascii?Q?MXHJIxJMdlT91JotbnQAihVxpbEiQjBc7ET7QkuHIx8Z6PB5D6+Yzj70Q1eQ?=
 =?us-ascii?Q?xZtXOH2b/M2XxwRl2JnEPHYkdyH6dPh9xB+0x09T7cvI/50yCWwHpmVbKnMp?=
 =?us-ascii?Q?qXn3+upZpkOdfcfAf8b13GzifyY08kRDqek2GDi4W7DwgqBlzlVimhWyzPPn?=
 =?us-ascii?Q?wYDoxpI9lzJMwHtU6CLRrZ4lTXk62GvhRa7e8Rq2FiarP2dHWMS4PhXLf2XK?=
 =?us-ascii?Q?sYciAsExdfT+eoo2hDZ2j6fgfbfAuoF0urwI0W7hB38od2qinSzVePya/8O5?=
 =?us-ascii?Q?QDh/kRsesAB3IMjJYoe+LNJ1bVn72VfdAEwylI+Sb5hkvdDAWz9XF6Pm8tLW?=
 =?us-ascii?Q?IPbpI+dq6S7KTcnmsUmznFcxiTeA/GsNWbOZufweYO6CIqbA1GtT2f68XeXt?=
 =?us-ascii?Q?bpHleXW8M54WOzeyMAbXY64LdMCGn2ZPUXtro/fXlE0J4RjjxOSSrMs1XDfr?=
 =?us-ascii?Q?rrMQNwwImPQe0q83BQJVG8nBji4xiVyPSPVUcakoeoDAnl3IR6T2fxA7T98B?=
 =?us-ascii?Q?daxD7MPegFzrpN+NSSeYTlL1dEMrbGtr+2idiMgXezq+YnacABWXZ72LkvFM?=
 =?us-ascii?Q?c2YU1rsojDbDdmSBVtFz2wXGcuux2yLOJGrqYkdTis6ycKTaqpmC0AWGf21U?=
 =?us-ascii?Q?ugUydk2r5GpDt9dhVmYf8qsXj2fI0RYKs9hWiuVc?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b63d4094-47df-4f5f-17ca-08db1540d3c3
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2023 01:53:57.9644
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 56I4QJdC8L1YDG6Hzkz+a/7yjDkLVs40jIukv0SxHe2fzYr6t9Usou9OdaHe7yPR
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR12MB8730
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Feb 23, 2023 at 09:59:35AM +1100, Alistair Popple wrote:
> 
> Jason Gunthorpe <jgg@nvidia.com> writes:
> 
> > On Wed, Feb 22, 2023 at 10:38:25PM +1100, Alistair Popple wrote:
> >> When a driver unpins a page we scan the pinners list and assign
> >> ownership to the next driver pinning the page by updating memcg_data and
> >> removing the vm_account from the list.
> >
> > I don't see how this works with just the data structure you outlined??
> > Every unique page needs its own list_head in the vm_account, it is
> > doable just incredibly costly.
> 
> The idea was every driver already needs to allocate a pages array to
> pass to pin_user_pages(), and by necessity drivers have to keep a
> reference to the contents of that in one form or another. So
> conceptually the equivalent of:
> 
> struct vm_account {
>        struct list_head possible_pinners;
>        struct mem_cgroup *memcg;
>        struct pages **pages;
>        [...]
> };
> 
> Unpinnig involves finding a new owner by traversing the list of
> page->memcg_data->possible_pinners and iterating over *pages[] to figure
> out if that vm_account actually has this page pinned or not and could
> own it.

Oh, you are focusing on Tejun's DOS scenario. 

The DOS problem is to prevent a pin users in cgroup A from keeping
memory charged to cgroup B that it isn't using any more.

cgroup B doesn't need to be pinning the memory, it could just be
normal VMAs and "isn't using anymore" means it has unmapped all the
VMAs.

Solving that problem means figuring out when every cgroup stops using
the memory - pinning or not. That seems to be very costly.

AFAIK this problem also already exists today as the memcg of a page
doesn't change while it is pinned. So maybe we don't need to address
it.

Arguably the pins are not the problem. If we want to treat the pin
like allocation then we simply charge the non-owning memcg's for the
pin as though it was an allocation. Eg go over every page and if the
owning memcg is not the current memcg then charge the current memcg
for an allocation of the MAP_SHARED memory. Undoing this is trivial
enoug.

This doesn't fix the DOS problem but it does sort of harmonize the pin
accounting with the memcg by multi-accounting every pin of a
MAP_SHARED page.

The other drawback is that this isn't the same thing as the current
rlimit. The rlimit is largely restricting the creation of unmovable
memory.

Though, AFAICT memcg seems to bundle unmovable memory (eg GFP_KERNEL)
along with movable user pages so it would be self-consistent.

I'm unclear if this is OK for libvirt..

> Agree this is costly though. And I don't think all drivers keep the
> array around so "iterating over *pages[]" may need to be a callback.

I think searching lists of pages is not reasonable. Things like VFIO &
KVM use cases effectively pin 90% of all system memory, that is
potentially TB of page lists that might need linear searching!

Jason