References: <Y+4lcq4Fge27TQIn@nvidia.com> <Y/T2pNQ70eMAl1sX@slm.duckdns.org>
 <Y/T+pw25oGmKqz1k@nvidia.com> <Y/T/bkcYc9Krw4rE@slm.duckdns.org>
 <Y/UEkNn0O65Pfi4e@nvidia.com> <Y/UIURDjR9pv+gzx@slm.duckdns.org>
 <Y/Ua6VcNe/DFh7X4@nvidia.com> <Y/UfS8TDIXhUlJ/I@slm.duckdns.org>
 <Y/UiQmuVwh2eqrfA@nvidia.com> <87o7pmnd0p.fsf@nvidia.com>
 <Y/YRJNwwvqp7nKKt@nvidia.com>
User-agent: mu4e 1.8.10; emacs 28.2
From:   Alistair Popple <apopple@nvidia.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Tejun Heo <tj@kernel.org>, Michal Hocko <mhocko@suse.com>,
        Yosry Ahmed <yosryahmed@google.com>, linux-mm@kvack.org,
        cgroups@vger.kernel.org, linux-kernel@vger.kernel.org,
        jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org,
        surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch,
        "Daniel P . Berrange" <berrange@redhat.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        Zefan Li <lizefan.x@bytedance.com>,
        Andrew Morton <akpm@linux-foundation.org>
Subject: Re: [PATCH 14/19] mm: Introduce a cgroup for pinned memory
Date:   Thu, 23 Feb 2023 09:59:35 +1100
In-reply-to: <Y/YRJNwwvqp7nKKt@nvidia.com>
Message-ID: <87k009nvnr.fsf@nvidia.com>
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?U3r/eDftl9wv29+Uxj90QSrqNG+qh7Ry4ZA2SF5Sqt0yOzdzOlNJYiDx8nPU?=
 =?us-ascii?Q?qxEy9zuR4D1n6+6z5GtFN7PUaLMFwIAQz1A1T/jIapKvKeeXhqgnCP8HdOwj?=
 =?us-ascii?Q?7wfX+XSf2uurUc92pZxVEv88CEfbD9qEGX6+G+iJKjwiFgj2Ta/dmDtPL6Pz?=
 =?us-ascii?Q?4sjiAtAAt23kshFTxOMeob+fktY+SgHn/ypWhZXUsAgGpF03W93IvjeEadsL?=
 =?us-ascii?Q?GRCFpj7CYwxyEKJOD39yTGZSD5FnXfSk8PZ6Cu41cb0r8PkJ5pMPGEzRRHh0?=
 =?us-ascii?Q?aXjmE4R9a+9OTHNXLbYjohGTiL6AR3aCJD0mz1c73sbOfRaGOM2TU89dRFOn?=
 =?us-ascii?Q?F0yJxcyHMogwf2GFKJSDWoU0biVi4zhCiq96fBXUp9eg14QJnLZm+W87de+E?=
 =?us-ascii?Q?Ps3ttJOwNj/ON1gtDEU2HT8pOiCQYXQ3PUbxh5nLE/pNCyKF7hjzQC3bdfvX?=
 =?us-ascii?Q?lS/uRs+vLUM234Tqh1pS6NuM6zyg+jeSl8JMqmiLuewKxVERFQmfXCcUHFn6?=
 =?us-ascii?Q?V7eiv4a9lq5APZtADpPTIlPk7EJWJCnMWJ78+f3dCDqxCKkPSx6oWG/NtcDS?=
 =?us-ascii?Q?L0KmCjdoLmZytxK9KS2JAzhqzoXFPZuAAkyce7ln4RDLZ75KZJmLYgPyoRIU?=
 =?us-ascii?Q?ALEgdBNpkuI+0fnGGoNlBDx8syUs1t9TluOZGYr++v3Pi6T/fbaIUb7hvcQO?=
 =?us-ascii?Q?CjIrHtMFFMr6kv5QmnRDme5xwF3sSDtpEfmmcudWOK/qk2Csg53QVxNxi1BU?=
 =?us-ascii?Q?tLQ6mlPoHUp+SWenHkdByUnsLRrdN0yt732DWAiPDwqMtuJxo0BMPJFaoKeb?=
 =?us-ascii?Q?Hlxhx/IyBLJVVXS2WlXI4cPgBCoGwBxYvWwvLVJPur3zE4Y3WxTycOHrfa37?=
 =?us-ascii?Q?zgbCLecIJKN1FRZ65LLCVzY11sWKRMr6QjqUzFKTBmrdOGpkXhrpUvrxtoW5?=
 =?us-ascii?Q?jjt0MOlvk8KR5a3V6hd4/FHKV0wt+oLXHDM3R7VPVct+POv0Zevom+AqB3JE?=
 =?us-ascii?Q?jItFVCCMzjVhwo2WLWJYZnoItzy7JrJN8E6f9oXZOUuzxnyl16jmabx1qlnO?=
 =?us-ascii?Q?LQStPCeUMtdbFY6QS8KIuwfDN2YZ7sZ/Ndg6m+TB02VYd8xujQz0FJLhOPOT?=
 =?us-ascii?Q?ja/YH+kTF4uLkpkKiPfOMVEIs50lsS/DXwoKIAoRGO0J7HA8KBetTXRJ9JEc?=
 =?us-ascii?Q?7kwFopHaPrs4yVjJwCS1wZl6/qe47zlLGuPIfYrYq2VOEwPu/DqtH1SbmbIc?=
 =?us-ascii?Q?UFUd/V164egvWky/fNkWlv6s7JFuvYPaAKlnBvjjZ/9+e6/0N+xVxkzdSxNR?=
 =?us-ascii?Q?BJJXZIFjs99z2bFoSBevd05+F8HWIoAG67wizHg3m6OstWkBLJEkH0RtOxZk?=
 =?us-ascii?Q?z2TxcE3JQQXS+AdWmbExWzKLJIeRItZiZ5XgJdH8EsTd5y4VJQfWPzPNQ1KD?=
 =?us-ascii?Q?pR64siy56wjMN07MN4QEG0Clr1UIcPyWGday+5ePa5IthGlv1xWA4knjqTDn?=
 =?us-ascii?Q?F29BiMxIAaWQtfDghUWltlu4pP39ubCDAXkX7TUBwCOf6xtYwaVNjpbYsdoj?=
 =?us-ascii?Q?Ez28TG6EZlVeRRSzr4h7C/esYTHgbx2uGjP27Cgu?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 86650f56-c5ba-4a19-5841-08db152a0cd1
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Feb 2023 23:10:55.2631
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pj3cAbMvyRjUg6Zl04BDzj+cUT+3uOIM51grHxnpQM8We8AexG74gPJHXsGLgbaO09aGQO8ppFx0zxpEUMxDkQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB5465
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


Jason Gunthorpe <jgg@nvidia.com> writes:

> On Wed, Feb 22, 2023 at 10:38:25PM +1100, Alistair Popple wrote:
>> When a driver unpins a page we scan the pinners list and assign
>> ownership to the next driver pinning the page by updating memcg_data and
>> removing the vm_account from the list.
>
> I don't see how this works with just the data structure you outlined??
> Every unique page needs its own list_head in the vm_account, it is
> doable just incredibly costly.

The idea was every driver already needs to allocate a pages array to
pass to pin_user_pages(), and by necessity drivers have to keep a
reference to the contents of that in one form or another. So
conceptually the equivalent of:

struct vm_account {
       struct list_head possible_pinners;
       struct mem_cgroup *memcg;
       struct pages **pages;
       [...]
};

Unpinnig involves finding a new owner by traversing the list of
page->memcg_data->possible_pinners and iterating over *pages[] to figure
out if that vm_account actually has this page pinned or not and could
own it.

Agree this is costly though. And I don't think all drivers keep the
array around so "iterating over *pages[]" may need to be a callback.

> Jason