Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
References: <20221123005752.161003-1-gshan@redhat.com>
 <e8a64b11-98f9-b571-dce9-a60df98e3e5@google.com>
 <c61612f7-b861-39cf-3e73-dbe4d134eec0@redhat.com>
 <871qptrvsw.fsf@nvidia.com> <Y37mC1+LQscJaOk4@casper.infradead.org>
 <51ffd399-7fa3-b2f2-b6e5-61a8b609e350@redhat.com>
 <87o7svreq0.fsf@nvidia.com>
 <2541344f-61d8-96d1-10e9-ba7e1743a299@redhat.com>
User-agent: mu4e 1.8.10; emacs 28.2
From:   Alistair Popple <apopple@nvidia.com>
To:     David Hildenbrand <david@redhat.com>
Cc:     Matthew Wilcox <willy@infradead.org>,
        Hugh Dickins <hughd@google.com>, Gavin Shan <gshan@redhat.com>,
        linux-mm@kvack.org, linux-kernel@vger.kernel.org,
        akpm@linux-foundation.org, william.kucharski@oracle.com,
        ziy@nvidia.com, kirill.shutemov@linux.intel.com,
        zhenyzha@redhat.com, shan.gavin@gmail.com, riel@surriel.com
Subject: Re: [PATCH] mm: migrate: Fix THP's mapcount on isolation
Date:   Fri, 02 Dec 2022 09:35:42 +1100
In-reply-to: <2541344f-61d8-96d1-10e9-ba7e1743a299@redhat.com>
Message-ID: <87zgc6buoq.fsf@nvidia.com>
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?hGmH/Iz17vJv1DOCOZw4U/c+uF/tarFV6atC+MFoqib0iL0Eilnitp4Oh5NV?=
 =?us-ascii?Q?9h/5TarXHAXHI4p6nlFxPslXV64q9+JUTTyMfBwwKMet3iJFzIqNYq/Ib49Z?=
 =?us-ascii?Q?CBO4WcxR3tM5B1CSua5+6dz6kaSDSLv85ZVak32Cx52l9dSqRC8/Xx9eyS8i?=
 =?us-ascii?Q?Zv8RHmyt2KCDOGh8WDx7jAD2EqZ51FutJfXOKwbPiPXapCAWBgNhlXkKr7U3?=
 =?us-ascii?Q?bWWs0jh3wq5orIiDOithfipKNwi0T2KGHziG65PNWmlbqEOD6WcWeXxk98KG?=
 =?us-ascii?Q?S0j5tr5JWELkyKzE42Y5X50nbMOSefQ0eLeASdEurthqyWSfTlV2Gf5V/yG8?=
 =?us-ascii?Q?JS2cHZcRv7vrAxemUyRToHFYH7f0/1yPcw01u5oepYzw0JNVcdDDnj8sjZ69?=
 =?us-ascii?Q?OsIMe8FpZmYhkXPfht+fE3E9v0cjLeI+cGAjORvGuM1YcwPhFMk7qZPHsw9z?=
 =?us-ascii?Q?bDac3GQpAF1T3ztQsnUCPk54bsxhYxZUIzUHwHvPLzoNKEfSHrOE+3OMayk6?=
 =?us-ascii?Q?/YE/oo2tK3Q8Pf0hoqrK0astSyPa98ByNFVLNMK2WBIPV9Te4Zx5f6r5rn06?=
 =?us-ascii?Q?2qnImPXxbdHedcyMiAZ86WhV/ig32XA36WinSIZrROMpidm3/e22HqL5am78?=
 =?us-ascii?Q?AqreywesUfstAxPALibFqmWRgZNUPkbCL3qUkxmSQC51d90EMPGa4OIro4II?=
 =?us-ascii?Q?sWd1EGmCYyzrC7GWJ0Ru4DL6LvhOm1uODIIGgJEz9JuCD+/CS3T74C3/KU9s?=
 =?us-ascii?Q?WT0nPWJnbu5Tibfy52dIexB0ICKqXo5EJhxdJYO4p56RR26Ba3KrdjnDa5we?=
 =?us-ascii?Q?B7Zca223hoWtWH0McAqsRYJoCcHLevcuuLdUX2xmGSxPtr84q24gVwUZPJxw?=
 =?us-ascii?Q?Pe9W72DBwdShDjbHEZkAzXY4eOXg3EVBS17Oid54CqtibUPddl8ERUN8c2Ce?=
 =?us-ascii?Q?LJFMuf2HTx0IiZIjrH4kZBCTshOoOgNh4Ws2GJvLERLZtNCSMQTaTNFbWwBH?=
 =?us-ascii?Q?gn2H70CFrZ7nDrHe/X/2Q2uBoGs61VZDWFmKbF1Cq3O4QMiDfKUIhUEQ0w7s?=
 =?us-ascii?Q?0etjrlEi3eleKIjmxlkKrYShi0VTXBQZ4b7EvblXTcxwVIUnho2aPxjSycbM?=
 =?us-ascii?Q?ApVkLBIcdBGP7UIXuRUXDQ9j6IWjIdTGNLg5MyACf8fsw4BmvNvKCMedbvEP?=
 =?us-ascii?Q?wkr8LFFHWTpwTaunSDdAwKCcgeRD88y+jGFbC9gwEnhMER4ZdAhB4cGjkwnl?=
 =?us-ascii?Q?waUS9n1dcT+7E+hu3pOJHU98u5dMyX5cYMoRc48YL0DNyLEe+DuhjP54Q7Ty?=
 =?us-ascii?Q?UzuT0Hmuycgi3kMGlg9ZyOh0xX9lgqhfdA9njBo0UZvb9YJ6YwTC0N54bhOd?=
 =?us-ascii?Q?S/JbdmTC9h01lb3C5ZTR8KQDPzjOXNOsvba2WMfhd2zfUQEQBVJ10fwFbdc7?=
 =?us-ascii?Q?6XWO8126nY5mPi8aPIRGZyUHZ5DZ1AKxJRIUMVQECuH19Z2cxVXkeYYQJjJ5?=
 =?us-ascii?Q?B+rq5lWmLgyzqeMUDEOJM8yPMMdczlV1CIQ0YON/1AcrhF6D2WIKhqaw0PVG?=
 =?us-ascii?Q?MH0o8vDwRLFxA/FULaCfOUZmAOyNnjO/54Rh4Wji?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e87887d8-02a9-44ef-6021-08dad3ef292e
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2022 22:55:37.0552
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3ELz4ZNaUMmw8BZ2mwhPCYam3aojNfWOzzf8InknIGOwje3VSp5TUKtJQV2a0RcEAnr9w8o273PqA9nL7Qonfw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5892
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


David Hildenbrand <david@redhat.com> writes:

>>>>> I think you're right. Without a page reference I don't think it is even
>>>>> safe to look at struct page, at least not without synchronisation
>>>>> against memory hot unplug which could remove the struct page. From a
>>>>> quick glance I didn't see anything here that obviously did that though.
>>>> Memory hotplug is the offending party here.  It has to make sure
>>>> that
>>>> everything else is definitely quiescent before removing the struct pages.
>>>> Otherwise you can't even try_get a refcount.
>> Sure, I might be missing something but how can memory hotplug
>> possibly
>> synchronise against some process (eg. try_to_compact_pages) doing
>> try_get(pfn_to_page(random_pfn)) without that process either informing
>> memory hotplug that it needs pages to remain valid long enough to get a
>> reference or detecting that memory hotplug is in the process of
>> offlining pages?
>
> It currently doesn't, and it's been mostly a known theoretical problem
> so far. We've been ignoring these kind of problems because they are
> not easy to sort out and so far never popped up in practice.
>
> First, the correct approach is using pfn_to_online_page() instead of
> pfn_to_page() when in doubt whether the page might already be
> offline. While we're using pfn_to_online_page() already in quite some
> PFN walkers, changes are good that we're still missing some.
>
> Second, essentially anybody (PFN walker) doing a pfn_to_online_page()
> is prone to races with memory offlining. I've discussed this in the
> past with Michal and Dan and one idea was to use RCU to synchronize
> PFN walkers and pfn_to_online_page(), essentially synchronizing
> clearing of the SECTION_IS_ONLINE flag.
>
> Nobody worked on that so far because we've never had actual BUG
> reports. These kind of races are rare, but they are theoretically
> possible.

Thanks for providing some background context here. I have been digging
into these details a bit because GPU drivers using device-private memory
tend to excersise memory hot unplug paths more frequently than what I
suspect is normally the case. I haven't actually seen any real BUG
reports caused by this though, and it's likely the places where they're
likely would involve device-private pages at the moment anyway. So agree
such races are rare.

[...]

>> try_to_compact_pages() -> compact_zone_order() -> compact_zone() ->
>> isolate_migratepages() -> isolate_migratepages_block() ->
>> PageHuge(pfn_to_page(pfn))
>> Again I couldn't see anything in that path that would hold the page
>> stable long enough to safely perform the pfn_to_page() and get a
>> reference. And after a bit of fiddling I was able to trigger the below
>> oops running the compaction_test selftest whilst offlining memory so I
>> don't think it is safe:
>
> Thanks for finding a reproducer. This is exactly the kind of BUG that
> we speculated about in the past but that we haven't seen in practice
> yet.

No worries. The reproducer consisted of running the compaction_test
selftest in a loop on a single NUMA node whilst cycling every memory
block in that node online/offline in parallel (ie. not a particularly
realistic real world workload). Eventually that led to a couple of
different kernel OOPS/BUG backtraces.

> Having that said, I'd be very happy if someone would have time to work
> on proper synchronization and I'd be happy to help
> brainstorming/reviewing :)

I would like to see the synchronisation cleaned up here and am happy to
work on it. I'm unlikely to have much bandwidth to do so before the new
year but any brainstorming/review would certainly be appreciated when I
do find some time though!