Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [PATCH V11 1/5] mm/hotplug: Introduce arch callback validating
 the hot remove range
To:     Anshuman Khandual <anshuman.khandual@arm.com>,
        David Hildenbrand <dhildenb@redhat.com>
Cc:     linux-mm@kvack.org, linux-kernel@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org, akpm@linux-foundation.org,
        catalin.marinas@arm.com, will@kernel.org, mark.rutland@arm.com,
        cai@lca.pw, logang@deltatee.com, cpandya@codeaurora.org,
        arunks@codeaurora.org, dan.j.williams@intel.com,
        mgorman@techsingularity.net, osalvador@suse.de,
        ard.biesheuvel@arm.com, steve.capper@arm.com, broonie@kernel.org,
        valentin.schneider@arm.com, robin.murphy@arm.com,
        steven.price@arm.com, suzuki.poulose@arm.com, ira.weiny@intel.com
References: <6f0efddc-f124-58ca-28b6-4632469cf992@arm.com>
 <3C3BE5FA-0CFC-4C90-8657-63EF5B680B0B@redhat.com>
 <6b8fb779-31e8-1b63-85a8-9f6c93a04494@arm.com>
From:   David Hildenbrand <david@redhat.com>
Autocrypt: addr=david@redhat.com; prefer-encrypt=mutual; keydata=
 mQINBFXLn5EBEAC+zYvAFJxCBY9Tr1xZgcESmxVNI/0ffzE/ZQOiHJl6mGkmA1R7/uUpiCjJ
 dBrn+lhhOYjjNefFQou6478faXE6o2AhmebqT4KiQoUQFV4R7y1KMEKoSyy8hQaK1umALTdL
 QZLQMzNE74ap+GDK0wnacPQFpcG1AE9RMq3aeErY5tujekBS32jfC/7AnH7I0v1v1TbbK3Gp
 XNeiN4QroO+5qaSr0ID2sz5jtBLRb15RMre27E1ImpaIv2Jw8NJgW0k/D1RyKCwaTsgRdwuK
 Kx/Y91XuSBdz0uOyU/S8kM1+ag0wvsGlpBVxRR/xw/E8M7TEwuCZQArqqTCmkG6HGcXFT0V9
 PXFNNgV5jXMQRwU0O/ztJIQqsE5LsUomE//bLwzj9IVsaQpKDqW6TAPjcdBDPLHvriq7kGjt
 WhVhdl0qEYB8lkBEU7V2Yb+SYhmhpDrti9Fq1EsmhiHSkxJcGREoMK/63r9WLZYI3+4W2rAc
 UucZa4OT27U5ZISjNg3Ev0rxU5UH2/pT4wJCfxwocmqaRr6UYmrtZmND89X0KigoFD/XSeVv
 jwBRNjPAubK9/k5NoRrYqztM9W6sJqrH8+UWZ1Idd/DdmogJh0gNC0+N42Za9yBRURfIdKSb
 B3JfpUqcWwE7vUaYrHG1nw54pLUoPG6sAA7Mehl3nd4pZUALHwARAQABtCREYXZpZCBIaWxk
 ZW5icmFuZCA8ZGF2aWRAcmVkaGF0LmNvbT6JAlgEEwEIAEICGwMFCQlmAYAGCwkIBwMCBhUI
 AgkKCwQWAgMBAh4BAheAFiEEG9nKrXNcTDpGDfzKTd4Q9wD/g1oFAl3pImkCGQEACgkQTd4Q
 9wD/g1o+VA//SFvIHUAvul05u6wKv/pIR6aICPdpF9EIgEU448g+7FfDgQwcEny1pbEzAmiw
 zAXIQ9H0NZh96lcq+yDLtONnXk/bEYWHHUA014A1wqcYNRY8RvY1+eVHb0uu0KYQoXkzvu+s
 Dncuguk470XPnscL27hs8PgOP6QjG4jt75K2LfZ0eAqTOUCZTJxA8A7E9+XTYuU0hs7QVrWJ
 jQdFxQbRMrYz7uP8KmTK9/Cnvqehgl4EzyRaZppshruKMeyheBgvgJd5On1wWq4ZUV5PFM4x
 II3QbD3EJfWbaJMR55jI9dMFa+vK7MFz3rhWOkEx/QR959lfdRSTXdxs8V3zDvChcmRVGN8U
 Vo93d1YNtWnA9w6oCW1dnDZ4kgQZZSBIjp6iHcA08apzh7DPi08jL7M9UQByeYGr8KuR4i6e
 RZI6xhlZerUScVzn35ONwOC91VdYiQgjemiVLq1WDDZ3B7DIzUZ4RQTOaIWdtXBWb8zWakt/
 ztGhsx0e39Gvt3391O1PgcA7ilhvqrBPemJrlb9xSPPRbaNAW39P8ws/UJnzSJqnHMVxbRZC
 Am4add/SM+OCP0w3xYss1jy9T+XdZa0lhUvJfLy7tNcjVG/sxkBXOaSC24MFPuwnoC9WvCVQ
 ZBxouph3kqc4Dt5X1EeXVLeba+466P1fe1rC8MbcwDkoUo65Ag0EVcufkQEQAOfX3n0g0fZz
 Bgm/S2zF/kxQKCEKP8ID+Vz8sy2GpDvveBq4H2Y34XWsT1zLJdvqPI4af4ZSMxuerWjXbVWb
 T6d4odQIG0fKx4F8NccDqbgHeZRNajXeeJ3R7gAzvWvQNLz4piHrO/B4tf8svmRBL0ZB5P5A
 2uhdwLU3NZuK22zpNn4is87BPWF8HhY0L5fafgDMOqnf4guJVJPYNPhUFzXUbPqOKOkL8ojk
 CXxkOFHAbjstSK5Ca3fKquY3rdX3DNo+EL7FvAiw1mUtS+5GeYE+RMnDCsVFm/C7kY8c2d0G
 NWkB9pJM5+mnIoFNxy7YBcldYATVeOHoY4LyaUWNnAvFYWp08dHWfZo9WCiJMuTfgtH9tc75
 7QanMVdPt6fDK8UUXIBLQ2TWr/sQKE9xtFuEmoQGlE1l6bGaDnnMLcYu+Asp3kDT0w4zYGsx
 5r6XQVRH4+5N6eHZiaeYtFOujp5n+pjBaQK7wUUjDilPQ5QMzIuCL4YjVoylWiBNknvQWBXS
 lQCWmavOT9sttGQXdPCC5ynI+1ymZC1ORZKANLnRAb0NH/UCzcsstw2TAkFnMEbo9Zu9w7Kv
 AxBQXWeXhJI9XQssfrf4Gusdqx8nPEpfOqCtbbwJMATbHyqLt7/oz/5deGuwxgb65pWIzufa
 N7eop7uh+6bezi+rugUI+w6DABEBAAGJAiUEGAECAA8FAlXLn5ECGwwFCQlmAYAACgkQTd4Q
 9wD/g1qA6w/+M+ggFv+JdVsz5+ZIc6MSyGUozASX+bmIuPeIecc9UsFRatc91LuJCKMkD9Uv
 GOcWSeFpLrSGRQ1Z7EMzFVU//qVs6uzhsNk0RYMyS0B6oloW3FpyQ+zOVylFWQCzoyyf227y
 GW8HnXunJSC+4PtlL2AY4yZjAVAPLK2l6mhgClVXTQ/S7cBoTQKP+jvVJOoYkpnFxWE9pn4t
 H5QIFk7Ip8TKr5k3fXVWk4lnUi9MTF/5L/mWqdyIO1s7cjharQCstfWCzWrVeVctpVoDfJWp
 4LwTuQ5yEM2KcPeElLg5fR7WB2zH97oI6/Ko2DlovmfQqXh9xWozQt0iGy5tWzh6I0JrlcxJ
 ileZWLccC4XKD1037Hy2FLAjzfoWgwBLA6ULu0exOOdIa58H4PsXtkFPrUF980EEibUp0zFz
 GotRVekFAceUaRvAj7dh76cToeZkfsjAvBVb4COXuhgX6N4pofgNkW2AtgYu1nUsPAo+NftU
 CxrhjHtLn4QEBpkbErnXQyMjHpIatlYGutVMS91XTQXYydCh5crMPs7hYVsvnmGHIaB9ZMfB
 njnuI31KBiLUks+paRkHQlFcgS2N3gkRBzH7xSZ+t7Re3jvXdXEzKBbQ+dC3lpJB0wPnyMcX
 FOTT3aZT7IgePkt5iC/BKBk3hqKteTnJFeVIT7EC+a6YUFg=
Organization: Red Hat GmbH
Message-ID: <19194427-1295-3596-2c2c-463c4adf4f35@redhat.com>
Date:   Mon, 13 Jan 2020 11:37:48 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.3.1
MIME-Version: 1.0
In-Reply-To: <6b8fb779-31e8-1b63-85a8-9f6c93a04494@arm.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 13.01.20 10:50, Anshuman Khandual wrote:
>=20
>=20
> On 01/13/2020 02:44 PM, David Hildenbrand wrote:
>>
>>
>>> Am 13.01.2020 um 10:10 schrieb Anshuman Khandual <anshuman.khandual@a=
rm.com>:
>>>
>>> =EF=BB=BF
>>>
>>>> On 01/10/2020 02:12 PM, David Hildenbrand wrote:
>>>>> On 10.01.20 04:09, Anshuman Khandual wrote:
>>>>> Currently there are two interfaces to initiate memory range hot rem=
oval i.e
>>>>> remove_memory() and __remove_memory() which then calls try_remove_m=
emory().
>>>>> Platform gets called with arch_remove_memory() to tear down require=
d kernel
>>>>> page tables and other arch specific procedures. But there are platf=
orms
>>>>> like arm64 which might want to prevent removal of certain specific =
memory
>>>>> ranges irrespective of their present usage or movability properties=
.
>>>>
>>>> Why? Is this only relevant for boot memory? I hope so, otherwise the
>>>> arch code needs fixing IMHO.
>>>
>>> Right, it is relevant only for the boot memory on arm64 platform. But=
 this
>>> new arch callback makes it flexible to reject any given memory range.
>>>
>>>>
>>>> If it's only boot memory, we should disallow offlining instead via a
>>>> memory notifier - much cleaner.
>>>
>>> Dont have much detail understanding of MMU notifier mechanism but fro=
m some
>>> initial reading, it seems like we need to have a mm_struct for a noti=
fier
>>> to monitor various events on the page table. Just wondering how a phy=
sical
>>> memory range like boot memory can be monitored because it can be used=
 both
>>> for for kernel (init_mm) or user space process at same time. Is there=
 some
>>> mechanism we could do this ?
>>>
>>>>
>>>>>
>>>>> Current arch call back arch_remove_memory() is too late in the proc=
ess to
>>>>> abort memory hot removal as memory block devices and firmware memor=
y map
>>>>> entries would have already been removed. Platforms should be able t=
o abort
>>>>> the process before taking the mem_hotplug_lock with mem_hotplug_beg=
in().
>>>>> This essentially requires a new arch callback for memory range vali=
dation.
>>>>
>>>> I somewhat dislike this very much. Memory removal should never fail =
if
>>>> used sanely. See e.g., __remove_memory(), it will BUG() whenever
>>>> something like that would strike.
>>>>
>>>>>
>>>>> This differentiates memory range validation between memory hot add =
and hot
>>>>> remove paths before carving out a new helper check_hotremove_memory=
_range()
>>>>> which incorporates a new arch callback. This call back provides pla=
tforms
>>>>> an opportunity to refuse memory removal at the very onset. In futur=
e the
>>>>> same principle can be extended for memory hot add path if required.
>>>>>
>>>>> Platforms can choose to override this callback in order to reject s=
pecific
>>>>> memory ranges from removal or can just fallback to a default implem=
entation
>>>>> which allows removal of all memory ranges.
>>>>
>>>> I suspect we want really want to disallow offlining instead. E.g., I
>>>
>>> If boot memory pages can be prevented from being offlined for sure, t=
hen it
>>> would indirectly definitely prevent hot remove process as well.
>>>
>>>> remember s390x does that with certain areas needed for dumping/kexec=
.
>>>
>>> Could not find any references to mmu_notifier in arch/s390 or any oth=
er arch
>>> for that matter apart from KVM (which has an user space component), c=
ould you
>>> please give some pointers ?
>>
>> Memory (hotplug) notifier, not MMU notifier :)
>=20
> They are so similarly named :)
>=20
>>
>> Not on my notebook right now, grep for MEM_GOING_OFFLINE, that should =
be it.
>>
>=20
> Got it, thanks ! But we will still need boot memory enumeration via MEM=
BLOCK_BOOT
> to reject affected offline requests in the callback.

Do you really need that?

We have SECTION_IS_EARLY. You could iterate all involved sections (for
which you are getting notified) and check if any one of these is marked
SECTION_IS_EARLY. then, it was added during boot and not via add_memory()=
.


--=20
Thanks,

David / dhildenb