Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <73db2622-4985-2f93-a118-d7d249094239@redhat.com>
Date:   Thu, 27 Jul 2023 11:41:54 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Subject: Re: collision between ZONE_MOVABLE and memblock allocations
Content-Language: en-US
To:     Michal Hocko <mhocko@suse.com>
Cc:     Ross Zwisler <zwisler@google.com>, linux-kernel@vger.kernel.org,
        linux-mm@kvack.org, Mike Rapoport <rppt@kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Matthew Wilcox <willy@infradead.org>,
        Mel Gorman <mgorman@suse.de>, Vlastimil Babka <vbabka@suse.cz>
References: <20230718220106.GA3117638@google.com>
 <ZLd/WEZTH5rlwYjP@dhcp22.suse.cz> <20230719224821.GC3528218@google.com>
 <9ef757dc-da4b-9fa1-de84-1328a74f18a7@redhat.com>
 <ZMIoQIS1t53XE4Kw@dhcp22.suse.cz>
From:   David Hildenbrand <david@redhat.com>
Organization: Red Hat
In-Reply-To: <ZMIoQIS1t53XE4Kw@dhcp22.suse.cz>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 27.07.23 10:18, Michal Hocko wrote:
> On Wed 26-07-23 10:44:21, David Hildenbrand wrote:
>> On 20.07.23 00:48, Ross Zwisler wrote:
>>> On Wed, Jul 19, 2023 at 08:14:48AM +0200, Michal Hocko wrote:
>>>> On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
>>>> [...]
>>>>> I do think that we need to fix this collision between ZONE_MOVABLE and memmap
>>>>> allocations, because this issue essentially makes the movablecore= kernel
>>>>> command line parameter useless in many cases, as the ZONE_MOVABLE region it
>>>>> creates will often actually be unmovable.
>>>>
>>>> movablecore is kinda hack and I would be more inclined to get rid of it
>>>> rather than build more into it. Could you be more specific about your
>>>> use case?
>>>
>>> The problem that I'm trying to solve is that I'd like to be able to get kernel
>>> core dumps off machines (chromebooks) so that we can debug crashes.  Because
>>> the memory used by the crash kernel ("crashkernel=" kernel command line
>>> option) is consumed the entire time the machine is booted, there is a strong
>>> motivation to keep the crash kernel as small and as simple as possible.  To
>>> this end I'm trying to get away without SSD drivers, not having to worry about
>>> encryption on the SSDs, etc.
>>
>> Okay, so you intend to keep the crashkernel area as small as possible.
>>
>>>
>>> So, the rough plan right now is:
>>>   > 1) During boot set aside some memory that won't contain kernel
>> allocations.
>>> I'm trying to do this now with ZONE_MOVABLE, but I'm open to better ways.
>>>
>>> We set aside memory for a crash kernel & arm it so that the ZONE_MOVABLE
>>> region (or whatever non-kernel region) will be set aside as PMEM in the crash
>>> kernel.  This is done with the memmap=nn[KMG]!ss[KMG] kernel command line
>>> parameter passed to the crash kernel.
>>>
>>> So, in my sample 4G VM system, I see:
>>>
>>>     # lsmem --split ZONES --output-all
>>>     RANGE                                  SIZE  STATE REMOVABLE BLOCK NODE   ZONES
>>>     0x0000000000000000-0x0000000007ffffff  128M online       yes     0    0    None
>>>     0x0000000008000000-0x00000000bfffffff  2.9G online       yes  1-23    0   DMA32
>>>     0x0000000100000000-0x000000012fffffff  768M online       yes 32-37    0  Normal
>>>     0x0000000130000000-0x000000013fffffff  256M online       yes 38-39    0 Movable
>>>     Memory block size:       128M
>>>     Total online memory:       4G
>>>     Total offline memory:      0B
>>>
>>> so I'll pass "memmap=256M!0x130000000" to the crash kernel.
>>>
>>> 2) When we hit a kernel crash, we know (hope?) that the PMEM region we've set
>>> aside only contains user data, which we don't want to store anyway.
>>
>> I raised that in different context already, but such assumptions are not
>> 100% future proof IMHO. For example, we might at one point be able to make
>> user page tables movable and place them on there.
>>
>> But yes, most kernel data structures (which you care about) will probably
>> never be movable and never end up on these regions.
>>
>>> We make a
>>> filesystem in there, and create a kernel crash dump using 'makedumpfile':
>>>
>>>     mkfs.ext4 /dev/pmem0
>>>     mount /dev/pmem0 /mnt
>>>     makedumpfile -c -d 31 /proc/vmcore /mnt/kdump
>>>
>>> We then set up the next full kernel boot to also have this same PMEM region,
>>> using the same memmap kernel parameter.  We reboot back into a full kernel.
>>>
>>> 3) The next full kernel will be a normal boot with a full networking stack,
>>> SSD drivers, disk encryption, etc.  We mount up our PMEM filesystem, pull out
>>> the kdump and either store it somewhere persistent or upload it somewhere.  We
>>> can then unmount the PMEM and reconfigure it back to system ram so that the
>>> live system isn't missing memory.
>>>
>>>     ndctl create-namespace --reconfig=namespace0.0 -m devdax -f
>>>     daxctl reconfigure-device --mode=system-ram dax0.0
>>>
>>> This is the flow I'm trying to support, and have mostly working in a VM,
>>> except up until now makedumpfile would crash because all the memblock
>>> structures it needed were in the PMEM area that I had just wiped out by making
>>> a new filesystem. :)
>>
>>
>> Thinking out loud (and remembering that some architectures relocate the
>> crashkernel during kexec, if I am not wrong), maybe the following would also
>> work and make your setup eventually easier:
>>
>> 1) Don't reserve a crashkernel area in the traditional way, instead reserve
>> that area using CMA. It can be used for MOVABLE allocations.
>>
>> 2) Let kexec load the crashkernel+initrd into ordinary memory only
>> (consuming as much as you would need there).
>>
>> 3) On kexec, relocate the crashkernel+initrd into the CMA area (overwriting
>> any movable data in there)
>>
>> 4) In makedumpfile, don't dump any memory that falls into the crashkernel
>> area. It might already have been overwritten by the second kernel
> 
> This is more or less what Jiri is looking into.
> 

Ah, very nice.

-- 
Cheers,

David / dhildenb