Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH] mm: don't rely on system state to detect hot-plug
 operations
To:     David Hildenbrand <david@redhat.com>,
        Michal Hocko <mhocko@suse.com>
Cc:     akpm@linux-foundation.org, Oscar Salvador <osalvador@suse.de>,
        rafael@kernel.org, nathanl@linux.ibm.com, cheloha@linux.ibm.com,
        stable@vger.kernel.org,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        linux-mm@kvack.org, LKML <linux-kernel@vger.kernel.org>
References: <5cbd92e1-c00a-4253-0119-c872bfa0f2bc@redhat.com>
 <20200908170835.85440-1-ldufour@linux.ibm.com>
 <20200909074011.GD7348@dhcp22.suse.cz>
 <9faac1ce-c02d-7dbc-f79a-4aaaa5a73d28@linux.ibm.com>
 <20200909090953.GE7348@dhcp22.suse.cz>
 <4cdb54be-1a92-4ba4-6fee-3b415f3468a9@linux.ibm.com>
 <20200909105914.GF7348@dhcp22.suse.cz>
 <74a62b00-235e-7deb-2814-f3b240fea25e@linux.ibm.com>
 <20200910072331.GB28354@dhcp22.suse.cz>
 <31cfdf35-618f-6f56-ef16-0d999682ad02@linux.ibm.com>
 <20200910111246.GE28354@dhcp22.suse.cz>
 <bd6f2d09-f4e2-0a63-3511-e0f9bf283fe3@linux.ibm.com>
 <65b2680d-b02e-16c9-66a4-e38b9d3fa52b@redhat.com>
From:   Laurent Dufour <ldufour@linux.ibm.com>
Message-ID: <ac9d3ea9-3735-8d38-e2d3-4eb69d5471b1@linux.ibm.com>
Date:   Thu, 10 Sep 2020 14:36:13 +0200
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0)
 Gecko/20100101 Thunderbird/68.12.0
MIME-Version: 1.0
In-Reply-To: <65b2680d-b02e-16c9-66a4-e38b9d3fa52b@redhat.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Le 10/09/2020 à 14:00, David Hildenbrand a écrit :
> On 10.09.20 13:35, Laurent Dufour wrote:
>> Le 10/09/2020 à 13:12, Michal Hocko a écrit :
>>> On Thu 10-09-20 09:51:39, Laurent Dufour wrote:
>>>> Le 10/09/2020 à 09:23, Michal Hocko a écrit :
>>>>> On Wed 09-09-20 18:07:15, Laurent Dufour wrote:
>>>>>> Le 09/09/2020 à 12:59, Michal Hocko a écrit :
>>>>>>> On Wed 09-09-20 11:21:58, Laurent Dufour wrote:
>>>>> [...]
>>>>>>>> For the point a, using the enum allows to know in
>>>>>>>> register_mem_sect_under_node() if the link operation is due to a hotplug
>>>>>>>> operation or done at boot time.
>>>>>>>
>>>>>>> Yes, but let me repeat. We have a mess here and different paths check
>>>>>>> for the very same condition by different ways. We need to unify those.
>>>>>>
>>>>>> What are you suggesting to unify these checks (using a MP_* enum as
>>>>>> suggested by David, something else)?
>>>>>
>>>>> We do have system_state check spread at different places. I would use
>>>>> this one and wrap it behind a helper. Or have I missed any reason why
>>>>> that wouldn't work for this case?
>>>>
>>>> That would not work in that case because memory can be hot-added at the
>>>> SYSTEM_SCHEDULING system state and the regular memory is also registered at
>>>> that system state too. So system state is not enough to discriminate between
>>>> the both.
>>>
>>> If that is really the case all other places need a fix as well.
>>> Btw. could you be more specific about memory hotplug during early boot?
>>> How that happens? I am only aware of https://lkml.kernel.org/r/20200818110046.6664-1-osalvador@suse.de
>>> and that doesn't happen as early as SYSTEM_SCHEDULING.
>>
>> That points has been raised by David, quoting him here:
>>
>>> IIRC, ACPI can hotadd memory while SCHEDULING, this patch would break that.
>>>
>>> Ccing Oscar, I think he mentioned recently that this is the case with ACPI.
>>
>> Oscar told that he need to investigate further on that.
>>
>> On my side I can't get these ACPI "early" hot-plug operations to happen so I
>> can't check that.
>>
>> If this is clear that ACPI memory hotplug doesn't happen at SYSTEM_SCHEDULING,
>> the patch I proposed at first is enough to fix the issue.
>>
> 
> Booting a qemu guest with 4 coldplugged DIMMs gives me:
> 
> :/root# dmesg | grep link_mem
> [    0.302247] link_mem_sections() during 1
> [    0.445086] link_mem_sections() during 1
> [    0.445766] link_mem_sections() during 1
> [    0.446749] link_mem_sections() during 1
> [    0.447746] link_mem_sections() during 1
> 
> So AFAICs everything happens during SYSTEM_SCHEDULING - boot memory and
> ACPI (cold)plug.
> 
> To make forward progress with this, relying on the system_state is
> obviously not sufficient.
> 
> 1. We have to fix this instance and the instance directly in
> get_nid_for_pfn() by passing in the context (I once had a patch to clean
> that up, to not have two state checks, but it got lost somewhere).
> 
> 2. The "system_state < SYSTEM_RUNNING" check in
> register_memory_resource() is correct. Actual memory hotplug after boot
> is not impacted. (I remember we discussed this exact behavior back then)
> 
> 3. build_all_zonelists() should work as expected, called from
> start_kernel() before sched_init().

I'm bit confused now.
Since hotplug operation is happening at SYSTEM_SCHEDULING like the regular 
memory registration, would it be enough to add a parameter to 
register_mem_sect_under_node() (reworking the memmap_context enum)?
That way the check is not based on the system state but on the calling path.