Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <83689f25-ca50-7ece-45f0-a936e704df7d@redhat.com>
Date:   Wed, 21 Jun 2023 10:11:43 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.12.0
To:     John Hubbard <jhubbard@nvidia.com>,
        Oscar Salvador <osalvador@suse.de>
Cc:     Andrew Morton <akpm@linux-foundation.org>,
        LKML <linux-kernel@vger.kernel.org>, linux-mm@kvack.org
References: <20230620011719.155379-1-jhubbard@nvidia.com>
 <ed83df65-f785-7077-ddd0-4e53d6fa6056@redhat.com>
 <80e01fa9-28c0-37e8-57f8-5bb4ce9a9db7@nvidia.com>
Content-Language: en-US
From:   David Hildenbrand <david@redhat.com>
Organization: Red Hat
Subject: Re: [PATCH] mm/memory_hotplug.c: don't fail hot unplug quite so
 eagerly
In-Reply-To: <80e01fa9-28c0-37e8-57f8-5bb4ce9a9db7@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 20.06.23 23:54, John Hubbard wrote:
> On 6/20/23 00:12, David Hildenbrand wrote:
>> On 20.06.23 03:17, John Hubbard wrote:
>>> mm/memory_hotplug.c: don't fail hot unplug quite so eagerly
>>>
>>> Some device drivers add memory to the system via memory hotplug. When
>>> the driver is unloaded, that memory is hot-unplugged.
>>
>> Which interfaces are they using to add/remove memory?
> 
> It's coming in from the kernel driver, like this:
> 
> offline_and_remove_memory()
>       walk_memory_blocks()
>           try_offline_memory_block()
>               device_offline()
>                   memory_subsys_offline()
>                       offline_pages()
> 
> ...and the above is getting invoked as part of killing a user space
> process that was helping (for performance reasons) holding the device
> nodes open. That triggers a final close of the file descriptors and
> leads to tearing down the driver. The teardown succeeds even though
> the memory was not offlined, and now everything is, to use a technical
> term, "stuck". :)
> 

Ah, I see, thanks! I thought it would just be offlining from user space.

> More below...
> 
>>
>>>
>>> However, memory hot unplug can fail. And these days, it fails a little
>>> too easily, with respect to the above case. Specifically, if a signal is
>>> pending on the process, hot unplug fails. This leads directly to: the
>>> user must reboot the machine in order to unload the driver, and
>>> therefore the device is unusable until the machine is rebooted.
>>
>> Why can't they retry in user space when offlining fails with -EINTR, or re-trigger driver unloading?
> 
> If someone uses "kill -9" to kill that process, then we get here,
> because user space cannot trap that signal.

Understood, thanks!

> 
> 
> ...
>>> --- a/mm/memory_hotplug.c
>>> +++ b/mm/memory_hotplug.c
>>> @@ -1879,12 +1879,6 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages,
>>>        do {
>>>            pfn = start_pfn;
>>>            do {
>>> -            if (signal_pending(current)) {
>>> -                ret = -EINTR;
>>> -                reason = "signal backoff";
>>> -                goto failed_removal_isolated;
>>> -            }
>>> -
>>>                cond_resched();
>>>                ret = scan_movable_pages(pfn, end_pfn, &pfn);
>>
>> No, we can't remove that. It's documented behavior that exists precisely for that reason:
>>
>> https://docs.kernel.org/admin-guide/mm/memory-hotplug.html#id21
>>
>> "
>> When offlining is triggered from user space, the offlining context can be terminated by sending a fatal signal. A timeout based offlining can easily be implemented via:
>>
>> % timeout $TIMEOUT offline_block | failure_handling
>> "
>>
>> Otherwise, there is no way to stop an userspace-triggered offline operation that loops forever in the kernel.
> 
> OK yes, I see.
> 
>>
>> I guess switching to fatal_signal_pending() might help to some degree, it should keep the timeout trick working.
>>
>> But it wouldn't help in your case because where root kills arbitrary processes. I'm not sure if that is something we should be paying attention to.
>>
> 
> Right. I think it would be more accurate perhaps, but it wouldn't help
> this particular complaint.
> 
> Perhaps it is reasonable to claim that, "well, kill -9 *means* that you
> end up here!" :) And the above patch clearly is not the way to go, but...
> 
> ...what about discerning between "user initiated offline_pages" and
> "offline pages as part of a driver shutdown/unload"?

Makes sense to me.

There are two ways for triggering it directly from user space:

1) drivers/base/core.c:online_store()
2) drivers/base/memory.c:state_store()

We cannot easily hook into 2) to indicate "we're offlining directly
from user space". SO we might have to do it the other way around.


Something along the following lines should do the trick (expect whitespace damage):


diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 53ee7654f009..acd4b739505a 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -152,6 +152,13 @@ void put_online_mems(void)
  
  bool movable_node_enabled = false;
  
+/*
+ * Protected by the device hotplug lock. Indicates whether device offlining
+ * is triggered from try_offline_memory_block() such that we don't fail memory
+ * offlining if a signal is pending.
+ */
+static bool mhp_in_try_offline_memory_block;
+
  #ifndef CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE
  int mhp_default_online_type = MMOP_OFFLINE;
  #else
@@ -1860,7 +1867,8 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages,
         do {
                 pfn = start_pfn;
                 do {
-                       if (signal_pending(current)) {
+                       if (!mhp_in_try_offline_memory_block &&
+                           signal_pending(current)) {
                                 ret = -EINTR;
                                 reason = "signal backoff";
                                 goto failed_removal_isolated;
@@ -2177,7 +2185,9 @@ static int try_offline_memory_block(struct memory_block *mem, void *arg)
         if (page && zone_idx(page_zone(page)) == ZONE_MOVABLE)
                 online_type = MMOP_ONLINE_MOVABLE;
  
+       mhp_in_try_offline_memory_block = true;
         rc = device_offline(&mem->dev);
+       mhp_in_try_offline_memory_block = false;
         /*
          * Default is MMOP_OFFLINE - change it only if offlining succeeded,
          * so try_reonline_memory_block() can do the right thing.


There is still arch/powerpc/platforms/pseries/hotplug-memory.c that calls
device_offline() and would fail on signals (not sure if relevant, like for virtio-mem it
shouldn't be that relevant).

I guess dlpar_remove_lmb() can now simply call offline_and_remove_memory().
[I might craft a patch later]

-- 
Cheers,

David / dhildenb