Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35;
Message-ID: <1d606139-9fff-a00e-c09b-587a8b6736f2@redhat.com>
Date:   Fri, 6 Oct 2023 14:33:27 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.1
Subject: Re: [PATCH v4 1/2] mm/memory_hotplug: split memmap_on_memory requests
 across memblocks
Content-Language: en-US
To:     "Verma, Vishal L" <vishal.l.verma@intel.com>,
        "Williams, Dan J" <dan.j.williams@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "osalvador@suse.de" <osalvador@suse.de>,
        "akpm@linux-foundation.org" <akpm@linux-foundation.org>
Cc:     "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
        "Huang, Ying" <ying.huang@intel.com>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "aneesh.kumar@linux.ibm.com" <aneesh.kumar@linux.ibm.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-cxl@vger.kernel.org" <linux-cxl@vger.kernel.org>,
        "Hocko, Michal" <mhocko@suse.com>,
        "nvdimm@lists.linux.dev" <nvdimm@lists.linux.dev>,
        "jmoyer@redhat.com" <jmoyer@redhat.com>,
        "Jonathan.Cameron@Huawei.com" <Jonathan.Cameron@Huawei.com>
References: <20230928-vv-kmem_memmap-v4-0-6ff73fec519a@intel.com>
 <20230928-vv-kmem_memmap-v4-1-6ff73fec519a@intel.com>
 <efe2acfd-f22f-f856-cd2a-32374af2053a@redhat.com>
 <7893b6a37a429e2f06f2b65009f044208f904b32.camel@intel.com>
From:   David Hildenbrand <david@redhat.com>
Organization: Red Hat
In-Reply-To: <7893b6a37a429e2f06f2b65009f044208f904b32.camel@intel.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 03.10.23 22:03, Verma, Vishal L wrote:
> On Mon, 2023-10-02 at 11:28 +0200, David Hildenbrand wrote:
>>
>>> +
>>> +static int __ref try_remove_memory(u64 start, u64 size)
>>> +{
>>> +       int rc, nid = NUMA_NO_NODE;
>>> +
>>> +       BUG_ON(check_hotplug_memory_range(start, size));
>>> +
>>> +       /*
>>> +        * All memory blocks must be offlined before removing memory.  Check
>>> +        * whether all memory blocks in question are offline and return error
>>> +        * if this is not the case.
>>> +        *
>>> +        * While at it, determine the nid. Note that if we'd have mixed nodes,
>>> +        * we'd only try to offline the last determined one -- which is good
>>> +        * enough for the cases we care about.
>>> +        */
>>> +       rc = walk_memory_blocks(start, size, &nid, check_memblock_offlined_cb);
>>> +       if (rc)
>>> +               return rc;
>>> +
>>> +       /*
>>> +        * For memmap_on_memory, the altmaps could have been added on
>>> +        * a per-memblock basis. Loop through the entire range if so,
>>> +        * and remove each memblock and its altmap.
>>> +        */
>>> +       if (mhp_memmap_on_memory()) {
>>> +               unsigned long memblock_size = memory_block_size_bytes();
>>> +               u64 cur_start;
>>> +
>>> +               for (cur_start = start; cur_start < start + size;
>>> +                    cur_start += memblock_size)
>>> +                       __try_remove_memory(nid, cur_start, memblock_size);
>>> +       } else {
>>> +               __try_remove_memory(nid, start, size);
>>> +       }
>>> +
>>>          return 0;
>>>    }
>>
>> Why is the firmware, memblock and nid handling not kept in this outer
>> function?
>>
>> We really shouldn't be doing per memory block what needs to be done per
>> memblock: remove_memory_block_devices() and arch_remove_memory().
> 
> 
> Ah yes makes sense since we only do create_memory_block_devices() and
> arch_add_memory() in the per memory block inner loop during addition.
> 
> How should the locking work in this case though?

Sorry, I had to process a family NMI the last couple of days.

> 
> The original code holds the mem_hotplug_begin() lock over
> arch_remove_memory() and all of the nid and memblock stuff. Should I
> just hold the lock and release it in the inner loop for each memory
> block, and then also acquire and release it separately for the memblock
> and nid stuff in the outer function?

I think we have to hold it over the whole operation.

I saw that you sent a v5, I'll comment there.


-- 
Cheers,

David / dhildenb