Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [PATCH] io-wq: fix handling of NUMA node IDs
From:   Jens Axboe <axboe@kernel.dk>
To:     Jann Horn <jannh@google.com>
Cc:     io-uring@vger.kernel.org,
        Andrew Morton <akpm@linux-foundation.org>,
        Michal Hocko <mhocko@suse.com>, linux-kernel@vger.kernel.org
References: <20191126181020.17593-1-jannh@google.com>
 <a61b62a2-8530-59ab-f96c-ccb4ad274d4a@kernel.dk>
Message-ID: <357aa558-0ec9-5a47-8540-c56d04a506b1@kernel.dk>
Date:   Tue, 26 Nov 2019 11:59:17 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.9.0
MIME-Version: 1.0
In-Reply-To: <a61b62a2-8530-59ab-f96c-ccb4ad274d4a@kernel.dk>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 11/26/19 11:17 AM, Jens Axboe wrote:
> On 11/26/19 11:10 AM, Jann Horn wrote:
>> There are several things that can go wrong in the current code on NUMA
>> systems, especially if not all nodes are online all the time:
>>
>>    - If the identifiers of the online nodes do not form a single contiguous
>>      block starting at zero, wq->wqes will be too small, and OOB memory
>>      accesses will occur e.g. in the loop in io_wq_create().
>>    - If a node comes online between the call to num_online_nodes() and the
>>      for_each_node() loop in io_wq_create(), an OOB write will occur.
>>    - If a node comes online between io_wq_create() and io_wq_enqueue(), a
>>      lookup is performed for an element that doesn't exist, and an OOB read
>>      will probably occur.
>>
>> Fix it by:
>>
>>    - using nr_node_ids instead of num_online_nodes() for the allocation size;
>>      nr_node_ids is calculated by setup_nr_node_ids() to be bigger than the
>>      highest node ID that could possibly come online at some point, even if
>>      those nodes' identifiers are not a contiguous block
>>    - creating workers for all possible CPUs, not just all online ones
>>
>> This is basically what the normal workqueue code also does, as far as I can
>> tell.
>>
>> Signed-off-by: Jann Horn <jannh@google.com>
>> ---
>>
>> Notes:
>>       compile-tested only.
>>       
>>       While I think I probably got this stuff right, it might be good if
>>       someone more familiar with the NUMA logic could give an opinion on this.
>>       
>>       An alternative might be to only allocate workers for online nodes, but
>>       then we'd have to either fiddle together logic to create more workers
>>       on demand or punt requests on newly-onlined nodes over to older nodes.
>>       Both of those don't seem very nice to me.
> 
> I don't think caring about not-online nodes in terms of savings is worth
> the trouble. I'll run this through the regular testing I have with no
> and 2 nodes, thanks.

Tests fine for me in all configurations, applied.

-- 
Jens Axboe