Received-SPF: pass (google.com: domain of linux-kernel+bounces-18749-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Message-ID: <19075505-b1a6-48d3-9732-7277c4697cf6@nvidia.com>
Date: Sun, 7 Jan 2024 02:40:49 +0200
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] nvme_core: scan namespaces asynchronously
Content-Language: en-US
To: Keith Busch <kbusch@kernel.org>, Stuart Hayes <stuart.w.hayes@gmail.com>
Cc: linux-kernel@vger.kernel.org, Jens Axboe <axboe@kernel.dk>,
 Christoph Hellwig <hch@lst.de>, Sagi Grimberg <sagi@grimberg.me>,
 linux-nvme@lists.infradead.org
References: <20240104163826.10561-1-stuart.w.hayes@gmail.com>
 <ZZbhKM0L8pFYX_zd@kbusch-mbp>
From: Max Gurtovoy <mgurtovoy@nvidia.com>
In-Reply-To: <ZZbhKM0L8pFYX_zd@kbusch-mbp>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?Rlc5dGFEZ2hSbk9FNUZtdjgwdDB1MzFPZk1lUXdiNjlnY0ZoWS9BVFJ5bncx?=
 =?utf-8?B?NnhKZXRNZWplV2dZRTNqRTh5MEdNTFZ5YnVuR0ZLZnNzM0NtZFNJQkJveXRh?=
 =?utf-8?B?WEQ1OWRMK3VxRHBCNjczOUpmY0dGUEhkUS93SmxhZWI4OWdWSDJUb0U5cFIy?=
 =?utf-8?B?SzZSVDgrZzJYS01UZFRScStxY3I2c0RKQUZXMFg1VDUrMTR0Sy93NGhyQy9m?=
 =?utf-8?B?NVdqeHZjRC9lWDFGK0FpV2p4UDZCTFhuR0lQa0dISTBNY0JTeU50b1RGdEZ0?=
 =?utf-8?B?b29CN2dRLzUzdklvUDNOcU5LUjY5ZzJSTHNBblhCSHFLSnRLTGd2WVBCenB3?=
 =?utf-8?B?cW5PaDlCWnkvYWh4OTZEQUtSZXdsdVF6M2VnZGp6ZE5vcnZDV0hucWl5Wk5T?=
 =?utf-8?B?bGtsb1pNcEg5Ni9vUWhMRkc4b0YycUk4UU9veVFPRm1LWGw4VGJ1WW10Q1Av?=
 =?utf-8?B?bFlWUTZGbVlERGdVNExta0J2emtTK2FPcWJlWEZnYUVJa3JNNmlGa3ZnWGlr?=
 =?utf-8?B?UWdISUkvSVkrbTdwR2grNW5SREJTd3RGd3BUeDNJSjNPL3BjdWMzVlRoTHRJ?=
 =?utf-8?B?cDVXZEtXVUJXUUNMdWx4SVBTZVFydnpKYzdUd2VwdWZJdU1lblNvNEdQNjJN?=
 =?utf-8?B?OUtIQWVTRkpkZU1NMHhsTnc1TkpOcklPN0hidTVXKzl3dklVVVpGa1VkVFZX?=
 =?utf-8?B?LzlxbW1QWHkrRmMxdUV1bDFJbW84Uy9XVm9EV0F3ek5xYU5YdGR1bDNHKzVN?=
 =?utf-8?B?Vnl2amphbTU3bzUxZVFrRmNKa3gvQkluc21vTGJhQzhSbVFQVy8rTGx6eGlv?=
 =?utf-8?B?V2xPbnZLZTRiRVRpYWw1U0hjdm0xeG9ROVRuNzdXZGxYc202dTd4NCtscWl6?=
 =?utf-8?B?ejNuaCtHQXdOSm81MWhtcGJpTFNVL0ZSNVBoc0JSbXVRQnFJb0JHY0hGWkRz?=
 =?utf-8?B?NmtlNENjYmpnSmswblA3TkJ3L2VHcUU3ZVFvdXRmSXZBb3NSdW9GTFY1dlBK?=
 =?utf-8?B?VG5paEhNbE9mZFRWeDRFZWlDa3N4Zmcvclg3RzJ3YWtDbjBhTzRBeGpMZmxG?=
 =?utf-8?B?bW9RbDR2Qk9TY2Z3ck9uZ3lmNE5EYjhtYnNqdkZvc21qTXo0cHNNRDVCWDF4?=
 =?utf-8?B?NllYZDdxRGZBNVU4RzB5d3FXRS9qVVNRMSthejlVa3paWUJkMTdWcnFUTEpv?=
 =?utf-8?B?akNGL0xYakZBWFczSUJ1QlphdDdkQ2c4VFBWZ1hkazVsRUpuUUlucXZOUWVU?=
 =?utf-8?B?L3dhblVxZ3Y4NFZBSEVveWd5b0N0cnI2L1NDaXBxcGpjZXVhU0hFdDE5UVdo?=
 =?utf-8?B?YkVHMFJRc0I5a1NDZlRLMUxrQ1lhYitzUENnK3lQeWZFNVNKLytwTGp2aEVx?=
 =?utf-8?B?U1R2L3FBbjBvM1ZwZndINS9jZW1VaG5kM1ZvaUxhTnFCNkNScFdxZy9uV2M2?=
 =?utf-8?B?VHZLdVNVa29TaVVYd1ZKNDB3WGVKQzJlM1g1b0M0K2t2T0RKbEdrWktKWFhJ?=
 =?utf-8?B?QVFpc040RUFpYWwrUENhQmdHTGVYWmVNYUd2ZGNXOGZTVktEck96dzlLdkYz?=
 =?utf-8?B?VHhqbTZkdHZFVmdMWWNjWFY5M1FSbVhBeW1WNXhuaXdKRlNRdlFiRHBqdWhr?=
 =?utf-8?B?N2VRSDkyZkczWms1QUwyaGpwRElRL0xzclFUZnJTSENqdXpucG9zYS9HNG95?=
 =?utf-8?B?QTFmeExkMjdTVFgxbi94ZzkrZm5EUi9HWC9KVWE3L0l4VTFOUEtJaStsaElB?=
 =?utf-8?B?ZWlEZzJIemFhVWplQjJ3cmRlMGh5UlNUV1JyNkk1ME42MkdtbG0wOGlhVzdU?=
 =?utf-8?B?MUZFNzJ4LzdDR3NYTTlWWFB0WW84TGZYMUpwQWJXR2NFRGphRU1OVVZIekxF?=
 =?utf-8?B?YTNCTktFZ0tuWFJhb3Y5TnJISlpuZG91ZUg1WVFZeXE4THlEL1RRcGNOZ0Jr?=
 =?utf-8?B?dHRpVlpoU2ExQVNOenhvMlJScnhSQUdhZVYwL0M0MFNlTHkva2xRUHVPZk1K?=
 =?utf-8?B?MSt1L0ZzL2FQdEhSckh1MU9BY1FmMmlBNUp6OGVsT1lZTWIzU3VwMWFUQ3J1?=
 =?utf-8?B?TngvQ211Yk5FNFBXZ2F6dktGNkNQbXY1RFN1SHhEMmY5SWVXUDloY1BJSDhZ?=
 =?utf-8?Q?wObSCr/6BYAPoeISzW3VmgbXp?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 39249b7a-b77c-4c0a-0625-08dc0f194e46
X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5040.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jan 2024 00:40:54.3748
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HVqflUqeXrt+L8L5C3WhZxf2VdGKYOlqx+AeYnMktdLxNSiKW84hhAJAWfUdCptPYLOXVIdUrEkuTHx25B1ZGg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5846


On 04/01/2024 18:47, Keith Busch wrote:
> On Thu, Jan 04, 2024 at 10:38:26AM -0600, Stuart Hayes wrote:
>> Currently NVME namespaces are scanned serially, so it can take a long time
>> for all of a controller's namespaces to become available, especially with a
>> slower (fabrics) interface with large number (~1000) of namespaces.
>>
>> Use async function calls to make namespace scanning happen in parallel,
>> and add a (boolean) module parameter "async_ns_scan" to enable this.
> 
> Hm, we're not doing a whole lot of blocking IO to bring up a namespace,
> so I'm a little surprised it makes a noticable difference. How much time
> improvement are you observing by parallelizing the scan? Is there a
> tipping point in Number of Namespaces where inline scanning is better
> than asynchronous? And if it is a meaningful gain, let's not introduce
> another module parameter to disable it.

I don't think it is a good idea since some of the namespace 
characteristics must be validated during re-connection time for example.
I actually prepared a patch that makes sure we sync the ns scanning 
before kicking the ns blk queue to avoid that situations.
for example, if for some reason ns1 change its uuid then we must remove 
it and open a new bdev instead. We can't kick old request to it...