Message-ID: <719eb5d2-680c-e596-1446-3ca8f47c3aea@oracle.com>
Date:   Tue, 4 Jan 2022 09:16:03 +1100
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.4.1
Subject: Re: [RFC PATCH v2 1/2] kernfs: use kernfs_node specific mutex and
 spinlock.
Content-Language: en-US
To:     Greg KH <gregkh@linuxfoundation.org>
Cc:     tj@kernel.org, linux-kernel@vger.kernel.org
References: <20220103084544.1109829-1-imran.f.khan@oracle.com>
 <20220103084544.1109829-2-imran.f.khan@oracle.com>
 <YdLH6qQNxa11YmRO@kroah.com>
From:   Imran Khan <imran.f.khan@oracle.com>
In-Reply-To: <YdLH6qQNxa11YmRO@kroah.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bnl3eVhFTFhCbnRrZGJ0UVZ6OVIyWStHRXhKbjFLTmhHVTA0anlidGo2NVdo?=
 =?utf-8?B?TjhxdTRmeDVpeVBLTUVnSGFXOVFRMVNKUWd3V2V2TDF2cVJ3Qk1FUHMwRitr?=
 =?utf-8?B?eTBraEl1NHpGUDg2bHlBYjZMQVVOWHpWekZYOXJvWExZbVM2TElMVEFpeUV3?=
 =?utf-8?B?eG9EcnM1bG0wSXBHd2F1cUN2alRudUFpYWJZV3ZNTWFIemxkWXc1Wk5iMlZi?=
 =?utf-8?B?cFE4VlV2UkNhNGhXSzE4enRaQ0VmdklqOWxWdTN5UWRVNUJ0R2R0SFQ3eHF1?=
 =?utf-8?B?RG9RaEFuQ2UycGswang0akljTEd0M2hOZWJ1aFVidU1CL2JMZlo3N1AxaG9i?=
 =?utf-8?B?cEowMFRzOUczY2lDdi8rbHRMdnd6ZVFHT3dPdkJ6TDJmcXVkVWxCVjFFOC8z?=
 =?utf-8?B?akdicFVLeDNxSjJlY05wWGNYTGdYMUJSWE13ekJwR3ZZSTZRNjJvbFZGSDR3?=
 =?utf-8?B?L0tGcjQza3JreE83elROSUxaenB2Q3YxMHpmZWxMOEZidS9GdlhOa09KQUhG?=
 =?utf-8?B?NEoycXp6UWZKMkhKVStuR25CejMxV1JCYnpIVHBSdkNlYmlFQjlpQUVYMktP?=
 =?utf-8?B?eGFiVkpjRjVnLzNueFltNGxRaGs2WUhvR0Z5NGR6Z2JxUlJlS2ZPREx6Y1VQ?=
 =?utf-8?B?Y2JuazJEaWZnMzZEWURJOC9kN1U4TEJrUXNlWmxOZGV1cWlvV0s1bE05dzk1?=
 =?utf-8?B?cUtKbktqcVAyRGV1bFF4TzZkeFhNV2Y5T2xDcW1LTzFzdENBUmFNWXJnejlG?=
 =?utf-8?B?eUxMVkgrVkVuY2FRZUxmeldJMVk4UnRNd3BLc1VSbDdXbGxnN0p6dWxkL0RJ?=
 =?utf-8?B?NDNRODNWVGNFMUVlcG5OUnhSNWFSTVk3RTJOT3B2NVJMaHZxbm5JakFKdXBL?=
 =?utf-8?B?M1d4eDF0K0VKRmE1YjBrUGY3VXFQVGFaRjRnd0FBM2xmSTRldVZZdWZOQ0E2?=
 =?utf-8?B?bmI2bWU1OFllbHJNRnozNjhJOGFYMDFNaklzU3BFeGZJZkZzd29PbmJjTFAr?=
 =?utf-8?B?TEdwOGlzajdOYmwvdFJTajRwY083NzhtSGUrVUhXSmlnODA0dG93V2c1WEtI?=
 =?utf-8?B?UzdyZWRpbzgzUkVGSUFhUFJyclRBZTZwT01rRmV5SDQ4cGJDendrLytRT0JC?=
 =?utf-8?B?blVIKzhnWnluaklFaEJkSzh3V3FKd2NUL3QzbnFUS1lFeE5DWm5pVkxieTNq?=
 =?utf-8?B?dFNPUnVja2NoMnIxK05ndHVJRXJleHdSbktMQnNMWjBmcU5UZGZLN1cwVnNB?=
 =?utf-8?B?Y1YzOUNpSWxSR2dyT245NURmZFhJV0w1bXVXdENta29RWlNhL2J2ZW1YSmZZ?=
 =?utf-8?B?M0JiTXV2RmVzSi9oRFpFZ0lwV3BCaDk0VmltelVHS1I4QWZiQUVEM1krOFJ5?=
 =?utf-8?B?V3JmM1VMTHZyZ1lFNjh0dkVES1d2aU5xRmorK0FXT09aenRyYzBoQnJLTVkr?=
 =?utf-8?B?Sk51T3NSSmpPRGxLQjdlUkdNSCs3ZjlQWmUvV25QSDdEV2ZVOHNheklMRjB6?=
 =?utf-8?B?WDh0eDhiTDgwWTlDMFdXc2ZZZDYvWkRXdlNwT3QrKzE4RWwzbWlrcGpZZHRE?=
 =?utf-8?B?RWZDR1kzMEExaUNpYllZMUN4WFRZSkNYalRIRURTT0V1VHVyQUN3cFd0SnZ1?=
 =?utf-8?B?alpHOHZwR01STk1uUmpORE9mdkdwYmNqdEJGUnhJOTRqWHRBVHREWTlyN0p6?=
 =?utf-8?B?dEhOc0V3UDU3d28wUEFxVFM1aG1TTXJTMytjOEZRUHFvaDc2dkwwRmQzcGZw?=
 =?utf-8?B?bytCQlZtTWtiaHZCNDVyRmtEamZacWNIYnlyYUVWQm5MYmpURUI4Ymw2cVk3?=
 =?utf-8?B?SVk0Y0tvUFVPQlBRR3pjeFAxSzdTdGh0dURHV3VwNDB0emdtbG85d2FVdDhS?=
 =?utf-8?B?UTZ3WTdIWkpqajFURmo1cllUc0g4Q2wwcU5OQVJFZ3ZIR2ZaVS80cVJYRFFI?=
 =?utf-8?B?LzN3eU1ZZ3JhZWlneUNNSnRCVzBqSDJmNEJqSy9WRHdmWU51MTFLMzN4eHR1?=
 =?utf-8?B?RUFOOFZTeXZZZDYxeGZnV1RDRWdkcXJSak5SR0Y5YzVQWUEvZ3pCdDUwbGVN?=
 =?utf-8?B?YTZEMkY4Slo5Vmcxek05VlY1NG55d3gxV0QzZnAxWFRINmZQbGM2MVFSZW9R?=
 =?utf-8?B?ZnFmazM5eUpCeTVjRmppdmNORnZMVVZqQVZINlNaZUlONjRtK2JGODN2M1dm?=
 =?utf-8?Q?Hy0JodgbWRRQK6QZ6TJAN6w=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7de69149-a162-4df2-0a13-08d9cf06a4f0
X-MS-Exchange-CrossTenant-AuthSource: CO1PR10MB4468.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2022 22:16:09.8114
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: L6nrI4nPHFA3x8h4Mlh3hCnB3pTXozCEnPcPNgBlVueItABkEmY9fDLhbXr9V/L3MmLjTksMlbpgHJrydt3uOw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO6PR10MB5410
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10216 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxscore=0 adultscore=0
 bulkscore=0 suspectscore=0 malwarescore=0 spamscore=0 mlxlogscore=999
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2201030150
X-Proofpoint-GUID: 8u5BF3uaGHs7tlwjJA0tSPV5LvdVX4hX
X-Proofpoint-ORIG-GUID: 8u5BF3uaGHs7tlwjJA0tSPV5LvdVX4hX
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 3/1/22 8:54 pm, Greg KH wrote:
> On Mon, Jan 03, 2022 at 07:45:43PM +1100, Imran Khan wrote:
>> diff --git a/include/linux/kernfs.h b/include/linux/kernfs.h
>> index 861c4f0f8a29..5ed4c9ed39af 100644
>> --- a/include/linux/kernfs.h
>> +++ b/include/linux/kernfs.h
>> @@ -164,6 +164,8 @@ struct kernfs_node {
>>  	unsigned short		flags;
>>  	umode_t			mode;
>>  	struct kernfs_iattrs	*iattr;
>> +	spinlock_t kernfs_open_node_lock;
>> +	struct mutex kernfs_open_file_mutex;
> 
> Did you just blow up the memory requirements of a system with lots of
> kobjects created?
>> We used to be able to support tens of thousands of scsi devices in a
> 32bit kernel, with this change, what is the memory difference that just
> happened?
> 
Indeed, this patch increases kernfs_node size by 36 bytes ( 28 bytes for
mutex + 4 bytes for spinlock). From current kernfs_node size of 128
bytes, this will be a ~25% increase in kobjects memory consumption.
I can replace the mutex object with a pointer to it, to bring down
the overall increase in size. Will the size change be acceptable then?

> There is a tradeoff of memory usage and runtime contention that has to
> be made here, and this might be pushing it in the wrong direction for
> a lot of systems.
> 
Agree. Could you please suggest if this should be made configurable via
kconfig ? I understand that this would result in 2 versions of some
functions but it will allow systems with large memories to avoid kernfs
contention.  We are seeing the launch time of some DB workloads
adversely getting affected with this contention.

Also based on recent movement of kernfs_rwsem into kernfs_root, do you
think that the above mentioned mutex and spinlock can be moved to
kernfs_root as well. Although that change would not help in my current
case, but it could avoid similar contentions between different users of
kernfs like cgroup and sysfs

Thanks.
 -- Imran

> thanks,
> 
> greg k-h