Message-ID: <989749c4-bae9-8055-39b4-ffc1cb6fc20b@oracle.com>
Date:   Tue, 11 Jan 2022 10:42:31 +1100
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.4.1
Subject: Re: [RFC PATCH v2 1/2] kernfs: use kernfs_node specific mutex and
 spinlock.
Content-Language: en-US
To:     Tejun Heo <tj@kernel.org>
Cc:     Greg KH <gregkh@linuxfoundation.org>, linux-kernel@vger.kernel.org
References: <20220103084544.1109829-1-imran.f.khan@oracle.com>
 <20220103084544.1109829-2-imran.f.khan@oracle.com>
 <YdLH6qQNxa11YmRO@kroah.com>
 <719eb5d2-680c-e596-1446-3ca8f47c3aea@oracle.com>
 <YdP57ij4JxahpdnC@kroah.com> <YddRVH4r6uNHt3xa@slm.duckdns.org>
 <03cb9939-efbb-1ce8-42f5-c167ac5246da@oracle.com>
 <YdivuA12i3VU8zO/@slm.duckdns.org>
From:   Imran Khan <imran.f.khan@oracle.com>
In-Reply-To: <YdivuA12i3VU8zO/@slm.duckdns.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dmtaSmMyZzU1cjByYU5pZ3d6ZFVwMFBkNExGcmpCVGlwS21kc3F0Y0p3czFN?=
 =?utf-8?B?MjVaVTVKUEFzKzBsWjI3VWxTay9jS3lVdi9ZWTBPbTkwT2dPbzYxMXNwTk96?=
 =?utf-8?B?TjQwNStSbncyQ2dCbDR0UUpvc1ZCeGRGNURGYTR4bkFHV2ZHWVN4YXVOWGVZ?=
 =?utf-8?B?QWNKQzlaUlFGQ2tzRlo0Q2p6dUxTZkhCQjRHT0JTREpTQnBnTHE2VnhpKzlF?=
 =?utf-8?B?em8xRUg0S1ljS1BqYlg0akRJbXpiNTJVNndXaXNQVForZnlEN040cGtVakNo?=
 =?utf-8?B?NVp4N0djZ3luRDJZS0laUFFZVmR6RksxV3RsM0wwK1JUckNWS1BoWUFTd29D?=
 =?utf-8?B?S1F1aWVKTDMxUlY5NWVDTm5KOTNjemtjR0VSWlJRTkVBZ0gxYUFMYWdORWRF?=
 =?utf-8?B?cWgxZkZVdU94ZW1RRXd2VTViWlR2ZkRPWTJrZGh3dU1ocjZmR0UwbXVHRWNL?=
 =?utf-8?B?N3dWMmFIVXFUVmhCTkt3VVVKNHUwUEpUZU0zQjdtbFRneXlHMHJEa2orbndG?=
 =?utf-8?B?Y0RKMDZobjVzSG0yZUdYaXplZXhwUEdCNXFUcHlSK0VNNXAvOFRFL3pVUjk3?=
 =?utf-8?B?VFpMam5GUzMyaDFvamZDd1ZoQ0ZxdkRpWEtudFdaLzAzeXdLNVVqYUIxZHFv?=
 =?utf-8?B?bnpycnpsWE5oVkVJZ2VkRXBSM0lFcW9sYk1EejFubnI5ajVhdC9ZMVlHcENF?=
 =?utf-8?B?VjBybVdFQUJqZG5GMUV6dnVvcjhSYVN4MHBSMW40cTk0djN5V01VVWVWWW1N?=
 =?utf-8?B?MVlKTm9FcnhlNkxLcFdDSlpYdnJTU0JreWpJSWdnQXNYakliOVR1RnRNNFNF?=
 =?utf-8?B?YWJJZndpRkdSR0xUbUo0aUVhbTgwR1NNVWh0NHNGd1dXQ0lRSk0rbUIzN2lw?=
 =?utf-8?B?NWMvb0FiTEY4NGNjTDRXaGFJUkQ5THZHTm84UzZteEVrNHNuWG9QT3F2bXpv?=
 =?utf-8?B?REQ0NUI2QmtRMmtCblIyUFhWMnVQS2MxT2VPVldSeFMyQkVIMVBIcmJTblJt?=
 =?utf-8?B?RStTR0dlMDFPL0NNYmxRNTlFUVlLZm9FUmY0OWx3SUQ1N2tmSkg0M0VEb2VE?=
 =?utf-8?B?ZFUxdDFwN2lUQUp0RDRXWHcxdWVucU1GeEtCOTNlcXNPL3hUUyt4SUEvQjh3?=
 =?utf-8?B?T2M0Wk5YQmZPYllySnovc2NYaytoRlUvMmF5UXpaaHJFM3ZOa1JuMnJqdWtH?=
 =?utf-8?B?S3dCMC9hSU56RGx5RHEvVXo2eGowbUlxWnNoVUY1dDlwUGhuU3o4YkFrMFlQ?=
 =?utf-8?B?cnhKR01udC9Ram8yYkpXQWFBM3k0RGhDNXFobjdNSUhlblN2ajlLM3lUSmJx?=
 =?utf-8?B?SlovcG9sR0tnS3JqMW1nRmI1anJDTGY3NXVLakxZYkRVOHZKK0t6R3c3MkRj?=
 =?utf-8?B?MGV0VXRUUCtlNWZrN2dwZ3FPVTlDSHFmNlIvWm9pN0FkYnBMTGZRYW1sRFR5?=
 =?utf-8?B?eHc0RjN2S1FhMzBRcjFESE5zbWJ5d1o2S0xKUnl1TFB6L0hZVDRZeUV3d0dl?=
 =?utf-8?B?R2pHNFdUemRmdTRXMThqcnFDS0swN0R2aG11a3A5NGozOXFUTDNmb1NVbnBQ?=
 =?utf-8?B?VUVFbVd5L2NHaklwU0U1Um5ISGhiMHg5QlpiNVpGWTZCWW1sbWhLdllRUlFD?=
 =?utf-8?B?YUhrSFJlbGlmV1QweWZpY3BJVEIrS25GNGw0VGJRcFFzTkNDc2R3dmt6VENk?=
 =?utf-8?B?UURFUWRUdVhVc2pEazlKY3J2aE8vUzQ4S0cxalk2aGlJSjJYb011Mjd6bGs2?=
 =?utf-8?B?YzdVbzlCTjZOcUliQzRlbUVOcjdUdnp5eUM2UStFQ3JGb3l1aHFvTWcyaGUw?=
 =?utf-8?B?Um9ZaU9kaEFLdnVYRkZMRmQvNDlXTCs3V1FKTjYreXJnTURNUmZ4YVlpRXNh?=
 =?utf-8?B?TlRBQjJaUUQ5cmlTbWVDZTU2K3JkK0d5cWpVVlIzSWRmRldxMVFia0t0V3Zk?=
 =?utf-8?B?a1dwZFpIbGdvalVBU21ESEZLc0U3N3ZyZ2p2UTBkbnpiZlhMSUpPUEtObFRt?=
 =?utf-8?B?dVQ1dkJTc0ptNzh6bHZ3cVhnWktnWG9IQy9OV2lqM1RwNFpEcDZPZkN2V2xy?=
 =?utf-8?B?VXVxYnQ3NGdXZVN0MzBSZkZLUjhKSXM4NDEvaWViWW1SWE1FSkcwZmc4UVFC?=
 =?utf-8?B?d0JJMzV4N1A3bVlIWUE5WHhMc29WZ3l5RGxmVmlGYjJtOGZhZVJMbGNoRnA2?=
 =?utf-8?Q?wCLsUalmcUabF6EEmMLmFFI=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 23713194-2af3-49cb-d428-08d9d492e282
X-MS-Exchange-CrossTenant-AuthSource: CO1PR10MB4468.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jan 2022 23:42:38.3467
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ymhegclBh1hR3iGD9cheTOjkSoIeiiSjUKbHKNCbTA7A2qJK1dCMK/8XtNXXNuNgKpqXeRnPTPm+5FBeKq6UfA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB5752
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10223 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 spamscore=0
 phishscore=0 adultscore=0 suspectscore=0 mlxscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2201100150
X-Proofpoint-GUID: -rkI6abyS14dFPGFo1SSZ1reUJr7L466
X-Proofpoint-ORIG-GUID: -rkI6abyS14dFPGFo1SSZ1reUJr7L466
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Tejun,

On 8/1/22 8:25 am, Tejun Heo wrote:
> Hello,
> 
> On Fri, Jan 07, 2022 at 11:01:55PM +1100, Imran Khan wrote:
>> Could you please suggest me some current users of hashed locks ? I can
>> check that code and modify my patches accordingly.
> 
> include/linux/blockgroup_lock.h seems to be one.
> 

Thanks for this.

>> As of now I have not found any standard benchmarks/workloads to show the
>> impact of this contention. We have some in house DB applications where
>> the impact can be easily seen.  Of course those applications can be
>> modified to get the needed data from somewhere else or access sysfs less
>> frequently but nonetheless I am trying to make the current locking
>> scheme more scalable.
> 
> I don't think it needs to show up in one of the common benchmarks but what
> the application does should make some sense. Which files are involved in the
> contentions?
> 

The database application has a health monitoring component which
regularly collects stats from sysfs. With small number of databases this
was not an issue but recently several customers did some consolidation
and ended up having hundreds of databases, all running on the same
server and in those setups the contention became more and more evident.
As more and more customers are consolidating we have started to get more
occurences of this issue and in this case it all depends on number of
running databases on the server.

I will have to reach out to application team to get a list of all sysfs
files being accessed but one of them is
"/sys/class/infiniband/<device>/ports/<port number>/gids/<gid index>".

Thanks
-- Imran