Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <e6a78b3b-27ae-ade9-77a9-f8677e0850ea@oracle.com>
Date:   Sat, 25 Jun 2022 19:15:10 +1000
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.10.0
Subject: Re: [PATCH] kernfs: Change kernfs_rwsem to a per-cpu rwsem.
Content-Language: en-US
To:     Tejun Heo <tj@kernel.org>
Cc:     gregkh@linuxfoundation.org, viro@zeniv.linux.org.uk,
        linux-kernel@vger.kernel.org
References: <20220620032634.1103822-1-imran.f.khan@oracle.com>
 <YraLOZb2SnKl0wUO@mtj.duckdns.org>
From:   Imran Khan <imran.f.khan@oracle.com>
In-Reply-To: <YraLOZb2SnKl0wUO@mtj.duckdns.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?anYzZUtCMC9NeU5KTWtUR0ZhTWs2ZUpLd0FLNkFhOFhmNjd3cWdkVk52MFBL?=
 =?utf-8?B?WmNHR2MwZ2ZHMDdpMDBVbXBwTUNaQ3Zna1dxMEh4dmJuQTd4eDFzZFJ3ZFc4?=
 =?utf-8?B?Wmh5bkFZVHpEKzNWQkdGQ1hKek5FclZYR045RU10eFVZcGNFcklDeGZUK3pH?=
 =?utf-8?B?NUhuS1BaM2o3WVRzMnJvdk1JdFFjaGdITm1aR2IzV0ZMbnFYOHZYS2FiaWYr?=
 =?utf-8?B?eXdnc1hVa1pGQjBnVlZLODE2c3ZCUzB1bTkweXlXalVYcURvQ0owSjFhRnlG?=
 =?utf-8?B?Zk5RMVozc29Fek43dnU3MzZaY3FZQW80UzFtTGtQK2gzb0t3NEE5ZnFMV2dz?=
 =?utf-8?B?LzIrbkw0M2RNRithKzRsWGtjTG9ZeUhMWmpmZlFyVHc5blgzbVZzWm0vb3py?=
 =?utf-8?B?Q3lCcTd0ZE5OTjlINDJOaDdHdmRPRWxPU210WXlQTUxDRGRDdmtWc1RXWXpK?=
 =?utf-8?B?OXFlaTF0dnhIUEEvNFdRYVc0OExoL3QxZHpIa3dLdTRKMXlNODRSejljTWVE?=
 =?utf-8?B?VHhUYjVncTA3S0U1SjdFZ0lPbG9sK242VXg2K0JzbGhITGtiQzZSZCtTWXBX?=
 =?utf-8?B?Um5IRXZnOEQyYmpFMGNRUjN2MXFvSGJKbjdRanpmWVNORW9NVUZZMUEzcHJT?=
 =?utf-8?B?ZzZacEc1SGI2ZlR1aE5EdUlVbkFVM1Z6eU1QTExiQWVEdnNFd01XMVlBOC9o?=
 =?utf-8?B?NXFxZnlkWTN0WGhKREhOeGdNL0NGNXhUbHdIdEFEUzkrUkF5OUN3UVZTYjB1?=
 =?utf-8?B?YVUrSGR5enFjRXdwUUVsRmxBWjVjNW9jc3liVGk4VkZ6RlhHT1VmV2JTN3pF?=
 =?utf-8?B?djB0amlLazZQdkVCZU93c3lEN2ZnVDFtNTBveUhVVkFORzd6ZTQxNHR2OWJl?=
 =?utf-8?B?SGFpc2FCb094eWdwMzliSm11VDl5S3c2VzAxb2dLN21QdXcwR2lDb2lNS21B?=
 =?utf-8?B?N1RZOG1pdWdVbXlDWHFNeW1VNUVZM1IxS0lUQzdjZVYvb3YwWFl2VzN2dVNp?=
 =?utf-8?B?VUUxSXorMzNZM2lwYWlxeCtleW9MS2w3Y0NwRDVwN2dRdStndzIvbzJveERP?=
 =?utf-8?B?TWo0WHArdHpIQ0x1NTlIWURTK2dPWkJ4SnRzWmllUkdFUHIzcEN0dkR5UEc0?=
 =?utf-8?B?ZWIyQW1EWkZwaXVGait5dTJ6WnIvMkFTRzVDWm94dnVHU1MyWnlUUjZkSWx6?=
 =?utf-8?B?WlpEUW8vdXI1d1dPWDFDMC9QZy81bGtNWHBHSE43WGEwY0Z1bkt0R3RmeEgz?=
 =?utf-8?B?YjVXNEhKOHVTK2l1N2FGakFqYXpJU3BDWlpJSE9IYkdkbnFGaERrN3N4WDJB?=
 =?utf-8?B?SjJSaFV1Q3JaRGlqWkhNWW9PY0FoZ1YvSEpMU0lQL1ZaRHowcmdYcU5RTXN4?=
 =?utf-8?B?aUwzcTlsRjh5MHpocUxlOFdhQWZQT1EwZUxqM3ZocE13VTZIL284aGpHM2Vk?=
 =?utf-8?B?c2ZFNlhkWnk4QXNCei9BVFd4ZndOQUFqV0RDS0lFUWxkYkpVajZHYW5NQ00v?=
 =?utf-8?B?RGdhei9WYkZEWldrNVVHTEVxSnVueFZGRWZWZmR6OFpWbnNob1VsV3k0Zk4x?=
 =?utf-8?B?NjlRZjhlTmZVR2Y5Q09VOENNdWx2UXR0WUpmV2hOM1IrdFIwMkhWUnFFUDkx?=
 =?utf-8?B?eTd1aWVCSGFvaklJbVJIMjdnMERhdWpBeTNvT3NBSnY4OFFyTnVOQVJyODVU?=
 =?utf-8?B?N2l5V28yUDBzQWcwU3FpYWhhV2kyRUJyQndGL2MvTkFUQUhTUCtnWkRlZWlN?=
 =?utf-8?B?bGoxRytYaFNYN3RCakVIK3BCa1NROGNLdXh1cmJYaWhBZkoxZjNodTlhNUJs?=
 =?utf-8?B?OGJkNmQvU1liWnBkQmI4ZWtic2UraGRnTlB6UzUzb0dzRXZZZFg4bU5hdGd2?=
 =?utf-8?B?a0c4Q2ovbFNkT2dDNGM4emtJQVRmRnRvbjlQUTUzdjBGalFCS2FKbVBlQkc2?=
 =?utf-8?B?QWpkQytkOFh1SDZGRzJ0Um56MmZrYTNSWWtFWHJpcGxIOFp6VWhNck9QZVcw?=
 =?utf-8?B?Y0ljQmgyU1ZQVlhDQSs2eFZ6dlB1NVB0cmNmNFIzSmpWYWhkekJMZ090N2RB?=
 =?utf-8?B?clhnWkVraHRDV090UlV2N2pZTmcvZ0RTUTFKdGVtbU1DNUNMUzc4WGZyYUxp?=
 =?utf-8?B?VDRuelpmWUtab2s5bVNWZnUwV1g1bEdKbjk0cDBocDdVQkFZMUlUaW13RVJz?=
 =?utf-8?B?NXc9PQ==?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7981d8f5-86f0-445e-576c-08da568b38d8
X-MS-Exchange-CrossTenant-AuthSource: CO1PR10MB4468.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jun 2022 09:15:18.5647
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: yxkAJH5Qu4YJ6+aAmt+Lmv26qGXsiLUG5P1By8TVp5j3vW8VrqivdU3aq2Fv6YYvhIpXqtJHQN1dQn94EPMZXw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR10MB4892
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883
 definitions=2022-06-25_03:2022-06-24,2022-06-25 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 spamscore=0 bulkscore=0
 phishscore=0 mlxlogscore=806 adultscore=0 mlxscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000
 definitions=main-2206250040
X-Proofpoint-ORIG-GUID: sYBk6D1Q6R2Iz-iKbehVchURQQU1L0LT
X-Proofpoint-GUID: sYBk6D1Q6R2Iz-iKbehVchURQQU1L0LT
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hello Tejun,

On 25/6/22 2:12 pm, Tejun Heo wrote:
> On Mon, Jun 20, 2022 at 01:26:34PM +1000, Imran Khan wrote:
>> On large systems when few hundred CPUs simulateously acquire kernfs_rwsem,
>> for reading we see performance degradation due to bouncing of cache line
>> that contains kernfs_rwsem. Changing kernfs_rwsem into a per-fs, per-cpu
>> rwsem can fix this degradation.
> ...
>> Moreover this run of 200 applications take more than 32 secs to finish on
>> this system.
>>
>> After changing kernfs_rwsem to a per-cpu rwsem, I can see that contention
>> for kernfs_rwsem is no longer visible in perf data and the test execution
>> time has reduced to almost half (17 secs or less from 32 secs or more).
>>
>> The overhead involving write operations with per-cpu rwsem will be higher
>> but frequency of creation/deletion of kernfs files is much less than
>> frequency at which kernfs (cgroup, sysfs) files are read.
> 
> The problem with percpu_rwsem is that write locking requires going
> through a RCU grace period, which can easily add two or more digit
> millisec latencies. I'm pretty sure there are code paths which are
> pretty heavy on write - e.g. during boot, depending on the machine
> configuration, we could be write-acquiring the rwsem hundreds of
> thousands of times and we'd be constantly doing RCU grace periods.
> 
> So, I don't think kernfs_rwsem is a good candidate for percpu rwsem.
> There likely are plenty of cases where write path isn't cold enough.
> 

Thanks for your feedback. I will continue with approach of hashed kernfs_rwsem then.

Thanks
-- Imran