Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Chen, Zhiyin" <zhiyin.chen@intel.com>
To:     Eric Biggers <ebiggers@kernel.org>,
        Christian Brauner <brauner@kernel.org>
CC:     "viro@zeniv.linux.org.uk" <viro@zeniv.linux.org.uk>,
        "linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "Zou, Nanhai" <nanhai.zou@intel.com>,
        "Feng, Xiaotian" <xiaotian.feng@intel.com>
Subject: RE: [PATCH] fs.h: Optimize file struct to prevent false sharing
Thread-Topic: [PATCH] fs.h: Optimize file struct to prevent false sharing
Thread-Index: AQHZkptsjNDfJ+8hD0ijYnqkdkJlwK9ygmoAgAEeaoCAAFkcQA==
Date:   Wed, 31 May 2023 10:31:09 +0000
Message-ID: <CO1PR11MB4931D767C5277A37F24C824DE4489@CO1PR11MB4931.namprd11.prod.outlook.com>
References: <20230530020626.186192-1-zhiyin.chen@intel.com>
 <20230530-wortbruch-extra-88399a74392e@brauner>
 <20230531015549.GA1648@quark.localdomain>
In-Reply-To: <20230531015549.GA1648@quark.localdomain>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?imNCKa8RsL3neU0em0FxHYvWDJTu8dXP/OoQazrRkiFVQpSUBLZWseRo6Ky7?=
 =?us-ascii?Q?KSU8Cqs72Egm1NP6W8AG5LVcol+TYS67a4SUzf7xVeo8M4jcN9TyMnNnRkm2?=
 =?us-ascii?Q?iY6gaEZl3B0uYT13ZseyebfImbSV4WWfjphjwIRKh5wDzUPGb6emDahuVF/x?=
 =?us-ascii?Q?rtgpbZuY4w1gSIK2wETP5mQpmUQzgo22czMAhY1SBfSpu5GRbnupv3EE3SyH?=
 =?us-ascii?Q?s4rXrS5qmby8HsLjWop7njAKunJWchSNvqxBZfy+/VKu3MEvPgyKnqX1Uj/0?=
 =?us-ascii?Q?oqJ81RuRgZQaxQdxNtezki4fflWnuBkBTvzE5ymSvlkcBA32UnOnfvivvXzc?=
 =?us-ascii?Q?h6q2GO7pfrYlMAgjFCvDySievuuZ0qPHVvzW90xHkxvwCEMJikDGL8duhDsN?=
 =?us-ascii?Q?6+VYm2gBAQOfl/3nFrLEDjOq56I3D1p6ush0+xBMuS51+3CDa7cqB9u7Egyb?=
 =?us-ascii?Q?/B1vKBZkb39oUPzg4OPkwAtDinOIKcgY1xc75HG1KtCvg4RpFI6IWFB4HLgJ?=
 =?us-ascii?Q?VLpW/u2UoQ8PaK8+fSmVAb6MiWqn+rwJY/HF8LWgUfB3fThvAcZZO4z85c9A?=
 =?us-ascii?Q?3NKk4I/T47K/KJxZovsiYjpZIDa44Wx4tOv1fE8Zopa/tvmZfnm3tNqFzuIw?=
 =?us-ascii?Q?uq4VSzj4xEmozEzG1zD03W+o7uMKTJ1TSIoziD/frFvaz5lX/GufvwWYS27O?=
 =?us-ascii?Q?Upk7YUX7V4owyShBQKaeiIvklgTheWjjHsLwYrW1rM0rFNA6nkQ55ZtnesMa?=
 =?us-ascii?Q?im/r166XYVwv3KnZHCtowDV84iF6/qoeTIFXdmgtqaeWqOqhefbw3Nu4dSQF?=
 =?us-ascii?Q?4EYkzag+IaK9VkbY4DSVrBQP09PRtVpIcEsBlXOIUgem/lPF+jiYFNzKqaVD?=
 =?us-ascii?Q?YRI6fQ9vbJIHV2S/4L7el6FM2gxS+J9p3m15ZkjfMg6gClcNWfNQ8qhWwzg0?=
 =?us-ascii?Q?lYn14BFqr6yXHmL1QnXm7G7cw7Hx3t5yTSaVGe43q2C6JGakBPdqJWOUZZZC?=
 =?us-ascii?Q?52ULHLQfqtXDHSWtifWa7oelgXbzxrR//2YMA2x4rlv1lmjm/QLQB0twxmQB?=
 =?us-ascii?Q?XEF5NrvhppPZX38QKrBBdC6JNJWPus8JZWeDvypoI/CHvTi0SiveWdMA+Ozf?=
 =?us-ascii?Q?kncFEkLhasSN+KotcfnD1g7mnNwUfqxZPFOyoBqUL2lY7GHFW3xwAIrzjKIy?=
 =?us-ascii?Q?LgEkvr13cwpt22mcxIh0IA2jtbFJuvognDHqhWfjXgJyrfCwkVybR8JGAN7G?=
 =?us-ascii?Q?CiGHLrLKd/C9GfPlOyo2PIJimFy79LRJKI2jJdlXRl/DSl1AGV1SlarIY7Hh?=
 =?us-ascii?Q?feI6lRXcouHU13gB3/LqBSFMBJHuW3poZ6VqhGGcXgcEVkbDCem0kNEAHyXl?=
 =?us-ascii?Q?WX5MtuiFo86zdm1PJyd7n/6s4ijEHk2q6UE6u8MsEPZBuCI8FyRWRFCfT0tv?=
 =?us-ascii?Q?0BcLZrW/ZaGNa2tWYSRdWII9SQkeEXuO5tSfisgri/unTLjJo/Xozc4ebWnX?=
 =?us-ascii?Q?YzUfce9jHkREJsR39X4XRc83IUQ6jTs+xOJathCyLlWo3HVf/lRD8ZcA5u3/?=
 =?us-ascii?Q?gmyKFTW/R/0EtdEefojPfC/yFp3Bb79GmB6UDhdt?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4931.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c3a25801-4084-4fa6-a7d0-08db61c22604
X-MS-Exchange-CrossTenant-originalarrivaltime: 31 May 2023 10:31:09.1958
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: wQo4uRrV5nah2UZkom/STGV2ZWkKzqgWc266zXKQ2+1LaaeIji0FOXh+gC4/hymZnbvY2yDtUPh+7owATLbDmA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR11MB7117
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

As Eric said, CONFIG_RANDSTRUCT_NONE is set in the default config=20
and some production environments, including Ali Cloud. Therefore, it=20
is worthful to optimize the file struct layout.

Here are the syscall test results of unixbench.

Command: numactl -C 3-18 ./Run -c 16 syscall

Without patch
------------------------
224 CPUs in system; running 16 parallel copies of tests
System Call Overhead                        5611223.7 lps   (10.0 s, 7 samp=
les)
System Benchmarks Partial Index              BASELINE       RESULT    INDEX
System Call Overhead                          15000.0    5611223.7   3740.8
                                                                   =3D=3D=
=3D=3D=3D=3D=3D=3D
System Benchmarks Index Score (Partial Only)                         3740.8

With patch
------------------------------------------------------------------------
224 CPUs in system; running 16 parallel copies of tests
System Call Overhead                        7567076.6 lps   (10.0 s, 7 samp=
les)
System Benchmarks Partial Index              BASELINE       RESULT    INDEX
System Call Overhead                          15000.0    7567076.6   5044.7
                                                                   =3D=3D=
=3D=3D=3D=3D=3D=3D
System Benchmarks Index Score (Partial Only)                         5044.7

> -----Original Message-----
> From: Eric Biggers <ebiggers@kernel.org>
> Sent: Wednesday, May 31, 2023 9:56 AM
> To: Christian Brauner <brauner@kernel.org>
> Cc: Chen, Zhiyin <zhiyin.chen@intel.com>; viro@zeniv.linux.org.uk; linux-
> fsdevel@vger.kernel.org; linux-kernel@vger.kernel.org; Zou, Nanhai
> <nanhai.zou@intel.com>
> Subject: Re: [PATCH] fs.h: Optimize file struct to prevent false sharing
>=20
> On Tue, May 30, 2023 at 10:50:42AM +0200, Christian Brauner wrote:
> > On Mon, May 29, 2023 at 10:06:26PM -0400, chenzhiyin wrote:
> > > In the syscall test of UnixBench, performance regression occurred
> > > due to false sharing.
> > >
> > > The lock and atomic members, including file::f_lock, file::f_count
> > > and file::f_pos_lock are highly contended and frequently updated in
> > > the high-concurrency test scenarios. perf c2c indentified one
> > > affected read access, file::f_op.
> > > To prevent false sharing, the layout of file struct is changed as
> > > following
> > > (A) f_lock, f_count and f_pos_lock are put together to share the
> > > same cache line.
> > > (B) The read mostly members, including f_path, f_inode, f_op are put
> > > into a separate cache line.
> > > (C) f_mode is put together with f_count, since they are used
> > > frequently at the same time.
> > >
> > > The optimization has been validated in the syscall test of
> > > UnixBench. performance gain is 30~50%, when the number of parallel
> > > jobs is 16.
> > >
> > > Signed-off-by: chenzhiyin <zhiyin.chen@intel.com>
> > > ---
> >
> > Sounds interesting, but can we see the actual numbers, please?
> > So struct file is marked with __randomize_layout which seems to make
> > this whole reordering pointless or at least only useful if the
> > structure randomization Kconfig is turned off. Is there any precedence
> > to optimizing structures that are marked as randomizable?
>=20
> Most people don't use CONFIG_RANDSTRUCT.  So it's still worth optimizing
> struct layouts for everyone else.
>=20
> - Eric