Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Neil Brown <neilb@suse.de>
CC:     Jeff Layton <jlayton@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH v3] nfsd: rework hashtable handling in
 nfsd_do_file_acquire
Thread-Topic: [PATCH v3] nfsd: rework hashtable handling in
 nfsd_do_file_acquire
Thread-Index: AQHY1xwiPOG/BlB/KUiRs5xLTvyF4638pTOAgACVooCAAQE+gA==
Date:   Tue, 4 Oct 2022 13:28:12 +0000
Message-ID: <4FF85113-6F17-4F3C-AD31-E2472A988618@oracle.com>
References: <20221003113436.24161-1-jlayton@kernel.org>
 <F4DF35B2-CE11-4BD9-8442-97852F57CE2E@oracle.com>
 <166483484979.14457.9448463531121052564@noble.neil.brown.name>
In-Reply-To: <166483484979.14457.9448463531121052564@noble.neil.brown.name>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Oo/egr1FyirwGm/yPjishEdW/KdTfapopvH5RnGcSE1Z5bWTPGYV0EVsZov1?=
 =?us-ascii?Q?ODlD1VSNticOMC+5UfJyprQBnovOU7ipsFMD20mULTUIvm9hxbvVM6AD9mzy?=
 =?us-ascii?Q?WlxcHYnmC92De3lGdi83Q9kStOtpzXd2ivcAqzjygRJ74NS7V63lf5wmvhJx?=
 =?us-ascii?Q?rOr/JJEaKHIIatwAkr4NpjHEpZzdakRRnkqZAtEqLxeNBnSRpi/OPJEkActV?=
 =?us-ascii?Q?4rl+am30mzIFcyU5J3RWjm52NQ40lqpAXd5rjS6kNslitAXIgw8TbxmPurWU?=
 =?us-ascii?Q?A5RkoPmX6I+XLWiN7XHbXSd/ykgFaclgmCZ3yWKCuBVtqDQoQpcH/1mlv2Kp?=
 =?us-ascii?Q?bz65FIaZuQfPjn3m23NmRHZqhzT3ISDYCod5DDTQ5ZhMNA8ZpIgETOa2zHed?=
 =?us-ascii?Q?8MUH/lNxjdtTPgrwJnQVeOsBpkbGDBv89Db9hBjA54GGFdS1TDEZIkbd//9w?=
 =?us-ascii?Q?3uUt3oxmamaIXIvhe3jXHTcAYXStYOtw2tb48JOb0Lz88TVj+Ru1+9b5Ltea?=
 =?us-ascii?Q?/4orik37hSdzlfPkyJKEiBjlU3YloeOgLOytoeX7W1ZACgtlJbZnv5zNHrYN?=
 =?us-ascii?Q?6W/EqH/6TDK4sB47jIrDAjY41fkRGKt05qiy/nkOZu5f8PU20a+mVj0Ruorf?=
 =?us-ascii?Q?ryuyWepLThZQmCh+PCAB8OmaCTYORBN9XwoSbLKSEPeQadc2a584l+kErUqH?=
 =?us-ascii?Q?s5SC8P4TOfXB2+wLVegSzJEOOkz+hWCdQxqjI7yX2U4BIgK6ko0TWlx0gHX/?=
 =?us-ascii?Q?EDabetM5MK6n0z3ArL5jy9gGfJEwsOuHC3Tni+IEVbr1SmP2ilKsWxJqD/pk?=
 =?us-ascii?Q?GKNykqEYNt9bK7yzAoXTtVxNrZFWkOoXApojQEWblZ8CoMm1pjvo7oKdg6PA?=
 =?us-ascii?Q?6XB//GTeOnDboOMMiXydBBYOCdYLUds3VdVI+VoHHTdofVCdijv71ShctSKI?=
 =?us-ascii?Q?Fwpmx+waL3hLskxlWjSiG0Pi4Vp09+RdcVdnKxL2z3M7CFLU/I8gLxteazx5?=
 =?us-ascii?Q?WnPYk38Qge76Kc9GdfpDQUD5kpnHkwGH9mXAbspTqK/nZoQPCvPdDmIniLOJ?=
 =?us-ascii?Q?dbsjuwiEtvZFV5UeRU8fAlEE61dttF9C3V2ZQIN514d4c8M5uozf8x2QgZsF?=
 =?us-ascii?Q?vdIK71Ojdki5G3XiH0JMuh4bk/e7lH3T9uJeKOcDnOMgInSmhkO5VXifg1L6?=
 =?us-ascii?Q?oIal8YhVvv0x+nsBAWwDiBpv4wEwrthfKDiAaQD7v4nKjq2SE1MeEMSgWXAU?=
 =?us-ascii?Q?6xOHT4/rjRgykJLQgQgDFWDCaPL8nKdp9ZYCkPGIaPdF34BJu7g6s7/EJKpR?=
 =?us-ascii?Q?lyJnXfKfLm+F5yP0LMmni8eAnqQm4jaSyUDjuaXV8jCux4bgN6YxDWRWl2QF?=
 =?us-ascii?Q?Zl3rcR2FvuPSiT+rvbRRNP11ggnd8mFpMndqXCqTCRspUho/Q6xKqoj5QQtu?=
 =?us-ascii?Q?elTErvgt5cswELeBKb3f+ud4MfPq/rqI4BKpSTuNjckZpX5DuWJk9TSSgIlf?=
 =?us-ascii?Q?FzD9ga2HOBHpIdf84uRrUf9S9ZRfN8uNPPeM4pLysNzw+3+soZgvPzZb7BAx?=
 =?us-ascii?Q?sIzx+zQ6tXMxW6zyvkXr6iCxzRdGSc5NoES2nr4IQ4IWnXobfhy3XagjtMN6?=
 =?us-ascii?Q?kQ=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <F1C0A7F749171B43BC4D46E93619DB04@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 407962a0-950b-43e8-ba5e-08daa60c4914
X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Oct 2022 13:28:12.1635
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: hFNznfWsl4K0QKPXDoJypwoWfGXuwlCM+nj1Vb3Hrj7QnujelClNN29YFi5qNUTFhjl+CXNLrd2dv7R4P1Pg0w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4726
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1
 definitions=2022-10-04_05,2022-09-29_03,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 mlxlogscore=999
 bulkscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000
 definitions=main-2210040087
X-Proofpoint-GUID: yMY6eBIHMYYfRAAxRZ4PH0hLC27BWPTj
X-Proofpoint-ORIG-GUID: yMY6eBIHMYYfRAAxRZ4PH0hLC27BWPTj
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Oct 3, 2022, at 6:07 PM, NeilBrown <neilb@suse.de> wrote:
>=20
> On Tue, 04 Oct 2022, Chuck Lever III wrote:
>>=20
>>> On Oct 3, 2022, at 7:34 AM, Jeff Layton <jlayton@kernel.org> wrote:
>>>=20
>>> nfsd_file is RCU-freed, so we need to hold the rcu_read_lock long enoug=
h
>>> to get a reference after finding it in the hash. Take the
>>> rcu_read_lock() and call rhashtable_lookup directly.
>>>=20
>>> Switch to using rhashtable_lookup_insert_key as well, and use the usual
>>> retry mechanism if we hit an -EEXIST. Eliminiate the insert_err goto
>>> target as well.
>>=20
>> The insert_err goto is there to remove a very rare case from
>> the hot path. I'd kinda like to keep that feature of this code.
>=20
> ????
> The fast path in the new code looks quite clean - what concerns you?
> Maybe a "likely()" annotation can be used to encourage the compiler to
> optimise for the non-error path so the error-handling gets moved
> out-of-line (assuming it isn't already), but don't think the new code
> needs that goto.

It's an instruction cache footprint issue.

A CPU populates its instruction cache by reading instructions from
memory in bulk (some multiple of the size of the cacheline). I
would like to keep the instructions involved with very rare cases
(like this one) out-of-line so they do not clutter the CPU's
instruction cache.

Unfortunately the compiler on my system has decided to place this
snippet of code right before the out_status: label, which defeats
my intention.


--
Chuck Lever