Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Benjamin Coddington <bcodding@redhat.com>
CC:     Theodor Mittermair <tmittermair@cvl.tuwien.ac.at>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: NFS performance problem (readdir, getattr, actimeo, lookupcache)
Thread-Topic: NFS performance problem (readdir, getattr, actimeo, lookupcache)
Thread-Index: AQHZCRoQ0sR8+FbrrEeyEVhIhKcs+q5g2RaAgAATxAA=
Date:   Tue, 6 Dec 2022 14:32:13 +0000
Message-ID: <22200778-0055-454B-84F2-E633E321C834@oracle.com>
References: <dc3b95d2-93d0-992f-8f02-75c5bbb3bdff@cvl.tuwien.ac.at>
 <0193C32D-D74A-423C-AF08-58EB436FABDD@redhat.com>
In-Reply-To: <0193C32D-D74A-423C-AF08-58EB436FABDD@redhat.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Zwf5YL7/j1s3OAqM2rj67MY47Hpa6ZoVwkTgvLbpE525LtCBOZNQfdUB/jmm?=
 =?us-ascii?Q?8ITwJwnUVRO4RVbzujwS/7+CqOLFluQuUvTH85+91CYyD3iAP/pdYkXWg7Ox?=
 =?us-ascii?Q?L00zff4p2ubH2fakwGcKzHAO0PmsOFiDdumnB7fDcImG59xhgg0qYE9GJO5s?=
 =?us-ascii?Q?Bm8Xwft+wGRpJUpcykRwLbSm0CN079wWSQZZ03kyZYMvcD4AAjp0PGOQ4D0p?=
 =?us-ascii?Q?ICv/oW7eyFAlOLbCL47Go9fFIhBtHHVxwx1rAbNxVXpnjM9Xj/9PxLfNPAbi?=
 =?us-ascii?Q?DhYu85EkxobtgGGEAXgcnDG75SlYsgvzxPO1vrz5DAKdsPg3OqqUGijeVzMr?=
 =?us-ascii?Q?jgHmmn5JP8Z44NxfswSg300VnY++IPHv+kzopb7A4nI5NMAMHaSLC4ig+eHI?=
 =?us-ascii?Q?mY/6UED26CE7iDIAWyrWtnqReYExL+0ZsfOCkm8uv21lsHh2MK6U6Ghfkw+E?=
 =?us-ascii?Q?tPl0RMGnh4gL1Bzi/kybBYTzIFNuiaHQAVouZi5q1cpVWUaR0tWalA9oAa6i?=
 =?us-ascii?Q?qz6sH7pYmUPwTJXCAMBfzvJzzOhU/xQI8se4lqEJ4aWvuw1R2kHaCOfIKhOy?=
 =?us-ascii?Q?eX0EduANiBxbVVECG37suy2BYMsjG+ASWBRLq7+SEr8bfQxAYdCLNDTEBmAy?=
 =?us-ascii?Q?f8vQD268VyPleVuUqO8zN4gGTRMT5Jybmuh+NSFktc8Gj4r4ModquSfTF4mw?=
 =?us-ascii?Q?EGaXPmoT6bLJCLdQszdYVWpUfr1bxHUdTivSFp4ll3wyAxDiCSK8D/jHBHuP?=
 =?us-ascii?Q?lZ6GBCYvARpjws9gQigE3aywTvk28rFstMabHSqtQqjX50uRwyes0p5kLqHp?=
 =?us-ascii?Q?13F97eS6n43K9fx7+R5g5e9uXs968A1Fje3GqXZNXaMivLYg/ZfcgRvZYxyt?=
 =?us-ascii?Q?UfdtAOdsxQ54yFyCRk9xDclCQ15MornBBBdgvHm8LYqzCPLAzN+0Xfjcn5em?=
 =?us-ascii?Q?bXtaUVnYijsTRxgfK/A/5j/17mbHh75T1ETALd//Og/y82RrEAjVhFjvEP/e?=
 =?us-ascii?Q?87JRutXQnhPjhca8sbu4vHPLT5NKR5qV90ax1BKRv08wqiOi9K+FeIAN9ELp?=
 =?us-ascii?Q?RLVpISGPNpDBcZ6TiDMixbks0n4TJneuOrAT6ze5pf965c53oCIFe1rOudv/?=
 =?us-ascii?Q?JzkOLALPkIIhxR+GaHMJonDPNAb8sOn/Tg1D4D/MhgiO7gyi9bWOwqAmyU4b?=
 =?us-ascii?Q?Ph5h+p1XXNNtBaQeGAK5KpHOfuHFnC5uvpQVhg72RxSNFjEtokqixmyfKMs3?=
 =?us-ascii?Q?NLQa/0XfRBHpeO9BvunjPLmvCmlKI6SGW02890BXYKI9E16Jqlescqmpbceq?=
 =?us-ascii?Q?9SVW1wKG2VXJ++bQVe2dcRJm5U3kiQwBNtmBjm+E1E0FLS1PyMAk53SErHL7?=
 =?us-ascii?Q?zXCtOtvz0Ize6h/+M2De5hb+/UlEA//6JGInq5hfSSrT6eqo8W0Oca3c0Jdq?=
 =?us-ascii?Q?IQFzZ7T617F3YYc1UYEmjLvM1gQ09231d3iNB2utIaqtB64Oao9XJ5945F6X?=
 =?us-ascii?Q?JNGSjn/KMiS64ZXFAXrBZlxqrq752ks2iPgsyCxSkKl6QP1J4f0emwcWBDlT?=
 =?us-ascii?Q?IALsTJiQRTEiqkOChr9QcmFSQtVqYBaxKu2aEuKWGLivufKwz4sdaWqZDJ94?=
 =?us-ascii?Q?Aw=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <44038C981FE0E0428DED9801AF81FBA3@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: nXmsxfj+z8hA72dVkBPzmnHo5LvziKtBr9t2QwWXduNOHkWKlkppN8IjIjIXZS62b8vuwsd0SdJX4IL5mJn96ocoodVs8csWRA3iUjRUL+clUpShSwIZ7x0gInL/ForJOymULJm2VVCBF3GvQLC1uiDk9OLsOmbk/SGRAq5+ty1N0iPgL27D04A4T30EmA+rx4+KsbuT6u7MZ5ijc8QbSHXsEevGMDTmeSOuC0KRyXk0FMR3DLg3bgo2GvG+3KDdEzCJPlh4xD9h9LANyXPTr3K5PhnxSmogHJUKyeDftN2dc+JjetDhSz9VToF2qX6RWCiY5ovOPpzqOemGrvHHpJzh3VTd5X80//MxXy/fnb89m5IJJP6xMEEf0AB7paP2+w0zSVBb+44q7CgYoxt94fyQDFTHiA/onC9zICvtT10ZiLzDJAF4M4+JyvEFmC6bXxgXYi/zE5J7vcHrrF+822BgcbqhEEMTzw+iphOp5wiLevisSsZVqnlmBRT9LdosAujikLIqx/RZUcTg32xB9fhssgwKgmCzNrl5hudsD1jRBJK+82mwJ+avvHd9278MOQ60nCcoT2mmefXfa52cw2pUkxt2APFeKH2oIzcWUYbigw2i1Y1B3hj+PvnOsZ18PUVnhWDheUPxRElvkdoA8sWPOc0XIP2L2InR/7cnXlDgslFYQR5PBAqjerpCjxGRjTl5XIvWBqE7WmtAct8EqQafn4CwNPvlQAgHnzSCJA6HbnxVihE5zXbFvHCwYShpZJAk08GRnB1if9e61jsMsC1Iu0p9I+1ZDr5yEvEp6/CwwE/v51GINeRQSxTYKs4xjAiQhfc1rmTmf6BvrRPm6A==
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: acae0554-8c88-4ee2-ee5e-08dad796aa8b
X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Dec 2022 14:32:13.2241
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: dLP/IDJ37FgLetlSlqhk3zgj7Sj+SLU2fqVtOQ0cFaEqRiyg7inPSwtoIiBZWGwTRtTm5hc4vUMDHnE69NlsDA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN0PR10MB4981
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2022-12-06_10,2022-12-06_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 phishscore=0
 mlxlogscore=999 mlxscore=0 spamscore=0 malwarescore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000
 definitions=main-2212060119
X-Proofpoint-ORIG-GUID: M9GNhkM1QKfG94-VG81Z1lEycVl1AMf0
X-Proofpoint-GUID: M9GNhkM1QKfG94-VG81Z1lEycVl1AMf0
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Dec 6, 2022, at 8:21 AM, Benjamin Coddington <bcodding@redhat.com> wro=
te:
>=20
> On 5 Dec 2022, at 21:18, Theodor Mittermair wrote:
>=20
>> Hello,
>=20
> Hi Theodor,
>=20
> .. snip ..
>=20
>> From what i gathered around the internet and understood, there seem to b=
e
>> heuristics involved when the client decides what operations to transmit =
to
>> the server.  Also, the timed-out cache seems to be creating what some
>> called a "getattr storm", which i understand in theory.
>=20
> When `du` gathers information, it does so by switching between two syscal=
ls:
> getdents() and stat() (or some equivalents).  The getdents() syscall caus=
es
> the NFS client to perform either READDIR or READDIRPLUS - the choice of
> which is governed by a heuristic.  The heuristic can only intelligently
> determine which readdir operation to use based on whether the program is
> performing this pattern of getdents(), stat(), stat(), stat(), getdents()=
,
> stat(), stat(), stat().  The way it can tell is by checking if each inode=
's
> attributes have been cached, so the cache timeouts end up coming into pla=
y.
>=20
>> But why does the first request manage to be smarter about it, since it
>> gathers the same information about the exact same files?
>=20
> It's not smarter, it just optimistically uses READDIRPLUS on the very fir=
st
> call of getdents() for a directory, but can only do so if the directory's
> dentries have not yet been cached.  If they /are/ cached, but each dentry=
's
> individual attributes have timed out, then the client must send an
> individual GETATTR for each entry.
>=20
> What is happening for you is that your attribute caches for each inode ar=
e
> timing out, but the overall directory's dentry list is not changing.
> There's no need to send /any/ readdir operations - so the heuristic doesn=
't
> send READDIRPLUS and you end up with one full pile of getdents() results =
of
> individual GETATTRs for every entry.

Are those GETATTRs emitted one at a time sequentially, or concurrently?

--
Chuck Lever