Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Bruce Fields <bfields@fieldses.org>
CC:     Daire Byrne <daire@dneg.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: parallel file create rates (+high latency)
Thread-Topic: parallel file create rates (+high latency)
Thread-Index: AQHYELR6RkgsqUSQAkGSQQ83MPCBH6xykl+AgAAI+4CAAAtagIABDMkAgAASyICAABlKAA==
Date:   Tue, 25 Jan 2022 15:30:31 +0000
Message-ID: <F7C721F7-D906-426F-814F-4D3F34AD6FB1@oracle.com>
References: <CAPt2mGOaRsKOiL_wuSK_D5oYYnn0R-pvVsZc5HYGdEbT2FngtQ@mail.gmail.com>
 <20220124193759.GA4975@fieldses.org>
 <CAPt2mGOCn5OaeZm24+zh92qRcWTF8h-H2WXqScz9RMfo4r_-Qw@mail.gmail.com>
 <20220124205045.GB4975@fieldses.org>
 <CAPt2mGPTGgXztawDJfAKsiYqnm6P_mn1rtquSDKjpnSgvJH1YA@mail.gmail.com>
 <20220125135959.GA15537@fieldses.org>
In-Reply-To: <20220125135959.GA15537@fieldses.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?QzX7Zq4z5xu0HZzSSXfrKikS3yMJ+BmVV598FICGlw36+5iNEpzEd5B16laz?=
 =?us-ascii?Q?MDvrcyvgj+y8KxsYtpu9onfV93BuWRVWfrLefNqqB7eY6oG3/SuAhUMBKwJK?=
 =?us-ascii?Q?w/VsUGHHw4eJrNIlDYWduESvHxMTLXden4uTXR1WdytV+VuVV3iIlUp9gF9g?=
 =?us-ascii?Q?yUE4Had3QN67sLeBP05/7BU91F3gVyt/EFFi1+wLpv8ea5QmQ5KwHcplFfJV?=
 =?us-ascii?Q?8IWvGtTjg0slRS7/PKoiv7TJWNbBvIHKGuAzPyChLMs0Xs2uAy7x9mKprN4k?=
 =?us-ascii?Q?e8ORMPxh1NbctA4B7nNHbViXfyOwHMNo3arewgtn6ipvynjTPaXNJrxt1oBn?=
 =?us-ascii?Q?jev3O0ZkOKmf+Jnzb0epk1QXuGepn5Il5ewnzpms4JCwEtno3m9dGUhsGxXP?=
 =?us-ascii?Q?fg29ZdC5TP2EQrGW/KH0t14+HZ9yaRtHjujwJlp08Nr617dCrJ48ZH+4KuxR?=
 =?us-ascii?Q?oz22Afvn+A8UMwFL4QwySpx6opx5gSPlBR1w9ys5pZrovNG9R/H98mXYqO0C?=
 =?us-ascii?Q?m8iyYidE7JWQKr68hJ1CkLJBv2rzUQX5dwQTEUCuDvB7XII6XMHy1z/Zpmj2?=
 =?us-ascii?Q?uVW2z415LKnzLAYS+AUwuEMAhezhPYObi1ZNVvWEKEKS03D2XoIC5ma5W0A9?=
 =?us-ascii?Q?RHHoa9PMAvu9tPqyHaOXF601vd5xd/9MKNZ5D+goFMZASdme1XjH3J853vGc?=
 =?us-ascii?Q?evhBvgyItO57O3YEhUC2u3zUyjJyLBT3TnBhk3JbgkBe3xVH1DctoF42FBnl?=
 =?us-ascii?Q?zhcOkpVK1DOJs8aYS1pSarLkVxfupF5IpqqClS7x+WGo4EE1MbinC8kw2V2M?=
 =?us-ascii?Q?MvcmfX1v2fVh2Hi4HMJag7wxeZ6676y7LDxO5DE9yhDJ34gok35jDdPrnL3R?=
 =?us-ascii?Q?mruGR9X3ag2UwxGMsgygNIlVHUDZERzp6msjDKtBqe+netp1AjFyNEix2Iyd?=
 =?us-ascii?Q?HXhFr9mrIWb1EKWiMgL3nv9nf1t27MxJxmzJb4TjnzRIA7YyQRq5Dqz7vxJU?=
 =?us-ascii?Q?qvTy/lvYPaZmgViMyrtHV8G/Lq7spPQaUMLO7gMRG3G9eWAu60E2V8GIgn04?=
 =?us-ascii?Q?PALgFrDuMlAb0g7aBQcaHJhknqV4ugrAeVXdZY6TYOSVCGPslUfO613y1sRO?=
 =?us-ascii?Q?VSMGcBXFHpYDH580Pwsfd+KfVdXBSNKsmMfUUa3erzQHM+L1UIn1q+I4ZLaV?=
 =?us-ascii?Q?tw0VmGKBz3mcjFdgrAU61DzPFoatXdor4zMOK1bDAgf/CrUtS4vLOcmIMsMF?=
 =?us-ascii?Q?ULw2RTUz68KxPFsBRp+O01TdjGTMre+yhQcmDdpoau5wVWf+Pf8kPePw1e64?=
 =?us-ascii?Q?TiiSwwBUNf5y/weDVTVqEBJcg8dI4I0jjYc3N0q3PpgMoSkxOqN1vK0ixPF3?=
 =?us-ascii?Q?hoSgndzd7bAWkZPXEMoh64KfpDF0duJ4khKHn+Rs4QdEOzzGQko4Wn+1qyll?=
 =?us-ascii?Q?Fng+PMXqsahJMW5wwYgdwl1VRa9wwvOA8dHqGrRC8Q05agBr7142P3Re0c8k?=
 =?us-ascii?Q?ONk/dmyHlqMuQqwTl5tqVz4rQxPgWF6ZclxYbLMjDbrkME8+uqex15hZMFqD?=
 =?us-ascii?Q?rlh/kJQZJUte6K+q/mswoE9WKkSLrEXVCgjZ/pnx++N8Mv2i5GBT53px1pc2?=
 =?us-ascii?Q?qTtHQ8AyDGIIi36WRkxenlc=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <0C0CA15DBB3EEF4990F0F086459ABB9F@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CH0PR10MB4858.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b2958671-eac4-4f5f-1871-08d9e0179f57
X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Jan 2022 15:30:31.0853
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 0Bd+IXWS+ep98eQuXZ0pZXtxoVSVioM3pCoDjAUG53BueIqn8pcqBZJWtRTyJdK9GNVXdI4LhWit/MWV9wHX9g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR10MB1919
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10237 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 phishscore=0 bulkscore=0
 suspectscore=0 malwarescore=0 adultscore=0 mlxlogscore=999 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000
 definitions=main-2201250099
X-Proofpoint-GUID: 5W1H970I5NDtjhqz6Hl2vGYP2ha8c8HS
X-Proofpoint-ORIG-GUID: 5W1H970I5NDtjhqz6Hl2vGYP2ha8c8HS
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jan 25, 2022, at 8:59 AM, J. Bruce Fields <bfields@fieldses.org> wrote=
:
>=20
> On Tue, Jan 25, 2022 at 12:52:46PM +0000, Daire Byrne wrote:
>> Yea, it does seem like the server is the ultimate arbitrar and the
>> fact that multiple clients can achieve much higher rates of
>> parallelism does suggest that the VFS locking per client is somewhat
>> redundant and limiting (in this super niche case).
>=20
> It doesn't seem *so* weird to have a server with fast storage a long
> round-trip time away, in which case the client-side operation could take
> several orders of magnitude longer than the server.
>=20
> Though even if the client locking wasn't a factor, you might still have
> to do some work to take advantage of that.  (E.g. if your workload is
> just a single "untar"--it still waits for one create before doing the
> next one).

Note that this is also an issue for data center area filesystems, where
back-end replication of metadata updates makes creates and deletes as
slow as if they were being done on storage hundreds of miles away.

The solution of choice appears to be to replace tar/rsync and such
tools with versions that are smarter about parallelizing file creation
and deletion.


--
Chuck Lever