Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Patrick Goetz <pgoetz@math.utexas.edu>
CC:     Bruce Fields <bfields@fieldses.org>, Daire Byrne <daire@dneg.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: parallel file create rates (+high latency)
Thread-Topic: parallel file create rates (+high latency)
Thread-Index: AQHYELR6RkgsqUSQAkGSQQ83MPCBH6xykl+AgAAI+4CAAAtagIABDMkAgAASyICAABlKAIAAag6AgAACQ4A=
Date:   Tue, 25 Jan 2022 21:58:11 +0000
Message-ID: <8BF65541-6AE3-45EA-A582-5773C0AAB399@oracle.com>
References: <CAPt2mGOaRsKOiL_wuSK_D5oYYnn0R-pvVsZc5HYGdEbT2FngtQ@mail.gmail.com>
 <20220124193759.GA4975@fieldses.org>
 <CAPt2mGOCn5OaeZm24+zh92qRcWTF8h-H2WXqScz9RMfo4r_-Qw@mail.gmail.com>
 <20220124205045.GB4975@fieldses.org>
 <CAPt2mGPTGgXztawDJfAKsiYqnm6P_mn1rtquSDKjpnSgvJH1YA@mail.gmail.com>
 <20220125135959.GA15537@fieldses.org>
 <F7C721F7-D906-426F-814F-4D3F34AD6FB1@oracle.com>
 <42867c2c-1ab3-9bb6-0e5a-57d13d667bc6@math.utexas.edu>
In-Reply-To: <42867c2c-1ab3-9bb6-0e5a-57d13d667bc6@math.utexas.edu>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?wZ/Q44UjaoR/rDqMCTtvVKKp7zqVI97/NUadLCNEAkIrnyTenCOQfe+JlMaz?=
 =?us-ascii?Q?YZcWT1M6A6WuevlMGA2EdYr/RFUOFs532muhWS1fi9KXDlKEqb7WkoAYREn4?=
 =?us-ascii?Q?93gi9fjxgK711/jDNYJf7nFk/SeermgYoO9JjUtiai65sfUPXW8MtkPSxqsN?=
 =?us-ascii?Q?XPKSbx2fQW1DwTRJlALHWF+nN5CE8GZMjI/7sy3VZvdu+l09m4SNAtBMTbMs?=
 =?us-ascii?Q?S8DyovbpZ2LaS2k5mWw5Uj7Ekcq6vJZTF37tud+ZDViqo98fNJtu6czchT2U?=
 =?us-ascii?Q?EEctO/MFvjTnZOe0kxsUotzpKG0WK5yX0rngDdf4qgGG2TYv0R8fjXtLZUjV?=
 =?us-ascii?Q?VpkJZOAp6dA6yZJd47ONf/ZcWd7w4sVizc9FMlOciPDzJBeHW3Y6sQo2BY5P?=
 =?us-ascii?Q?to2rlL5RGJ3bgxIZ30FZyccJah797I+ClimGREzXz8FpVxMJxEFyBnlNO4se?=
 =?us-ascii?Q?iq/EQJ53yXYI0Iqqk5sjAAJPdnvp+uapu+AZv3edav/g7ov+uh+C9JE80h2e?=
 =?us-ascii?Q?olv/fjlxzH6OwIh+UpCYbZ+44vkcLTUIjItktiANS8qiPm3kssosI8Bd+TjV?=
 =?us-ascii?Q?ZjNQfuT7bMOv5vOlVB1zy6j8PTTCD0PzkwhNQviBri984DwQunoDRTnjimzu?=
 =?us-ascii?Q?qBynQEoWFYvv68XF86G7CDMOiSqpMa0CXJ2De8Lfd3ORVh8LlxY5GSlkrH46?=
 =?us-ascii?Q?Nzf+52U1BIEBnX8eoopEqaMtewYrEEMC+9FdRChVA5G58EsgrbIYYbgN8uJO?=
 =?us-ascii?Q?EcFhy76NnloLICIqcHug7woa/sUo5ltbV6IQh2QTsZZt2BOE4JBDrfE1tnbg?=
 =?us-ascii?Q?HImg5e00b7rsQcfv129YXOnnwcUt+Wwf9Vn90miFIGS7kbh/rwDqRNi8UHTW?=
 =?us-ascii?Q?GXH63QapU6Se2oMrevIbunu2kBGzu7B0OVnyZ5u+743sZjxfC1/S+R8RKCGF?=
 =?us-ascii?Q?ZygOieLJdomGbvtbTUhhS1KrQvGR+OWMryN4ddNDt0JKDo4Y3Foka405/TR1?=
 =?us-ascii?Q?B5u44a3/6r9S4zXe6b3V9NUDcNXuSEmWCglv+7sJrvlRzm26l9aot9m4hIJP?=
 =?us-ascii?Q?XQqwscLWs9fDdp8dCjHMg06U3/QlD/l6Orx3mm02qZAJPaUEMWGn0hsMSw3+?=
 =?us-ascii?Q?QfZir6ElH4xKyZzJeWgl8/3iUa1iXqfLBRFC9UDfkY4p/hDbXE6A3TtKiaC/?=
 =?us-ascii?Q?3a+ahJVqMkuiodvnMy/tt3zZyskz4Hksk70tlJ7AdV7ew6snPCuEeAaAx/uO?=
 =?us-ascii?Q?+WRtWmuwkkLeUtObM9HPgswd1/GyuHnfqr8TKMXE7xArd1LgBUDGKyakt4XZ?=
 =?us-ascii?Q?eSd+xMefRVw3+sDrDkNRBPkzztTgqManLm6J4BekBljURKan5HvfmMznf1CE?=
 =?us-ascii?Q?lEd+ARg44ogtWNRlh/0n8pFhCz3NzDhuoEGMGCo4tITGPudp4AMslmbJ+av5?=
 =?us-ascii?Q?kaVqxsjD767UJkpxALfMyrLpObUv/F6vXB6OrEkWLxnL+mFVykZbgh19DuDs?=
 =?us-ascii?Q?PGufqOQYje+x+zapK255MyBa87s4vk2bagGXlkkAqzNDBogCwB8INweqLC91?=
 =?us-ascii?Q?hsWIb5n8z87PU0SSG49TMVaNDzpJY+0gcYszHILnMxbOjZr1BmYtYdh2D2tR?=
 =?us-ascii?Q?yv6S7rMqsQxB2qEwn7OFSP8=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <F961F86E1D3A144C990A42A4302AD6EA@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CH0PR10MB4858.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4083d223-42cc-4c44-257a-08d9e04dc7a0
X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Jan 2022 21:58:11.5432
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: nUBlCSJn6ravX0KCvSYgH90izGIHP6e6oHh0463pXGUKxdH7nSRWWntDeJX4Et437N9XqcedGe8AzolSbxlIQw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2432
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10238 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 malwarescore=0
 spamscore=0 phishscore=0 suspectscore=0 adultscore=0 mlxlogscore=999
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000
 definitions=main-2201250131
X-Proofpoint-GUID: 4tr4s87QifYbjXWp02tQZru5jf6KrIZI
X-Proofpoint-ORIG-GUID: 4tr4s87QifYbjXWp02tQZru5jf6KrIZI
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jan 25, 2022, at 4:50 PM, Patrick Goetz <pgoetz@math.utexas.edu> wrote=
:
>=20
>=20
>=20
> On 1/25/22 09:30, Chuck Lever III wrote:
>>> On Jan 25, 2022, at 8:59 AM, J. Bruce Fields <bfields@fieldses.org> wro=
te:
>>>=20
>>> On Tue, Jan 25, 2022 at 12:52:46PM +0000, Daire Byrne wrote:
>>>> Yea, it does seem like the server is the ultimate arbitrar and the
>>>> fact that multiple clients can achieve much higher rates of
>>>> parallelism does suggest that the VFS locking per client is somewhat
>>>> redundant and limiting (in this super niche case).
>>>=20
>>> It doesn't seem *so* weird to have a server with fast storage a long
>>> round-trip time away, in which case the client-side operation could tak=
e
>>> several orders of magnitude longer than the server.
>>>=20
>>> Though even if the client locking wasn't a factor, you might still have
>>> to do some work to take advantage of that.  (E.g. if your workload is
>>> just a single "untar"--it still waits for one create before doing the
>>> next one).
>> Note that this is also an issue for data center area filesystems, where
>> back-end replication of metadata updates makes creates and deletes as
>> slow as if they were being done on storage hundreds of miles away.
>> The solution of choice appears to be to replace tar/rsync and such
>> tools with versions that are smarter about parallelizing file creation
>> and deletion.
>=20
> Are these tools available to mere mortals?  If so, what are they called. =
 This is a problem I'm currently dealing with; trying to back up hundreds o=
f terabytes of image data.

They are available to cloud customers (like Oracle and Amazon) I believe,
and possibly for Azure folks too. Try Google, I'm sorry I don't have a
link handy.

parcp? Something like that.

--
Chuck Lever