Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Geert Jansen <gerardu@amazon.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: RFC: return d_type for non-plus READDIR
Thread-Topic: RFC: return d_type for non-plus READDIR
Thread-Index: AQHXH4An0zoT5whq90uvEbXY1zJqrqqRsqoAgACtjwCAAMovAA==
Date:   Wed, 24 Mar 2021 13:50:52 +0000
Message-ID: <C70E02C8-DCB9-4096-B917-591A4EE21D76@oracle.com>
References: <20210323010057.GA129497@dev-dsk-gerardu-1d-3da90cb4.us-east-1.amazon.com>
 <689DD4DF-17C0-4776-BE53-7F10F7FFE720@oracle.com>
 <20210324014713.GA44499@gerardu>
In-Reply-To: <20210324014713.GA44499@gerardu>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <4805D3F4644DB241BE90A7752E5A41A2@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: f6501397-e583-4136-2c88-08d8eecbd71b
X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Mar 2021 13:50:52.6065
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: wqWYc2jtQzJfjYe0M7R29jcGCod3G1LIg4rjdBoGpDNAi9IO/BFOjnPj+BZjYsmOjMsyXrYM9WTif8O6/tN06g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4544
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9932 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=911
 malwarescore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2009150000 definitions=main-2103240105
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9932 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0
 lowpriorityscore=0 suspectscore=0 clxscore=1015 priorityscore=1501
 spamscore=0 adultscore=0 impostorscore=0 mlxlogscore=999 mlxscore=0
 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2009150000 definitions=main-2103240105
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

Hi Geert -

> On Mar 23, 2021, at 9:47 PM, Geert Jansen <gerardu@amazon.com> wrote:
>=20
> On Tue, Mar 23, 2021 at 03:26:02PM +0000, Chuck Lever III wrote:
>=20
>>> Since not all file servers may be able to produce the directory entry t=
ype
>>> efficiently, this could be implemented as a mount option that defaults =
off.
>>=20
>> Can you say more about the impact of requesting this attribute
>> from servers that cannot efficiently provide it? Which servers
>> and filesystems find it a problem, and how much of a problem is
>> it?
>=20
> The ability to satisfy a non-plus READDIR by reading just the directory
> pages, instead of having to read all dirent inodes as well, can be worth =
it
> for certain use cases (especially those with large directories). If a fil=
e
> system does not store d_type in the directory, and the client would alway=
s
> request the type attribute even for non-plus READDIR, then you lose the
> ability to make this optimization.
>=20
> From a review of the man pages, most local file systems appear to be able=
 to
> store d_type within the directory, including ext4, xfs and zfs. Both ext4
> and xfs have options to turn this behavior off. If you'd export such a fi=
le
> system using nfsd, then this would cause additional IO on the file system=
 if
> we would always request the type attribute.
>=20
> I do not know how other commercial servers handle this.

"How much of a problem is it" -- I guess what I really want to
see is some quantification of the problem, in numbers.

- Exactly which workloads benefit from having the DT information?
- How much do they improve?
- Which workloads are negatively impacted, and how much?
- How are workloads impacted if the client requests DT
information from servers that cannot support it efficiently?

Seems to me there will be some caching effects -- there are at
least two caches between the server's persistent storage and the
application. So I expect this will be a complex situation, at
best.

I totally agree that directory operations are a performance
and scalability sore spot for NFS, so I personally am interested
in hearing any and all suggestions in this area. In this case,
the proposed mechanism is intriguing and sensible, but I would
suggest that without measurement data, the proposal seems
incomplete so far.


>> I'd rather avoid adding another administrative knob unless it is
>> absolutely necessary... are there other options for controlling
>> whether the client requests this attribute?
>>=20
>> For example, is there a way for a server to decide not to provide
>> it if it would be burdensome to do so? ie, the client always asks,
>> but it would be up to the server to provide it if it can do so.
>=20
> I looked in the RFCs but I am not sure if there is a way today? Both 4.0 =
and
> 4.1 define "type" as a required attribute that needs to be returned if th=
e
> client asks for it. There also does not appear to be an enum value
> corresponding to DT_UNKNOWN. Were you thinking about something specifical=
ly?

I wasn't thinking of a particular protocol mechanism, though that
is certainly a possibility. I'm more interested in seeing if there
are ways to enable the proposed improvement without adding more
administrative complexity. Yet one more thing that can be set
incorrectly and has to be maintained in perpetuity.

So, alternatives might be:
- Always requesting the DT information
- Leveraging an existing mount option, like lookupcache=3D
- A sysfs setting or a module parameter
- A heuristic to guess when requesting the information is harmful
- Enabling the request based on directory size or some other static
feature of the directory
- If this information is of truly great benefit, approaching server
vendors to support it efficiently, and then have it always enabled
on clients

Adding an administrative knob means we don't have a good understanding
of how this setting is going to work. As an experimental feature, this
is a great way to go, but for a permanent, long-term thing, let's keep
in mind that client administration is a resource that has to scale
well to cohorts of 100s of thousands of systems. The simpler and more
automatic we can make it, the better off it will be for everyone.


> If there's no way to do this today, then I guess a per-file system attrib=
ute
> that indicates support for "can produce file type efficient when reading =
a
> directory" would would be a relatively clean solution. I presume it would
> require an RFC to define this attribute. Would you have a recommendation =
given
> your your experience with the RFC process?

My recommendation is to look for other alternatives first ;-)

It can't hurt to ask for advice from the nfsv4 working group, but
I would go in armed with some performance numbers.

--
Chuck Lever