Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Nagendra Tomar <Nagendra.Tomar@microsoft.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Trond Myklebust <trond.myklebust@hammerspace.com>,
        Anna Schumaker <anna.schumaker@netapp.com>
Subject: Re: [PATCH 0/5] nfs: Add mount option for forcing RPC requests for
 one file over one connection
Thread-Topic: [PATCH 0/5] nfs: Add mount option for forcing RPC requests for
 one file over one connection
Thread-Index: Adcfp3hcmhRvn3IHQv25MQlIHJPMUAARGOoAAAMHR+AAAek9AAAAU0WQAAIhaYAAALhG4AABaUIA
Date:   Tue, 23 Mar 2021 18:25:57 +0000
Message-ID: <D13EEDD3-522C-4AA3-9FC8-907A582EC57D@oracle.com>
References: <SG2P153MB0361E2D69F8AD8AFBF9E7CD89E649@SG2P153MB0361.APCP153.PROD.OUTLOOK.COM>
 <CD47CE3E-0C07-4019-94AF-5474D531B12D@oracle.com>
 <SG2P153MB0361BCD6D1F8F86E0648E3EB9E649@SG2P153MB0361.APCP153.PROD.OUTLOOK.COM>
 <5B030422-09B7-470D-9C7A-18C666F5817D@oracle.com>
 <SG2P153MB0361DC1EF8EDD02356D29E579E649@SG2P153MB0361.APCP153.PROD.OUTLOOK.COM>
 <575FF32B-463E-41D8-882F-6A746DF7FDFA@oracle.com>
 <SG2P153MB0361455966B8436516A355729E649@SG2P153MB0361.APCP153.PROD.OUTLOOK.COM>
In-Reply-To: <SG2P153MB0361455966B8436516A355729E649@SG2P153MB0361.APCP153.PROD.OUTLOOK.COM>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <3C7F832C29FF844890F2BF7F93FDCE3E@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 42dbb916-5298-4fbb-f295-08d8ee291a5f
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Mar 2021 18:25:57.4837
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: s1pQeC47iMGAzJntq7dbbkG/HY+5fuvTwA+NYidJ0Fl2fGL8Vr60rrUo2vDY6V3B89oVdPH0G868Zygaw25bpw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3223
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9932 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 spamscore=0
 mlxscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000
 definitions=main-2103230135
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9932 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 lowpriorityscore=0 phishscore=0
 mlxlogscore=999 priorityscore=1501 impostorscore=0 bulkscore=0 spamscore=0
 adultscore=0 clxscore=1015 malwarescore=0 mlxscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000
 definitions=main-2103230135
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Mar 23, 2021, at 2:01 PM, Nagendra Tomar <Nagendra.Tomar@microsoft.com=
> wrote:
>=20
>>> On Mar 23, 2021, at 12:29 PM, Nagendra Tomar
>> <Nagendra.Tomar@microsoft.com> wrote:
>>>=20
>>>>> On Mar 23, 2021, at 11:57 AM, Nagendra Tomar
>>>> <Nagendra.Tomar@microsoft.com> wrote:
>>>>>=20
>>>>>>> On Mar 23, 2021, at 1:46 AM, Nagendra Tomar
>>>>>> <Nagendra.Tomar@microsoft.com> wrote:
>>>>>>>=20
>>>>>>> From: Nagendra S Tomar <natomar@microsoft.com>
>>>>>>=20
>>>>>> The flexfiles layout can handle an NFSv4.1 client and NFSv3 data
>>>>>> servers. In fact it was designed for exactly this kind of mix of
>>>>>> NFS versions.
>>>>>>=20
>>>>>> No client code change will be necessary -- there are a lot more
>>>>>> clients than servers. The MDS can be made to work smartly in
>>>>>> concert with the load balancer, over time; or it can adopt other
>>>>>> clever strategies.
>>>>>>=20
>>>>>> IMHO pNFS is the better long-term strategy here.
>>>>>=20
>>>>> The fundamental difference here is that the clustered NFSv3 server
>>>>> is available over a single virtual IP, so IIUC even if we were to use
>>>>> NFSv41 with flexfiles layout, all it can handover to the client is th=
at single
>>>>> (load-balanced) virtual IP and now when the clients do connect to the
>>>>> NFSv3 DS we still have the same issue. Am I understanding you right?
>>>>> Can you pls elaborate what you mean by "MDS can be made to work
>>>>> smartly in concert with the load balancer"?
>>>>=20
>>>> I had thought there were multiple NFSv3 server targets in play.
>>>>=20
>>>> If the load balancer is making them look like a single IP address,
>>>> then take it out of the equation: expose all the NFSv3 servers to
>>>> the clients and let the MDS direct operations to each data server.
>>>>=20
>>>> AIUI this is the approach (without the use of NFSv3) taken by
>>>> NetApp next generation clusters.
>>>=20
>>> Yeah, if could have clients access all the NFSv3 servers then I agree, =
pNFS
>>> would be a viable option. Unfortunately that's not an option in this ca=
se. The
>>> cluster has 100's of nodes and it's not an on-prem server, but a cloud =
service,
>>> so the simplicity of the single LB VIP is critical.
>>=20
>> The clients mount only the MDS. The MDS provides the DS addresses, they =
are
>> not exposed to client administrators. If the MDS adopts the load balance=
r's IP
>> address, then the clients would simply mount that same server address us=
ing
>> NFSv4.1.
>=20
> I understand/agree with the "client mounts the single MDS IP" part. What =
I meant=20
> by "simplicity of the single LB VIP" is to not having to have so many rou=
table=20
> IP addresses, since the clients could be on a (very) different network th=
an the=20
> storage cluster they are accessing, even though client admins will not de=
al with
> those addresses themselves, as you mention.

Got it.


>> The other alternative is to make the load balancer sniff the FH from eac=
h
>> NFS request and direct it to a consistent NFSv3 DS. I still prefer that
>> over adding a very special-case mount option to the Linux client. Again,
>> you'd be deploying a code change in one place, under your control, inste=
ad
>> of on 100's of clients.
>=20
> That is one option but that makes LB application aware and potentially le=
ss=20
> performant. Appreciate your suggestion, though!

You might get part of the way there by having the LB direct
traffic from a particular client to a particular backend NFS
server. The client and its applications are bound to have a
narrow file working set.


> I was hoping that such a client side change could be useful to possibly m=
ore=20
> users with similar setups, after all file->connection affinity doesn't so=
und too=20
> arcane and one can think of benefits of one node processing one file. No?

That's where I'm getting hung up (outside the personal preference
that we not introduce yes another mount option). While I understand
what's going on now (thanks!) I'm not sure this is a common usage
scenario for NFSv3. Other opinions welcome here!

Nor does it seem like one that we want to encourage over solutions
like pNFS. Generally the Linux community has taken the position
that server bugs should be addressed on the server, and this seems
like a problem that is introduced by your middlebox and server
combination. The client is working properly and is complying with
spec.

If the server cluster prefers particular requests to go to particular
targets, a layout is the way to go, IMHO.

(I'm not speaking for the NFS client maintainers, just offering an
opinion and hoping my comments clarify the scenario for others on
the list paying attention to this thread).

--
Chuck Lever