Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Neil Brown <neilb@suse.de>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [RFC PATCH 0/4] nfsd: allow NFSv4 state to be revoked.
Thread-Topic: [RFC PATCH 0/4] nfsd: allow NFSv4 state to be revoked.
Thread-Index: AQHYEzqzINafEUb13kObHCDc0cTYeKx3GDuAgABfVQCAABgdAIAAR7yAgACdGgA=
Date:   Fri, 28 Jan 2022 13:46:45 +0000
Message-ID: <C462D2BB-BD5B-4372-B644-FD4D6D877072@oracle.com>
References: <164325908579.23133.4781039121536248752.stgit@noble.brown>
 <7B388FE8-1109-4EDD-B716-661870B446C7@oracle.com>
 <164332328424.5493.16812905543405189867@noble.neil.brown.name>
 <A933D67A-0C1B-4700-97E7-0DBEF4458A77@oracle.com>
 <164334386787.5493.637178363398104896@noble.neil.brown.name>
In-Reply-To: <164334386787.5493.637178363398104896@noble.neil.brown.name>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?VfGJwhXdIOZDSfU6N6WNcE1SNZ7ZZVDwYt82ftCDCQ+Sorv/K6zTFrmhfpvo?=
 =?us-ascii?Q?eq1S2uRxi5poXSkGgT5bAfPfze1GLwZn6uRcrJ7/cXEDwwryH7KPyr30B47w?=
 =?us-ascii?Q?LBsj44+fXnD7u7OLuBqN3VzoG4djTNPijc3xef30yKRBF26xz3Ln3QMTajYD?=
 =?us-ascii?Q?k513/5FKrGD+HJMMMUsq7pQCLm/0vV42VaCCeiry6Av5KThEnxVx9wm2fqRm?=
 =?us-ascii?Q?QFYBh01UX8qBB2QsLNJnchJBXXteWYCm9jDXH48hLpvRuaCV6lOfNYyzEegT?=
 =?us-ascii?Q?voOmQv48/HLOjJXTXaDMt+pEVXazWnpFivrtvGS5GSvIecQLnDEwi9RTia54?=
 =?us-ascii?Q?eCCm+tGwlQk/91OWBgJTd3fE6GcuByGWPpP7hp9uGRHJsT76kIigbMPT3rja?=
 =?us-ascii?Q?C/5IXoKfeJrBZFYmyXVEc41sIVubjXRccW32HThhcLmKITYjVki+1PzY3yeO?=
 =?us-ascii?Q?FxlFeIT96MrzhAlKINbQt1NnfCDYlwPfLS3aOmLxzI/G0Nn9r75sbyg5J3iV?=
 =?us-ascii?Q?s7IJDlZRzP2ks9DHPRJKVJ1w41ouqk6/+VOgpWUiLod/KkydkjRy5hVDtEe0?=
 =?us-ascii?Q?eXtpPQbhjMMGip9+y/ozOCCM0OJQfiYqhL+kfjb1fEWBU65IR9ob3xlWlbu8?=
 =?us-ascii?Q?mWEI34D+y6N4jtjK6zm6UOmz7v+/CPRgc6xMFWXuKkfiefn8h0tgjxfChYDA?=
 =?us-ascii?Q?XEY4XT0pqyvypGwBZDg+S4djZtsZrAp1G6zhB9KpiUPLkgSzXu0ASJJwlBSl?=
 =?us-ascii?Q?tyMSOvf4hsEKEGdxuBI7QGGNC894VVsMBauNdc2iN05PyPOrDCAHI9ydl+/S?=
 =?us-ascii?Q?VKiulUxLFT5s2ePk2tCOGr53rUdWsFMJcFPEnDbjARjA6aVROsV1fBE6AiVV?=
 =?us-ascii?Q?CRVB3HzsD1sBXsFDfIlYgwLEf+tEQxIdukIRUCtRcCuYWYFm2sBK9NKHMh7M?=
 =?us-ascii?Q?dH5uIY8CnUgBxD0XoTWkUSxkt1ZuPZj3mpxlxA8KwvorVm5PLNYmSBlioXjF?=
 =?us-ascii?Q?PWdfdJy7eiWXBrTeqGmNfNYC+fyAkbnxuWiBCWDfg4B8iRcoryniZb4A/TAn?=
 =?us-ascii?Q?2FPVdgooD1ic2GB1gIpHv2aBpE3V+Mmw0HYSKTfaQJRwi3kDJ374Oc6tbpFs?=
 =?us-ascii?Q?bloGPqnFU4SFJmNV/7xrVrp5P53uSPb0n3zR0wQjmXzB+corV3sdy0x4W/P4?=
 =?us-ascii?Q?RTzVXbE2B0VdW7RjvtzMzJMG6XgDefLoYBHkIvN9s4QXa3WYidswIHNMVmiA?=
 =?us-ascii?Q?Sca5dVJp1LcERaSFeQd5RWIvHVlQxUFLl+nfhtqYlFLKJw2WdDwa681VKEWY?=
 =?us-ascii?Q?+vueMOsPl4Dmd8RWB2dtK164oM14Nw9qDpbzgX8Qh/7b8j4oxvPdQz06mBGG?=
 =?us-ascii?Q?GYO4BbwaWnPwJhx6VCLoWhZFYfyC1xEj5zaVp1CDWs8F8Sfe6c98GRoaVLUj?=
 =?us-ascii?Q?G6AO6aNgwVD23CY+BvWnf9KLllX6wnpRWtGoWgHiJpnt93G0GPDj/tyVw7Vu?=
 =?us-ascii?Q?cev76bnpAGbSlXJkCxgwsiFtuzSZOZdwI2xzqOu3IF9mVJ3o6Qe02IYHddqa?=
 =?us-ascii?Q?Hm8cCcNvdoo/oG5Q1yTQXsZNhuC5GKZROEYPkeYyGpw8qmGOe0s+jruOqgTN?=
 =?us-ascii?Q?TxfT3zGw2dwkQZpVzU5CeyU=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <BBF92D005D0FCB479520CBB373DB4DF0@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CH0PR10MB4858.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ffbaeb1e-438f-4d84-d84b-08d9e2649fd2
X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Jan 2022 13:46:45.4232
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 94ZIp4aoTsPrm/ZichYiQhMbyV7LxWcJHEqkyMrXsVNm1o0dkX/YU7Q0G7Ok0H2AedzeKBODAAffJYrR9cI4HA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB3444
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10240 signatures=669575
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0
 mlxlogscore=999 mlxscore=0 adultscore=0 bulkscore=0 malwarescore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2201110000 definitions=main-2201280087
X-Proofpoint-GUID: tyXu4ZOwW1PcZ29d-AOQ33no0rV-8Tds
X-Proofpoint-ORIG-GUID: tyXu4ZOwW1PcZ29d-AOQ33no0rV-8Tds
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jan 27, 2022, at 11:24 PM, NeilBrown <neilb@suse.de> wrote:
>=20
> On Fri, 28 Jan 2022, Chuck Lever III wrote:
>>=20
>>> On Jan 27, 2022, at 5:41 PM, NeilBrown <neilb@suse.de> wrote:
>>>=20
>>> On Fri, 28 Jan 2022, Chuck Lever III wrote:
>>>> Hi Neil-
>>>>=20
>>>>> On Jan 26, 2022, at 11:58 PM, NeilBrown <neilb@suse.de> wrote:
>>>>>=20
>>>>> If a filesystem is exported to a client with NFSv4 and that client ho=
lds
>>>>> a file open, the filesystem cannot be unmounted without either stoppi=
ng the
>>>>> NFS server completely, or blocking all access from that client
>>>>> (unexporting all filesystems) and waiting for the lease timeout.
>>>>>=20
>>>>> For NFSv3 - and particularly NLM - it is possible to revoke all state=
 by
>>>>> writing the path to the filesystem into /proc/fs/nfsd/unlock_filesyst=
em.
>>>>>=20
>>>>> This series extends this functionality to NFSv4.  With this, to unmou=
nt
>>>>> an exported filesystem is it sufficient to disable export of that
>>>>> filesystem, and then write the path to unlock_filesystem.
>>>>>=20
>>>>> I've cursed mainly on NFSv4.1 and later for this.  I haven't tested
>>>>> yet with NFSv4.0 which has different mechanisms for state management.
>>>>>=20
>>>>> If this series is seen as a general acceptable approach, I'll look in=
to
>>>>> the NFSv4.0 aspects properly and make sure it works there.
>>>>=20
>>>> I've browsed this series and need to think about:
>>>> - whether we want to enable administrative state revocation and
>>>> - whether NFSv4.0 can support that reasonably
>>>>=20
>>>> In particular, are there security consequences for revoking
>>>> state? What would applications see, and would that depend on
>>>> which minor version is in use? Are there data corruption risks
>>>> if this facility were to be misused?
>>>=20
>>> The expectation is that this would only be used after unexporting the
>>> filesystem.  In that case, the client wouldn't notice any difference
>>> from the act of writing to unlock_filesystem, as the entire filesystem
>>> would already be inaccessible.
>>>=20
>>> If we did unlock_filesystem a filesystem that was still exported, the
>>> client would see similar behaviour to a network partition that was of
>>> longer duration than the lease time.   Locks would be lost.
>>>=20
>>>>=20
>>>> Also, Dai's courteous server work is something that potentially
>>>> conflicts with some of this, and I'd like to see that go in
>>>> first.
>>>=20
>>> I'm perfectly happy to wait for the courteous server work to land befor=
e
>>> pursuing this.
>>>=20
>>>>=20
>>>> Do you have specific user requests for this feature, and if so,
>>>> what are the particular usage scenarios?
>>>=20
>>> It's complicated....
>>>=20
>>> The customer has an HA config with multiple filesystem resource which
>>> they want to be able to migrate independently.  I don't think we really
>>> support that,
>>=20
>> With NFSv4, the protocol has mechanisms to indicate to clients that
>> a shared filesystem has migrated, and to indicate that the clients'
>> state has been migrated too. Clients can reclaim their state if the
>> servers did not migrate that state with the data. It deals with the
>> edge cases to prevent clients from stealing open/lock state during
>> the migration.
>>=20
>> Unexporting doesn't seem like the right approach to that.
>=20
> No, but it something that should work, and should allow the filesystem
> to be unmounted.  You get to keep both halves.
>=20
>>=20
>>=20
>>> but they seem to want to see if they can make it work (and
>>> it should be noted that I talk to an L2 support technician who talks to
>>> the customer representative, so I might be getting the full story).
>>>=20
>>> Customer reported that even after unexporting a filesystem, they cannot
>>> then unmount it.
>>=20
>> My first thought is that probably clients are still pinning
>> resources on that shared filesystem. I guess that's what the
>> unlock_ interface is supposed to deal with. But that suggests
>> to me that unexporting first is not as risk-free as you
>> describe above. I think applications would notice and there
>> would be edge cases where other clients might be able to
>> grab open/lock state before the original holders could
>> re-establish their lease.
>=20
> Unexporting isn't risk free.  It just absorbs all the risks - none are
> left of unlock_filesystem to be blamed for.
>=20
> Expecting an application to recover if you unexport a filesystem and
> later re-export it is certainly not guaranteed.  That isn't the use-case
> I particularly want to fix.  I want to be able to unmount a filesystem
> without visiting call clients and killing off applications.

OK. The top level goal then is simply to provide another
arrow in the administrator's quiver to manage a large
NFS server. It brings NFSv4 closer to par with the NFSv3
toolset.

I say we have enough motivation for a full proof of
concept. I would like to see support for minor version 0
added, and a fuller discussion of the consequences for
clients and applications will be needed (at least for
the purpose of administrator documentation).


>>> Whether or not we think that independent filesystem
>>> resources is supportable, I do think that the customer should have a
>>> clear path for unmounting a filesystem without interfering with service
>>> provided from other filesystems.
>>=20
>> Maybe. I guess I put that in the "last resort" category
>> rather than "this is something safe that I want to do as
>> part of daily operation" category.
>=20
> Agree.  Definitely "last resort".
>=20
>>=20
>>=20
>>> Stopping nfsd would interfere with
>>> that service by forcing a grace-period on all filesystems.
>>=20
>> Yep. We have discussed implementing a per-filesystem
>> grace period in the past. That is probably a pre-requisite
>> to enabling filesystem migration.
>>=20
>>=20
>>> The RFC explicitly supports admin-revocation of state, and that would
>>> address this specific need, so it seemed completely appropriate to
>>> provide it.
>>=20
>> Well the RFC also provides for migrating filesystems without
>> stopping the NFS service. If that's truly the goal, then I
>> think we want to encourage that direction instead of ripping
>> out open and lock state.
>=20
> I suspect that virtual IPs and network namespaces is the better approach
> for migrating exported filesystems.  It isn't clear to me that
> integrated migration support in NFS would add anything of value.
>=20
> But as I think I said to Bruce - seamless migration support is not my
> goal here.  In the context where a site has multiple filesystems that
> are all NFS exported, there is a case for being able to forcibly
> unexport/unmount one filesystem without affecting the others.  That is
> my aim here.

My initial impulse is to better understand what is preventing
the unexported filesystem from being unmounted. Better
observability there could potentially be of value.


> Thanks,
> NeilBrown
>=20
>=20
>>=20
>> Also, it's not clear to me that clients support administrative
>> revocation as broadly as we might like. The Linux NFS client
>> does have support for NFSv4 migration, though it's a bit
>> fallow these days.
>>=20
>>=20
>>> As an aside ...  I'd like to be able to suggest that the customer use
>>> network namespaces for the different filesystem resources.  Each could
>>> be in its own namespace and managed independently.  However I don't
>>> think we have good admin infrastructure for that do we?
>>=20
>> None that I'm aware of. SteveD is the best person to ask.
>>=20
>>=20
>>> I'd like to be able to say "set up these 2 or 3 config files and run=20
>>> systemctl start nfs-server@foo and the 'foo' network namespace will be
>>> created, configured, and have an nfs server running".
>>> Do we have anything approaching that?  Even a HOWTO ??
>>=20
>> Interesting idea! But doesn't ring a bell.
>>=20
>> --
>> Chuck Lever

--
Chuck Lever