Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <4be6f388-9fa0-a54a-8c78-8f1761284e35@talpey.com>
Date:   Mon, 3 Oct 2022 10:44:03 -0400
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.3.1
Subject: Re: nfs4.1+: workaround for defunct clientaddr?
Content-Language: en-US
To:     Manfred Schwarb <manfred99@gmx.ch>,
        Jeff Layton <jlayton@kernel.org>, linux-nfs@vger.kernel.org
References: <39bf58c7-47d9-744b-6d26-d672aa713024@gmx.ch>
 <8cd63730f7b5f3e2aa3bde98587de0c6a42b384b.camel@kernel.org>
 <8550c032-2ef8-4ddb-19a0-307777bd9645@gmx.ch>
 <c163b299164cd9abe21325a947c9efe908a04331.camel@kernel.org>
 <e3842fe7-da95-98e2-b0e8-fa3aede673cc@gmx.ch>
From:   Tom Talpey <tom@talpey.com>
In-Reply-To: <e3842fe7-da95-98e2-b0e8-fa3aede673cc@gmx.ch>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dGh3MExQQ3RKNDRGSkx5dHE1NDZzN1QweDZGV3ZOczJ1QzF6cXlXMkR6WWlm?=
 =?utf-8?B?TmZzTzJWZ29NRDBBMVB2bDBYYjhUQmtIN21wR3ZiaFowaTZxbEpkdlF0TkNC?=
 =?utf-8?B?M0NZTWc0bVdrRkVrRXBvN2JNaWk3TDNMYVlOdk92UWR0VFh2MFFrSEsvSTZI?=
 =?utf-8?B?alNYQTJMVWhnNURLeG84RGdOUDRXdC9yVml5VnJYZmFaVDJ0Mm84ZlRaRjR0?=
 =?utf-8?B?UllEUkFMQSs0NHUvQjE1RXJSSjZYSm54S2lXZzNYY2RVS2tudW9BUjlOUjBP?=
 =?utf-8?B?SW4reko2WU5kV3BwMWZTc2xWLzl3MDhYMGdaaUlRTkhSVURsb0xvS1JZd3N4?=
 =?utf-8?B?NTlTSisvYXFwR0puUCtVbit3bUdQWHNJbTRXOXBZRVdJc2tsaDRMVG5uSWov?=
 =?utf-8?B?eW1USVRmZmlaclVybEYvNzlLSzVqeDNvQnEyWTN2RC9rRURhZHB3eVQ0M29U?=
 =?utf-8?B?Rk14QVRxSVR0MWhGczFLcGpvVDliTXhTU3k4YVU5NHRMUVRuLyt4VngvcXl6?=
 =?utf-8?B?L2NZNDlvTVpqdHdTQTZwUUxBWmN6WTA4VUtQemtzRnVrNXlkNEt6Tk94dWND?=
 =?utf-8?B?Z0YrZjRIWXhUK2c5RWlXNUQraXAvVHZ3anVKa2dCNnFWZEYwTG9NOXRabXhK?=
 =?utf-8?B?aGlLdVg1R3c2dnlETGt3K3pVQTZDM3RJMlFWTElFMmc5anl0aDc2UER3OWd5?=
 =?utf-8?B?cCt3OG9uUHJrdFJpRXpVeFVyWGJQQ1ZBYUxwcVQ2Mlcyb2IwLzdMRE90T2kv?=
 =?utf-8?B?UXV5KytRZFVsMWVGNWJHaHN2Zk0xeThqK2E2RVM3WGZxYi9DL2ZoQ1d1SVdi?=
 =?utf-8?B?SjRpVCsxeGpvVnJacTBGbk1iQ0lGYUZuVFhteWVVMGxpYTBScnRzSWdyVmNU?=
 =?utf-8?B?RTJncTFSRGRnM0VGMStKK2JDek9Bd2VSM3dtOU5SL1E2NWtZZUpBNmhyc1Nn?=
 =?utf-8?B?NzlxOVFQK3Yxem9yYngzeG9JcFhzTTE0Wk1hb1ZCUUVOM3FUcGVkek56Y1NR?=
 =?utf-8?B?MzFzQXpPenZCSUhIcnZjTDVxb3VTYS9JNGxMVDVQMVdoRlBvamlySnViVDJM?=
 =?utf-8?B?NE1qWkgreW5jVkhBSTJrSVhOekQyRGZTYTcvUDN5eTR5aWlJaWY4RzZSeXp3?=
 =?utf-8?B?TEQ2bHpVVnJHNTJwQVNqcXkxVTV5ME9zdTZ1alJBRHhJNCsvaVhydFJLcXc2?=
 =?utf-8?B?S3ZqSzFna3dQMlhqV0dxQWhMOEp1L1RYOWdLVWN1eGFaaE15cm51QncyK0pk?=
 =?utf-8?B?TzB2b0tWbjhHOXczZ1duMzFzRlhwZ201RzFMbTltcW4ySHVVQ0tIWmZGUWdn?=
 =?utf-8?B?dTlGS1BwM2VaRHAveWM1cE42U0VQNDN2U0I4bnRjeHFYWG1NRkFSbFdrTTlu?=
 =?utf-8?B?d0lYTlR5YjA1OVVlYkhhSTFKeGozNTVGRytvL3FpZlloUGNsU2ZqYkVURU1K?=
 =?utf-8?B?dEZBVVVEbjBOcDFzQ2VDREhaMFlJeDdzeEVnU2pQT1JjZDJsY3NoMi9ObEdI?=
 =?utf-8?B?alRGejdrODk0eEo2RGRKOFhFaFgwTUdnTjgyb3VxUC9lelp5aDltc2g2bGkz?=
 =?utf-8?B?MEhwdUxFZDFSNmp3QmRIK1lTa3lRQW4wMlhhZitkb3Zra1o3dm5nRWlwMGE4?=
 =?utf-8?B?UDMvUy8vbjYraERDWDJBcTVydGkrVUUwcHc1RWUwZkF3ajVWekUvN3ZjYUxZ?=
 =?utf-8?B?b3djTTUveGJtM2QzT09WSlhmWlJ6ajBpVlhVdVJDMXRmaDdka0FXdlpoRU1C?=
 =?utf-8?B?Nm9XcW5KUnh5RmVOVEVTN29DRlo1TkdyTGtNUUdtc3hjYUVpNDlSRVFXUkU4?=
 =?utf-8?B?U0pqUmJ0Qy8xS3gvRUFVOFpFWktDWnpIWndkcG1mcTFoa1ZJUXRST0tvMmM1?=
 =?utf-8?B?eGRRZkNWMEI2aGRNWmhuSGdNUE1DRlRqVEtLa0hwa0s2am5KYkE2VnNWL3hq?=
 =?utf-8?B?VWRIbXQxL0lJSkNGaFp4YnBMQlI1YjgyZ01lMERDa0QzM0RnZVpSclVZaVBU?=
 =?utf-8?B?eU81VnZqaVpqNklQeVNQSk5HVjRtanhJSDhkbHJXMEx1eVptNEtuNkhySHd6?=
 =?utf-8?B?S2cxejBwS2NSZ21abmtQYWc4YThOTGpOdnBiU29jc0QwL0p1bGxqY3EwRU1W?=
 =?utf-8?Q?TAc93vgOA/IUS1Ohs+f61cfPu?=
X-OriginatorOrg: talpey.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b8472dcb-d3f6-4e69-a5cd-08daa54db87a
X-MS-Exchange-CrossTenant-AuthSource: SN6PR01MB4445.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Oct 2022 14:44:05.3706
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 2b2dcae7-2555-4add-bc80-48756da031d5
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: cLGgzkxSO+rZcqLGWsT/p7wa1pc55+j7CGgBs7sw4NeLvx3T580+IUvt4RROcDWq
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR01MB4980
X-Spam-Status: No, score=-3.4 required=5.0 tests=BAYES_00,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

On 10/3/2022 9:24 AM, Manfred Schwarb wrote:
> Am 03.10.22 um 14:26 schrieb Jeff Layton:
>> On Mon, 2022-10-03 at 13:55 +0200, Manfred Schwarb wrote:
>>> Am 03.10.22 um 13:39 schrieb Jeff Layton:
>>>> On Sun, 2022-10-02 at 14:35 +0200, Manfred Schwarb wrote:
>>>>> Hi,
>>>>>
>>>>> I have 2 boxes connected with 2 network cards each, one
>>>>> crossover connection and one connection via LAN.
>>>>> I want to use the crossover connection for backup,
>>>>> so I want to be able to select exactly this wire when
>>>>> doing my NFS backup transfers. Everything interconnected via NFS4.1
>>>>> and automount.
>>>>>
>>>>> Now the thing is, if there is an already existing connection
>>>>> via LAN, I am not able to select the crossover connection,
>>>>> there is some session reuse against my will.
>>>>>
>>>>> automount config:
>>>>> /net/192.168.99.1  -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.99.100   /  192.168.99.1:/
>>>>> /net2/192.168.98.1 -fstype=nfs4,nfsvers=4,minorversion=1,clientaddr=192.168.98.100   /  192.168.98.1:/
>>>>>
>>>>> mount -l:
>>>>> 192.168.99.1:/data on /net/192.168.99.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
>>>>> 192.168.99.1:/data on /net2/192.168.98.1/data type nfs4 (...,clientaddr=192.168.99.100,addr=192.168.99.1)
>>>>>
>>>>> As you see, both connections are on "192.168.99.1:/data", and the backup runs
>>>>> over the same wire as all user communication, which is not desired.
>>>>> This even happens if I explicitly set some clientaddr= option.
>>>>>
>>>>> Now I found two workarounds:
>>>>> - downgrade to NFS 4.0, clientaddr seems to work with it
>>>>> - choose different NFS versions, i.e. one connection with
>>>>>    minorversion=1 and the other with minorversion=2
>>>>>
>>>>> Both possibilities seem a bit lame to me.
>>>>> Are there some other (recommended) variants which do what I want?
>>>>>
>>>>> It seems different minor versions result in different "nfs4_unique_id" values,
>>>>> and therefore no session sharing occurs. But why do different network
>>>>> interfaces (via explicitly set clientaddr= by user) not result in different
>>>>> "nfs4_unique_id" values?
>>>>>
>>>>> Thanks for any comments and advice,
>>>>> Manfred
>>>>
>>>> That sounds like a bug. We probably need to compare the clientaddr
>>>> values in nfs_compare_super or nfs_compare_mount_options so that it
>>>> doesn't match if the clientaddrs are different.
>>>>
>>
>>
>> Actually, I take it back, clientaddr is specifically advertised as being
>> for NFSv4.0 only. The workaround for you is "nosharecache", which will
>> force the mount under /net2 to get a new superblock altogether.
> 
> But clientaddr is silently accepted on NFS4.1+, and seemingly silently does nothing.
> 
> The point is, RFC5661 explicitely tells
> "NFS minor version 1 is deemed superior to NFS minor version 0 with no loss of functionality".

NFSv4.1 doesn't have a clientaddr in the protocol, only 4.0.

I believe the reason that the clientaddr option is accepted by mount is
to handle multi-protocol negotiate. If the admin requests "any NFSv4"
and provides an explicit clientaddr, it is only needed when falling
all the way down to 4.0, but ignoring it would be incorrect. So perhaps
the mount command could use a tweak.

> So this behavior comes as a surprise.

Well, it wouldn't have been an issue if the missing comparison bug that
Jeff mentions was fixed... I think that's the real issue.

Tom.


>>>> As a workaround, you can probably mount the second mount with
>>>> -o nosharecache and get what you want.
>>>
>>> Indeed, nosharecache works. But the man page has some scary words for it:
>>>    "This is considered a data risk".
>>>
>>
>> Yeah, it does sound scary but it's not a huge issue unless you're doing
>> I/O to the same files at the same time via both mounts. With
>> "sharecache" (the default) you get better cache coherency in that
>> situation since the inode and its pagecache are the same.
>>
> 
> So I guess this is equivalent to the minorversion=1/minorversion=2 trick
> cache coherency wise then?
> 
> 
>> With "nosharecache" you need to be more careful to flush caches, etc. if
>> you are doing reads and writes to the same files via different paths. If
>> you need careful coordination there, then you probably want to use file
>> locking.
> 
> Thanks for these explanations, it is appreciated!
> Manfred
> 
>> --
>> Jeff Layton <jlayton@kernel.org>
> 
>