Received-SPF: pass (google.com: domain of linux-nfs+bounces-3338-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <d11d884b-8933-4a3a-a16b-8437530da1e7@talpey.com>
Date: Wed, 22 May 2024 13:18:55 -0400
User-Agent: Mozilla Thunderbird
Subject: Re: sm notify (nlm) question
To: Trond Myklebust <trondmy@hammerspace.com>,
 "ffilzlnx@mindspring.com" <ffilzlnx@mindspring.com>,
 "aglo@umich.edu" <aglo@umich.edu>
Cc: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
 "chuck.lever@oracle.com" <chuck.lever@oracle.com>
References: <CAN-5tyFBn3C_CTrsftuYeWJHe7KWxd82YFCyrN9t=az8J4RU0w@mail.gmail.com>
 <2C80B5BC-AAEC-41F8-BEB6-C920F88C89BB@oracle.com>
 <0b1101daa646$d26a6300$773f2900$@mindspring.com>
 <CAN-5tyGECFmtzFsYNSZicPcH4SMKF0yovk6V20sWJ1LrZKzzyA@mail.gmail.com>
 <0b1401daa64b$f5831ee0$e0895ca0$@mindspring.com>
 <CAN-5tyHWeQykx0cFpOFf2hTBRk9_NaZarzeeAFdSu2NW0zqobA@mail.gmail.com>
 <33bdba418d7ccd7d9b87fa478ea71a14e979aaf1.camel@hammerspace.com>
Content-Language: en-US
From: Tom Talpey <tom@talpey.com>
In-Reply-To: <33bdba418d7ccd7d9b87fa478ea71a14e979aaf1.camel@hammerspace.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?WEJaemloYjdRV3NVY1UvWTZibUJVcmdpUVhSWThYMGtGbUtIUVRnN2l5TU5q?=
 =?utf-8?B?ZVlMcm9yc0RQcFV4Y3d1eUI0MlN6N3M2dWpiTnJGS21ENElqR1dSM1ZXQThZ?=
 =?utf-8?B?dUcvbjBSQ3dkckxOUjhPSHp2NThJaDd1bjJ2TEZhTnRhay9NTkJtSlp5clFy?=
 =?utf-8?B?a0dYamVCV2QyMkF5K1pPNUg0ZFlKRDZZT0NBbndidkNTVmFveDRpaVRjTkxU?=
 =?utf-8?B?WWZzbUhEeTB4UitMemZCak14YTNCbmdOam9XZkYwSE1oSkNsNFlObWtNNEpP?=
 =?utf-8?B?UEViZ3pTTnRjb3dWMWM3cHhWRW9QdEhuVmE1TnNQMFU1TjFiRVlXQmJNWWY4?=
 =?utf-8?B?NHJQR0FRb0Rld2pPY29RU1ZNTmlrdnY1OEtnOGV1QVN1c2xEK1hxblN0Y3Bx?=
 =?utf-8?B?djhFcHowcDZtM1FRcDlqZ052YzQ1SFRhWURkWTB4RkhXblk4SUcxTWRUcjBi?=
 =?utf-8?B?d2hvN2tqQVJJVThNZThHbUg1VUo2SU1LeDk0U1pDSTBCUWhrZHZNY1U4K2pJ?=
 =?utf-8?B?ZWJtZkZYbW9ucVVqdzVjRkYrWjM5Nk5zMm1ycW9ncGd6Y08yVDlxamU2UzNT?=
 =?utf-8?B?d2RiNXorUE1Yc0FlZHZ3a0dlVXcrYkZNak9aSEVVc1YxWEV4enBaRW9qNnd5?=
 =?utf-8?B?RVdOQ3ZJL2JnVlhKT1BvanZqSTF3SjkyeU5MNHovVWgwR2RYZ0RuS2VHZE1Z?=
 =?utf-8?B?QW1NK1lXVUNnVWJCRXJoWnB5Y1ZjNGFaSWpIVWZFeXdoK1VxY2dhb1ppWTgx?=
 =?utf-8?B?ZUVFVno5WGprS0FRenMrVFBqRjlkVFZlYlAwanJCSTdtRDYxdlcwK1RwZlNR?=
 =?utf-8?B?SE5lVmJaQnRFTHVzYUczeVdieWp1ZG9SMDAxMVZ6eVZ2S0paZ3A5VEdmSERU?=
 =?utf-8?B?eHJhNUEwLy9HRUJXN3FYUzZkbTBDT1Y2bjVlZ3BMLzNuc3BsRkNvQ21BQkhX?=
 =?utf-8?B?V0J5MXNJekc3TW4xTHpPcDBTWTFuMktFTW1mbUFjVGorVE1GMDh3WlhLUStN?=
 =?utf-8?B?aEw4T3ZiSXFLY2xVNCsyWHZCV21jS1hpbUhEKyt4YmRUdU5sWTNTNlNGSktE?=
 =?utf-8?B?NGVBMXBCdFpZQ0N5WVFlUFZkeEg4MzN3eDF3NzZIMUxsMWVYbEhyN3dPNjJq?=
 =?utf-8?B?dmlRWi9DaGpMZkNkeGM0aUpOeXpGNU9EOTdtWWx3NnFxV0pWUE5UdUNzamc1?=
 =?utf-8?B?b2pYWnBvUmhJS1NNZ3NOWGhLa1FQYzcxbzhXV203R0J5NDJIMGNaYjU1SVpT?=
 =?utf-8?B?Y0NpUEd5UWVkWFVCMFBPdWNmRmF2N0R0cytPRUNYa09MYWRwT0NNVlZPWk9s?=
 =?utf-8?B?Q1I4YzdUMkxickRtcXpjZFJoNXowenNINmNRek5JclNxQ2hwZGtBM1F0Q3JP?=
 =?utf-8?B?VXFzUC96WHYrbGU1TkQwT1hXcC9tdUN2cnFhb1JvdkJMdWxIUlE2ekxubEtw?=
 =?utf-8?B?WnlWdU5XTUxCNmFSQ0ozeGUrMHJCRkJOcWZpcmtOY0V2anBFelI5UVVIVVUz?=
 =?utf-8?B?VXcxMmwwbDgrcSt5TUhRZk16WjhpdzdIcHJmU3BxS3JUa1VCMFZ3SUx1anRt?=
 =?utf-8?B?T0VNaEc2TTE2Rk8yQ0RFSmFXV0hrSXlkQU1CMUUyV3ZCVWpHQ3NIZGRTM1Zn?=
 =?utf-8?B?eWxPVlAzREN5b0xvV2VHem95cCs2NFlSOGVaNVNETlYvenNkS0pMNTRxQzBp?=
 =?utf-8?B?ak12WmJrZFlpdmswWTFoNmZaWkNBajBlR2tzaisxcitQNlJHZ3R6aXNnSVdr?=
 =?utf-8?B?K3JLY3kwdzdOV0ZZeU4xZnl3dERwYytCNXhaRVlFS2ExN3NmMjhlY2pHUlN1?=
 =?utf-8?B?SFoxRGR4d1FTbGUyb3ZHek9XNnlVUEdvY2k0WWVRNmwyUlByakFqZ2NlSmdr?=
 =?utf-8?B?N1FHcFhzNjUzV3k0eTNScXhIKzQ3d1F0S1V3YXNhaDNwM1F1TzdrQTNWYzB4?=
 =?utf-8?B?ME5zNjJLcmVid24rdzVHTDMyZGo5aXYxY1M1alk1c2NHdEpKYWhoNHB1LzZu?=
 =?utf-8?B?UWQzTXNYaDlGTHhhMWtJMlRMTTNITHQxY25rUjJkVVpMV0pIaDdpeWRhc0Y4?=
 =?utf-8?B?bERMcGRGV2RWM24rUnZaNzNHT3E2Q1h0QU4vcGFIS1FsMU40MVp6MTIvbGIr?=
 =?utf-8?Q?Bh7O5MTrJM9R7LdC9xVZV+XbO?=
X-OriginatorOrg: talpey.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 50525220-71bc-4f13-ab7a-08dc7a83439c
X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB7170.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 May 2024 17:18:57.4741
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 2b2dcae7-2555-4add-bc80-48756da031d5
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: +HGVsvNLbB44cgW6t3Iv3CjuZJ3o7Iz1UyzYN7kFXWrpCaBZK6bHTwG1f4nKeh+c
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR01MB7909

On 5/22/2024 12:20 PM, Trond Myklebust wrote:
> On Wed, 2024-05-22 at 09:57 -0400, Olga Kornievskaia wrote:
>> On Tue, May 14, 2024 at 6:13 PM Frank Filz <ffilzlnx@mindspring.com>
>> wrote:
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: Olga Kornievskaia [mailto:aglo@umich.edu]
>>>> Sent: Tuesday, May 14, 2024 2:50 PM
>>>> To: Frank Filz <ffilzlnx@mindspring.com>
>>>> Cc: Chuck Lever III <chuck.lever@oracle.com>; Linux NFS Mailing
>>>> List <linux-
>>>> nfs@vger.kernel.org>
>>>> Subject: Re: sm notify (nlm) question
>>>>
>>>> On Tue, May 14, 2024 at 5:36 PM Frank Filz
>>>> <ffilzlnx@mindspring.com> wrote:
>>>>>
>>>>>>> On May 14, 2024, at 2:56 PM, Olga Kornievskaia
>>>>>>> <aglo@umich.edu>
>>>> wrote:
>>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> Given that not everything for NFSv3 has a specification, I
>>>>>>> post a
>>>>>>> question here (as it concerns linux v3 (client)
>>>>>>> implementation)
>>>>>>> but I ask a generic question with respect to NOTIFY sent by
>>>>>>> an NFS server.
>>>>>>
>>>>>> There is a standard:
>>>>>>
>>>>>> https://pubs.opengroup.org/onlinepubs/9629799/chap11.htm
>>>>>>
>>>>>>
>>>>>>> A NOTIFY message that is sent by an NFS server upon reboot
>>>>>>> has a
>>>>>>> monitor name and a state. This "state" is an integer and is
>>>>>>> modified on each server reboot. My question is: what about
>>>>>>> state
>>>>>>> value uniqueness? Is there somewhere some notion that this
>>>>>>> value
>>>>>>> has to be unique (as in say a random value).
>>>>>>>
>>>>>>> Here's a problem. Say a client has 2 mounts to ip1 and ip2
>>>>>>> (both
>>>>>>> representing the same DNS name) and acquires a lock per
>>>>>>> mount. Now
>>>>>>> say each of those servers reboot. Once up they each send a
>>>>>>> NOTIFY
>>>>>>> call and each use a timestamp as basis for their "state"
>>>>>>> value --
>>>>>>> which very likely is to produce the same value for 2
>>>>>>> servers
>>>>>>> rebooted at the same time (or for the linux server that
>>>>>>> looks like
>>>>>>> a counter). On the client side, once the client processes
>>>>>>> the 1st
>>>>>>> NOTIFY call, it updates the "state" for the monitor name
>>>>>>> (ie a
>>>>>>> client monitors based on a DNS name which is the same for
>>>>>>> ip1 and
>>>>>>> ip2) and then in the current code, because the 2nd NOTIFY
>>>>>>> has the
>>>>>>> same "state" value this NOTIFY call would be ignored. The
>>>>>>> linux
>>>>>>> client would never reclaim the 2nd lock (but the
>>>>>>> application
>>>>>>> obviously would never know it's missing a lock)
>>>>>>> --- data corruption.
>>>>>>>
>>>>>>> Who is to blame: is the server not allowed to send "non-
>>>>>>> unique"
>>>>>>> state value? Or is the client at fault here for some
>>>>>>> reason?
>>>>>>
>>>>>> The state value is supposed to be specific to the monitored
>>>>>> host. If
>>>>>> the client is indeed ignoring the second reboot notification,
>>>>>> that's incorrect
>>>> behavior, IMO.
>>>>>
>>>>> If you are using multiple server IP addresses with the same DNS
>>>>> name, you
>>>> may want to set:
>>>>>
>>>>> sysctl fs.nfs.nsm_use_hostnames=0
>>>>>
>>>>> The NLM will register with statd using the IP address as name
>>>>> instead of host
>>>> name. Then your two IP addresses will each have a separate
>>>> monitor entry and
>>>> state value monitored.
>>>>
>>>> In my setup I already have this set to 0. But I'll look around
>>>> the code to see what
>>>> it is supposed to do.
>>>
>>> Hmm, maybe it doesn't work on the client side. I don't often test
>>> NLM clients with my Ganesha work because I only run one VM and NLM
>>> clients can’t function on the same host as any server other than
>>> knfsd...
>>
>> I've been staring and tracing the code and here's what I conclude:
>> the
>> use of nsm_use_hostname toggles nothing that helps. No matter what
>> statd always stores whatever it is monitoring based on the DSN name
>> (looks like git blame says it's due to nfs-utils's commit
>> 0da56f7d359475837008ea4b8d3764fe982ef512 "statd - use dnsname to
>> ensure correct matching of NOTIFY requests". Now what's worse is that
>> when statd receives a 2nd monitoring request from lockd for something
>> that maps to the same DNS name, statd overwrites the previous
>> monitoring information it had. When a NOTIFY arrives from an IP
>> matching the DNS name, the statd does the downcall and it will send
>> whatever the last monitoring information lockd gave it. Therefore all
>> the other locks will never be recovered.
>>
>> What I struggle with is how to solve this problem. Say ip1 and ip2
>> run
>> an NFS server and both are known under the same DNS name:
>> foo.bar.com.
>> Does it mean that they represent the "same" server? Can we assume
>> that
>> if one of them "rebooted" then the other rebooted as well?  It seems
>> like we can't go backwards and go back to monitoring by IP. In that
>> case I can see that we'll get in trouble if the rebooted server
>> indeed
>> comes back up with a different IP (same DNS name) and then it would
>> never match the old entry and the lock would never be recovered (but
>> then also I think lockd will only send the lock to the IP is stored
>> previously which in this case would be unreachable). If statd
>> continues to monitor by DNS name and then matches either ips to the
>> stored entry, then the problem comes with "state" update. Once statd
>> processes one NOTIFY which matched the DNS name its state "should" be
>> updated but then it would leads us back into the problem if ignoring
>> the 2nd NOTIFY call. If statd were to be changed to store multiple
>> monitor handles lockd asked to monitor, then when the 1st NOTIFY call
>> comes we can ask lockd to recover "all" the store handles. But then
>> it
>> circles back to my question: can we assume that if one IP rebooted
>> does it imply all IPs rebooted?
>>
>> Perhaps it's lockd that needs to change in how it keeps track of
>> servers that hold locks. The behaviour seems to have changed in 2010
>> (with commit 8ea6ecc8b0759756a766c05dc7c98c51ec90de37 "lockd: Create
>> client-side nlm_host cache") when nlm_host cache was introduced
>> written to be based on hash of IP. It seems that before things were
>> based on a DNS name making it in line with statd.
>>
>> Anybody has any thoughts as to whether statd or lockd needs to
>> change?
>>
> 
> I believe Tom Talpey is to blame for the nsm_use_hostname stuff. That
> all came from his 2006 Connectathon talk
> https://nfsv4bat.org/Documents/ConnectAThon/2006/talpey-cthon06-nsm.pdf

I deny that!! :) All that talk intended to do was to point out how
deeply flawed the statmon protocol is, and how badly it was then
implemented. However, hostnames may be a slight improvement over
the mess that was 2006. And it's been kinda sorta working since then.

Personally I still think trying to "fix" nsm is a fool's errand.
It's just never ever going to succeed. Particularly if both the
clients *and* servers have to change. NFS4.1 is the better way.

Tom.