Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: Locking issue between NFSv4 and SMB client
From:   dai.ngo@oracle.com
To:     Bruce Fields <bfields@fieldses.org>
Cc:     "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
        CIFS <linux-cifs@vger.kernel.org>
References: <5b7be2c0-95a6-048c-581f-17e5e3750daa@oracle.com>
 <20210923215056.GH18334@fieldses.org>
 <90a8f89b-e8ac-2187-2926-d723ebbcb839@oracle.com>
 <4c98a686-3be9-6f95-ea1a-8f03fbf3ea0c@oracle.com>
Message-ID: <adb4f18d-1b2a-1cf6-3209-f34cdc95d4f0@oracle.com>
Date:   Thu, 7 Oct 2021 10:38:45 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0)
 Gecko/20100101 Thunderbird/78.14.0
In-Reply-To: <4c98a686-3be9-6f95-ea1a-8f03fbf3ea0c@oracle.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dG1yVit2T2tIblQ2UnFSNTE0VlIxRG82MzlHSnVFODZvTE4vVjU1aTB6cm96?=
 =?utf-8?B?Y0E3cGUzOHlEeTc1RW85Zlh1emRMOGxuYk1sWnJJQkFIMDdPMFk0U1Ayb2V0?=
 =?utf-8?B?TTkybzlpeFEzWWF6S3VHSksrY1VXejY1d3JrRjY3WDMveCtadzZXZFZ5UVd2?=
 =?utf-8?B?S2g1S1pGcHlGczRvaFBTSzJLcFpNdzhUSENReG9OQTA4S3E3V2Z4UGFVTHRF?=
 =?utf-8?B?b2wybVQrWFl0VFI1aGtSUFg5QjFpeXpQQThNUXRsVlZpNVFxUkpCYS9LVlZj?=
 =?utf-8?B?UHAvNll0TlNPeURVbEtwaW1veEpaamQzYnNMZVdVUUlrdSs0OEdEMGtSejhN?=
 =?utf-8?B?YW5SMFZXMEMyYm9RSzh2VnpKaDd6cU1Ua0ErclpadHRHakJ4N05qclF3K2Fn?=
 =?utf-8?B?K0lnb1ZIVHNlMmZwWTlFOW1JSTVFQXhvN3hHbVVrTk15RS9MNFI0S04xYmZ5?=
 =?utf-8?B?UkhVNUFyRjUzYWhHUE5FUXFNblJZMS9oN0U3YW1mSFpneFhCbGJuVkovdFd1?=
 =?utf-8?B?S2dyYSs4NFBjd09ka0V2QmIwTUtWNzd6aDRzYnl2SHEvdWVsUUV6amRGbDlH?=
 =?utf-8?B?VElBM2lGMFdZMENia0FjZENjMUpkVGxqZEwrQm9lQkJYUlVzWFE3MXE3ZE9p?=
 =?utf-8?B?dkh4eEhmSGRpYm5xdkFoUDJoelZERURQMS93dzh0dGV6ci8vVmZtWkZGdWZa?=
 =?utf-8?B?RFFhQ3VoN01Zbmd6WWhrMnFIYnMzZUVqdjRqQ1NtL1F5NnJnZHJFNlpSdG1K?=
 =?utf-8?B?d2FuR2pzOGltd0U3eWNtYnljNmtkTnVwaVJtMmFGSzhGbEZRQ0l6UE5jM2cx?=
 =?utf-8?B?ZEVVZjdscjRQNzg5amVESGg3YWp5VG5JSDZjY0xFYjViOWRhUHZyUmxpRVl1?=
 =?utf-8?B?bDhiOUFmNFozWHk3UkMrOTd3NS9WSlNoVHJnbHhyZ1lraFQrYlJBSHhFYXpU?=
 =?utf-8?B?ZDczcWNxblI4bU1KNjhFbVEwdi9rRVZoM2J5aFAxWlN1ZFRCWlBjVll4eGVk?=
 =?utf-8?B?bGdQTldQL2dIejJrRWR6Z0FLclI3UlFGRzRIdDZnTEhVMERGdEJjVEl2TlA2?=
 =?utf-8?B?ZU0rUkc5WGExcVpmMW0xKzJ6cng0UnNkbUFpbS9OWWRLYTV6QnV5VzhZd0Uz?=
 =?utf-8?B?TnUrYnNIUWdsV084SDRjWXQvWFY3c1QzVDlqa2VwenJ2ZGNsc3psdE1YQStY?=
 =?utf-8?B?TjZNUDlreFludDRiM2lnWWFaQ1V5ZDBJRDUwWFNVVFZ0M0thc3UvNHVOTGk0?=
 =?utf-8?B?QS9uRTdxVkxhSlEybkZKR2Q2QnJnTlQrTUtOSE9CQkxGcHN0QUNvc0FQbmsr?=
 =?utf-8?B?a3dZVEcxZWtiUlBNNzlQNlNQSk1sTEhleU1xYnpQdnNHZDZIMktQKzZlTnhq?=
 =?utf-8?B?SExIWDJobkN3V1VpQlovdWNuTkdIcXQ3cUNRMjNPYnMrR0J6K1pOcmpFUVl3?=
 =?utf-8?B?QTN2ZWU2dm9TdVlsa09LTTdUSE43azdGVXp2V0pYMUJ5cVJhYUNiQ1VkNHpu?=
 =?utf-8?B?UGwwVTQ5d0JFVlhUWmFHRXVtVVFNYzlKczIxQXU3M1F3eWNQWHp5TlhxaHBD?=
 =?utf-8?B?aGJINlNjR2UyYkRQcFpyc1N6cnFQR0VFNWk3M2lZcnhRVzJ4cFlMdlVYVjc1?=
 =?utf-8?B?ZHhzZzJNQmptWWh6Njk0NllUSkF1bEFOYkN2M1hoTjFUc0E2cnkxaHA5alFQ?=
 =?utf-8?B?UldGVzVabTRERklaR2lJMU5Yc2l6VE8rcEUxcEVqZ0FrYUlVQVlnYjFkMkFy?=
 =?utf-8?Q?+WuPSq5KHnipn49lyOf6/3OVItb06bRI4kaW4iL?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 54f18caf-b388-4113-2f7d-08d989b95194
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Oct 2021 17:38:48.1610
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: tdIz5r0zWHWa8V+8Vlk3YpP7I1qE2fOAEOr415vqkmLdbN6uxmMnicUMmSWUScn3WROF2HhE32kj5QgkTZv3Ew==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB4387
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10130 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 phishscore=0
 spamscore=0 mlxscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109230001
 definitions=main-2110070113
X-Proofpoint-ORIG-GUID: RqjucoQmYzRKRgeTu5EAvVm2Fn_oQbGG
X-Proofpoint-GUID: RqjucoQmYzRKRgeTu5EAvVm2Fn_oQbGG
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


On 10/7/21 10:03 AM, dai.ngo@oracle.com wrote:
>
> On 9/23/21 3:39 PM, dai.ngo@oracle.com wrote:
>>
>> On 9/23/21 2:50 PM, Bruce Fields wrote:
>>> On Thu, Jul 15, 2021 at 04:45:22PM -0700, dai.ngo@oracle.com wrote:
>>>> Hi Bruce,
>>> Oops, sorry for neglecting this.
>>>
>>>> I'm doing some locking testing between NFSv4 and SMB client and
>>>> think there are some issues on the server that allows both clients
>>>> to lock the same file at the same time.
>>> It's not too surprising to me that getting consistent locks between the
>>> two would be hard.
>>>
>>> Did you get any review from a Samba expert?  I seem to recall it having
>>> a lot of options, and I wonder if it's configured correctly for this
>>> case.
>>
>> No, I have not heard from any Samba expert.
>>
>>>
>>> It sounds like Samba may be giving out oplocks without getting a lease
>>> from the kernel.
>>
>> I will have to circle back to this when we're done with the 1st
>> phase of courteous server.
>
> I disabled oplock for the SMB share and locking between NFSv4 and SMB
> client works as expected. It appears that smbd does not set the VFS
> lease on the file after granting oplock to smb client.

Enabling kernel oplocks has the same effect, smbd does not grant oplock
to client forcing it to send lock request.

-Dai

>
> -Dai
>
>> -Dai
>>
>>>
>>> --b.
>>>
>>>> Here is what I did:
>>>>
>>>> NOTE: lck is a simple program that use lockf(3) to lock a file from
>>>> offset 0 to the length specified by '-l'.
>>>>
>>>> On NFSv4 client
>>>> ---------------
>>>>
>>>> [root@nfsvmd07 ~]# nfsstat -m
>>>> /tmp/mnt from nfsvmf24:/root/smb_share
>>>> Flags: 
>>>> rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,
>>>> proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.80.62.47,
>>>>         local_lock=none,addr=10.80.111.94
>>>> [root@nfsvmd07 ~]#
>>>>
>>>>
>>>> [root@nfsvmd07 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
>>>> Lck/file: 1, Maxlocks: 10000000
>>>> Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing 
>>>> F_LOCK..
>>>> LOCKED...
>>>>
>>>> Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
>>>>
>>>> [NFS client successfully locks the file]
>>>>
>>>> On SMB client
>>>> -------------
>>>>
>>>> [root@nfsvme24 ~]# mount |grep cifs
>>>> //nfsvmf24/smb_share on /tmp/mnt type cifs 
>>>> (rw,relatime,vers=3.1.1,cache=strict,username=root,uid=0,noforceuid,gid=0,noforcegid,addr=10.80.111.94,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
>>>> [root@nfsvme24 ~]#
>>>>
>>>> [root@nfsvme24 ~]# smbclient -L nfsvmf24
>>>> Enter SAMBA\root's password:
>>>>
>>>>     Sharename       Type      Comment
>>>>     ---------       ----      -------
>>>>     print$          Disk      Printer Drivers
>>>>     smb_share       Disk      Test Samba Share <<===== share to mount
>>>>     IPC$            IPC       IPC Service (Samba 4.10.16)
>>>>     root            Disk      Home Directories
>>>> Reconnecting with SMB1 for workgroup listing.
>>>>
>>>>     Server               Comment
>>>>     ---------            -------
>>>>
>>>>     Workgroup            Master
>>>>     ---------            -------
>>>> [root@nfsvme24 ~]#
>>>>
>>>> [root@nfsvme24 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
>>>> Lck/file: 1, Maxlocks: 10000000
>>>> Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing 
>>>> F_LOCK..
>>>> LOCKED...
>>>>
>>>> Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
>>>>
>>>> [SMB client successfully locks the file]
>>>>
>>>> The same issue happens when either client locks the file first.
>>>> I think this is what has happened:
>>>>
>>>> 1. NFSv4 client opens and locks the file first
>>>>
>>>>      . NFSv4 client send OPEN and LOCK to server, server replies
>>>>        OK on both requests.
>>>>
>>>>      . SMB client sends create request with Oplock==Lease for
>>>>        the same file.
>>>>
>>>>      . server holds off on replying to SMB client's create request,
>>>>        recalls delegation from NFSv4 client, waits for NFSv4 client
>>>>        to return the delegation then replies success to SMB client's
>>>>        create request with lease granted (Oplock==Lease).
>>>>
>>>>        NOTE: I think SMB server should replies the create request
>>>>        with Oplock==None to force the SMB client to sends the
>>>>        lock request.
>>>>
>>>>      . Once SMB client receives the reply of the create with
>>>>        'Oplock==Lease', it assumes it has full control of the file
>>>>        therefor it does not need to send the lock request.
>>>>
>>>>      . both NFSv4 and SMB client now think they have locked the file.
>>>>
>>>> pcap:  nfs_lock_smb_lock.pcap
>>>>
>>>> 2. SMB client creates the file with 'Oplock==Lease' first
>>>>
>>>>      . SMB sends create request with 'Oplock==Lease' to server,
>>>>        server replies OK with 'Oplock==Lease'. SMB client skips
>>>>        sending lock request since it assumes it has full control
>>>>        of the file with the lease.
>>>>
>>>>      . NFSv4 client sends OPEN to server, server replies OK with
>>>>        delagation is none. NFSv4 client sends LOCK request, since
>>>>        no lock was created in the kernel for the SMB client, the
>>>>        lock was granted to the NFSv4 client.
>>>>
>>>>       NOTE: I think the SMB server should send lease break
>>>>       notification to the SMB client, wait for the lease break
>>>>       acknowledgment from SMB client before replying to the
>>>>       OPEN of the NFSv4 client. This will force the SMB client
>>>>       to send the lock request to the server.
>>>>
>>>>      . both NFSv4 and SMB client now think they have locked the file.
>>>>
>>>> Your thought?
>>>>
>>>> Thanks,
>>>>
>>>> -Dai