Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: Locking issue between NFSv4 and SMB client
From:   dai.ngo@oracle.com
To:     Bruce Fields <bfields@fieldses.org>
Cc:     "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
        CIFS <linux-cifs@vger.kernel.org>
References: <5b7be2c0-95a6-048c-581f-17e5e3750daa@oracle.com>
 <20210923215056.GH18334@fieldses.org>
 <90a8f89b-e8ac-2187-2926-d723ebbcb839@oracle.com>
Message-ID: <4c98a686-3be9-6f95-ea1a-8f03fbf3ea0c@oracle.com>
Date:   Thu, 7 Oct 2021 10:03:34 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0)
 Gecko/20100101 Thunderbird/78.14.0
In-Reply-To: <90a8f89b-e8ac-2187-2926-d723ebbcb839@oracle.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Q3VnaTR0anVyZ1hFZUkxblkwNjR3V2taUzc1S2ZQcyt6Q0pPY2RHK2llY3ZC?=
 =?utf-8?B?Z2FmbTcvQ2R1NmMzNWM1cE9SMVBycG1YK252QzcvRERUZUN6VStiaFNYNWpS?=
 =?utf-8?B?MWNPRFVpUzZvNGRLNkg5Zld2WkRyaDZtZ2tkQnZyYXRJcW5BMHVUVmNENURH?=
 =?utf-8?B?T2xrR0NYdG5hK25UcVJlN1N0aUdHZ3dHdHlMZlRmaGwvamU1djJPek9PRXJo?=
 =?utf-8?B?bmZrMDh0OGpVemkzaFYvSEpIOW1QYTNjaDNCVnlsemhsMDVwYjJXbjVLME1l?=
 =?utf-8?B?TVhmc2NHOUM1ZkM3M0tyS0FaUzYxY0tiN1Z1anJ0WG1qSVNHR291SVBsdzJE?=
 =?utf-8?B?elhNMFkvSWM0dTZodzlweVU5WlA0Y1F2S0FVaW0vUkNYT2dWVUxOSzJkZzcw?=
 =?utf-8?B?L2svZTJ5TUliVmJjTXBPR09VSE8yYUJzVlpONUVwc1V6RHBZbENmZmE5KzhZ?=
 =?utf-8?B?ekpLcCt5c0NzaHREVGJFOFg2OEJERHZDSGpEUHE2R0pyc3JrVWRicnROLys3?=
 =?utf-8?B?dVhaUnk2TCtLRW9BbzUra0V3azhNdWJuL2hxbnhMN2RyQUlsNDA1cG9CbWhu?=
 =?utf-8?B?c3UxZnFYekJ3KzdkcERXUEgySVBSTktUa1IwNDZNbEk5alZFWkYybzdLMlRU?=
 =?utf-8?B?WG55RzhDV0dHcENhMFFzVnRVNEZyZU5YQktPZDkrU0Zub3l6czFhbFNsN1VE?=
 =?utf-8?B?N1dIYW9PWjdEeU9wSWF6ZGFKWlVuWUhXY1dISWNBdzRsbVBzNm5sWk93bnFr?=
 =?utf-8?B?NGhNdmpsVWx5eGl0ZUwwOVlJUHc4c0N3L05GTUxtQmkzdmltZWRrenlMajBk?=
 =?utf-8?B?SUFVUGRpbXRYTDdFakw5NUQ2QlZaRC9pd3hmVnlzb1F3SXZyUEpGUkgxN2Zp?=
 =?utf-8?B?dkRZbE1sU1pnMlM2QmRoQXo0dHd5QjgvVnBrS0FLMnhoUWduSW1uV0xhelU2?=
 =?utf-8?B?cmFIUU1KR2xKNEtWQTU1dTE3a0syWTlrTGdUS21oYWxSZVRTWUlra0t3MHdZ?=
 =?utf-8?B?ek02eEJlc3dFdTBubnVJaFYvd3RtcWFIMi9QVXh5Z2U3VWJvTXFMK01RL2Mz?=
 =?utf-8?B?MEdOQ1J6bnNPby9Wb0hTRFdtTEpCODlob1BFczJKQ3JZQjFLVFYvMkxvdHNP?=
 =?utf-8?B?U1g5Q0FXVHZPa2puay92NzBPdFJlZUNIM3UvQWg2OUVpR2daNmcvbk1maVVR?=
 =?utf-8?B?WUE2Si82cEhYZ25idWxBaDV5L1k3cGJxaldMN1E5elhnK040ZUMvZzkraEdY?=
 =?utf-8?B?M3d2dGJaTWd1YkdyejRTc1ZleVJIUmU3N2hiWlBrekVvZnE2cGNPWjhUa1h0?=
 =?utf-8?B?aHZzNGUrT1F0VFYzQ0dpMjhkTW0rUm5XcEcxNS9peFdtN0F1TTRQNWF4MlVV?=
 =?utf-8?B?WVA3OWRBN0JtZnNhY3JXYTJOck9OTWhIRjhNVWtEclE2NXRPelg3ZUdzUmRz?=
 =?utf-8?B?OE9NUnJHRnlKNVNMeDN5V3JIbkw4UklpTVVqRkQvbDk0bHJDNWd6TXhnNXJp?=
 =?utf-8?B?NTQ0Q0VrZkttUGpON2J3eHJpUFBBNVVqVlZSWlpiaXNMS2JpcGlQYTNVeFJM?=
 =?utf-8?B?VFVndy9NeWk0a29vaWZrVWtvYnc0Y1dIM0ppQnFEOWlGaXVlN2pyVUFjenFR?=
 =?utf-8?B?aHdrNlRHdWt5VzN4Rm1ZbHFnUVV4NUZTZ0tqaHo4UmVrczNrMURQTVdheUN2?=
 =?utf-8?B?VnhFUG1kL2grYnVCWTREQWMvTHlOSmpyNU0vbVQyZUtQaldmYXk0bjNsbU5U?=
 =?utf-8?Q?/D1RSkMA8kLHrXKOPI8AriFvsrMy1mjvDRXfS2Q?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9a032551-0118-4d99-5010-08d989b46787
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Oct 2021 17:03:37.5995
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YRZHpsjJJhcuIy5YQX9v56JKybLRLVS+P43V4AMZ10H3LnqEAWN9wTYVwCCGRViDmyFhodm7njG0KgXgJfwZvw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3605
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10130 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0
 phishscore=0 malwarescore=0 bulkscore=0 mlxlogscore=999 mlxscore=0
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2109230001 definitions=main-2110070110
X-Proofpoint-GUID: 0cU2Zm5ltxy2sHuOIlCwKSszVZrUYcCz
X-Proofpoint-ORIG-GUID: 0cU2Zm5ltxy2sHuOIlCwKSszVZrUYcCz
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


On 9/23/21 3:39 PM, dai.ngo@oracle.com wrote:
>
> On 9/23/21 2:50 PM, Bruce Fields wrote:
>> On Thu, Jul 15, 2021 at 04:45:22PM -0700, dai.ngo@oracle.com wrote:
>>> Hi Bruce,
>> Oops, sorry for neglecting this.
>>
>>> I'm doing some locking testing between NFSv4 and SMB client and
>>> think there are some issues on the server that allows both clients
>>> to lock the same file at the same time.
>> It's not too surprising to me that getting consistent locks between the
>> two would be hard.
>>
>> Did you get any review from a Samba expert?  I seem to recall it having
>> a lot of options, and I wonder if it's configured correctly for this
>> case.
>
> No, I have not heard from any Samba expert.
>
>>
>> It sounds like Samba may be giving out oplocks without getting a lease
>> from the kernel.
>
> I will have to circle back to this when we're done with the 1st
> phase of courteous server.

I disabled oplock for the SMB share and locking between NFSv4 and SMB
client works as expected. It appears that smbd does not set the VFS
lease on the file after granting oplock to smb client.

-Dai

> -Dai
>
>>
>> --b.
>>
>>> Here is what I did:
>>>
>>> NOTE: lck is a simple program that use lockf(3) to lock a file from
>>> offset 0 to the length specified by '-l'.
>>>
>>> On NFSv4 client
>>> ---------------
>>>
>>> [root@nfsvmd07 ~]# nfsstat -m
>>> /tmp/mnt from nfsvmf24:/root/smb_share
>>> Flags: 
>>> rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,
>>> proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.80.62.47,
>>>         local_lock=none,addr=10.80.111.94
>>> [root@nfsvmd07 ~]#
>>>
>>>
>>> [root@nfsvmd07 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
>>> Lck/file: 1, Maxlocks: 10000000
>>> Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing 
>>> F_LOCK..
>>> LOCKED...
>>>
>>> Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
>>>
>>> [NFS client successfully locks the file]
>>>
>>> On SMB client
>>> -------------
>>>
>>> [root@nfsvme24 ~]# mount |grep cifs
>>> //nfsvmf24/smb_share on /tmp/mnt type cifs 
>>> (rw,relatime,vers=3.1.1,cache=strict,username=root,uid=0,noforceuid,gid=0,noforcegid,addr=10.80.111.94,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
>>> [root@nfsvme24 ~]#
>>>
>>> [root@nfsvme24 ~]# smbclient -L nfsvmf24
>>> Enter SAMBA\root's password:
>>>
>>>     Sharename       Type      Comment
>>>     ---------       ----      -------
>>>     print$          Disk      Printer Drivers
>>>     smb_share       Disk      Test Samba Share <<===== share to mount
>>>     IPC$            IPC       IPC Service (Samba 4.10.16)
>>>     root            Disk      Home Directories
>>> Reconnecting with SMB1 for workgroup listing.
>>>
>>>     Server               Comment
>>>     ---------            -------
>>>
>>>     Workgroup            Master
>>>     ---------            -------
>>> [root@nfsvme24 ~]#
>>>
>>> [root@nfsvme24 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000
>>> Lck/file: 1, Maxlocks: 10000000
>>> Locking[/tmp/mnt/messages] Offset[0] Len[100000000] N[0]...doing 
>>> F_LOCK..
>>> LOCKED...
>>>
>>> Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop
>>>
>>> [SMB client successfully locks the file]
>>>
>>> The same issue happens when either client locks the file first.
>>> I think this is what has happened:
>>>
>>> 1. NFSv4 client opens and locks the file first
>>>
>>>      . NFSv4 client send OPEN and LOCK to server, server replies
>>>        OK on both requests.
>>>
>>>      . SMB client sends create request with Oplock==Lease for
>>>        the same file.
>>>
>>>      . server holds off on replying to SMB client's create request,
>>>        recalls delegation from NFSv4 client, waits for NFSv4 client
>>>        to return the delegation then replies success to SMB client's
>>>        create request with lease granted (Oplock==Lease).
>>>
>>>        NOTE: I think SMB server should replies the create request
>>>        with Oplock==None to force the SMB client to sends the
>>>        lock request.
>>>
>>>      . Once SMB client receives the reply of the create with
>>>        'Oplock==Lease', it assumes it has full control of the file
>>>        therefor it does not need to send the lock request.
>>>
>>>      . both NFSv4 and SMB client now think they have locked the file.
>>>
>>> pcap:  nfs_lock_smb_lock.pcap
>>>
>>> 2. SMB client creates the file with 'Oplock==Lease' first
>>>
>>>      . SMB sends create request with 'Oplock==Lease' to server,
>>>        server replies OK with 'Oplock==Lease'. SMB client skips
>>>        sending lock request since it assumes it has full control
>>>        of the file with the lease.
>>>
>>>      . NFSv4 client sends OPEN to server, server replies OK with
>>>        delagation is none. NFSv4 client sends LOCK request, since
>>>        no lock was created in the kernel for the SMB client, the
>>>        lock was granted to the NFSv4 client.
>>>
>>>       NOTE: I think the SMB server should send lease break
>>>       notification to the SMB client, wait for the lease break
>>>       acknowledgment from SMB client before replying to the
>>>       OPEN of the NFSv4 client. This will force the SMB client
>>>       to send the lock request to the server.
>>>
>>>      . both NFSv4 and SMB client now think they have locked the file.
>>>
>>> Your thought?
>>>
>>> Thanks,
>>>
>>> -Dai