Message-ID: <6aea870f-d51e-ed42-6f96-6b5b78edfcc3@oracle.com>
Date:   Mon, 6 Dec 2021 12:36:26 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.3.2
Subject: Re: [PATCH RFC v6 1/2] fs/lock: add new callback, lm_expire_lock, to
 lock_manager_operations
Content-Language: en-US
To:     "bfields@fieldses.org" <bfields@fieldses.org>,
        Trond Myklebust <trondmy@hammerspace.com>
Cc:     "jlayton@redhat.com" <jlayton@redhat.com>,
        "viro@zeniv.linux.org.uk" <viro@zeniv.linux.org.uk>,
        "chuck.lever@oracle.com" <chuck.lever@oracle.com>,
        "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
        "linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>
References: <20211206175942.47326-1-dai.ngo@oracle.com>
 <20211206175942.47326-2-dai.ngo@oracle.com>
 <133AE467-0990-469D-A8A9-497C1C1BD09A@oracle.com>
 <254f1d07c02a5b39d8b7743af4ceb9b5f69e4e07.camel@hammerspace.com>
 <20211206200540.GD20244@fieldses.org>
From:   dai.ngo@oracle.com
In-Reply-To: <20211206200540.GD20244@fieldses.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?M0I3UTcvTVl1YVdnZ1FaUlFuU0dVcGtDaTAyYWg4anZVcFN3SEtnaVFwdS90?=
 =?utf-8?B?ZWxWa082UTYzRjZsSVVvWVdveElLUWFGVHBWVnNQNHNab2RGc0Y4aXMzbEN4?=
 =?utf-8?B?bDJkNEVYbzB2Q2ZhQlJtbFVPR2t6ZGhDRFhvOEEycVZpcExrbEFEUTYrd25h?=
 =?utf-8?B?V3F1NFBPUVBMR0k4Um1PRzRDQ1I2WnM5dzI4cXZRUzVOMm1Gdldnb09BUjd4?=
 =?utf-8?B?TnI3M2lmYnptMFByLzV3R0cyNWVVVFJDalYvbDFOU2dDb3Z1SG1kR0Q5SnZV?=
 =?utf-8?B?aU5yZW9KQWxXajN1QWlacUNtUmtUdnZ1MUQxTXI1MGJOZ09tcWNpZ0FTOVF6?=
 =?utf-8?B?a0pWRkRUNUI0NVFRZHZCa0FXMExmdTZVbHU3Z203ak9EU0puK09pdGJFSFp6?=
 =?utf-8?B?OU4zTEg0bW5ieFl6YUxsT01NVjlEaW1tNE5ma2h2NENuOFFKME5PalJWeFA3?=
 =?utf-8?B?NDNNbDFYWEFnZEhIRlBEY2FycGs1WWh5TUFKNDI1c2h6Tytmb0p6WERyVkRX?=
 =?utf-8?B?bTA2ZkNweXdRRjR5YkRlR1QyZkd4RkVsZmhIYmErOE1va0xaWUs5bnRXNEdE?=
 =?utf-8?B?aVlPZStQckphYXEweDJEM2cwTzJRTFpzN1lpOVhpYlY4WVlPUzV2TWVVdENi?=
 =?utf-8?B?T0VYcTQ2dWIyUkpTMmp4VHQ3emgvODZMTXlQaFJvaW1ZMFhVdnpZT0tIM1Zt?=
 =?utf-8?B?VnQxMVJFcmtqeEdJNEJtWjhNSWNLS3VvaHB4bE8rNkEyTkNuMVJmOEgrWlVV?=
 =?utf-8?B?YStiWmxaVFFqcUpobkpFekxOVVh0NktydGNkeUNnc3ZEYit0WkpPWHhFY2hG?=
 =?utf-8?B?SWczUzNCSkY2UWFsWlZ4YUZEYlJWcHVJS0U0WUhNcmdCTitoSFZxT2xCS0dQ?=
 =?utf-8?B?eXovUTkxMDNwc082LzloUGFvdzFoY2FKamhlelhKNDVqTHViL2pUVWYwUTJl?=
 =?utf-8?B?Sk80cXRzY2h3Z0grU2xrdUtUblhtU2dUOW5sL2dNN2k5bmlMV3A5Q2xtMTNn?=
 =?utf-8?B?T0hreFhHbHRvNDUzQUZTaWdqNVJMUzNmMGJ2Kzg3WTAreW8yMEFydWR0ZDZV?=
 =?utf-8?B?eFdQOFNZVks1eVdOUFZHRjczK01BTUdRWGcwbE5BMTc3OWRFR1RaSlk2K05S?=
 =?utf-8?B?cTdEWlJ0MHNHWEVReUd6dWdCOGxBOTIyeWpiUWFrVWFuelpRalJ4NnJFQUJv?=
 =?utf-8?B?R0lxaWJlMGdmcWU2V1lRd212ZXNFZXJwZjFaUWFLYytNNy9FWnNvU1JMeTZF?=
 =?utf-8?B?RTg4bStDUkZDazZwSWVxSEprT1dUaXBqTTBTa0NwL3dKd0d1R2ZDRk9nd1Rh?=
 =?utf-8?B?SkRxdVFCQ0tYZkEwdEZjV3h6Zk1NNXVOWmNoSEhXWlFXaGUrOVppVjRUQmVJ?=
 =?utf-8?B?Y1hVbGpBOWFXN2NvNVY2ZEs3RHJEYzBqUzA4bGRKMTBHcU4yZGI2c2dZOTBw?=
 =?utf-8?B?anRtVFYyanEyMXpvdE5wMU94RjdpTXdDMEk1aEZMMkkwb3ZEbDI4WWF4QUNW?=
 =?utf-8?B?T2I5NFkrdHhIMEpQMlN4cEpTYlpkNlA4VkJMaXE3YUdoQzVCdko2Zzl4eC9P?=
 =?utf-8?B?NUNNcGtENmUwWWoramkreXAwUFB0LytRUDBjZnU5VVkrT2EyRzlGTDdjczJ2?=
 =?utf-8?B?Tkh5Vnh5clJodzJLNnlaUml0NjBPazZ6WkNCMzNYZXdVMkNncVdJRXQxYkZS?=
 =?utf-8?B?dzZMazc3VjBFckpIRGdxR01wQUhrR0dpaElhVEVnVjhKb0RuV1V5dklKdHZo?=
 =?utf-8?B?eS9uRG9QTTU2UXNFMFU3NkEzd2hreWR0cXZVUUQ2T1VYZUJ5MGFsR0FKSHBR?=
 =?utf-8?B?RDhZTElPM005VzYxOGliRjRnM2Z0dWpsN3BLQThuODFld1p3S1NNZk1ndHJC?=
 =?utf-8?B?a25KWVVwMSsrM1R3OStPY0Uxa0NwV3J3KzY2N2NIb2NpMHNWVE1qMlF0cVY4?=
 =?utf-8?B?RmZGZHZVSFAyZUVtUExFOFBwK1FENWpKRHlhNUxBWmVVMTFoQ0kvQVBWY0xP?=
 =?utf-8?B?Y3ZYcjJTVEkvK1J5UnBKUkdjcjJrMTlJTTB2MlExQmdDSkZDSmo3a3JtSkNn?=
 =?utf-8?B?TlZqZ2hTUm1NQU1CSmtaNjNmdkFsbDMwVVZRZ2R3d0J0R05ZMkhVclNGTS8v?=
 =?utf-8?B?d1hTRitrM0pmQm1SalRHcXYyMzFUbTdPeGZtQ2VScGNhc25EcHEwMk5PcFFW?=
 =?utf-8?Q?0tzt9k+K45Q1I+IpXciWVc8=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 23fc51a0-887a-4e95-b5d4-08d9b8f8146b
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2021 20:36:28.5057
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: hThbgwWZjH6KSvtydq/OGJsV/O/BlGk/F6tUoOQ+PGw05/QRc7lk+gjQVOm1uq0AkQTKXVmFVYUhsHWCudlRNw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3415
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10190 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxlogscore=999
 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 suspectscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2110150000 definitions=main-2112060126
X-Proofpoint-GUID: SvopE1KMIugFEhayBejfd6vvIxx2YV7n
X-Proofpoint-ORIG-GUID: SvopE1KMIugFEhayBejfd6vvIxx2YV7n
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


On 12/6/21 12:05 PM, bfields@fieldses.org wrote:
> On Mon, Dec 06, 2021 at 07:52:29PM +0000, Trond Myklebust wrote:
>> On Mon, 2021-12-06 at 18:39 +0000, Chuck Lever III wrote:
>>>
>>>> On Dec 6, 2021, at 12:59 PM, Dai Ngo <dai.ngo@oracle.com> wrote:
>>>>
>>>> Add new callback, lm_expire_lock, to lock_manager_operations to
>>>> allow
>>>> the lock manager to take appropriate action to resolve the lock
>>>> conflict
>>>> if possible. The callback takes 2 arguments, file_lock of the
>>>> blocker
>>>> and a testonly flag:
>>>>
>>>> testonly = 1  check and return true if lock conflict can be
>>>> resolved
>>>>               else return false.
>>>> testonly = 0  resolve the conflict if possible, return true if
>>>> conflict
>>>>               was resolved esle return false.
>>>>
>>>> Lock manager, such as NFSv4 courteous server, uses this callback to
>>>> resolve conflict by destroying lock owner, or the NFSv4 courtesy
>>>> client
>>>> (client that has expired but allowed to maintains its states) that
>>>> owns
>>>> the lock.
>>>>
>>>> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
>>> Al, Jeff, as co-maintainers of record for fs/locks.c, can you give
>>> an Ack or Reviewed-by? I'd like to take this patch through the nfsd
>>> tree for v5.17. Thanks for your time!
>>>
>>>
>>>> ---
>>>> fs/locks.c         | 28 +++++++++++++++++++++++++---
>>>> include/linux/fs.h |  1 +
>>>> 2 files changed, 26 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/fs/locks.c b/fs/locks.c
>>>> index 3d6fb4ae847b..0fef0a6322c7 100644
>>>> --- a/fs/locks.c
>>>> +++ b/fs/locks.c
>>>> @@ -954,6 +954,7 @@ posix_test_lock(struct file *filp, struct
>>>> file_lock *fl)
>>>>          struct file_lock *cfl;
>>>>          struct file_lock_context *ctx;
>>>>          struct inode *inode = locks_inode(filp);
>>>> +       bool ret;
>>>>
>>>>          ctx = smp_load_acquire(&inode->i_flctx);
>>>>          if (!ctx || list_empty_careful(&ctx->flc_posix)) {
>>>> @@ -962,11 +963,20 @@ posix_test_lock(struct file *filp, struct
>>>> file_lock *fl)
>>>>          }
>>>>
>>>>          spin_lock(&ctx->flc_lock);
>>>> +retry:
>>>>          list_for_each_entry(cfl, &ctx->flc_posix, fl_list) {
>>>> -               if (posix_locks_conflict(fl, cfl)) {
>>>> -                       locks_copy_conflock(fl, cfl);
>>>> -                       goto out;
>>>> +               if (!posix_locks_conflict(fl, cfl))
>>>> +                       continue;
>>>> +               if (cfl->fl_lmops && cfl->fl_lmops->lm_expire_lock
>>>> &&
>>>> +                               cfl->fl_lmops->lm_expire_lock(cfl,
>>>> 1)) {
>>>> +                       spin_unlock(&ctx->flc_lock);
>>>> +                       ret = cfl->fl_lmops->lm_expire_lock(cfl,
>>>> 0);
>>>> +                       spin_lock(&ctx->flc_lock);
>>>> +                       if (ret)
>>>> +                               goto retry;
>>>>                  }
>>>> +               locks_copy_conflock(fl, cfl);
>> How do you know 'cfl' still points to a valid object after you've
>> dropped the spin lock that was protecting the list?
> Ugh, good point, I should have noticed that when I suggested this
> approach....
>
> Maybe the first call could instead return return some reference-counted
> object that a second call could wait on.
>
> Better, maybe it could add itself to a list of such things and then we
> could do this in one pass.

I think we adjust this logic a little bit to cover race condition:

The 1st call to lm_expire_lock returns the client needs to be expired.

Before we make the 2nd call, we save the 'lm_expire_lock' into a local
variable then drop the spinlock, and use the local variable to make the
2nd call so that we do not reference 'cfl'. The argument of the second
is the opaque return value from the 1st call.

nfsd4_fl_expire_lock also needs some adjustment to support the above.

-Dai

>
> --b.
>
>>>> +               goto out;
>>>>          }
>>>>          fl->fl_type = F_UNLCK;
>>>> out:
>>>> @@ -1140,6 +1150,7 @@ static int posix_lock_inode(struct inode
>>>> *inode, struct file_lock *request,
>>>>          int error;
>>>>          bool added = false;
>>>>          LIST_HEAD(dispose);
>>>> +       bool ret;
>>>>
>>>>          ctx = locks_get_lock_context(inode, request->fl_type);
>>>>          if (!ctx)
>>>> @@ -1166,9 +1177,20 @@ static int posix_lock_inode(struct inode
>>>> *inode, struct file_lock *request,
>>>>           * blocker's list of waiters and the global blocked_hash.
>>>>           */
>>>>          if (request->fl_type != F_UNLCK) {
>>>> +retry:
>>>>                  list_for_each_entry(fl, &ctx->flc_posix, fl_list) {
>>>>                          if (!posix_locks_conflict(request, fl))
>>>>                                  continue;
>>>> +                       if (fl->fl_lmops && fl->fl_lmops-
>>>>> lm_expire_lock &&
>>>> +                                       fl->fl_lmops-
>>>>> lm_expire_lock(fl, 1)) {
>>>> +                               spin_unlock(&ctx->flc_lock);
>>>> +                               percpu_up_read(&file_rwsem);
>>>> +                               ret = fl->fl_lmops-
>>>>> lm_expire_lock(fl, 0);
>>>> +                               percpu_down_read(&file_rwsem);
>>>> +                               spin_lock(&ctx->flc_lock);
>>>> +                               if (ret)
>>>> +                                       goto retry;
>>>> +                       }
>> ditto.
>>
>>>>                          if (conflock)
>>>>                                  locks_copy_conflock(conflock, fl);
>>>>                          error = -EAGAIN;
>>>> diff --git a/include/linux/fs.h b/include/linux/fs.h
>>>> index e7a633353fd2..1a76b6451398 100644
>>>> --- a/include/linux/fs.h
>>>> +++ b/include/linux/fs.h
>>>> @@ -1071,6 +1071,7 @@ struct lock_manager_operations {
>>>>          int (*lm_change)(struct file_lock *, int, struct list_head
>>>> *);
>>>>          void (*lm_setup)(struct file_lock *, void **);
>>>>          bool (*lm_breaker_owns_lease)(struct file_lock *);
>>>> +       bool (*lm_expire_lock)(struct file_lock *fl, bool
>>>> testonly);
>>>> };
>>>>
>>>> struct lock_manager {
>>>> -- 
>>>> 2.9.5
>>>>
>>> --
>>> Chuck Lever
>>>
>>>
>>>
>> -- 
>> Trond Myklebust
>> Linux NFS client maintainer, Hammerspace
>> trond.myklebust@hammerspace.com
>>
>>