Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Trond Myklebust <trondmy@hammerspace.com>,
        "jlayton@kernel.org" <jlayton@kernel.org>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH RFC] NFSD: Fix possible sleep during
 nfsd4_release_lockowner()
Thread-Topic: [PATCH RFC] NFSD: Fix possible sleep during
 nfsd4_release_lockowner()
Thread-Index: AQHYbfIMzaj7dJY/skSZB97V0kxLh60seYcAgAAWLQCAAAdIAIAABFwAgAAPm4CAABJ8gIAATM0A
Date:   Mon, 23 May 2022 22:18:45 +0000
Message-ID: <85DDDD2E-CCEE-4EA4-8271-4C75C409E106@oracle.com>
References: <165323344948.2381.7808135229977810927.stgit@bazille.1015granger.net>
 <fe3f9ece807e1433631ee3e0bd6b78238305cb87.camel@kernel.org>
 <510282CB-38D3-438A-AF8A-9AC2519FCEF7@oracle.com>
 <c3d053dc36dd5e7dee1267f1c7107bbf911e4d53.camel@kernel.org>
 <1A37E2B5-8113-48D6-AF7C-5381F364D99E@oracle.com>
 <c357e63272de96f9e7595bf6688680879d83dc83.camel@kernel.org>
 <93d11e12532f5a10153d3702100271f70373bce6.camel@hammerspace.com>
In-Reply-To: <93d11e12532f5a10153d3702100271f70373bce6.camel@hammerspace.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?59jNzMHpqIx5DEjgYUTXqVHzDZGsJJ8ZrV2smpghoZafnMKlgrfYjqMD8dId?=
 =?us-ascii?Q?2bvFkrEn0PWZ7FpmsGBNxWHxzSY1IUiSbdeXn93CQLZOhxT1s+f9rRfcbxVN?=
 =?us-ascii?Q?mf7kT82c/eUrckhNiDKKziQHFmFO5Ic0mUO1GnIhpCPDc3VDR6xBFYByoJE+?=
 =?us-ascii?Q?5R3xR/1+Qffil1K512bTIEwMbuUMLbjfV6RxQf5EGDX7p6EUJwQpnRSJjtZ/?=
 =?us-ascii?Q?BzVm54jaMis1GFMG6bxx98jyexjgKDOvAS4g9dpHu07dWaUiUI4LBCb30kwb?=
 =?us-ascii?Q?h0ozRWTjppWT7wXLWqWrwkdlI1UX1EJPfNu7yIxcBHhggLXNPnFriq8FtfB8?=
 =?us-ascii?Q?lLkvzfIlZoZ7hhz0fjrbn9h4v0+D4/rnLjDfFjFwA6EeGOSSbS5s7hL5He6/?=
 =?us-ascii?Q?43RYyDbKIsiS8TlIAZuJbsPqdA5VpjfOCkS3xfbSPxaNOJTUoo+DY1hIG96L?=
 =?us-ascii?Q?jfkjQv5gZBI8PzntlnrsedpT8HDe5YEWGIb40eH+M8r+CmLNZkrCp12/RQ4B?=
 =?us-ascii?Q?Jdxd/pDx0moPCr7ROHWPFu8HGoFDbhor4oYCfOHSL+KoY1/tf3q/wnJrJ6at?=
 =?us-ascii?Q?dpLCDEZHXX7HnCTF+kE4pmZgigVMSpp0nEmqXAFki20LsRXr2L2qP4erObOn?=
 =?us-ascii?Q?yB22lhyve/uZ5I+tyMlgt1T0+tyOB6gkzzZzMcppN8CC+WBln2C43xGm6Xwb?=
 =?us-ascii?Q?2WaH2EGKCQlxNQBbd+nqvyr6HqUff596WUJk/aK+G0yGSSYKcUJQlAH83cQV?=
 =?us-ascii?Q?FpHyV3qDiFTd8nGMQ0koRdPtGP1o0I/inft8pj9lqQt/Gu9jQvxrZvW4Ov99?=
 =?us-ascii?Q?SiJQBhrddfexkQiBMAMZ9SuClNTI0AmJzo1CPbVHSj6vEd24va5B5vP+/Osd?=
 =?us-ascii?Q?dm+eH2UqCfPgcnxfZRAVcA9baxvAderjtgjgYYibpQ4/SUTd2fzTj3qhUGUj?=
 =?us-ascii?Q?hcFxYUjQnFiE7wk3UmroJBiTy2kco9ZXZHvdUhbCPNMGhpbAk4lstizHe7X+?=
 =?us-ascii?Q?i1H3bLnXKfcNKH64FCyCoWs9RcICBAwjyvSg/YJQrG9bPPKXFwBWKaQsFExV?=
 =?us-ascii?Q?6UPZNo6e4alYjI2x8fsf2KCO8APo/aL8E/5ngrCsZhxDn3ncbSYv4hLTpayF?=
 =?us-ascii?Q?N6JVkeSidQxBFrwmyfTlyM8wWNk7kP/x/g1vgiikzrODLvYOoMlV1U5MmqkK?=
 =?us-ascii?Q?QiCpEvhiQtSkZDUOFNasCBT4hnNs4wfsovctmEaOms3DfmVukXexvzWZq+px?=
 =?us-ascii?Q?o43Je3/2jL9OiXwYovYGgDTJHi5KV41TtefvXQK74CGCsdVcCAA9E8z3Ws8t?=
 =?us-ascii?Q?7AslrnnbBAQPWXq05raXPZrT+X4gG2iV4J70EZYkcl2O4eV4hHBunTFcSQts?=
 =?us-ascii?Q?PyF8Uw2okBFD2ctpPwLwOmExon91TPvVlCVWobBkqBHHQlAXrkyrsQ20WkQn?=
 =?us-ascii?Q?riP2MZG89lO91wbn14kVa608+yfc0c6m7eEOH5ubw+zz9Jbw0CFEzzMUevc0?=
 =?us-ascii?Q?C9D4aU3t8qZx72F/PQudCc1ojIrHfejjs5QvamGwKiPHfazjcjhZi2hEsvpu?=
 =?us-ascii?Q?lXiGnkRZ5WHdYSDVj8NRFtxKJ8LmIkMCy/+azVWR8SE85V5tI8nnsLjnaxZT?=
 =?us-ascii?Q?tHHzYF+njMemtqOgv0CvF3iJJiZ1i5BuMydR8l2l/wxQ+crYEFhG2ua+rB0L?=
 =?us-ascii?Q?fWkzcMhwNj4hisEt9W6TAqzzrQAf3bW8ZjUdNtgJNkeuTJSBy6WzMkZO/eJx?=
 =?us-ascii?Q?hEW1XBJBg8O60ESYNb43RN877l1uFl4=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <E446F90FEB309649A68609A0D52F7A02@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d5d84b36-3769-4a00-df63-08da3d0a3393
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 May 2022 22:18:45.0174
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: XEuOZBrCrBwcAlEwE+8SGSoYAZbOhOKEV+BWAgjVA+IuFKImKOc+R0XOOaEWdC/13ws+XYI6LEfDraBe4ROk3g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR10MB1585
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.874
 definitions=2022-05-23_09:2022-05-23,2022-05-23 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 phishscore=0
 spamscore=0 malwarescore=0 mlxlogscore=824 suspectscore=0 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000
 definitions=main-2205230114
X-Proofpoint-GUID: OfKcfw4lltUBGT2YjclOD6h1Jr3wcgk-
X-Proofpoint-ORIG-GUID: OfKcfw4lltUBGT2YjclOD6h1Jr3wcgk-
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On May 23, 2022, at 1:43 PM, Trond Myklebust <trondmy@hammerspace.com> wr=
ote:
>=20
> IOW: the number of lm_get_owner and lm_put_owner calls should always be
> 100% balanced once all the locks belonging to a specific lock owner are
> removed.

It doesn't look like that's the case. If we rely solely on
lm_get/put_owner, the lockcnt quickly becomes negative, even
without the presence of lock conflicts:

            nfsd-999   [001]   155.888211: nfsd_compound:        xid=3D0x00=
00001c opcnt=3D2
            nfsd-999   [001]   155.888260: nfsd_compound_status: op=3D1/2 O=
P_PUTFH status=3D0
            nfsd-999   [001]   155.888283: nfsd_preprocess:      seqid=3D2 =
client 628c04a3:f2adef0d stateid 00000002:00000002
            nfsd-999   [001]   155.888419: bprint:               nfsd4_lm_g=
et_owner: lo=3D0xffff8881955b1050 conflict=3Dfalse lockcnt=3D1 so_count=3D3
            nfsd-999   [001]   155.888431: bprint:               nfsd4_lm_p=
ut_owner.part.0: lo=3D0xffff8881955b1050 conflict=3Dfalse lockcnt=3D1 so_co=
unt=3D3
            nfsd-999   [001]   155.888441: nfsd_file_put:        hash=3D0x3=
39 inode=3D0xffff88816770d908 ref=3D3 flags=3DHASHED|REFERENCED may=3DWRITE=
|READ file=3D0xffff88811232f7c0
            nfsd-999   [001]   155.888453: nfsd_compound_status: op=3D2/2 O=
P_LOCK status=3D0
            nfsd-999   [003]   155.890388: nfsd_compound:        xid=3D0x00=
00001d opcnt=3D2
            nfsd-999   [003]   155.890457: nfsd_compound_status: op=3D1/2 O=
P_PUTFH status=3D0
            nfsd-999   [003]   155.890459: nfsd_preprocess:      seqid=3D1 =
client 628c04a3:f2adef0d stateid 00000003:00000001
            nfsd-999   [003]   155.890479: bprint:               nfsd4_lm_p=
ut_owner.part.0: lo=3D0xffff8881955b1050 conflict=3Dfalse lockcnt=3D0 so_co=
unt=3D4
            nfsd-999   [003]   155.890490: nfsd_file_put:        hash=3D0x3=
39 inode=3D0xffff88816770d908 ref=3D3 flags=3DHASHED|REFERENCED may=3DWRITE=
|READ file=3D0xffff88811232f7c0
            nfsd-999   [003]   155.890492: bprint:               nfsd4_lm_p=
ut_owner.part.0: lo=3D0xffff8881955b1050 conflict=3Dfalse lockcnt=3D-1 so_c=
ount=3D3
            nfsd-999   [003]   155.890502: nfsd_compound_status: op=3D2/2 O=
P_LOCKU status=3D0
            nfsd-999   [003]   155.892494: nfsd_compound:        xid=3D0x00=
00001e opcnt=3D5
            nfsd-999   [003]   155.892498: nfsd_compound_status: op=3D1/5 O=
P_PUTROOTFH status=3D0
            nfsd-999   [003]   155.892528: nfsd_compound_status: op=3D2/5 O=
P_LOOKUP status=3D0
            nfsd-999   [003]   155.892552: nfsd_compound_status: op=3D3/5 O=
P_LOOKUP status=3D0
            nfsd-999   [003]   155.892576: nfsd_compound_status: op=3D4/5 O=
P_LOOKUP status=3D0
            nfsd-999   [003]   155.892604: nfsd_file_fsnotify_handle_event:=
 inode=3D0xffff88816770d908 nlink=3D0 mode=3D0100644 mask=3D0x4
            nfsd-999   [003]   155.892605: nfsd_file_unhash_and_release_loc=
ked: hash=3D0x339 inode=3D0xffff88816770d908 ref=3D2 flags=3DHASHED|REFEREN=
CED may=3DWRITE|READ file=3D0xffff88811232f7c0
            nfsd-999   [003]   155.892606: nfsd_file_unhash:     hash=3D0x3=
39 inode=3D0xffff88816770d908 ref=3D2 flags=3DREFERENCED may=3DWRITE|READ f=
ile=3D0xffff88811232f7c0
            nfsd-999   [003]   155.892607: nfsd_file_close_inode: hash=3D0x=
339 inode=3D0xffff88816770d908 found=3D0
            nfsd-999   [003]   155.892610: nfsd_compound_status: op=3D5/5 O=
P_REMOVE status=3D0
            nfsd-999   [001]   155.894663: nfsd_compound:        xid=3D0x00=
00001f opcnt=3D1
            nfsd-999   [001]   155.894667: bprint:               nfsd4_rele=
ase_lockowner: lo=3D0xffff8881955b1050 lockcnt=3D-2 so_count=3D1
            nfsd-999   [001]   155.894670: nfsd_file_put:        hash=3D0x3=
39 inode=3D0xffff88816770d908 ref=3D2 flags=3DREFERENCED may=3DWRITE|READ f=
ile=3D0xffff88811232f7c0
            nfsd-999   [001]   155.894706: nfsd_compound_status: op=3D1/1 O=
P_RELEASE_LOCKOWNER status=3D0

There are other places inside NFSD that call nfs4_get/
put_stateowner on a lockowner. That seems to be what keeps
the so_count above water.

Or, another way to look at it: simply saving the value of
the conflict boolean in file_lock and passing that to
lm_put_owner() doesn't seem to be adequate.


--
Chuck Lever