Received-SPF: pass (google.com: domain of linux-nfs+bounces-505-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Tue, 12 Dec 2023 09:05:43 -0500
From: Chuck Lever <chuck.lever@oracle.com>
To: Jeff Layton <jlayton@kernel.org>, NeilBrown <neilb@suse.de>
Cc: Olga Kornievskaia <kolga@netapp.com>, Dai Ngo <Dai.Ngo@oracle.com>,
        Tom Talpey <tom@talpey.com>, linux-nfs@vger.kernel.org,
        linux-kernel@vger.kernel.org, Zhi Li <yieli@redhat.com>
Subject: Re: [PATCH] nfsd: properly tear down server when write_ports fails
Message-ID: <ZXhot6zUt6G1xaos@tissot.1015granger.net>
References: <20231211-nfsd-fixes-v1-1-c87a802f4977@kernel.org>
 <170233558429.12910.17902271117186364002@noble.neil.brown.name>
 <a2b59634a697ae07a315d6f663afaff5cd5bf375.camel@kernel.org>
 <ZXhlNtQ9o+howGbH@tissot.1015granger.net>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZXhlNtQ9o+howGbH@tissot.1015granger.net>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?PUnwLMWZsESROWvtNnuoodxCOHPmkwzOnCWcGGm6sf0d3l/OqC9fRD4hX6sC?=
 =?us-ascii?Q?RSWtIqSw7JvsvX1feERxOcr1JFZBd+V+TFZaWnZtiUS2wg67+Fh/Y4V1TFj+?=
 =?us-ascii?Q?7e23NfCC0mC+EVo6XQx/KKph2C4JUR7+CQMTbP4vmfqnFsyWvsWy5mNKjkZp?=
 =?us-ascii?Q?Ocr12Vs9UZ9TncOlXFS71NuwbscQwKu/3nBvbArlKtRoLh9BOJ3ZDMyecVC4?=
 =?us-ascii?Q?2i+fGdgdp9C9GogDuK3UK9/JIcDH4/PMmvkETjJuxd8yKSSsWndeixcGHkhm?=
 =?us-ascii?Q?Ytej5QFUMz5UxsONl9AB5TVWuhfvJlqKRAOoz879frlqj5yc9ldKg1oxOF6p?=
 =?us-ascii?Q?8N/eEkWGDgf3lvQk3nYkmkzOa4uCM4tTHrExB4Xmj6k5mDVk6aReyLEaD1TG?=
 =?us-ascii?Q?RSAZJkeSdI+46UEMWXYUuaAxMn60OOMKbak5n35GqOdvp4RCzDran24aaGIy?=
 =?us-ascii?Q?Ulsszymh3RuG7xuaJC2O9Fr++yRjiORNZZTcYu5OrS0+bP2qQCRAoDjmPJBi?=
 =?us-ascii?Q?kFdOz7AhJz56oO6Hlvod/07+iVAFpUewF+kibrM4MJDl+5Y6ZH+HGSDd6gjx?=
 =?us-ascii?Q?BPN94DCHmNhCJTOFV0WXv3PXCQc4UkUoxw94i5gGNowLzUFqi9o5b6/uxkMT?=
 =?us-ascii?Q?XqNwEFxy8qbBkx/NEFXChG1s7l5pe5TDmb4IfkWPW+8Zqz0ydRs9o1tkoOTz?=
 =?us-ascii?Q?afGtQIOZ93lfW5p+oJLHKEAQ4Oxsbi1+VXSzF+P6egHZn/moxkq73zfjBUky?=
 =?us-ascii?Q?D7MoRxZr/V8OW9jlEwJWoogy/3X1/lc2ZcDN7FEFDQ4zJ5R5Pypy/L7ob1Em?=
 =?us-ascii?Q?qqrNyYKwbc4N9GyOpnKv6ZMOGiyhAZyNcbK32Yf2qUhNYRlAXX/NZIjKP9U7?=
 =?us-ascii?Q?Xbv8eb0/Wog7Ioc0EAIyqSifyefKb/Hascr6tjrnGZz3bAeKEk0AIOWWij7K?=
 =?us-ascii?Q?b64YNWTmz1QbQAiRxKx+8JgpoXcTgEf/BA5W8aheWc2rYDzguzobI49ugnlA?=
 =?us-ascii?Q?G87qI96vk2xt85rLCRQ3QBBOCobT8QIQoqepeCfsFxy56sGiytMavQTlGJ7R?=
 =?us-ascii?Q?rAogFfPFZA4pxNd/4kROd4+Eki+cpPWszMXpjs3rUXU2/TnjffsHa6cVu7I/?=
 =?us-ascii?Q?pOhyqErtHIxruvksdfJpnCqXr/u/BKVhmHIfLcyb9HwdJwtgz3bhuejNHXXO?=
 =?us-ascii?Q?EMRFze/l5M1i6vm088p7CpUBOcgQBUT+ygy5Q0bZ1QTY+02L30y7nCEeQUqD?=
 =?us-ascii?Q?15bN+mfu2XeIfjP18OTqVGF5ud+L0SzGZI4QQKzUjkUfIv87TS33jQWPgW8+?=
 =?us-ascii?Q?+hAnorLUuGroXdBxwxt/QZu59MCh7u/NCk6YslZdHhK6gMBgP9TDAxfqzeDi?=
 =?us-ascii?Q?ULCp7AHLilyrl4TTOR1ufMbt56kz3RC/jxQjmXSb9PKKs05LBcvKxx/jcKRO?=
 =?us-ascii?Q?UvUGZKdyh/4FNfcvX5lctKtwNKBztMYF4APa4G7/XZm6F2t/+l6v211wtfcK?=
 =?us-ascii?Q?YNSzxtbvTo+ZBzuLPtqEqwnj7ZBytaw+/bM9CiJfJlamQZuim/hOra0Kvkv8?=
 =?us-ascii?Q?GV7FSNENdOK2BnfUhZh1yfA2ZSq2d18HsxDuJUsYqI1GwG5nIdI8cDw7VIp3?=
 =?us-ascii?Q?hg=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	QNjl+h8jbyeFbl+Z2juqNHNofirwBGO3qVEcjizWErTvUe5wcCYCWc6QW981DhGm25BC1B5h6uZlVnpzjqCEcfiiMiC+hd/5NvlswaPyOtDMZeBNbKtzz2RKqnq0rFOno9Za5BA/EY2PKI3QdvoImuTP0KzdI/flBEFcVYzQVg/iYerKCx22st2LXfj/qRpjgWgYSgfJTv9yTGb8S3Lud1gBKM6/z/qd+iSiCxN2LHTBp1787RZ/EF/wVc80lwyDEEwZx6h2wpcfVnhHZZ8+FHq/AX8cYI78+duTZEcKawMuYIr0KMCqYk9F2GUGurqAgQOfAfCUpGoq244/bnOEehD0ZmiIRMmcjgQIs1Icl5PXf63GqVDobsUb7Dyod90I9LIYDSLuWdaSSOd5TZeN5VjNNTls1Qq6lFlLB02Si7IYXpmlFnIpfOF4Zpzv5nt2/aMj4zZMyATYJFKGY7jQg+lGUg9Qvm7Hsl1z4Zpimn2lHuNEPpY4O5kRUPy4qxowWCz+bicHC+lrvRw3HU/2QDcwOsfRyC9DuuhMKszA4uFsIW7mkUDIS2VvggjWuFLAoZVxL7iO6vaJa1D2t8e2j3T9quZz76UiWAMWD6Rt6qJTlz49sn3IogZRftlO1idYiYyKm/TPLd2qgKEi1WyVPf7z8+ks6E5ut65u/zCih1v1QFVUB1Zn8vKzRrf61pc0iAeFxX3mRGYXFLBpKglqCxKhqFpXFIN0htSn8lcvx5XOdDbnadYBj3F7enw+05FBQypxzU8TZ3KxFh06b5x9TCR4o/CfibaOzkm3uSuyNMsSLPm8zRxbJ6Mop13MYl9EajjSCmCS13Obj0Gwr8+dvnXsG/fhiAKYXH/haGk5A5IkBh7fc7OdRQ65zcwli2syJlbsqJLs/EhUdr5Gx9tuXg==
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 11e0b59d-6ad0-49f0-3da5-08dbfb1b6ff0
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Dec 2023 14:05:46.5176
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: zWFQ9qRAdRBsSC4G7tD/ECLMH9vB+IsHBesPNMbe+xntWLNhO+jH++5gcJYb4s2AbVa8ETQWrQpVC+NRVuscXA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR10MB4444
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-12_07,2023-12-12_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 spamscore=0
 mlxscore=0 adultscore=0 phishscore=0 malwarescore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2312120107
X-Proofpoint-ORIG-GUID: tI4DgAjcx6FJQc_v511zv_-Mq0U1cGq-
X-Proofpoint-GUID: tI4DgAjcx6FJQc_v511zv_-Mq0U1cGq-

On Tue, Dec 12, 2023 at 08:50:46AM -0500, Chuck Lever wrote:
> On Mon, Dec 11, 2023 at 06:11:04PM -0500, Jeff Layton wrote:
> > On Tue, 2023-12-12 at 09:59 +1100, NeilBrown wrote:
> > > On Tue, 12 Dec 2023, Jeff Layton wrote:
> > > > When the initial write to the "portlist" file fails, we'll currently put
> > > > the reference to the nn->nfsd_serv, but leave the pointer intact. This
> > > > leads to a UAF if someone tries to write to "portlist" again.
> > > > 
> > > > Simple reproducer, from a host with nfsd shut down:
> > > > 
> > > >     # echo "foo 2049" > /proc/fs/nfsd/portlist
> > > >     # echo "foo 2049" > /proc/fs/nfsd/portlist
> > > > 
> > > > The kernel will oops on the second one when it trips over the dangling
> > > > nn->nfsd_serv pointer. There is a similar bug in __write_ports_addfd.
> > > > 
> > > > This patch fixes it by adding some extra logic to nfsd_put to ensure
> > > > that nfsd_last_thread is called prior to putting the reference when the
> > > > conditions are right.
> > > > 
> > > > Fixes: 9f28a971ee9f ("nfsd: separate nfsd_last_thread() from nfsd_put()")
> > > > Closes: https://issues.redhat.com/browse/RHEL-19081
> > > > Reported-by: Zhi Li <yieli@redhat.com>
> > > > Signed-off-by: Jeff Layton <jlayton@kernel.org>
> > > > ---
> > > > This should probably go to stable, but we'll need to backport for v6.6
> > > > since older kernels don't have nfsd_nl_rpc_status_get_done. We should
> > > > just be able to drop that hunk though.
> > > > ---
> > > >  fs/nfsd/nfsctl.c | 32 ++++++++++++++++++++++++++++----
> > > >  fs/nfsd/nfsd.h   |  8 +-------
> > > >  fs/nfsd/nfssvc.c |  2 +-
> > > >  3 files changed, 30 insertions(+), 12 deletions(-)
> > > 
> > > This is much the same as
> > > 
> > > https://lore.kernel.org/linux-nfs/20231030011247.9794-2-neilb@suse.de/
> > > 
> > > It seems that didn't land.  Maybe I dropped the ball...
> > 
> > Indeed it is. I thought the problem seemed familiar. Your set is
> > considerably more comprehensive. Looks like I even sent some Reviewed-
> > bys when you sent it.
> > 
> > Chuck, can we get these in or was there a problem with them?
> 
> Offhand, I'd say either I was waiting for some review comments
> to be addressed or the mail got lost (vger or Exchange or I
> accidentally deleted the series). I'll go take a look.

I reviewed the thread:

https://lore.kernel.org/linux-nfs/20231030011247.9794-1-neilb@suse.de/

From the looks of it, I was expecting Neil to address a couple of
review comments and repost. These are the two comments that stand
out to me now:

On 1/5:

> > Then let's add
> > 
> > Fixes: ec52361df99b ("SUNRPC: stop using ->sv_nrthreads as a refcount")
> > 
> > to this one, since it addresses a crasher seen in the wild.
> 
> Sounds good.
> 
> > > but it won't fix the hinky error cleanup in nfsd_svc. It looks like that
> > > does get fixed in patch #4 though, so I'm not too concerned.
> > 
> > Does that fix also need to be backported?
> 
> I think so, but we might want to split that out into a more targeted
> patch and apply it ahead of the rest of the series. Our QA folks seem to
> be able to hit the problem somehow, so it's likely to be triggered by
> people in the field too.

This last paragraph requests a bit of reorganization to enable an
easier backport.

And on 2/5:

> > > > +struct pool_private {
> > > > +	struct svc_serv *(*get_serv)(struct seq_file *, bool);
> > > 
> > > This bool is pretty ugly. I think I'd rather see two operations here
> > > (get_serv/put_serv). Also, this could use a kerneldoc comment.
> > 
> > I agree that bool is ugly, but two function pointers as function args
> > seemed ugly, and stashing them in 'struct svc_serv' seemed ugly.
> > So I picked one.  I'd be keen to find an approach that didn't require a
> > function pointer.
> > 
> > Maybe sunrpc could declare
> > 
> >    struct svc_ref {
> >          struct mutex mutex;
> >          struct svc_serv *serv;
> >    }
> > 
> > and nfsd could use one of those instead of nfsd_mutex and nfsd_serv, and
> > pass a pointer to it to the open function.
> > 
> > But then the mutex would have to be in the per-net structure.  And maybe
> > that isn't a bad idea, but it is a change...
> > 
> > I guess I could pass pointers to nfsd_mutex and nn->nfsd_serv to the
> > open function....
> > 
> > Any other ideas?
> 
> I think just passing two function pointers to svc_pool_stats_open, and
> storing them both in the serv is the best solution (for now). Like you
> said, there are no clean options here. That function only has one caller
> though, so at least the nastiness will be confined to that.
> 
> Moving the mutex to be per-net does make a lot of sense, but I think
> that's a separate project. If you decide to do that and it allows you to
> make a simpler interface for handling the get/put_serv pointers, then
> the interface can be reworked at that point.

The other requests I see in that thread have already been answered
adequately.


-- 
Chuck Lever