Received-SPF: pass (google.com: domain of linux-kernel+bounces-44850-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Tue, 30 Jan 2024 10:34:18 -0500
From: Rodrigo Vivi <rodrigo.vivi@intel.com>
To: Mukesh Ojha <quic_mojha@quicinc.com>
CC: <linux-kernel@vger.kernel.org>, Johannes Berg <johannes@sipsolutions.net>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>, "Rafael J . Wysocki"
	<rafael@kernel.org>, Jose Souza <jose.souza@intel.com>
Subject: Re: [PATCH 2/2] devcoredump: Remove the mutex serialization
Message-ID: <ZbkW-kJ8DTgCZphm@intel.com>
References: <20240126151121.1076079-1-rodrigo.vivi@intel.com>
 <20240126151121.1076079-2-rodrigo.vivi@intel.com>
 <e356adc5-937f-08ba-68bc-975ef96d6332@quicinc.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <e356adc5-937f-08ba-68bc-975ef96d6332@quicinc.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Hk4gwRdRzxtOvyt7Fs3ye5s7JHqAQQsYhADGj3C1tAown89ek463PgwgeSRG?=
 =?us-ascii?Q?ATb+x5zPx1ccHWnZkvFsW5GUJYcVfTLNoHDODOmjbJjgNqpkH2SYm70lG5Ux?=
 =?us-ascii?Q?t2RcUOL4xAsgnGZa+mUphD9Xfv4Pt/u3dIToNMRVH4LGmMrKZNa55BBZ7H7v?=
 =?us-ascii?Q?faa/yKPQGHRU6f2hVOwOAF2y2nLK1kcetxxPU94HmzbRhke7izn4LjkFaR/C?=
 =?us-ascii?Q?M3U8kXpurNyl0k6NItqELuo8/Qga6A1+4qvU7FlQXYRf7ZeCOmqLJe1kHTD1?=
 =?us-ascii?Q?9Cezkm+WBrEkWlY3QzpmAVzHoeQCAYBRdcdM5cUX1ic4Gk6VwPsz9rfjOzps?=
 =?us-ascii?Q?eaw9sQFgBgOvy+2pmddlEOQRFLiu7ObOxmby5Yma+zyXpqepZQI4bQ43++Fu?=
 =?us-ascii?Q?sFD9wY31i2zKvR99KIePcx20BLGnKPaq+Ck2jROWcfZVoBcaJj30QgmQsdhk?=
 =?us-ascii?Q?By7WQ9vKDyNGsTpgjjPJPOY6TrPBD0KVgrqtmrolYCSgclXfJtVX08Zi8ow3?=
 =?us-ascii?Q?rhFrObm02pOS4GlObALL+DMI8OD6yDCEUqSr5z0bZGr0kW0i/b1ad4+XAcwu?=
 =?us-ascii?Q?bloMOiE8js2E/EnvYbAeYeSBZOhObz56+jHDgl2BUQ1LOVRrUPHxthr3T1MC?=
 =?us-ascii?Q?fx5YQCd0b2VZ14sJ2/jlQRcAGihcl8yCSb9UrVGFCm5jbj1OBqjXuaXimnAk?=
 =?us-ascii?Q?ealjZ9IOlebVQB6f/R6x0dNcD1LOv6g5q4GxfAJ/fD1nzCoGoAzvBx5JPzym?=
 =?us-ascii?Q?gQ5Y9SghbVgl25jb4i9QOBKzJMIvAnehzn3G7fmlS6EtGH9BoIXToYkZwKCe?=
 =?us-ascii?Q?9K86hBSaaIisPRxaOVRy9LK7yWus7WD6VMq6Dj0aQKCwqB/sF3JamU6ZlCsQ?=
 =?us-ascii?Q?cwG+NNef06Js2s2HYDtCaWeIyIzXIabHIdJra3+5ievsiZz86m/yx+OHCVC5?=
 =?us-ascii?Q?IDBThLdmlWiz0qm7rVhOMeTS6os5lOR/jwDPnKa65s01jmih70IE1HHmix5b?=
 =?us-ascii?Q?TjAdUyVErz8aQBcvNWKx/U+mkGjeSQLjk71Y7eMUHr4RaEF0kkGtBnut05BJ?=
 =?us-ascii?Q?FaEV8IFxDbhBqkHHDIYffur9xx82vaMFzWE/rHnowQP6gbLOx78RbRsEdfxe?=
 =?us-ascii?Q?UYhafBLK9xfc389JVpmJtzOBQclx+zL2w+FU6xLJAeeYynaw5JpJtvqiR/Bj?=
 =?us-ascii?Q?QzHsvcIKZq/kSzhGSaaIgdgp9mnyjFBK8NK2lF8usqsDH2/jTurmI4uF0Wvg?=
 =?us-ascii?Q?IfOKdFeu50qG7EEswPT/TldnX5hssdKSQopDGC6eQM9nkzoeHCeiv7FjD4cO?=
 =?us-ascii?Q?qrelCYGsMgrEg31Ni97JwxPAnjfEv8CN1Qnonb2L4auW96phgvOn2sPGHRw8?=
 =?us-ascii?Q?4PKX2MIGirk3b83RV4TaEDrnb5j4LETvGyYn5nNPlFfODBQK/HwXaKChzaMS?=
 =?us-ascii?Q?KhaD0nT5wczBcXP0itLfPpu2dzVRVamVxKHDVp3Wy6ae++jBJKuoJC1OqXji?=
 =?us-ascii?Q?3r81pyC5zu/rTYm5QS/s+Jp3YhiLryPGWhBSHRdYB5p6n5FRkWlQQF0AydZg?=
 =?us-ascii?Q?fWVLGaS6RR4R+rdWbs2lnZ6z5xdoDV+99nT6CNyx?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 2a45117a-6470-4e90-0f44-08dc21a8ee8c
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6059.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Jan 2024 15:34:22.3148
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 9/gQEzOuAcvwmuws5+6ibJzzbmKss1Egp/pAHdgxgZKbIkq5Z9tWsBgcdbVLCOFnh5Ulw7LFBK/rN8MVbgWG7w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR11MB7273
X-OriginatorOrg: intel.com

On Tue, Jan 30, 2024 at 05:32:24PM +0530, Mukesh Ojha wrote:
> 
> 
> On 1/26/2024 8:41 PM, Rodrigo Vivi wrote:
> > The commit 01daccf74832 ("devcoredump : Serialize devcd_del work")
> > introduced the mutex to protect the case where mod_delayed_work
> > could be called before the delayed work even existed.
> > 
> > Instead, we can simply initialize the delayed work before the device
> > is added, so the race condition doesn't exist at first place.
> > 
> > The mutex_unlock is very problematic here. Although mod_delayed_work
> > is async, we have no warranty that the work is not finished before
> > the mutex_unlock(devcd->mutex), and if that happen 'devcd' is used
> > after freed.
> 
> I agree, Mutex is bad and last time there was only a situation of UAF from
> disable_store() and that can not occur as it keeps its ref, so
> we went ahead with the change.,

my concern was with:

flush_delayed_work(&devcd->del_wk);
mutex_unlock(&devcd->mutex);

at devcd_free().

flush_work always wait for the work to finish it's execution,
which will delete the device.
The with the release, the devcd should be gone soon and
this is at risk of the UAF, no?

maybe I'm missing something.


> 
> > 
> > Cc: Mukesh Ojha <quic_mojha@quicinc.com>
> > Cc: Johannes Berg <johannes@sipsolutions.net>
> > Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> > Cc: Rafael J. Wysocki <rafael@kernel.org>
> > Cc: Jose Souza <jose.souza@intel.com>
> > Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
> > ---
> >   drivers/base/devcoredump.c | 97 +++-----------------------------------
> >   1 file changed, 6 insertions(+), 91 deletions(-)
> > 
> > diff --git a/drivers/base/devcoredump.c b/drivers/base/devcoredump.c
> > index 678ecc2fa242..0e26b1273920 100644
> > --- a/drivers/base/devcoredump.c
> > +++ b/drivers/base/devcoredump.c
> > @@ -25,47 +25,6 @@ struct devcd_entry {
> >   	struct device devcd_dev;
> >   	void *data;
> >   	size_t datalen;
> > -	/*
> > -	 * Here, mutex is required to serialize the calls to del_wk work between
> > -	 * user/kernel space which happens when devcd is added with device_add()
> > -	 * and that sends uevent to user space. User space reads the uevents,
> > -	 * and calls to devcd_data_write() which try to modify the work which is
> > -	 * not even initialized/queued from devcoredump.
> > -	 *
> > -	 *
> > -	 *
> > -	 *        cpu0(X)                                 cpu1(Y)
> > -	 *
> > -	 *        dev_coredump() uevent sent to user space
> > -	 *        device_add()  ======================> user space process Y reads the
> > -	 *                                              uevents writes to devcd fd
> > -	 *                                              which results into writes to
> > -	 *
> > -	 *                                             devcd_data_write()
> > -	 *                                               mod_delayed_work()
> > -	 *                                                 try_to_grab_pending()
> > -	 *                                                   del_timer()
> > -	 *                                                     debug_assert_init()
> > -	 *       INIT_DELAYED_WORK()
> > -	 *       schedule_delayed_work()
> > -	 *
> > -	 *
> > -	 * Also, mutex alone would not be enough to avoid scheduling of
> > -	 * del_wk work after it get flush from a call to devcd_free()
> > -	 * mentioned as below.
> > -	 *
> > -	 *	disabled_store()
> > -	 *        devcd_free()
> > -	 *          mutex_lock()             devcd_data_write()
> > -	 *          flush_delayed_work()
> > -	 *          mutex_unlock()
> > -	 *                                   mutex_lock()
> > -	 *                                   mod_delayed_work()
> > -	 *                                   mutex_unlock()
> > -	 * So, delete_work flag is required.
> > -	 */
> > -	struct mutex mutex;
> > -	bool delete_work;
> >   	struct module *owner;
> >   	ssize_t (*read)(char *buffer, loff_t offset, size_t count,
> >   			void *data, size_t datalen);
> > @@ -125,13 +84,8 @@ static ssize_t devcd_data_write(struct file *filp, struct kobject *kobj,
> >   	struct device *dev = kobj_to_dev(kobj);
> >   	struct devcd_entry *devcd = dev_to_devcd(dev);
> > -	mutex_lock(&devcd->mutex);
> > -	if (!devcd->delete_work) {
> > -		devcd->delete_work = true;
> > -		mod_delayed_work(system_wq, &devcd->del_wk, 0);
> > -	}
> > -	mutex_unlock(&devcd->mutex);
> > -
> > +	/* This file needs to be closed before devcd can be deleted */
> > +	mod_delayed_work(system_wq, &devcd->del_wk, 0);
> >   	return count;
> >   }
> > @@ -158,12 +112,7 @@ static int devcd_free(struct device *dev, void *data)
> >   {
> >   	struct devcd_entry *devcd = dev_to_devcd(dev);
> > -	mutex_lock(&devcd->mutex);
> > -	if (!devcd->delete_work)
> > -		devcd->delete_work = true;
> > -
> >   	flush_delayed_work(&devcd->del_wk);
> > -	mutex_unlock(&devcd->mutex);
> >   	return 0;
> >   }
> > @@ -173,30 +122,6 @@ static ssize_t disabled_show(const struct class *class, const struct class_attri
> >   	return sysfs_emit(buf, "%d\n", devcd_disabled);
> >   }
> > -/*
> > - *
> > - *	disabled_store()                                	worker()
> > - *	 class_for_each_device(&devcd_class,
> > - *		NULL, NULL, devcd_free)
> > - *         ...
> > - *         ...
> > - *	   while ((dev = class_dev_iter_next(&iter))
> > - *                                                             devcd_del()
> > - *                                                               device_del()
> > - *                                                                 put_device() <- last reference
> > - *             error = fn(dev, data)                           devcd_dev_release()
> > - *             devcd_free(dev, data)                           kfree(devcd)
> > - *             mutex_lock(&devcd->mutex);
> > - *
> > - *
> > - * In the above diagram, It looks like disabled_store() would be racing with parallely
> > - * running devcd_del() and result in memory abort while acquiring devcd->mutex which
> > - * is called after kfree of devcd memory  after dropping its last reference with
> > - * put_device(). However, this will not happens as fn(dev, data) runs
> > - * with its own reference to device via klist_node so it is not its last reference.
> > - * so, above situation would not occur.
> > - */
> > -
> >   static ssize_t disabled_store(const struct class *class, const struct class_attribute *attr,
> >   			      const char *buf, size_t count)
> >   {
> > @@ -308,13 +233,7 @@ static void devcd_remove(void *data)
> >   {
> >   	struct devcd_entry *devcd = data;
> > -	mutex_lock(&devcd->mutex);
> > -	if (!devcd->delete_work) {
> > -		devcd->delete_work = true;
> > -		/* XXX: Cannot flush otherwise the mutex below will hit a UAF */
> > -		mod_delayed_work(system_wq, &devcd->del_wk, 0);
> > -	}
> > -	mutex_unlock(&devcd->mutex);
> > +	flush_delayed_work(&devcd->del_wk);
> >   }
> >   /**
> > @@ -365,16 +284,15 @@ void dev_coredumpm(struct device *dev, struct module *owner,
> >   	devcd->read = read;
> >   	devcd->free = free;
> >   	devcd->failing_dev = get_device(dev);
> > -	devcd->delete_work = false;
> > -	mutex_init(&devcd->mutex);
> >   	device_initialize(&devcd->devcd_dev);
> >   	dev_set_name(&devcd->devcd_dev, "devcd%d",
> >   		     atomic_inc_return(&devcd_count));
> >   	devcd->devcd_dev.class = &devcd_class;
> > -	mutex_lock(&devcd->mutex);
> > +	INIT_DELAYED_WORK(&devcd->del_wk, devcd_del);
> > +	schedule_delayed_work(&devcd->del_wk, DEVCD_TIMEOUT);
> 
> Last time, we discussed [1] here, involves a assumption
> about timeout can not happen before device_add() succeeds.
> It is rare but it is there.
> 
> https://lore.kernel.org/all/87ilr15ekx.ffs@tglx/

hmm... I couldn't imagine a case where a device_add could
take longer then 5 minutes, at least not without other bigger
problems...

I'm wondering that multiple subsequent calls of dev_coredumpm()
would fail to find the failing_device with the class_find_device
and all, but maybe I'm overthinking here or missing something else.

> 
> -Mukesh
> 
> >   	dev_set_uevent_suppress(&devcd->devcd_dev, true);
> >   	if (device_add(&devcd->devcd_dev))
> >   		goto put_device;
> > @@ -392,15 +310,12 @@ void dev_coredumpm(struct device *dev, struct module *owner,
> >   	dev_set_uevent_suppress(&devcd->devcd_dev, false);
> >   	kobject_uevent(&devcd->devcd_dev.kobj, KOBJ_ADD);
> > -	INIT_DELAYED_WORK(&devcd->del_wk, devcd_del);
> > -	schedule_delayed_work(&devcd->del_wk, DEVCD_TIMEOUT);
> >   	if (devm_add_action(dev, devcd_remove, devcd))
> >   		dev_warn(dev, "devcoredump managed auto-removal registration failed\n");
> > -	mutex_unlock(&devcd->mutex);
> >   	return;
> >    put_device:
> > +	cancel_delayed_work(&devcd->del_wk);
> >   	put_device(&devcd->devcd_dev);
> > -	mutex_unlock(&devcd->mutex);
> >    put_module:
> >   	module_put(owner);
> >    free: