Received-SPF: pass (google.com: domain of linux-kernel+bounces-39123-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Thu, 25 Jan 2024 19:29:42 +0100 (CET)
From: Mikulas Patocka <mpatocka@redhat.com>
To: Linus Torvalds <torvalds@linux-foundation.org>, 
    Thomas Gleixner <tglx@linutronix.de>
cc: linux-kernel@vger.kernel.org, dm-devel@lists.linux.dev, 
    Mike Snitzer <msnitzer@redhat.com>, Ignat Korchagin <ignat@cloudflare.com>, 
    Damien Le Moal <damien.lemoal@wdc.com>, Bob Liu <bob.liu@oracle.com>, 
    Hou Tao <houtao1@huawei.com>, Nathan Huckleberry <nhuck@google.com>, 
    Peter Zijlstra <peterz@infradead.org>, Ingo Molnar <mingo@elte.hu>
Subject: [PATCH] softirq: fix memory corruption when freeing tasklet_struct
Message-ID: <82b964f0-c2c8-a2c6-5b1f-f3145dc2c8e5@redhat.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII

Hi

There's a problem with the tasklet API - there is no reliable way how to
free a structure that contains tasklet_struct. The problem is that the
function tasklet_action_common calls task_unlock(t) after it called the
callback. If the callback does something that frees tasklet_struct,
task_unlock(t) would write into free memory.

dm-crypt does this - it schedules a tasklet with tasklet_schedule, it does
encryption inside the tasklet handler (because it performs better than
doing the encryption in a workqueue), then it submits a workqueue entry
and calls bio_endio from the workqueue entry.

However, if the workqueue preempts ksoftirqd, this race condition happens:

ksoftirqd:
* tasklet_action_common
* t->func(t->data)	(that points to kcryptd_crypt_tasklet)
* kcryptd_crypt_tasklet
* kcryptd_crypt
* kcryptd_crypt_read_convert
* crypt_dec_pending
* queue_work(cc->io_queue, &io->work);
now we switch to the workqueue process:
* kcryptd_io_bio_endio
* bio_endio(io->base_bio)	(this calls clone_endio)
* clone_endio
* free_tio
* bio_put(clone) - the bio is freed
now we switch back to ksoftirqd:
* tasklet_action_common calls task_unlock(t) 
* task_unlock(t) touches memory that was already freed when the bio was freed

dm-verity has a similar problem.

In order to fix this bug, I am proposing to add a new flag
TASKLET_STATE_ONESHOT. The flag indicates that the tasklet will be
submitted only once and it prevents tasklet_action_common from touching
the tasklet after the callback completed.

If you have another idea how to solve this bug, let me know.

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Fixes: 39d42fa96ba1 ("dm crypt: add flags to optionally bypass kcryptd workqueues")
Fixes: 5721d4e5a9cd ("dm verity: Add optional "try_verify_in_tasklet" feature")
Cc: stable@vger.kernel.org	# v5.9+

---
 drivers/md/dm-crypt.c         |    1 +
 drivers/md/dm-verity-target.c |    1 +
 include/linux/interrupt.h     |    9 ++++++++-
 kernel/softirq.c              |   22 +++++++++++++++-------
 4 files changed, 25 insertions(+), 8 deletions(-)

Index: linux-2.6/drivers/md/dm-crypt.c
===================================================================
--- linux-2.6.orig/drivers/md/dm-crypt.c	2024-01-18 19:18:30.000000000 +0100
+++ linux-2.6/drivers/md/dm-crypt.c	2024-01-25 16:42:17.000000000 +0100
@@ -2265,6 +2265,7 @@ static void kcryptd_queue_crypt(struct d
 		if (in_hardirq() || irqs_disabled()) {
 			io->in_tasklet = true;
 			tasklet_init(&io->tasklet, kcryptd_crypt_tasklet, (unsigned long)&io->work);
+			tasklet_set_oneshot(&io->tasklet);
 			tasklet_schedule(&io->tasklet);
 			return;
 		}
Index: linux-2.6/include/linux/interrupt.h
===================================================================
--- linux-2.6.orig/include/linux/interrupt.h	2023-11-13 17:39:50.000000000 +0100
+++ linux-2.6/include/linux/interrupt.h	2024-01-25 16:41:52.000000000 +0100
@@ -684,7 +684,9 @@ struct tasklet_struct name = {				\
 enum
 {
 	TASKLET_STATE_SCHED,	/* Tasklet is scheduled for execution */
-	TASKLET_STATE_RUN	/* Tasklet is running (SMP only) */
+	TASKLET_STATE_RUN,	/* Tasklet is running (SMP only) */
+	TASKLET_STATE_ONESHOT	/* Don't unlock the tasklet after the callback
+				   to avoid writing to free memory */
 };
 
 #if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT_RT)
@@ -756,6 +758,11 @@ extern void tasklet_init(struct tasklet_
 extern void tasklet_setup(struct tasklet_struct *t,
 			  void (*callback)(struct tasklet_struct *));
 
+static inline void tasklet_set_oneshot(struct tasklet_struct *t)
+{
+	__set_bit(TASKLET_STATE_ONESHOT, &t->state);
+}
+
 /*
  * Autoprobing for irqs:
  *
Index: linux-2.6/kernel/softirq.c
===================================================================
--- linux-2.6.orig/kernel/softirq.c	2023-10-31 15:31:42.000000000 +0100
+++ linux-2.6/kernel/softirq.c	2024-01-25 17:10:03.000000000 +0100
@@ -774,18 +774,26 @@ static void tasklet_action_common(struct
 
 		if (tasklet_trylock(t)) {
 			if (!atomic_read(&t->count)) {
+				/*
+				 * If oneshot is set, we must not touch the
+				 * tasklet after the callback.
+				 */
+				bool oneshot = test_bit(TASKLET_STATE_ONESHOT, &t->state);
 				if (tasklet_clear_sched(t)) {
 					if (t->use_callback) {
-						trace_tasklet_entry(t, t->callback);
-						t->callback(t);
-						trace_tasklet_exit(t, t->callback);
+						void (*callback)(struct tasklet_struct *) = t->callback;
+						trace_tasklet_entry(t, callback);
+						callback(t);
+						trace_tasklet_exit(t, callback);
 					} else {
-						trace_tasklet_entry(t, t->func);
-						t->func(t->data);
-						trace_tasklet_exit(t, t->func);
+						void (*func)(unsigned long) = t->func;
+						trace_tasklet_entry(t, func);
+						func(t->data);
+						trace_tasklet_exit(t, func);
 					}
 				}
-				tasklet_unlock(t);
+				if (!oneshot)
+					tasklet_unlock(t);
 				continue;
 			}
 			tasklet_unlock(t);
Index: linux-2.6/drivers/md/dm-verity-target.c
===================================================================
--- linux-2.6.orig/drivers/md/dm-verity-target.c	2024-01-18 19:18:30.000000000 +0100
+++ linux-2.6/drivers/md/dm-verity-target.c	2024-01-25 18:12:09.000000000 +0100
@@ -676,6 +676,7 @@ static void verity_end_io(struct bio *bi
 
 	if (static_branch_unlikely(&use_tasklet_enabled) && io->v->use_tasklet) {
 		tasklet_init(&io->tasklet, verity_tasklet, (unsigned long)io);
+		tasklet_set_oneshot(&io->tasklet);
 		tasklet_schedule(&io->tasklet);
 	} else {
 		INIT_WORK(&io->work, verity_work);