Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   Mike Christie <michael.christie@oracle.com>
To:     geert@linux-m68k.org, hdanton@sina.com, hch@infradead.org,
        stefanha@redhat.com, jasowang@redhat.com, mst@redhat.com,
        sgarzare@redhat.com, virtualization@lists.linux-foundation.org,
        christian.brauner@ubuntu.com, axboe@kernel.dk,
        ebiederm@xmission.com, linux-kernel@vger.kernel.org
Cc:     Mike Christie <michael.christie@oracle.com>
Subject: [PATCH V8 6/8] vhost_task: Allow vhost layer to use copy_process
Date:   Wed,  9 Feb 2022 17:29:37 -0600
Message-Id: <20220209232939.9169-7-michael.christie@oracle.com>
In-Reply-To: <20220209232939.9169-1-michael.christie@oracle.com>
References: <20220209232939.9169-1-michael.christie@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?KrNGaVcmbfsvjU583W/+IEk8/avYSuOHZBMGfkymmmUL1tB7+jwxsdD8J2MY?=
 =?us-ascii?Q?DSycVtWTVeYnThoO1Ac1hZxTLNUB7zXx53k9bLpUlQ6Kvua7D3IdAuSxKRfC?=
 =?us-ascii?Q?HHdefHfh3T6c6HI4MZB7mBU8egpqQrn4NkkUWVkAmk9pUN3Fv3Zg2GBtfYit?=
 =?us-ascii?Q?SeOEt/hOEYyDD1CS5p9owaUAIvk2xkv8ft7JjfeIdnYprkckhQX+KPkMgP48?=
 =?us-ascii?Q?7nww5F1xw5HqiL/25wl/gUNfz2SDACU4RMwOaugQcAfIF8xwaF6rCIXh5osg?=
 =?us-ascii?Q?LCmJUuRPuWZEZESXmyccB4RV6gHvAkE5yz1HVkowFYLAGhno+P9UNS81EIzy?=
 =?us-ascii?Q?ONbonQdPwU9Djdgj3x/BFqtqoLLC+Ul8cr8nVJxNTy/gBmyuM7HaOb+qdUKK?=
 =?us-ascii?Q?zrVl4l+i1JSNwilXBLlfi1dsSCm5GwsVFQI3/4qNH2NBMFqzwIT6jBvSZZI6?=
 =?us-ascii?Q?H5t9fkmtEBmNKqag1BaCoS4hTCc0ujiX6cefsG418y9uIUJbZ522PDZSFcgh?=
 =?us-ascii?Q?zxQAOfRW+/oMYawh+GtppqxjbpeA58oyrcyBdC1u71bN0QB9mbsWK5VJrtgm?=
 =?us-ascii?Q?njIYRIAu/7TC8nSz4Soo4mlng+GIewKu+CLYNJvonM60zr8o6ei835DLHCf3?=
 =?us-ascii?Q?PuFqErW6HCKu7QYQ++NcVjuP9PnPL3QdSyD5psh8Ie5jtDCwCbEbWYp8j5l3?=
 =?us-ascii?Q?G+6Ymza5jnwoghoq4jFdf3mSLNvOjlaMZpwvxCYpLzM98rg0+aXZN12850kv?=
 =?us-ascii?Q?yztwg9Hzjzr4cX8hxQgnYrMUNOLhMzvDWveG3LwBRDesWHRbZxm3CBd4qt1L?=
 =?us-ascii?Q?oloWH5f9SoNVzJ71jps/YYOkHAK3S+hFh3OsHyKDhBLh9EH3ZZQ7x0O+yf0g?=
 =?us-ascii?Q?VynSWkZeZaUmwTu+/ATLXreuAM8RNZ55ypc9ZS/zMTpxtCKE7LRhi3ZrAHur?=
 =?us-ascii?Q?lEuBjttYWiMmuV10rUDHc5RcX6M+ViR0MDJwKaTBN7/+cY7UkjnrxVPUVXjP?=
 =?us-ascii?Q?Alko51LUvjVU4MKzWSswlZurFJGD6xH5GM5//GR3/D+Jc0Ui6N0qAZ6Z3YRc?=
 =?us-ascii?Q?4g0M3jYhyGx0Tpa8bshMAx/cDaJKq+hdWsheGkcYjZJ8HSeFK1h8Mu7NbW/a?=
 =?us-ascii?Q?RRBo6aczTl2JnX37XJfK1PiqhmV3XCfGX+YtgYV9aBiDSEQ72/RFZzPLC4Fn?=
 =?us-ascii?Q?m50Dd6R+9XihbULw46R8yPD1fH92frxTK+l5DIcfWjgfzUTqepB+uAM+owZk?=
 =?us-ascii?Q?DLOdSpGPCXFKwxZZ9SlGwdXco5CiKwIfzJ3zCO58kiGnkjAepWujfTUEQ602?=
 =?us-ascii?Q?WUE47PkSnfAu+Urio/LtLj+ZwZsIgAgzy/mjQIf2mubJDJZlBAD47KBf1FQY?=
 =?us-ascii?Q?yVfjTWPlEcKRSA9LrE2Ekw2ylgfZdlyePhT2bUI/+YJ7SP92tWfdRoy2OhGE?=
 =?us-ascii?Q?+vbwSy/nm2RUEJz18pnGOao3z1gLsKaKlHQKW6SF+y76JIcMNveUFC9KNpkz?=
 =?us-ascii?Q?fkNchtKub40hhvyHp+VXpW+n6VuipkNVE42WC/147bxtb21gkfLOynChaFiH?=
 =?us-ascii?Q?sHbGMDSACDQWo1iWRHFNmvWZZEIG29hzILlkAFkqmSims3jTczBim0ClXnc4?=
 =?us-ascii?Q?Zj/yD/HfW3i9jP9LdaTyBOGrin6gCxNbmXff9+U4IsZ0Kh86PSQWAq2HaEcq?=
 =?us-ascii?Q?8lB+eA=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4a2feccb-9005-4255-0d30-08d9ec24131f
X-MS-Exchange-CrossTenant-AuthSource: DM5PR10MB1466.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2022 23:29:53.4978
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: nW5edfpJf7M9cxRbB/U8bl710U5iP0L9ZmFPfFxQQPTskpD06if7xfRtFb3/utSeaXmOX/axZs8V40HBDxxSv0SB8Olppe2TT084c3I8S64=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB3007
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10253 signatures=673431
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 suspectscore=0
 mlxlogscore=999 mlxscore=0 adultscore=0 malwarescore=0 spamscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000
 definitions=main-2202090122
X-Proofpoint-GUID: HJ6CKZzgio3Gs4DxOE79OeuPAvQt7GlB
X-Proofpoint-ORIG-GUID: HJ6CKZzgio3Gs4DxOE79OeuPAvQt7GlB
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Qemu will create vhost devices in the kernel which perform network, SCSI,
etc IO and management operations from worker threads created by the
kthread API. Because the kthread API does a copy_process on the kthreadd
thread, the vhost layer has to use kthread_use_mm to access the Qemu
thread's memory and cgroup_attach_task_all to add itself to the Qemu
thread's cgroups, and it bypasses the RLIMIT_NPROC limit.

This patch adds a new struct vhost_task which can be used instead of
kthreads. They allow the vhost layer to use copy_process and inherit
the userspace process's mm and cgroups and the task is accounted for
under the userspace's nproc count.

Signed-off-by: Mike Christie <michael.christie@oracle.com>
---
 MAINTAINERS                      |   2 +
 drivers/vhost/Kconfig            |   5 ++
 include/linux/sched/vhost_task.h |  23 ++++++
 kernel/Makefile                  |   1 +
 kernel/vhost_task.c              | 123 +++++++++++++++++++++++++++++++
 5 files changed, 154 insertions(+)
 create mode 100644 include/linux/sched/vhost_task.h
 create mode 100644 kernel/vhost_task.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 3e461db9cd91..4a3d9541e3cc 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -20477,7 +20477,9 @@ L:	virtualization@lists.linux-foundation.org
 L:	netdev@vger.kernel.org
 S:	Maintained
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git
+F:	kernel/vhost_task.c
 F:	drivers/vhost/
+F:	include/linux/sched/vhost_task.h
 F:	include/linux/vhost_iotlb.h
 F:	include/uapi/linux/vhost.h
 
diff --git a/drivers/vhost/Kconfig b/drivers/vhost/Kconfig
index 587fbae06182..b455d9ab6f3d 100644
--- a/drivers/vhost/Kconfig
+++ b/drivers/vhost/Kconfig
@@ -13,9 +13,14 @@ config VHOST_RING
 	  This option is selected by any driver which needs to access
 	  the host side of a virtio ring.
 
+config VHOST_TASK
+	bool
+	default n
+
 config VHOST
 	tristate
 	select VHOST_IOTLB
+	select VHOST_TASK
 	help
 	  This option is selected by any driver which needs to access
 	  the core of vhost.
diff --git a/include/linux/sched/vhost_task.h b/include/linux/sched/vhost_task.h
new file mode 100644
index 000000000000..50d02a25d37b
--- /dev/null
+++ b/include/linux/sched/vhost_task.h
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_VHOST_TASK_H
+#define _LINUX_VHOST_TASK_H
+
+#include <linux/completion.h>
+
+struct task_struct;
+
+struct vhost_task {
+	int (*fn)(void *data);
+	void *data;
+	struct completion exited;
+	unsigned long flags;
+	struct task_struct *task;
+};
+
+struct vhost_task *vhost_task_create(int (*fn)(void *), void *arg, int node);
+__printf(2, 3)
+void vhost_task_start(struct vhost_task *vtsk, const char namefmt[], ...);
+void vhost_task_stop(struct vhost_task *vtsk);
+bool vhost_task_should_stop(struct vhost_task *vtsk);
+
+#endif
diff --git a/kernel/Makefile b/kernel/Makefile
index 56f4ee97f328..d82f388082b8 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -15,6 +15,7 @@ obj-y     = fork.o exec_domain.o panic.o \
 obj-$(CONFIG_USERMODE_DRIVER) += usermode_driver.o
 obj-$(CONFIG_MODULES) += kmod.o
 obj-$(CONFIG_MULTIUSER) += groups.o
+obj-$(CONFIG_VHOST_TASK) += vhost_task.o
 
 ifdef CONFIG_FUNCTION_TRACER
 # Do not trace internal ftrace files
diff --git a/kernel/vhost_task.c b/kernel/vhost_task.c
new file mode 100644
index 000000000000..4ab1c195bc76
--- /dev/null
+++ b/kernel/vhost_task.c
@@ -0,0 +1,123 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2021 Oracle Corporation
+ */
+#include <linux/slab.h>
+#include <linux/completion.h>
+#include <linux/sched/task.h>
+#include <linux/sched/vhost_task.h>
+#include <linux/sched/signal.h>
+
+enum vhost_task_flags {
+	VHOST_TASK_FLAGS_STOP,
+};
+
+static void vhost_task_fn(void *data)
+{
+	struct vhost_task *vtsk = data;
+	int ret;
+
+	ret = vtsk->fn(vtsk->data);
+	complete(&vtsk->exited);
+	do_exit(ret);
+}
+
+/**
+ * vhost_task_stop - stop a vhost_task
+ * @vtsk: vhost_task to stop
+ *
+ * Callers must call vhost_task_should_stop and return from their worker
+ * function when it returns true;
+ */
+void vhost_task_stop(struct vhost_task *vtsk)
+{
+	pid_t pid = vtsk->task->pid;
+
+	set_bit(VHOST_TASK_FLAGS_STOP, &vtsk->flags);
+	wake_up_process(vtsk->task);
+	/*
+	 * Make sure vhost_task_fn is no longer accessing the vhost_task before
+	 * freeing it below. If userspace crashed or exited without closing,
+	 * then the vhost_task->task could already be marked dead so
+	 * kernel_wait will return early.
+	 */
+	wait_for_completion(&vtsk->exited);
+	/*
+	 * If we are just closing/removing a device and the parent process is
+	 * not exiting then reap the task.
+	 */
+	kernel_wait4(pid, NULL, __WCLONE, NULL);
+	kfree(vtsk);
+}
+EXPORT_SYMBOL_GPL(vhost_task_stop);
+
+/**
+ * vhost_task_should_stop - should the vhost task return from the work function
+ */
+bool vhost_task_should_stop(struct vhost_task *vtsk)
+{
+	return test_bit(VHOST_TASK_FLAGS_STOP, &vtsk->flags);
+}
+EXPORT_SYMBOL_GPL(vhost_task_should_stop);
+
+/**
+ * vhost_task_create - create a copy of a process to be used by the kernel
+ * @fn: thread stack
+ * @arg: data to be passed to fn
+ * @node: numa node to allocate task from
+ *
+ * This returns a specialized task for use by the vhost layer or NULL on
+ * failure. The returned task is inactive, and the caller must fire it up
+ * through vhost_task_start().
+ */
+struct vhost_task *vhost_task_create(int (*fn)(void *), void *arg, int node)
+{
+	struct kernel_clone_args args = {
+		.flags		= CLONE_FS | CLONE_UNTRACED | CLONE_VM,
+		.exit_signal	= 0,
+		.stack		= (unsigned long)vhost_task_fn,
+		.worker_flags	= USER_WORKER | USER_WORKER_NO_FILES |
+				  USER_WORKER_SIG_IGN,
+	};
+	struct vhost_task *vtsk;
+	struct task_struct *tsk;
+
+	vtsk = kzalloc(sizeof(*vtsk), GFP_KERNEL);
+	if (!vtsk)
+		return ERR_PTR(-ENOMEM);
+
+	init_completion(&vtsk->exited);
+	vtsk->data = arg;
+	vtsk->fn = fn;
+	args.stack_size =  (unsigned long)vtsk;
+
+	tsk = copy_process(NULL, 0, node, &args);
+	if (IS_ERR(tsk)) {
+		kfree(vtsk);
+		return NULL;
+	}
+
+	vtsk->task = tsk;
+
+	return vtsk;
+}
+EXPORT_SYMBOL_GPL(vhost_task_create);
+
+/**
+ * vhost_task_start - start a vhost_task created with vhost_task_create
+ * @vtsk: vhost_task to wake up
+ * @namefmt: printf-style format string for the thread name
+ */
+void vhost_task_start(struct vhost_task *vtsk, const char namefmt[], ...)
+{
+	char name[TASK_COMM_LEN];
+	va_list args;
+
+	va_start(args, namefmt);
+	vsnprintf(name, sizeof(name), namefmt, args);
+	set_task_comm(vtsk->task, name);
+	va_end(args);
+
+	wake_up_new_task(vtsk->task);
+}
+EXPORT_SYMBOL_GPL(vhost_task_start);
-- 
2.25.1