Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758687AbbGHMyP (ORCPT ); Wed, 8 Jul 2015 08:54:15 -0400 Received: from mx1.redhat.com ([209.132.183.28]:55596 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758650AbbGHMyM (ORCPT ); Wed, 8 Jul 2015 08:54:12 -0400 Message-ID: <1436360051.1391.131.camel@redhat.com> Subject: Re: [RFC PATCH] irq: IRQ bypass manager From: Alex Williamson To: "Wu, Feng" Cc: "linux-kernel@vger.kernel.org" , "kvm@vger.kernel.org" , "eric.auger@st.com" , "eric.auger@linaro.org" , "joro@8bytes.org" , "avi.kivity@gmail.com" , "pbonzini@redhat.com" Date: Wed, 08 Jul 2015 06:54:11 -0600 In-Reply-To: References: <20150707212725.9250.21077.stgit@gimli.home> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 13938 Lines: 404 On Wed, 2015-07-08 at 12:22 +0000, Wu, Feng wrote: > > > > -----Original Message----- > > From: Alex Williamson [mailto:alex.williamson@redhat.com] > > Sent: Wednesday, July 08, 2015 5:40 AM > > To: linux-kernel@vger.kernel.org; kvm@vger.kernel.org > > Cc: eric.auger@st.com; eric.auger@linaro.org; joro@8bytes.org; > > avi.kivity@gmail.com; pbonzini@redhat.com; Wu, Feng > > Subject: [RFC PATCH] irq: IRQ bypass manager > > > > When a physical I/O device is assigned to a virtual machine through > > facilities like VFIO and KVM, the interrupt for the device generally > > bounces through the host system before being injected into the VM. > > However, hardware technologies exist that often allow the host to be > > bypassed for some of these scenarios. Intel Posted Interrupts allow > > the specified physical edge interrupts to be directly injected into a > > guest when delivered to a physical processor while the vCPU is > > running. ARM IRQ Forwarding allows the hypervisor to handle level > > triggered device interrupts as edge interrupts, by giving the guest > > control of de-asserting and unmasking the interrupt line. > > > > The IRQ bypass manager here is meant to provide the shim to connect > > interrupt producers, generally the host physical device driver, with > > interrupt consumers, generally the hypervisor, in order to configure > > these bypass mechanism. To do this, we base the connection on a > > shared, opaque token. For KVM-VFIO this is expected to be an > > eventfd_ctx since this is the connection we already use to connect an > > eventfd to an irqfd on the in-kernel path. When a producer and > > consumer with matching tokens is found, callbacks via both registered > > participants allow the bypass facilities to be automatically enabled. > > My Pi patches can work well based on this one and the one Eric sent > out earlier. Alex, what should we do in the next step to speed up the > upstreaming process? Hi Feng, Post the patches. Define how the update() callback is used. Help to address the issues I've outlined below. Thanks, Alex > > > > Signed-off-by: Alex Williamson > > Cc: Eric Auger > > --- > > > > This is the current draft of the IRQ bypass manager, I've made the > > following changes: > > > > - Incorporated Eric's extensions (I would welcome Sign-offs from all > > involved in the development, especially Eric - I've gone ahead and > > added Linaro copyright for the contributions so far) > > - Module support with module reference tracking > > - might_sleep() as suggested by Paolo > > - kerneldoc as suggested by Paolo > > - Renamed file s/bypass/irqbypass/ because a module named "bypass" > > is strange > > > > Issues: > > - The update() callback is defined but not used > > - We can't have *all* the callbacks be optional. I assume add/del > > are required > > - Naming consistency, stop is to start as suspend is to resume, not > > stop/resume > > - Callback descriptions including why we need separate stop/start > > hooks when it seems like the callee could reasonably assume such > > around the add/del callbacks > > - Need functional prototypes for both PI and forwarding > > > > include/linux/irqbypass.h | 75 ++++++++++++++++ > > kernel/irq/Kconfig | 3 + > > kernel/irq/Makefile | 1 > > kernel/irq/irqbypass.c | 206 > > +++++++++++++++++++++++++++++++++++++++++++++ > > 4 files changed, 285 insertions(+) > > create mode 100644 include/linux/irqbypass.h > > create mode 100644 kernel/irq/irqbypass.c > > > > diff --git a/include/linux/irqbypass.h b/include/linux/irqbypass.h > > new file mode 100644 > > index 0000000..cc7ce45 > > --- /dev/null > > +++ b/include/linux/irqbypass.h > > @@ -0,0 +1,75 @@ > > +/* > > + * IRQ offload/bypass manager > > + * > > + * Copyright (C) 2015 Red Hat, Inc. > > + * Copyright (c) 2015 Linaro Ltd. > > + * > > + * This program is free software; you can redistribute it and/or modify > > + * it under the terms of the GNU General Public License version 2 as > > + * published by the Free Software Foundation. > > + */ > > +#ifndef IRQBYPASS_H > > +#define IRQBYPASS_H > > + > > +#include > > + > > +struct irq_bypass_consumer; > > + > > +/** > > + * struct irq_bypass_producer - IRQ bypass producer definition > > + * @node: IRQ bypass manager private list management > > + * @token: opaque token to match between producer and consumer > > + * @irq: Linux IRQ number for the producer device > > + * @stop: > > + * @resume: > > + * @add_consumer: > > + * @del_consumer: > > + * > > + * The IRQ bypass producer structure represents an interrupt source for > > + * participation in possible host bypass, for instance an interrupt vector > > + * for a physical device assigned to a VM. > > + */ > > +struct irq_bypass_producer { > > + struct list_head node; > > + void *token; > > + int irq; /* linux irq */ > > + void (*stop)(struct irq_bypass_producer *); > > + void (*resume)(struct irq_bypass_producer *); > > + void (*add_consumer)(struct irq_bypass_producer *, > > + struct irq_bypass_consumer *); > > + void (*del_consumer)(struct irq_bypass_producer *, > > + struct irq_bypass_consumer *); > > +}; > > + > > +/** > > + * struct irq_bypass_consumer - IRQ bypass consumer definition > > + * @node: IRQ bypass manager private list management > > + * @token: opaque token to match between producer and consumer > > + * @stop: > > + * @resume: > > + * @add_consumer: > > + * @del_consumer: > > + * @update: > > + * > > + * The IRQ bypass consumer structure represents an interrupt sink for > > + * participation in possible host bypass, for instance a hypervisor may > > + * support offloads to allow bypassing the host entirely or offload > > + * portions of the interrupt handling to the VM. > > + */ > > +struct irq_bypass_consumer { > > + struct list_head node; > > + void *token; > > + void (*stop)(struct irq_bypass_consumer *); > > + void (*resume)(struct irq_bypass_consumer *); > > + void (*add_producer)(struct irq_bypass_consumer *, > > + struct irq_bypass_producer *); > > + void (*del_producer)(struct irq_bypass_consumer *, > > + struct irq_bypass_producer *); > > + void (*update)(struct irq_bypass_consumer *); > > +}; > > + > > +int irq_bypass_register_producer(struct irq_bypass_producer *); > > +void irq_bypass_unregister_producer(struct irq_bypass_producer *); > > +int irq_bypass_register_consumer(struct irq_bypass_consumer *); > > +void irq_bypass_unregister_consumer(struct irq_bypass_consumer *); > > +#endif /* IRQBYPASS_H */ > > diff --git a/kernel/irq/Kconfig b/kernel/irq/Kconfig > > index 9a76e3b..de5afe3 100644 > > --- a/kernel/irq/Kconfig > > +++ b/kernel/irq/Kconfig > > @@ -100,4 +100,7 @@ config SPARSE_IRQ > > > > If you don't know what to do here, say N. > > > > +config IRQ_BYPASS_MANAGER > > + tristate > > + > > endmenu > > diff --git a/kernel/irq/Makefile b/kernel/irq/Makefile > > index d121235..a6cfec7 100644 > > --- a/kernel/irq/Makefile > > +++ b/kernel/irq/Makefile > > @@ -7,3 +7,4 @@ obj-$(CONFIG_PROC_FS) += proc.o > > obj-$(CONFIG_GENERIC_PENDING_IRQ) += migration.o > > obj-$(CONFIG_PM_SLEEP) += pm.o > > obj-$(CONFIG_GENERIC_MSI_IRQ) += msi.o > > +obj-$(CONFIG_IRQ_BYPASS_MANAGER) += irqbypass.o > > diff --git a/kernel/irq/irqbypass.c b/kernel/irq/irqbypass.c > > new file mode 100644 > > index 0000000..3fd828d > > --- /dev/null > > +++ b/kernel/irq/irqbypass.c > > @@ -0,0 +1,206 @@ > > +/* > > + * IRQ offload/bypass manager > > + * > > + * Copyright (C) 2015 Red Hat, Inc. > > + * Copyright (c) 2015 Linaro Ltd. > > + * > > + * This program is free software; you can redistribute it and/or modify > > + * it under the terms of the GNU General Public License version 2 as > > + * published by the Free Software Foundation. > > + * > > + * Various virtualization hardware acceleration techniques allow bypassing > > + * or offloading interrupts received from devices around the host kernel. > > + * Posted Interrupts on Intel VT-d systems can allow interrupts to be > > + * received directly by a virtual machine. ARM IRQ Forwarding can allow > > + * level triggered device interrupts to be de-asserted directly by the VM. > > + * This manager allows interrupt producers and consumers to find each other > > + * to enable this sort of bypass. > > + */ > > + > > +#include > > +#include > > +#include > > +#include > > + > > +MODULE_LICENSE("GPL v2"); > > +MODULE_DESCRIPTION("IRQ bypass manager utility module"); > > + > > +static LIST_HEAD(producers); > > +static LIST_HEAD(consumers); > > +static DEFINE_MUTEX(lock); > > + > > +/* @lock must be held when calling connect */ > > +static void __connect(struct irq_bypass_producer *prod, > > + struct irq_bypass_consumer *cons) > > +{ > > + if (prod->stop) > > + prod->stop(prod); > > + if (cons->stop) > > + cons->stop(cons); > > + if (prod->add_consumer) > > + prod->add_consumer(prod, cons); > > + if (cons->add_producer) > > + cons->add_producer(cons, prod); > > + if (cons->resume) > > + cons->resume(cons); > > + if (prod->resume) > > + prod->resume(prod); > > +} > > + > > +/* @lock must be held when calling disconnect */ > > +static void __disconnect(struct irq_bypass_producer *prod, > > + struct irq_bypass_consumer *cons) > > +{ > > + if (prod->stop) > > + prod->stop(prod); > > + if (cons->stop) > > + cons->stop(cons); > > + if (cons->del_producer) > > + cons->del_producer(cons, prod); > > + if (prod->del_consumer) > > + prod->del_consumer(prod, cons); > > + if (cons->resume) > > + cons->resume(cons); > > + if (prod->resume) > > + prod->resume(prod); > > +} > > + > > +/** > > + * irq_bypass_register_producer - register IRQ bypass producer > > + * @producer: pointer to producer structure > > + * > > + * Add the provided IRQ producer to the list of producers and connect > > + * with any matching tokens found on the IRQ consumers list. > > + */ > > +int irq_bypass_register_producer(struct irq_bypass_producer *producer) > > +{ > > + struct irq_bypass_producer *tmp; > > + struct irq_bypass_consumer *consumer; > > + > > + might_sleep(); > > + > > + if (!try_module_get(THIS_MODULE)) > > + return -ENODEV; > > + > > + mutex_lock(&lock); > > + > > + list_for_each_entry(tmp, &producers, node) { > > + if (tmp->token == producer->token) { > > + mutex_unlock(&lock); > > + module_put(THIS_MODULE); > > + return -EINVAL; > > + } > > + } > > + > > + list_add(&producer->node, &producers); > > + > > + list_for_each_entry(consumer, &consumers, node) { > > + if (consumer->token == producer->token) { > > + __connect(producer, consumer); > > + break; > > + } > > + } > > + > > + mutex_unlock(&lock); > > + return 0; > > +} > > +EXPORT_SYMBOL_GPL(irq_bypass_register_producer); > > + > > +/** > > + * irq_bypass_unregister_producer - unregister IRQ bypass producer > > + * @producer: pointer to producer structure > > + * > > + * Remove a previously registered IRQ producer from the list of producers > > + * and disconnected from any connected IRQ consumers. > > + */ > > +void irq_bypass_unregister_producer(struct irq_bypass_producer *producer) > > +{ > > + struct irq_bypass_consumer *consumer; > > + > > + might_sleep(); > > + > > + mutex_lock(&lock); > > + > > + list_for_each_entry(consumer, &consumers, node) { > > + if (consumer->token == producer->token) { > > + __disconnect(producer, consumer); > > + break; > > + } > > + } > > + > > + list_del(&producer->node); > > + > > + mutex_unlock(&lock); > > + module_put(THIS_MODULE); > > +} > > +EXPORT_SYMBOL_GPL(irq_bypass_unregister_producer); > > + > > +/** > > + * irq_bypass_register_consumer - register IRQ bypass consumer > > + * @consumer: pointer to consumer structure > > + * > > + * Add the provided IRQ consumer to the list of consumers and connect > > + * with any matching tokens found on the IRQ producer list. > > + */ > > +int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer) > > +{ > > + struct irq_bypass_consumer *tmp; > > + struct irq_bypass_producer *producer; > > + > > + might_sleep(); > > + > > + if (!try_module_get(THIS_MODULE)) > > + return -ENODEV; > > + > > + mutex_lock(&lock); > > + > > + list_for_each_entry(tmp, &consumers, node) { > > + if (tmp->token == consumer->token) { > > + mutex_unlock(&lock); > > + module_put(THIS_MODULE); > > + return -EINVAL; > > + } > > + } > > + > > + list_add(&consumer->node, &consumers); > > + > > + list_for_each_entry(producer, &producers, node) { > > + if (producer->token == consumer->token) { > > + __connect(producer, consumer); > > + break; > > + } > > + } > > + > > + mutex_unlock(&lock); > > + return 0; > > +} > > +EXPORT_SYMBOL_GPL(irq_bypass_register_consumer); > > + > > +/** > > + * irq_bypass_unregister_consumer - unregister IRQ bypass consumer > > + * @consumer: pointer to consumer structure > > + * > > + * Remove a previously registered IRQ consumer from the list of consumers > > + * and disconnected from any connected IRQ producers. > > + */ > > +void irq_bypass_unregister_consumer(struct irq_bypass_consumer > > *consumer) > > +{ > > + struct irq_bypass_producer *producer; > > + > > + might_sleep(); > > + > > + mutex_lock(&lock); > > + > > + list_for_each_entry(producer, &producers, node) { > > + if (producer->token == consumer->token) { > > + __disconnect(producer, consumer); > > + break; > > + } > > + } > > + > > + list_del(&consumer->node); > > + > > + mutex_unlock(&lock); > > + module_put(THIS_MODULE); > > +} > > +EXPORT_SYMBOL_GPL(irq_bypass_unregister_consumer); > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/