Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Thu, 5 Apr 2018 00:17:39 +0200 (CEST)
From:   Thomas Gleixner <tglx@linutronix.de>
To:     Dexuan Cui <decui@microsoft.com>
cc:     'Greg KH' <gregkh@linuxfoundation.org>,
        "Michael Kelley (EOSG)" <Michael.H.Kelley@microsoft.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        KY Srinivasan <kys@microsoft.com>,
        Stephen Hemminger <sthemmin@microsoft.com>,
        Vitaly Kuznetsov <vkuznets@redhat.com>
Subject: Re: Any standard kernel API to dynamically allocate/free per-cpu
 vectors on x86?
In-Reply-To: <KL1P15301MB000638051CC8DC7FD2A2E74BBFA40@KL1P15301MB0006.APCP153.PROD.OUTLOOK.COM>
Message-ID: <alpine.DEB.2.21.1804050004230.1492@nanos.tec.linutronix.de>
References: <KL1P15301MB000638051CC8DC7FD2A2E74BBFA40@KL1P15301MB0006.APCP153.PROD.OUTLOOK.COM>
User-Agent: Alpine 2.21 (DEB 202 2017-01-01)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Dexuan,

On Wed, 4 Apr 2018, Dexuan Cui wrote:

> Hi,
> Recently, two Hyper-V specific vectors were introduced in
> arch/x86/include/asm/irq_vectors.h:
> 
> #if IS_ENABLED(CONFIG_HYPERV)
> #define HYPERV_REENLIGHTENMENT_VECTOR   0xee
> #define HYPERV_STIMER0_VECTOR           0xed
> #endif
> 
> What if we need more such vectors on Hyper-V? This static global reservation
> mechanism doesn't scale, IMO.
> 
> Actually I believe we don't really need to reserve the same vector on all CPUs,
> because the model-specific registers (MSRs) defined for Hyper-V synthetic 
> interrupt controller (SynIC) are per-cpu, meaning each virtual processor has
> its own copy of these registers, so they can be programmed independently.
> 
> That is to say, we can use different vectors for the same Synthetic Interrupt,
> as long as we're able to dynamically get a per-cpu vector for each CPU.
> 
> I'm wondering if there is such a standard kernel API on x86. 
> As I skimmed through the code, I haven't found it yet, if any.
> 
> PS, the background of my question is that: in the future Hyper-V will
> support multiple VMBus drivers running simultaneously in one virtual
> machine, and I think it would be better for the future secondary VMBus
> driver to use a different synthetic interrupt, which I hope can be
> configured with dynamic vectors rather than a new static globally-reserved
> vector.

We don't have a simple way to do such allocations because they involve IDT
entry manipulation.

If there is no hard requirement that this stuff runs through an direct
vector, then we could simply make these interrupts go through the normal
interrupt path. That means you have to request them like regular device
interrupts and the handling path is slightly longer than the direct vector
mode. You'd get virtual interrupt numbers per cpu which also show up in
/proc/interrupts as separate lines.

That needs a very simple and minimal virtual interrupt controller driver
which is mostly a dummy implementation except for the activation function
which would allow you to retrieve the vector number and store it in the
MSR.

There are a few details to be hashed out vs. CPU hotplug, but we have all
the infrastructure in place to deal with that.

Hmm?

Thanks,

	tglx