Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 15 Nov 2022 17:44:41 -0500
From:   "Michael S. Tsirkin" <mst@redhat.com>
To:     Thomas Gleixner <tglx@linutronix.de>
Cc:     Angus Chen <angus.chen@jaguarmicro.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Ming Lei <ming.lei@redhat.com>,
        Jason Wang <jasowang@redhat.com>
Subject: Re: IRQ affinity problem from virtio_blk
Message-ID: <20221115174152-mutt-send-email-mst@kernel.org>
References: <TY2PR06MB3424CB11DB57CA1FAA16F10D85049@TY2PR06MB3424.apcprd06.prod.outlook.com>
 <87v8nfrhbw.ffs@tglx>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87v8nfrhbw.ffs@tglx>
Precedence: bulk

Thanks Thomas, I have a question:

On Tue, Nov 15, 2022 at 11:19:47PM +0100, Thomas Gleixner wrote:
> On Tue, Nov 15 2022 at 03:40, Angus Chen wrote:
> > Before probe one virtio_blk.
> > crash_cts> p *vector_matrix
> > $44 = {
> >   matrix_bits = 256,
> >   alloc_start = 32,
> >   alloc_end = 236,
> >   alloc_size = 204,
> >   global_available = 15354,
> >   global_reserved = 154,
> >   systembits_inalloc = 3,
> >   total_allocated = 411,
> >   online_maps = 80,
> >   maps = 0x46100,
> >   scratch_map = {1160908723191807, 0, 1, 18435222497520517120},
> >   system_map = {1125904739729407, 0, 1, 18435221191850459136}
> > }
> > After probe one virtio_blk.
> > crash_cts> p *vector_matrix
> > $45 = {
> >   matrix_bits = 256,
> >   alloc_start = 32,
> >   alloc_end = 236,
> >   alloc_size = 204,
> >   global_available = 15273,
> >   global_reserved = 154,
> >   systembits_inalloc = 3,
> >   total_allocated = 413,
> >   online_maps = 80,
> >   maps = 0x46100,
> >   scratch_map = {25769803776, 0, 0, 14680064},
> >   system_map = {1125904739729407, 0, 1, 18435221191850459136}
> > }
> >
> > We can see global_available drop from 15354 to 15273, is 81.
> > And the total_allocated increase from 411 to 413. One config irq,and
> > one vq irq.
> 
> Right. That's perfectly fine. At the point where you looking at it, the
> matrix allocator has given out 2 vectors as can be seen via
> total_allocated.
> 
> But then it also has another 79 vectors put aside for the other queues,


What makes it put these vectors aside? pci_alloc_irq_vectors_affinity ?


> but those queues have not yet requested the interrupts so there is no
> allocation yet. But the vectors are guaranteed to be available when
> request_irq() for those queues runs, which does the actual allocation.
> 
> Btw, you can enable CONFIG_GENERIC_IRQ_DEBUGFS and then look at the
> content of /sys/kernel/debug/irq/domain/VECTOR which gives you a very
> clear picture of what's going on. No need for gdb.
> 
> > It is easy to expend the irq resource ,because virtio_blk device could
> > be more than 512.
> 
> How so? virtio_blk allocates a config interrupt and one queue interrupt
> per CPU. So in your case a total of 81.
> 
> How would you exhaust the vector space? Each CPU has about ~200 (in your
> case exactly 204) vectors which can be handed out to devices. You'd need
> to instantiate about 200 virtio_blk devices to get to the point of
> vector exhaustion.
> 
> So what are you actually worried about and which problem are you trying
> to solve?
> 
> Thanks,
> 
>         tglx
>