Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Thomas Gleixner <tglx@linutronix.de>
To:     Angus Chen <angus.chen@jaguarmicro.com>,
        "Michael S. Tsirkin" <mst@redhat.com>
Cc:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Ming Lei <ming.lei@redhat.com>,
        Jason Wang <jasowang@redhat.com>,
        Christoph Hellwig <hch@lst.de>, Jens Axboe <axboe@kernel.dk>
Subject: RE: IRQ affinity problem from virtio_blk
In-Reply-To: <TY2PR06MB3424E673A1374CAC37E647CA85079@TY2PR06MB3424.apcprd06.prod.outlook.com>
References: <TY2PR06MB3424CB11DB57CA1FAA16F10D85049@TY2PR06MB3424.apcprd06.prod.outlook.com>
 <87v8nfrhbw.ffs@tglx> <20221115174152-mutt-send-email-mst@kernel.org>
 <87sfijrf9o.ffs@tglx> <87o7t7rec7.ffs@tglx>
 <20221115183339-mutt-send-email-mst@kernel.org>
 <TY2PR06MB34241333364AA5843CD4BAA885079@TY2PR06MB3424.apcprd06.prod.outlook.com>
 <87fsejqicd.ffs@tglx>
 <TY2PR06MB3424E673A1374CAC37E647CA85079@TY2PR06MB3424.apcprd06.prod.outlook.com>
Date:   Wed, 16 Nov 2022 14:27:11 +0100
Message-ID: <877czvqbbk.ffs@tglx>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: bulk

On Wed, Nov 16 2022 at 11:24, Angus Chen wrote:
>> >> On Wed, Nov 16, 2022 at 12:24:24AM +0100, Thomas Gleixner wrote:
>> > Any other information I need to provide,pls tell me.
>> 
>> A sensible use case for 180+ virtio block devices in a single guest.
>> 
> Our card can provide more than 512 virtio_blk devices .
> one virtio_blk passthrough to one container,like docker.

I'm not sure whether that's sensible, but that's how your hardware is
designed. You could have provided this information upfront instead of
random memory dumps of the irq matrix internals.

> So we need so much devices.
> In the first patch ,I del the IRQD_AFFINITY_MANAGED in virtio_blk .

There is no IRQD_AFFINITY_MANAGED in virtio_blk. That flag is internal
to the interrupt core code and you can neither delete it nor fiddle with
it from inside virtio_blk.

You can do that in your private kernel, but that's not an option for
mainline as it will break existing setups and it's fundamentally wrong.

The block-mq code has assumptions about the semantics of managed
interrupts. It happens to work for the single queue case because that
always ends up with queue affinity == cpu_possible_mask.

For anything else which assigns the queues to partitions of the CPU
space it definitely expects the semantics of managed interrupts.

> As you know, if we just use small queues number ,like 1or 2,we Still
> occupy 80 vector ,that is kind of waste,and it is easy to eahausted
> the Irq resource.

We know that by now. No point in repeating this over and over. Aside of
that it's not that easy because this is the first time within 5 years
that someone ran into this problem.

The real question is how to solve this proper without creating problems
for other scenarios. That needs involvment of the blk-mq people.

Thanks,

        tglx