Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Fri, 3 Jan 2020 19:29:08 +0800
From:   Ming Lei <ming.lei@redhat.com>
To:     John Garry <john.garry@huawei.com>
Cc:     Marc Zyngier <maz@kernel.org>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "chenxiang (M)" <chenxiang66@hisilicon.com>,
        "bigeasy@linutronix.de" <bigeasy@linutronix.de>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "hare@suse.com" <hare@suse.com>, "hch@lst.de" <hch@lst.de>,
        "axboe@kernel.dk" <axboe@kernel.dk>,
        "bvanassche@acm.org" <bvanassche@acm.org>,
        "peterz@infradead.org" <peterz@infradead.org>,
        "mingo@redhat.com" <mingo@redhat.com>,
        Zhang Yi <yi.zhang@redhat.com>
Subject: Re: [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity
 for managed interrupt
Message-ID: <20200103112908.GA20353@ming.t460p>
References: <20191220233138.GB12403@ming.t460p>
 <fffcd23dd8286615b6e2c99620836cb1@www.loen.fr>
 <d5774e2f-bb60-c27c-bf00-267b88400a12@huawei.com>
 <e815b5451ea86e99d42045f7067f455a@www.loen.fr>
 <20191224015926.GC13083@ming.t460p>
 <7a961950624c414bb9d0c11c914d5c62@www.loen.fr>
 <20191225004822.GA12280@ming.t460p>
 <72a6a738-f04b-3792-627a-fbfcb7b297e1@huawei.com>
 <20200103004625.GA5219@ming.t460p>
 <2b070d25-ee35-aa1f-3254-d086c6b872b1@huawei.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <2b070d25-ee35-aa1f-3254-d086c6b872b1@huawei.com>
User-Agent: Mutt/1.12.1 (2019-06-15)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Fri, Jan 03, 2020 at 10:41:48AM +0000, John Garry wrote:
> On 03/01/2020 00:46, Ming Lei wrote:
> > > > > d the
> > > > > DMA API more than an architecture-specific problem.
> > > > > 
> > > > > Given that we have so far very little data, I'd hold off any conclusion.
> > > > We can start to collect latency data of dma unmapping vs nvme_irq()
> > > > on both x86 and arm64.
> > > > 
> > > > I will see if I can get a such box for collecting the latency data.
> > > To reiterate what I mentioned before about IOMMU DMA unmap on x86, a key
> > > difference is that by default it uses the non-strict (lazy) mode unmap, i.e.
> > > we unmap in batches. ARM64 uses general default, which is strict mode, i.e.
> > > every unmap results in an IOTLB fluch.
> > > 
> > > In my setup, if I switch to lazy unmap (set iommu.strict=0 on cmdline), then
> > > no lockup.
> > > 
> > > Are any special IOMMU setups being used for x86, like enabling strict mode?
> > > I don't know...
> > BTW, I have run the test on one 224-core ARM64 with one 32-hw_queue NVMe, the
> > softlock issue can be triggered in one minute.
> > 
> > nvme_irq() often takes ~5us to complete on this machine, then there is really
> > risk of cpu lockup when IOPS is > 200K.
> 
> Do you have a typical nvme_irq() completion time for a mid-range x86 server?

~1us.

It is done via bcc script, and ebpf itself may introduce some overhead.

> 
> > 
> > The soft lockup can be triggered too if 'iommu.strict=0' is passed in,
> > just takes a bit longer by starting more IO jobs.
> > 
> > In above test, I submit IO to one single NVMe drive from 4 CPU cores via 8 or
> > 12 jobs(iommu.strict=0), meantime make the nvme interrupt handled just in one
> > dedicated CPU core.
> 
> Well a problem with so many CPUs is that it does not scale (well) with MQ
> devices, like NVMe.
> 
> As CPU count goes up, device queue count doesn't and we get more contention.

The problem is worse on ARM64 system, in which there are more CPU cores,
and each single CPU core is often slower than x86's. Meantime each
hardware interrupt has to be handled on single CPU target.

Also the storage device(such as NVMe) itself should be same for both
from performance viewpoint.

> 
> > 
> > Is there lock contention among iommu dma map and unmap callback?
> 
> There would be the IOVA management, but that would be common to x86. Each
> CPU keeps an IOVA cache, and there is a central pool of cached IOVAs, so
> that reduces any contention, unless the caches are exhausted.
> 
> I think most contention/bottleneck is at the SMMU HW interface, which has a
> single queue interface.

Not sure if it is related with single queue interface, given my test just
uses single hw queue by pushing several CPU cores to submit IO and
handling the single queue's interrupt on one dedicated CPU core.


Thanks,
Ming