Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 22 Jun 2022 19:51:26 -0700
From:   Jerry Snitselaar <jsnitsel@redhat.com>
To:     Baolu Lu <baolu.lu@linux.intel.com>
Cc:     Steve Wahl <steve.wahl@hpe.com>, Joerg Roedel <jroedel@suse.de>,
        Kyung Min Park <kyung.min.park@intel.com>,
        David Woodhouse <dwmw2@infradead.org>,
        Will Deacon <will@kernel.org>,
        iommu <iommu@lists.linux-foundation.org>,
        Kevin Tian <kevin.tian@intel.com>,
        Mike Travis <mike.travis@hpe.com>,
        Dimitri Sivanich <sivanich@hpe.com>,
        Russ Anderson <russ.anderson@hpe.com>,
        Linux List Kernel Mailing <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v3] iommu/vt-d: Make DMAR_UNITS_SUPPORTED a config setting
Message-ID: <20220623025126.ld45k72c2okodvvk@cantor>
References: <9d6177ac-802f-eb11-4307-b0e49d8126b5@linux.intel.com>
 <20220615183650.32075-1-steve.wahl@hpe.com>
 <96984350-3a04-812a-60f1-a09138014fac@linux.intel.com>
 <CALzcdduhpOebY18hiOpMcXLcH83bY_u4fZT09pBcoz2ObqXnGg@mail.gmail.com>
 <f0e4adc8-5d67-b76a-d0f1-2df83bd69a82@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <f0e4adc8-5d67-b76a-d0f1-2df83bd69a82@linux.intel.com>
Precedence: bulk

On Thu, Jun 23, 2022 at 10:29:35AM +0800, Baolu Lu wrote:
> On 2022/6/22 23:05, Jerry Snitselaar wrote:
> > On Wed, Jun 22, 2022 at 7:52 AM Baolu Lu<baolu.lu@linux.intel.com>  wrote:
> > > On 2022/6/16 02:36, Steve Wahl wrote:
> > > > To support up to 64 sockets with 10 DMAR units each (640), make the
> > > > value of DMAR_UNITS_SUPPORTED adjustable by a config variable,
> > > > CONFIG_DMAR_UNITS_SUPPORTED, and make it's default 1024 when MAXSMP is
> > > > set.
> > > > 
> > > > If the available hardware exceeds DMAR_UNITS_SUPPORTED (previously set
> > > > to MAX_IO_APICS, or 128), it causes these messages: "DMAR: Failed to
> > > > allocate seq_id", "DMAR: Parse DMAR table failure.", and "x2apic: IRQ
> > > > remapping doesn't support X2APIC mode x2apic disabled"; and the system
> > > > fails to boot properly.
> > > > 
> > > > Signed-off-by: Steve Wahl<steve.wahl@hpe.com>
> > > > Reviewed-by: Kevin Tian<kevin.tian@intel.com>
> > > > ---
> > > > 
> > > > Note that we could not find a reason for connecting
> > > > DMAR_UNITS_SUPPORTED to MAX_IO_APICS as was done previously.  Perhaps
> > > > it seemed like the two would continue to match on earlier processors.
> > > > There doesn't appear to be kernel code that assumes that the value of
> > > > one is related to the other.
> > > > 
> > > > v2: Make this value a config option, rather than a fixed constant.  The default
> > > > values should match previous configuration except in the MAXSMP case.  Keeping the
> > > > value at a power of two was requested by Kevin Tian.
> > > > 
> > > > v3: Make the config option dependent upon DMAR_TABLE, as it is not used without this.
> > > > 
> > > >    drivers/iommu/intel/Kconfig | 7 +++++++
> > > >    include/linux/dmar.h        | 6 +-----
> > > >    2 files changed, 8 insertions(+), 5 deletions(-)
> > > > 
> > > > diff --git a/drivers/iommu/intel/Kconfig b/drivers/iommu/intel/Kconfig
> > > > index 39a06d245f12..07aaebcb581d 100644
> > > > --- a/drivers/iommu/intel/Kconfig
> > > > +++ b/drivers/iommu/intel/Kconfig
> > > > @@ -9,6 +9,13 @@ config DMAR_PERF
> > > >    config DMAR_DEBUG
> > > >        bool
> > > > 
> > > > +config DMAR_UNITS_SUPPORTED
> > > > +     int "Number of DMA Remapping Units supported"
> > > > +     depends on DMAR_TABLE
> > > > +     default 1024 if MAXSMP
> > > > +     default 128  if X86_64
> > > > +     default 64
> > > With this patch applied, the IOMMU configuration looks like:
> > > 
> > > [*]   AMD IOMMU support
> > > <M>     AMD IOMMU Version 2 driver
> > > [*]     Enable AMD IOMMU internals in DebugFS
> > > (1024) Number of DMA Remapping Units supported   <<<< NEW
> > > [*]   Support for Intel IOMMU using DMA Remapping Devices
> > > [*]     Export Intel IOMMU internals in Debugfs
> > > [*]     Support for Shared Virtual Memory with Intel IOMMU
> > > [*]     Enable Intel DMA Remapping Devices by default
> > > [*]     Enable Intel IOMMU scalable mode by default
> > > [*]   Support for Interrupt Remapping
> > > [*]   OMAP IOMMU Support
> > > [*]     Export OMAP IOMMU internals in DebugFS
> > > [*]   Rockchip IOMMU Support
> > > 
> > > The NEW item looks confusing. It looks to be a generic configurable
> > > value though it's actually Intel DMAR specific. Any thoughts?
> > > 
> > > Best regards,
> > > baolu
> > > 
> > Would moving it under INTEL_IOMMU at least have it show up below
> > "Support for Intel IOMMU using DMA Remapping Devices"? I'm not sure it
> > can be better than that, because IRQ_REMAP selects DMAR_TABLE, so we
> > can't stick it in the if INTEL_IOMMU section.
> 
> It's more reasonable to move it under INTEL_IOMMU, but the trouble is
> that this also stands even if INTEL_IOMMU is not configured.

My thought only was with it after the 'config INTEL_IOMMU' block and before 'if INTEL_IOMMU'
it would show up like:

[*]   Support for Intel IOMMU using DMA Remapping Devices
(1024) Number of DMA Remapping Units supported   <<<< NEW

> 
> The real problem here is that the iommu sequence ID overflows if
> DMAR_UNITS_SUPPORTED is not big enough. This is purely a software
> implementation issue, I am not sure whether user opt-in when building a
> kernel package could help a lot here.
> 

Is this something that could be figured out when parsing the dmar
table? It looks like currently iommu_refcnt[], iommu_did[], and
dmar_seq_ids[] depend on it.

Regards,
Jerry


> If we can't find a better way, can we just step back?
> 
> Best regards,
> baolu