Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 28 Apr 2022 13:55:56 -0500
From:   Bjorn Helgaas <helgaas@kernel.org>
To:     "Maciej W. Rozycki" <macro@orcam.me.uk>
Cc:     Palmer Dabbelt <palmer@dabbelt.com>,
        Bjorn Helgaas <bhelgaas@google.com>, linux-pci@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH] PCI: Avoid handing out address 0 to devices
Message-ID: <20220428185556.GA35255@bhelgaas>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <alpine.DEB.2.21.2204192214310.9383@angie.orcam.me.uk>
Precedence: bulk

On Wed, Apr 27, 2022 at 11:18:12PM +0100, Maciej W. Rozycki wrote:
> On Mon, 18 Apr 2022, Bjorn Helgaas wrote:
> ...

> > Sparc uses the MMIO I/O port address directly in the struct resource,
> > so it will never try to allocate [io 0x0000], and there's no problem
> > with using PCI I/O port 0:
> > 
> >   pci_bus 0000:00: root bus resource [io  0x804000000000-0x80400fffffff] (bus address [0x0000-0xfffffff])
> >   mpt2sas0: ioport(0x0000804000001000), size(256)
> > 
> > The sparc ioport interfaces are basically:
> > 
> >   ioport_map(port)  { return port; }
> >   ioread8(addr)     { return readb(addr); }
> >   inb(addr)         { return readb(addr); }
> > 
> > RISC-V uses the generic ones, i.e.,
> > 
> >   ioport_map(port)  { return PIO_OFFSET + port; }
> >   ioread8(addr)     { if (addr) >= PIO_RESERVED)
> >                         return readb(addr);
> >                       else
> >                         return inb(addr & PIO_MASK); }
> >   inb(addr)         { return __raw_readb(PCI_IOBASE + addr); }
> > 
> > Obviously RISC-V gives you prettier I/O port addresses, but at the
> > cost of having PCI I/O port 0 be 0 in the struct resource as well,
> > which makes it basically unusable.  Is it worth it?
> 
>  Well, the SPARC port may be able to get away with that, but overall I 
> think using PCI bus addresses for port I/O resources is the only sane 
> thing to do.

That only works if you have a single host bridge where you care about
I/O ports, or if you're willing to split up the single space across
the multiple host bridges, e.g., [io 0x0000-0x7fff] to one host
bridge, [0x8000-0xffff] to another.

> In fact I think for MMIO resources we probably ought to do 
> the same, though it may be actually more difficult to implement, because 
> it's more likely there are systems out there with multiple per-bus MMIO 
> address spaces.

I might be missing your point here, but multiple host bridges are
common on ia64, sparc, and (I think) powerpc.  It probably would be
feasible to make the struct resource address identical to the PCI bus
address for 64-bit MMIO space because they're both constrained to the
same-sized space.

But doing it for the PCI 32-bit non-prefetchable space would be an
issue because there's usually RAM in that area of CPU address space,
so we'd have to reserve a hole for PCI, put the MMIO in the hole, then
split the MMIO space across all the host bridges.  This sparc system
has 16 host bridges, so even if we had no hole, each one could only
have 256MB of identity-mapped non-prefetchable space:

  pci_bus 0000:00: root bus resource [mem 0x800000000000-0x80007effffff] (bus address [0x00000000-0x7effffff])
  pci_bus 0001:00: root bus resource [mem 0x801000000000-0x80107effffff] (bus address [0x00000000-0x7effffff])
  ...
  pci_bus 000f:00: root bus resource [mem 0x839000000000-0x83907effffff] (bus address [0x00000000-0x7effffff])

(This is from https://bugzilla.kernel.org/show_bug.cgi?id=96241)

>  The reason why I think using bus addresses here is the right thing to do 
> is that systems may have multiple decode windows exposed to the CPU(s) 
> mapping to the same I/O bus addresses, often for specific purposes. E.g. 
> Alpha has the sparse and the dense address space and some systems (I have 
> one with a MIPS CPU) have spaces with different endianness swap policies 
> (for matching either bit or byte lanes) wired in the bus logic hardware.  
> So the same port I/O location can be mapped at different MMIO addresses 
> simultaneously in one system.

Is this feature used by Linux?  I guess it would require some generic
mechanism drivers could use to get the desired endianness?  I don't
know whether such a thing exists.

Bjorn