Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 14 Jun 2022 16:47:35 -0700
From:   Keith Busch <kbusch@kernel.org>
To:     Bjorn Helgaas <helgaas@kernel.org>
Cc:     "Guilherme G. Piccoli" <gpiccoli@igalia.com>,
        Hans de Goede <hdegoede@redhat.com>,
        "Rafael J . Wysocki" <rafael@kernel.org>,
        Mika Westerberg <mika.westerberg@linux.intel.com>,
        Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= <kw@linux.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Myron Stowe <myron.stowe@redhat.com>,
        Juha-Pekka Heikkila <juhapekka.heikkila@gmail.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        "H . Peter Anvin" <hpa@zytor.com>,
        Benoit =?iso-8859-1?Q?Gr=E9goire?= <benoitg@coeus.ca>,
        Hui Wang <hui.wang@canonical.com>, linux-acpi@vger.kernel.org,
        linux-pci@vger.kernel.org, x86@kernel.org,
        linux-kernel@vger.kernel.org, Jens Axboe <axboe@fb.com>,
        Christoph Hellwig <hch@lst.de>,
        Sagi Grimberg <sagi@grimberg.me>,
        linux-nvme@lists.infradead.org
Subject: Re: [PATCH] x86/PCI: Revert: "Clip only host bridge windows for E820
 regions"
Message-ID: <YqkeF2uqAyyxiZrQ@kbusch-mbp.dhcp.thefacebook.com>
References: <6eae37ce-dd44-9c32-3f68-2b4e102dce8e@igalia.com>
 <20220614230128.GA901712@bhelgaas>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220614230128.GA901712@bhelgaas>
Precedence: bulk

On Tue, Jun 14, 2022 at 06:01:28PM -0500, Bjorn Helgaas wrote:
> [+cc NVMe folks]
> 
> On Tue, Jun 14, 2022 at 07:49:27PM -0300, Guilherme G. Piccoli wrote:
> > On 14/06/2022 12:47, Hans de Goede wrote:
> > > [...]
> > > 
> > > Have you looked at the log of the failed boot in the Steam Deck kernel
> > > bugzilla? Everything there seems to work just fine and then the system
> > > just hangs. I think that maybe it cannot find its root disk, so maybe
> > > an NVME issue ?
> > 
> > *Exactly* that - NVMe device is the root disk, it cannot boot since the
> > device doesn't work, hence no rootfs =)
> 
> Beginning of thread: https://lore.kernel.org/r/20220612144325.85366-1-hdegoede@redhat.com
> 
> Steam Deck broke because we erroneously trimmed out the PCI host
> bridge window where BIOS had placed most devices, successfully
> reassigned all the PCI bridge windows and BARs, but some devices,
> apparently including NVMe, didn't work at the new addresses.
> 
> Do you NVMe folks know of gotchas in this area?  I want to know
> because we'd like to be able to move devices around someday to make
> room for hot-added devices.
> 
> This reassignment happened before drivers claimed the devices, so from
> a PCI point of view, I don't know why the NVMe device wouldn't work at
> the new address.

The probe status quickly returns ENODEV. Based on the output (we don't log
much, so this is just an educated guesss), I think that means the driver read
all F's from the status register, which indicates we can't read it when using
the reassigned memory window.

Why changing memory windows may not work tends to be platform or device
specific. Considering the renumbered windows didn't cause a problem for other
devices, it sounds like this nvme device may be broken.