Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 5 Jan 2023 15:38:40 -0800
From:   Dan Williams <dan.j.williams@intel.com>
To:     Bjorn Helgaas <helgaas@kernel.org>,
        "Liang, Kan" <kan.liang@linux.intel.com>
CC:     <bhelgaas@google.com>, <hdegoede@redhat.com>,
        <kernelorg@undead.fr>, <kjhambrick@gmail.com>, <2lprbe78@duck.com>,
        <nicholas.johnson-opensource@outlook.com.au>, <benoitg@coeus.ca>,
        <mika.westerberg@linux.intel.com>, <wse@tuxedocomputers.com>,
        <mumblingdrunkard@protonmail.com>, <linux-pci@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <david.e.box@intel.com>,
        <yunying.sun@intel.com>, Tony Luck <tony.luck@intel.com>,
        Dan Williams <dan.j.williams@intel.com>, <dave.jiang@intel.com>
Subject: Re: Bug report: the extended PCI config space is missed with 6.2-rc2
Message-ID: <63b75f8083b79_c81f02949b@dwillia2-xfh.jf.intel.com.notmuch>
References: <ac2693d8-8ba3-72e0-5b66-b3ae008d539d@linux.intel.com>
 <20230105223257.GA1177387@bhelgaas>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230105223257.GA1177387@bhelgaas>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?/8MyZJFtYCHr/xiVsYrLuJ7uud4T9p61PprO7eMISAdavycktAzzcLgLy3/3?=
 =?us-ascii?Q?ueEpqwKjOy23saT3z6Tw2/o0Ly5S4f3G8ym1+mwJovBh66dMMLRrOA0F6/ud?=
 =?us-ascii?Q?IRQOtJaxGb6D8n6gnPPnzh0gZiD8flJ9FXQ7gss9Gn2SupU9iqw3s/c13eXE?=
 =?us-ascii?Q?7Wu8lVG4sUVze2tc+DlE39rWWLJ3oTusRrb9BcWIbzEQ6OzC9DH2V1xCaMjg?=
 =?us-ascii?Q?X5P03ImIWDifgzoeNu3FDnwOe//sDKHsTCVXVAHqpvV5Zc++/B+nfUrbzhAx?=
 =?us-ascii?Q?MIoKI20KdIo2OJFTshXvRyeOj8xsFE2JSOsK7O79EluYJ3U4xvH61J/f1+Fl?=
 =?us-ascii?Q?Mf35hBVgG7kK4r45equnOmzUaT35ngOkxGaQOxGlv4bUWAG0e/yiXgQsNBK+?=
 =?us-ascii?Q?DnUlMNoJ1zjZuoWDwpi8gANTYh93+dByk8BAVy7Een31VeSlgOAeEEKWXCQr?=
 =?us-ascii?Q?MLrc3b1gf2yqXsQwuo1M+G5+7HVGK9MeHokgiRVCzhU05cK4aWoqEE2Y8etH?=
 =?us-ascii?Q?OAz6H5RPU6YXvX/n+XGFly/hcSJ9B0VJzsH1xSRsqB5b3TABrwpdRC307l7c?=
 =?us-ascii?Q?9AUPt2fZuDgN/zE1Q/j1Q1MtsMjrwfWatyFf2vl5ocqNOTNJqiGk0ETygdwe?=
 =?us-ascii?Q?1MTsYixaasxzzEJe8PCK+GUAs/lWSC8dtT//R3rXmLteyfLwQVH2fif7LR7/?=
 =?us-ascii?Q?TZ9jNsHArdTY/hhIBJ5ttvYH6f4RrAXJVZmZh3DtmsfI86lWhbb05q8onKb3?=
 =?us-ascii?Q?eiwI7yRV4wwa3zPu1/fyX70T9Ez25B9C0gxApE6tl3y6SWoT9q90XDSVAuqF?=
 =?us-ascii?Q?PdA++lC7RgNemZqowqGESzye76/XKnVlHerRGXiQzTA8wiXSmSeIxGo/0K6H?=
 =?us-ascii?Q?Mqt534Esl6U39ou/fBfoMJm1vVxh+dI6Mqum6fFjacveJCG+vzth9sTowGKr?=
 =?us-ascii?Q?JEPunBp0EiALOV6STIcf8B+ZF9HS6tFuHvJfuW80Ha3LsJhfl37V+nE0Eqmb?=
 =?us-ascii?Q?ekeC+O5mMhPnV5sks3dn31VFn39DmqS6mmK5rQGxEsNNO5n2RFMJLDMEtVF7?=
 =?us-ascii?Q?fC60DOdqvX3Qsmc/u+/ZjkUPruPoGB+xCcHMW76iJc/LIfGyTsyvyukEkzOO?=
 =?us-ascii?Q?yVtOGnYRTi+sRrlMSeFaRFRUqnM4yyUOZN/3ng8M0/rn4g4XaW950NSEv6wr?=
 =?us-ascii?Q?QMZhPvaoroK8giDTA+iv7HKQS6nthpqJQePMtcEaHmRhIJgsAa0MzeOWVLCV?=
 =?us-ascii?Q?Xk/y5NAcit0n+ZCsHJPkk/TF3xbqyARR28HGc5Gz5uKuhlPX2cVV1vXHp9ph?=
 =?us-ascii?Q?4Dd6cbbdH7sALvWEhKwwSxSKzIM7j+Kf4wH5p8Znz0NrwfIi3hiO9H/OYFWX?=
 =?us-ascii?Q?JrFrF7qmhOSxNrBkrzCjFaKtPRHpGzW0nLVwk3ae+o824Z3yRmdqAwlP/l7E?=
 =?us-ascii?Q?dzDEvVYSSC8+1Mtw67WXTLCnpyBuHQUwKpQgV9rtif3Ukab/B/aBwf1BfKy/?=
 =?us-ascii?Q?eQU8pMd9zRJsT//2+IFZOuDsorIR1mAlEqMYzKRM4ds3cWMCF/YC5uOX+iFx?=
 =?us-ascii?Q?X232g9q5Rcbi7/d/5e5szAEisBja6uSWEJ/c+yNwh3KfpFSTOHvnOVd3cbJ4?=
 =?us-ascii?Q?sg=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 55255d4d-0750-4f6c-5a07-08daef75fb43
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jan 2023 23:38:43.3616
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: RWKqpEr4XeKHC0p9V5cWfrN3rQftaVKVS+FHk8Q9vr3C1BIYPla4i9YI2g1SHk2ifqbaoC6kTbQQ/d1rDMPnvh3OML45Blp3rW7ElsEGC+E=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB4871
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Bjorn Helgaas wrote:
> [+cc Tony, Dan]
> 
> On Wed, Jan 04, 2023 at 09:39:56AM -0500, Liang, Kan wrote:
> > Hi Bjorn,
> > 
> > Happy new year!
> > 
> > We found some PCI issues with the latest 6.2-rc2.
> > 
> > - Using the lspci -xxxx, the extended PCI config space of all PCI
> > devices are missed with the latest 6.2-rc2. The system we used had 932
> > PCI devices, at least 800 which have extended space as seen when booted
> > into a 5.15 kernel. But none of them appeared in 6.2-rc2.
> > - The drivers which rely on the information in the extended PCI config
> > space don't work anymore. We have confirmed that the perf uncore driver
> > (uncore performance monitoring) and Intel VSEC driver (telemetry) don't
> > work in 6.2-rc2. There could be more drivers which are impacted.
> > 
> > After a bisect, we found the regression is caused by the below commit
> > 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map").
> > After reverting the commit, the issues are gone.
> 
> Can you try this patch (based on v6.2-rc1):

Looks good to me, one question below, but either way:

Reviewed-by: Dan Williams <dan.j.williams@intel.com>

...and Dave, who reported that CXL enumeration was busted in -rc2, says
this patch fixes that. So you can also add:

Tested-by: Dave Jiang <dave.jiang@intel.com>

> 
> 
> commit 89a0067217b0 ("x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space")
> parent 1b929c02afd3
> Author: Bjorn Helgaas <bhelgaas@google.com>
> Date:   Thu Jan 5 16:02:58 2023 -0600
> 
>     x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
>     
>     Normally we reject ECAM space unless it is reported as reserved in the E820
>     table or via a PNP0C02 _CRS method (PCI Firmware, r3.3, sec 4.1.2).  This
>     means extended config space (offsets 0x100-0xfff) may not be accessible.
>     
>     Some firmware doesn't report ECAM space via PNP0C02 _CRS methods, but does
>     mention it as an EfiMemoryMappedIO region via EFI GetMemoryMap(), which is
>     normally converted to an E820 entry by a bootloader or EFI stub.
>     
>     07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes
>     E820 entries that correspond to EfiMemoryMappedIO regions because some
>     other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the
>     E820 entries prevent Linux from allocating BAR space for hot-added devices.
>     
>     Allow use of ECAM for extended config space when the region is covered by
>     an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02
>     _CRS.
>     
>     Fixes: 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map")
>     Link: https://lore.kernel.org/r/ac2693d8-8ba3-72e0-5b66-b3ae008d539d@linux.intel.com
> 
> diff --git a/arch/x86/pci/mmconfig-shared.c b/arch/x86/pci/mmconfig-shared.c
> index 758cbfe55daa..4adc587a4c94 100644
> --- a/arch/x86/pci/mmconfig-shared.c
> +++ b/arch/x86/pci/mmconfig-shared.c
> @@ -12,6 +12,7 @@
>   */
>  
>  #include <linux/acpi.h>
> +#include <linux/efi.h>
>  #include <linux/pci.h>
>  #include <linux/init.h>
>  #include <linux/bitmap.h>
> @@ -442,6 +443,25 @@ static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used)
>  	return mcfg_res.flags;
>  }
>  
> +static bool is_efi_reserved(u64 start, u64 end, enum e820_type not_used)
> +{
> +	efi_memory_desc_t *md;
> +	u64 size, mmio_start, mmio_end;
> +
> +	for_each_efi_memory_desc(md) {
> +		if (md->type == EFI_MEMORY_MAPPED_IO) {

Should this also consider EFI_RESERVED_TYPE? Not that any known BIOS
needs that accommodation. This is more a question than a suggestion.