Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Received-SPF: Pass (protection.outlook.com: domain of amd.com designates
 165.204.84.17 as permitted sender) receiver=protection.outlook.com;
 client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C
Date:   Fri, 14 Apr 2023 13:21:37 +0200
From:   Robert Richter <rrichter@amd.com>
To:     Ira Weiny <ira.weiny@intel.com>
CC:     Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Terry Bowman <terry.bowman@amd.com>,
        <alison.schofield@intel.com>, <vishal.l.verma@intel.com>,
        <bwidawsk@kernel.org>, <dan.j.williams@intel.com>,
        <dave.jiang@intel.com>, <linux-cxl@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <bhelgaas@google.com>,
        Oliver O'Halloran <oohall@gmail.com>,
        "Mahesh J Salgaonkar" <mahesh@linux.ibm.com>,
        <linuxppc-dev@lists.ozlabs.org>, <linux-pci@vger.kernel.org>
Subject: Re: [PATCH v3 6/6] PCI/AER: Unmask RCEC internal errors to enable
 RCH downstream port error handling
Message-ID: <ZDk3QeWZDOP8sr4s@rric.localdomain>
References: <20230411180302.2678736-7-terry.bowman@amd.com>
 <20230412212901.GA81099@bhelgaas>
 <20230413180122.00007471@Huawei.com>
 <643887b44b2d4_3a1882949d@iweiny-mobl.notmuch>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <643887b44b2d4_3a1882949d@iweiny-mobl.notmuch>
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2023 11:21:54.2082
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 63a70abd-3998-4231-22f8-08db3cda7392
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com]
X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT063.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5074
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 13.04.23 15:52:36, Ira Weiny wrote:
> Jonathan Cameron wrote:
> > On Wed, 12 Apr 2023 16:29:01 -0500
> > Bjorn Helgaas <helgaas@kernel.org> wrote:
> > 
> > > On Tue, Apr 11, 2023 at 01:03:02PM -0500, Terry Bowman wrote:
> > > > From: Robert Richter <rrichter@amd.com>
> > > > 

> > > > +static int __cxl_unmask_internal_errors(struct pci_dev *rcec)
> > > > +{
> > > > +	int aer, rc;
> > > > +	u32 mask;
> > > > +
> > > > +	/*
> > > > +	 * Internal errors are masked by default, unmask RCEC's here
> > > > +	 * PCI6.0 7.8.4.3 Uncorrectable Error Mask Register (Offset 08h)
> > > > +	 * PCI6.0 7.8.4.6 Correctable Error Mask Register (Offset 14h)
> > > > +	 */  
> > > 
> > > Unmasking internal errors doesn't have anything specific to do with
> > > CXL, so I don't think it should have "cxl" in the function name.
> > > Maybe something like "pci_aer_unmask_internal_errors()".
> > 
> > This reminds me.  Not sure we resolved earlier discussion on changing
> > the system wide policy to turn these on 
> > https://lore.kernel.org/linux-cxl/20221229172731.GA611562@bhelgaas/
> > which needs pretty much the same thing.
> > 
> > Ira, I think you were picking this one up?
> > https://lore.kernel.org/linux-cxl/63e5fb533f304_13244829412@iweiny-mobl.notmuch/
> 
> After this discussion I posted an RFC to enable those errors.
> 
> https://lore.kernel.org/all/20230209-cxl-pci-aer-v1-1-f9a817fa4016@intel.com/
> 
> Unfortunately the prevailing opinion was that this was unsafe.  And no one
> piped up with a reason to pursue the alternative of a pci core call to enable
> them as needed.
> 
> So I abandoned the work.
> 
> I think the direction things where headed was to have a call like:
> 
> int pci_enable_pci_internal_errors(struct pci_dev *dev)
> {
> 	int pos_cap_err;
> 	u32 reg;
> 
> 	if (!pcie_aer_is_native(dev))
> 		return -EIO;
> 
> 	pos_cap_err = dev->aer_cap;
> 
> 	/* Unmask correctable and uncorrectable (non-fatal) internal errors */
> 	pci_read_config_dword(dev, pos_cap_err + PCI_ERR_COR_MASK, &reg);
> 	reg &= ~PCI_ERR_COR_INTERNAL;
> 	pci_write_config_dword(dev, pos_cap_err + PCI_ERR_COR_MASK, reg);
> 	
> 	pci_read_config_dword(dev, pos_cap_err + PCI_ERR_UNCOR_SEVER, &reg);
> 	reg &= ~PCI_ERR_UNC_INTN;
> 	pci_write_config_dword(dev, pos_cap_err + PCI_ERR_UNCOR_SEVER, reg);
> 	
> 	pci_read_config_dword(dev, pos_cap_err + PCI_ERR_UNCOR_MASK, &reg);
> 	reg &= ~PCI_ERR_UNC_INTN;
> 	pci_write_config_dword(dev, pos_cap_err + PCI_ERR_UNCOR_MASK, reg);
> 
> 	return 0;
> }
> 
> ... and call this from the cxl code where it is needed.

The version I have ready after addressing Bjorn's comments is pretty
much the same, apart from error checking of the read/writes.

From your patch proposed you will need it in aer.c too and we do not
need to export it.

This patch only enables it for (CXL) RCECs. You might want to extend
this for CXL endpoints (and ports?) then.

> 
> Is this an acceptable direction?  Terry is welcome to steal the above from my
> patch and throw it into the PCI core.
> 
> Looking at the current state of things I think cxl_pci_ras_unmask() may
> actually be broken now without calling something like the above.  For that I
> dropped the ball.

Thanks,

-Robert

> 
> Ira