Received: by 2002:a05:6a10:413:0:0:0:0 with SMTP id 19csp960541pxp; Wed, 16 Mar 2022 22:42:40 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxBOUIxvZcAa4xdbvK4xQIXZdh0RCiBK+afNIseyxLdvUDKH2GeWZ/gwCgcdDcp61V3CPLW X-Received: by 2002:a17:902:ca0b:b0:153:b9f5:83b6 with SMTP id w11-20020a170902ca0b00b00153b9f583b6mr3419710pld.166.1647495760001; Wed, 16 Mar 2022 22:42:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1647495759; cv=none; d=google.com; s=arc-20160816; b=l6+c4eJGlTjtVNZJJxFgophNzPN6uryvGs+U+CyP+XiXPhaNbUra6PEsVaAEZB7ele kg4MFwdbFPfllg+OPIdK+Cd/eie+jGcTNUNahVhqjIrYuPYCQWIrvfU2aS1nhLrCsxbg R3iZ6Kue0Ka1Pn3GKMXB/yFMHjXbGCjEFkuBtsjiMlDiXmypCUgJaBpCsKvx/t4dfv0q ODJtWXUTHkXqSeUT8Qa7JPnbJ9nG4RvlHqxRp6Gw7VUD4MZm2IuwPGlWXFOES4WaVAXf wOIFwyY8VEGf+tz7aCg78qIUlKtkvIu5uVkGSmH5m2WIJENPpqdIxuVm+2yT9Ix6BB4d X+JQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=YdfDwLcCxjBdFThwwHos9x013CjhNXUSI2ePOCAUr0A=; b=vSu/aMWtA6HBC32LM43EpVG0N+7mL9ugvCw5jXjgJwl6bYZeNlLGvOr9ARrrMZTl4I 1V7cNPatXI1cBiVQl6JRI90x2POTWTWPFoCCnrdcWcgLZ7TU9rheRl/tt9D9rieagLo9 VaQcmog9BGq9N0s84emTcOLQ4IfFR4vmtj2huEYyD3zyoh/tMHJcWlEa72Uzf6v956mw ZY1n9+G7IXxr5yg653dMYcDKJozMlhDB1DolyZRq6LYT/Xdfw8+FxWFpdXJMH8vEAGgp RWa8TYAPNkfKJqpj6y4ftg30s21tDDoxBfXgKgcnQr0lv71RXY/QpIili5DNq0K6sE+b c+QA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@purestorage.com header.s=google header.b=iIS9XuZt; spf=softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=purestorage.com Return-Path: Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net. [23.128.96.19]) by mx.google.com with ESMTPS id e13-20020a170903240d00b00151a8923013si3689999plo.198.2022.03.16.22.42.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Mar 2022 22:42:39 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19; Authentication-Results: mx.google.com; dkim=pass header.i=@purestorage.com header.s=google header.b=iIS9XuZt; spf=softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=purestorage.com Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 7F7DB1AD1D2; Wed, 16 Mar 2022 21:52:57 -0700 (PDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351544AbiCPQ3G (ORCPT + 99 others); Wed, 16 Mar 2022 12:29:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48652 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236791AbiCPQ3E (ORCPT ); Wed, 16 Mar 2022 12:29:04 -0400 Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B0F9255B8 for ; Wed, 16 Mar 2022 09:27:48 -0700 (PDT) Received: by mail-pf1-x435.google.com with SMTP id d19so4414893pfv.7 for ; Wed, 16 Mar 2022 09:27:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=YdfDwLcCxjBdFThwwHos9x013CjhNXUSI2ePOCAUr0A=; b=iIS9XuZtPG0W5Be9kuSTERIiint31DBY4mh3ylmZoH9Dg3HorEwyYM9h/pjdpx5yX3 fgMnEdo7U7Wyr2osuRjdEQBgoEgAuiqHSPJ9lIxbUaP+h+2tiylVp7bvWSvcnIuTd/6F I27AW6cl8/e0TlxzfCTZtE9O4Y+oX4c/otCS4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=YdfDwLcCxjBdFThwwHos9x013CjhNXUSI2ePOCAUr0A=; b=bseztXOMedTRsGDmctmli/RzlRZAPs84OQ7E0vXoJHmkAh9TLNwvoUlR846vq2KIBZ kqiV1kJvza9AktY1InmfiNz34jnUe8GiH64pHz/Na1FPRajcyytZjZRNrxO0SIIH7rnV ZWgyelrwyEySfv7P+nr3O74WenlGVSOvyf3dp5DCsAQOv01wJcUIt+Rw1Yr4KVoD8ecz H/WNBLliZ54ic2Ft9Fbp1YlkiLJDnRN7NHpkzdeKQcESGFm6glLwjlLfLSAVeP2VixB4 kygHHoLiYwxm0r34h3hEeirGhCHZrlqGt8t1pA+M0hR1aQipRzVW/lVP81RMeXJzTHEa Un4A== X-Gm-Message-State: AOAM530okOVW+8PGX3GeXEiA6EeOKqm20nWKc5EqyLxMFiu7g0aJWP7i nW7mCQIPaW7q9W7UIym5t81GzQ== X-Received: by 2002:aa7:9ad8:0:b0:4f7:78d4:de8c with SMTP id x24-20020aa79ad8000000b004f778d4de8cmr121241pfp.38.1647448068048; Wed, 16 Mar 2022 09:27:48 -0700 (PDT) Received: from ebps (vpn.purestorage.com. [192.30.189.1]) by smtp.gmail.com with ESMTPSA id n14-20020a17090a394e00b001c670d67b8esm847798pjf.32.2022.03.16.09.27.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Mar 2022 09:27:47 -0700 (PDT) Date: Wed, 16 Mar 2022 09:27:44 -0700 From: Eric Badger To: Sathyanarayanan Kuppuswamy Cc: Bjorn Helgaas , Russell Currey , Oliver OHalloran , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, ebadger@purestorage.com Subject: Re: [PATCH v2] PCI/AER: Handle Multi UnCorrectable/Correctable errors properly Message-ID: <20220316162744.GA1585319@ebps> References: <20220315050842.120063-1-sathyanarayanan.kuppuswamy@linux.intel.com> <20220315171425.GA1521135@ebps> <2d4e8811-dce6-c891-e92d-e3746434685e@linux.intel.com> <20220315195255.GA1523195@ebps> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Mar 15, 2022 at 02:29:23PM -0700, Sathyanarayanan Kuppuswamy wrote: > On 3/15/22 12:52 PM, Eric Badger wrote: > > On Tue, Mar 15, 2022 at 10:26:46AM -0700, Sathyanarayanan Kuppuswamy wrote: > > > On 3/15/22 10:14 AM, Eric Badger wrote: > > > > > # Prep injection data for a correctable error. > > > > > $ cd /sys/kernel/debug/apei/einj > > > > > $ echo 0x00000040 > error_type > > > > > $ echo 0x4 > flags > > > > > $ echo 0x891000 > param4 > > > > > > > > > > # Root Error Status is initially clear > > > > > $ setpci -s ECAP0001+0x30.w > > > > > 0000 > > > > > > > > > > # Inject one error > > > > > $ echo 1 > error_inject > > > > > > > > > > # Interrupt received > > > > > pcieport : AER: Root Error Status 0001 > > > > > > > > > > # Inject another error (within 5 seconds) > > > > > $ echo 1 > error_inject > > > > > > > > > > # No interrupt received, but "multiple ERR_COR" is now set > > > > > $ setpci -s ECAP0001+0x30.w > > > > > 0003 > > > > > > > > > > # Wait for a while, then clear ERR_COR. A new interrupt immediately > > > > > fires. > > > > > $ setpci -s ECAP0001+0x30.w=0x1 > > > > > pcieport : AER: Root Error Status 0002 > > > > > > > > > > Currently, the above issue has been only reproduced in the ICL server > > > > > platform. > > > > > > > > > > [Eric: proposed reproducing steps] > > > > Hmm, this differs from the procedure I described on v1, and I don't > > > > think will work as described here. > > > > > > I have attempted to modify the steps to reproduce it without returning > > > IRQ_NONE for all cases (which will break the functionality). But I > > > think I did not correct the last few steps. > > > > Well, the thinking in always returning IRQ_NONE was so that only setpci > > modified the register and we could clearly see how writes to the > > register affect interrupt generation. > > Got it. Makes sense. > > > > > > How about replacing the last 3 steps with following? > > > > > > # Inject another error (within 5 seconds) > > > $ echo 1 > error_inject > > > > > > # You will get a new IRQ with only multiple ERR_COR bit set > > > pcieport : AER: Root Error Status 0002 > > > > This seems accurate. Though it does muddy a detail that I think was > > clearer in the original procedure: was the second interrupt triggered by > > the second error, or by the write of 0x1 to Root Error Status? > > I think you are talking about the following command, right? > > setpci -s ECAP0001+0x30.w=0x1 > > If yes, my previously modified instructions already removed it. So > no confusion. The confusion I mention is: "what actually triggers the second interrupt?" Since I can't find a description of the observed behavior in the PCIe spec, I find it interesting to know what's actually happening. Since the procedure we've discussed in this thread stalls in aer_irq(), you can't distinguish clearly which event causes the second interrupt. > > To summarize, > > In your case, you have controlled both register read/write of Root > error status register to simulate the interrupt with only multi > ERR_COR bit set. > > In my case, I have attempted to simulate it without changing the > default behavior of aer_irq() in the kernel. > > Both seem ok to me. Although my personal preference is to trigger > the error without changing the code behavior, if both you and Bjorn > prefer to revert to old instructions, I will fix this in the next version. I think the amended procedure from this thread is fine to demonstrate how to play with the patch. The other procedure is available on the list if anyone has a need for it. Cheers, Eric