Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Bharat Bhushan <bharat.bhushan@nxp.com>
To:     Alex Williamson <alex.williamson@redhat.com>
CC:     Bjorn Helgaas <bhelgaas@google.com>,
        Bjorn Helgaas <helgaas@kernel.org>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        "bharatb.yadav@gmail.com" <bharatb.yadav@gmail.com>,
        David Daney <david.daney@cavium.com>,
        "jglauber@cavium.com" <jglauber@cavium.com>,
        "mbroemme@libmpq.org" <mbroemme@libmpq.org>,
        "chrisrblake93@gmail.com" <chrisrblake93@gmail.com>
Subject: RE: [PATCH] PCI: Mark NXP LS1088 to avoid bus reset bus
Thread-Topic: [PATCH] PCI: Mark NXP LS1088 to avoid bus reset bus
Thread-Index: AQHUhi2yh4+Qm28J50ixacdSn4eZIqVjwNwAgAAJqACAALvmYIABG3kAgACEALCAAbCFAIAAAq1ggACrf4CABF3uQA==
Date:   Mon, 3 Dec 2018 11:01:29 +0000
Message-ID: <VI1PR04MB4845A5AAE3692EC284147E419AAE0@VI1PR04MB4845.eurprd04.prod.outlook.com>
References: <20181127083454.26560-1-Bharat.Bhushan@nxp.com>
        <20181127153356.GA112381@google.com>    <20181127090830.084fedf1@x1.home>
        <VI1PR04MB48457E23AA3354316C7782D69AD10@VI1PR04MB4845.eurprd04.prod.outlook.com>
        <CAErSpo7By1t5X=M=x_+Fs8TeROACBnS3f5ujsi=Ewiogh=-wRg@mail.gmail.com>
        <VI1PR04MB4845D4ED02E38246E71DD1A49AD30@VI1PR04MB4845.eurprd04.prod.outlook.com>
        <20181129225606.328f7386@x1.home>
        <VI1PR04MB48458DF3A1EBB73A4E2CDCD19AD30@VI1PR04MB4845.eurprd04.prod.outlook.com>
 <20181130091929.3eae4e99@x1.home>
In-Reply-To: <20181130091929.3eae4e99@x1.home>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: nxp.com does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: a7304b93-21f5-4ea9-d5ce-08d6590ead8a
X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Dec 2018 11:01:29.3763
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR04MB4992
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Alex,

> -----Original Message-----
> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Friday, November 30, 2018 9:49 PM
> To: Bharat Bhushan <bharat.bhushan@nxp.com>
> Cc: Bjorn Helgaas <bhelgaas@google.com>; Bjorn Helgaas
> <helgaas@kernel.org>; linux-pci@vger.kernel.org; Linux Kernel Mailing Lis=
t
> <linux-kernel@vger.kernel.org>; bharatb.yadav@gmail.com; David Daney
> <david.daney@cavium.com>; jglauber@cavium.com;
> mbroemme@libmpq.org; chrisrblake93@gmail.com
> Subject: Re: [PATCH] PCI: Mark NXP LS1088 to avoid bus reset bus
>=20
> On Fri, 30 Nov 2018 06:24:16 +0000
> Bharat Bhushan <bharat.bhushan@nxp.com> wrote:
>=20
> > Hi Alex,
> >
> > > -----Original Message-----
> > > From: Alex Williamson <alex.williamson@redhat.com>
> > > Sent: Friday, November 30, 2018 11:26 AM
> > > To: Bharat Bhushan <bharat.bhushan@nxp.com>
> > > Cc: Bjorn Helgaas <bhelgaas@google.com>; Bjorn Helgaas
> > > <helgaas@kernel.org>; linux-pci@vger.kernel.org; Linux Kernel
> > > Mailing List <linux-kernel@vger.kernel.org>;
> > > bharatb.yadav@gmail.com; David Daney <david.daney@cavium.com>;
> > > jglauber@cavium.com; mbroemme@libmpq.org;
> chrisrblake93@gmail.com
> > > Subject: Re: [PATCH] PCI: Mark NXP LS1088 to avoid bus reset bus
> > >
> > > On Fri, 30 Nov 2018 05:29:47 +0000
> > > Bharat Bhushan <bharat.bhushan@nxp.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > > -----Original Message-----
> > > > > From: Bjorn Helgaas <bhelgaas@google.com>
> > > > > Sent: Thursday, November 29, 2018 1:46 AM
> > > > > To: Bharat Bhushan <bharat.bhushan@nxp.com>
> > > > > Cc: alex.williamson@redhat.com; Bjorn Helgaas
> > > > > <helgaas@kernel.org>;
> > > > > linux- pci@vger.kernel.org; Linux Kernel Mailing List <linux-
> > > > > kernel@vger.kernel.org>; bharatb.yadav@gmail.com; David Daney
> > > > > <david.daney@cavium.com>; jglauber@cavium.com;
> > > mbroemme@libmpq.org;
> > > > > chrisrblake93@gmail.com
> > > > > Subject: Re: [PATCH] PCI: Mark NXP LS1088 to avoid bus reset bus
> > > > >
> > > > > On Tue, Nov 27, 2018 at 10:32 PM Bharat Bhushan
> > > > > <bharat.bhushan@nxp.com> wrote:
> > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Alex Williamson <alex.williamson@redhat.com>
> > > > > > > Sent: Tuesday, November 27, 2018 9:39 PM
> > > > > > > To: Bjorn Helgaas <helgaas@kernel.org>
> > > > > > > Cc: Bharat Bhushan <bharat.bhushan@nxp.com>;
> > > > > > > linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org;
> > > > > > > bharatb.yadav@gmail.com; David Daney
> > > <david.daney@cavium.com>;
> > > > > Jan
> > > > > > > Glauber <jglauber@cavium.com>; Maik Broemme
> > > > > <mbroemme@libmpq.org>;
> > > > > > > Chris Blake <chrisrblake93@gmail.com>
> > > > > > > Subject: Re: [PATCH] PCI: Mark NXP LS1088 to avoid bus reset
> > > > > > > bus
> > > > > > >
> > > > > > > On Tue, 27 Nov 2018 09:33:56 -0600 Bjorn Helgaas
> > > > > > > <helgaas@kernel.org> wrote:
> > > > >
> > > > > > > > 4) Is there a hardware erratum for this?  If so, please
> > > > > > > > include the URL here.
> > > > > >
> > > > > > No h/w errata as of now.
> > > > >
> > > > > Does that mean (a) the HW folks agree this is a hardware problem
> > > > > but they haven't written an erratum, (b) there is an erratum but
> > > > > it isn't public, (c) we don't have any concrete evidence of a
> > > > > hardware problem, but things just don't work if we do a bus
> > > > > reset, (d) something
> > > else?
> > > >
> > > > I will say it is (c) - not concluded to be hardware h/w issue.
> > > >
> > > > >
> > > > > > In pci_reset_secondary_bus() I have tried to increase the
> > > > > > delay after reset
> > > > > but not helped.
> > > > > > Do I need to add delay at some other place as well?
> > > > >
> > > > > No, I think the place you tried should be enough.
> > > > >
> > > > > You should also be able to exercise this from user-space by
> > > > > using "setpci" to set and clear the Secondary Bus Reset bit in
> > > > > the Bridge Control register.  Then you can also use setpci to
> > > > > read/write config space of the NIC.  The kernel would normally
> > > > > read the Vendor and Device IDs as the first access to the device
> during enumeration.
> > > > > You also might be able to learn something by using "lspci -vv"
> > > > > on the bridge before and after the reset to see if it logs any
> > > > > AER bits (if it
> > > supports AER) or the other standard error logging bits.
> > > >
> > > > I tried below sequence for Secondary bus reset and device config
> > > > space show 0xff
> > > >
> > > > root@localhost:~# lspci -x
> > > > 0002:00:00.0 PCI bridge: Freescale Semiconductor Inc Device 80c0
> > > > (rev
> > > > 10)
> > > > 00: 57 19 c0 80 07 01 10 00 10 00 04 06 08 00 01 00
> > > > 10: 00 00 00 00 00 00 00 00 00 01 ff 00 01 01 00 00
> > > > 20: 00 40 00 40 f1 ff 01 00 00 00 00 00 00 00 00 00
> > > > 30: 00 00 00 00 40 00 00 00 00 00 00 40 63 01 00 00
> > > >
> > > > 0002:01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit
> > > > Network Connection
> > > > 00: 86 80 d3 10 06 04 10 00 00 00 00 02 10 00 00 00
> > > > 10: 00 00 0c 40 00 00 00 40 01 00 00 00 00 00 0e 40
> > > > 20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 1f a0
> > > > 30: 00 00 24 40 c8 00 00 00 00 00 00 00 63 01 00 00
> > > >
> > > > root@localhost:~#  setpci -s 0002:00:00.0 0x3e.b=3D0x40
> > > > root@localhost:~#  setpci -s 0002:00:00.0 0x3e.b=3D0x00
> > > >
> > > > root@localhost:~# lspci -x
> > > > 0002:00:00.0 PCI bridge: Freescale Semiconductor Inc Device 80c0
> > > > (rev
> > > > 10)
> > > > 00: 57 19 c0 80 07 01 10 00 10 00 04 06 08 00 01 00
> > > > 10: 00 00 00 00 00 00 00 00 00 01 ff 00 01 01 00 00
> > > > 20: 00 40 00 40 f1 ff 01 00 00 00 00 00 00 00 00 00
> > > > 30: 00 00 00 00 40 00 00 00 00 00 00 40 63 01 00 00
> > >
> > > Just for curiosity sake, what if you re-write the secondary and
> > > subordinate bus registers here:
> > >
> > > # setpci -s 0002:00:00.0 0x19.b=3D0x01 # setpci -s 0002:00:00.0
> > > 0x1a.b=3D0xff
> >
> > Result is same, here are logs
> >
> > root@localhost:~# setpci -s 0002:00:00.0 0x3e.b=3D0x40 root@localhost:~=
#
> > setpci -s 0002:00:00.0 0x3e.b=3D0x00 root@localhost:~# setpci -s
> > 0002:00:00.0 0x19.b=3D0x01 root@localhost:~# setpci -s 0002:00:00.0
> > 0x1a.b=3D0xff root@localhost:~# lspci -x
> > 0002:00:00.0 PCI bridge: Freescale Semiconductor Inc Device 80c0 (rev
> > 10)
> > 00: 57 19 c0 80 07 01 10 00 10 00 04 06 08 00 01 00
> > 10: 00 00 00 00 00 00 00 00 00 01 ff 00 01 01 00 00
> > 20: 00 40 00 40 f1 ff 01 00 00 00 00 00 00 00 00 00
> > 30: 00 00 00 00 40 00 00 00 00 00 00 40 63 01 00 00
> >
> > 0002:01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit
> > Network Connection (rev ff)
> > 00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > 10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > 20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > 30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>=20
> Ok, thanks for scratching my itch.
>=20
> > > IIRC the users that debugged the AMD bus reset issue re-wrote the
> > > entire 64 bytes of the bridge config header and then further
> > > narrowed the issue down to the two registers above.  If one bridge
> > > implementation can have such an issue, maybe others do too.  Perhaps
> there's common IP in use.
> >
> > >  Are you able
> > > to test other endpoints besides this e1000e device with this setpci
> > > technique?  Thanks,
> >
> > I tried  with " Broadcom Limited NetXtreme BCM5722 Gigabit Ethernet PCI
> Express" I observe same issue.
>=20
> Personally I'd exhaust talking with your hardware folks before blocking b=
us
> resets at the software level, it seems like a gap in PCIe compliance of t=
he
> device.  Thanks,

I will continue to work with our h/w team on this.

Thanks
-Bharat

>=20
> Alex