Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   George Cherian <gcherian@marvell.com>
To:     Jakub Kicinski <kuba@kernel.org>
CC:     "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "davem@davemloft.net" <davem@davemloft.net>,
        Sunil Kovvuri Goutham <sgoutham@marvell.com>,
        Linu Cherian <lcherian@marvell.com>,
        "Geethasowjanya Akula" <gakula@marvell.com>,
        "masahiroy@kernel.org" <masahiroy@kernel.org>,
        "willemdebruijn.kernel@gmail.com" <willemdebruijn.kernel@gmail.com>,
        "saeed@kernel.org" <saeed@kernel.org>,
        "jiri@resnulli.us" <jiri@resnulli.us>
Subject: RE: [PATCHv5 net-next 2/3] octeontx2-af: Add devlink health reporters
 for NPA
Thread-Topic: [PATCHv5 net-next 2/3] octeontx2-af: Add devlink health
 reporters for NPA
Thread-Index: AdbHkfXh13uAAxORSn68kJWCJhFAHgABlcMA
Date:   Tue, 1 Dec 2020 05:18:42 +0000
Message-ID: <BYAPR18MB26793451EC000695A56D3B00C5F40@BYAPR18MB2679.namprd18.prod.outlook.com>
References: <BYAPR18MB2679898E4AB4122CE2E566D6C5F40@BYAPR18MB2679.namprd18.prod.outlook.com>
In-Reply-To: <BYAPR18MB2679898E4AB4122CE2E566D6C5F40@BYAPR18MB2679.namprd18.prod.outlook.com>
Accept-Language: en-IN, en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: cdd99e41-182c-4a8f-0989-08d895b8920f
X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Dec 2020 05:18:42.9285
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Hsr7Y8vCLCTlgNS6wcgk84/tIN24lJGKZ65z8LaVfvH/ZKxaGpGe1J2Ij4VJGwwJsBuDFzlq64OoZ7taISkdAw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR18MB2422
X-OriginatorOrg: marvell.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312,18.0.737
 definitions=2020-11-30_12:2020-11-30,2020-11-30 signatures=0
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Jakub,

> -----Original Message-----
> From: George Cherian
> Sent: Tuesday, December 1, 2020 9:06 AM
> To: Jakub Kicinski <kuba@kernel.org>
> Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org;
> davem@davemloft.net; Sunil Kovvuri Goutham <sgoutham@marvell.com>;
> Linu Cherian <lcherian@marvell.com>; Geethasowjanya Akula
> <gakula@marvell.com>; masahiroy@kernel.org;
> willemdebruijn.kernel@gmail.com; saeed@kernel.org; jiri@resnulli.us
> Subject: Re: [PATCHv5 net-next 2/3] octeontx2-af: Add devlink health
> reporters for NPA
>=20
> Hi Jakub,
>=20
> > -----Original Message-----
> > From: Jakub Kicinski <kuba@kernel.org>
> > Sent: Tuesday, December 1, 2020 7:59 AM
> > To: George Cherian <gcherian@marvell.com>
> > Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org;
> > davem@davemloft.net; Sunil Kovvuri Goutham
> <sgoutham@marvell.com>;
> > Linu Cherian <lcherian@marvell.com>; Geethasowjanya Akula
> > <gakula@marvell.com>; masahiroy@kernel.org;
> > willemdebruijn.kernel@gmail.com; saeed@kernel.org; jiri@resnulli.us
> > Subject: Re: [PATCHv5 net-next 2/3] octeontx2-af: Add devlink health
> > reporters for NPA
> >
> > On Thu, 26 Nov 2020 19:32:50 +0530 George Cherian wrote:
> > > Add health reporters for RVU NPA block.
> > > NPA Health reporters handle following HW event groups
> > >  - GENERAL events
> > >  - ERROR events
> > >  - RAS events
> > >  - RVU event
> > > An event counter per event is maintained in SW.
> > >
> > > Output:
> > >  # devlink health
> > >  pci/0002:01:00.0:
> > >    reporter hw_npa
> > >      state healthy error 0 recover 0  # devlink  health dump show
> > > pci/0002:01:00.0 reporter hw_npa
> > >  NPA_AF_GENERAL:
> > >         Unmap PF Error: 0
> > >         NIX:
> > >         0: free disabled RX: 0 free disabled TX: 0
> > >         1: free disabled RX: 0 free disabled TX: 0
> > >         Free Disabled for SSO: 0
> > >         Free Disabled for TIM: 0
> > >         Free Disabled for DPI: 0
> > >         Free Disabled for AURA: 0
> > >         Alloc Disabled for Resvd: 0
> > >   NPA_AF_ERR:
> > >         Memory Fault on NPA_AQ_INST_S read: 0
> > >         Memory Fault on NPA_AQ_RES_S write: 0
> > >         AQ Doorbell Error: 0
> > >         Poisoned data on NPA_AQ_INST_S read: 0
> > >         Poisoned data on NPA_AQ_RES_S write: 0
> > >         Poisoned data on HW context read: 0
> > >   NPA_AF_RVU:
> > >         Unmap Slot Error: 0
> >
> > You seem to have missed the feedback Saeed and I gave you on v2.
> >
> > Did you test this with the errors actually triggering? Devlink should
> > store only
> Yes, the same was tested using devlink health test interface by injecting
> errors.
> The dump gets generated automatically and the counters do get out of sync=
,
> in case of continuous error.
> That wouldn't be much of an issue as the user could manually trigger a du=
mp
> clear and Re-dump the counters to get the exact status of the counters at
> any point of time.

Now that recover op is added the devlink error counter and recover counter =
will be=20
proper. The internal counter for each event is needed just to understand wi=
thin a specific reporter, how=20
many such events occurred.=20

Following is the log snippet of the devlink health test being done on hw_ni=
x reporter.
# for i in `seq 1 33` ; do  devlink health test pci/0002:01:00.0 reporter h=
w_nix; done
//Inject 33 errors (16  of NIX_AF_RVU and 17 of NIX_AF_RAS and  NIX_AF_GENE=
RAL errors)
# devlink health=20
pci/0002:01:00.0:
  reporter hw_npa
    state healthy error 0 recover 0 grace_period 0 auto_recover true auto_d=
ump true
  reporter hw_nix
    state healthy error 250 recover 250 last_dump_date 1970-01-01 last_dump=
_time 00:04:16 grace_period 0 auto_recover true auto_dump true
# devlink health dump show pci/0002:01:00.0 reporter hw_nix
NIX_AF_GENERAL:
        Memory Fault on NIX_AQ_INST_S read: 1=20
        Memory Fault on NIX_AQ_RES_S write: 1=20
        AQ Doorbell error: 1=20
        Rx on unmapped PF_FUNC: 1=20
        Rx multicast replication error: 1=20
        Memory fault on NIX_RX_MCE_S read: 1=20
        Memory fault on multicast WQE read: 1=20
        Memory fault on mirror WQE read: 1=20
        Memory fault on mirror pkt write: 1=20
        Memory fault on multicast pkt write: 1
  NIX_AF_RAS:
        Poisoned data on NIX_AQ_INST_S read: 1=20
        Poisoned data on NIX_AQ_RES_S write: 1=20
        Poisoned data on HW context read: 1=20
        Poisoned data on packet read from mirror buffer: 1=20
        Poisoned data on packet read from mcast buffer: 1=20
        Poisoned data on WQE read from mirror buffer: 1=20
        Poisoned data on WQE read from multicast buffer: 1=20
        Poisoned data on NIX_RX_MCE_S read: 1
  NIX_AF_RVU:
        Unmap Slot Error: 0
# devlink health dump clear pci/0002:01:00.0 reporter hw_nix
# devlink health dump show pci/0002:01:00.0 reporter hw_nix
NIX_AF_GENERAL:
        Memory Fault on NIX_AQ_INST_S read: 17=20
        Memory Fault on NIX_AQ_RES_S write: 17=20
        AQ Doorbell error: 17=20
        Rx on unmapped PF_FUNC: 17=20
        Rx multicast replication error: 17=20
        Memory fault on NIX_RX_MCE_S read: 17=20
        Memory fault on multicast WQE read: 17=20
        Memory fault on mirror WQE read: 17=20
        Memory fault on mirror pkt write: 17=20
        Memory fault on multicast pkt write: 17
  NIX_AF_RAS:
        Poisoned data on NIX_AQ_INST_S read: 17=20
        Poisoned data on NIX_AQ_RES_S write: 17=20
        Poisoned data on HW context read: 17=20
        Poisoned data on packet read from mirror buffer: 17=20
        Poisoned data on packet read from mcast buffer: 17=20
        Poisoned data on WQE read from mirror buffer: 17=20
        Poisoned data on WQE read from multicast buffer: 17=20
        Poisoned data on NIX_RX_MCE_S read: 17
  NIX_AF_RVU:
        Unmap Slot Error: 16
>=20
> > one dump, are the counters not going to get out of sync unless
> > something clears the dump every time it triggers?
Also, note that auto_dump is something which can be turned off by user.
# devlink health set pci/0002:01:00.0 reporter hw_nix auto_dump false
So that user can dump whenever required, which will always return the corre=
ct counter values.

>=20
> Regards,
> -George