Message-ID: <98b2a8d6-c5bf-a782-7fc1-8874f94edc25@os.amperecomputing.com>
Date:   Thu, 16 Feb 2023 10:22:14 +0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.6.1
Subject: Re: [PATCH 2/2] misc: smpro-errmon: Add dimm training failure
 syndrome
Content-Language: en-CA
To:     Paul Menzel <pmenzel@molgen.mpg.de>
Cc:     Arnd Bergmann <arnd@arndb.de>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Joel Stanley <joel@jms.id.au>,
        Andrew Jeffery <andrew@aj.id.au>, linux-kernel@vger.kernel.org,
        openbmc@lists.ozlabs.org,
        Open Source Submission <patches@amperecomputing.com>,
        Thang Nguyen <thang@os.amperecomputing.com>,
        Phong Vo <phong@os.amperecomputing.com>
References: <20230214064509.3622044-1-quan@os.amperecomputing.com>
 <20230214064509.3622044-3-quan@os.amperecomputing.com>
 <866fe1b3-8044-6581-9711-452550f91198@molgen.mpg.de>
From:   Quan Nguyen <quan@os.amperecomputing.com>
In-Reply-To: <866fe1b3-8044-6581-9711-452550f91198@molgen.mpg.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cnQ4ZlljbjdaT1FLWDdNNHl4cmQwekc4UmZuUjhlMXN2MHF1RXdJSWI1NmxH?=
 =?utf-8?B?VlRTaGw2cEd5SU9qNE5Ca3NhSUJ0Y3FKdSszczdVSlJwU2ZhclhHVUxiNDA0?=
 =?utf-8?B?WStJVW54NXMyU1hzaHJubmdqRVFzUHN4M1plczZUdThuY0lxZ2IxaHBac2tj?=
 =?utf-8?B?SlozdFppenBkeW1ESFRUSENSQm45NGJveHA3WGxteDUvc2tCaVNjY0NQencr?=
 =?utf-8?B?T1cvNzVwekVvcmxhNk85MDN3Z3lCQkR6RDFEVHhqUkhRNVIwRGtReU5HU3l1?=
 =?utf-8?B?WU9SL2thZkgranUwUHlqOVJpSHEyOS93cnh3dCtoWUtlS3p1K2RyRWxWcVI4?=
 =?utf-8?B?WU5Mbm50bWV4L3pPQlVnUU5tZUtiM1A2dnFZKzAzRjFVQjlhdFRXM3c5Wkls?=
 =?utf-8?B?elRiZkg0TmM2dkJKOVNoN1RIalNCM083cjFMcmlad3FNNVBIWVFmWWtqZ1Ex?=
 =?utf-8?B?eDVWY0R3a2tqRlI2c2NOem8zTUNKZUJMbUdOcUYrUVlMMk9sWHA2dFMxS2du?=
 =?utf-8?B?M3F6UGNtd3Rhb0lHTnoyV0Z4Y1RpaFV6WVpRb0NqY1hjeXZxakZOWEFMbU5k?=
 =?utf-8?B?anJkTVBSNm5lV3hFaHhOajNXOUNML2RnbWdKRDdrRFViT0JhRXNMSDR5WFhI?=
 =?utf-8?B?RldwVnlUWitVeUE4L05KdFhMTlIyN3VZbTN5bHlmOTFBdVZ0OGZBTXZjOUtD?=
 =?utf-8?B?RTMra1VUSk1HWkpYdmI3S2djc1U1QkJybzBSWWdVRU9uMXoxSGFKbGlFS2lV?=
 =?utf-8?B?aFM5TzZlejVjMElWWERmR2Z6M1F0c3FTNVdOTmhUSURQYndndlJEMlA3N0xv?=
 =?utf-8?B?SmhjOENzU3lRRDlPTWtGaUQrTlk4UnRzb2wrSkwvajlFNXhPODVaN253UkVU?=
 =?utf-8?B?dzB2RTBTdDZDeU9zY2ZjR2I2OExRdGNNeDQxYXYvMzhuU05RL2tFeG5iK2hq?=
 =?utf-8?B?eTJXS0JTakZGRWJneVdyU3Q5aElsbEM3dEx5eHpna2pxTXF4V3VHb3hiT1JI?=
 =?utf-8?B?QWdvY3dxMmUyRGRBWWx0RUx3bjhDQ0V2Q1lDUzQ0dkMwVGFicXVqNUZBbjZ0?=
 =?utf-8?B?Z3Roc0RqbzJNV1F2bGhOU1pKdkZFWFRaR2wzRjd3VzJ1b0VPSzNORkxhZ3hp?=
 =?utf-8?B?M2JVTVkyRnpKNXZGZVREUEtoREZiYXAvYlZkQkMvL2ZKNGZ3NkhCMVArK3Zq?=
 =?utf-8?B?WkNpQThObnFEZ0s0WE5xbS9Eak5QSjk0N2hiZFZoclZLQlMvLzNXT0Y3WCsy?=
 =?utf-8?B?K2xVc1h1akNTVXgyRm90REtxMXNDeUdPWXc0ME1XdS9RYWVxSGM0ZkQ3Nkdh?=
 =?utf-8?B?UUd6cGNMTGpKTDJVVTN2RCs4Q2ZZSjNlUldjQTVTWkZnVjJTQS9mZ0hxSmNX?=
 =?utf-8?B?cUdNQVcrVDNFaDFFVmdwVVphVkxiaWh3Ym12eFdBTFVpelR0eCtFT1FSaGVU?=
 =?utf-8?B?NUdOZ0xucFRXQ3MvU0NoWDhBbEZZUDcrQnVaYlg2Z1FidzRFZ1VlUHhmVTJM?=
 =?utf-8?B?aGF0Y1p1eUhWQ21LYmVyTG9nWlRkbzhMMlREcDNEck9CRDB6MHpaU1dKdTFG?=
 =?utf-8?B?VkludThINkw0dlVWYnkxQmpYV21KSUJHdnh1VWFzYU9BNEFSZjRJS1ZtbmNz?=
 =?utf-8?B?b3NIdTJHNjlCRXI5MktmQWF3cGRTUWJiZU5ScFNabDhldEdwMFl2L29WSjFK?=
 =?utf-8?B?b2F0ME54Mzd5UUlwRElpaWllbFE0eUYxVzBHN1RuVEhVdC9HaE5weGNld213?=
 =?utf-8?B?RktsaUIyS1d2MTd3ZERWMzF1MFNqZGpvUytVTkg0ZDUwdFVXYVE5NitibGsv?=
 =?utf-8?B?cFFxbDdya0xscVk4eFNoU3NZU2poZlhGK2ZjK2lScFV3aldlUjBqNWRCV3BT?=
 =?utf-8?B?WXVySFRhL2RQL0Q0Tmp6bWtMZ2ZDa3lRU3psMHZMZGwydjIyajMvVDZlTWVs?=
 =?utf-8?B?WFFGVTRFY0p0ZVcyTnNYSGNRaFZ0dlVoRFVpcVM4Qm9mVUtqYXc1SHk1N3ZW?=
 =?utf-8?B?UEU5MjhPcjMvaWtRZjZHMUNqcUlMTXp6RjAxbDAwMmc4am1wOWRoNkNPUlVt?=
 =?utf-8?B?THRXY1pST2l5WEQyM3dYNUJhSzgyWTVTTEpWTXZrZXpZZklhUjAwMU9iWksz?=
 =?utf-8?B?MFkxMytMbmhxMzN1bzh5bG52eitjSXJiN2pFTXFGL1VmSFFGTXI2T3A2dVRI?=
 =?utf-8?Q?m4gA2BzUKClqaDfTbqbRkCA=3D?=
X-OriginatorOrg: os.amperecomputing.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8e3dc8fd-e6c8-4cd6-2807-08db0fcd0991
X-MS-Exchange-CrossTenant-AuthSource: SN4PR01MB7455.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2023 03:22:31.0808
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: xrUjYj+GaKf0D8Rdm7FHrrKkmIYYsDaT2v/ocRPU7SEjEvIaQA+LNlQTLQM6k6tFjuOz+dMeBAvXr8AWLpkU3phgod/tuSphMoqsrNExg1o=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR01MB7992
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 15/02/2023 14:33, Paul Menzel wrote:
> Dear Quan,
> 
> 
> Thank you for your patch.
> 

Thanks Paul for the review.

> Am 14.02.23 um 07:45 schrieb Quan Nguyen:
>> Adds event_dimm[0-15]_syndrome sysfs to report the failure syndrome
>> to BMC when DIMM training failed.
> 
> Where you able to verify that it works? Out of curiosity, how?
> 

Yes, we verified it by injecting DIMM errors and confirm that errors was 
reported correctly via sysfs.
For about how to do error injection, we may  need to refer to section 
3.2 Memory Error Group in Altra Family RAS Error Injection User Manual. 
It is shared in our Ampere Customer Connect [1]. The latest version is 
v1.00_20220329.

[1] https://connect.amperecomputing.com

>> Signed-off-by: Quan Nguyen <quan@os.amperecomputing.com>
>> ---
>>   .../sysfs-bus-platform-devices-ampere-smpro   | 10 +++
>>   drivers/misc/smpro-errmon.c                   | 77 +++++++++++++++++++
>>   2 files changed, 87 insertions(+)
>>
>> diff --git 
>> a/Documentation/ABI/testing/sysfs-bus-platform-devices-ampere-smpro 
>> b/Documentation/ABI/testing/sysfs-bus-platform-devices-ampere-smpro
>> index d4e3f308c451..c35f1d45e656 100644
>> --- a/Documentation/ABI/testing/sysfs-bus-platform-devices-ampere-smpro
>> +++ b/Documentation/ABI/testing/sysfs-bus-platform-devices-ampere-smpro
>> @@ -265,6 +265,16 @@ Description:
>>           For more details, see section `5.7 GPI Status Registers and 
>> 5.9 Memory Error Register Definitions,
>>           Altra Family Soc BMC Interface Specification`.
>> +What:        
>> /sys/bus/platform/devices/smpro-errmon.*/event_dimm[0-15]_syndrome
>> +KernelVersion:    6.1
> 
> Should it be 6.2, as it probably won’t make it into 6.1?
> 

Thanks for the catch. Will fix in next version.

>> +Contact:    Quan Nguyen <quan@os.amperecomputing.com>
>> +Description:
>> +        (RO) The sysfs returns the 2-byte DIMM failure syndrome data 
>> for slot
>> +        0-15 if it failed to initialized.
> 
> to initialize
> 

Will fix in next version.

>> +
>> +        For more details, see section `5.11 Boot Stage Register 
>> Definitions,
>> +        Altra Family Soc BMC Interface Specification`.
>> +
>>   What:        /sys/bus/platform/devices/smpro-misc.*/boot_progress
>>   KernelVersion:    6.1
>>   Contact:    Quan Nguyen <quan@os.amperecomputing.com>
>> diff --git a/drivers/misc/smpro-errmon.c b/drivers/misc/smpro-errmon.c
>> index 1635e881aefb..3e8570cbb740 100644
>> --- a/drivers/misc/smpro-errmon.c
>> +++ b/drivers/misc/smpro-errmon.c
>> @@ -47,6 +47,12 @@
>>   #define WARN_PMPRO_INFO_LO    0xAC
>>   #define WARN_PMPRO_INFO_HI    0xAD
>> +/* Boot Stage Register */
>> +#define BOOTSTAGE        0xB0
>> +#define DIMM_SYNDROME_SEL    0xB4
>> +#define DIMM_SYNDROME_ERR    0xB5
>> +#define DIMM_SYNDROME_STAGE    4
>> +
>>   /* PCIE Error Registers */
>>   #define PCIE_CE_ERR_CNT        0xC0
>>   #define PCIE_CE_ERR_LEN        0xC1
>> @@ -468,6 +474,61 @@ EVENT_RO(vrd_hot, VRD_HOT_EVENT);
>>   EVENT_RO(dimm_hot, DIMM_HOT_EVENT);
>>   EVENT_RO(dimm_2x_refresh, DIMM_2X_REFRESH_EVENT);
>> +static ssize_t smpro_dimm_syndrome_read(struct device *dev, struct 
>> device_attribute *da,
>> +                    char *buf, int slot)
> 
> Could `slot` be passed as `unsigned int`?
> 

Yes, I will update in next version.
Indeed, the "slot" will be passed as the last param to regmap_write() 
which is in unsigned int as below.

int regmap_write(struct regmap *map, unsigned int reg, unsigned int val);

So, change to "unsigned int" is corrected.

>> +{
>> +    struct smpro_errmon *errmon = dev_get_drvdata(dev);
>> +    s32 data;
>> +    int ret;
>> +
>> +    ret = regmap_read(errmon->regmap, BOOTSTAGE, &data);
> 
> The function signature is:
> 
>      int regmap_read(struct regmap *map, unsigned int reg, unsigned int 
> *val)
> 
> So why not use unsigned int as data type for `data`?
> 

Agree, you comment is correct. Will change data to "unsigned int" in new 
version.

>> +    if (ret)
>> +        return ret;
>> +
>> +    /* check for valid stage */
>> +    data = (data >> 8) & 0xff;
>> +    if (data != DIMM_SYNDROME_STAGE)
>> +        return ret;
> 
> Isn’t now success returned? Should a debug message be printed?
> 

Yes, this is a success case, there is no dimm training error and has 
nothing to return in sysfs.
If no dimm training error, host will never stay in DIMM_SYNDROME_STAGE.

I dont think printing a debug message in this case is good because 
printing out something like "No dimm traning error" is not necessary as 
I think.

>> +
>> +    /* Write the slot ID to retrieve Error Syndrome */
>> +    ret = regmap_write(errmon->regmap, DIMM_SYNDROME_SEL, slot);
>> +    if (ret)
>> +        return ret;
>> +
>> +    /* Read the Syndrome error */
>> +    ret = regmap_read(errmon->regmap, DIMM_SYNDROME_ERR, &data);
>> +    if (ret || !data)
>> +        return ret;
>> +
>> +    return sysfs_emit(buf, "%04x\n", data);
>> +}
>> +
>> +#define EVENT_DIMM_SYNDROME(_slot) \
>> +    static ssize_t event_dimm##_slot##_syndrome_show(struct device 
>> *dev,          \
>> +                             struct device_attribute *da, \
>> +                             char *buf)                   \
>> +    
>> {                                                                             \
>> +        return smpro_dimm_syndrome_read(dev, da, buf, 
>> _slot);                 \
>> +    
>> }                                                                             \
>> +    static DEVICE_ATTR_RO(event_dimm##_slot##_syndrome)
>> +
>> +EVENT_DIMM_SYNDROME(0);
>> +EVENT_DIMM_SYNDROME(1);
>> +EVENT_DIMM_SYNDROME(2);
>> +EVENT_DIMM_SYNDROME(3);
>> +EVENT_DIMM_SYNDROME(4);
>> +EVENT_DIMM_SYNDROME(5);
>> +EVENT_DIMM_SYNDROME(6);
>> +EVENT_DIMM_SYNDROME(7);
>> +EVENT_DIMM_SYNDROME(8);
>> +EVENT_DIMM_SYNDROME(9);
>> +EVENT_DIMM_SYNDROME(10);
>> +EVENT_DIMM_SYNDROME(11);
>> +EVENT_DIMM_SYNDROME(12);
>> +EVENT_DIMM_SYNDROME(13);
>> +EVENT_DIMM_SYNDROME(14);
>> +EVENT_DIMM_SYNDROME(15);
>> +
>>   static struct attribute *smpro_errmon_attrs[] = {
>>       &dev_attr_overflow_core_ce.attr,
>>       &dev_attr_overflow_core_ue.attr,
>> @@ -493,6 +554,22 @@ static struct attribute *smpro_errmon_attrs[] = {
>>       &dev_attr_event_vrd_hot.attr,
>>       &dev_attr_event_dimm_hot.attr,
>>       &dev_attr_event_dimm_2x_refresh.attr,
>> +    &dev_attr_event_dimm0_syndrome.attr,
>> +    &dev_attr_event_dimm1_syndrome.attr,
>> +    &dev_attr_event_dimm2_syndrome.attr,
>> +    &dev_attr_event_dimm3_syndrome.attr,
>> +    &dev_attr_event_dimm4_syndrome.attr,
>> +    &dev_attr_event_dimm5_syndrome.attr,
>> +    &dev_attr_event_dimm6_syndrome.attr,
>> +    &dev_attr_event_dimm7_syndrome.attr,
>> +    &dev_attr_event_dimm8_syndrome.attr,
>> +    &dev_attr_event_dimm9_syndrome.attr,
>> +    &dev_attr_event_dimm10_syndrome.attr,
>> +    &dev_attr_event_dimm11_syndrome.attr,
>> +    &dev_attr_event_dimm12_syndrome.attr,
>> +    &dev_attr_event_dimm13_syndrome.attr,
>> +    &dev_attr_event_dimm14_syndrome.attr,
>> +    &dev_attr_event_dimm15_syndrome.attr,
>>       NULL
>>   };
> 
> 
> Kind regards,
> 
> Paul