Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32;
Message-ID: <1c598798-5b28-4a17-bf86-042781808021@amd.com>
Date:   Mon, 16 Oct 2023 10:14:05 -0400
User-Agent: Mozilla Thunderbird
Cc:     yazen.ghannam@amd.com, "tony.luck@intel.com" <tony.luck@intel.com>,
        "bp@alien8.de" <bp@alien8.de>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
        "x86@kernel.org" <x86@kernel.org>, "hpa@zytor.com" <hpa@zytor.com>,
        "linux-edac@vger.kernel.org" <linux-edac@vger.kernel.org>
Subject: Re: [PATCH] x86/mce: Increase the size of the MCE pool from 2 to 8
 pages
To:     Dave Hansen <dave.hansen@intel.com>,
        "Sironi, Filippo" <sironi@amazon.de>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
References: <20231011163320.79732-1-sironi@amazon.de>
 <afaef377-25e0-49f6-a99f-3e5bd4b44f87@intel.com>
 <EDD08AA3-C404-4DB6-96BA-2B25519B2496@amazon.de>
 <6591377b-7911-444b-abf9-cfc978472d76@intel.com>
Content-Language: en-US
From:   Yazen Ghannam <yazen.ghannam@amd.com>
In-Reply-To: <6591377b-7911-444b-abf9-cfc978472d76@intel.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bE5kUllhNlpPTzA1emVqVndxK1UxRml6SnlZV29La0hRUmFTY1NpcWpDcFAv?=
 =?utf-8?B?R1lzM1VJLzNxajlObnV1Y2VHQkpwRGlSbUNkOW8zYW9QTHBOTE45OGgzUWJz?=
 =?utf-8?B?YXdwVFh5cEZlaXB0K0N0MldYZklMbFZmRFEzS3NJWFJBWFg1TkxJYnlRUUZw?=
 =?utf-8?B?Q3dqeExDZ0F3MkJ2L0hiZnM1TTZBOWQyZ1JZQ1dkdFZFWGNrWDdMTytOYldm?=
 =?utf-8?B?VDlIYWY0L0I5cVM4bWoyeHQ2eEFyVTlMMGRsc2J5UkphSi9ucmVDY041U3lF?=
 =?utf-8?B?Z2FGUENsVjk1TVhLUmlxbkl2dHMzYmttSVArYjRoN0o0RlJoMmpFRTNtTjhq?=
 =?utf-8?B?SFU4Y0tmREpiVURvYVE4MExJVWdtKzdIaWtBZHJDSnU1M2JVMTdIbnQzYXpn?=
 =?utf-8?B?cnlMSHhkRFN3WXpXQmhiRno3RDJkSm1ONGg0RnkwdjFMa3B1VFNZT3hOekFs?=
 =?utf-8?B?UnNxeFBmUXF5ODJCbFdwaXZMTUtld3ZoaXBjSlhpWnZORC8wU2JsbVBRelFV?=
 =?utf-8?B?UXNsdm1Gb1lyQ0ZBVGEvM2tuaWx2RW9ScGNQZ1pUeDhYNWtSdG91amhVazFT?=
 =?utf-8?B?Z3FKdEw2ZjB2M1FNRVF2VFlML1JXMEhZdnFkSXZjRlpIdHJlRnduY1NKbmU3?=
 =?utf-8?B?WmlSdmJMTndLRXZoSk4veGZWWUtsblR0MHR4cStoWU01ZXZpbEJrQzA0azM5?=
 =?utf-8?B?TVFNSy9PRDhkZDZZb0tzL1JCaGtrdlB0aHZybk1KT1dVbDY2dnhnMlhtaGt5?=
 =?utf-8?B?WTUwYmpna2REU2VEbHVURTdNNmpCajMvblZHQy9TZDJnZVFPc0dzZ2VNSVBp?=
 =?utf-8?B?VW5hR2hnU0xPQ3dJU3V0eWFySW9IaW1HTk1nRzJTTXZLMmk4Z2cxM2JjRlF2?=
 =?utf-8?B?SGJOR3N5RkZZTml5NXZRNDhYaTZxNXp4cXh3YXpOQ3BieVBCdUNqSm1hblpz?=
 =?utf-8?B?ZmpoL0RhZDlxanZzRjhUS1RjdFNRL0pzWGwxVUh2OXkyeTlDRG1UcitvTUxu?=
 =?utf-8?B?aFVLY3BGbW0zV2FZblI2SUh1TG1SMDlNQm0wVTZzUXhtOFZ2TkhyTzRHb2FP?=
 =?utf-8?B?SzJmSTArbE1kSlAvcjU0b1VTR2FmRXBVeTcwSmdZa3J4SDVtdGFzcG9QekM5?=
 =?utf-8?B?VWp4T0Q4RWlyeXpXaDFNSklCSWlkZEpZL2pZUHNJMVJ2RWNKUGxWekQ5dVhj?=
 =?utf-8?B?Um1JT3E0SVkzTzUyaENVMStJUHd4VjQ2NWlMUlA0aklTbHQwemE2NU1DdlBT?=
 =?utf-8?B?YXoyeFd3dGZyakRpTFYyeXR0ekRkcituTjIyeUUrWko2TzVZVGgxU0dQQVhU?=
 =?utf-8?B?R0lJZHZ1NUZjUHNTZkdtaTFzM2NTSjc3R2kwaGx1R2wyUnZTOUZwTGN2T1RU?=
 =?utf-8?B?b2pJMUQ3L3prYTRHekNBUjJCS2J3VFZnNitkZG56T3RQWTBkaGNNOUFtNGhv?=
 =?utf-8?B?SXovRHpXUWdiTTJRWStqaUhUUElrbDNxMFRTZUhCSzVVZlR2L0Z1dHIycklH?=
 =?utf-8?B?dWEyWWJPZ1ZjamxqbitJUkhlVVNabUVFemVRRlpUdzdQOU00Ujl0ZXZqekEw?=
 =?utf-8?B?UEhySzlKMExNeUdYUnFqNWZJSWdhbjhwTmJ2eVBYdTJuV3M1N01zY0JxVjdw?=
 =?utf-8?B?T1R5YTlrRVVjWHorSVVLdmc0ejJ6WDhKdjdURWg2OGlDb0FFc3ZiUCtjT1Jp?=
 =?utf-8?B?NkFUd3pReWFDT0kvQXFnaEFIWEY5TE04UXFCU3ZxRG9nL3VZMEQwaTJmOFc3?=
 =?utf-8?B?ckhKWFVienVkS3pkNGFpT0pLdWNvbXI2aW5PT3RxbGNsN0NFVUVyMldESGth?=
 =?utf-8?B?VDBQcnIwMUFab0RWdnZ5eDhwZnZmSkJWK3dMdERjUjN0U0JCMVAvZVpLK3FI?=
 =?utf-8?B?djBIWEJFZmx5TzlES2kwWEZVY2lJN2JyS0RGdWRCRUxKZlFVSWVrVEdsZTZZ?=
 =?utf-8?B?dSs2UjVvaEJPWnF5MEZNcGl3emM2eVc0dW41MXRyUGR2MDhXejdCWnVXVFdE?=
 =?utf-8?B?YmJuM3ArZTRTbjlvZi9oUm42Y2RKcjhrR28rOFBxNHliWWFpWXk3dS9zcHJC?=
 =?utf-8?B?LzByKzVma25NV1I5ZTVNVnNSVzl1dE84Y051RmNTQ1lyOFR0WWJvanY2anN1?=
 =?utf-8?Q?nO1qnBKM2shS7hM53+aSdETFS?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ca2895ec-a0dd-42de-5413-08dbce52295a
X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB3108.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Oct 2023 14:14:08.6657
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1PKB6D2dOObjZ8bxXfBsJ9IMGZjGOnwU/FgqLPTXWjMASfDNQREiBIXHXpquPKlCgk/OP7Zkq55dEAmu7uFwSQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5752
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 16 Oct 2023 07:14:45 -0700 (PDT)

On 10/12/23 11:49 AM, Dave Hansen wrote:
> On 10/12/23 04:46, Sironi, Filippo wrote:
>> There's correlation across the errors that we're seeing, indeed,
>> we're looking at the same row being responsible for multiple CPUs
>> tripping and running into #MC. I still don't like the full lack of
>> visibility; it's not uncommon in a large fleet to see to take a
>> server out of production, replace a DIMM and shortly after taking it
>> out of production again to replace another DIMM just because some of
>> the errors weren't properly logged.
> 
> So you had two nearly simultaneous DIMM failures.  The first failed,
> filled up the buffer and then the second failed, but there was no room.
> The second failed *SO* soon after the first that there was no
> opportunity to empty the buffer between.
> 
> Right?
> 
> How do you know that storing 8 pages of records will catch this case as
> opposed to storing 2?
> 
>>> Is there any way that the size of the pool can be more automatically
>>> determined? Is the likelihood of a bunch errors proportional to the
>>> number of CPUs or amount of RAM or some other aspect of the hardware?
>>>
>>> Could the pool be emptied more aggressively so that it does not fill up?
> 
> You didn't really address the additional questions I posed there.
> 
> I'll add one more: how many of the messages are duplicates or
> *effectively* duplicates?  Or is that hard to determine at the time that
> the entries are being made that they are duplicates?
> 
> It _should_ also be fairly easy to enlarge the buffer on demand, say, if
> it got half full.  What's the time scale over which the buffer filled
> up?  Did a single #MC fill it up?
> 
> I really think we need to understand what the problem is and have _some_
> confidence that the proposed solution will fix that, even if we're just
> talking about a new config option.

I've seen a similar issue, and it's not just related to memory errors.
In my experience it was MCA errors from a variety of hardware blocks.
For example, a bad link internal to an SoC could spew MCA errors
regardless of the scale of RAM or CPUs. Same thing is possible for a bad
cache, etc.

These were during pre-production testing, and the easy workaround is to
increase the MCE genpool size at build time.

I don't think this needs to be the default though.

How about this to start?

1) Keep the current config size for boot time.
2) Add a kernel parameter and/or sysfs file to allow users to request
additional genpool capacity.
3) Use gen_pool_add(), or whichever, to add the capacity based on user
input.

Maybe this can be expanded later to be automatic. But I think it simpler
to start with explicit user input.

Thanks,
Yazen