Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34;
Message-ID: <5b6bdf6a-760c-4ba3-95ec-2d4482ad9bac@intel.com>
Date:   Tue, 10 Oct 2023 08:56:38 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH RESEND v2] x86/mce: Set PG_hwpoison page flag to avoid the
 capture kernel panic
To:     Ingo Molnar <mingo@kernel.org>
CC:     <x86@kernel.org>, <linux-edac@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <patches@lists.linux.dev>,
        <bp@alien8.de>, <tony.luck@intel.com>, <naoya.horiguchi@nec.com>
References: <20230914030539.1622477-1-zhiquan1.li@intel.com>
 <ZRsUpM/XtPAE50Rm@gmail.com>
Content-Language: en-US
From:   Zhiquan Li <zhiquan1.li@intel.com>
In-Reply-To: <ZRsUpM/XtPAE50Rm@gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SUhTRjBaWlR0S050MWVodHN0NW4zNHFTSmZma3F1MVlPNGh2MnMweVVKTnRz?=
 =?utf-8?B?S3lhTnlZeWF5SnBuSEp4SjNranByeWZ6RlNIbnhHVTVqdDdlcFQyUEFCc21T?=
 =?utf-8?B?OElDemtyOGtwNTd1T3IyMXhwYTd6YVdvTHM1M3hQemJuRzBzaFhhc2t5cUg1?=
 =?utf-8?B?dURIRFZLb01XVUdnVnhPWllVd3EwL2RKMjdxMnV1YUxTQ2doMVMrQ0dCd0lT?=
 =?utf-8?B?VUZxczNVOE5JRUR0VTYyTEovS0JJc1h6cUdGOGhrVjBQdHJWZ3F5YUpjb0tG?=
 =?utf-8?B?TXZFQmVWWk5IMkJ2em0yb0tVeHM0LysvMkRQMXVTR0hVT3cvQlo0cmJQUFQr?=
 =?utf-8?B?dUczUnVkc1o1Z3Mwb2FWQkNCVllmQ2gwbTVla1BQSXJqMUZHR0o1SjE0WGxl?=
 =?utf-8?B?ZmlrY1FCaGxsNWN6VDRVM0ZoSlZkTVdNb1pVSnp6MXdHMVNheFA1MURxc2FZ?=
 =?utf-8?B?ZzY0Zk54KytlZHBkUnZmdGlhdXFwMkNySmI1OXNLZEVsOHE2ZUNnODVwTjhs?=
 =?utf-8?B?eldQNmx1NERROUwyTW1KNGN5MG5QVmlvVkNxVTI1cTV6ZzdSM2pjUG5XcG5O?=
 =?utf-8?B?cGwxL2dGMW4vdHlucURZTjQ5Z2paL2ZJMldFSHFITFEvV3FiMmNKeEFxL1pX?=
 =?utf-8?B?YWw4c3VQdzRjY1Q3S3VxVmhRd1lyM2pldlJ6SFkyenFvemN4M1UvOUU5OFAv?=
 =?utf-8?B?Nzc5eXQxKzBQS1ErMmh3eFpLRzlzcVNGSUJoNHMwT0ZrSGUzYjNSTUZzcnNV?=
 =?utf-8?B?OEFscDBrdGVVK1J4NURiRU1pdllzY2p3WmhsUGQ2QzNScTQvZGo3cjhQbUVx?=
 =?utf-8?B?dUdzeXcwMmM2NDMxRWZiNHd0OVhQdzgyc0c5RVlJZzVMYU93bVhBd3duUFR2?=
 =?utf-8?B?UkNpYldTOUJpTDdHclEvaXZOa0FkYWhwTUdHamRySFRzbW9lKzBTS1hFK2wz?=
 =?utf-8?B?RW5DOVV3bzkzMCtCWmMrNGZMU2VJM2NEYzVUSXQzOVpJemw4SU16Rllwdnds?=
 =?utf-8?B?YXowTyt4VDF6NFdsYVVJZHRodjdiejk5czZzc0ZnMkV1NWo1cHJFSkRGSWpI?=
 =?utf-8?B?QnFWOUhhVHZ0YWZxL0ZDT0Y0eXFpZFRySTRQSFFaTVA2QTNma0RNTDFZaHV2?=
 =?utf-8?B?eERFL3F6RTZnT0wyWVhQdXQzVlpqM1NZeEt3V3kzNDlJdmVwUjkvYnVpdTdr?=
 =?utf-8?B?bGhWMjlsSHpteDRXUzBhbS9SVHRaeG5McTFqZjZjUzFGMGUyZDQwQ2hOOUpC?=
 =?utf-8?B?c3BhcWpzcWE2TElKWnZhcWY0alh3TXNyNnFtZ0piNEZUczZxa0RDVDhhVytN?=
 =?utf-8?B?RVBDcHFTaUVQS1AvNFBUcnd0Ulk4V3RQL3Fia2dERGVDaWRrZE5NdTZDQU4y?=
 =?utf-8?B?T2YrTXB1WGxYa2Y2NFR2OXAyTzFCcUhjQW9EU2tXc2IzOC8za2haemkxaWUv?=
 =?utf-8?B?V3pmVDdMMGIreHkvL3gvTFlNS3VVZGxnQUhnTTRoYWRGSUJoL2NnV3p1ZFdG?=
 =?utf-8?B?cmZ6K2t3MlhRRkQvZEVkcTFySFFXNWVVSHczazlXNGtpeEtGdTVSNkJwVHZE?=
 =?utf-8?B?QUxVRUFGQTQ4MTFXWjdPb0pUbmxoRWtQL001TmltdjFxd2FIRFo4T1V2SEpq?=
 =?utf-8?B?ckRKMm1lK0lzREJOdE5YUkgzeGxwdHppdXBOcDdpYUpuS2FjMGlESHBBLy92?=
 =?utf-8?B?MHZTa1pGQVV6NHQrWnZlam1yT3BMVzlQRC90K3BaZS9UN2NNS0VLVFZERmNZ?=
 =?utf-8?B?RVhPYXBpUWlrMGpGZHMyaXlQMjRIN0lLaUJCVVg4TGxTenFNVCt5dHVPaVN3?=
 =?utf-8?B?OG9DajBlOUFQR2w5SUo1ZzNmdHdSNEtGT1IraktTUkM1TXhGV2VaZm5xM1Z4?=
 =?utf-8?B?UkJlR3BzcmE2aFlNZENYWDFnejVPOEJDR2tKZ05ucGI1UlBaNWtMeVhJa2ZK?=
 =?utf-8?B?Yk5Cb1BoeFFVRzJiYlVoN2xjRi90bXYrcTMzT1E5S3RwMTQ1cmZPcGlJRFUy?=
 =?utf-8?B?YlhPU2lTanc2bFNnVnJISHRROWVjMUk3RHUrMGQyb1NNeVg4eWlSZkx3MTB3?=
 =?utf-8?B?L2dOOEF2bURjeXlwbE4zZHhJbCtqMnV2ZGlYOUpLMjcxa21IKzkwOWNXQjZM?=
 =?utf-8?Q?M/JwbaDRFbBK8zFltyp/trlfW?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 80a97251-18f3-4450-85fe-08dbc9291ce0
X-MS-Exchange-CrossTenant-AuthSource: MN2PR11MB3728.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Oct 2023 00:37:42.0953
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: +19IfD8vwPEnekX7++tmtKr9jcU3dZfI1277OKBlNbkup3q1+2WT1IaYmkd1HMMiianmhV1kDCZqj+X/IhIONA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB8504
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 09 Oct 2023 17:38:22 -0700 (PDT)
X-Spam-Level: **


On 2023/10/3 03:06, Ingo Molnar wrote:
> The English in this commit is *atrocious*, both in the changelog and in
> the comments - how on Earth did 'Posion' typo and half a dozen other
> typos and bad grammar survive ~3 iterations and a Reviewed-by tag?? The
> version below fixes up the worst, but I suspect that's not the only
> problem with this patch...

Many thanks for your attention and fixes up, Ingo.

I’d like to introduce more background of this patch.

Memory errors don’t happen very often, especially the severity is fatal.
 However, in large-scale scenarios, such as data centers, it might still
happen.  For some MCE fatal error cases, the kernel might call
mce_panic() to terminate the production kernel directly, but not try to
make the kernel survive via memory_failure() handling.  Unfortunately,
the capture kernel will panic for the same reason if it touches the
error memory again.  The consequence is that only an incomplete vmcore
is left for sustaining engineers, it’s a big headache for them to make
clear what happened in the past.


We had considered 3 solutions and finally chose the last one.

1. When the capture kernel boots up, re-scans the MCE banks to check if
   there are fatal errors, set the PG_hwpoison flag for each error
   pages.
   We can foresee this solution is heavy.  It needs to find the struct
   page of error pages from old memory and set the flag.  Looks like we
   need to remake the wheel, so we gave up it.

2. Replace the function copy_to_iter() at __copy_oldmem_page() with the
   function _copy_mc_to_iter(), which is a #MC safe version.
   This solution is lightweight but has following drawbacks:

   1) Such issues are quite rare events; we don’t want to use a #MC safe
      copy to accommodate it. Especially, if the problem can be deal
      with by MCE handling rather than touching the Kdump stuff.

   2) The #MC safe copy is conditionally, whether it can fix the #MC
      error depends on MCE handling can reach the fixup_exception()
      function at do_machine_check().  However, in fatal error case, it
      might invoke mce_panic() to crash the capture kernel earlier than
      fixing up the error.

3. The solution in this patch overcomes all above drawbacks.  It set the
   flag just before the production kernel calls panic(), which would not
   introduce additional overhead in capture kernel or conflict with
   other hwpoision-related code in production kernel.  Furthermore, it
   leverages the already existing mechanisms to fix the issue as much as
   possible, the code changes are also lightweight.


To verify the fix is not difficult.  The issue can be simulated by
ras-tools
(https://git.kernel.org/pub/scm/linux/kernel/git/aegl/ras-tools.git),
"copyout" test case.  It can inject a fatal memory error in kernel space
via APEI ENIJ interface (need hardware platform support), and then it
touches the error page to produce the issue.  The patch has been
validated by this tool.

Any idea is welcome!

Best Regards,
Zhiquan