Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Message-ID: <79ad6be3-a817-b6bf-b32b-74b80a512c14@oracle.com>
Date:   Tue, 31 May 2022 17:22:40 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.3.1
Subject: Re: [PATCH v8 0/7] crash: Kernel handling of CPU and memory hot
 un/plug
Content-Language: en-US
To:     David Hildenbrand <david@redhat.com>,
        Sourabh Jain <sourabhjain@linux.ibm.com>,
        linux-kernel@vger.kernel.org, x86@kernel.org,
        kexec@lists.infradead.org, ebiederm@xmission.com,
        dyoung@redhat.com, bhe@redhat.com, vgoyal@redhat.com
Cc:     tglx@linutronix.de, mingo@redhat.com, bp@alien8.de,
        dave.hansen@linux.intel.com, hpa@zytor.com,
        nramas@linux.microsoft.com, thomas.lendacky@amd.com,
        robh@kernel.org, efault@gmx.de, rppt@kernel.org,
        konrad.wilk@oracle.com, boris.ostrovsky@oracle.com
References: <20220505184603.1548-1-eric.devolder@oracle.com>
 <311b0834-c675-fd15-8184-82b122f4a9cc@linux.ibm.com>
 <bffb3d9e-1946-f4b6-d58c-9c44bc0bee26@oracle.com>
 <94fba107-a425-7cf6-2a7b-0562c2dcfce4@linux.ibm.com>
 <5358be66-e545-4482-bad6-00d3d53aac8a@redhat.com>
From:   Eric DeVolder <eric.devolder@oracle.com>
In-Reply-To: <5358be66-e545-4482-bad6-00d3d53aac8a@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Ry8vTEFzejhxWXhFQnlnalhjLzExWjJJSEpwMmRCUG9ySUJJcDdEUFdEYkVp?=
 =?utf-8?B?K2J6N1cyMnFKTHZ0NXBZajFLektSMEtUeVg4REpZSDBqTjNqQ0pZczJOU1FI?=
 =?utf-8?B?ZVkyMUEwa1ZVWjJoa1FRTHNBU3BIcVJNNjc1MFlmSC9QbE02Um5YTlFjZjdQ?=
 =?utf-8?B?WU1IZHNlSWNzejMreW1UUzUvaVVqK2dIZG5UV2VaYm95OEUwV2NOY1lLekYx?=
 =?utf-8?B?cXV1MmJ5czJRSDkrMXpMU2Fuc1Ezdm9jeG4xUGh6NlhlYUJlWGFxb0p6YVRI?=
 =?utf-8?B?WHVHdDdnZlhDRnVYNGd0MzF6eGZxODJtbC9FVkFNK1ByZWNFTXFkMVVXdXU5?=
 =?utf-8?B?Y2pFL25jbEMyQjUvbjhqKzIvS2ZveTFkTFZrblFUN1pRbEVuTnI1ZXV3d0pY?=
 =?utf-8?B?dkJDMTVORkpJMERXaU43VkVpbEJpUEs3RmxxMTlIQ0UvQzRadGpWTHJ5bWE3?=
 =?utf-8?B?NnFpZUJUUzVWaWdpWDhDMGlRQ2wyOGx3UzdaZElMeFBwUnlwd0dPaTlibHA2?=
 =?utf-8?B?SzlreTlvS3BPSVovdXNHSDROb0F0bS9OREVrZmwzMC9yb2lhWXdFRzBGMk9m?=
 =?utf-8?B?TWRzczhlN3p1c000UnpQdERtTVBWOGo1ZWdaQjI3bHVnRllQSG9keXNveE5Z?=
 =?utf-8?B?YU0yWHRGNkVTN2I0Smk2MEVySlNPODJHc1ZGZVVjNktpWSt4cFFXb3NaNHZi?=
 =?utf-8?B?OVQ5ZEx3QUVZMW1HZ1ExTXpxSy9aL3Q0ZWg3VmxwRWlxTzdKNzM3VkM4Yjh3?=
 =?utf-8?B?RGdqSWtrTG4wNzREcDVNWlNsa1oycitsZzFuVTB2MC9ra1RXQWlwZFNpSXpY?=
 =?utf-8?B?ZlgwMWtUdmdUdjhTb0ZSSnkvb25GeVNlOWdhT01RR0RmamxsKzBoTm9pN253?=
 =?utf-8?B?cEVNdFBBMVZycm51ajVXdlZ4RFlDMU1PRjlwdDkxZDBUSCtYRUV4ZjdpMGs1?=
 =?utf-8?B?bWdDWDlKcDJZSHZTT2hIZ0JtWWNqUEJ2NjJreGpRazNmaVhJTDFBK05FcE54?=
 =?utf-8?B?eG9SZ0Z5QjYxNHhVaXg3RDFOOGExdGNYYWJjOEdJUC9IanRicWNtM1VhbW5y?=
 =?utf-8?B?eEpXU0NVejl1dHJ3ckE4ZkszdkEvKzdvWW90d0dVSSs4aHVhZnRiNUV6SHFM?=
 =?utf-8?B?ZndZT3ZnTDl2ckwvSmZTWFdpMnRmUjVyQTdQNFoxbDc0RXFieGhBWk1xWDYv?=
 =?utf-8?B?UlBPd2YvUHRTY1d1K20xbkFSSlJ0RHNpdmtTT0ROSWpqMWxpNnVnWGU0QW9O?=
 =?utf-8?B?UGZOKzNMMTBOMm5aaFpwWVE2c0NiUzI1aFA5eFZRaU9qWXlHU3BjeFZFaTd1?=
 =?utf-8?B?UVFrRHFvQncwbGVoaWtuQlRUZG9XZWFyY3k2dDlsemM2bTB6ZzZXWkI5UnZT?=
 =?utf-8?B?UWdsTFJjbnVWMW0raThCYm91QWFCcURBRC9xRWI4ZWNFNW9ueVZBMzBJSGJp?=
 =?utf-8?B?cHdyUFVoS1RFUmpnby81Y0Y1RUFpbE5BV1h4NnFES01UMWNHOFUzc1NFZ2tq?=
 =?utf-8?B?dTYzekVDanMzL1JXVmxrNU9FSFBBdmt6NXcxcmN6WHAzMEhjdmxHSFJ5ZVFL?=
 =?utf-8?B?RkpPbzM3bkdrRXowdmRKSWs3RWhyZUdVVi9DYjFzKzZpY0Y3RHp3SGQxNjJq?=
 =?utf-8?B?OHJ4T1d2YVF1NXc2UmNKbWdDMGFrYlJuOUx5emRkeE9jSEpxNXhXMzRuMTda?=
 =?utf-8?B?S3BlM0UvK2pQVjhVZC8reVF6Sno5R2szcXgrenpnWXZNaXV6WkZqemNFeUxP?=
 =?utf-8?B?ait0ZGE0aGxyTnpDSXA0TlVxdVgxTzBUdWNJL1hzY0hrRklJRDN4Nm5mYWF4?=
 =?utf-8?B?NjVKOVNwK0VySVp3bVcxdWlzUFNaemtZenlwczliTFhkcHEyQ2VIU01BMTZC?=
 =?utf-8?B?SVNyUk5BTlM4ZzhSaGpTcWtNRWlXeW1RYzgxcGtYbDgwTFNic0FoYUExUkJI?=
 =?utf-8?B?VGtBZW5tUTdKR0VCMEFYODk4SjgwTkIzdmdER3oyUjZtekx2cWk2YllFRGtv?=
 =?utf-8?B?ZkJqanY2SXFMSERSTXJhOHhvQmlaTWp3K1pJSXowMTNFenJsVGNtTGt4TjN2?=
 =?utf-8?B?ZVdZaHJYNmxGTEw3MFgrTFNGNUVZOEZpcVVoVGt2UUh2WTByMXdleU55VllM?=
 =?utf-8?B?dExCYzk5Q2RKRDVmM2FuOGNpRCtiQkc5UHgrbnk2aWJWak4ybE5tZzVYRU45?=
 =?utf-8?B?aUo0cXM0REtzUy9HWXJ0Y3JKc1ZhSlNid1NYV0FnU1Iwc3JRbDkza1VoWkJ1?=
 =?utf-8?B?dEdCTzhnSE9zUXFHMFpzWlk3aHVySGxOUnBBcmpKZHdKWVhaaE5XNTU4V1Zj?=
 =?utf-8?B?ZE5FTHVBK2hZVnhoTk9pNVQwM0NKUFlHdktlUU0yTW9rNTZvSmpQU3ZZZ2ds?=
 =?utf-8?Q?ea6FSUiK3TychRGGHNYsuIxRfpn6YnDYsvCSK?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 195d77cd-d9d6-4190-ef7f-08da4354170b
X-MS-Exchange-CrossTenant-AuthSource: CO1PR10MB4531.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2022 22:22:47.1362
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Vqh4jSXyB1euh3WBY3r4fbfC+PR+5cJVwTJS7PkOlZiWVl4YXGMkahinWdRXMIjJTivD4gpDTwMDyWK/Ro6mdXoYbyeeraJSZIIJ9aVAeec=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR10MB6118
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.874
 definitions=2022-05-31_08:2022-05-30,2022-05-31 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0
 suspectscore=0 spamscore=0 phishscore=0 malwarescore=0 bulkscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2204290000 definitions=main-2205310097
X-Proofpoint-GUID: g3AclTNAY4kSHlRsx85CWzGnn1XrWADO
X-Proofpoint-ORIG-GUID: g3AclTNAY4kSHlRsx85CWzGnn1XrWADO
X-Spam-Status: No, score=-3.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 5/31/22 08:18, David Hildenbrand wrote:
> On 26.05.22 15:39, Sourabh Jain wrote:
>> Hello Eric,
>>
>> On 26/05/22 18:46, Eric DeVolder wrote:
>>>
>>>
>>> On 5/25/22 10:13, Sourabh Jain wrote:
>>>> Hello Eric,
>>>>
>>>> On 06/05/22 00:15, Eric DeVolder wrote:
>>>>> When the kdump service is loaded, if a CPU or memory is hot
>>>>> un/plugged, the crash elfcorehdr (for x86), which describes the CPUs
>>>>> and memory in the system, must also be updated, else the resulting
>>>>> vmcore is inaccurate (eg. missing either CPU context or memory
>>>>> regions).
>>>>>
>>>>> The current solution utilizes udev to initiate an unload-then-reload
>>>>> of the kdump image (e. kernel, initrd, boot_params, puratory and
>>>>> elfcorehdr) by the userspace kexec utility. In previous posts I have
>>>>> outlined the significant performance problems related to offloading
>>>>> this activity to userspace.
>>>>>
>>>>> This patchset introduces a generic crash hot un/plug handler that
>>>>> registers with the CPU and memory notifiers. Upon CPU or memory
>>>>> changes, this generic handler is invoked and performs important
>>>>> housekeeping, for example obtaining the appropriate lock, and then
>>>>> invokes an architecture specific handler to do the appropriate
>>>>> updates.
>>>>>
>>>>> In the case of x86_64, the arch specific handler generates a new
>>>>> elfcorehdr, and overwrites the old one in memory. No involvement
>>>>> with userspace needed.
>>>>>
>>>>> To realize the benefits/test this patchset, one must make a couple
>>>>> of minor changes to userspace:
>>>>>
>>>>>    - Disable the udev rule for updating kdump on hot un/plug changes.
>>>>>      Add the following as the first two lines to the udev rule file
>>>>>      /usr/lib/udev/rules.d/98-kexec.rules:
>>>>
>>>> If we can have a sysfs attribute to advertise this feature then
>>>> userspace
>>>> utilities (kexec tool/udev rules) can take action accordingly. In
>>>> short, it will
>>>> help us maintain backward compatibility.
>>>>
>>>> kexec tool can use the new sysfs attribute and allocate additional
>>>> buffer space
>>>> for elfcorehdr accordingly. Similarly, the checksum-related changes
>>>> can come
>>>> under this check.
>>>>
>>>> Udev rule can use this sysfs file to decide kdump service reload is
>>>> required or not.
>>>
>>> Great idea. I've been working on the corresponding udev and
>>> kexec-tools changes and your input/idea here is quite timely.
>>>
>>> I have boolean "crash_hotplug" as a core_param(), so it will show up as:
>>>
>>> # cat /sys/module/kernel/parameters/crash_hotplug
>>> N
>>
>> How about using 0-1 instead Y/N?
>> 0 = crash hotplug not supported
>> 1 = crash hotplug supported
>>
>> Also how about keeping sysfs here instead?
>> /sys/kernel/kexec_crash_hotplug
> 
> It's not only about hotplug, though. And actually we care about
> onlining/offlining. Hmm, I wonder if there is a better name for this
> automatic handling of cpu and memory devices.
> 
In the upcoming v9, there is no /sys/kernel/crash/kexec_crash_hotplug; I have sysfs attributes for 
memory blocks and CPUs named 'crash_hotplug' that can be utilized directly in udev rule as 
ATTR{crash_hotplug} to determine if the kernel is handling this for crash kernel update purposes.

Here's the current commit message for that change:

====
crash: memory and CPU hotplug sysfs attributes

This introduces the crash_hotplug attribute for memory and CPUs
for use by userspace.  This change directly facilitates the udev
rule for managing userspace re-loading of the crash kernel.

For memory, this changeset introduces the crash_hotplug attribute
to the /sys/devices/system/memory directory. For example:

  # udevadm info --attribute-walk /sys/devices/system/memory/memory81
   looking at device '/devices/system/memory/memory81':
     KERNEL=="memory81"
     SUBSYSTEM=="memory"
     DRIVER==""
     ATTR{online}=="1"
     ATTR{phys_device}=="0"
     ATTR{phys_index}=="00000051"
     ATTR{removable}=="1"
     ATTR{state}=="online"
     ATTR{valid_zones}=="Movable"

   looking at parent device '/devices/system/memory':
     KERNELS=="memory"
     SUBSYSTEMS==""
     DRIVERS==""
     ATTRS{auto_online_blocks}=="offline"
     ATTRS{block_size_bytes}=="8000000"
     ATTRS{crash_hotplug}=="1"

For CPUs, this changeset introduces the crash_hotplug attribute
to the /sys/devices/system/cpu directory. For example:

  # udevadm info --attribute-walk /sys/devices/system/cpu/cpu0
   looking at device '/devices/system/cpu/cpu0':
     KERNEL=="cpu0"
     SUBSYSTEM=="cpu"
     DRIVER=="processor"
     ATTR{crash_notes}=="277c38600"
     ATTR{crash_notes_size}=="368"
     ATTR{online}=="1"

   looking at parent device '/devices/system/cpu':
     KERNELS=="cpu"
     SUBSYSTEMS==""
     DRIVERS==""
     ATTRS{crash_hotplug}=="1"
     ATTRS{isolated}==""
     ATTRS{kernel_max}=="8191"
     ATTRS{nohz_full}=="  (null)"
     ATTRS{offline}=="4-7"
     ATTRS{online}=="0-3"
     ATTRS{possible}=="0-7"
     ATTRS{present}=="0-3"

With these changes in place, and by using the same attribute
crash_hotplug name, it is possible to efficiently instruct the
udev rule to skip crash kernel reloading.

For example, the following is the proposed udev rule change for RHEL
system 98-kexec.rules (as the first two lines of the rule file):

  # The kernel handles updates to crash elfcorehdr
  ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"

When examined in the context of 98-kexec.rules, the above change
tests if crash_hotplug is set, and if so, it skips the userspace
initiated unload-then-reload of the crash kernel.
=====

Does that work for you?
Eric