From:   Eric DeVolder <eric.devolder@oracle.com>
To:     linux-kernel@vger.kernel.org, x86@kernel.org,
        kexec@lists.infradead.org, ebiederm@xmission.com,
        dyoung@redhat.com, bhe@redhat.com, vgoyal@redhat.com
Cc:     tglx@linutronix.de, mingo@redhat.com, bp@alien8.de,
        dave.hansen@linux.intel.com, hpa@zytor.com,
        nramas@linux.microsoft.com, thomas.lendacky@amd.com,
        robh@kernel.org, efault@gmx.de, rppt@kernel.org, david@redhat.com,
        sourabhjain@linux.ibm.com, konrad.wilk@oracle.com,
        boris.ostrovsky@oracle.com, eric.devolder@oracle.com
Subject: [PATCH v18 0/7] crash: Kernel handling of CPU and memory hot un/plug
Date:   Tue, 31 Jan 2023 17:42:29 -0500
Message-Id: <20230131224236.122805-1-eric.devolder@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?/KS1bAXUx7LhxowDizzzj1Swi44sy8oaYvGhagSc5pcfy71KFeDPSHb5/znX?=
 =?us-ascii?Q?3ERrftJAUQVo0R0l+YL7WJI7rsUDu73FFberW1fgp9QOwJehME629oYlLh2q?=
 =?us-ascii?Q?PrXuEjeYBK+g2zmn4L0yQ5d5VT8G9F3sMsqWvn/eYkCSC0YDzyw5ckD9lurC?=
 =?us-ascii?Q?WI47sBE1k392FdKzCf9FZbJ27wrdQLik941J+WC+WkEHkce/YcTXGbJCynsx?=
 =?us-ascii?Q?z8utvrdszKRmiyUcdYXjNkNsiPu2TaHZAM97pvwwdCobfohsSEF0lk+lzPJz?=
 =?us-ascii?Q?LGj264zPrrNFO8lYLyuCBi8N1COoJY/ipphBTVGEpTnHHVFtAKoQb5w6jkLJ?=
 =?us-ascii?Q?bczT59kcwOhCTbqlgaV+nNOk7GlziHUV9HNN0i9vNVKm/ZCZhGIL4C9dSHYw?=
 =?us-ascii?Q?xy027i+K6mWgDmfdKMOWZjSZw1OdQWdjUDjriB4Gh96uS9gKd7/VLL10wi2T?=
 =?us-ascii?Q?qyQou0BVbqMD0/JSDilitZm/0MJBXVqxrsMUktWz4u779Dt6hu/PnUK32qsL?=
 =?us-ascii?Q?gr4j6vfRA1OsosVg2CJsR+/qJ9T8QVpN/VJVo5GatJJgCuPi5V74o4TG5BYy?=
 =?us-ascii?Q?L5XewbHvxezujjKqj0xvzFnfEwemoqFb9Nir610syP5PTdnfOY6iOpI7sIzT?=
 =?us-ascii?Q?7jp4sZtuxnp9TtEiRWKEfv+iiVBz42qg+uW+vennnC9MgvMvx2lL7sEDZ5Sg?=
 =?us-ascii?Q?1hpDgcaqieMppxTT68XaI9j27cr4m7di+VI36WXkdgt2tpk6M1pMDv+NWlV5?=
 =?us-ascii?Q?1/Aea20btl9U62a0c+RkOgEUCgHQTR+waFnybLArwz3A5RW6a8DLNTYEmQ0S?=
 =?us-ascii?Q?oM5aozVeH7ULRiVnExeplsgUrGD0kVYi49e21oPhQz15F+AJdo2Ac/GD7wI7?=
 =?us-ascii?Q?8EY5nbfNARkoduw74S/3ownVBiFwHBvhuMZkuOc+PhvmlCwQ9zuwmDyWojbQ?=
 =?us-ascii?Q?5VNVTGMdK4Bj/bBAjESmEWMtx7Uvs6+tvAvw7YfvvyYSpvwMPqEbdBqT/qvi?=
 =?us-ascii?Q?oh0BgnYew38i+p1SQ3DB3Gi418cX+ErfFgtfHUbjLI0/1JpP2qq+ZpOhrdtO?=
 =?us-ascii?Q?c121BTyxykyrogxa2IBG2baq5yOtN6/6t3JoKsnYo7zJbyWq43fti92eEN3/?=
 =?us-ascii?Q?6DG8Qly8Md6WXqCRzoexlRax9Hl7Km9o9CmIe3kRaXNdq/VvxS9kzIimECwX?=
 =?us-ascii?Q?xeu1wWhLbddtq4Tys3yaN4ufBLtdO7oZgME8MaWkGjztPEQo6A6omvQQxTZ0?=
 =?us-ascii?Q?z9O3c33tWTk9GnoZpc+ga1W8IoJYoO75UWfSt0ZZf8F5/JXcb0+TfCgPKqcz?=
 =?us-ascii?Q?QlBOdvHAWQnansq5aablumRLjuWugeqnU9Kwi5rNEYUUO0TMskjfP/Zn0rNl?=
 =?us-ascii?Q?m4yDq9m4+4WBrGlM9Hm47M30ulAshLDhYDMpfrnJqWlRAMGGGEqrszSGT11x?=
 =?us-ascii?Q?gr53hMaDhqzbDNKUj7tdFBW7cCxDnCRioV8lnJkG3tqWjj/MYHgQvEqLJxWA?=
 =?us-ascii?Q?3yN56mNWBpKcCq01fDsIF08By5MflhrWtUVzFyo67+NzG9eMzUdmk3oN1ATi?=
 =?us-ascii?Q?xWX0XwzaWroWQIlch5gnqaI8IcVj+8v8/ykpJ1AboC3ZcvDktLNLBpaSzcle?=
 =?us-ascii?Q?wg=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?9JO+3S1Fs9GLxs6V9sa7mfdH7uOG/BvBxQM14oFcoHSTcUURce8uDwJdzKX0?=
 =?us-ascii?Q?hPTa0cXfvyZqhMwSTSn4Q1polbywx2FmU6J3dDj41r0P8IFazYA2T6/IIkra?=
 =?us-ascii?Q?WqmKnGOWj7qxi93a2ogXxdiFXJOy9j5r9okLy+qkkVB5WBveVkiOoYGk/ygA?=
 =?us-ascii?Q?59LAvOTTB3YLAjW8wFijZi+1sDy2EFTQ1+HpC4jeYtcwFyJX73RhcKb88Hjj?=
 =?us-ascii?Q?BT6oRZSYVPLTnqAuygMMsJBJxSoHdZY1ZfR6eXIw9pLCshoc7N6IucWVSg7q?=
 =?us-ascii?Q?ylNs1pG8hJxtU9XLgI/pCZ/JkGqLE4Ko4KBsyQY+qtDychEwa1SupTouM4jS?=
 =?us-ascii?Q?8NHw3nzeCK7MGuKooJPvS/UMVDfwItt1ianPzXLSwj9Rq0vXqnIFb0HtQ09h?=
 =?us-ascii?Q?XwCpKFbUFzktakT4L3wlBONDbkfGylBFpDgYrVA05unCcT3bp74TokuXXw35?=
 =?us-ascii?Q?r74virgsXiLS+EXPx3X1dF2OHAtml8FDhWqpAmO8g5sAIdOfbSRJG7AuMWBe?=
 =?us-ascii?Q?n/7lsWpuRStGyTRTAP/ldJ8P/OmURUf8VLpT3znbJd3kI8eesOPjgEWQr9wa?=
 =?us-ascii?Q?wPZUNXfN8it632ZdNLNAxOICUhgOypjvHLpS+n9227xRF3BF6nTtRb/bX3rR?=
 =?us-ascii?Q?A+/Zf0iLfYTZTvOVbthWW2sY4yqJobaXUYK7u0QF5JSvTfoytql76V6SPbN/?=
 =?us-ascii?Q?cQAjRoOX/iacmc6/6VIBVRo9ayBlLwK9uvUV1ZyGg37iq+bRO62LXDCwIfNL?=
 =?us-ascii?Q?k4nGUQg8Zl87N7/M1fy//xovsMQArqSASRVet/myeJIPmeXRyyZsT0DZOFca?=
 =?us-ascii?Q?+68pQnaZsP0vyHruxgNuFEioa4celyw523hJJxpuDAQ4o3Wr+uRWoj/T8PJV?=
 =?us-ascii?Q?FoMW6krOKvqErgWOr6mm6N//1c+kVyacluI74TQnW4VJBf2yEIwkwbXynPzQ?=
 =?us-ascii?Q?/6B+zCjJPmmTxGpQUoxRjP/SKykeQT6b9zIzYs1uEMtEC3/AGfQ6WadDVE1X?=
 =?us-ascii?Q?f82heLk+JzAi4UNQ5TbKGUPuDiix18Icr59pSJlfgu9HF3bl2OeSPDlOuHCq?=
 =?us-ascii?Q?YihBKMdw5Vir8IuqTXNQRvubjHbHTYQxMsHGh8jWClmTNGep+0MUNuOGo2JD?=
 =?us-ascii?Q?33Yxug1PAZKibYT6qLPKtM4do9sCF7aZKcT6V37h8oJueE5GyYQgkfI=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f9671054-121e-47d9-ba49-08db03dc7827
X-MS-Exchange-CrossTenant-AuthSource: CO1PR10MB4531.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Jan 2023 22:42:45.5533
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: xLwSfqF7gXMsoWfKvlNm3CIRnGOHlU14tghMISOGt+G2+6WXR1gW0Z8dDHRzD5+yzJ0n6I5AGAU7f8XPRDjbRvKLv3RzX2ENx4RKmqjkgvc=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB7241
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1
 definitions=2023-01-31_08,2023-01-31_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0
 suspectscore=0 adultscore=0 mlxscore=0 phishscore=0 mlxlogscore=999
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2212070000 definitions=main-2301310196
X-Proofpoint-ORIG-GUID: vEHEHN_Z2y9GLvEwcPNjSpSn-4LzftC3
X-Proofpoint-GUID: vEHEHN_Z2y9GLvEwcPNjSpSn-4LzftC3
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Once the kdump service is loaded, if changes to CPUs or memory occur,
either by hot un/plug or off/onlining, the crash elfcorehdr must also
be updated.

The elfcorehdr describes to kdump the CPUs and memory in the system,
and any inaccuracies can result in a vmcore with missing CPU context
or memory regions.

The current solution utilizes udev to initiate an unload-then-reload
of the kdump image (eg. kernel, initrd, boot_params, puratory and
elfcorehdr) by the userspace kexec utility. In previous posts I have
outlined the significant performance problems related to offloading
this activity to userspace.

This patchset introduces a generic crash handler that registers with
the CPU and memory notifiers. Upon CPU or memory changes, from either
hot un/plug or off/onlining, this generic handler is invoked and
performs important housekeeping, for example obtaining the appropriate
lock, and then invokes an architecture specific handler to do the
appropriate elfcorehdr update.

In the case of x86_64, the arch specific handler generates a new
elfcorehdr, and overwrites the old one in memory; thus no involvement
with userspace needed.

To realize the benefits/test this patchset, one must make a couple
of minor changes to userspace:

 - Prevent udev from updating kdump crash kernel on hot un/plug changes.
   Add the following as the first lines to the RHEL udev rule file
   /usr/lib/udev/rules.d/98-kexec.rules:

   # The kernel handles updates to crash elfcorehdr for cpu and memory changes
   SUBSYSTEM=="cpu", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"
   SUBSYSTEM=="memory", ATTRS{crash_hotplug}=="1", GOTO="kdump_reload_end"

   With this changeset applied, the two rules evaluate to false for
   cpu and memory change events and thus skip the userspace
   unload-then-reload of kdump.

 - Change to the kexec_file_load for loading the kdump kernel:
   Eg. on RHEL: in /usr/bin/kdumpctl, change to:
    standard_kexec_args="-p -d -s"
   which adds the -s to select kexec_file_load() syscall.

This kernel patchset also supports kexec_load() with a modified kexec
userspace utility. A working changeset to the kexec userspace utility
is posted to the kexec-tools mailing list here:

 http://lists.infradead.org/pipermail/kexec/2022-October/026032.html

To use the kexec-tools patch, apply, build and install kexec-tools,
then change the kdumpctl's standard_kexec_args to replace the -s with
--hotplug. The removal of -s reverts to the kexec_load syscall and
the addition of --hotplug invokes the changes put forth in the
kexec-tools patch.

Regards,
eric
---
v18: 31jan2023
 - Rebased onto 6.2.0-rc6
 - Renamed struct kimage member hotplug_event to hp_action, and
   re-enumerated the KEXEC_CRASH_HP_x items, adding _NONE at 0.
 - Moved to cpuhp state CPUHP_BP_PREPARE_DYN instead of
   CPUHP_AP_ONLINE_DYN in order to minimize window of time CPU
   is not reflected in elfcorehdr.
 - Reworked some of the comments and commit messages to offer
   more of the why, than what, per Thomas Gleixner.

v17: 18jan2023
 - Rebased onto 6.2.0-rc4
 - Moved a bit of code around so that kexec_load()-only builds
   work, per Sourabh.
 - Corrected computation of number of memory region Phdrs needed
   when x86 memory hotplug is not enabled, per Baoquan.

v16: 5jan2023
 https://lkml.org/lkml/2023/1/5/673
 - Rebased onto 6.2.0-rc2
 - Corrected error identified by Baoquan.

v15: 9dec2022
 https://lkml.org/lkml/2022/12/9/520
 - Rebased onto 6.1.0-rc8
 - Replaced arch_un/map_crash_pages() with direct use of
   kun/map_local_pages(), per Boris.
 - Some x86 changes, per Boris.

v14: 16nov2022
 https://lkml.org/lkml/2022/11/16/1645
 - Rebased onto 6.1.0-rc5
 - Introduced CRASH_HOTPLUG Kconfig item to better fine tune
   compilation of feature components, per Boris.
 - Removed hp_action parameter to arch_crash_handle_hotplug_event()
   as it is unused.

v13: 31oct2022
 https://lkml.org/lkml/2022/10/31/854
 - Rebased onto 6.1.0-rc3, which means converting to use the new
   kexec_trylock() away from mutex_lock(kexec_mutex).
 - Moved arch_un/map_crash_pages() into kexec.h and default
   implementation using k/unmap_local_pages().
 - Changed more #ifdef's into IS_ENABLED()
 - Changed CRASH_MAX_MEMORY_RANGES to 8192 from 32768, and it moved
   into x86 crash.c as #define rather Kconfig item, per Boris.
 - Check number of Phdrs against PN_XNUM, max possible.

v12: 9sep2022
 https://lkml.org/lkml/2022/9/9/1358
 - Rebased onto 6.0-rc4
 - Addressed some minor formatting items, per Baoquan

v11: 26aug2022
 https://lkml.org/lkml/2022/8/26/963
 - Rebased onto 6.0-rc2
 - Redid the rework of __weak to use asm/kexec.h, per Baoquan
 - Reworked some comments and minor items, per Baoquan

v10: 21jul2022
 https://lkml.org/lkml/2022/7/21/1007
 - Rebased to 5.19.0-rc7
 - Per Sourabh, corrected build issue with arch_un/map_crash_pages()
   for architectures not supporting this feature.
 - Per David Hildebrand, removed the WARN_ONCE() altogether.
 - Per David Hansen, converted to use of kmap_local_page().
 - Per Baoquan He, replaced use of __weak with the kexec technique.

v9: 13jun2022
 https://lkml.org/lkml/2022/6/13/3382
 - Rebased to 5.18.0
 - Per Sourabh, moved crash_prepare_elf64_headers() into common
   crash_core.c to avoid compile issues with kexec_load only path.
 - Per David Hildebrand, replaced mutex_trylock() with mutex_lock().
 - Changed the __weak arch_crash_handle_hotplug_event() to utilize
   WARN_ONCE() instead of WARN(). Fix some formatting issues.
 - Per Sourabh, introduced sysfs attribute crash_hotplug for memory
   and CPUs; for use by userspace (udev) to determine if the kernel
   performs crash hot un/plug support.
 - Per Sourabh, moved the code detecting the elfcorehdr segment from
   arch/x86 into crash_core:handle_hotplug_event() so both kexec_load
   and kexec_file_load can benefit.
 - Updated userspace kexec-tools kexec utility to reflect change to
   using CRASH_MAX_MEMORY_RANGES and get_nr_cpus().
 - Updated the new proposed udev rules to reflect using the sysfs
   attributes crash_hotplug.

v8: 5may2022
 https://lkml.org/lkml/2022/5/5/1133
 - Per Borislav Petkov, eliminated CONFIG_CRASH_HOTPLUG in favor
   of CONFIG_HOTPLUG_CPU || CONFIG_MEMORY_HOTPLUG, ie a new define
   is not needed. Also use of IS_ENABLED() rather than #ifdef's.
   Renamed crash_hotplug_handler() to handle_hotplug_event().
   And other corrections.
 - Per Baoquan, minimized the parameters to the arch_crash_
   handle_hotplug_event() to hp_action and cpu.
 - Introduce KEXEC_CRASH_HP_INVALID_CPU definition, per Baoquan.
 - Per Sourabh Jain, renamed and repurposed CRASH_HOTPLUG_ELFCOREHDR_SZ
   to CONFIG_CRASH_MAX_MEMORY_RANGES, mirroring kexec-tools change
   by David Hildebrand. Folded this patch into the x86
   kexec_file_load support patch.

v7: 13apr2022
 https://lkml.org/lkml/2022/4/13/850
 - Resolved parameter usage to crash_hotplug_handler(), per Baoquan.

v6: 1apr2022
 https://lkml.org/lkml/2022/4/1/1203
 - Reword commit messages and some comment cleanup per Baoquan.
 - Changed elf_index to elfcorehdr_index for clarity.
 - Minor code changes per Baoquan.

v5: 3mar2022
 https://lkml.org/lkml/2022/3/3/674
 - Reworded description of CRASH_HOTPLUG_ELFCOREHDR_SZ, per
   David Hildenbrand.
 - Refactored slightly a few patches per Baoquan recommendation.

v4: 9feb2022
 https://lkml.org/lkml/2022/2/9/1406
 - Refactored patches per Baoquan suggestsions.
 - A few corrections, per Baoquan.

v3: 10jan2022
 https://lkml.org/lkml/2022/1/10/1212
 - Rebasing per Baoquan He request.
 - Changed memory notifier per David Hildenbrand.
 - Providing example kexec userspace change in cover letter.

RFC v2: 7dec2021
 https://lkml.org/lkml/2021/12/7/1088
 - Acting upon Baoquan He suggestion of removing elfcorehdr from
   the purgatory list of segments, removed purgatory code from
   patchset, and it is signficiantly simpler now.

RFC v1: 18nov2021
 https://lkml.org/lkml/2021/11/18/845
 - working patchset demonstrating kernel handling of hotplug
   updates to x86 elfcorehdr for kexec_file_load

RFC: 14dec2020
 https://lkml.org/lkml/2020/12/14/532
 - proposed concept of allowing kernel to handle hotplug update
   of elfcorehdr
---


Eric DeVolder (7):
  crash: move a few code bits to setup support of crash hotplug
  crash: prototype change for crash_prepare_elf64_headers()
  crash: add generic infrastructure for crash hotplug support
  kexec: exclude elfcorehdr from the segment digest
  kexec: exclude hot remove cpu from elfcorehdr notes
  crash: memory and cpu hotplug sysfs attributes
  x86/crash: add x86 crash hotplug support

 .../admin-guide/mm/memory-hotplug.rst         |   8 +
 Documentation/core-api/cpu_hotplug.rst        |  18 +
 arch/arm64/kernel/machine_kexec_file.c        |   6 +-
 arch/powerpc/kexec/file_load_64.c             |   2 +-
 arch/riscv/kernel/elf_kexec.c                 |   7 +-
 arch/x86/Kconfig                              |  13 +
 arch/x86/include/asm/kexec.h                  |  15 +
 arch/x86/kernel/crash.c                       | 124 ++++++-
 drivers/base/cpu.c                            |  14 +
 drivers/base/memory.c                         |  13 +
 include/linux/crash_core.h                    |   9 +
 include/linux/kexec.h                         |  53 ++-
 kernel/crash_core.c                           | 337 ++++++++++++++++++
 kernel/kexec_file.c                           | 187 +---------
 14 files changed, 595 insertions(+), 211 deletions(-)

-- 
2.31.1