From:   Eric DeVolder <eric.devolder@oracle.com>
To:     linux-kernel@vger.kernel.org, x86@kernel.org,
        kexec@lists.infradead.org, ebiederm@xmission.com,
        dyoung@redhat.com, bhe@redhat.com, vgoyal@redhat.com
Cc:     tglx@linutronix.de, mingo@redhat.com, bp@alien8.de,
        dave.hansen@linux.intel.com, hpa@zytor.com,
        nramas@linux.microsoft.com, thomas.lendacky@amd.com,
        robh@kernel.org, efault@gmx.de, rppt@kernel.org, david@redhat.com,
        sourabhjain@linux.ibm.com, konrad.wilk@oracle.com,
        boris.ostrovsky@oracle.com, eric.devolder@oracle.com
Subject: [PATCH v20 6/8] crash: change crash_prepare_elf64_headers() to for_each_possible_cpu()
Date:   Fri, 17 Mar 2023 17:21:27 -0400
Message-Id: <20230317212128.21424-7-eric.devolder@oracle.com>
In-Reply-To: <20230317212128.21424-1-eric.devolder@oracle.com>
References: <20230317212128.21424-1-eric.devolder@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?J/xzcdYAsT2roT1IVtsKHRfNJUWTJ7kZ5pQq5YcJtmR3XTFNV3a2rADrGr9i?=
 =?us-ascii?Q?GJ9tio2Cks45XNZLR5yrsem7G/jQQ1khAocByUwbQstGtd0xP7NbAw7/vkO8?=
 =?us-ascii?Q?+VCgx8ceHWxOE9/HsD7WmZ4p99pNO5FlwONCDMnHE5H7ogR8t3uMbrWyxij/?=
 =?us-ascii?Q?ncmuOQMCjJ4AEZXl3TyC41Bx/iyKFpJSYpNz0VTk0XRKIU+batQnMO25e6To?=
 =?us-ascii?Q?fmtZAjdGy7QRAGYGmY/zeZRKiJABS5gOnGGHZifEDT9VmdTBz+psSJzcUvSl?=
 =?us-ascii?Q?nTEPu/9ghG4GnoKUqO5mGav4oNFsxNf97UYJznD3qG399Fu5+P7Woe7SGtGz?=
 =?us-ascii?Q?SiU/Ct8Z9RZehjdV9qE1sj5oUHh4NU91mDEorNUWTzO/x7pWpmHFvpd4/wSL?=
 =?us-ascii?Q?Y8oMcyCebmNC0NSp583apisFBNaXY1JD/+RGP+vu9dhQxsSB9AjLRdeyCM1Q?=
 =?us-ascii?Q?o0xQHPwF7so4jgnbmnPyrvc7xrLg2Q5eI6laTYG745MSr8Cob3qtfM5H40KC?=
 =?us-ascii?Q?TfnxoihUCVLMkJ+QHBxhjG99FUk057Gafn0Scm8gcX4kW2Z8j223gPWS0Hpe?=
 =?us-ascii?Q?tKPwPVqENUV5QhmPfAXTaNngffmHxTovIUUE0VDDTmrdRyflGjFuRCT7zbNk?=
 =?us-ascii?Q?ZVExOyBXwvXL9RA8XK35mtOqXhUILTz2nWNSb6VNVmBKn1p63XChs9jlf2X8?=
 =?us-ascii?Q?suJtlmLnQhJjsajIHcIcyymq6uPIMXdH0ucmtRdQaCiUTFjX+XTEfvB08NvJ?=
 =?us-ascii?Q?trGJipesgs+xJXH1sRMG5BCZF1IZJJTPSd5Q4aa12Os5ekltPhIFe+jf6t5k?=
 =?us-ascii?Q?NFu9dw/rguDu0hYtzNYmoaboD+OQj7qEGkLvPk2bSWEwNZg9l6mkajb6IdPA?=
 =?us-ascii?Q?d8EDHiquD1ACqie9ju5z/xTZwwwWOu7cRvxXcLDDuRO7LTS9n00AMysHCvK3?=
 =?us-ascii?Q?W3BFiA/YefbTj+C8gmEj5am4d00MUVp6k5wj7IzKqAQHpEQL06ISJnbl1pDN?=
 =?us-ascii?Q?J7rkvPlGvGmrKtOg7mQipNQd2IRekrlnAjKhbvFiJY5D2OtEkYqb0r852acd?=
 =?us-ascii?Q?wGVCR88nce4kyUlf5M/+e00y0Jzj0/uf6/9FhNPTGxhyJWdnT3zTy1icd5F2?=
 =?us-ascii?Q?Mnjs+8qVzzC1vu/N5qugXj8/+hKQJ6qMhkCeibn/Gplstm7BdyT5BOzaXCrl?=
 =?us-ascii?Q?4RexQPsFgqrcR8IsvPkjTyzQRPu2Q9h2i+JW4XQ4GJ6sNKW0DWMpRwP6C7Qd?=
 =?us-ascii?Q?46NYEG54MhaK1FSooanoYhQArZSFhxvkLuGYyPQTOdYodU7b5L/IDcaMSENU?=
 =?us-ascii?Q?um/yTaJeGh5ce7jHetrtA1hIv4vCTBCbBzJK6td334Mi4Ujj9Yadmxsb9fd+?=
 =?us-ascii?Q?8mj6wL01+frZ7OhkMi8zGqHecjfzYduc0iOkZvnWBxXfPQRj2twqtI4SUCps?=
 =?us-ascii?Q?/rbehSL9gCKgityC28ymSMHSJxFwubA16hF5DJeIMUZsjgBlKj95ZuU7Oa3T?=
 =?us-ascii?Q?MoGrQepzHuBCnM/z7qpyF2BIA2ju2bgXZBC3PQMtxBJabY2uN2dNyme5TF9+?=
 =?us-ascii?Q?ScKwYKpooSebRwFEplZDmw7F0XXyioD3+MD2cMLDmxlSmt7lbcMEBPuJ51vE?=
 =?us-ascii?Q?CA=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?4gDFUtdc/FSrySf/pWkgB7TWyADbzmI6H4MNTk5fBt8SZg/8mHDldPHnfIpC?=
 =?us-ascii?Q?kdlnaYw51tSiLpilhOvM5aYVg1qm4aEQ4NJHgpi27/yS8XpXlbKr8rUILAGv?=
 =?us-ascii?Q?2iuLR/31N7bHHhChA+g8Wx7i/7uRlX95I09ZKVqdxoqjplDCUHj8VSw0E6OV?=
 =?us-ascii?Q?Evu4BxCHHuhe7j/PqluSf/kpgBjK8BA3EP7bCTEcBxfyz8mnlJfGnMeyOJ5D?=
 =?us-ascii?Q?gXNf/ZieaRAZ5i2+yA1/F0N6Tyjz26fT149u8NU0KePpkiGlwBy6XRYoSdxB?=
 =?us-ascii?Q?Q6wrRyxrs0FcwWi2P3TqA2mcxhSmRRcRz76v+vY7ZQJQ9eohc5VXQw0rZBJZ?=
 =?us-ascii?Q?jpO0gQptZOg8WhzWh8iZFdP3h9T44yUZxbWPLiLUSfoeVKGJ0gTZyLZAf6w1?=
 =?us-ascii?Q?BQvcOsA6+zvaK2hScwYWNo5oKqFktdAUCRIknDuRKxrzDTm2jE+KEWLZLAyS?=
 =?us-ascii?Q?CkKCdBT3bacWPhDEYkJECo8iMe9xesbNbl3eZe0zRTZDVTaUIPqB1n91NGqI?=
 =?us-ascii?Q?/OUl+Yb6tUjiPSVsn3hGS6xVqCKz0Y0fbzviKifeAtOGn8cKZAjR2mPut/0A?=
 =?us-ascii?Q?XYiOekRH6SICSj5ILGuANCXDT5SsDpfDKYmwgyArP6T1MsJ5PXI265cupR0V?=
 =?us-ascii?Q?AJKDT0q9eTwHS1NBqfTGBmKNZIH5OM63GP5E1mmtCm81j8dl3bhOx7hBJHO2?=
 =?us-ascii?Q?aqLGhb18TiIx0GbaAR05rhzGMVEz4RwdXOEXpx4T0ot/XFPQN0f0NjHzrVud?=
 =?us-ascii?Q?f8mgpqY/iHI60ZH0vMAEOeI3QSXVmSE8WUvSPneWCj5ojbzDEnLE0HuewAKL?=
 =?us-ascii?Q?hLfJu+W3Egzx3vqi0sbjgL7lWAcYJhKhColHp34GGZQFTMY7Q3RydeKolhq/?=
 =?us-ascii?Q?ktMMZ0oUk94TETDF9svI4WrDDtkUF0o3Vi9+XMvu7Eq8e/3cJ5eV6Zw4HkMk?=
 =?us-ascii?Q?y0gMeWKKxGNVGdvlUon8/8m5BCBg+uEawCRrpaY0c9sAUg9wMf+DExZ6/CRi?=
 =?us-ascii?Q?pFJt74ekPAsTcgzpL+HoRZ+6hfjQPMk70El/1zJh4l1ct1GtLx1Pd741l9rY?=
 =?us-ascii?Q?/FcWBf/GBx+w6JgeiHeumTGf1iYEhTG0ch6JrlyB1301LcFLJ6UF1VReqKsv?=
 =?us-ascii?Q?t5VCnXuEKbq9CEUr7iMew269TgPZ9yUm/kmGsjJgUwY3soWJUGwpPtw=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f679fa2a-3d6e-414e-4b78-08db272da280
X-MS-Exchange-CrossTenant-AuthSource: CO1PR10MB4531.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Mar 2023 21:21:55.8260
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: h8Ejamc+RWZFE2meZ0BR5Rip9MyodBx0MrmZE9BQXZyNqpNyU5ZlDf+hwGSz7qKCakf0zdsSmXACPffxifTeprIp8sjNEHjeM773OjGkAKg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5901
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22
 definitions=2023-03-17_17,2023-03-16_02,2023-02-09_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 phishscore=0
 mlxscore=0 malwarescore=0 mlxlogscore=999 spamscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303150002
 definitions=main-2303170150
X-Proofpoint-GUID: Dhls-A_tCITodXZ66z23di2RPLG2I2UL
X-Proofpoint-ORIG-GUID: Dhls-A_tCITodXZ66z23di2RPLG2I2UL
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

The function crash_prepare_elf64_headers() generates the elfcorehdr
which describes the cpus and memory in the system for the crash kernel.
In particular, it writes out ELF PT_NOTEs for memory regions and the
cpus in the system.

With respect to the cpus, the current implementation utilizes
for_each_present_cpu() which means that as cpus are added and removed,
the elfcorehdr must again be updated to reflect the new set of cpus.

The reasoning behind the change to use for_each_possible_cpu(), is:

- At kernel boot time, all percpu crash_notes are allocated for all
  possible cpus; that is, crash_notes are not allocated dynamically
  when cpus are plugged/unplugged. Thus the crash_notes for each
  possible cpu are always available.

- The crash_prepare_elf64_headers() creates an ELF PT_NOTE per cpu.
  Changing to for_each_possible_cpu() is valid as the crash_notes
  pointed to by each cpu PT_NOTE are present and always valid.

Furthermore, examining a common crash processing path of:

 kernel panic -> crash kernel -> makedumpfile -> 'crash' analyzer
           elfcorehdr      /proc/vmcore     vmcore

reveals how the ELF cpu PT_NOTEs are utilized:

- Upon panic, each cpu is sent an IPI and shuts itself down, recording
 its state in its crash_notes. When all cpus are shutdown, the
 crash kernel is launched with a pointer to the elfcorehdr.

- The crash kernel via linux/fs/proc/vmcore.c does not examine or
 use the contents of the PT_NOTEs, it exposes them via /proc/vmcore.

- The makedumpfile utility uses /proc/vmcore and reads the cpu
 PT_NOTEs to craft a nr_cpus variable, which is reported in a
 header but otherwise generally unused. Makedumpfile creates the
 vmcore.

- The 'crash' dump analyzer does not appear to reference the cpu
 PT_NOTEs. Instead it looks-up the cpu_[possible|present|onlin]_mask
 symbols and directly examines those structure contents from vmcore
 memory. From that information it is able to determine which cpus
 are present and online, and locate the corresponding crash_notes.
 Said differently, it appears that 'crash' analyzer does not rely
 on the ELF PT_NOTEs for cpus; rather it obtains the information
 directly via kernel symbols and the memory within the vmcore.

(There maybe other vmcore generating and analysis tools that do use
these PT_NOTEs, but 'makedumpfile' and 'crash' seems to be the most
common solution.)

This change results in the benefit of having all cpus described in
the elfcorehdr, and therefore reducing the need to re-generate the
elfcorehdr on cpu changes, at the small expense of an additional
56 bytes per PT_NOTE for not-present-but-possible cpus.

On systems where kexec_file_load() syscall is utilized, all the above
is valid. On systems where kexec_load() syscall is utilized, there
may be the need for the elfcorehdr to be regenerated once. The reason
being that some archs only populate the 'present' cpus in the
/sys/devices/system/cpus entries, which the userspace 'kexec' utility
uses to generate the userspace-supplied elfcorehdr. In this situation,
one memory or cpu change will rewrite the elfcorehdr via the
crash_prepare_elf64_headers() function and now all possible cpus will
be described, just as with kexec_file_load() syscall.

Suggested-by: Sourabh Jain <sourabhjain@linux.ibm.com>
Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
Reviewed-by: Sourabh Jain <sourabhjain@linux.ibm.com>
---
 kernel/crash_core.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/crash_core.c b/kernel/crash_core.c
index ec1d68569f6b..30ba30cadf2f 100644
--- a/kernel/crash_core.c
+++ b/kernel/crash_core.c
@@ -364,8 +364,8 @@ int crash_prepare_elf64_headers(struct crash_mem *mem, int need_kernel_map,
 	ehdr->e_ehsize = sizeof(Elf64_Ehdr);
 	ehdr->e_phentsize = sizeof(Elf64_Phdr);
 
-	/* Prepare one phdr of type PT_NOTE for each present CPU */
-	for_each_present_cpu(cpu) {
+	/* Prepare one phdr of type PT_NOTE for each possible CPU */
+	for_each_possible_cpu(cpu) {
 		phdr->p_type = PT_NOTE;
 		notes_addr = per_cpu_ptr_to_phys(per_cpu_ptr(crash_notes, cpu));
 		phdr->p_offset = phdr->p_paddr = notes_addr;
-- 
2.31.1