Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 24 Oct 2022 17:10:01 +0800
From:   Baoquan He <bhe@redhat.com>
To:     Eric DeVolder <eric.devolder@oracle.com>,
        Sourabh Jain <sourabhjain@linux.ibm.com>
Cc:     linux-kernel@vger.kernel.org, x86@kernel.org,
        kexec@lists.infradead.org, ebiederm@xmission.com,
        dyoung@redhat.com, vgoyal@redhat.com, tglx@linutronix.de,
        mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com,
        hpa@zytor.com, nramas@linux.microsoft.com, thomas.lendacky@amd.com,
        robh@kernel.org, efault@gmx.de, rppt@kernel.org, david@redhat.com,
        konrad.wilk@oracle.com, boris.ostrovsky@oracle.com
Subject: Re: [PATCH v12 3/7] crash: add generic infrastructure for crash
 hotplug support
Message-ID: <Y1ZWaSeGk53QqZHX@MiWiFi-R3L-srv>
References: <20220909210509.6286-1-eric.devolder@oracle.com>
 <20220909210509.6286-4-eric.devolder@oracle.com>
 <8cc31b22-a061-d07c-77dc-c555b8b35af3@linux.ibm.com>
 <97f2daae-f34a-86bb-6d28-8aa8314321bc@oracle.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <97f2daae-f34a-86bb-6d28-8aa8314321bc@oracle.com>
Precedence: bulk

Hi Eric, Sourabh,

On 10/07/22 at 02:14pm, Eric DeVolder wrote:
> 
> 
> On 10/3/22 12:51, Sourabh Jain wrote:
> > Hello Eric,
> > 
> > On 10/09/22 02:35, Eric DeVolder wrote:
......
> > > +static void handle_hotplug_event(unsigned int hp_action, unsigned int cpu)
> > > +{
> > > +??? /* Obtain lock while changing crash information */
> > > +??? mutex_lock(&kexec_mutex);
> > > +
> > > +??? /* Check kdump is loaded */
> > > +??? if (kexec_crash_image) {
> > > +??????? struct kimage *image = kexec_crash_image;
> > > +
> > > +??????? if (hp_action == KEXEC_CRASH_HP_ADD_CPU ||
> > > +??????????? hp_action == KEXEC_CRASH_HP_REMOVE_CPU)
> > > +??????????? pr_debug("crash hp: hp_action %u, cpu %u\n", hp_action, cpu);
> > > +??????? else
> > > +??????????? pr_debug("crash hp: hp_action %u\n", hp_action);
> > > +
> > > +??????? /*
> > > +???????? * When the struct kimage is allocated, it is wiped to zero, so
> > > +???????? * the elfcorehdr_index_valid defaults to false. Find the
> > > +???????? * segment containing the elfcorehdr, if not already found.
> > > +???????? * This works for both the kexec_load and kexec_file_load paths.
> > > +???????? */
> > > +??????? if (!image->elfcorehdr_index_valid) {
> > > +??????????? unsigned char *ptr;
> > > +??????????? unsigned long mem, memsz;
> > > +??????????? unsigned int n;
> > > +
> > > +??????????? for (n = 0; n < image->nr_segments; n++) {
> > > +??????????????? mem = image->segment[n].mem;
> > > +??????????????? memsz = image->segment[n].memsz;
> > > +??????????????? ptr = arch_map_crash_pages(mem, memsz);
> > > +??????????????? if (ptr) {
> > > +??????????????????? /* The segment containing elfcorehdr */
> > > +??????????????????? if (memcmp(ptr, ELFMAG, SELFMAG) == 0) {
> > > +??????????????????????? image->elfcorehdr_index = (int)n;
> > > +??????????????????????? image->elfcorehdr_index_valid = true;
> > > +??????????????????? }
> > > +??????????????? }
> > > +??????????????? arch_unmap_crash_pages((void **)&ptr);
> > > +??????????? }
> > > +??????? }
> > > +
> > > +??????? if (!image->elfcorehdr_index_valid) {
> > > +??????????? pr_err("crash hp: unable to locate elfcorehdr segment");
> > > +??????????? goto out;
> > > +??????? }
> > > +
> > > +??????? /* Needed in order for the segments to be updated */
> > > +??????? arch_kexec_unprotect_crashkres();
> > > +
> > > +??????? /* Flag to differentiate between normal load and hotplug */
> > > +??????? image->hotplug_event = true;
> > > +
> > > +??????? /* Now invoke arch-specific update handler */
> > > +??????? arch_crash_handle_hotplug_event(image, hp_action);
> > > +
> > > +??????? /* No longer handling a hotplug event */
> > > +??????? image->hotplug_event = false;
> > > +
> > > +??????? /* Change back to read-only */
> > > +??????? arch_kexec_protect_crashkres();
> > > +??? }
> > > +
> > > +out:
> > > +??? /* Release lock now that update complete */
> > > +??? mutex_unlock(&kexec_mutex);
> > > +}
> > > +
> > > +static int crash_memhp_notifier(struct notifier_block *nb, unsigned long val, void *v)
> > > +{
> > > +??? switch (val) {
> > > +??? case MEM_ONLINE:
> > > +??????? handle_hotplug_event(KEXEC_CRASH_HP_ADD_MEMORY, 0);
> > > +??????? break;
> > > +
> > > +??? case MEM_OFFLINE:
> > > +??????? handle_hotplug_event(KEXEC_CRASH_HP_REMOVE_MEMORY, 0);
> > > +??????? break;
> > > +??? }
> > > +??? return NOTIFY_OK;
> > 
> > Can we pass v (memory_notify) argument to arch_crash_handle_hotplug_event function
> > via handle_hotplug_event?
> > 
> > Because the way memory hotplug is handled on PowerPC, it is hard to update the elfcorehdr
> > without memory_notify args.
> > 
> > On PowePC memblock data structure is used to prepare elfcorehdr for kdump. Since the notifier
> > used for memory hotplug crash handler get initiated before the memblock data structure update
> > happens (as depicted below), the newly prepared elfcorehdr still holds the old memory regions.
> > So if the system crash with obsolete elfcorehdr, makedumpfile failed to collect vmcore.
> > 
> > Sequence of actions done on PowerPC to server the memory hotplug:
> > 
> >  ?Initiate memory hot remove
> >  ????????? |
> >  ????????? v
> >  ?offline pages
> >  ????????? |
> >  ????????? v
> >  ?initiate memory notify call chain
> >  ?for MEM_OFFLINE event.
> >  ?(same is used for crash update)
> >  ????????? |
> >  ????????? v
> >  ?prepare new elfcorehdr for kdump using
> >  ?memblock data structure
> >  ????????? |
> >  ????????? v
> >  ?update memblock data structure
> > 
> > How passing memory_notify to arch crash hotplug handler will help?
> > 
> > memory_notify holds the start PFN and page count, with that we can get
> > the base address and size of hot unplugged memory and can use the same
> > to avoid hot unplugged memeory region to get added in the elfcorehdr..
> > 
> > Thanks,
> > Sourabh Jain
> > 
> 
> Sourabh, let's see what Baoquan thinks.
> 
> Baoquan, are you OK with this request? I once had these parameters to the
> crash hotplug handler and since they were unused at the time, you asked
> that I remove them, which I did.

Sorry to miss this mail. I thought both of you were talking about
somthing, and didn't notice this question to me.

I think there are two ways to solve the issue Sourabh raised:
1) make handle_hotplug_event() get and pass down the memory_notify as
Sourabh said, or the hp_action, mem_start|size as Eric suggested. I
have to admit I haven't carefully checked which one is better.

2) let the current code as is since it's aiming at x86 only. Later
Sourabh can modify code according to his need on ppc. This can give
satisfying why on code change each time.

I personally like the 2nd way, while also like seeing 1st one if the
code change and log is convincing to any reviewer.

> 
> To accommodate this, how about this:
> 
> static void handle_hotplug_event(unsigned int hp_action, unsigned int cpu,
>      unsigned long mem_start, unsigned long mem_size)
> 
> For CPU events, I would just pass zeros for mem_start/size. For memory events,
> I would pass KEXEC_CRASH_HP_INVALID_CPU.
> 
> Thanks,
> eric