MIME-Version: 1.0
References: <20211106013312.26698-1-ricardo.neri-calderon@linux.intel.com>
 <20211106013312.26698-5-ricardo.neri-calderon@linux.intel.com>
 <CAJZ5v0gemmV1Lz3+9iKz1eiXtkyDc3+4+po4Eidchzk+J2=ceA@mail.gmail.com> <20211130132137.GA25524@ranerica-svr.sc.intel.com>
In-Reply-To: <20211130132137.GA25524@ranerica-svr.sc.intel.com>
From:   "Rafael J. Wysocki" <rafael@kernel.org>
Date:   Tue, 30 Nov 2021 14:32:42 +0100
Message-ID: <CAJZ5v0iXBn1o9ZFzNaYU4ft=JcRfNv7AJ8Sq-9HbBJbp60LpWQ@mail.gmail.com>
Subject: Re: [PATCH 4/7] thermal: intel: hfi: Handle CPU hotplug events
To:     Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Cc:     "Rafael J. Wysocki" <rafael@kernel.org>,
        "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>,
        Daniel Lezcano <daniel.lezcano@linaro.org>,
        Linux PM <linux-pm@vger.kernel.org>,
        "the arch/x86 maintainers" <x86@kernel.org>,
        "open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
        Len Brown <len.brown@intel.com>,
        Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>,
        Aubrey Li <aubrey.li@linux.intel.com>,
        Amit Kucheria <amitk@kernel.org>,
        Andi Kleen <ak@linux.intel.com>,
        Tim Chen <tim.c.chen@linux.intel.com>,
        "Ravi V. Shankar" <ravi.v.shankar@intel.com>,
        Ricardo Neri <ricardo.neri@intel.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Tue, Nov 30, 2021 at 2:22 PM Ricardo Neri
<ricardo.neri-calderon@linux.intel.com> wrote:
>
> On Wed, Nov 24, 2021 at 03:48:49PM +0100, Rafael J. Wysocki wrote:
> > On Sat, Nov 6, 2021 at 2:34 AM Ricardo Neri
> > <ricardo.neri-calderon@linux.intel.com> wrote:

[cut]

> > > +/**
> > > + * intel_hfi_offline() - Disable HFI on @cpu
> > > + * @cpu:       CPU in which the HFI will be disabled
> > > + *
> > > + * Remove @cpu from those covered by its HFI instance.
> > > + *
> > > + * On some processors, hardware remembers previous programming settings even
> > > + * after being reprogrammed. Thus, keep HFI enabled even if all CPUs in the
> > > + * die/package of @cpu are offline. See note in intel_hfi_online().
> > > + */
> > > +void intel_hfi_offline(unsigned int cpu)
> > > +{
> > > +       struct cpumask *die_cpumask = topology_core_cpumask(cpu);
> > > +       struct hfi_cpu_info *info = &per_cpu(hfi_cpu_info, cpu);
> > > +       struct hfi_instance *hfi_instance;
> > > +
> > > +       if (!boot_cpu_has(X86_FEATURE_INTEL_HFI))
> > > +               return;
> > > +
> > > +       hfi_instance = info->hfi_instance;
> > > +       if (!hfi_instance)
> > > +               return;
> > > +
> > > +       if (!hfi_instance->initialized)
> > > +               return;
> > > +
> > > +       mutex_lock(&hfi_lock);
> > > +
> > > +       /*
> > > +        * We were using the core cpumask of @cpu to track CPUs in the same
> > > +        * die/package. Now it is going offline and we need to find another
> > > +        * CPU we can use.
> > > +        */
> > > +       if (die_cpumask == hfi_instance->cpus) {
> > > +               int new_cpu;
> > > +
> > > +               new_cpu = cpumask_any_but(hfi_instance->cpus, cpu);
> > > +               if (new_cpu >= nr_cpu_ids)
> > > +                       /* All other CPUs in the package are offline. */
> > > +                       hfi_instance->cpus = NULL;
> > > +               else
> > > +                       hfi_instance->cpus = topology_core_cpumask(new_cpu);
> >
> > Hmmm.  Is topology_core_cpumask() updated when CPUs go offline and online?
>
> Yes. A CPU going offline is cleared from its siblings' cpumask [1] and its own [2]
> in remove_siblinginfo() via cpu_disable_common(). A CPU going online is set
> in its siblings' cpumask and its own in set_cpu_sibling_map() [3].

OK, so it is necessary to ensure that intel_hfi_offline() will always
run after remove_siblinginfo() so it sees the updated mask.  How do we
ensure that?

> [1]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kernel/smpboot.c?h=v5.16-rc3#n1592
> [2]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kernel/smpboot.c?h=v5.16-rc3#n1617
> [3]. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kernel/smpboot.c?h=v5.16-rc3#n657