Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751105AbcKMXnv (ORCPT ); Sun, 13 Nov 2016 18:43:51 -0500 Received: from aibo.runbox.com ([91.220.196.211]:53380 "EHLO aibo.runbox.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750738AbcKMXnu (ORCPT ); Sun, 13 Nov 2016 18:43:50 -0500 From: "M. Vefa Bicakci" Subject: Re: [PATCH] x86/cpuid: Deal with broken firmware once more To: Boris Ostrovsky References: <20161102122557.qs4rl6mb7n7l7j7p@linutronix.de> <24e69019-60d0-29e7-e31f-c6f00f9ed98a@brocade.com> <58e229e2-91f4-a97f-1b9f-089f48ef994a@brocade.com> <86609338-2b45-ed7e-fb07-99421e43a2f1@brocade.com> <49fe8cc5-0f0f-6cac-7a5c-803e81f5667d@runbox.com> <68840c0b-44c9-ddd8-bfab-f4fd8bacbaf0@oracle.com> Cc: "Charles (Chas) Williams" , Thomas Gleixner , Sebastian Andrzej Siewior , "x86@kernel.org" , LKML , Peter Zijlstra , Borislav Petkov , David Vrabel , Juergen Gross , xen-devel Message-ID: <41978b7b-2880-4ea5-14c3-7185422261e7@runbox.com> Date: Mon, 14 Nov 2016 02:42:43 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <68840c0b-44c9-ddd8-bfab-f4fd8bacbaf0@oracle.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 5054 Lines: 118 On 11/13/2016 09:04 PM, Boris Ostrovsky wrote: > On 11/12/2016 05:05 PM, M. Vefa Bicakci wrote: >> On 11/10/2016 06:31 PM, Boris Ostrovsky wrote: >>> On 11/10/2016 10:05 AM, Charles (Chas) Williams wrote: >>>> >>>> On 11/10/2016 09:02 AM, Boris Ostrovsky wrote: >>>>> On 11/10/2016 06:13 AM, Thomas Gleixner wrote: >>>>>> On Thu, 10 Nov 2016, M. Vefa Bicakci wrote: >>>>>> >>>>>>> I have found that your patch unfortunately does not improve the >>>>>>> situation >>>>>>> for me. Here is an excerpt obtained from the dmesg of a kernel >>>>>>> compiled >>>>>>> with this patch *as well as* Sebastian's patch: >>>>>>> [ 0.002561] CPU: Physical Processor ID: 0 >>>>>>> [ 0.002566] CPU: Processor Core ID: 0 >>>>>>> [ 0.002572] [Firmware Bug]: CPU0: APIC id mismatch. Firmware: >>>>>>> ffff CPUID: 2 >>>>>> So apic->cpu_present_to_apicid() gives us a completely bogus APIC id >>>>>> which >>>>>> translates to a bogus package id. And looking at the XEN code: >>>>>> >>>>>> xen_pv_apic.cpu_present_to_apicid = xen_cpu_present_to_apicid, >>>>>> >>>>>> and xen_cpu_present_to_apicid does: >>>>>> >>>>>> static int xen_cpu_present_to_apicid(int cpu) >>>>>> { >>>>>> if (cpu_present(cpu)) >>>>>> return xen_get_apic_id(xen_apic_read(APIC_ID)); >>>>>> else >>>>>> return BAD_APICID; >>>>>> } >>>>>> >>>>>> So independent of which present CPU we query we get just some random >>>>>> information, in the above case we get BAD_APICID from >>>>>> xen_apic_read() not >>>>>> from the else path as this CPU _IS_ present. >>>>>> >>>>>> What's so wrong with storing the fricking firmware supplied APICid as >>>>>> everybody else does and report it back when queried? >>>>> By firmware you mean ACPI? It is most likely not available to PV guests. >>>>> How about returning cpu_data(cpu).initial_apicid? >>>>> >>>>> And what was the original problem? >>>> The original issue I found was that VMware was returning a different set >>>> of APIC id's in the ACPI tables than what it advertised on the CPU's. >>>> >>>> http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1266716.html >>> For Xen, we recently added a6a198bc60e6 ("xen/x86: Update topology map >>> for PV VCPUs") to at least temporarily work around some topology map >>> problems that PV guests have with RAPL (which I think is what Vefa's >>> problem was). >> Hello Boris, >> >> (Sorry for the delay!) >> >> It appears that the problem is a bit different compared to the one >> corrected by a6a198bc60e6, because my kernel tree -- based on 4.8.6 -- >> already includes the -stable backport of that commit, i.e. >> 88540ad0820ddfb05626e0136c0e5a79cea85fd1 >> >> The patch I included in my previous e-mail (dated 2016-11-10) corrects >> root cause of the issue I am having with 4.8.6. Sebastian's original >> patch adding error checking to the RAPL module prevents the RAPL module >> from causing a kernel oops without my patch. > > I don't see any messages from you on that date. Can you provide a link > to it (and to Sebastian's patch)? > > (BTW, generally it's a good idea to copy xen-devel list on any > Xen-related issues). As I explain below, it turns out that my issue was 'only' a kernel configuration issue. For reference, I had unknowingly solved my kernel-configuration-induced issue via the patch at: https://marc.info/?l=linux-kernel&m=147875027314638&w=2 Sebastian's patch (which adds error handling to the RAPL module) is at: https://marc.info/?l=linux-kernel&m=147739814217598&w=2 >> The issue I am experiencing is caused by the boot-up code in the >> 'init_apic_mappings' function switching the APIC ops structure from >> Xen's structure to a no-op structure by calling the 'apic_disable' >> function. Please let me know if I can clarify or elaborate. > > apic_disable() is only invoked if there is no APIC present (i.e. > detect_init_APIC() returns a non-zero value) and I don't think this can > happen. Is your CPUID[1].edx[9] not set? I found out that my domU kernels invoke the 'apic_disable' function because CONFIG_X86_MPPARSE was not enabled in my kernel configuration, which would cause the 'smp_found_config' bit to be unset at boot-up. This would cause 'init_apic_mappings' to call 'apic_disable', which would cause Xen's 'apic' ops structure pointer to be replaced with the no-op APIC ops structure's pointer. The use of the no-op APIC ops structure would in turn cause invalid virtual CPU package identifiers to be generated. Invalid CPU package identifiers would in turn cause the RAPL module to produce a kernel oops due to potentially missing error handling. It looks like I have been ignoring the following kernel warning which I should have noticed a long time ago: MPS support code is not built-in. Using acpi=off or acpi=noirq or pci=noacpi may have problem To all on this e-mail thread, I learned a bit through this exercise, but I have also taken a lot of everyone's time and created quite a bit of e-mail traffic because of a kernel configuration issue on my end. My apologies. Vefa