Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: FSGSBASE causing panic on 5.9-rc1
From:   Tom Lendacky <thomas.lendacky@amd.com>
To:     Andy Lutomirski <luto@kernel.org>
Cc:     Sean Christopherson <sean.j.christopherson@intel.com>,
        Joerg Roedel <joro@8bytes.org>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Vitaly Kuznetsov <vkuznets@redhat.com>,
        Wanpeng Li <wanpengli@tencent.com>,
        Jim Mattson <jmattson@google.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        X86 ML <x86@kernel.org>,
        "Chang S. Bae" <chang.seok.bae@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Sasha Levin <sashal@kernel.org>,
        Borislav Petkov <bp@alien8.de>,
        Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@kernel.org>
References: <39d57e4d-5db4-f536-6f5e-6b0414c0dccc@amd.com>
 <7dedb0ab-56a6-5d96-577b-21ab1ecdad24@amd.com>
 <CALCETrUArZ04UqJjp2mEheuMpZyPwLeEgGM7mdDdmWxmx=GN4Q@mail.gmail.com>
 <CALCETrWcXgp+GzFNrOKcX740+R-57FgNu3Wt3g=_sRCMYFzbMg@mail.gmail.com>
 <20200820151049.GA26595@sjchrist-ice>
 <362f5a45-b808-25ad-b99b-449e05717c4a@amd.com>
 <CALCETrW8SWgWxDdMXuMPD-A1avztvNcpwLAKKRQ4ojmcz07B6A@mail.gmail.com>
 <45e16c5c-ce99-33da-99c8-ea52ef0945db@amd.com>
Message-ID: <c7f65dbc-316b-efb9-d2c3-c7c212289dce@amd.com>
Date:   Thu, 20 Aug 2020 11:30:50 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.10.0
In-Reply-To: <45e16c5c-ce99-33da-99c8-ea52ef0945db@amd.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from 255.255.255.255 (255.255.255.255) by SN6PR04CA0076.namprd04.prod.outlook.com (2603:10b6:805:f2::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3305.24 via Frontend Transport; Thu, 20 Aug 2020 16:30:51 +0000
X-Originating-IP: [67.79.209.213]
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-HT: Tenant
X-MS-Office365-Filtering-Correlation-Id: 04d61a52-61a1-4585-3bd0-08d8452667d5
X-MS-TrafficTypeDiagnostic: DM6PR12MB4433:
X-Microsoft-Antispam-PRVS: <DM6PR12MB4433C63F9BFE2EA174BB596EEC5A0@DM6PR12MB4433.namprd12.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:843;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: rHxlLDpQTwIBPmSzNzm5zAOSAC5ASYH/jCTr8qcqEHViINOBMelfmI16MYxgzfgKIyeI9bfvkWHbjCnBpEvCN8lZXfkeqfpmhkZMgHBlihqdowdFbZ6ZkOOoePRFyi7H7RnRcKo3eF+0qQ3euBzFqyC1QcRBP/dCOuQEt1KY6OPG9XSolYGcfBKkQ8PRWfaj9f4WoX1fCfUHgx9BmZ4DurBQHgeXmGOXn5OXEsXdYBLiLP3xi2wmQ8mZ1BgYCGNivniFMFC1izAf0+WN0Oh8HvrENppylbkTNSYGL3CxgfLleI4DsIgzU2HUPj6stNbsC0vW6v08bpzQwIgdOp8v+ArFSO02HQmsMlPeHd9TVviYbvHITd0Ox5VoQT9VJCMptSXw4cccLLt15G3qwVmvO+v2+r6anmplE/eLNV5lpxU=
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM5PR12MB1355.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(366004)(39860400002)(346002)(376002)(396003)(136003)(6916009)(54906003)(66556008)(5660300002)(66476007)(16576012)(316002)(7416002)(2906002)(478600001)(36756003)(66946007)(31696002)(110011004)(956004)(26005)(8676002)(52116002)(8936002)(86362001)(53546011)(31686004)(4326008)(186003)(6486002)(2616005)(43740500002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData: sBbjJv0cWjQOa/N/U4kBHtlxbTgWLBAtkPwKILg+vrRPAlZC/FhytBxmFzUZreETyN1mrzeBcNRTN87uU2fG7BN8jxMB3DG+gqy32QYcLU2oK1rLSMS0nX17RFYGbxSaBJEFKQaGZmEvnEQ2lu39kVz35fl0Ph8IUbzXEiK1LDUHof8slys7lJOcfUiW32L8Y+hXMr+zAzo9SnmxU9pRaMRmy9+mt2bAyZffJPYV9OjYUwgnYglOI6cnr1D7pLGihOmq2AXjmd+nM6Z2BlBy+IVFbTwyJXEfD2REuKJYV6mzUBvnVKSDwP0suTN1TM3Y3Fb3C2yJ18w+Wz4pre5EUf6vB1r6pYK4OGRsJPILr8e5I5yi8Z5OTQ7IfDyhPhSasVmCanvgQgNFB1TCXUgf2aJc/Be20xXVNxHWLnn8shxRBfoTT5lujC3th1x9rSgA7RpLZmFLj5wavtBic2iZWr7UBfwGkGiyqqUuC6ggIwUo2f0wc1IcsWkkj2n94eusa/ph1huHCdULjL+Wt8rO12RIAM0hwLBsZPV/U9hKFuW+Cmzx5B8vlXXbnlzWgPBjJcS9W8oJFK6R4QR4+FeJ4LRTIu2HsDNEHia9eCv0q/SDc6M14frW03yoFgu8sW6+qoHqyjN6T/hB3clkhVsMsQ==
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 04d61a52-61a1-4585-3bd0-08d8452667d5
X-MS-Exchange-CrossTenant-AuthSource: DM5PR12MB1355.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Aug 2020 16:30:52.7306
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: H1YwF+gPVsoaVfGGpodRzankItTRXIULIfZFfR1SYIMhutnzaEV39UH5NJKvv2wAe90kQ25QE9hm4OMwjs7L5g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4433
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 8/20/20 11:17 AM, Tom Lendacky wrote:
> On 8/20/20 10:55 AM, Andy Lutomirski wrote:
>> On Thu, Aug 20, 2020 at 8:21 AM Tom Lendacky <thomas.lendacky@amd.com> 
>> wrote:
>>>
>>> On 8/20/20 10:10 AM, Sean Christopherson wrote:
>>>> On Wed, Aug 19, 2020 at 05:21:33PM -0700, Andy Lutomirski wrote:
>>>>> On Wed, Aug 19, 2020 at 2:25 PM Andy Lutomirski <luto@kernel.org> wrote:
>>>>>>
>>>>>> On Wed, Aug 19, 2020 at 11:19 AM Tom Lendacky 
>>>>>> <thomas.lendacky@amd.com> wrote:
>>>>>>>
>>>>>>> On 8/19/20 1:07 PM, Tom Lendacky wrote:
>>>>>>>> It looks like the FSGSBASE support is crashing my second 
>>>>>>>> generation EPYC
>>>>>>>> system. I was able to bisect it to:
>>>>>>>>
>>>>>>>> b745cfba44c1 ("x86/cpu: Enable FSGSBASE on 64bit by default and 
>>>>>>>> add a chicken bit")
>>>>>>>>
>>>>>>>> The panic only happens when using KVM. Doing kernel builds or stress
>>>>>>>> on bare-metal appears fine. But if I fire up, in this case, a 64-vCPU
>>>>>>>> guest and do a kernel build within the guest, I get the following:
>>>>>>>
>>>>>>> I should clarify that this panic is on the bare-metal system, not 
>>>>>>> in the
>>>>>>> guest. And that specifying nofsgsbase on the bare-metal command 
>>>>>>> line fixes
>>>>>>> the issue.
>>>>>>
>>>>>> I certainly see some oddities:
>>>>>>
>>>>>> We have this code:
>>>>>>
>>>>>> static void svm_vcpu_put(struct kvm_vcpu *vcpu)
>>>>>> {
>>>>>>           struct vcpu_svm *svm = to_svm(vcpu);
>>>>>>           int i;
>>>>>>
>>>>>>           avic_vcpu_put(vcpu);
>>>>>>
>>>>>>           ++vcpu->stat.host_state_reload;
>>>>>>           kvm_load_ldt(svm->host.ldt);
>>>>>> #ifdef CONFIG_X86_64
>>>>>>           loadsegment(fs, svm->host.fs);
>>>>>>           wrmsrl(MSR_KERNEL_GS_BASE, current->thread.gsbase);
>>>>
>>>> Pretty sure current->thread.gsbase can be stale, i.e. this needs:
>>>>
>>>>        current_save_fsgs();
>>>
>>> I did try adding current_save_fsgs() in svm_vcpu_load(), saving the
>>> current->thread.gsbase value to a new variable in the svm struct. I then
>>> used that variable in the wrmsrl below, but it still crashed.
>>
>> Can you try bisecting all the way back to:
>>
>> commit dd649bd0b3aa012740059b1ba31ecad28a408f7f
>> Author: Andy Lutomirski <luto@kernel.org>
>> Date:   Thu May 28 16:13:48 2020 -0400
>>
>>      x86/cpu: Add 'unsafe_fsgsbase' to enable CR4.FSGSBASE
>>
>> and adding the unsafe_fsgsbase command line option while you bisect.
> 
> I'll give that a try.
> 
>>
>> Also, you're crashing when you run a guest, right?  Can you try
> 
> Right, when the guest is running. The guest boots fine and only when I put 
> some stress on it (kernel build) does it cause the issue. It might be 
> worth trying to pin all the vCPUs and see if the crash still happens.
> 
>> running the x86 sefltests on a bad kernel without running any guests?
> 
> I'll give that a try.

All the selftests passed.

Thanks,
Tom

> 
> Thanks,
> Tom
> 
>>
>> --Andy
>>