Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <5e70b4e9-94e4-e673-db37-03dcb5551a62@nvidia.com>
Date:   Thu, 19 Jan 2023 08:16:59 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.2.2
Subject: Re: [PATCH] KVM: arm64: vgic: Fix soft lockup during VM teardown
Content-Language: en-US
To:     Marc Zyngier <maz@kernel.org>
Cc:     James Morse <james.morse@arm.com>,
        Catalin Marinas <catalin.marinas@arm.com>,
        Will Deacon <will@kernel.org>,
        linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev,
        linux-kernel@vger.kernel.org, Vikram Sethi <vsethi@nvidia.com>,
        Zenghui Yu <yuzenghui@huawei.com>,
        Oliver Upton <oliver.upton@linux.dev>,
        Suzuki K Poulose <suzuki.poulose@arm.com>,
        Ard Biesheuvel <ardb@kernel.org>
References: <20230118022348.4137094-1-sdonthineni@nvidia.com>
 <863588njmt.wl-maz@kernel.org>
 <28061ceb-a7ce-0aca-a97d-8227dcfe6800@nvidia.com>
 <87bkmvdmna.wl-maz@kernel.org>
 <2e0c971a-0199-ff0d-c13c-d007d9f03122@nvidia.com>
 <86wn5imxm9.wl-maz@kernel.org>
From:   Shanker Donthineni <sdonthineni@nvidia.com>
In-Reply-To: <86wn5imxm9.wl-maz@kernel.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VU5OSm5IdDQza3dNTVlqSi91cjJUSThhR0RLVXcwTmEvU3JDZ2hKaWxpODQv?=
 =?utf-8?B?YTFYUVBqK1ZrRUdMQXd0YzQ3V2s1N1FwK2ZaTVRZVnBtS1FFVmwyYWdRcjVZ?=
 =?utf-8?B?d25vY0dhMHJENlNkRGN2a1lxaVM4T3ZVOEdoRE9xdVFhNGdjKzhuc3E2YnN1?=
 =?utf-8?B?TkprbE5UOTZ4Mk41Wk9kVkQvNjRXRHYyekRVa0IvS0xpT0NueUFSTlV0S1lv?=
 =?utf-8?B?M2Q4WXFsbUZWVGlraXpicS8yUUhYYk45UTNlYmJnK3haTkZVY0FpNlNhZHUw?=
 =?utf-8?B?c0xXMXZ5UGRLL1RlSmNVNlh4RWlod2NSZk1nc2hycnltb2txeHV3RHpxMktJ?=
 =?utf-8?B?RXVTWkFTdGlyN2JRNEJzS09pb0ZWd1J0ck1HUm5TV1AzbG5aWFBaTkN0M0g4?=
 =?utf-8?B?SFBMK3hUSWlGWXdzV00wT3RaQzA3YlQ1N3hrT0g5bDNwcGZVTjIwQnFhV3lD?=
 =?utf-8?B?ZW1odGpkSE85Zjc1SlNteU0wWStucUNyL25Bam9tcWkrV3MvQVpEamI1WDE2?=
 =?utf-8?B?SWxzR0pmU21ETitHRFEwUDIvaWhHTWVkTW9kYUgzYWZ5amRRRW5LcGFzeGJM?=
 =?utf-8?B?ZGM3R3F4Q0thSnd4T0hIcFM5eGlpL0hSekozdENGUi9MSC9MWHRSM21VRnNH?=
 =?utf-8?B?VmtHWTZsWVFMc0I3L0hqN3gzSWxzTXlOYTlWQjZWcXhBQlFJRiszNXBnd1My?=
 =?utf-8?B?TmY1a1dQWkN0SGRrbm5QZ2JMT2tlODltZTFWRjB6STRlTXZWU1dvUVhXZTRN?=
 =?utf-8?B?RkhXRXZSMTdndkswL3Bab2p3UTdlMTAxZGVxOVdlUW9oeTNmNFprUmVMc25E?=
 =?utf-8?B?TmJiNk81ZVk4RlMxZ1VXc2dvblRHRE5qeE90SUNLTjNDTExleStUTU5MVXNG?=
 =?utf-8?B?TWpSZU5DNkJlbnJGbm1zMVF2WkpHdWFOUW1NdE5Ob3dWbmFvdDg5alBDYzJ1?=
 =?utf-8?B?aDlpb3lmV1ExaTVZVGFuelVwTWpTNjVGWUJzMXZwTjVYZGx1UUNCNHZtQU9L?=
 =?utf-8?B?MHU1SWtaS2JPU041TFJ0UzlDMTI0aHVoTzVJSTA2Z00wVTRWSmQvaGFtZlJW?=
 =?utf-8?B?VXorNi9XWVJZQ3BiREdLZmhOWWtiNXpPMmpZWXJBdVNuQ2w2VE1tRlFZVHMv?=
 =?utf-8?B?UnpPNnhHUlE5b1M1TzlBdWJNZ3lEQVZOVmhqd2k2NXJIdVZIQlZRbnFjeTIz?=
 =?utf-8?B?b0dla20wM0dONE4wM2JTNkFPblpQdGI3UXlJZDdUMGt1Y0FYT1N5Wk5VeVJi?=
 =?utf-8?B?Wmh1TWpaaSsyZFZrclNxRi9QYUpTRklNaS9CYXhnV003YWpkdTZGT3Rhc0Zn?=
 =?utf-8?B?Y09WdlVGeE13TkN2YVRhUjkvbFpWc0JPM2I1Sk1aYnVhMWpnYTF1cE5IZFIv?=
 =?utf-8?B?YWJHeDZraUJwbTB0NmZyeU5LUUIzU0JOS2t3VmpxRnB1cjVleFJSUkZLMkJM?=
 =?utf-8?B?RnZUUUNrbDJhSHJqMU15U3V1TXhlaDdQc0tvZHB4QUUxM215dE9QRld1MkFk?=
 =?utf-8?B?Rm85dlMvZjdWeElwcjhrNEFSc0N2dEMxbEZoc25uTDJ4Qm1WL0paTXBpeThG?=
 =?utf-8?B?cEM3OWhQVHZKT0hGWTFpU0VMaTUrdFpnckdYeTBGNDJEMEJVT2hoYzduZWVq?=
 =?utf-8?B?cHNuWURBV1dRM3N5U3FRbm14VkZMQWFNcmdHNG55OERycDVZM3Erai9aWGZm?=
 =?utf-8?B?SldXd1AzL0Y1VDd5VVM3Mm54U2NjVW8wUU9QaDFkWjZ4bEIvbTVmdHBzVUNw?=
 =?utf-8?B?SzdJWkcyS3ZmU2hiYUxTZXVqdHU1RkRCS3JCREk1RmZseml3TGkxL1A5d1ZC?=
 =?utf-8?B?MHhnNDY3TlpXTEJLUTFKdkR0STVyajdCcmIzcU9OSjljTk1kRGdNb29pSnZP?=
 =?utf-8?B?NGN3cm1oSFZCSlFDRjZ0ZDJOb0dhSmIwa3VseWdBY3RLaU85aTgxRnRGaUJq?=
 =?utf-8?B?eDN6bEV1alVQYzZZNXM1dlBPSGQ5WndxUGtWYnE3NXBOb05lcEpWRGVlOXJY?=
 =?utf-8?B?dmNjSUMvcmhxVDBDWHprZzdrdjBjY29UdkNCTEpUOGp6WnpSVkZXbkFmS0sx?=
 =?utf-8?B?UEdzOFFiU3p5c29OV1paT0lnMGo5OE83YWxnN2l3VTQ2RUhOUk1CN1ZuUWpJ?=
 =?utf-8?B?V09SRHNCaDFlblRaTnVGZmRRZmxRdXI1aEh2UGxrUnZ4RXNkOUh6eHhtd0Q4?=
 =?utf-8?B?WXc9PQ==?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 361320c8-070f-48ef-40a2-08dafa27d892
X-MS-Exchange-CrossTenant-AuthSource: MN0PR12MB5762.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jan 2023 14:17:07.2353
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LUSXGLnq+sLXFJWs9apba/MuRaQ1lS8Fjy4qigqCr4kOQj12kawMzeOKoyZlM+hEAhzN3T4dLd7PuaKL1Kw58g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7350
X-Spam-Status: No, score=-1.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        NICE_REPLY_A,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,
        SPF_NONE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Marc,

On 1/19/23 08:01, Marc Zyngier wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Thu, 19 Jan 2023 13:00:49 +0000,
> Shanker Donthineni <sdonthineni@nvidia.com> wrote:
>>
>>
>>
>> On 1/19/23 01:11, Marc Zyngier wrote:
>>> So you can see the VM being torn down while the vgic save sequence is
>>> still in progress?
>>>
>>> If you can actually see that, then this is a much bigger bug than the
>>> simple race you are describing, and we're missing a reference on the
>>> kvm structure. This would be a *MAJOR* bug.
>>>
>> How do we know vGIC save sequence is in progress while VM is being
>> teardown?  I'm launching/terminating ~32 VMs in a loop to reproduce
>> the issue.
> 
> Errr... *you* know when you are issuing the save ioctl, right? You
> also know when you are terminating the VM (closing its fd or killing
> the VMM).
> 
>>
>>> Please post the full traces, not snippets. The absolutely full kernel
>>> log, the configuration, what you run, how you run it, *EVERYTHING*. I
>>> need to be able to reproduce this.
>> Sure, I'll share the complete boot log messages of host kernel next run.
>>
>>>
>>>>
>>>>>>
>>>>>> irqreturn_t handle_irq_event(struct irq_desc *desc)
>>>>>> {
>>>>>>        irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS);
>>>>>>        raw_spin_unlock(&desc->lock);
>>>>>>
>>>>>>        ret = handle_irq_event_percpu(desc);
>>>>>>
>>>>>>        raw_spin_lock(&desc->lock);
>>>>>>        irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS);
>>>>>> }
>>>>>
>>>>> How is that relevant to this trace? Do you see this function running
>>>>> concurrently with the teardown? If it matters here, it must be a VPE
>>>>> doorbell, right? But you claim that this is on a GICv4 platform, while
>>>>> this would only affect GICv4.1... Or are you using GICv4.1?
>>>>>
>>>> handle_irq_event() is running concurrently with irq_domain_activate_irq()
>>>> which happens before free_irq() called. Corruption at [78.983544] and
>>>> teardown started at [87.360891].
>>>
>>> But that doesn't match the description you made of concurrent
>>> events. Does it take more than 9 seconds for the vgic state to be
>>> saved to memory?
>>
>> Are there any other possibilities of corrupting IRQD_IRQ_INPROGRESS
>> state bit other than concurrent accesses?
> 
> Forget about this bit. You said that we could see the VM teardown
> happening *at the same time* as the vgic state saving, despite the
> vgic device holding a reference on the kvm structure. If that's the
> case, this bit is the least of our worries. Think of the consequences
> for a second...
> 
> [...]
> 
>> Using the below steps for launching/terminating 32 VMs in loop. The
>> failure is intermittent. The same issue is reproducible with KVMTOOL
>> also.
> 
> kvmtool never issue a KVM_DEV_ARM_VGIC_GRP_CTRL with the
> KVM_DEV_ARM_ITS_SAVE_TABLES argument, so the code path we discussed is
> never used. What is the exact problem you're observing with kvmtool
> as the VMM?
> 
Unfortunately I didn't capture the log messages of kvmtool based VM boot.
Saw once kernel crash. Now I'm trying to reproduce the issue with QEMU tool
and share complete log messages.

>          M.
> 
> --
> Without deviation from the norm, progress is not possible.