Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753127AbcCUHwG (ORCPT ); Mon, 21 Mar 2016 03:52:06 -0400 Received: from mx1.redhat.com ([209.132.183.28]:47135 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752706AbcCUHv5 (ORCPT ); Mon, 21 Mar 2016 03:51:57 -0400 From: Vitaly Kuznetsov To: KY Srinivasan Cc: "devel\@linuxdriverproject.org" , "linux-kernel\@vger.kernel.org" , "Haiyang Zhang" , "Alex Ng \(LIS\)" , "Radim Krcmar" , Cathy Avery Subject: Re: [PATCH] Drivers: hv: vmbus: handle various crash scenarios References: <1458304404-8347-1-git-send-email-vkuznets@redhat.com> Date: Mon, 21 Mar 2016 08:51:54 +0100 In-Reply-To: (KY Srinivasan's message of "Fri, 18 Mar 2016 18:02:53 +0000") Message-ID: <874mc02rqd.fsf@vitty.brq.redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4071 Lines: 121 KY Srinivasan writes: >> -----Original Message----- >> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com] >> Sent: Friday, March 18, 2016 5:33 AM >> To: devel@linuxdriverproject.org >> Cc: linux-kernel@vger.kernel.org; KY Srinivasan ; >> Haiyang Zhang ; Alex Ng (LIS) >> ; Radim Krcmar ; Cathy >> Avery >> Subject: [PATCH] Drivers: hv: vmbus: handle various crash scenarios >> >> Kdump keeps biting. Turns out CHANNELMSG_UNLOAD_RESPONSE is always >> delivered to CPU0 regardless of what CPU we're sending >> CHANNELMSG_UNLOAD >> from. vmbus_wait_for_unload() doesn't account for the fact that in case >> we're crashing on some other CPU and CPU0 is still alive and operational >> CHANNELMSG_UNLOAD_RESPONSE will be delivered there completing >> vmbus_connection.unload_event, our wait on the current CPU will never >> end. > > What was the host you were testing on? > I was testing on both 2012R2 and 2016TP4. The bug is easily reproducible by forcing crash on a secondary CPU, e.g.: # cat crash.sh #! /bin/sh echo c > /proc/sysrq-trigger # taskset -c 1 ./crash.sh >> >> Do the following: >> 1) Check for completion_done() in the loop. In case interrupt handler is >> still alive we'll get the confirmation we need. >> >> 2) Always read CPU0's message page as CHANNELMSG_UNLOAD_RESPONSE >> will be >> delivered there. We can race with still-alive interrupt handler doing >> the same but we don't care as we're checking completion_done() now. >> >> 3) Cleanup message pages on all CPUs. This is required (at least for the >> current CPU as we're clearing CPU0 messages now but we may want to >> bring >> up additional CPUs on crash) as new messages won't be delivered till we >> consume what's pending. On boot we'll place message pages somewhere >> else >> and we won't be able to read stale messages. >> >> Signed-off-by: Vitaly Kuznetsov >> --- >> drivers/hv/channel_mgmt.c | 30 +++++++++++++++++++++++++----- >> 1 file changed, 25 insertions(+), 5 deletions(-) >> >> diff --git a/drivers/hv/channel_mgmt.c b/drivers/hv/channel_mgmt.c >> index b10e8f74..5f37057 100644 >> --- a/drivers/hv/channel_mgmt.c >> +++ b/drivers/hv/channel_mgmt.c >> @@ -512,14 +512,26 @@ static void init_vp_index(struct vmbus_channel >> *channel, const uuid_le *type_gui >> >> static void vmbus_wait_for_unload(void) >> { >> - int cpu = smp_processor_id(); >> - void *page_addr = hv_context.synic_message_page[cpu]; >> + int cpu; >> + void *page_addr = hv_context.synic_message_page[0]; >> struct hv_message *msg = (struct hv_message *)page_addr + >> VMBUS_MESSAGE_SINT; >> struct vmbus_channel_message_header *hdr; >> bool unloaded = false; >> >> - while (1) { >> + /* >> + * CHANNELMSG_UNLOAD_RESPONSE is always delivered to CPU0. >> When we're >> + * crashing on a different CPU let's hope that IRQ handler on CPU0 is >> + * still functional and vmbus_unload_response() will complete >> + * vmbus_connection.unload_event. If not, the last thing we can do >> is >> + * read message page for CPU0 regardless of what CPU we're on. >> + */ >> + while (!unloaded) { >> + if (completion_done(&vmbus_connection.unload_event)) { >> + unloaded = true; >> + break; >> + } >> + >> if (READ_ONCE(msg->header.message_type) == >> HVMSG_NONE) { >> mdelay(10); >> continue; >> @@ -530,9 +542,17 @@ static void vmbus_wait_for_unload(void) >> unloaded = true; >> >> vmbus_signal_eom(msg); >> + } >> >> - if (unloaded) >> - break; >> + /* >> + * We're crashing and already got the UNLOAD_RESPONSE, cleanup >> all >> + * maybe-pending messages on all CPUs to be able to receive new >> + * messages after we reconnect. >> + */ >> + for_each_online_cpu(cpu) { >> + page_addr = hv_context.synic_message_page[cpu]; >> + msg = (struct hv_message *)page_addr + >> VMBUS_MESSAGE_SINT; >> + msg->header.message_type = HVMSG_NONE; >> } >> } >> >> -- >> 2.5.0 -- Vitaly