From: KY Srinivasan <kys@microsoft.com>
To: Vitaly Kuznetsov <vkuznets@redhat.com>,
        "devel@linuxdriverproject.org" <devel@linuxdriverproject.org>
CC: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "Haiyang Zhang" <haiyangz@microsoft.com>,
        "Alex Ng (LIS)" <alexng@microsoft.com>,
        "Radim Krcmar" <rkrcmar@redhat.com>, Cathy Avery <cavery@redhat.com>
Subject: RE: [PATCH] Drivers: hv: vmbus: handle various crash scenarios
Thread-Topic: [PATCH] Drivers: hv: vmbus: handle various crash scenarios
Thread-Index: AQHRgRJgH+GbSkQfjk2MKtlbBA5OMZ9ffm9g
Date: Fri, 18 Mar 2016 18:02:53 +0000
Message-ID: <SN2PR03MB214246D9A9DAFA36E392AE78A08C0@SN2PR03MB2142.namprd03.prod.outlook.com>
References: <1458304404-8347-1-git-send-email-vkuznets@redhat.com>
In-Reply-To: <1458304404-8347-1-git-send-email-vkuznets@redhat.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 8BIT
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Mar 2016 18:02:53.3390
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN2PR03MB1917
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 3713
Lines: 109


> -----Original Message-----
> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> Sent: Friday, March 18, 2016 5:33 AM
> To: devel@linuxdriverproject.org
> Cc: linux-kernel@vger.kernel.org; KY Srinivasan <kys@microsoft.com>;
> Haiyang Zhang <haiyangz@microsoft.com>; Alex Ng (LIS)
> <alexng@microsoft.com>; Radim Krcmar <rkrcmar@redhat.com>; Cathy
> Avery <cavery@redhat.com>
> Subject: [PATCH] Drivers: hv: vmbus: handle various crash scenarios
> 
> Kdump keeps biting. Turns out CHANNELMSG_UNLOAD_RESPONSE is always
> delivered to CPU0 regardless of what CPU we're sending
> CHANNELMSG_UNLOAD
> from. vmbus_wait_for_unload() doesn't account for the fact that in case
> we're crashing on some other CPU and CPU0 is still alive and operational
> CHANNELMSG_UNLOAD_RESPONSE will be delivered there completing
> vmbus_connection.unload_event, our wait on the current CPU will never
> end.

What was the host you were testing on?

K. Y
> 
> Do the following:
> 1) Check for completion_done() in the loop. In case interrupt handler is
>    still alive we'll get the confirmation we need.
> 
> 2) Always read CPU0's message page as CHANNELMSG_UNLOAD_RESPONSE
> will be
>    delivered there. We can race with still-alive interrupt handler doing
>    the same but we don't care as we're checking completion_done() now.
> 
> 3) Cleanup message pages on all CPUs. This is required (at least for the
>    current CPU as we're clearing CPU0 messages now but we may want to
> bring
>    up additional CPUs on crash) as new messages won't be delivered till we
>    consume what's pending. On boot we'll place message pages somewhere
> else
>    and we won't be able to read stale messages.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  drivers/hv/channel_mgmt.c | 30 +++++++++++++++++++++++++-----
>  1 file changed, 25 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/hv/channel_mgmt.c b/drivers/hv/channel_mgmt.c
> index b10e8f74..5f37057 100644
> --- a/drivers/hv/channel_mgmt.c
> +++ b/drivers/hv/channel_mgmt.c
> @@ -512,14 +512,26 @@ static void init_vp_index(struct vmbus_channel
> *channel, const uuid_le *type_gui
> 
>  static void vmbus_wait_for_unload(void)
>  {
> -	int cpu = smp_processor_id();
> -	void *page_addr = hv_context.synic_message_page[cpu];
> +	int cpu;
> +	void *page_addr = hv_context.synic_message_page[0];
>  	struct hv_message *msg = (struct hv_message *)page_addr +
>  				  VMBUS_MESSAGE_SINT;
>  	struct vmbus_channel_message_header *hdr;
>  	bool unloaded = false;
> 
> -	while (1) {
> +	/*
> +	 * CHANNELMSG_UNLOAD_RESPONSE is always delivered to CPU0.
> When we're
> +	 * crashing on a different CPU let's hope that IRQ handler on CPU0 is
> +	 * still functional and vmbus_unload_response() will complete
> +	 * vmbus_connection.unload_event. If not, the last thing we can do
> is
> +	 * read message page for CPU0 regardless of what CPU we're on.
> +	 */
> +	while (!unloaded) {
> +		if (completion_done(&vmbus_connection.unload_event)) {
> +			unloaded = true;
> +			break;
> +		}
> +
>  		if (READ_ONCE(msg->header.message_type) ==
> HVMSG_NONE) {
>  			mdelay(10);
>  			continue;
> @@ -530,9 +542,17 @@ static void vmbus_wait_for_unload(void)
>  			unloaded = true;
> 
>  		vmbus_signal_eom(msg);
> +	}
> 
> -		if (unloaded)
> -			break;
> +	/*
> +	 * We're crashing and already got the UNLOAD_RESPONSE, cleanup
> all
> +	 * maybe-pending messages on all CPUs to be able to receive new
> +	 * messages after we reconnect.
> +	 */
> +	for_each_online_cpu(cpu) {
> +		page_addr = hv_context.synic_message_page[cpu];
> +		msg = (struct hv_message *)page_addr +
> VMBUS_MESSAGE_SINT;
> +		msg->header.message_type = HVMSG_NONE;
>  	}
>  }
> 
> --
> 2.5.0