Received: by 2002:ad5:474a:0:0:0:0:0 with SMTP id i10csp6415731imu; Mon, 21 Jan 2019 08:31:31 -0800 (PST) X-Google-Smtp-Source: ALg8bN6kcwhING7rtUxnhdTauYHozhxYFtRAo+RFGVXPNVnAHIG9jc7e3b/2tmxrlsoU8h+DDm67 X-Received: by 2002:a62:cd1:: with SMTP id 78mr30276987pfm.219.1548088291872; Mon, 21 Jan 2019 08:31:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1548088291; cv=none; d=google.com; s=arc-20160816; b=Op4IqLpyjPbLkMcLdldOl+x2/f9FcvIr+CsWrE5mS636UO0IzWHnOmr0PLcwwF84Bz /O5nvD9AmyKGp4uE4Sjm/8Ejkty8HingQdigYsLUfQLXy9xPr7O/cf618AVKpyVhqbo1 NUdZXzR+S+XAOaVRmUvKzAI7QlpfPPlQ/vsBfubE3qsBiSGYuGuGf3gwrH2/mT9zx1Xd zP3LXX0Y0Qo5Gej1nMtdfCInjIq0yC9SjvGAOVEMnwI307Z8q/tKjo31NrY9wM0i5Ou/ NhoemXc2Ys0okcH6+W6heUsjJuCwtJ397WzdFh+rAJdOrXsO0XTciYTbFvTG4Uf4djpY jHvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:dkim-signature; bh=BPMrAw0VYjVeTLtgHSgwvU76nQdwPVRJhRGf5cyWYnU=; b=Uo1LUwk7rCTIKWqxXD9AoCdtM+nSYg8yyovrsQ5B62hDCJhS75RWtoxOH59uPNaDI/ 57s2Y0jZUzijIsAkT/S/CslfUP7yx9Wk/xB/WPRWVI0NQChWh3IUFSd+V4c7gxDVlL+O FuiQ+u7M8ccOcuYPgC9bE1sQtxi+2qYhoCWLt5XUDkLtVFBlEv5tFtLsib1oslz3t+ly FTPaCZJG8VpJL874xm1UpMVvlVTKMtdQeI7viZIc4anKOpgZcCZ1ka8ul8dbbeM1E76M IMSmlMFMO1vj8Nf0AJMxmwTKIKjUKk4/nP+J4KTl1gcz29n3SPmA0UAGoj8HOx+Q83dv C7bw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=lEq3HqOn; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id r81si13853924pfr.164.2019.01.21.08.31.14; Mon, 21 Jan 2019 08:31:31 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=lEq3HqOn; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730379AbfAUQ3r (ORCPT + 99 others); Mon, 21 Jan 2019 11:29:47 -0500 Received: from mail-it1-f196.google.com ([209.85.166.196]:54791 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727205AbfAUQ3r (ORCPT ); Mon, 21 Jan 2019 11:29:47 -0500 Received: by mail-it1-f196.google.com with SMTP id i145so17161230ita.4 for ; Mon, 21 Jan 2019 08:29:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=BPMrAw0VYjVeTLtgHSgwvU76nQdwPVRJhRGf5cyWYnU=; b=lEq3HqOnCIHKAhCVvOWHBUAqlTInb5n29l1d0Lczqtee0bVpT5F1JleSKkAUTvbizy z48G1YoRGFz2AzapKtcJGnKVCPmpcvyaacvtYpbnE6X25w0BKp6GNt5dRcOT3E1UTtd0 BbRKJHsR3x32AZsIOqZMvBNDoaAlX36GIvgTfu+haqDLz9MUlFYMzrDlrLIdwBJaFvwt N7HISpeC9PJsMSKh1xrmStQ1RtFN7kuEXPr8XKL8s+8heeaYQu6CGY5Y0G5MhA6fXBSw ces+zCN9O3meFto6XFUSo89W2KjtGPhxHvUieasEZNbGfZRsUfZc0+4c3Tc8R0+0ShjE IqPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=BPMrAw0VYjVeTLtgHSgwvU76nQdwPVRJhRGf5cyWYnU=; b=YmI0hoXqLr/pznXPL2KawvXjDe7tpYqwtW3GK6a3boGAyU8RMe2kd5eYFfU0SsNcPq ih5q8zPW0ej8vHv4JOB32yklQnT6W8KMnU3WHiLvet8uHM1nBhPiHX56kynHjOaF1x+d EhTGGvJ+xoExmexZf2T2Crzc2CR4REXWr1+E7gJ9wg3wI1dhHZNk0zjqJlX5WsuIlYKH iYz7pUbedDxD9p42TQYaVy/meTi/lIbP4rApvJwnPhY2Ex62amCZxhDAG9/wf8Zdz4QL JtbDsVGTPZfw9nqcT3PHVwX0uxaC7tzIQbA+5baVBC4js6T+qlyt8FUBl9SDxZv85CUY KZ8g== X-Gm-Message-State: AJcUukdNr1/NWeT/UBwcboDes8kpHQFgi8iy0qLmnM8muSwBB3FB/BX8 bPTRQ3YMUIawCydFUsngCX8= X-Received: by 2002:a24:1a90:: with SMTP id 138mr42619iti.171.1548088185910; Mon, 21 Jan 2019 08:29:45 -0800 (PST) Received: from ubu-Virtual-Machine (66-188-57-61.dhcp.bycy.mi.charter.com. [66.188.57.61]) by smtp.gmail.com with ESMTPSA id h139sm6525077ith.24.2019.01.21.08.29.44 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 21 Jan 2019 08:29:45 -0800 (PST) Date: Mon, 21 Jan 2019 11:29:38 -0500 From: Kimberly Brown To: Stephen Hemminger Cc: Dexuan Cui , Michael Kelley , Long Li , Sasha Levin , "devel@linuxdriverproject.org" , Haiyang Zhang , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH v3] Drivers: hv: vmbus: Expose counters for interrupts and full conditions Message-ID: <20190121162938.GA3544@ubu-Virtual-Machine> References: <20190105043518.GA4072@ubu-Virtual-Machine> <20190117043759.GA3395@ubu-Virtual-Machine> <20190117091019.1fb0c186@hermes.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190117091019.1fb0c186@hermes.lan> User-Agent: Mutt/1.9.4 (2018-02-28) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Jan 17, 2019 at 09:11:03AM -0800, Stephen Hemminger wrote: > > > > +static ssize_t channel_intr_in_full_show(const struct vmbus_channel > > *channel, > > + char *buf) > > +{ > > + return sprintf(buf, "%llu\n", channel->intr_in_full); > > +} > > > intr_in_full is u64, which is not the same as unsigned long long. > to be correct you need a cast here. > Thanks for the feedback. I'll fix this issue in all four of the "_show" functions that are added in this patch. > > > diff --git a/include/linux/hyperv.h b/include/linux/hyperv.h > > > index dcb6977afce9..7e5239123276 100644 > > > --- a/include/linux/hyperv.h > > > +++ b/include/linux/hyperv.h > > > @@ -751,6 +751,27 @@ struct vmbus_channel { > > > u64 interrupts; /* Host to Guest interrupts */ > > > u64 sig_events; /* Guest to Host events */ > > > > > > + /* Interrupt counts for 2 types of Guest to Host interrupts */ > > > + u64 intr_in_full; /* in ring buffer, full to not full */ > > > + u64 intr_out_empty; /* out ring buffer, empty to not empty */ > > > + > > > + /* > > > + * The total number of write operations that encountered a full > > > + * outbound ring buffer. > > > + */ > > > + u64 out_full_total; > > > + /* > > > + * The number of write operations that were the first to encounter a > > > + * full outbound ring buffer. > > > + */ > > > + u64 out_full_first; > > Adding more fields changes cache layout which can cause > additional cache miss in the hot path. > Good point. I think that the "intr_out_empty" field is in a good location, but the "intr_in_full", "out_full_first", and "out_full_total" fields could be moved to the end of the struct. These variables are used only when ring buffer full conditions occur. Ring buffer full conditions shouldn't be encountered often, and, if they are, they're a signal that changes should probably be made to prevent them. If you have any other suggestions for this, please let me know. > > > + /* > > > + * Indicates that a full outbound ring buffer was encountered. The flag > > > + * is set to true when a full outbound ring buffer is encountered and > > > + * set to false when a write to the outbound ring buffer is completed. > > > + */ > > > + bool out_full_flag; > > Discussion on kernel mailing list. Recommends against putting bool > in structures since that pads to full sizeof(int). Could this be > part of a bitfield? > There are currently 4 other bool variables in this struct. Maybe some or all of the bool variables could be placed adjacent to each other and changed into bitfields. I'll need to look into this. > > > /* Channel callback's invoked in softirq context */ > > > struct tasklet_struct callback_event; > > > void (*onchannel_callback)(void *context); > > > @@ -936,6 +957,23 @@ static inline void *get_per_channel_state(struct > > > vmbus_channel *c) > > > static inline void set_channel_pending_send_size(struct vmbus_channel *c, > > > u32 size) > > > { > > > + unsigned long flags; > > > + > > > + spin_lock_irqsave(&c->outbound.ring_lock, flags); > > > + > > > + if (size) { > > > + ++c->out_full_total; > > > + > > > + if (!c->out_full_flag) { > > > + ++c->out_full_first; > > > + c->out_full_flag = true; > > > + } > > > + } else { > > > + c->out_full_flag = false; > > > + } > > > + > > > + spin_unlock_irqrestore(&c->outbound.ring_lock, flags); > > If this is called often, the additional locking will impact performance. > In hv_sock, each call of "hvs_stream_has_space()" results in a call to "channel_set_pending_send_size()", so this could be a concern. I'll work on addressing this issue. > > > c->outbound.ring_buffer->pending_send_sz = size; > > > } > > > > > Could I propose another alternative. > > It might be more useful to count the guest to host interaction events > rather than the ring buffer. > > For example the number of calls to: > vmbus_set_event which means host exit call > vmbus_setevent fastpath using sync_set_bit > calls to rinbuffer_write that returned -EAGAIN > > These would require less locking, reuse existing code paths > and not require additional state. > I'm not sure that this approach would provide the data that we're looking for. For example, we're interested in evaluating how often ring buffer write operations encounter full conditions. For this, we need to know how many interaction events were caused by the ring buffer being full. Counting the number of calls to "vmbus_set_event()" and "vmbus_setevent()" wouldn't allow us to determine what caused the events. For counting the full conditions, the number of calls to "ring_buffer_write()" that returned -EAGAIN isn't sufficient because hv_sock doesn't use the -EAGAIN path to determine that the ring buffer is full. Therefore, we need to count the number of full conditions in both "ring_buffer_write()" and "set_channel_pending_send_size()".