Date: Tue, 28 Jul 2015 00:02:14 +0300
From: "Michael S. Tsirkin" <mst@redhat.com>
To: Bandan Das <bsd@redhat.com>
Cc: kvm@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
        Eyal Moscovici <EYALMO@il.ibm.com>, Razya Ladelsky <RAZYA@il.ibm.com>,
        cgroups@vger.kernel.org, jasowang@redhat.com
Subject: Re: [RFC PATCH 0/4] Shared vhost design
Message-ID: <20150727235818-mutt-send-email-mst@redhat.com>
References: <1436760455-5686-1-git-send-email-bsd@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1436760455-5686-1-git-send-email-bsd@redhat.com>
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 4840
Lines: 108

On Mon, Jul 13, 2015 at 12:07:31AM -0400, Bandan Das wrote:
> Hello,
> 
> There have been discussions on improving the current vhost design. The first
> attempt, to my knowledge was Shirley Ma's patch to create a dedicated vhost
> worker per cgroup.
> 
> http://comments.gmane.org/gmane.linux.network/224730
> 
> Later, I posted a cmwq based approach for performance comparisions
> http://comments.gmane.org/gmane.linux.network/286858
> 
> More recently was the Elvis work that was presented in KVM Forum 2013
> http://www.linux-kvm.org/images/a/a3/Kvm-forum-2013-elvis.pdf
> 
> The Elvis patches rely on common vhost thread design for scalability
> along with polling for performance. Since there are two major changes
> being proposed, we decided to split up the work. The first (this RFC),
> proposing a re-design of the vhost threading model and the second part
> (not posted yet) to focus more on improving performance. 
> 
> I am posting this with the hope that we can have a meaningful discussion
> on the proposed new architecture. We have run some tests to show that the new
> design is scalable and in terms of performance, is comparable to the current
> stable design. 
> 
> Test Setup:
> The testing is based on the setup described in the Elvis proposal.
> The initial tests are just an aggregate of Netperf STREAM and MAERTS but
> as we progress, I am happy to run more tests. The hosts are two identical
> 16 core Haswell systems with point to point network links. For the first 10 runs,
> with n=1 upto n=10 guests running in parallel, I booted the target system with nr_cpus=8
> and mem=12G. The purpose was to do a comparision of resource utilization
> and how it affects performance. Finally, with the number of guests set at 14,
> I didn't limit the number of CPUs booted on the host or limit memory seen by
> the kernel but boot the kernel with isolcpus=14,15 that will be used to run
> the vhost threads. The guests are pinned to cpus 0-13 and based on which
> cpu the guest is running on, the corresponding I/O thread is either pinned
> to cpu 14 or 15.
> Results
> # X axis is number of guests
> # Y axis is netperf number
> # nr_cpus=8 and mem=12G
> #Number of Guests        #Baseline            #ELVIS
> 1                        1119.3		      1111.0
> 2			 1135.6		      1130.2
> 3			 1135.5		      1131.6
> 4			 1136.0		      1127.1
> 5			 1118.6		      1129.3
> 6			 1123.4		      1129.8
> 7			 1128.7		      1135.4
> 8			 1129.9		      1137.5
> 9			 1130.6		      1135.1
> 10			 1129.3		      1138.9
> 14*			 1173.8		      1216.9

I'm a bit too busy now, with 2.4 and related stuff, will review once we
finish 2.4.  But I'd like to ask two things:
- did you actually test a config where cgroups were used?
- does the design address the issue of VM 1 being blocked
  (e.g. because it hits swap) and blocking VM 2?

> 
> #* Last run with the vCPU and I/O thread(s) pinned, no CPU/memory limit imposed.
> #  I/O thread runs on CPU 14 or 15 depending on which guest it's serving
> 
> There's a simple graph at
> http://people.redhat.com/~bdas/elvis/data/results.png
> that shows how task affinity results in a jump and even without it,
> as the number of guests increase, the shared vhost design performs
> slightly better.
> 
> Observations:
> 1. In terms of "stock" performance, the results are comparable.
> 2. However, with a tuned setup, even without polling, we see an improvement
> with the new design.
> 3. Making the new design simulate old behavior would be a matter of setting
> the number of guests per vhost threads to 1.
> 4. Maybe, setting a per guest limit on the work being done by a specific vhost
> thread is needed for it to be fair.
> 5. cgroup associations needs to be figured out. I just slightly hacked the
> current cgroup association mechanism to work with the new model. Ccing cgroups
> for input/comments.
> 
> Many thanks to Razya Ladelsky and Eyal Moscovici, IBM for the initial
> patches, the helpful testing suggestions and discussions.
> 
> Bandan Das (4):
>   vhost: Introduce a universal thread to serve all users
>   vhost: Limit the number of devices served by a single worker thread
>   cgroup: Introduce a function to compare cgroups
>   vhost: Add cgroup-aware creation of worker threads
> 
>  drivers/vhost/net.c    |   6 +-
>  drivers/vhost/scsi.c   |  18 ++--
>  drivers/vhost/vhost.c  | 272 +++++++++++++++++++++++++++++++++++--------------
>  drivers/vhost/vhost.h  |  32 +++++-
>  include/linux/cgroup.h |   1 +
>  kernel/cgroup.c        |  40 ++++++++
>  6 files changed, 275 insertions(+), 94 deletions(-)
> 
> -- 
> 2.4.3
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/