Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <PH0PR11MB51748333769D9002912FF63C84E89@PH0PR11MB5174.namprd11.prod.outlook.com>
In-Reply-To: <PH0PR11MB51748333769D9002912FF63C84E89@PH0PR11MB5174.namprd11.prod.outlook.com>
From:   Joel Fernandes <joelaf@google.com>
Date:   Mon, 26 Jul 2021 11:57:21 -0400
Message-ID: <CAJWu+oq98Yh4kknze7u6SOYZNw6hR5aFkj61aXbTV3JPq=7RNg@mail.gmail.com>
Subject: Re: Core Scheduling - Concurrent VMs
To:     "Rajendran, Jaishankar" <jaishankar.rajendran@intel.com>,
        Dario Faggioli <dfaggioli@suse.com>,
        Vineeth Pillai <vineethrp@gmail.com>
Cc:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "peterz@infradead.org" <peterz@infradead.org>,
        "Mallikarjun Chegaraddi, Raju" 
        <raju.mallikarjun.chegaraddi@intel.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

Hi,
+Dario Faggioli and +Vineeth Pillai as well.

On Mon, Jul 26, 2021 at 1:41 AM Rajendran, Jaishankar
<jaishankar.rajendran@intel.com> wrote:
>
> Refer to the below experiments performed using Core Scheduling for Concurrent VMs and we found the benchmark scores executed in different VMs are
> degrading after enablement of Core Scheduling. Both the host and guest are enabled with Core Scheduler

Why are you running core scheduling within the guest? Only the host
needs it and you tag vCPU threads in the host. Within the guest you
don't need it (and it probably doesn't make sense unless you pin the
vCPU threads to individual physical hardware threads).

>
> Environment:
> ++++++++++
> Platform - CML-NUC - i5-10600 @ 3.3 Ghz
> Core Assignment for Host OS : No .of Cores     : 4   No .of Threads : 4 ( i.e. 8 Logical Cores)
> Host OS - Ubuntu 20.1 with Chromium Kernel 5.4
> Guest OS - Android OS running as VM using qemu/kvm with Chromium Kernel 5.4
> VM's are assigned with 4 cores and 4 Threads (i.e. 8 Logical Cores)
>
> Kernel Config:
> +++++++++++
> Host and Guest OS Kernel is enabled with CONFIG_SCHED_CORE=y
>
> CGROUP Mapping:
> ++++++++++++++++
> We did follow the CGroup approach for Core Scheduling based on the below document
> https://lkml.org/lkml/2021/1/22/1469
>  and configured the CGroup. But not able to find cpu.core_tag file in CGroups?
>
> VM1 (Android OS running using qemu/KVM) is executed and its default cgroup is changed to caas1
> VM2 (Android OS running using qemu/KVM) is executed and its default cgroup is changed to caas2

It is not clear to me how the individual vCPU threads are tagged so it
is hard to say much.

> The changes are validated using htop command.
>
> WorkLoad:
> +++++++++
> GeekBench 5.3.1 - Multi core Test executed on both the VMs
>
> Observations:
> ++++++++++++
> As per core CORE_SCHEDULER documentation, (vCPU) Threads of VM1 & VM2 (belonging to two different cgroups ) should not be scheduled on the same core at a given time.
> But we observe that vCPU Threads of different VMs are scheduled in the same core ( Used HTOP and ps commands)

I don't think these commands can really tell you that core scheduler
is working or not. They are too coarse grained. You would need to look
at it through ftrace scheduling events. Core scheduling will still
allow differently tagged threads to use a core, it is just that it
wont allow it simultaneously.

Also just to note, in ChromeOS, we are working on some changes to tag
the whole Android VM with same tag (all vCPU threads in a single VM
can share a core) which should improve performance. Previously we were
putting different tags for each vCPU thread of the Android VM which
seemed overkill.