Subject: Re: [PATCH 7/9] x86/intel_rdt: Implement scheduling support for Intel
 RDT
To: Vikas Shivappa <vikas.shivappa@linux.intel.com>, vikas.shivappa@intel.com
References: <1438898117-3692-1-git-send-email-vikas.shivappa@linux.intel.com>
 <1438898117-3692-8-git-send-email-vikas.shivappa@linux.intel.com>
Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com,
        tglx@linutronix.de, mingo@kernel.org, tj@kernel.org,
        peterz@infradead.org, matt.fleming@intel.com, will.auld@intel.com,
        glenn.p.williamson@intel.com, kanaka.d.juvva@intel.com
From: Andy Lutomirski <luto@kernel.org>
Message-ID: <55C3E7B0.2010609@kernel.org>
Date: Thu, 6 Aug 2015 16:03:12 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.1.0
MIME-Version: 1.0
In-Reply-To: <1438898117-3692-8-git-send-email-vikas.shivappa@linux.intel.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 2688
Lines: 51

On 08/06/2015 02:55 PM, Vikas Shivappa wrote:
> Adds support for IA32_PQR_ASSOC MSR writes during task scheduling. For
> Cache Allocation, MSR write would let the task fill in the cache
> 'subset' represented by the task's intel_rdt cgroup cache_mask.
>
> The high 32 bits in the per processor MSR IA32_PQR_ASSOC represents the
> CLOSid. During context switch kernel implements this by writing the
> CLOSid of the cgroup to which the task belongs to the CPU's
> IA32_PQR_ASSOC MSR.
>
> This patch also implements a common software cache for IA32_PQR_MSR
> (RMID 0:9, CLOSId 32:63) to be used by both Cache monitoring (CMT) and
> Cache allocation. CMT updates the RMID where as cache_alloc updates the
> CLOSid in the software cache. During scheduling when the new RMID/CLOSid
> value is different from the cached values, IA32_PQR_MSR is updated.
> Since the measured rdmsr latency for IA32_PQR_MSR is very high (~250
>   cycles) this software cache is necessary to avoid reading the MSR to
> compare the current CLOSid value.
>
> The following considerations are done for the PQR MSR write so that it
> minimally impacts scheduler hot path:
>   - This path does not exist on any non-intel platforms.
>   - On Intel platforms, this would not exist by default unless CGROUP_RDT
>   is enabled.
>   - remains a no-op when CGROUP_RDT is enabled and intel SKU does not
>   support the feature.
>   - When feature is available and enabled, never does MSR write till the
>   user manually creates a cgroup directory *and* assigns a cache_mask
>   different from root cgroup directory. Since the child node inherits
>   the parents cache mask, by cgroup creation there is no scheduling hot
>   path impact from the new cgroup.
>   - MSR write is only done when there is a task with different Closid is
>   scheduled on the CPU. Typically if the task groups are bound to be
>   scheduled on a set of CPUs, the number of MSR writes is greatly
>   reduced.
>   - A per CPU cache of CLOSids is maintained to do the check so that we
>   dont have to do a rdmsr which actually costs a lot of cycles.
>   - For cgroup directories having same cache_mask the CLOSids are reused.
>   This minimizes the number of CLOSids used and hence reduces the MSR
>   write frequency.

What happens if a user process sets a painfully restrictive CLOS and 
then spends most of its time in the kernel doing work on behalf of 
unrelated tasks?  Does performance suck?

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/