Date: Fri, 27 Feb 2015 11:34:16 -0800 (PST)
From: Vikas Shivappa <vikas.shivappa@intel.com>
To: Tejun Heo <tj@kernel.org>
cc: Vikas Shivappa <vikas.shivappa@linux.intel.com>,
        linux-kernel@vger.kernel.org, vikas.shivappa@intel.com,
        matt.fleming@intel.com, hpa@zytor.com, tglx@linutronix.de,
        mingo@kernel.org, peterz@infradead.org, will.auld@intel.com,
        dave.hansen@intel.com, andi.kleen@intel.com, tony.luck@intel.com,
        kanaka.d.juvva@intel.com
Subject: Re: [PATCH 3/7] x86/intel_rdt: Support cache bit mask for Intel
 CAT
In-Reply-To: <20150227121222.GE3964@htj.duckdns.org>
Message-ID: <alpine.DEB.2.10.1502271118040.31647@vshiva-Udesk>
References: <1424819804-4082-1-git-send-email-vikas.shivappa@linux.intel.com> <1424819804-4082-4-git-send-email-vikas.shivappa@linux.intel.com> <20150227121222.GE3964@htj.duckdns.org>
User-Agent: Alpine 2.10 (DEB 1266 2009-07-14)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 3250
Lines: 75


Hello Tejun,

On Fri, 27 Feb 2015, Tejun Heo wrote:

> Hello,
>
> On Tue, Feb 24, 2015 at 03:16:40PM -0800, Vikas Shivappa wrote:
>> Add support for cache bit mask manipulation. The change adds a file to
>> the RDT cgroup which represents the CBM(cache bit mask) for the cgroup.
>>
>> The RDT cgroup follows cgroup hierarchy ,mkdir and adding tasks to the
>> cgroup never fails.  When a child cgroup is created it inherits the
>> CLOSid and the CBM from its parent.  When a user changes the default
>> CBM for a cgroup, a new CLOSid may be allocated if the CBM was not
>> used before. If the new CBM is the one that is already used, the
>> count for that CLOSid<->CBM is incremented. The changing of 'cbm'
>> may fail with -ENOSPC once the kernel runs out of maximum CLOSids it
>> can support.
>> User can create as many cgroups as he wants but having different CBMs
>> at the same time is restricted by the maximum number of CLOSids
>> (multiple cgroups can have the same CBM).
>> Kernel maintains a CLOSid<->cbm mapping which keeps count
>> of cgroups using a CLOSid.
>>
>> The tasks in the CAT cgroup would get to fill the LLC cache represented
>> by the cgroup's 'cbm' file.
>>
>> Reuse of CLOSids for cgroups with same bitmask also has following
>> advantages:
>> - This helps to use the scant CLOSids optimally.
>> - This also implies that during context switch, write to PQR-MSR is done
>> only when a task with a different bitmask is scheduled in.
>
> I feel a bit underwhelmed about this new controller and its interface.
> It is evidently at a lot lower level and way more niche than what
> other controllers are doing, even cpuset.  At the same time, as long
> as it's well isolated, it piggybacking on cgroup should be okay.

This cgroup subsystem would basically let the user partition one of the Platform 
shared resource , the LLC cache. This could be extended in future to partition 
more shared resources when there is hardware support that way we may eventually have more 
files in the cgroup. RDT is a generic term for platform resource sharing.
For more information you can refer to section 17.15 of Intel SDM.
We did go through quite a bit of discussion on lkml regarding adding 
the cgroup interface for CAT and the patches were posted only after that.
This cgroup would not interact with other cgroups in the sense would not modify 
or add any elements to existing cgroups - there was such a proposal but was 
removed as we did not get agreement on lkml.

the original lkml thread is here from 10/2014 for your reference -
https://lkml.org/lkml/2014/10/16/568

   I
> take it that the feature implemented is too coarse to allow for weight
> based distribution?
>
Could you please clarify more on this ? However there is a limitation from 
hardware that there have to be a minimum of 2 bits in the cbm if thats what you 
referred to. Otherwise the bits in the cbm directly map to the number of cache 
ways and hence the cache capacity ..

Thanks,
Vikas

> Thanks.
>
> -- 
> tejun
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/