From: Vikas Shivappa <vikas.shivappa@linux.intel.com>
To: vikas.shivappa@intel.com
Cc: vikas.shivappa@linux.intel.com, x86@kernel.org,
        linux-kernel@vger.kernel.org, hpa@zytor.com, tglx@linutronix.de,
        mingo@kernel.org, peterz@infradead.org, matt.fleming@intel.com,
        will.auld@intel.com, glenn.p.williamson@intel.com,
        kanaka.d.juvva@intel.com, mtosatti@redhat.com
Subject: [DESIGN] x86/intel_rdt: Intel Cache Allocation interface proposal
Date: Sun, 23 Aug 2015 11:00:01 -0700
Message-Id: <1440352801-25013-1-git-send-email-vikas.shivappa@linux.intel.com>
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 6811
Lines: 195

This document tries to propose alternative interface for the Intel cache
allocation compared to the cgroup interface in the current patchset - 
http://marc.info/?l=linux-kernel&m=143889814520578

More info about cache allocation can be found in Intel SDM june 2015
volume 3, section 17.16.

Design overview:
---------------

OS maintains a mapping between task_struct and the class of service it
belongs to. This is done by adding a new field 'closid' in the
task_struct. Each closid is mapped to unique capacity bit mask(cbm)
  which indicates the cache capacity associated to the closid.

During scheduing, kernel writes this closid into IA32_PQOS_MSR to
indicate the hardware as to what Class of service(CLOS) the task belongs
to. 

It makes following changes to the current patch series :
 - Add kernel mode API to control cache allocations from with in the OS.
 - we dont use cgroup and instead expose controls through using sysfs in
 /sys/kernel directory for the administrator to configure the cache
 allocations. 
 - And optionally it also adds capabilities to add a control where
 process can change the cache allocation under the defined allocations
 by administrator.

The usecases targeted is mainly server clusters, cloud and container
based services and HPC workloads.  Users of cloud or containers would
get a VM/container to run the workloads and its most appropriate to
setup the static cache allocations for these units like VM/Containers.
For containers, many of the container based products like
Rancher/stackengine etc are docker based and allocate/manage
resources through a centralized orchestration/deployment tool.
Containers are quickly picking up in usage given the ease of deployment
of new containers and the scaling. These cache alloc interfaces try to
build a framework so that such use cases like cloud and container based
can easily adapt. 

Apps are restricted to self control the cache allocations as cache is
orders of magnitude scarce resource when we compare to other resources
like memory and will quickly run out of the resource if the apps
naturally try to use more of the resource to increase their own
performance.

kernel mode API:
---------------

enum cache_resource{
  l3_shared,
};

struct cache_alloc_config
{
  u32 max_cbm; 		    
  u32 max_closid; 	   
  unsigned long cache_size;
  int cdp_mode;
};

struct clos_cbm_table {
  	unsigned long l3_cbm;
	unsigned int clos_refcnt;
};

void cache_alloc_get_info(enum cache_resource cr, struct
cache_alloc_config &config); 

This returns the cache allocation configuration information along with
the cache size. Additionational capabilities can be added for example
the current mode whether code data prioritization(supporting both
    icache/dcache or legacy cache alloc).

int cache_alloc_set_cdpmode(bool setcdp); 

By default cdp(code data prioritization which supports allocation of
code and data seperately instead of common cache allocation) is not
enabled and can be set/reset with this API. Enabling cdp would reset all
the capacity bit masks and reduce the number of CLOSids to half. 
With cdp enabled the cbm can be extended to represent data and code
capacity mask (by having two u32).

void cache_alloc_get_cbm_table( struct clos_cbm_table *cctable, int
    size);
Returns the mapping of the current closids to the capacity bit masks. 

u32 cache_alloc_set_cbm( u32 preferred_closid, u32 cbm);

This reconfigures the capacity bitmask(cbm) for a preferred closid. If
the cbm is already present in the table, that closid is returned. That
way each unique cbm has one closid.

sysfs interface
---------------

This exposes files changeble by root in /sys/kernel/cache_alloc
directory.

clos_cbm_table :
Reading - this shows the max_cbm and the current snapshot of the
clos_cbm table.
writing - user can write the 'preferred closid' 'cbm' to change the
existing entry in the set of CLOS configs. 
If user writes a bitmask that already
exists it outputs indicating what closid has the cbm.
  $ echo <closid> <cbm> > /sys/kernel/cache_alloc/clos_cbm_table

Alternatively , instead of clos_cbm_table a directory for each clos
would be created with a file cbm in each directory. 

add_task :
write only: Can change the closid of any task by writing the 'pid'
'closid'. eg:
  $ echo <pid> <closid> > /sys/kernel/cache_alloc/add_task

threshold_clos : Can have two values 'lowest', 'all'. default to lowest.
When it lowest , a process can self change its closid to a different
closid but the new closid has to have the lowest capacity bitmask among
all the bitmasks. When its 'all' the process can change to any closid.
the interface is indicated below.

cdp_enable : takes 1/0 and by default is 0. Used to set cdp mode.

  $ ls /sys/kernel/cache_alloc
    add_task
    threshold_clos
    cdp_enable
    clos0/
    clos1/
    ...
    closn/

  
The closid of the task can be viewed in the /proc/<tid>/ stats.
The tasks would have closid 0 by default and would inherit parents
closid upon fork.

prctl/ syscall interface for process to change cache alloc
----------------------------------------------------------

This lets a process change its own cache allocation. However the amount
of change that can be done is limited. This is because L3 cache is a
very limited/scarce resource and can easily be exhausted by the first
few processes requesting more amount of cache. And this also lets one
centralized entity or a system-controlled mechanism which can be used
only by administrator to have a higher control in deciding the cache
allocation which is more useful in the scenarios described above.

struct cat_config {
  u32 max_cbm;
  u32 max_clos;
  unsigned long chunk_size;
  int any_clos_allowed;    
};

void cat_get_current_config(struct cat_config &config, struct
    clos_cbm_table &cctable);

This returns the max clos and cbm length and the current mappings of the
closid and the capacity masks. It also returns the chunk_size which
specifies the size of cache capacity that corresponds to one bit of cbm.
any_clos_allowed will be true if the threshold_clos is set
to 'any'. 

prctl(PR_SET_CLOSID, <new_closid>, ... );

Cache can be allocated in terms of bytes or percentages using this
interface. One can calculate the chunk size from the APIs and then
convert the size required to mask easily by using bitmask length = (size
    required/ chunk size). Also the bitmask gives the flexibility to
have exclusive, completely overlapping or partially overlapping cache
areas which can be adjusted based on the requirements of the workloads.


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/