Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752301AbbKPOkx (ORCPT ); Mon, 16 Nov 2015 09:40:53 -0500 Received: from mx1.redhat.com ([209.132.183.28]:47174 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752366AbbKPOhb (ORCPT ); Mon, 16 Nov 2015 09:37:31 -0500 Date: Mon, 16 Nov 2015 11:03:57 -0200 From: Marcelo Tosatti To: Peter Zijlstra Cc: Luiz Capitulino , Thomas Gleixner , Vikas Shivappa , Tejun Heo , Yu Fenghua , linux-kernel@vger.kernel.org Subject: Re: [PATCH RFC] ioctl based CAT interface Message-ID: <20151116130357.GA25951@amt.cnet> References: <20151113163933.GA10222@amt.cnet> <20151113165100.GI17308@twins.programming.kicks-ass.net> <20151113172740.GA13490@amt.cnet> <20151113174300.GA14726@amt.cnet> <20151116085926.GK17308@twins.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151116085926.GK17308@twins.programming.kicks-ass.net> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4864 Lines: 163 On Mon, Nov 16, 2015 at 09:59:26AM +0100, Peter Zijlstra wrote: > On Fri, Nov 13, 2015 at 03:43:02PM -0200, Marcelo Tosatti wrote: > > Subject: cat cgroup interface proposal (non hierarchical) was Re: [PATCH > > V15 00/11] x86: Intel Cache > > Allocation Technology Support > > > > > > https://lkml.org/lkml/2015/11/2/700 > > I've really no idea what you're trying to say there. That just doesn't > parse. I've posted this internally at people understood and commented on it. What part you don't understand? Its a proposal for a cgroups based interface. The description starts from the main mount point of the cgroup. So you would: # mount -t cgroup -ointel_cat intel_cat /sys/fs/cgroup/cat/ # cd /sys/fs/cgroup/cat/ # ls cat_hw_info # mkdir cache_reservation_for_forwarding_app # ls cat_hw_info cache_reservation_for_forwarding_app # cd cache_reservation_for_forwarding_app # ls type size socketmask enable_reservation # ... Yes? cgroup CAT interface (non hierarchical): --------------------------------------- 0) Main directory files: cat_hw_info ----------- CAT HW information: CBM length, CDP supported, etc. Information reported per-socket, as sockets can have different configurations. Perhaps should be inside sysfs. 1) Sub-directories represent cache reservations (size,type). mkdir cache_reservation_for_forwarding_app cd cache_reservation_for_forwarding_app echo "80KB" > size echo "data_and_code" > type echo "socketmask=0xfffff" > socketmask (optional) echo "1" > enable_reservation echo "pid-of-forwarding-main-thread pid-of-helper-thread ..." > tasks Files: type ---------------- {data_and_code, only_code, only_data}. Type of L3 CAT cache allocation to use. only_code,only_data only supported on CDP capable processors. size ---- size of L3 cache reservation. rounding -------- {round_down,round_up} whether to round up / round down allocation size in kbytes, to cache-way size. Default: round_up socketmask ---------- Mask of sockets where the reservation is in effect. A zero bit means the task will not have the L3 cache portion that the cgroup references reserved on that socket. Default: all sockets set. enable ------ Allocate reservation with parameters set above. When a reservation is enabled, it reserves L3 cache space on any socket thats specified in "socketmask". After cgroup has been enabled by a write of "1" to "enable_reservation" file, only the "tasks" file can be modified. To change the size of a cgroup reservation, recreate the directory. tasks ----- Contains the list of tasks which use this cache reservation. Error reporting --------------- Errors are reported in response to write as appropriate: for example, write 1 > enable when there is not enough space for "socketmask" would return -ENOSPC, etc. Write to "enable" without size being set would return -EINVAL, etc. Listing ------- To list which reservations are in place, search for subdirectories where "enabled" file has value 1. Semantics: A task has guaranteed cache reservation on any CPU where its scheduled in, for the lifetime of the cgroup, as long as that task is not attached to further cgroups. That is, a task belonging to cgroup-A can have its cache reservation invalidated when attached to cgroup-B, (reasoning: it might be necessary to reallocate the CBMs to respect contiguous bits in cache, a restriction of the CAT HW interface). ------- BLOCKER ------- Can't use cgroups for CAT because: "Control Groups extends the kernel as follows: - Each task in the system has a reference-counted pointer to a css_set. - A css_set contains a set of reference-counted pointers to cgroup_subsys_state objects, one for each cgroup subsystem registered in the system." You need a task to be part of two cgroups at one time, to support the following configuration: Task-A: 70% of cache reserved exclusively (reservation-0). 20% of cache reserved (reservation-1). Task-B: 20% of cache reserved (reservation-1). Unless reservations are created separately, then added to cgroups: mount -t cgroup ... /../catcgroup/ cd /../catcgroup/ # create reservations cd reservations mkdir reservation-1 echo "80K" > size echo "socketmask" > ... echo "1" > enable mkdir reservation-2 echo "160K" > size echo "socketmask" > ... echo "1" > enable # attach reservation to cgroups cd /../catcgroup/ mkdir cgroup-for-threaded-app echo reservation-1 reservation-2 > reservations echo "mainthread" > tasks cd .. mkdir cgroup-for-helper-thread echo reservation-1 > reservations echo "helperthread" > tasks cd .. This way mainthread and helperthread can share "reservation-1". -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/