Received: by 2002:a05:6359:c8b:b0:c7:702f:21d4 with SMTP id go11csp751873rwb; Fri, 7 Oct 2022 03:50:01 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6d5EaczX3FdQhOTS8Vf5W9yxDn6AtmlS1OcAT+bF0WGPjgvLopo3lQiBbPEVAzJIt8B23j X-Received: by 2002:a63:6c01:0:b0:429:ea6e:486d with SMTP id h1-20020a636c01000000b00429ea6e486dmr3970254pgc.247.1665139801553; Fri, 07 Oct 2022 03:50:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665139801; cv=none; d=google.com; s=arc-20160816; b=eDZhHfHXhBjVYbWar6vke5XzPPQngl6w/xgZhnEs2ddY4tTjra/ya8wECtUlJJcVTg jnFocnEP4IHtjuqV703fR9HQwd8QpCp6/oOwCnV/aDoPH9Qpy+dyUyTHEsW9fCEjLBuO yZA66h92Dh8hMUZANVcCbjK8NGvNacdb76588gJedEeYpaI1OlMchEYp+D0j7DRvN54P Z9aAc75QPGS1NV7e886yDUgi+QgCFgFY81NqniFPetGIfapI0ByXW4pStBt4o1lr3Kmm ZMku6FCJ/ElLd63i+CJY+SRLjbhBKTgqRFSmFnNtvVLUF7IpzdFLgTqIvhe8ZBZWtnk0 7nuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:subject:message-id:date:from:mime-version :dkim-signature; bh=ISJQz79DPSkm66hvmvBdJ6e82SZwboKuGqw3xkwjuPw=; b=p67fuG4gvfa7htIuqXnRmmMxx9wIt5/8dR+nD6Mhz+7AhfZlFOcw/5mcH+HNcmBTSB tT7ZbfJv2a9M2acglNt8bmYyf1ozb4GKVDQXiZjgTKbarSXzOUjaPOYDTsvZAnUR/jYy W8HtbkHYk+y1pRmI7xLnFCHEjaAMUa4J+RMu32j4sDfSZK2zvl3k9kCiesnEShUT1Dv7 AXUzijockzI5zvzmpZYZh+ahWtCAmMRapaheD8vZVuy0YN53bN6rlgWxv2oLXEjZAImO paH+ualrB7jPIUcznsvk5vAjsWLqAF+5FXaLKgJa82OIQOGivGixl6xrvU5mmkjo2tSk 9wcQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=B+ULrpmv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c19-20020a63ea13000000b0044af51e7d4bsi2200619pgi.75.2022.10.07.03.49.46; Fri, 07 Oct 2022 03:50:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=B+ULrpmv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229581AbiJGKjt (ORCPT + 99 others); Fri, 7 Oct 2022 06:39:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37872 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229543AbiJGKjs (ORCPT ); Fri, 7 Oct 2022 06:39:48 -0400 Received: from mail-yw1-x1130.google.com (mail-yw1-x1130.google.com [IPv6:2607:f8b0:4864:20::1130]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7ECF1DEA8 for ; Fri, 7 Oct 2022 03:39:45 -0700 (PDT) Received: by mail-yw1-x1130.google.com with SMTP id 00721157ae682-354c7abf786so41465837b3.0 for ; Fri, 07 Oct 2022 03:39:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=ISJQz79DPSkm66hvmvBdJ6e82SZwboKuGqw3xkwjuPw=; b=B+ULrpmvR9yACLV9AGYO9l9qA4tUtNbBrI1JPJTlEH36kugEdg4LLcpsNnVWpG9Xlg L7w7Wn0ylufwWWf71d2Z/6G7JFmAxnQo98PO3vKHKbkiPE/DyxZo2W4+5YjyhwGBvgfK n3doFefS2Tc5lQ+fmNIk+HS4peX/otJ17cHhh6D+y98mgsynG+nC4I4fE5MCdp5R6Z48 w93GDPDlNIc9+WsHATYdASLtv/84gYRECPQpN72clbiOclLoxI036icbureEucIctuu4 NneXrnWlKtCk4rUaRlHFbQdSuQaTNHN9AnByq+CxFYTR0a4YT2/TldCJHO4vTUX1vGFM 4UOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ISJQz79DPSkm66hvmvBdJ6e82SZwboKuGqw3xkwjuPw=; b=aO9pgezuZF7jyKsN5qINIU2ilkATRIQQ1OK+3h8Hq1lC4+oj/fbIlDl5mhad4ITcUz oCzBrGQTZCPJc3RzT866FG4ZvPfRiWNsaF3aRF6txFChsNMpm8LVl+XHVVx0FiMwZYBh /NzTbWAsNR2MorWrlOFH7oe/0Q0DJ1mqBbNMqDIJLbpQK7I7eQchRUMDUovFZHkXINUF MB0o84Ejf1B8UUagVQ1bzdU99FMXaEjyioSNm70C/mzQ0bOfC/rJQIT48soU3o4ohjiq COv8ebxs5Vf7c9weh/lxKazlfNEgL0iOnCqotx0BUQY6c7QGS3WnpD/Kd/9a1a0vIYNN 7cVw== X-Gm-Message-State: ACrzQf0ZxdpACadQmK7ub7+CzpU2lISjAyXh5uiEYEELjpiOppigfn4G 1KjWypUAHeuZmEHb/v1bruSL4yZXnoVAdSJ/wkZZdA== X-Received: by 2002:a81:1204:0:b0:356:d2ae:a29 with SMTP id 4-20020a811204000000b00356d2ae0a29mr3835103yws.398.1665139184656; Fri, 07 Oct 2022 03:39:44 -0700 (PDT) MIME-Version: 1.0 From: Peter Newman Date: Fri, 7 Oct 2022 12:39:33 +0200 Message-ID: Subject: [RFD] resctrl: reassigning a running container's CTRL_MON group To: Reinette Chatre , Fenghua Yu Cc: Stephane Eranian , linux-kernel@vger.kernel.org, Thomas Gleixner , James Morse , Babu Moger Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Reinette, Fenghua, I'd like to talk about the tasks file interface in CTRL_MON and MON groups. For some background, we are using the memory-bandwidth monitoring and allocation features of resctrl to maintain QoS on external memory bandwidth for latency-sensitive containers to help enable batch containers to use up leftover CPU/memory resources on a machine. We also monitor the external memory bandwidth usage of all hosted containers to identify ones which are misusing their latency-sensitive CoS assignment and downgrade them to the batch CoS. The trouble is, container manager developers working with the tasks interface have complained that it's not usable for them because it takes many (or an unbounded number of) passes to move all tasks from a container over, as the list is always changing. Our solution for them is to remove the need for moving tasks between CTRL_MON groups. Because we are mainly using MB throttling to implement QoS, we only need two classes of service. Therefore we've modified resctrl to reuse existing CLOSIDs for CTRL_MON groups with identical configurations, allowing us to create a CTRL_MON group for every container. Instead of moving the tasks over, we only need to update their CTRL_MON group's schemata. Another benefit for us is that we do not need to also move all of the tasks over to a new monitoring group in the batch CTRL_MON group, and the usage counts remain intact. The CLOSID management rules would roughly be: 1. If an update would cause a CTRL_MON group's config to match that of an existing group, the CTRL_MON group's CLOSID should change to that of the existing group, where the definition of "match" is: all control values match in all domains for all resources, as well as the cpu masks matching. 2. If an update to a CTRL_MON group sharing a CLOSID with another group causes that group to no longer match any others, a new CLOSID must be allocated. 3. An update to a CTRL_MON group using a non-shared CLOSID which continues to not match any others follows the current resctrl behavior. Before I prepare any patches for review, I'm interested in any comments or suggestions on the use case and solution. Are there simpler strategies for reassigning a running container's tasks to a different CTRL_MON group that we should be considering first? Any concerns about the CLOSID-reusing behavior? The hope is existing users who aren't creating identically-configured CTRL_MON groups would be minimally impacted. Would it help if the proposed behavior were opt-in at mount-time? Thanks! -Peter