Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <d3c06fa3-83a4-7ade-6b08-3a7259aa6c4b@intel.com>
Date:   Tue, 1 Nov 2022 09:48:56 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.4.1
Subject: Re: [RFD] resctrl: reassigning a running container's CTRL_MON group
To:     Peter Newman <peternewman@google.com>
CC:     James Morse <james.morse@arm.com>, Tony Luck <tony.luck@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "Eranian, Stephane" <eranian@google.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "Thomas Gleixner" <tglx@linutronix.de>,
        Babu Moger <Babu.Moger@amd.com>,
        "Gaurang Upasani" <gupasani@google.com>
References: <CALPaoCj-zav4x6H3ffXo_O+RFan8Qb-uLy-DdtkaQTfuxY4a0w@mail.gmail.com>
 <Y0BhzKkksSjSeE3W@agluck-desk3.sc.intel.com>
 <81a7b4f6-fbb5-380e-532d-f2c1fc49b515@intel.com>
 <CALPaoCjdeRjyX5L6BBX688ZM21eMwetuL9QLF1+GEDUskGcU2w@mail.gmail.com>
 <76bb4dc9-ab7c-4cb6-d1bf-26436c88c6e2@arm.com>
 <CALPaoCiKUQC+LxDwKQ0gE5AQniJi_nbzrXi_HA9ZBRtiXdw_dg@mail.gmail.com>
 <835d769b-3662-7be5-dcdd-804cb1f3999a@arm.com>
 <CALPaoCgiwtvxZMJ6f-BOTNMPXDbMB8QM42HyCAvYRi4grPiphQ@mail.gmail.com>
 <CALPaoCg2-9ARbK+MEgdvdcjJtSy_2H6YeRkLrT97zgy8Aro3Vg@mail.gmail.com>
 <715e4123-fdb3-a71e-4069-91d16a56a308@arm.com>
 <CALPaoCjUk31HvhaqWKfkLNLyZJzbDLJOnEqTsqn882CToLCGcw@mail.gmail.com>
 <317b4a96-f28d-aab5-57cc-f0222b7e4901@intel.com>
 <CALPaoChNAaoJ7uzuxFFDn-f5nvUCXyJ4jkbETHXNdDpXNnh9pQ@mail.gmail.com>
 <08c0e91a-a17a-5dad-0638-800a4db5034f@intel.com>
 <CALPaoCj8bVo3Z1r9_Ag=6KvGuR2wzQesArwZKEDvudGPYbbwaA@mail.gmail.com>
 <CALPaoCikWKz8VLf5QdeYm5k58tePTpb3nTKMGJ+qeGmevjvMpg@mail.gmail.com>
Content-Language: en-US
From:   Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <CALPaoCikWKz8VLf5QdeYm5k58tePTpb3nTKMGJ+qeGmevjvMpg@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?K2FPS2JGUU5ITUpRUXMyaXl5N1VmeVRLemdDY1RyaDFVNkhpNDVTUklCYkt1?=
 =?utf-8?B?Q1JYYjVYSW5pZGV4ZnI2SC9IRjZKNFA0V1JOK1cvc1pJa250RWhDUktJUWFV?=
 =?utf-8?B?TDNKNU9OcUhwUC9VakY3TWRXd0h6SUpFU05BWWYwNnVaUDRLcXhyeGs5ZHlq?=
 =?utf-8?B?NGV5QlQ0QVNRVUtVMUVDUzgxazZZaE1laWhmUm0vZ1lvYVRlTkVPN05zNkdG?=
 =?utf-8?B?V3UyZVJlZXhTeXJxNlBGYlFNUjkxc1pBZStKdWRreE1GWFRDb2Q2QWFhSFhR?=
 =?utf-8?B?N2FuaVIxQ2NxQVpQd2xRUVNXM3A2N3VSOXhQTTN2QmlZaC8xOHY3OWwyeTk0?=
 =?utf-8?B?QS9kKzJKNnlGVjc0d05VMFhFQlNGa29tZ21kYy9zTVJ2dzFuZTFKWFJaYVZ5?=
 =?utf-8?B?SDJPZGdtS2swKytPMnljN3p2SXQyMExiU3A4Ylp6YWxaTTVsd2k4SncrUG1N?=
 =?utf-8?B?TFpJa2FSMW9oeFFvZFdIWkxDdmJ2Y1IvSUUxM3dqRlZ1NlNVUkNCVWFHTXNm?=
 =?utf-8?B?RmJzOHp4cnArak9QTy92d2huK2pRcmNMcjRrUWIvWUd6WkU2UjNyUHpldUdi?=
 =?utf-8?B?RUk0aHVhaFlmbU1oTEFaT1pYNTQrVHlYN3NHYWdtMVc2dEVHM3N2OEJyODJD?=
 =?utf-8?B?STZQOGFMdGdXUEp5bUhCVjhWOG5xc2hrdmphV2ZObzhGK01TVkg3YWZCN05H?=
 =?utf-8?B?K2hnVkRhM1lRNzlSZk9KRkxmOG5kelZWNDZ2dkpyVEhLenFFbERYZVd5Tndz?=
 =?utf-8?B?RkhsUGNkT3ErWjN2Znk2end1RnM3bVd0TFNVNC9OYlFXR1JMVFkxMWx6NjFB?=
 =?utf-8?B?L3RvdzdjbXBEcXEwNGRiWjlqVVIrRW1tK2pwOXZXNWxDTEVDMWdqbXdPZ2tX?=
 =?utf-8?B?bXBHaVJrUnQvdGZac2p4S1V5bUFnZE1qbkFON09VMmtrM29kMHI3RW1KNnZi?=
 =?utf-8?B?anNYRitkdHdiZGEra2psSVFMRnRUOUpIUTRYNFBoUk1TcnZCTjF4a01PR2ZH?=
 =?utf-8?B?ZWlEUzZVN1dFQk1nU3g1NGhhRnNUeFFEMlR1c0N1YmttRC9vZzN4N3Zwbjlk?=
 =?utf-8?B?VlRvVlc5WEVBV3lrQ0xUNS9YSUpaTmYzSmNhZEh4Qy9yeTM5OUlYUVhKVmpz?=
 =?utf-8?B?STYrQ0FzV1hSR1NJZFY5UmpvYkRmMjNyZ0FTdE43VWl1cmpzNytNT1I1aTRn?=
 =?utf-8?B?WmdvZU5UL1JYQ01wcVJBTTVNZ3M1c1F5b1diWktySFJOM1NVd0VjZEJyZ3Jh?=
 =?utf-8?B?WGdDSmpVNzU0bjJEcHhqYng0aklZMlM5cktyRjNVQTNGcVBVWVM0a2dnbFZr?=
 =?utf-8?B?azRFblNiMnUyZlhtOE1RMUNSWEovK3F4ZVdaMlFPN2sva3VDMStNUFMzeDJp?=
 =?utf-8?B?Q3pmS3hEc242ZEpISlB3TUIxMjZmTERzbnUwaFhTYWtQRVgzbEpqSUMyNzZY?=
 =?utf-8?B?QkIxWFpLcmdxM3FVQ3BZUlBrbXUxZGttNXp4NlFwdWxUSHllaVpZY3VWWjlT?=
 =?utf-8?B?UTdiZE9YRHR3MERiaHM5WCt1NUd3aENSNlc1Y1ViOExDSE5TRXplRDZwZ2tH?=
 =?utf-8?B?OHdVUHMxL1NsNk9BSkNIanh1UEsyM2trQm8wVS9wTDBxRGFXWGJjTnlHVHZM?=
 =?utf-8?B?RWF5Kys5anFFUW1BT3dheWZzTjRvV2JWdDllaEViNERFRmpZRnExYWZ6c3c3?=
 =?utf-8?B?eFc5Mzl5a0NHSk04L3Z5MGZVM2ErYTlrbDZzdklmOGRmSitQbGxIdzF3dHVB?=
 =?utf-8?B?ZU15alI4VGxIcDRHTUJRdm1qV0c5TXY0SEg2cDkwMUNHVXZJbHV0SFV1T1dE?=
 =?utf-8?B?cDNHMkVkZzYvV2k3SEQrMDlrSTk0M1BPTTlmQXpXbStCMzJBWUw0dDYrUzJo?=
 =?utf-8?B?TFdyeW54SHNGamtoZHhqWkpqY2FlNStEMzVTOWk4TnJ1blRkT3JraGhqU3Vv?=
 =?utf-8?B?Q2VyM0xOVWgxWnc0Sk1DMGxoQ0tMS1ZsWmVtNlhlQUlQYURyNTR0V3dITFpq?=
 =?utf-8?B?aHFCam5HZmI3SkhUYkluc293MmZZOWNGeUVnVDRpZm9razhDY3l3eS96aHBF?=
 =?utf-8?B?dHF3WUtILzlndXUwcW1vVFgzOXlRY3RxV0s3VEFpcjMvczB1MXRmdHErN01V?=
 =?utf-8?B?cXF5UTFmN3NrYU9GUEFyVjRkSFpGMVRSVkVQUkdsVE5HbXZ0OERvWjAybUI5?=
 =?utf-8?B?WUE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c1235ed1-6159-4d95-f703-08dabc28f918
X-MS-Exchange-CrossTenant-AuthSource: CY4PR11MB1862.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Nov 2022 16:48:59.1785
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: X6pJlrcjK7+bQAsjCwuuIxFsxc+OtrAAGevLJp+BbXoeszQy8wRtkxh0fJXM9PYlzDGFChrztvfjhXV/UB40iqnnGM9ETY0C903RyrMZr3w=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB5708
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Peter,

On 11/1/2022 8:53 AM, Peter Newman wrote:
> On Tue, Nov 1, 2022 at 4:23 PM Peter Newman <peternewman@google.com> wrote:
>> Yes it looks like the task's rq_lock would provide the necessary
>> ordering. It's not feasible to ensure the IPI arrives before the target
>> task migrates away, but the task would need to obtain the same lock in
>> order to migrate off of its current CPU, so that alone would ensure the
>> next migration would observe the updates.
>>
>> The difficulty is this lock is private to sched/, so I'd have to propose
>> some API.

I thought that we could do something similar to cgroup_move_task(). Instead
of new API it seems that the custom is for subsystems to move their scheduler
related code to kernel/sched/. For example, a new
kernel/sched/resctrl.c that implements the task moving code that benefits
from the private sched/ APIs.

But ...

> Actually it looks like I can just use task_call_func() to lock down the
> task while we do our updates and decide if or where to send IPIs. That
> seems easy enough.

Indeed, this does look promising. Thanks for finding that.

If you do pursue something like this I assume that you have some
challenging environments in which to try it out? I am curious about the
user space visible impact of the additional locking on a task move when
the number of tasks being moved is high.

Reinette