Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <aab04cfb-2dd5-89dc-213d-7fa253615864@amd.com>
Date:   Tue, 2 Aug 2022 11:46:32 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.1
From:   Ravi Bangoria <ravi.bangoria@amd.com>
Subject: Re: [RFC v2] perf: Rewrite core context handling
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     acme@kernel.org, alexander.shishkin@linux.intel.com,
        jolsa@redhat.com, namhyung@kernel.org, songliubraving@fb.com,
        eranian@google.com, alexey.budankov@linux.intel.com,
        ak@linux.intel.com, mark.rutland@arm.com, megha.dey@intel.com,
        frederic@kernel.org, maddy@linux.ibm.com, irogers@google.com,
        kim.phillips@amd.com, linux-kernel@vger.kernel.org,
        santosh.shukla@amd.com, ravi.bangoria@amd.com
References: <20220113134743.1292-1-ravi.bangoria@amd.com>
 <YqdLH+ZU/sf4n0pa@hirez.programming.kicks-ass.net>
 <YqdNKJllCVMci3ov@hirez.programming.kicks-ass.net>
Content-Language: en-US
In-Reply-To: <YqdNKJllCVMci3ov@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SHBqclgzNE84aEdvaGo4ejlmWlhBMjQ1UThIMldvc2xmbTNnKytBMUVDQ3Z1?=
 =?utf-8?B?cEh3M0E3bC9yNmw3K0Ura0JXNDROU1E5aU5SZGJmQjQ4MW5HZzZYZ055Rkpy?=
 =?utf-8?B?S2d5ZXJJZlZIeUEzd0JzWTA0eE42MG5EemRTZVV6OEJMUHlDS2tjSjRPUGxh?=
 =?utf-8?B?ZUV3L2hDaGZlaW95ekJKaFM4aXdVY2wwRTRWbGswZXh4WkExeVF0ckZaSHVN?=
 =?utf-8?B?eU9XRStGMGpLbGVXc0hQcE9sRjQwMzZ3VWMwVnBLNzZKZFo2L2hTaWxVZmRr?=
 =?utf-8?B?aVB0cllvNnVPbnVsWmhKdWJOdGsvZXhuMTBsYzRhdVdGMmhNVEdqV3NJRStX?=
 =?utf-8?B?K2JBK0E3ZmJhcUxxU1NCMWFHanpOaHhCYTg4QXNnQmo3Q2U2UG1XN1RkekJp?=
 =?utf-8?B?cGthQ09YUEtaOFUwYzMyc3l6THA3eWVlYUYzdW9rZUFuSEY1MHUzSlZTVEhG?=
 =?utf-8?B?UnFRVHBIOU9oMXRwNlJ6SlpFS2xVRFhEallaSXh2Rmk5ZjhyVUJBT3pqWGJw?=
 =?utf-8?B?RzdqcjFYTFBFdzlDMWVTL3pDMlRLVHVTaCtxNzQ1MWpzK0VKQyt3d0R1cjlK?=
 =?utf-8?B?L1pxQ0RBaUNaTEdGUmEyZ1djS053bldPaXY3NXdiWGtIcW9QV0hvbVVkeVlO?=
 =?utf-8?B?ck5oS2o1Z2MxYTRVZnMwcDg2TWZrcjZzeUFVandXME5ycGY1aEIvMkZnMDZk?=
 =?utf-8?B?L3RIZzhVVVExMkhUZ21jRHlEcGxZUTdjanhJSHJFK2FRRVg1TFBrR2xSbFdq?=
 =?utf-8?B?Y1hLZ3gwUEpUOXlxTyswN3ZlZThNUDl1MEE4dGlaSDdVdkxZM0xyQ3JmS0Ix?=
 =?utf-8?B?dlJZZkpyQmNFNmRZcXZRT0wvd1lSelFWMmlvRXhjbEFUZFFSMmh0YjRzdDZC?=
 =?utf-8?B?cWlnbmFaMUE4SnoyNVQ0S1ZzeXkrQ3BuZkI4L3RraFhNaFpPUG1meTd1a3gw?=
 =?utf-8?B?bTBEaUJabkFPN1ZjN2lVVENuN1JHa0o3ZmFjQWlIdWZUR1dObVIxanJ0dHJK?=
 =?utf-8?B?SUlBdWZyY3BFSFdpMXVDREFqaFFHWnRIeFZ6YWVhdTh5NlZFa0dqR21nZStU?=
 =?utf-8?B?aFM5VEJ4SmVKSVBXdWozQ0J5ZHpPc3dpR2M0dzMrSFZHdzdkeHV4K240dEJx?=
 =?utf-8?B?NDdpd1VjL0txd1ZkUXk4Y3g4SlQrWjYxaEd4MGRsZXRXdVVGcVd1RnlRelky?=
 =?utf-8?B?Qm0xalo1R0ZTeFlKM1FzZ2swNDhvTEx0TUdtVWZHOVJ0SmJzei9CellQMkVr?=
 =?utf-8?B?OWlkYTV0aGcyUXlnMEdUQmRYYUZRTUFlczNiakNpNkpDeWRwZFZub2JRYTV4?=
 =?utf-8?B?ZzlhUk5GSW1GUFhEeUUwenhDK3J1Nmp0ampXY2REOGU4cXZENEVaU0VrYkFG?=
 =?utf-8?B?MHluanhCNFBYdk5SVGRvc0t0SUgvQVRZcTdRaCs0VCtJMHdCUWErMndia1ZF?=
 =?utf-8?B?S3ZpTWVEWHFhQms4cWE4aVE0M3FnTW44eHJOMWhtY3kxZFVWTDk5ZVN6MS80?=
 =?utf-8?B?NTdVZUdxU0dUK0FWRDVVWWdJVHN6ckNLM3hYY3d0cWkyUlBXTm8wa0p6TFZp?=
 =?utf-8?B?R1hCRW8yN0o5SC9CaUxrQTVueDloeWlkQ2hsdmZlbGNYNHZORDRrVnNFMWdZ?=
 =?utf-8?B?SEZGdk5SbUszM2I0SFdUU01MNFdUckI2VjJjVE9ZRlJxVjQ4Y0U3b1lMYlVv?=
 =?utf-8?B?bE5GSmVSdVZiVk9hZktQYnZtUzk1Q3c2cEhwVEVxWUpHWU9UeVpvODlQVUxw?=
 =?utf-8?B?MlFRbTVZcDE3cHcwRWt4V0R0dkVzVUpudENRZm5ZOXV1SFFGYUxZQkpXSExD?=
 =?utf-8?B?U1p0V2tyMEhDU0FBTVZkd055dVQ3R205bVNhWnpDa3l2SXNMVFphaFRld3RH?=
 =?utf-8?B?dnJOenhtNTBvcy9QQk9QbFY1SUJSY2hnQ3RVSXpwVk1TS09WN2tiUFJuZ3Fp?=
 =?utf-8?B?WDBkSGNOUnFwMm0zK0dIbFBzVzBsY2Rqa1pTVFVVRzcydmFHVUp5VGlVRUxr?=
 =?utf-8?B?OUJSS01oeVN2bWV0N0ZvakcvUTVWTjhqQ2FvYVBFbVJkL1AzT2t3cjlBK1d3?=
 =?utf-8?B?WVRra09BQnlNTWZzSzdTV2hiekRRWWV4Y2k5R2E0cWgrVVl4K2V2RmlFRDEx?=
 =?utf-8?Q?Kllv9MpMVQtfMkEeaYG9rA2HA?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b51be613-1d61-4968-cddb-08da744e948e
X-MS-Exchange-CrossTenant-AuthSource: PH7PR12MB6588.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2022 06:16:47.8141
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: X5Ldf8B+5v/tMEo88Ae0gMvKQIIPmyDwOZW1AIHFW24YkPYl5nOscXAEOFiRLHNZeesDF0SWET4G6xxQ5CaVSw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB3095
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 13-Jun-22 8:13 PM, Peter Zijlstra wrote:
> On Mon, Jun 13, 2022 at 04:35:11PM +0200, Peter Zijlstra wrote:
> 
>> @@ -3652,17 +3697,28 @@ static noinline int visit_groups_merge(s
>>  			.size = ARRAY_SIZE(itrs),
>>  		};
>>  		/* Events not within a CPU context may be on any CPU. */
>> -		__heap_add(&event_heap, perf_event_groups_first(groups, -1, NULL));
>> +		__heap_add(&event_heap, perf_event_groups_first(groups, -1, pmu, NULL));
>>  	}
>>  	evt = event_heap.data;
>>  
>> -	__heap_add(&event_heap, perf_event_groups_first(groups, cpu, NULL));
>> +	__heap_add(&event_heap, perf_event_groups_first(groups, cpu, pmu, NULL));
>>  
>>  #ifdef CONFIG_CGROUP_PERF
>>  	for (; css; css = css->parent)
>> -		__heap_add(&event_heap, perf_event_groups_first(groups, cpu, css->cgroup));
>> +		__heap_add(&event_heap, perf_event_groups_first(groups, cpu, pmu, css->cgroup));
>>  #endif
>>  
>> +	if (event_heap.nr) {
>> +		/*
>> +		 * XXX: For now, visit_groups_merge() gets called with pmu
>> +		 * pointer never NULL. But these functions needs to be called
>> +		 * once for each pmu if I implement pmu=NULL optimization.
>> +		 */
>> +		__link_epc((*evt)->pmu_ctx);
>> +		perf_assert_pmu_disabled((*evt)->pmu_ctx->pmu);
>> +	}
>> +
>> +
>>  	min_heapify_all(&event_heap, &perf_min_heap);
>>  
>>  	while (event_heap.nr) {
> 
>> @@ -3741,39 +3799,67 @@ static int merge_sched_in(struct perf_ev
>>  	return 0;
>>  }
>>  
>> -static void
>> -ctx_pinned_sched_in(struct perf_event_context *ctx,
>> -		    struct perf_cpu_context *cpuctx)
>> +static void ctx_pinned_sched_in(struct perf_event_context *ctx, struct pmu *pmu)
>>  {
>> +	struct perf_event_pmu_context *pmu_ctx;
>>  	int can_add_hw = 1;
>>  
>> -	if (ctx != &cpuctx->ctx)
>> -		cpuctx = NULL;
>> -
>> -	visit_groups_merge(cpuctx, &ctx->pinned_groups,
>> -			   smp_processor_id(),
>> -			   merge_sched_in, &can_add_hw);
>> +	if (pmu) {
>> +		visit_groups_merge(ctx, &ctx->pinned_groups,
>> +				   smp_processor_id(), pmu,
>> +				   merge_sched_in, &can_add_hw);
>> +	} else {
>> +		/*
>> +		 * XXX: This can be optimized for per-task context by calling
>> +		 * visit_groups_merge() only once with:
>> +		 *   1) pmu=NULL
>> +		 *   2) Ignoring pmu in perf_event_groups_cmp() when it's NULL
>> +		 *   3) Making can_add_hw a per-pmu variable
>> +		 *
>> +		 * Though, it can not be opimized for per-cpu context because
>> +		 * per-cpu rb-tree consist of pmu-subtrees and pmu-subtrees
>> +		 * consist of cgroup-subtrees. i.e. a cgroup events of same
>> +		 * cgroup but different pmus are seperated out into respective
>> +		 * pmu-subtrees.
>> +		 */
>> +		list_for_each_entry(pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) {
>> +			can_add_hw = 1;
>> +			visit_groups_merge(ctx, &ctx->pinned_groups,
>> +					   smp_processor_id(), pmu_ctx->pmu,
>> +					   merge_sched_in, &can_add_hw);
>> +		}
>> +	}
>>  }
> 
> I'm not sure I follow.. task context can have multiple PMUs just the
> same as CPU context can, that's more or less the entire point of the
> patch.

Current rbtree key is {cpu, cgroup_id, group_idx}. However, effective key for
task specific context is {cpu, group_idx} because cgroup_id is always 0. And
effective key for cpu specific context is {cgroup_id, group_idx} because cpu
is same for entire rbtree.

With New design, rbtree key will be {cpu, pmu, cgroup_id, group_idx}. But as
explained above, effective key for task specific context will be {cpu, pmu,
group_idx}. Thus, we can handle pmu=NULL in visit_groups_merge(), same as you
did in the very first RFC[1]. (This may make things more complicated though
because we might also need to increase heap size to accommodate all pmu events
in single heap. Current heap size is 2 for task specific context, which is
sufficient if we iterate over all pmus).

Same optimization won't work for cpu specific context because, it's effective
key would be {pmu, cgroup_id, group_idx} i.e. each pmu subtree is made up of
cgroup subtrees.

Please correct me if my understanding is wrong.

Thanks,
Ravi

[1]:
https://lore.kernel.org/lkml/20181010104559.GO5728@hirez.programming.kicks-ass.net