Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Message-ID: <48fb48fa-c65d-8e38-dabb-cf9be21365ca@oracle.com>
Date:   Mon, 8 Nov 2021 12:20:26 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.3.0
Subject: Re: [PATCH] x86/smp: Factor out parts of native_smp_prepare_cpus()
Content-Language: en-US
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org,
        tglx@linutronix.de, bp@alien8.de, dave.hansen@linux.intel.com,
        x86@kernel.org, hpa@zytor.com, jgross@suse.com
References: <1635896196-18961-1-git-send-email-boris.ostrovsky@oracle.com>
 <YYk+D17oIOGKWNtN@hirez.programming.kicks-ass.net>
From:   Boris Ostrovsky <boris.ostrovsky@oracle.com>
In-Reply-To: <YYk+D17oIOGKWNtN@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eHFZTjlWeFFXaU5VV2JyQkdRWTRJbExFWTViZ2IvQWhkd2RlNFh1d1NkMGl3?=
 =?utf-8?B?dzR6OExIYXQ5TVRPNUxkdTF1Y3loUGlpVHFBZWdldWZCNklUd3NMZm9UNUJK?=
 =?utf-8?B?emVnTTl4QklUSnliRlk2QzJEdTlIckpnZUZhVGJEeWZQUW03d3dQZDloVnFx?=
 =?utf-8?B?UjRJU2NNditXclkxRWpxSWhqQ3Q2dFM4Tmx6NVhLZU1Mcng3ajZIVjBncmxi?=
 =?utf-8?B?dFQzRndZRjdCdDR2djBxQitYVFZVR0VvazBhKzlESzY1d29kRWpJai9vZnJI?=
 =?utf-8?B?Tk9KYWcxWXR2VUFFRUcrTHJDTkk1RVA4eXVhRGJVeXFSaCtGM09QamJuQ0Y3?=
 =?utf-8?B?cjcvV3MxVmdUWTB2Y3JhOWZSU1FXRitlL29DTzhZVFZPc3ZWalF0OFh4a3U1?=
 =?utf-8?B?YUUzL0tpbHMzOGRjOHNSeDJWZjF6LzQxN2JpSEJPK3BqNmdBTHk0S0NIQW9u?=
 =?utf-8?B?Vm5jbU50ZVRCaGtIZk53TkhmdG53dGQyd0Rqb0RMVnBRYWNaM09ia2xhSHow?=
 =?utf-8?B?OWNuRncrNHhUL0FyY3QrYlpZUmsrRlJqc1VDWHNRZ3prd3V6QkxNTDcwRXhz?=
 =?utf-8?B?cFVJQlEwNWhnWmJsQW83bmdTWUpZWFZOTTR6b1doUUFlNGFpMnZrTWUwZGRC?=
 =?utf-8?B?TkNGcWR4cmZ2d2VObDYxblB2dnhQM0ZTWmlFdmllanVtdk95T2RpdlRVTzlz?=
 =?utf-8?B?RVdvTWZVc05iL1VrN1pOeU4xcW9oUnFKR0pzTUZYcmlwV3k1SDdiLy9KMTZQ?=
 =?utf-8?B?bDdjMEN5S2ZRQ1hPY3dRMHJPSUtGZnBQMTB6dHBoMGV6eGxWRGkrdUtpbmRK?=
 =?utf-8?B?N1lKMTBtVHR6SWFlYXB2SCsvU050Z1FPeTFUZ2JrMXFySDdBSzRELzVnOFpk?=
 =?utf-8?B?MVZObjFOUytWcXp2UWpEdm0wOE93VWlzZGhFcDhkeHdhTmhXYzZ6dzRRVDUx?=
 =?utf-8?B?c05BZ0VNS2wyaSt1YWFyQjJoY1g5eHZqU1FkNUxrYkwrN2dDdVVBR1cvcDFD?=
 =?utf-8?B?bnFCTnFnUndSeDg4SFIyN0R1eEZpbG1ZNytNcTFwT1FsWUhpTUFpMHF1VndE?=
 =?utf-8?B?dTNoNk5rNnVkOEZoYXF3eVlFQ0dBbUg4eFFTSkdCMXVqVWZDUEQ4LzYva2VN?=
 =?utf-8?B?c2YrRUwxUUxTdTZkSVVuTFRFNklORFdyOWI5YjI2R2Y3alVaNS81aEVTbXJ3?=
 =?utf-8?B?K0RJRE52aWdJWENMdGRrMmc4TWgrb0ZvWVZlQjNCeXJrTnJRdnVOdTZ4a3NU?=
 =?utf-8?B?MnNBeHR5OTk4Z1NwS3MzWnN5U2RyQVQvMEYzbDU3T0t4L255aGdEUmR3T29G?=
 =?utf-8?B?NjNLeHQ0VTNIWEJldEdOM3dvRUJsQ1RReVMxS3ZqY2l5MTI2S1E2UXByc3Rw?=
 =?utf-8?B?MUJ4SmJLdkJpUDh4T1VzMGFGQTNEVDQvM245U2JVbFhZSFdlNEhrYWNmdVhO?=
 =?utf-8?B?eVBPMkRmQzhRM1NsMW9heUVRRUxGeGRaVXVYTi85SG04eU9SWDVLbnRSSnY0?=
 =?utf-8?B?V1YvbXkzbVVmMm4rNjZuNW5QRzFONHBHeUxrUlJObm5jNWc4SlNVbE01dkR4?=
 =?utf-8?B?Q0NCZExUMDg0TjVNUzQweEhIRFpud1FyaHZTR1I2NnQ2dXJzUjl1ZUIrZFUz?=
 =?utf-8?B?eklnbkkvbEtkUU1QRUxrbkVEODljRFh1VnZTRlV1bEVvTGtZSzFkRTBpUFRF?=
 =?utf-8?B?Z2FjS0J0aE1SQng5MEZGL2xCMDRraFFqb1YzQ0tQSEJOK3hKTFQwRFcwSE1p?=
 =?utf-8?B?ZFdzMDVHdVprdG1zcHVFd1ozWnc3OFA2enhtVnpvOWlZdG1LdEQ2eE9Pd2Vq?=
 =?utf-8?B?Yk1JaWNHKzFDVUhSNCtTcnQ0VGpTLzlCNDB0YUpPN2dINWx3V1NNNHRwdThp?=
 =?utf-8?B?d1h5c0RGYzVyZTY2c3h4dklteUZDY0gyUTRFRkU5V1Q1UGtCRkRVOXhZbjlD?=
 =?utf-8?B?dERIOUJ5KzFacWtoZkFSUmNQL3FSMFdaRjdqTXlFYTA2UGhtVCtHZHl2b2lu?=
 =?utf-8?B?dlc3VDJESmdUZ2dxd1NtM0JiU2xyNlIvNTdrc3ozUjZmVmZhczZIVkZLRnFV?=
 =?utf-8?B?ckd2UnEzVUo3em5XMXRuRHdMMGtyUEM3L1VPSWZZVzB2b29TT2Yva2hPTjlt?=
 =?utf-8?B?dytwZFRBSk1ERkZKc2xZZDJFTTdOazRjTFNRNkhlNXAybEZEZ3pFM0Jab3pv?=
 =?utf-8?B?Vk1vMFhvN2NXVWVKZ3FzNmlKMkM3YUFTQ2NWK2hMb3h4TmhtWDJqd0pIaFRH?=
 =?utf-8?B?NC8xcExvN0ZMOS9BMlNxQ1BzYUVnPT0=?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1d0e2fb8-4f73-4dbb-0969-08d9a2dc11f1
X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB5009.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Nov 2021 17:20:32.8105
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: OcdYYg6zioBhcAH7k7cMf7nPCSAStzCLb9q3SPR9NFPsF2d2LnUzXfKgvZaUrKpMSoVA5N4Jg1KOT6KT9xZ2GV/LNsClQxyBEVZBhk03cbI=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB3886
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10162 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 spamscore=0 mlxscore=0
 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2111080104
X-Proofpoint-GUID: WwJ6HiBW37uzhIH0qk1HR2IZoU-95bAv
X-Proofpoint-ORIG-GUID: WwJ6HiBW37uzhIH0qk1HR2IZoU-95bAv
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 11/8/21 10:11 AM, Peter Zijlstra wrote:
> On Tue, Nov 02, 2021 at 07:36:36PM -0400, Boris Ostrovsky wrote:
>> Commit 66558b730f25 ("sched: Add cluster scheduler level for x86")
>> introduced cpu_l2c_shared_map mask which is expected to be initialized
>> by smp_op.smp_prepare_cpus(). That commit only updated
>> native_smp_prepare_cpus() version but not xen_pv_smp_prepare_cpus().
>> As result Xen PV guests crash in set_cpu_sibling_map().
>>
>> While the new mask can be allocated in xen_pv_smp_prepare_cpus() one can
>> see that both versions of smp_prepare_cpus ops share a number of common
>> operations that can be factored out. So do that instead.
>>
>> Fixes: 66558b730f25 ("sched: Add cluster scheduler level for x86")
>> Signed-off-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> Thanks! I'll go stick that somewhere /urgent (I've had another report on
> that here:
>
>    https://lkml.kernel.org/r/20211105074139.GE174703@worktop.programming.kicks-ass.net
> )


Thank you. (I don't see this message btw)


>
> But looking at those functions; there seems to be more spurious
> differences. For example, the whole sched_topology thing.


I did look at that and thought this should be benign given that Xen PV is not really topology-aware. I didn't see anything that would be a cause for concern but perhaps you can point me to things I missed.


>
> Should we re-architect this whole smp_prepare_cpus() thing instead? Have
> a common function and a guest function? HyperV for instance seems to
> call native_smp_prepare_cpus() and then does something extra (as does
> xen_hvm).


Something like


void smp_prepare_cpus()

{

     // Code that this patch moved to smp_prepare_cpus_common();


    smp_ops.smp_prepare_cpus();  // Including baremetal

}


?


XenHVM and hyperV will need to call native smp_op too. Not sure this will be prettier than what it is now?


-boris


-boris