Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <9dfe87f0-fc95-6c28-6695-62f1f5403df6@oracle.com>
Date:   Tue, 13 Dec 2022 16:25:32 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.5.1
Content-Language: en-US
To:     Krister Johansen <kjlx@templeofstupid.com>
Cc:     Juergen Gross <jgross@suse.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>, x86@kernel.org,
        "H. Peter Anvin" <hpa@zytor.com>, xen-devel@lists.xenproject.org,
        linux-kernel@vger.kernel.org,
        Marcelo Tosatti <mtosatti@redhat.com>,
        Anthony Liguori <aliguori@amazon.com>,
        David Reaver <me@davidreaver.com>,
        Brendan Gregg <brendan@intel.com>
References: <20221208163650.GA3225@templeofstupid.com>
 <1e6c1b08-d573-fba9-61fd-d40a74427d46@oracle.com>
 <20221212155730.GA1973@templeofstupid.com>
 <20221212160524.GB1973@templeofstupid.com>
 <d1a2b785-edc7-b7da-d2f2-123d1555022e@oracle.com>
 <20221212220959.GB1935@templeofstupid.com>
From:   Boris Ostrovsky <boris.ostrovsky@oracle.com>
Subject: Re: [PATCH linux-next v2] x86/xen/time: prefer tsc as clocksource
 when it is invariant
In-Reply-To: <20221212220959.GB1935@templeofstupid.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YTBJVHhNQVpvUEVvdTIyTHNmQlVYM0EwcW1scklJbDNXRTA1YzJ3SWd1T1hr?=
 =?utf-8?B?OFdNUWdKT3d6cXo5Vm5CN3U4eGliM2pHM0tiOXBQWHZCbHd5Rk5Gd0s0QXQz?=
 =?utf-8?B?RVYreDVhZmhHOHJ4ditrWG9lYWxnc3JvbHBiMVdkRHNzVHFQWGtNY09mRkZ4?=
 =?utf-8?B?aFpnS1Vwck5lTVlZQmNHYmVyMm80Z1VJSjhEckxONVY4YjBmc3ZPMlNDK2Jq?=
 =?utf-8?B?cWtLc1ZIVGRWL3R6QjNOZytiVnpXamtZWjBDTkovMElZdUk4VDhibHRzenNB?=
 =?utf-8?B?L0xUR3NWZ1hpcmN5eWRhWFhkTTRxOFRKOEg5Y3NIM3V5RUhxNldJeTJUeGVE?=
 =?utf-8?B?dlE2SjVzeXdZaG12N0o0dDkySUVVOXRlZ0JzMnNIN1U1NEw5cHNPTXBCZVE2?=
 =?utf-8?B?REJsVm9yNHNDSjltQ2NGWjJTdXBUK1EvckdQcEgrUU4wbXA1UTlUVVJUM0Vp?=
 =?utf-8?B?YlF5VTIyTVRMbWhxZy8vNEVMN0xTZGdCbUxpUGRFN2Fib0hyUUEyU1VnU1Rk?=
 =?utf-8?B?SE45cDI1TWZHWDBTeExKMDZxYWo2cDZFWThXenhzQXd1RGhzcXgrc09sUlgw?=
 =?utf-8?B?Q3dZR3h6Q0o4cXpkUVBWc3J3VGhJemlua0hzR2NzUUR2d25DTFhRZ1J6dG9y?=
 =?utf-8?B?OGh6WEU1ZlZ3RWEybzdKcXVpcG1Na0lXajViQ0dNQmI0Rzh0MzFyWThUZXgv?=
 =?utf-8?B?NndUc2g3NlkrMU1YRG9rcjhZajlkUHMzcmxOT0d0UHdzcXhJVG5DQkkvNlRw?=
 =?utf-8?B?d3gvTEFkdW15b2FPR1JocUtDMUFXb2daS1Y1YkY2MFQ3SmRZQSsxbzVjbTRH?=
 =?utf-8?B?aHE1MjRiVzBEMW5BMjNFNDVsbXNzL0wyK2pENVhUUytEWFpjNWJGZmVmR0pZ?=
 =?utf-8?B?Ujl4R3RWakpiK1RzemVYdGlvbHVEKzd2SHIxK1laWldaRWpxY2tmUERobytY?=
 =?utf-8?B?STE0aWJRcWZqZlVOTkdPSlp0bjdleUpMTDdQbW1sRWxKdDUxOWt5dTh1VWY0?=
 =?utf-8?B?OUYxb2tXSkZEYVRZWDJSS2VyOWhocjlKbUhOclhUZEZaT0pkWUk5ZzVTVlgx?=
 =?utf-8?B?VnE1a242WG84QVhIVW9mbzJ1Y0RmaG9QK1FPVUk1UkQ5OEtNV01PYzRSY1dC?=
 =?utf-8?B?VUxOMFhKZGNxZXdBOUtMSER0aUdLUkNYbVZwS3hlNTlDcTNQQjZTblZVckN2?=
 =?utf-8?B?dDZpMkpxRmR6eTJkZEM1ZVZleUhUa1ZUMHpJK0Vod2tTdHlYdi8rNi9STjEw?=
 =?utf-8?B?am96aitHQUFnMUdlY0RxWGVHSkFBNjVnTjE1U3kyTFpDUlJTWHpmMUNVYzdV?=
 =?utf-8?B?WUFPbSt1cHhTWFQ3ejV6ZjA1enVIQWtVRXpXWC9NTFVPSFowc1VZTCttTHJy?=
 =?utf-8?B?THFVVlF5K1NSMGhJR2lkRFFXZ0FkZStmRjh5bm5xQm42dVdLcGlNYXNra3pK?=
 =?utf-8?B?L0tOWkZBWnN0RE9VV2w5WFNod1pXVFhUKzFMekhwdzFnOFloZmtnczc5T0hH?=
 =?utf-8?B?TmlwMjc1MjVzdSs1ZVU1dXNGckJyZDdwbUFPcUZ5NTNnQ0ZGZVhjcCtPaXFB?=
 =?utf-8?B?aDN6WU9tSnhIV1FWcTZGQTFPeXdpdjV5VElpME94SXlmc3hDTENFbWo2NzVs?=
 =?utf-8?B?YzZERWVzTjRoUlF6azVkMXRqZWVVbFdHVFh5ZnJTQUNmUzhhRGZvdllISVFH?=
 =?utf-8?B?d1hMVmVRQWJGMHArYjYvZGRGMG1sQytLUWxTS2hpTVlITEVxWVVZdW5LaVBi?=
 =?utf-8?B?aUhZM0RWV3Rvd05uT2ZWVjF3R1hEcUdGcGhMM0UrcDQ2QVkxOHB2RDUwN05O?=
 =?utf-8?B?cUNwOGREakNKZ3AyTzk3VHBKVDhqY1BBWTVVRm4zNlpIU3d2NWk3WS8rS01r?=
 =?utf-8?B?L0hmNmdVeW93aXU3eXU5RC9PZ3dxMDFiZmhaN2ttdmVLYWU1U0oxS1V3SGEx?=
 =?utf-8?B?dlpIc0pSY0lHRU0yQjZWQ21HWnU4VzFMdjlXZmdnU0FzZDJpa0owK1pYRENo?=
 =?utf-8?B?YWVNUTJvUGhVMlluYkZCbTF6OVFlNGcrUDFyMUVPanJWbC80Qlh1K2tGVWV3?=
 =?utf-8?B?TVVremx5a21CNUVxOGxMd1hUTmNyb29SUUE1TW5ocDQvMGZqdk9INWdRRnJW?=
 =?utf-8?B?M1lWUXNCVWhMemRVbzNTZFNJcFdBa3MvNUdYSmIzZ3Q1MXEzQlFndDRKRHNC?=
 =?utf-8?B?UWc9PQ==?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 47ec1c74-08d6-4a2a-0a56-08dadd50939d
X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB5009.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Dec 2022 21:25:37.6728
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: PmIzWHe1g8klWZnw/xEtef0Q5xkObCXuWeGy0vX3WA6URpl+lBibHVSABwLCaVhP0zUEdutw9d3xCc1RJVu6vCLQWcFi9d2bcv84CEJJxe0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR10MB6194
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2022-12-13_03,2022-12-13_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 mlxscore=0
 phishscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000
 definitions=main-2212130186
X-Proofpoint-ORIG-GUID: sNlQXmlEEKrlm4HgpJ-YgBR8i48z7t0e
X-Proofpoint-GUID: sNlQXmlEEKrlm4HgpJ-YgBR8i48z7t0e
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 12/12/22 5:09 PM, Krister Johansen wrote:
> On Mon, Dec 12, 2022 at 01:48:24PM -0500, Boris Ostrovsky wrote:
>> On 12/12/22 11:05 AM, Krister Johansen wrote:
>>> diff --git a/arch/x86/include/asm/xen/cpuid.h b/arch/x86/include/asm/xen/cpuid.h
>>> index 6daa9b0c8d11..d9d7432481e9 100644
>>> --- a/arch/x86/include/asm/xen/cpuid.h
>>> +++ b/arch/x86/include/asm/xen/cpuid.h
>>> @@ -88,6 +88,12 @@
>>>     *             EDX: shift amount for tsc->ns conversion
>>>     * Sub-leaf 2: EAX: host tsc frequency in kHz
>>>     */
>>> +#define XEN_CPUID_TSC_EMULATED       (1u << 0)
>>> +#define XEN_CPUID_HOST_TSC_RELIABLE  (1u << 1)
>>> +#define XEN_CPUID_RDTSCP_INSTR_AVAIL (1u << 2)
>>> +#define XEN_CPUID_TSC_MODE_DEFAULT   (0)
>>> +#define XEN_CPUID_TSC_MODE_EMULATE   (1u)
>>> +#define XEN_CPUID_TSC_MODE_NOEMULATE (2u)
>> This file is a copy of Xen public interface so this change should go to Xen first.
> Ok, should I split this into a separate patch on the linux side too?


Yes. Once the Xen patch has been accepted you will either submit the same patch for Linux or sync Linux file with Xen (if there are more differences).


>
>>> +static int __init xen_tsc_safe_clocksource(void)
>>> +{
>>> +	u32 eax, ebx, ecx, edx;
>>> +
>>> +	if (!(xen_hvm_domain() || xen_pvh_domain()))
>>> +		return 0;
>>> +
>>> +	if (!(boot_cpu_has(X86_FEATURE_CONSTANT_TSC)))
>>> +		return 0;
>>> +
>>> +	if (!(boot_cpu_has(X86_FEATURE_NONSTOP_TSC)))
>>> +		return 0;
>>> +
>>> +	if (check_tsc_unstable())
>>> +		return 0;
>>> +
>>> +	cpuid(xen_cpuid_base() + 3, &eax, &ebx, &ecx, &edx);
>>> +
>>> +	if (eax & XEN_CPUID_TSC_EMULATED)
>>> +		return 0;
>>> +
>>> +	if (ebx != XEN_CPUID_TSC_MODE_NOEMULATE)
>>> +		return 0;
>> Why is the last test needed?
> I was under the impression that if the mode was 0 (default) it would be
> possible for the tsc to become emulated in the future, perhaps after a
> migration.  The presence of the tsc_mode noemulate meant that we could
> count on the falseneess of the XEN_CPUID_TSC_EMULATED check remaining
> constant.


This will filter out most modern processors with TSC scaling support where in default mode we don't intercept RDTCS after migration. But I don't think we have proper interface to determine this so we don't have much choice but to indeed make this check.


-boris