Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32;
Message-ID: <b56a1eb7-4e31-4806-9f5e-31efe7212e04@amd.com>
Date:   Mon, 6 Nov 2023 17:23:44 +0530
User-Agent: Mozilla Thunderbird
Reply-To: nikunj@amd.com
Subject: Re: [PATCH v5 13/14] x86/tsc: Mark Secure TSC as reliable clocksource
Content-Language: en-US
To:     "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
        seanjc@google.com, pbonzini@redhat.com
Cc:     Dave Hansen <dave.hansen@intel.com>, linux-kernel@vger.kernel.org,
        thomas.lendacky@amd.com, x86@kernel.org, kvm@vger.kernel.org,
        bp@alien8.de, mingo@redhat.com, tglx@linutronix.de,
        dave.hansen@linux.intel.com, dionnaglaze@google.com,
        pgonda@google.com
References: <20231030063652.68675-1-nikunj@amd.com>
 <20231030063652.68675-14-nikunj@amd.com>
 <57d63309-51cd-4138-889d-43fbdf5ec790@intel.com>
 <ae267e31-5722-4784-9146-28bb13ca7cf5@amd.com>
 <20231102103306.v7ydmrobd5ibs4yn@box.shutemov.name>
 <5d8040b2-c761-4cea-a2ec-39319603e94a@amd.com>
 <cf92b26e-d940-4dc8-a339-56903952cee2@amd.com>
 <20231102123851.jsdolkfz7sd3jys7@box>
From:   "Nikunj A. Dadhania" <nikunj@amd.com>
In-Reply-To: <20231102123851.jsdolkfz7sd3jys7@box>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MVlvdk9WaWlUcXg0SUpxNWhTUU5vKytTaVl3UE5sRlRyQmhxdlpyU1FoMnpm?=
 =?utf-8?B?TGZrK3FlZit5anVqQXB3b3dUVUtmNlRFZVBkbC95MS9sNFR1ZWhiM3RXSjc1?=
 =?utf-8?B?R0tWZFdJcy9wSXplejJoY1JPWlVCTmJNenVBeXFqWjEzeXphVVpBSjBqVFdW?=
 =?utf-8?B?ZnhHUHpvK3E4aFRXNklnN1d2WVowVkdJMkZVazNMSkNmYWdSajk5a1RkRGVu?=
 =?utf-8?B?ZkJTWkUyTWVWSXEvQllhQ3dkSXROTHlHdTZsNWtjK1NNZXB3SnRqU3p5ZVlR?=
 =?utf-8?B?emRSL1E2K05ZdWdhNzc0RlM3S1E0V0ZvakY5Tzc0Y2pUQU1EaGxwa2NnUytL?=
 =?utf-8?B?QXdpQU9UNmZFQ004Z0UycThXOW1sT0NXOWMyNkR2OFhBNTVUZWcvS2NzZ3BU?=
 =?utf-8?B?cUtrQ1hvdFF3ZnN2Vk95dGs5SlVsSi8zbDZ3ZU5TL1NYREhSNitqVHJhZ1Vu?=
 =?utf-8?B?VVRGWStzYnUvUlVDZkFlMXJoeng0aWE2VEhxSnFLY1I2MVRmNVAxcmh1Qjlx?=
 =?utf-8?B?Y0xpOEhIZW1naXl5aG82bDZYTmVtS3g2OU14b0lDckMwQnh3WExDWHJSYkNX?=
 =?utf-8?B?dlU2anFVeHA0YkNHYk45SER3WGo3dElsN3ZLcThmZTU4THJIQjBTY1lFUXYx?=
 =?utf-8?B?dHhPbU1jK0lOTVFlaG13anF4MzNFdU5SWElpcTNnSkw4dE9FSDNRdzVtNkFQ?=
 =?utf-8?B?MWUvYjU1RktyNjB2c0k1RGJ6Rkk2WXNHUUEwNnhiZEdyTmRSSlI3WkdrVExx?=
 =?utf-8?B?OU9OQkhDRGU4eHlIRUxETXBGNXdNNHUxSXlsbkp0M3NKT1RnS1FET3hzeXM3?=
 =?utf-8?B?OWs5MFlDbm1zbC9BYmNyWWtzWmtmeStqTVhIb3V0azAvMGRLenY0T3ZtMFRI?=
 =?utf-8?B?ZzE0ZUFFRndUWFdqUUtJV3VLM0E3b0V4SU9ZU1pRbFJ1NXlTYWtSRzdlVDRv?=
 =?utf-8?B?aFR4SHZqUHZjSENvUlFYRU5FZzdhUTF6T3R0dG5YRlJ4eXpRSnlJNCtzeVB5?=
 =?utf-8?B?K2ZwS3cxSXdRUzYzOXc5MXB3K2dPajBoZkRnS21PNzdzWVpna0RFYzZuc3lo?=
 =?utf-8?B?OVdhK2E1cWRpYU1JNUdDRWY5OTEvdXYxNnVFUktKR2lZcndJUWRCUElGTHZ0?=
 =?utf-8?B?VGFCWHdEVnNVM3FCbmMvdlVURzhWTGFMLy9TSVFCbFhvWmYzYSthdmhEQndR?=
 =?utf-8?B?cW9VQkh3VHVjeDMyQkNEUDdSMnhia0F0Wllkb25oYVJENEFCbHR1NHFKOGsw?=
 =?utf-8?B?WEtkTGYxcDUvMmlScVFuUnk2SVQzSXJxdkFkdXRzaTBlM3ZhbWVNQkxTL0hi?=
 =?utf-8?B?M29icWowd2lScXFDOG55NUhDVHdzdkpha2pDd05nT094MlR6RGV6ZnEvaHRH?=
 =?utf-8?B?cU14M3JXbVpBQjVCd1ZPZTF2cTI5L3FlMnlZUVo4OEhRdjU4THI1U2V6RUtC?=
 =?utf-8?B?Z016N3pYTVdhYW0xZVdYaG1lY1BiNjU2bWtTQkJGRGJ3Z3hXV3NGZjdEenlI?=
 =?utf-8?B?OEFhR1p4WVRUcUlXcGIxOVVoTTlMT2tnZ0VpdlU3L1lrQVE2M2h5K3NRdmVD?=
 =?utf-8?B?aFZWbVhrUWdhNG5MbERXeFZYVHZVazRLOHFVRmUzZTduTGhiMWp3RHdOZGQ2?=
 =?utf-8?B?aXd5ZVBEb3Z5K0hacXFCempkbWdXSVZnQkRFeGpaUXZ2MksxZ21RdW53YnVU?=
 =?utf-8?B?ZDFza0c1d0JlZ3J4OXZJajlpSTVDVmNDYTlKcWNNV1NkWFVkWklXSGxKZmQz?=
 =?utf-8?B?TG03OE5iZk56UUpGenZ4L0xaMDR0bklJZzBjQzJMYlY1MkYrbVUyaUhPc2tx?=
 =?utf-8?B?Y3hUdmUzUFZMOGZDYyszU2UxWUNHYmhmMlVERjZYaHl4QWZKbzdCMFpPTUJ0?=
 =?utf-8?B?dCs2RStGYWpKMVdPSVBoZzFVNTY4SGhtNDE4UzhFMWtTeEJhRTNQMjh1K1VV?=
 =?utf-8?B?b2QrV3Y5dXVOOGh6SFVaaHB4U0JVNkl6SXdKQVllb29rZE9NRmpCc01KWUVJ?=
 =?utf-8?B?NWdsTFAzd0twY0ZjTGE3T3BCRDZ1UFBDcWR3MHc0a2VnSVVzSDcxaXExWlZX?=
 =?utf-8?B?OFlhWEZMS2xYVEpvVmk4enZtb0VZU3pqNnBQQ1lUVTFqanhnUmlzQjR1Rytq?=
 =?utf-8?Q?x7tFvrSnoyLBYoJ9igaOmDk89?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d576e8f2-9b19-4d59-3da8-08dbdebf0ccf
X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB6309.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Nov 2023 11:53:54.1533
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: S3Wu/f6v+nlA6qL27o+ow8oxt6C/otuPAp0TY/QTdEyd7HmjECV0l4lYX03b1dcRyWg6W8WouAsVAZc2gaP9rw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5283
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 06 Nov 2023 03:54:19 -0800 (PST)

On 11/2/2023 6:08 PM, Kirill A. Shutemov wrote:
> On Thu, Nov 02, 2023 at 05:46:26PM +0530, Nikunj A. Dadhania wrote:
>> On 11/2/2023 5:37 PM, Nikunj A. Dadhania wrote:
>>> On 11/2/2023 4:03 PM, Kirill A. Shutemov wrote:
>>>> On Thu, Nov 02, 2023 at 11:23:34AM +0530, Nikunj A. Dadhania wrote:
>>>>> On 10/30/2023 10:48 PM, Dave Hansen wrote:
>>>>>> On 10/29/23 23:36, Nikunj A Dadhania wrote:
>>>>>> ...
>>>>>>> diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c
>>>>>>> index 15f97c0abc9d..b0a8546d3703 100644
>>>>>>> --- a/arch/x86/kernel/tsc.c
>>>>>>> +++ b/arch/x86/kernel/tsc.c
>>>>>>> @@ -1241,7 +1241,7 @@ static void __init check_system_tsc_reliable(void)
>>>>>>>  			tsc_clocksource_reliable = 1;
>>>>>>>  	}
>>>>>>>  #endif
>>>>>>> -	if (boot_cpu_has(X86_FEATURE_TSC_RELIABLE))
>>>>>>> +	if (boot_cpu_has(X86_FEATURE_TSC_RELIABLE) || cc_platform_has(CC_ATTR_GUEST_SECURE_TSC))
>>>>>>>  		tsc_clocksource_reliable = 1;
>>>>>>
>>>>>> Why can't you just set X86_FEATURE_TSC_RELIABLE?
>>>>>
>>>>> Last time when I tried, I had removed my kvmclock changes and I had set
>>>>> the X86_FEATURE_TSC_RELIABLE similar to Kirill's patch[1], this did not
>>>>> select the SecureTSC.
>>>>>
>>>>> Let me try setting X86_FEATURE_TSC_RELIABLE and retaining my patch for
>>>>> skipping kvmclock.
>>>>
>>>> kvmclock lowers its rating if TSC is good enough:
>>>>
>>>> 	if (boot_cpu_has(X86_FEATURE_CONSTANT_TSC) &&
>>>> 	    boot_cpu_has(X86_FEATURE_NONSTOP_TSC) &&
>>>> 	    !check_tsc_unstable())
>>>> 		kvm_clock.rating = 299;
>>>>
>>>> Does your TSC meet the requirements?
>>>
>>> I have set TscInvariant (bit 8) in CPUID_8000_0007_edx and TSC is set as reliable.
>>>
>>> With this I see kvm_clock rating being lowered, but kvm-clock is still being picked as clock-source.
>>
>> Ah.. at later point TSC is picked up, is this expected ?
>>
>> [    2.564052] clocksource: Switched to clocksource kvm-clock
>> [    2.678136] clocksource: Switched to clocksource tsc
> 
> On bare metal I see switch from tsc-early to tsc. tsc-early rating is
> equal to kvmclock rating after it gets lowered.

For SNP guest with secure tsc enabled, kvm-clock and tsc-early both are at 299.
Initially, kvm-clock is selected as clocksource and when tsc with 300 rating is enqueued, 
clocksource then switches to tsc.

[    0.004231] clocksource: clocksource_enqueue: name kvm-clock rating 299
[...]
[    2.046319] clocksource: clocksource_enqueue: name tsc-early rating 299
[...]
[    3.399179] clocksource: Switched to clocksource kvm-clock
[...]
[    3.513652] clocksource: clocksource_enqueue: name tsc rating 300
[    3.517314] clocksource: Switched to clocksource tsc
 
> Maybe kvmclock rating has to be even lower after detecting sane TSC?

If I set kvmclock rating to 298, I do see exact behavior as you have seen on the bare-metal.

[    0.004520] clocksource: clocksource_enqueue: name kvm-clock rating 298
[...]
[    1.827422] clocksource: clocksource_enqueue: name tsc-early rating 299
[...]
[    3.485059] clocksource: Switched to clocksource tsc-early
[...]
[    3.623625] clocksource: clocksource_enqueue: name tsc rating 300
[    3.628954] clocksource: Switched to clocksource tsc

Regards
Nikunj