Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <52c6a8a6-3a0a-83ba-173d-0833e16b64fd@amd.com>
Date:   Fri, 18 Aug 2023 15:32:34 -0500
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.2.1
Subject: Re: [PATCH 4/8] KVM: gmem: protect kvm_mmu_invalidate_end()
Content-Language: en-US
To:     Sean Christopherson <seanjc@google.com>, isaku.yamahata@intel.com
Cc:     kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
        isaku.yamahata@gmail.com, Michael Roth <michael.roth@amd.com>,
        Paolo Bonzini <pbonzini@redhat.com>, erdemaktas@google.com,
        Sagi Shahar <sagis@google.com>,
        David Matlack <dmatlack@google.com>,
        Kai Huang <kai.huang@intel.com>,
        Zhi Wang <zhi.wang.linux@gmail.com>, chen.bo@intel.com,
        linux-coco@lists.linux.dev,
        Chao Peng <chao.p.peng@linux.intel.com>,
        Ackerley Tng <ackerleytng@google.com>,
        Vishal Annapurve <vannapurve@google.com>,
        Yuan Yao <yuan.yao@linux.intel.com>,
        Jarkko Sakkinen <jarkko@kernel.org>,
        Xu Yilun <yilun.xu@intel.com>,
        Quentin Perret <qperret@google.com>, wei.w.wang@intel.com,
        Fuad Tabba <tabba@google.com>
References: <cover.1692119201.git.isaku.yamahata@intel.com>
 <b37fb13a9aeb8683d5fdd5351cdc5034639eb2bb.1692119201.git.isaku.yamahata@intel.com>
 <ZN+whX3/lSBcZKUj@google.com>
From:   "Kalra, Ashish" <ashish.kalra@amd.com>
In-Reply-To: <ZN+whX3/lSBcZKUj@google.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OWNJL3E0OEN3cVkyQ285aVRycDQ2U0h0MlJQeU9tOXFvcytUaXpVeXR2VTMv?=
 =?utf-8?B?cFNIZjA5amFFeGJYbmtMZzl4cGxDTUdVZlp3L1BRWTdZUFphZXdFOVV1aHEz?=
 =?utf-8?B?Y1JtWGtBZ2hkbHo0ZGEvdzZyK1lYeERIK2xxWlVHLzF4N0dJQTdySW1WYSt4?=
 =?utf-8?B?SUZ6L2VZUWpTeHlDb2Zwb0s3T0ZXZlFZbThMWGtRbnp4Z2Fxd1BuUXJPdEF3?=
 =?utf-8?B?ckh4Nk96UU5QNE55dlVrdVFDajNrYWd6MVF2RXJGTVdaVXN0d293QlpxWDN1?=
 =?utf-8?B?SloyeVJCazFkbitIaHZOSlRqNC9qbWUyc1RYaUhpWTg4aVlPY0oySTM3cktV?=
 =?utf-8?B?RC9OdS9ha1JRMFpKVTZ4ZnExWVNpTFIxSkFPL2xVOHhvM0FzUnZrRWRDam1o?=
 =?utf-8?B?UXBYWFF2dm1TYWJlaWhBNUwzbC9KQkdHOTNiVXdiVmNVOE9tTHVablEzWmg4?=
 =?utf-8?B?NlQ2TWFxZW51VEpJeWx0NE9ZV0pHNEdwd1ZjVUNFc1FIaE9LRFRlMGI2QmM0?=
 =?utf-8?B?dFdZMDVPR2VFeEFyVUo3MkdTK0J0WHVqYjRwK0V6ekNEVXltZVVFZ1VvRWZs?=
 =?utf-8?B?KzhQVmN0K2VjcTBGTDZWL3IzdzFXRm5vTHBybzZqMVRkT1VBK3RBMXU3MUFk?=
 =?utf-8?B?R2pkWC9kM2RVRVFoYkd1MkxqdTNHbVQ3UWdPM3JqclB0T2RZa2lIZWgvaURx?=
 =?utf-8?B?blFYY2lDYm5sbGQwbitkZzZLS3dLTi90QWphNDRtbVAraVo3SXI2eGdIbE5U?=
 =?utf-8?B?eDAyZGdnWVord24wRDR0WVBISHZPWHpPSE82SUZUdXBYV2hDUnBhc3lTVFFh?=
 =?utf-8?B?SllRWENTL1k4SnNndjh4WGdMVi9TVWxDbCtkbEFDWW8vU1ZFbnFNL1NlWG4z?=
 =?utf-8?B?VVlGWUtSaFZyb0xxbStVYVMzcHU5MmYxSmwvY0YyUVhnMG9LZHhTUXZZaDIx?=
 =?utf-8?B?T1Y5YnJVQVBMdUZkQTIweU1kc2FwOS9WUEN2RUowVVZZbHhjWldKaEhselo1?=
 =?utf-8?B?d29JYmlFQWdzcVI2bTJwKzFXUzVMUDVqNVVjcG9tSVJ1SE9mUlh4NTZ5UXhj?=
 =?utf-8?B?b1lNWTBybE9EVkw5Vm0rM01hNmJvQWR3SUl5TVh2c3poRFEvZ2dHRDkwakF3?=
 =?utf-8?B?NkVWVHNVVUtBQnJGb3J4bFRYSEJQS1RtaTQvSG5tZGhjYURPTmFIZDQzZEFQ?=
 =?utf-8?B?cFYxWDFlSW5BQXoxblVSWGp4SXYycDdITHlSWFYydU8vK25MNWR2WWVQNlZ4?=
 =?utf-8?B?aXMrenNFMUNJclpWUU5EQUZLYmpnaEtLc1Y1TTc0YkNDVEtBMFJPVzc5NEV6?=
 =?utf-8?B?dHg4RUt6a1dMRUJ6OVFFMGg5T051clpacTFTdVdybHNoRkg5cStXd2l5WkRx?=
 =?utf-8?B?K29oVVVlakNldnBZSm9nNnpoY0xMYytENEVJTE02VEp1djRocWhoMzMwVita?=
 =?utf-8?B?ZEphcjVnNnZNUUV1elFCd2tkWHlrcU1Md1pPWkdua0JUb0dTalgvdlIzZGEw?=
 =?utf-8?B?ZzlxaG1ZOUczT0owaXovdkFoSGpjWDlVakFMRnRLaXRTekgycFpNdWwzRzd3?=
 =?utf-8?B?ck1DM1VpcFMrazRZWEdEaUFkSWdnV3V5QlFkS1M5eDFxVXJJRkRZd21SeFpJ?=
 =?utf-8?B?Y1FMSTVyR05JSDNhcnFzZjJkZmlkQzdYQkljY29zRXhlUkpLd21Ub01Oc3VI?=
 =?utf-8?B?UU50bGQ1bDliYldmOG1WZG1pLzVnajRuYkprSVZ4VFJCZlhzOExwcW11YktN?=
 =?utf-8?B?N3lObjd3eWVZK3QvQjhrOThOVHg4QlkyWWx5TTRLeTIzRHZ2alV4MDlUbXRq?=
 =?utf-8?B?Rms0NXJLQzJiejFYM2tyNkE0Ym5md0dHT2FIZkZIMGJFMGFLMGxaRGxIVzUz?=
 =?utf-8?B?VHhxd0xxa2lkcUJsSkdMeGNtN3liaGZhOFgwRWh2VkVYYXlMUWlzUUtXMjl5?=
 =?utf-8?B?MDl5dVplWFNzRDhzRHdyeFVLemtXSS9lb1pURlVXZEhCTXEvMTlzMkxRYXFr?=
 =?utf-8?B?K0NrR09LeHBpRjJvTzE1aVRPa2pCS2liQzR2ZTg0OVFjd1FlNFhOWU4zWUx2?=
 =?utf-8?B?Y1dBZ1JtUCtjVkgzNm1PZDJRR01XaW5nQ21qZG5BemJyYmF2MzV6Rm4yYjhR?=
 =?utf-8?Q?VrZ0amPiFn5IGZ67LiXW0ge9O?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a434dcb1-5668-4530-702b-08dba02a42f6
X-MS-Exchange-CrossTenant-AuthSource: SN6PR12MB2767.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Aug 2023 20:32:37.6636
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Bujt/hoV4vbyqn4iH044SAi9X1JdQSeroQ8tmcT9yF8BEoE+i5R3eaR6NN3VDg/QchVbHQQgqB+WO9UY0ImV0g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB6464
X-Spam-Status: No, score=-4.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        NICE_REPLY_A,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,
        SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 8/18/2023 12:55 PM, Sean Christopherson wrote:
> On Tue, Aug 15, 2023, isaku.yamahata@intel.com wrote:
>> From: Isaku Yamahata <isaku.yamahata@intel.com>
>>
>> kvm_mmu_invalidate_end() updates struct kvm::mmu_invalidate_in_progress
>> and it's protected by kvm::mmu_lock.  call kvm_mmu_invalidate_end() before
>> unlocking it. Not after the unlock.
>>
>> Fixes: 8e9009ca6d14 ("KVM: Introduce per-page memory attributes")
> 
> This fixes is wrong.  It won't matter in the long run, but it makes my life that
> much harder.
> 
>> Signed-off-by: Isaku Yamahata <isaku.yamahata@intel.com>
>> ---
>>   virt/kvm/kvm_main.c | 15 ++++++++++++++-
>>   1 file changed, 14 insertions(+), 1 deletion(-)
>>
>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>> index 8bfeb615fc4d..49380cd62367 100644
>> --- a/virt/kvm/kvm_main.c
>> +++ b/virt/kvm/kvm_main.c
>> @@ -535,6 +535,7 @@ struct kvm_mmu_notifier_range {
>>   	} arg;
>>   	gfn_handler_t handler;
>>   	on_lock_fn_t on_lock;
>> +	on_unlock_fn_t before_unlock;
>>   	on_unlock_fn_t on_unlock;
> 
> Ugh, shame on my past me.  Having on_lock and on_unlock be asymmetrical with respect
> to the lock is nasty.
> 
> I would much rather we either (a) be explicit, e.g. before_(un)lock and after_(un)lock,
> or (b) have just on_(un)lock, make them symetrical, and handle the SEV mess a
> different way.
> 
> The SEV hook doesn't actually care about running immediately after unlock, it just
> wants to know if there was an overlapping memslot.  It can run after SRCU is dropped,
> because even if we make the behavior more precise (right now it blasts WBINVD),
> just having a reference to memslots isn't sufficient, the code needs to guarantee
> memslots are *stable*.  And that is already guaranteed by the notifier code, i.e.
> the SEV code could just reacquire SRCU.

On a separate note here, the SEV hook blasting WBINVD is still causing 
serious performance degradation issues with SNP triggered via 
AutoNUMA/numad/KSM, etc. With reference to previous discussions related 
to it, we have plans to replace WBINVD with CLFLUSHOPT.

Pasting your previous thoughts on the same:

For SNP guests, KVM should use CLFLUSHOPT and not WBINVD.
That will slow down the SNP guest itself, but it should eliminate the 
noisy neighbor problems.

In theory, KVM could do the same for SEV/SEV-ES guests, but that's 
subtly quite difficult, because in order to use CLFLUSHOPT, the kernel 
needs a valid VA=>PA mapping.
Because mmu_notifier_invalidate_range_start() calls aren't fully 
serialized, KVM would encounter situations where there is no valid 
mapping for the userspace VA.
KVM could ignore those, but IIRC when Mingwei and I last looked at this, 
we weren't super confident that KVM wouldn't miss edge cases.

Using KVM's SPTEs to get the PA isn't a great option, as that would 
require KVM to flush whenever a leaf SPTE were zapped, i.e. even when 
_KVM_ initiates the zap.

UPM is supposed to make this easier because the notifier should be able 
to provide the PFN(s) being unmapped and the use the direct map to 
flush.  I don't think the proposed series actually provides the PFN, but 
it should not be difficult to add.

Thanks,
Ashish