Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Wang, Wei W" <wei.w.wang@intel.com>
To:     "Yamahata, Isaku" <isaku.yamahata@intel.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
CC:     "Yamahata, Isaku" <isaku.yamahata@intel.com>,
        "isaku.yamahata@gmail.com" <isaku.yamahata@gmail.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        "Aktas, Erdem" <erdemaktas@google.com>,
        "Christopherson,, Sean" <seanjc@google.com>,
        "Shahar, Sagi" <sagis@google.com>,
        David Matlack <dmatlack@google.com>
Subject: RE: [PATCH v10 052/108] KVM: x86/tdp_mmu: Ignore unsupported mmu
 operation on private GFNs
Thread-Topic: [PATCH v10 052/108] KVM: x86/tdp_mmu: Ignore unsupported mmu
 operation on private GFNs
Thread-Index: AQHY7CiJasKThqRaDEOPq0Yx+uNQTK5exgOw
Date:   Mon, 5 Dec 2022 14:23:35 +0000
Message-ID: <DS0PR11MB6373E6562F6F0CCC92CF8A22DC189@DS0PR11MB6373.namprd11.prod.outlook.com>
References: <cover.1667110240.git.isaku.yamahata@intel.com>
 <32e2f5f567e1af3858e2896d705b66f90a908ff0.1667110240.git.isaku.yamahata@intel.com>
In-Reply-To: <32e2f5f567e1af3858e2896d705b66f90a908ff0.1667110240.git.isaku.yamahata@intel.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?o8rvTcC2SA8ERJTUlGs7/WdWcLJ33zqPmogYE+ONaa1jgVYGtQNo9azxrlV3?=
 =?us-ascii?Q?9qXbMCJdEVA4XlvKQ9Nz8IdxUzUMuMcgbe6Oy05laaAzxTSZCGfy+CtvcgU6?=
 =?us-ascii?Q?gEi5n8Efn6it/V1X6Vss6Lkxp/0GYxgx0/kukUmA1y17fDdbdjaak991140G?=
 =?us-ascii?Q?MN5NerIrdYzataCFQZVSRJgMPqs0L4YlIr1Szs/od4Fa1VmcBJnkik/T76sH?=
 =?us-ascii?Q?J0VB35aJf4XNxkg/n6qVOMmuubXwj4UOwe7LK1OYPeNphq5XwdfnT/SjTtV7?=
 =?us-ascii?Q?5O7VqQCm4KT+53q+XrBqnmi4zS5k6tOR+PRysqRCHUvrDIexxhdnS7xII5lc?=
 =?us-ascii?Q?hG8iu5D5ccRV+sZ0p/l3MDPdK+RUhZVpKhO5dRUvuKql28EnP7APMOYywgXH?=
 =?us-ascii?Q?zdcvjP0mlNZezvjjur4eHd55lcG/I/fgotTs2tf39URja2vbRj1yFCgb13Ry?=
 =?us-ascii?Q?JkyPi/GKGhtkNL/XPiKmeynM6/TSTtBgRjHIGuhrZbskmIAyJKSe0oFXQvt5?=
 =?us-ascii?Q?TtkzCK+DHMJ5hA7VSjgpBb7c91A/2903MD42RXF+QdIHbyA2Lv5aVMv7KAma?=
 =?us-ascii?Q?S8uOlOsMxq7tS0l7EPbJ9PyjcfuOB1t8cIOMmgJ60iyrLL7VALej9kyhb45F?=
 =?us-ascii?Q?ovnBwoVhwHBQC9kCFXIzCwKu3jVHsrQwU+8TtI7FJHjNLhFUKtLcZ6YcKF88?=
 =?us-ascii?Q?5TRLChJdeCcs6DhUgGEKxifW23QNlgk6DdBESTrW84O7cLPizMSwiUTxtk3G?=
 =?us-ascii?Q?9ZlUTL3K912WucbZ5Upe6OUaUIWaXzMouUNJHdFmoh2mWep8yWV0ou2zJ87X?=
 =?us-ascii?Q?HLHqNHk9AXaNAoRJG6ElPWHQ0knowD5IxhBOZ+/hLtn6iXERNCMIAUIA9I1J?=
 =?us-ascii?Q?I1j1m9Ucp1ajj44qfZg/HZjJ6uptznL8bj/1ulXDqimRWwPXfEe0ywCRdhIC?=
 =?us-ascii?Q?Mxcd6GUdpUAMKj91OvcmgeyVo07mP/wwlRBIYDK0lhCnbyYwWYeLtQVqiq9+?=
 =?us-ascii?Q?CceT8VFiktFMUqRMDuBWKCRh3NSuVH/xRiLG3RZ9U9E1KIXf0uLyiUAyXl/C?=
 =?us-ascii?Q?iGbyVJ0nxk0akJJbbeOBBKfrEVQdhNixDtBZapThRUjXTnUblnizkQjtM+tk?=
 =?us-ascii?Q?Qkc1USxT4Zqi8WTnhtT0ZtMUxX9uDLcg/v6Z+ZRwaxpTJYIoT6lcRrwOol1O?=
 =?us-ascii?Q?Br3qKEyZ8d8MWdKKCl9451OO+2LPv3El4yuOT6WbkPO7b8NjDxvW6Vz/S5wP?=
 =?us-ascii?Q?GaEGwQCpt8uJg8y/iceIq1hLVb9cl/qMUJ+LmGlp3fgui1lzExac7rODl8H8?=
 =?us-ascii?Q?Kb1TyI/zu7ttYvGdqUKM8fkM0pzQkz80KQxj2hhWeZ5A4TMJ/rX7rbARf4xe?=
 =?us-ascii?Q?+HVfoxMExIGAWZNnzO0s4PVBdeZlq9KzVi70Oc7SzST4OBURCxIqsfkaxByz?=
 =?us-ascii?Q?iPHo/7mXgn36I219wc8qS1BHBFID/XXWHFapom08KHf3xravdU60y9dhJrHV?=
 =?us-ascii?Q?zRoFKPyR+psZ5woMwH0ZhPNq8IKlZmph3d8GM0C8HxZF7/JtoLJtK5eSisoP?=
 =?us-ascii?Q?tEiB+ADKgzIA0z9TLxKjrVz05KpOq0YZ2Pk2/OLx?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB6373.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 42bae555-0074-485d-8597-08dad6cc4b70
X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Dec 2022 14:23:35.3371
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: koUY+/asPDSWMjj888zMdT4ePOzadVS7ldlv+ep5oToaWnLJbv5XCJiANOmB7jIodubntonaPCRnMIGKHt68lw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5904
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Sunday, October 30, 2022 2:23 PM, Yamahata, Isaku wrote:
> From: Isaku Yamahata <isaku.yamahata@intel.com>
>=20
> Some KVM MMU operations (dirty page logging, page migration, aging page)
> aren't supported for private GFNs (yet) with the first generation of TDX.
> Silently return on unsupported TDX KVM MMU operations.
>=20
> Signed-off-by: Isaku Yamahata <isaku.yamahata@intel.com>
> ---
>  arch/x86/kvm/mmu/mmu.c     |  3 ++
>  arch/x86/kvm/mmu/tdp_mmu.c | 73
> +++++++++++++++++++++++++++++++++++---
>  arch/x86/kvm/x86.c         |  3 ++
>  3 files changed, 74 insertions(+), 5 deletions(-)
>=20
> diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index
> 02e7b5cf3231..efc3b3f2dd12 100644
> --- a/arch/x86/kvm/mmu/mmu.c
> +++ b/arch/x86/kvm/mmu/mmu.c
> @@ -6588,6 +6588,9 @@ static bool kvm_mmu_zap_collapsible_spte(struct
> kvm *kvm,
>  	for_each_rmap_spte(rmap_head, &iter, sptep) {
>  		sp =3D sptep_to_sp(sptep);
>=20
> +		/* Private page dirty logging is not supported yet. */
> +		KVM_BUG_ON(is_private_sptep(sptep), kvm);
> +
>  		/*
>  		 * We cannot do huge page mapping for indirect shadow pages,
>  		 * which are found on the last rmap (level =3D 1) when not using diff =
--git
> a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index
> 0e053b96444a..4b207ce83ffe 100644
> --- a/arch/x86/kvm/mmu/tdp_mmu.c
> +++ b/arch/x86/kvm/mmu/tdp_mmu.c
> @@ -1469,7 +1469,8 @@ typedef bool (*tdp_handler_t)(struct kvm *kvm,
> struct tdp_iter *iter,
>=20
>  static __always_inline bool kvm_tdp_mmu_handle_gfn(struct kvm *kvm,
>  						   struct kvm_gfn_range *range,
> -						   tdp_handler_t handler)
> +						   tdp_handler_t handler,
> +						   bool only_shared)
>  {
>  	struct kvm_mmu_page *root;
>  	struct tdp_iter iter;
> @@ -1480,9 +1481,23 @@ static __always_inline bool
> kvm_tdp_mmu_handle_gfn(struct kvm *kvm,
>  	 * into this helper allow blocking; it'd be dead, wasteful code.
>  	 */
>  	for_each_tdp_mmu_root(kvm, root, range->slot->as_id) {
> +		gfn_t start;
> +		gfn_t end;
> +
> +		if (only_shared && is_private_sp(root))
> +			continue;
> +
>  		rcu_read_lock();
>=20
> -		tdp_root_for_each_leaf_pte(iter, root, range->start, range->end)
> +		/*
> +		 * For TDX shared mapping, set GFN shared bit to the range,
> +		 * so the handler() doesn't need to set it, to avoid duplicated
> +		 * code in multiple handler()s.
> +		 */
> +		start =3D kvm_gfn_for_root(kvm, root, range->start);
> +		end =3D kvm_gfn_for_root(kvm, root, range->end);
> +
> +		tdp_root_for_each_leaf_pte(iter, root, start, end)
>  			ret |=3D handler(kvm, &iter, range);
>=20
>  		rcu_read_unlock();
> @@ -1526,7 +1541,12 @@ static bool age_gfn_range(struct kvm *kvm, struct
> tdp_iter *iter,
>=20
>  bool kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range
> *range)  {
> -	return kvm_tdp_mmu_handle_gfn(kvm, range, age_gfn_range);
> +	/*
> +	 * First TDX generation doesn't support clearing A bit for private
> +	 * mapping, since there's no secure EPT API to support it.  However
> +	 * it's a legitimate request for TDX guest.
> +	 */
> +	return kvm_tdp_mmu_handle_gfn(kvm, range, age_gfn_range, true);
>  }
>=20
>  static bool test_age_gfn(struct kvm *kvm, struct tdp_iter *iter, @@ -153=
7,7
> +1557,8 @@ static bool test_age_gfn(struct kvm *kvm, struct tdp_iter *ite=
r,
>=20
>  bool kvm_tdp_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range
> *range)  {
> -	return kvm_tdp_mmu_handle_gfn(kvm, range, test_age_gfn);
> +	/* The first TDX generation doesn't support A bit. */
> +	return kvm_tdp_mmu_handle_gfn(kvm, range, test_age_gfn, true);
>  }
>=20
>  static bool set_spte_gfn(struct kvm *kvm, struct tdp_iter *iter, @@ -158=
2,8
> +1603,11 @@ bool kvm_tdp_mmu_set_spte_gfn(struct kvm *kvm, struct
> kvm_gfn_range *range)
>  	 * No need to handle the remote TLB flush under RCU protection, the
>  	 * target SPTE _must_ be a leaf SPTE, i.e. cannot result in freeing a
>  	 * shadow page.  See the WARN on pfn_changed in
> __handle_changed_spte().
> +	 *
> +	 * .change_pte() callback should not happen for private page, because
> +	 * for now TDX private pages are pinned during VM's life time.
>  	 */
> -	return kvm_tdp_mmu_handle_gfn(kvm, range, set_spte_gfn);
> +	return kvm_tdp_mmu_handle_gfn(kvm, range, set_spte_gfn, true);
>  }
>=20
>  /*
> @@ -1637,6 +1661,14 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm,
>=20
>  	lockdep_assert_held_read(&kvm->mmu_lock);
>=20
> +	/*
> +	 * Because first TDX generation doesn't support write protecting privat=
e
> +	 * mappings and kvm_arch_dirty_log_supported(kvm) =3D false, it's a bug
> +	 * to reach here for guest TD.
> +	 */
> +	if (WARN_ON_ONCE(!kvm_arch_dirty_log_supported(kvm)))
> +		return false;
> +
>  	for_each_valid_tdp_mmu_root_yield_safe(kvm, root, slot->as_id, true)
>  		spte_set |=3D wrprot_gfn_range(kvm, root, slot->base_gfn,
>  			     slot->base_gfn + slot->npages, min_level); @@ -1902,6
> +1934,14 @@ bool kvm_tdp_mmu_clear_dirty_slot(struct kvm *kvm,
>=20
>  	lockdep_assert_held_read(&kvm->mmu_lock);
>=20
> +	/*
> +	 * First TDX generation doesn't support clearing dirty bit,
> +	 * since there's no secure EPT API to support it.  It is a
> +	 * bug to reach here for TDX guest.
> +	 */
> +	if (WARN_ON_ONCE(!kvm_arch_dirty_log_supported(kvm)))
> +		return false;
> +

It might not be a good choice to intercept everywhere in kvm_mmu just as td=
x
doesn't support it. I'm thinking maybe we could do the check in tdx.c, whic=
h is
much simpler. For example:

@@ -2592,6 +2605,12 @@ static void tdx_handle_changed_private_spte(struct k=
vm *kvm,
        lockdep_assert_held(&kvm->mmu_lock);

        if (change->new.is_present) {
+               /* Only flags change. This isn't supported currently. */
+               KVM_BUG_ON(change->old.is_present, kvm);

Then we can have kvm_arch_dirty_log_supported completely removed.