Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 26 Jun 2023 11:38:34 +0800
From:   Yan Zhao <yan.y.zhao@intel.com>
To:     Yuan Yao <yuan.yao@linux.intel.com>
CC:     Xiaoyao Li <xiaoyao.li@intel.com>, Chao Gao <chao.gao@intel.com>,
        <kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <pbonzini@redhat.com>, <seanjc@google.com>, <kai.huang@intel.com>,
        <robert.hoo.linux@gmail.com>
Subject: Re: [PATCH v3 07/11] KVM: VMX: drop IPAT in memtype when CD=1 for
 KVM_X86_QUIRK_CD_NW_CLEARED
Message-ID: <ZJkIOlWoBGI/AJHG@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <20230616023101.7019-1-yan.y.zhao@intel.com>
 <20230616023815.7439-1-yan.y.zhao@intel.com>
 <ZJESMaG5Thb5LWtt@chao-email>
 <ZJEQNTvfwOSsSzrf@yzhao56-desk.sh.intel.com>
 <45be3429-f352-920a-d1af-7f3a07930f5d@intel.com>
 <ZJjW9Kstbfa+LmAV@yzhao56-desk.sh.intel.com>
 <20230626034027.g75sliwf46impob3@yy-desk-7060>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230626034027.g75sliwf46impob3@yy-desk-7060>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?+eHdMHgChnxLHVlojwHqla1EEqi/SLwt+jQOfbO3aOuAg8+iRRGUfE2/WtM1?=
 =?us-ascii?Q?dRahFCzshqAoHTpoWveVxYGmzYAKTU21gc3UCbdR0vsjbz/XhVCEQ2oSWp+T?=
 =?us-ascii?Q?HzOALBoMnepNhSYyxi/5sh9dYE9FiqKip6/hdQ18YafdU/0VnSTMSOQu7vlv?=
 =?us-ascii?Q?j0wBVQ8dW7DQdH91Ewykkhn54ge4GvpjnNFme2Vyb1v3dBoVbv8Ereyk0s3K?=
 =?us-ascii?Q?Dmbj68p7lc5LO/AK+7sFUqCBzHQTXnlSwFsy/7Hh0MikSfciR1bi8/YSQIQ/?=
 =?us-ascii?Q?oUE7a9HOLaNKi/H4JPqkT8DclvA/d1J5f++jUerms/6Er4octtoLZPn3+UwP?=
 =?us-ascii?Q?DH55yG6l74yRa2k1SzSD8p1XBNCGpx5G82Og6wi8xejHtYEGWz/oDAdCbn4n?=
 =?us-ascii?Q?UREbzbzMXaVHcHl8jvo82SCg60QqpEXyNQ8ya9Jt2KI2eoss9IkoK+VxIbmD?=
 =?us-ascii?Q?30HajKH2MNTude9ciOJkBbhBlg1KBiFTZyCe5sGG/1sleAozpmCgmJaoi8XF?=
 =?us-ascii?Q?l77SZlL/nePhdJIYSPJwDyTbbI3wUGj80KWpzykgn1yu4WO6bK+rkHXo5L1A?=
 =?us-ascii?Q?Qbsu/DwaN8qXiZ0XUdIcL2NR5trIAce3npJF4f7ngd0q2PfgoqQwwvEduGiC?=
 =?us-ascii?Q?teOXpPK0WHayW44RYWepZFXp7DRWAsQKEZKs2E06sL/m0U0mCOZZrrywAeea?=
 =?us-ascii?Q?d/uPJn0okh+ybOHPbKj3xGgc9PWfxXgi49yPZHkWk/3MHzB/wmSFoYYodulp?=
 =?us-ascii?Q?2eINJWxOK0+wiyuab7yV8F6CcEtK7b75Ci0NFvcnytJTvHzn1bYm9Fiz9dtL?=
 =?us-ascii?Q?lRpKKr/OjY1pFoQL3qnuc3oFRmZNMPhQ3n7nhuKn+7vFTN8EhJYJC5l5lbtS?=
 =?us-ascii?Q?tH/XtD30xsXcsrsFAueKbonFD+yFfq4hiWwBzDb8dBm9J0x+Yj8gn/CXUYXE?=
 =?us-ascii?Q?GrSWgKCp7jHWiVQbjnCb9+Boeq7RqydmzowoHf/V/v2CiRW7j7JrNL/YSFRm?=
 =?us-ascii?Q?GrPWwjRvvauXHivJ8eVcZoTSi27Ti5BzCKqMKvt3JIZDScU4r1GiIWOiziXn?=
 =?us-ascii?Q?jNRMIL6KF5UUKLmMD6c0RyuN0AS4fpz/Y5ojBd68O5xSt4nR5+qYK02wktMO?=
 =?us-ascii?Q?jlb0rycSqVSR6k7DMYWPPOKZsd4tv1wir+APJvklDLZoRMBlKPjksJqBqSq8?=
 =?us-ascii?Q?2wqKfntvUpGd5iGITQi4LzYQbrfSUYX0SelORYDojnQRlOSHs8gEgsaz4iKE?=
 =?us-ascii?Q?eWfA6EMTH/fVz8UTr0GTj7mluQi+164vmKCPwuoOx5q3w0Q1Pb0J85LHYynI?=
 =?us-ascii?Q?Ye/sX7w6PR6Bi/U0aG+MUCP3BFyLNW5keZWKcdyj+vp6cZQduBTzriQLDWyL?=
 =?us-ascii?Q?5GwIqvsPhEdi0JRQ/H8o6SE6PHiKp1gwTywkmBAhBel6OQEVwC9xtoUX6trk?=
 =?us-ascii?Q?sIgaljFXm9nwg52FdRctC5+uZNHq1vZ0K93ES92mDuCl0rVsMix/JkdmZ9s1?=
 =?us-ascii?Q?9KHJyEX+TGTdKmbpFzc+QNTbGQtvsB9pVQNqia0ppvdQzvsQvAovNB/DvxcC?=
 =?us-ascii?Q?znVy8duT0ULlZxbrTjaRXonke4cwKTEki2qA511m?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 0fe42517-4c07-477f-c07a-08db75fa5b0c
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2023 04:03:53.4655
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: j6n/9NrPLfbQUu8WbUwhNDNyls55c+L7DMzaXAvkwoPbU94mTQvVb20ehw+/Bya+e7fJ8CTWfAKrgRe9KUa/kA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR11MB6732
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Jun 26, 2023 at 11:40:28AM +0800, Yuan Yao wrote:
> On Mon, Jun 26, 2023 at 08:08:20AM +0800, Yan Zhao wrote:
> > On Sun, Jun 25, 2023 at 03:14:37PM +0800, Xiaoyao Li wrote:
> > > On 6/20/2023 10:34 AM, Yan Zhao wrote:
> > > > On Tue, Jun 20, 2023 at 10:42:57AM +0800, Chao Gao wrote:
> > > > > On Fri, Jun 16, 2023 at 10:38:15AM +0800, Yan Zhao wrote:
> > > > > > For KVM_X86_QUIRK_CD_NW_CLEARED, remove the ignore PAT bit in EPT memory
> > > > > > types when cache is disabled and non-coherent DMA are present.
> > > > > >
> > > > > > With the quirk KVM_X86_QUIRK_CD_NW_CLEARED, WB + IPAT are returned as the
> > > > > > EPT memory type when guest cache is disabled before this patch.
> > > > > > Removing the IPAT bit in this patch will allow effective memory type to
> > > > > > honor PAT values as well, which will make the effective memory type
> > > > > Given guest sets CR0.CD, what's the point of honoring (guest) PAT? e.g.,
> > > > > which guests can benefit from this change?
> > > > This patch is actually a preparation for later patch 10 to implement
> > > > fine-grained zap.
> > > > If when CR0.CD=1 the EPT type is WB + IPAT, and
> > > > when CR0.CD=0 + mtrr enabled, EPT type is WB or UC or ..., which are
> > > > without IPAT, then we have to always zap all EPT entries.
> > > >
> > > > Given removing the IPAT bit when CR0.CD=1 only makes the quirk
> > > > KVM_X86_QUIRK_CD_NW_CLEARED more strict (meaning it could be WC/UC... if
> > > > the guest PAT overwrites it), it's still acceptable.
> > >
> > > Per my understanding, the reason why KVM had KVM_X86_QUIRK_CD_NW_CLEARED is
> > > to ensure the memory type is WB to achieve better boot performance for old
> > > OVMF.
> > It works well for OVMF c9e5618f84b0cb54a9ac2d7604f7b7e7859b45a7,
> > which is Apr 14 2015.
> >
> >
> > > you need to justify the original purpose is not broken by this patch.
> >
> > Hmm, to dig into the history, the reason for this quirk is explained below:
> >
> > commit fb279950ba02e3210a16b11ecfa8871f3ee0ca49
> > Author: Xiao Guangrong <guangrong.xiao@intel.com>
> > Date:   Thu Jul 16 03:25:56 2015 +0800
> >
> >     KVM: vmx: obey KVM_QUIRK_CD_NW_CLEARED
> >
> >     OVMF depends on WB to boot fast, because it only clears caches after
> >     it has set up MTRRs---which is too late.
> >
> >     Let's do writeback if CR0.CD is set to make it happy, similar to what
> >     SVM is already doing.
> >
> >
> > which means WB is only a must for fast boot before OVMF has set up MTRRs.
> > At that period, PAT is default to WB.
> >
> > After OVMF setting up MTRR, according to the definition of no-fill cache
> > mode, "Strict memory ordering is not enforced unless the MTRRs are
> > disabled and/or all memory is referenced as uncached", it's valid to
> > honor PAT in no-fill cache mode.
> 
> Does it also mean that, the honor PAT in such no-fill cache mode should
> also happen for non-quirk case ? e.g. the effective memory type can be
> WC if EPT is UC + guest PAT is WC for CD=1.
No. Only the quirk KVM_X86_QUIRK_CD_NW_CLEARED indicates no-fill cache
mode (CD=1 and NW=0).
Without the quirk, UC + IPAT is desired.

> 
> > Besides, if the guest explicitly claim UC via PAT, why should KVM return
> > WB?
> > In other words, if it's still slow caused by a UC value in guest PAT,
> > it's desired to be fixed in guest instead of a workaround in KVM.
> 
> the quirk may not work after this patch if the guest PAT is
> stronger than WB for CD=1, we don't if any guest "works correctly" based
> on this quirk, I hope no. How about highlight this in commit message
At least for Seabios and OVMF, the PAT is WB by default.
Even after MTRRs enabled, if there are UC ranges, they are small in size
and are desired to be UC.
So, I think it's ok.

> explicitly ?
Will try to explain the background and possible influence.

Thanks

> 
> Also I agree that such issue should be fixed in guest not in KVM.
>