Received-SPF: pass (google.com: domain of linux-kernel+bounces-2968-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Mon, 18 Dec 2023 10:58:40 +0800
From: Yan Zhao <yan.y.zhao@intel.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
CC: "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"dri-devel@lists.freedesktop.org" <dri-devel@lists.freedesktop.org>,
	"pbonzini@redhat.com" <pbonzini@redhat.com>, "seanjc@google.com"
	<seanjc@google.com>, "olvaffe@gmail.com" <olvaffe@gmail.com>, "Lv, Zhiyuan"
	<zhiyuan.lv@intel.com>, "Wang, Zhenyu Z" <zhenyu.z.wang@intel.com>, "Ma,
 Yongwei" <yongwei.ma@intel.com>, "vkuznets@redhat.com" <vkuznets@redhat.com>,
	"wanpengli@tencent.com" <wanpengli@tencent.com>, "jmattson@google.com"
	<jmattson@google.com>, "joro@8bytes.org" <joro@8bytes.org>,
	"gurchetansingh@chromium.org" <gurchetansingh@chromium.org>,
	"kraxel@redhat.com" <kraxel@redhat.com>
Subject: Re: [RFC PATCH] KVM: Introduce KVM VIRTIO device
Message-ID: <ZX+1YNlgqYA/N06T@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <20231214103520.7198-1-yan.y.zhao@intel.com>
 <BN9PR11MB5276BE04CBB6D07039086D658C93A@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <BN9PR11MB5276BE04CBB6D07039086D658C93A@BN9PR11MB5276.namprd11.prod.outlook.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?JUTaRuxVIvpcqgggRRvNXq5Mh5cu0yvjGtbHBsSO8iZmpNs1I7HDFaktU3dB?=
 =?us-ascii?Q?KN828I+pQi8HQLoKyyplrE4v8bce74DC7Cjyf7fuoN0fhsZXwoGNiLFDe3O4?=
 =?us-ascii?Q?MSFOTGl7RtuZnRchAX1ipSijPfuRJB6l3qBfULEQMJwzextu9seMX/V2q8XN?=
 =?us-ascii?Q?zpfXBCdCMMciPff9Jbs4TflyetjvSaqLsV5CGh0tVJ8f+Se8RDHdy2F2zjVT?=
 =?us-ascii?Q?zOU7VX5RFLjZX+dg4jydRiK6HdZAYIrjreJOg61xLxjBnofzkAj5qMwXqhsY?=
 =?us-ascii?Q?7q/28eeNlewVbKIFzoOCw2AeE6qLXnLwYhunHQFckGqiQeLpIr461/XaSEJH?=
 =?us-ascii?Q?1TnsOUIHcnuyZCLDuTbiOgcToMYfiLGGg4aXMDEdGrdByhfvl4SR1n7lbTUY?=
 =?us-ascii?Q?VK4aa3CIxQ18YjaTaDO7mzgi7fpwJ0QCxiT824wnOIjrilPQo1smltV3l/9u?=
 =?us-ascii?Q?ueu+8FKsDDzaEDlFR4VFw5Js8D/+2knbWdmVdX+Gq7sT+GyryJS0XoRDRW+O?=
 =?us-ascii?Q?4rLvKspviaLyFCx+knEYaqTNJgxAmhJr12cXkLOurqW599XLXyICed0QDzAl?=
 =?us-ascii?Q?kNxPxXB4JcyXhnYc2FO6F15uoHYrn0qldKOubem5ny5e6R8+3LOpgf8FmhI5?=
 =?us-ascii?Q?MmeNOb53LEtG8+d+xrrMIOY0p51dPRvfFbIS3nkpQ7cgXiXwbEfH8016YcHY?=
 =?us-ascii?Q?EwcxZWmuoe+UlyJConVbTFWc1FdrqTdFR/S7kMdw9so0cwbvqr1t2F+Oi71L?=
 =?us-ascii?Q?hzuRivhwokSRUs1Qf0Bfgv3DUTmuALrCvMAi3+bgIZjg6WqPIvKn5bcMGg4W?=
 =?us-ascii?Q?uU0gXvssp7qBLK/EqP+VBmHRz6QQVmGhNuNKDKb+KCFbyhQ+pdIvc9UAyvS3?=
 =?us-ascii?Q?/+ztniqkf+ksSebg5WzW4iyD2xBCaXqRjdRYhjXVLBRvBWcYjyskeZGC3wGQ?=
 =?us-ascii?Q?WBSBI/JWSZCBfqSMDTMaoBg2/1fVl53AFH9mY/Dhb8kuxhQFZSVdEBFH/VYn?=
 =?us-ascii?Q?XOegqzD1kK4CZllK3ViZQ8aDAIfdZ354udZJxmCLwVAMKfhWFJkoF7XwooxE?=
 =?us-ascii?Q?VqiQXxGFd5Bb4/PW9LSPIFw5K/52IwaeHc+mD1dBmoyr68g7ePKqltF88oBm?=
 =?us-ascii?Q?iIvkvilV46A0C22V5kZpKWO1w8Z7nUzUuYh/EBSfOyWb6b3iDyfI6a5OlFYc?=
 =?us-ascii?Q?lE3WAhy8Cvhi/lhDw4YCZ3GCAXCDanzoyTxvD5E6OcS2sNbeXwYvEg8LA1gY?=
 =?us-ascii?Q?v24W3tMFkTdErwtKl1c/r4NYRfOV2Zljfh4ge/byuYkHPwr4IUGEspl1Lcou?=
 =?us-ascii?Q?YwVKbMkLpKhFT4vnbtyqR2z0i5qZubL5A00C/VlbRl7Pl2FlxJn2vcXW45gi?=
 =?us-ascii?Q?29uz6r9bzXWV091F8joX1zlNG7xUPkuuCDamAClVoFxxxT6gzFf/TMWb9bHd?=
 =?us-ascii?Q?wic8PEJcqKxN5s+4HQ0B267iWQFCVGXBE/AQ5RGHVudyNsjibtNQ373OzhcX?=
 =?us-ascii?Q?P53lg9H/cPC8vclVEuskTQx9A6WnbOviRS1hvK1sek+QHqNt0Agp24W8h7zl?=
 =?us-ascii?Q?4E3GH21I4Mfh696+dV8fNwWiK4kN5QFHDE0ehCeg?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a763b18a-32de-4556-b7c7-08dbff794fe9
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Dec 2023 03:27:50.6286
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: h7n9JAOGBepNPpsOLz7zMfVKsEy8YwnqMh2INhC9E8u/VahIHms11E54MkebJetoc7Q6SzSXrBFB/Pp5K+bl6A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR11MB4702
X-OriginatorOrg: intel.com

On Fri, Dec 15, 2023 at 02:23:48PM +0800, Tian, Kevin wrote:
> > From: Zhao, Yan Y <yan.y.zhao@intel.com>
> > Sent: Thursday, December 14, 2023 6:35 PM
> > 
> > - For host non-MMIO pages,
> >   * virtio guest frontend and host backend driver should be synced to use
> >     the same memory type to map a buffer. Otherwise, there will be
> >     potential problem for incorrect memory data. But this will only impact
> >     the buggy guest alone.
> >   * for live migration,
> >     as QEMU will read all guest memory during live migration, page aliasing
> >     could happen.
> >     Current thinking is to disable live migration if a virtio device has
> >     indicated its noncoherent state.
> >     As a follow-up, we can discuss other solutions. e.g.
> >     (a) switching back to coherent path before starting live migration.
> 
> both guest/host switching to coherent or host-only?
> 
> host-only certainly is problematic if guest is still using non-coherent.
Both.

> on the other hand I'm not sure whether the host/guest gfx stack is
> capable of switching between coherent and non-coherent path in-fly
> when the buffer is right being rendered.
> 
Yes. I'm also not sure about it. But it's an option though.

> >     (b) read/write of guest memory with clflush during live migration.
> 
> write is irrelevant as it's only done in the resume path where the
> guest is not running.
Given host write is with PAT WB and hardware is in no-snoop mode, is it
better to perform cache flush after host write?
(can do more investigation to check if it's necessary).

BTW, there's also post-copy live migration, in which case the guest is
running :)

> 
> > 
> > Implementation Consideration
> > ===
> > There is a previous series [1] from google to serve the same purpose to
> > let KVM be aware of virtio GPU's noncoherent DMA status. That series
> > requires a new memslot flag, and special memslots in user space.
> > 
> > We don't choose to use memslot flag to request honoring guest memory
> > type.
> 
> memslot flag has the potential to restrict the impact e.g. when using
> clflush-before-read in migration? Of course the implication is to
> honor guest type only for the selected slot in KVM instead of applying
> to the entire guest memory as in previous series (which selects this
> way because vmx_get_mt_mask() is in perf-critical path hence not
> good to check memslot flag?)
>
I think checking memslot flag in itself is all right.
But memslot flag does not contain the memory type that host is using for
the memslot.
On the other hand, virtio GPU is not the only source of non-coherent DMAs.
Memslot flag way is not applicable to pass-through GPUs, due to lacking of
coordination between guest and host.


> > Instead we hope to make the honoring request to be explicit (not tied to a
> > memslot flag). This is because once guest memory type is honored, not only
> > memory used by guest virtio device, but all guest memory is facing page
> > aliasing issue potentially. KVM needs a generic solution to take care of
> > page aliasing issue rather than counting on memory type of a special
> > memslot being aligned in host and guest.
> > (we can discuss what a generic solution to handle page aliasing issue will
> > look like in later follow-up series).
> > 
> > On the other hand, we choose to introduce a KVM virtio device rather than
> > just provide an ioctl to wrap kvm_arch_[un]register_noncoherent_dma()
> > directly, which is based on considerations that
> 
> I wonder it's over-engineered for the purpose.
> 
> why not just introducing a KVM_CAP and allowing the VMM to enable?
As we hope to increase non-coherent DMA count on hot-plug of a non-coherent
device and decrease non-coherent DMA count on hot-unplug of the non-coherent
device, a KVM_CAP looks requiring user to maintain a ref count before turning
on/off, which is less desired. Agree?

> KVM doesn't need to know the exact source of requiring it...
Maybe we can use the source info in a way like this:
1. indicate the source is not a passthrough device
2. record relationship between GPA and memory type.

Then, if KVM knows non-coherent DMAs do not contain any passthrough 
devices, it can force a GPA's memory type (by ignoring guest PAT) to the
one specified by host (in 2), so as to avoid cache flush operations before
live migration.

If there are passthrough devices involved later, we can zap the EPT and
rebuild memory type to honor guest PAT, resorting to cache flush before
live migration to maintain coherency.