Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@ziepe.ca>, Marc Zyngier <maz@kernel.org>
CC:     xieming <xieming@kylinos.cn>,
        "sashal@kernel.org" <sashal@kernel.org>,
        "catalin.marinas@arm.com" <catalin.marinas@arm.com>,
        "linux@armlinux.org.uk" <linux@armlinux.org.uk>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "will@kernel.org" <will@kernel.org>,
        "kvmarm@lists.cs.columbia.edu" <kvmarm@lists.cs.columbia.edu>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>
Subject: RE: [PATCH v2] kvm/arm64: fixed passthrough gpu into vm on arm64
Thread-Topic: [PATCH v2] kvm/arm64: fixed passthrough gpu into vm on arm64
Thread-Index: AQHYRbFhb+/o5gfYPEODwyIzUkp2lKzbRWGAgAR9v4CAABc4gIAAJa0AgAKH4mA=
Date:   Wed, 6 Apr 2022 08:17:00 +0000
Message-ID: <BN9PR11MB52767F1DF0A3895C65F2A67B8CE79@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20220401090828.614167-1-xieming@kylinos.cn>
 <87tubcbvgk.wl-maz@kernel.org> <20220404132405.GQ64706@ziepe.ca>
 <87o81gc3dc.wl-maz@kernel.org> <20220404170202.GR64706@ziepe.ca>
In-Reply-To: <20220404170202.GR64706@ziepe.ca>
Accept-Language: en-US
Content-Language: en-US
dlp-version: 11.6.401.20
dlp-product: dlpe-windows
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?N7Yzp6SAUgp5bLAsllTD3LVsjdP9i1aoDNpCh11iPFARprg8qsVzRYPL4SNL?=
 =?us-ascii?Q?Y8KGwXjLipwKJH9qYrjFqCFyCmcwS0JNNdmWMeAA//dTj/RRKFHFv+GawJ9z?=
 =?us-ascii?Q?pDfJjz9EY4bDj9b2kdUlRWJDlREaDGRh08ySCtUi8YrMJ/k5E2MiRK7DxqSo?=
 =?us-ascii?Q?cBl2c+tejNHs2qH57IZKRE9q2NpLmcqQz2rxJzYhez8oAcv0dWVGyc17HMSv?=
 =?us-ascii?Q?+tKLcFkBTd1gH+DWOgKdRFyUUd3/8j3/BpYotuqayU5FByaJQbkUDUQ9uaGp?=
 =?us-ascii?Q?p4RfFzC78i0JTzMLNmxNl+rx4HM58cKjszne7iV5m4JndULd/5OdRVTNzFLn?=
 =?us-ascii?Q?HXf9wiPgvFyEu73zxBxam20DveHhGKFi0kPFLS1J+MecaV/dJJDAdhR0YqnQ?=
 =?us-ascii?Q?Re6dmBYiZbU3hs8dK9yX3Kqyrl+VVmm6GppaA3Cjk019HS+SOCY7LzCo/mxQ?=
 =?us-ascii?Q?oklo9oXpjuLtR3HEc9uoI1izWLXGTUmWOTf5rOAXM/W7FbzuBlUTUZb0kUR1?=
 =?us-ascii?Q?9ouGxnhHKDhZGCeM59KbmDwz4GvBFjKIQukbsfOLeiO8WFnln6lnd2rJIw7X?=
 =?us-ascii?Q?lCKNX1Rrhd0q3r+vhLYv6ZzzhmnhnfCaBgdWD6wbjk7mtUe1MPN4XpaO6D6A?=
 =?us-ascii?Q?LcNMVlkGlSy5t0owlP1//WaG/Rb1KEpJ1WkpGVRVk8Cu8J2R3r9Sxs1YGiz5?=
 =?us-ascii?Q?0pXa/c3PjyR5bAAbxmCpLZjrK1yBeiCP/6mTkuET3p5HJzzNg6/2xPoNcsBx?=
 =?us-ascii?Q?bsfHrfGR93I3Fzbg1Lw9zm1dlxkxdvzK0/1Zd97BxpfEUd+2piKWK5eVjyLn?=
 =?us-ascii?Q?YIAUh4hXuz27WvMUhr0XTL0hfhMqo1CHg7sQwlyyLvCM4LTxNbT9ktqiVGjq?=
 =?us-ascii?Q?EuWsZU3qcqPqLOiUlFLW905NWpc4e0ZEhfotgD0QenqZdBze+N+PqO4Po9aO?=
 =?us-ascii?Q?FZjD+gznYkfuCAv7LV4trdmEMuL3ye8qCCYLvRdmSXAjELbfW1h65c6X0Ao3?=
 =?us-ascii?Q?5AaUIfqUCln/yGnTTJX6Dr/k6u/KTo9k/Mk7x64qvnUmPoRRxkZfsqfBGaTv?=
 =?us-ascii?Q?Q0r66Zefd4oHSYysTv0wnUvROds3Vrk9NMiG8lANxHxyvfsrZ4tk7+tZ2Oc4?=
 =?us-ascii?Q?0ObtFpDYYWqth4NIiob7Fm0nQaLkQ/UzAlscgSSHgVj7Agd5EZsSepr4IqO8?=
 =?us-ascii?Q?02ZxrTUDaM8IG6uATr1s6kSXJ168qE9o5UNDirVD9fiK6foJj0SkCVERcv+m?=
 =?us-ascii?Q?+ttrrbMHZQ/3Xv75xk+KOXpSwbxN+16PajRBBIzuMjlO3ddXADTMYUipv8TB?=
 =?us-ascii?Q?mZrcZJlDW9pvvjzRfdunDVVkKjlDkG20hWq1gh7Sgq6dhCZw4JQQ1ZmVTSgP?=
 =?us-ascii?Q?SjakQ342jdIahUZLLGhdJly5TtdkQDR4ZdYpAbtIdIAQCnYiOR4Yu91MOYfz?=
 =?us-ascii?Q?o13Cq6g5OH/4JhX9sOsm+Ce1214u7V8CJH/xwohGp2gNGZbWBqCM59PV4rj+?=
 =?us-ascii?Q?wZnfXH9TJwBhaJkGpbsC3eQyH54aqqiXsjdwKutSs7zVTqdqhXAe1BFtLmB4?=
 =?us-ascii?Q?+WEafTNhekVmCE8rrl+HWuA/Cf6P4VXPhbAGcswqHOHpRyKMdd6BeTjF8jnJ?=
 =?us-ascii?Q?nNn1qz1tFSRTNWHMtg1N1LFVHqNxOi29k1cVktaR0tjdrVjGdLFB2RieB3c6?=
 =?us-ascii?Q?OEmqPebuR9J4d7ByRL59THUytUDJiyQK9K3H5fovAIWnmFlndQQ96+sxUS6f?=
 =?us-ascii?Q?JP94+RzjtA=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d71ff164-ae77-40e0-cdd1-08da17a5d312
X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Apr 2022 08:17:00.3833
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Hv6vXaRKJEQY3CItvkLsl+Qy0ciN5Xwnp4WKoFuwhZfwJdNQ9MYgJgwXobypYAr9ui/2uydG7JUyggBgLdm3JA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR11MB2593
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jason Gunthorpe <jgg@ziepe.ca>
> Sent: Tuesday, April 5, 2022 1:02 AM
>=20
> On Mon, Apr 04, 2022 at 03:47:11PM +0100, Marc Zyngier wrote:
> > > I'm guessing it will turn into a SBSA like thing where the ARM ARM is
> > > kind of vauge but a SOC has to implement Normal-NC in a certain way t=
o
> > > be functional for the server market.
> >
> > The main issue is that this equivalence isn't architected, so people
> > can build whatever they want. SBSA means nothing to KVM (or Linux at
> > large), and there is currently no way to describe which devices are
> > safe to map as Normal-NC vs Device.
>=20
> And people have, we know of some ARM SOC's that don't work fully with
> NORMAL_NC for this usage. That is already a problem for baremetal
> Linux, let alone KVM..
>=20
> That is why I likened it to SBSA - if you want to build a server SOC
> that works with existing server software, you have to support
> NORMAL_NC in this way. Even if it isn't architected.
>=20
> The KVM challenge, at least, is to support a CPU with working
> NORMAL_NC to create VM that emulates the same CPU with working
> NORMAL_NC.
>=20
> I didn't quite understand your other remarks though - is there a
> problem here? It seems like yes from the other thread you pointed at?
>=20
> I would think that KVM should mirror the process page table
> configuration into the KVM page table and make this into a userspace
> problem?
>=20
> That turns it into a VFIO problem to negotiate with userspace and set
> the proper pgprot. At least VFIO has a better chance than KVM to
> consult DT or something to learn about the specific device's
> properties.
>=20
> I don't know how VFIO/qemu/etc can make this all work automatically
> correctly 100% of the time. It seems to me it is the same problem as
> just basic baremetal "WC" is troubled on ARM in general today. Maybe
> some tables and a command line option in qemu is the best we can hope
> for.
>=20

Not knowing those ARM details and how they differ from x86. Just FYI
how it works on Intel platform.

If no assigned device the KVM page table (EPT) PTE is always set to
'forced WB' in a way overriding the guest cache attributes.

If having assigned device, EPT PTE is set to:

  1) UC for mmio regions. The effective memory type is UC or WC
     depending on guest attributes;

  2) forced WB for memory pages if the device cannot do noncoherent
     dma. Guest cache attributes are overridden (same as no device assigned=
);

  3) a type making the guest cache attributes effective for memory
     pages if noncoherent dma is possible.

All above logic is contained in vmx_get_mt_mask().

Thanks
Kevin