Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 26 Oct 2021 00:16:43 +1100
From:   David Gibson <david@gibson.dropbear.id.au>
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     "Tian, Kevin" <kevin.tian@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "hch@lst.de" <hch@lst.de>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "jean-philippe@linaro.org" <jean-philippe@linaro.org>,
        "parav@mellanox.com" <parav@mellanox.com>,
        "lkml@metux.net" <lkml@metux.net>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "lushenming@huawei.com" <lushenming@huawei.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "yi.l.liu@linux.intel.com" <yi.l.liu@linux.intel.com>,
        "Tian, Jun J" <jun.j.tian@intel.com>, "Wu, Hao" <hao.wu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "jacob.jun.pan@linux.intel.com" <jacob.jun.pan@linux.intel.com>,
        "kwankhede@nvidia.com" <kwankhede@nvidia.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "dwmw2@infradead.org" <dwmw2@infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>
Subject: Re: [RFC 13/20] iommu: Extend iommu_at[de]tach_device() for multiple
 devices group
Message-ID: <YXauO+YSR7ivz1QW@yekko>
References: <20210919063848.1476776-1-yi.l.liu@intel.com>
 <20210919063848.1476776-14-yi.l.liu@intel.com>
 <YWe+88sfCbxgMYPN@yekko>
 <BN9PR11MB54337A8E65C789D038D875C68CB89@BN9PR11MB5433.namprd11.prod.outlook.com>
 <YWzwmAQDB9Qwu2uQ@yekko>
 <20211018163238.GO2744544@nvidia.com>
 <YXY9UIKDlQpNDGax@yekko>
 <20211025121410.GQ2744544@nvidia.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
        protocol="application/pgp-signature"; boundary="O2NMk5xF7UYH/YH7"
Content-Disposition: inline
In-Reply-To: <20211025121410.GQ2744544@nvidia.com>
Precedence: bulk


--O2NMk5xF7UYH/YH7
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Mon, Oct 25, 2021 at 09:14:10AM -0300, Jason Gunthorpe wrote:
> On Mon, Oct 25, 2021 at 04:14:56PM +1100, David Gibson wrote:
> > On Mon, Oct 18, 2021 at 01:32:38PM -0300, Jason Gunthorpe wrote:
> > > On Mon, Oct 18, 2021 at 02:57:12PM +1100, David Gibson wrote:
> > >=20
> > > > The first user might read this.  Subsequent users are likely to just
> > > > copy paste examples from earlier things without fully understanding
> > > > them.  In general documenting restrictions somewhere is never as
> > > > effective as making those restrictions part of the interface signat=
ure
> > > > itself.
> > >=20
> > > I'd think this argument would hold more water if you could point to
> > > someplace in existing userspace that cares about the VFIO grouping.
> >=20
> > My whole point here is that the proposed semantics mean that we have
> > weird side effects even if the app doesn't think it cares about
> > groups.
> >=20
> > e.g. App's input is a bunch of PCI addresses for NICs.  It attaches
> > each one to a separate IOAS and bridges packets between them all.  As
> > far as the app is concerned, it doesn't care about groups, as you say.
> >=20
> > Except that it breaks if any two of the devices are in the same group.
> > Worse, it has a completely horrible failure mode: no syscall returns
>=20
> Huh? If an app requests an IOAS attach that is not possible then the
> attachment IOCTL will fail.
>=20
> The kernel must track groups and know that group A is on IOAS A and
> any further attach of a group A device must specify IOAS A or receive
> a failure.

Ok, I misunderstood the semantics that were suggested.

So, IIUC what you're suggested is that if group X is attached to IOAS
1, then attaching the group to IOAS 1 again should succeed (as a
no-op), but attaching to any other IOAS should fail?

That's certainly an improvement, but there's still some questions.

If you attach devices A and B (both in group X) to IOAS 1, then detach
device A, what happens?  Do you detach both devices?  Or do you have a
counter so you have to detach as many time as you attached?

> The kernel should never blindly acknowledge a failed attachment.
>=20
> Jason
>=20

--=20
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

--O2NMk5xF7UYH/YH7
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEdfRlhq5hpmzETofcbDjKyiDZs5IFAmF2rjkACgkQbDjKyiDZ
s5JwahAA391JynQuavWmhBOhDzb14OvOhpDcitGZ2dFWubVWtz4qL5lRWS/02Q0N
9dpFnqINn3fxXAVKM//9YP5KRFeAS5v5C7ScUL0GDjKLMvQ5cl/YR/t8J2i+q1q5
osWlUmUofwjTKMzxrV+fRaY8W8Ro2vdp9NV3xJsE4ipZVIxU4X2XgIJ8Zp37ED+R
WPftlJukAKQIJmdiqaEyNah+eG0Jl4MmqOsGLRXrD4KFiblKwXX2AlQgxb0hzo6P
eScpIA26XA/v3ch/Llh7mo4obgb0PAsqeGhKUQpfFb7p+APamwdtjGqe8vE1UW+E
jVkaZwKTJ6Y6Dc88FBseS0sFvJDCqzGri023feLbP16V8pwhyweegqx75b0rcL1/
piUbQnSt8ZRidBRKq87NusMkBpV221+uqLvranfuoeqgbt9U0vYIB8J5jW+fyrCp
qu5Ch+wAmJZHr1/WmCAPmHV8pmGKoGUVYwc5vStAKkBTru0qAh5ImN7kTlpd8OpS
ktThhH1jVYpejts5quqAtDi/swpjCIBb209clpsidn8nVPBvI0I3diVWnN7qsXI4
vnmi3K+Y8bjANYPaLBtUVX4motWGBXLbOfbPkAua5bmHIYjZMSSm9E81R+SHzsPY
FOecGoVdowCrUloQPRA0YHxXocyX3ZSYvBWX0a8pZMpfMOuec30=
=dI80
-----END PGP SIGNATURE-----

--O2NMk5xF7UYH/YH7--