Date:   Thu, 2 Dec 2021 10:45:14 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc:     Thomas Gleixner <tglx@linutronix.de>,
        Logan Gunthorpe <logang@deltatee.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Marc Zygnier <maz@kernel.org>,
        Alex Williamson <alex.williamson@redhat.com>,
        Kevin Tian <kevin.tian@intel.com>,
        Megha Dey <megha.dey@intel.com>,
        Ashok Raj <ashok.raj@intel.com>, linux-pci@vger.kernel.org,
        Jon Mason <jdmason@kudzu.us>,
        Dave Jiang <dave.jiang@intel.com>,
        Allen Hubbe <allenbh@gmail.com>, linux-ntb@googlegroups.com,
        linux-s390@vger.kernel.org, Heiko Carstens <hca@linux.ibm.com>,
        Christian Borntraeger <borntraeger@de.ibm.com>, x86@kernel.org,
        Joerg Roedel <jroedel@suse.de>,
        iommu@lists.linux-foundation.org
Subject: Re: [patch 21/32] NTB/msi: Convert to msi_on_each_desc()
Message-ID: <20211202144514.GQ4670@nvidia.com>
References: <87o861banv.ffs@tglx>
 <20211201001748.GF4670@nvidia.com>
 <87mtlkaauo.ffs@tglx>
 <20211201130023.GH4670@nvidia.com>
 <87y2548byw.ffs@tglx>
 <20211201181406.GM4670@nvidia.com>
 <87mtlk84ae.ffs@tglx>
 <87r1av7u3d.ffs@tglx>
 <20211202135502.GP4670@nvidia.com>
 <YajW6veanK4GZUkv@kroah.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <YajW6veanK4GZUkv@kroah.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?DXnrJ1GJqiu0t8/khzdJQDoSDdzCtvRrFwbqY8rzmO7h/oAole7zRZfb7FpZ?=
 =?us-ascii?Q?g7nKpgNzWeoqTjJjOtfgC0uf7vpk+hBbfVhPvzOnkbDCV+6OOH/x5KyCsQdY?=
 =?us-ascii?Q?2qb495f6QwxtiAKqSn560iwBERfqkNE0wctYflvA01DmxABLUShuPrI43BBN?=
 =?us-ascii?Q?IeyABwvs455MJO59iI2idlbt7irS80HuqzMUQM1y8oGq83c3C4fX7GSKhYV6?=
 =?us-ascii?Q?5aRUw/3nqlp6wl74WWZ2Apy/kKJ5qMe0MORoAbgjt02C3K170VABUfT8w5hi?=
 =?us-ascii?Q?BikoIhj63Ew1/a+s04KwTtWomBeHwMUaIWLsPQVjijq1oNF3sJZcdTj3EG5n?=
 =?us-ascii?Q?tNbgyP/XdQzV9LVwmRR3OK/8p1HfxkIMI0dSbfKvxRBcl2Sv5j9hvbDZ5KMj?=
 =?us-ascii?Q?TsXCgnA1vjeWLZG+SB1RGvnsByJV9sumCqMCx2F3xsOKoqiOCVm/Bvqod5tj?=
 =?us-ascii?Q?uAbCyLf+T4Cvc5ZWhCno1G2a49Q42YwmwWng8BOBduM9Xnd+bbZYzL6Ntydw?=
 =?us-ascii?Q?qY0FIh/l44gWaQuL+aSWqqUzhQr5QU8tOKqKEqd33mqopP6hPd6KHCPuaJm4?=
 =?us-ascii?Q?vIfOi4pHZSmBGCs49vc02oB9bdM+XuiUnuh3S8CLJGkmQelHvcxFjwxiZPuh?=
 =?us-ascii?Q?ahDPqPOXbUv/IuZ2th++NhGXfzloW+pvoAetIjPFgCPxIw53AsbwI0cVhxZd?=
 =?us-ascii?Q?pkgKgKqUrG8UdUxbydOFfuNl57iSz43F8AfMOgs4XIU/HMs0F+MDvUCPGYaa?=
 =?us-ascii?Q?mghjzw01jhn3HbJB0FwMLR5IWApuSloqyh2b7qWXQ4oVDpJT4ajkxOKlBOA4?=
 =?us-ascii?Q?iNCC3sLafJb23c3F3656NZ7Egb8XOjMeEdq0GNEWxodSQmjU/NVFCEAJv8py?=
 =?us-ascii?Q?P2cWeLskqLpSAwiK55rDkfqfUsuVXq4GHnxSVkRkChC9mMNwWj3gfZ7iTGwP?=
 =?us-ascii?Q?nkvT8TEkqpuVRO9t5I1bGh1hqgtvjCVWDukr+3zaRJM1LyTfRnyoSx/WliPY?=
 =?us-ascii?Q?Gf5OlkbqLGJoSCGveIHjOh7PmG21ZWwBiSFXTniW66EObS58BzldL/A74zVJ?=
 =?us-ascii?Q?/+oHtDOhGQhG4gl9wRrbId5+uH2WYqF/3l7htWJNNEa1gEiMzjrgJESc9e2o?=
 =?us-ascii?Q?A0AQPAQRJTU8MVkurW34gg0hjVnav5+QBnHLuqnJsRubRfw3uLQOx+YxFpa0?=
 =?us-ascii?Q?VwEjK1qrflNyPZiA4+C0pQRqa+BH+OLr9oa9OsDjU6Rrpy2YW1HfVo4ICN6Y?=
 =?us-ascii?Q?Cm0KS+9FQsQZpPIRrilQGx8WapW7Vt81OLT9L7cPhg9lowg4tftO4wmS5DU0?=
 =?us-ascii?Q?Crq/dFvdiSafou8AwP3eoYkMijLTOG6zawmSsXOjcaZ/5UIaeMCzIRQ6Mqnv?=
 =?us-ascii?Q?kfBO/JtmhgiIre8rkrlFdfLmeVETFHEyQ5acJOQv28sgqOZrO+rggNQNj3bA?=
 =?us-ascii?Q?Xs8hmM9BMnoLvhPzbI28cWHpazNsXq3asTRKdJ9KTv19DWR7PUXMjYMYj7Ga?=
 =?us-ascii?Q?ALQuEcKuD+ql1/NsRtBaG4Wgk99xDVc9ZliG4+n57mkOlxDZRIN6PgrHmJzK?=
 =?us-ascii?Q?LaShTJCSyJ3CtZT8Wek=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: cd0a7af6-db9d-424c-ff6c-08d9b5a25a4a
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Dec 2021 14:45:15.5860
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: hfPU1iuT4TXLtSDHxF28C9Z+zdeCqb0rqRpoRGObfoVoNsSvXIcgCWahbUwSO/3l
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5288
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Dec 02, 2021 at 03:23:38PM +0100, Greg Kroah-Hartman wrote:
> On Thu, Dec 02, 2021 at 09:55:02AM -0400, Jason Gunthorpe wrote:
> > Further, there is no reason why IMS should be reserved exclusively for
> > VFIO! Why shouldn't the cdev be able to use IMS vectors too? It is
> > just a feature of the PCI device like MSI. If the queue has a PASID it
> > can use IDXD's IMS.
> 
> No, sorry, but a cdev is not for anything resembling any real resource
> at all.

My point is that when the user asks the driver to allocate a queue
through a cdev ioctl it should be able to get the queue attached to an
IMS, today it can only get a queue attached to a MSI.

> It is ONLY for the /dev/NODE interface that controls the character
> device api to userspace.  The struct device involved in it is ONLY for
> that, nothing else.  Any attempt to add things to it will be gleefully
> rejected.

I agree with you!
 
> > If we really need a 2nd struct device to turn on IMS then, I'd suggest
> > picking the cdev, as it keeps IMS and its allocator inside the IDXD
> > PCIe driver and not in the VFIO world.
> 
> No!  Again, a cdev is to control the lifespan/lifecycle of the /dev/NODE
> only.  Anything other than that is not ok to do at all.

Said the same thing in a prior email - which is why I think the only
logical choice here is to make IMS work on the pci_device

FWIW I feel the same way about the VFIO mdev - its *ONLY* purpose is
to control the lifecycle and we are close to stripping away all the
other abuses using it for other things.

Jason