Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 8 Jul 2021 09:08:46 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     "Raj, Ashok" <ashok.raj@intel.com>
Cc:     Thomas Gleixner <tglx@linutronix.de>,
        "Dey, Megha" <megha.dey@intel.com>, linux-kernel@vger.kernel.org,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "Tian, Kevin" <kevin.tian@intel.com>,
        "Pan, Jacob jun" <jacob.jun.pan@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "Kumar, Sanjay K" <sanjay.k.kumar@intel.com>,
        "Van De Ven, Arjan" <arjan.van.de.ven@intel.com>,
        "Williams, Dan J" <dan.j.williams@intel.com>,
        "Shankar, Ravi V" <ravi.v.shankar@intel.com>
Subject: Re: Programming PASID in IMS entries
Message-ID: <20210708120846.GD4459@nvidia.com>
References: <bd509e3d-f59d-1200-44ce-93cf9132bd8c@intel.com>
 <87k0m2qzgz.ffs@nanos.tec.linutronix.de>
 <20210707221216.GA56594@otc-nc-03>
 <20210707235822.GB4459@nvidia.com>
 <20210708003335.GC56594@otc-nc-03>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20210708003335.GC56594@otc-nc-03>
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from mlx.ziepe.ca (47.55.113.94) by BL0PR1501CA0012.namprd15.prod.outlook.com (2603:10b6:207:17::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4308.20 via Frontend Transport; Thu, 8 Jul 2021 12:08:47 +0000
Received: from jgg by mlx with local (Exim 4.94)        (envelope-from <jgg@nvidia.com>)        id 1m1Spi-006FaI-Cl; Thu, 08 Jul 2021 09:08:46 -0300
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 8cb2f25b-f2e0-4fb2-6456-08d9420923bf
X-MS-TrafficTypeDiagnostic: BL1PR12MB5351:
X-Microsoft-Antispam-PRVS: <BL1PR12MB5351F3DAA1006C682D9559F1C2199@BL1PR12MB5351.namprd12.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:8882;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: 6FOmqo5Xq1LFCpiGqfNXmevKZPYbYPqjofN4Ku6UWG55vvzvOUiCYi5moEMLNdko6HJEDbmMDMgSLBxfbhs1XUbclXA/+VC0FLxsay2IeWD85bBeNpUZtHE0T7HPSOQ1CzzafKcsy7WfskiMG55I4R8lwF9k++tOGhaaoeyFMt1NvWjEUP3bYHNO1hCVG+rWPzkdrgE1jF39UVrFD9RTB/LY7Sf1/IVeYVljhHVCpD4pAB4vFlJR5tI1lIZsBVSWfcvp6Jw/ZocqS0Toc4izA5PNJ+IhrKjg0ILjJGjQnfDfPHT1WCNZmu4s4fGnUXtaASEoQYjVQAXZH7nPEUG0LYRSpHHd4czE5gYZ3+/8HhUZJolCR23QbMAkj3lJPeDO3bhvG4rHMMb/nZqAHU0i7vnJ0A+KBeOothmVj+YUEzpQNGbrX5YAifojxuhVF8Zp+BDj+WjZzEX+4QaR+WjPD2vDlseHCAdZ5Bnz46d+v1pyBoIJ8FK7xKhOlpzfMNFH9suUEQ8ObpQ/Eb/RwUcd7KKYU2rlv/2oN+EG5HyHSPeZTDlSOnj3T4Hfqn0gJD+s5McIdIYWBFbI54yljeCKhc5rtKVEtuWahKjv4iBVEI95qOU89UrJ+3pkVh/VgY3kpY4BWZdPTJtvFiD1Ifjy6Q==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BL0PR12MB5506.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(346002)(396003)(136003)(366004)(39860400002)(376002)(8936002)(9786002)(33656002)(7416002)(66556008)(66476007)(54906003)(8676002)(316002)(66946007)(83380400001)(9746002)(6916009)(36756003)(26005)(38100700002)(4326008)(478600001)(186003)(86362001)(2906002)(5660300002)(1076003)(426003)(2616005);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Wqfi8XBRyCNM+DscLZ0bFkAcdfqGnoOlHAZHxSx7N3uyEcnB6XOzXBi+bnG2?=
 =?us-ascii?Q?yWqT+AAqWjubrXi5frER9Fqm/l/I2JKcYU85+e8l0hlyOE88610fXcd0/0mR?=
 =?us-ascii?Q?25w7ZqyMiHod+yIDcsLQFl+TjvfNyK1BmHl4nFyRYI6cHffLalTwunjvobpM?=
 =?us-ascii?Q?lVoU8QRmiuEkeUMjl6wS4z1zbmssKnFmOjEutW3IiedbzGVWQ7znb8vhI54q?=
 =?us-ascii?Q?hwpM9H9C5q2tkOnVckmN1uAdW65eMh3uJ0Qpl+Z+iKaEkv/6jwmemSlP3f0U?=
 =?us-ascii?Q?VFtSaQYCVKV/wYHRn4TJe7NBBH5lnscRUYjN73baJg1yVXF+Bm/bXW411Mrl?=
 =?us-ascii?Q?yW/uKcW+hc5lJl8z7ABEB8w3R3JrgeG/cw2Nyk/1et5HOEMvXe53iF9Zp4lT?=
 =?us-ascii?Q?AlQUD0JcfFcqY7RxPXN9vpNRvftS2NijDx/f83pojv3TsLaJ1aW0PnHgkOME?=
 =?us-ascii?Q?VaMagVzPTPVnaY4sxpO9Xa/2eXHv1AMxFtM6yvIP10mCfgYvZGOwfpOJTykp?=
 =?us-ascii?Q?K8POtDwmNT7hvCA1k5v1uIIPbf1cgDrG/WgUzYtFM7WeSOKM24mQ7F2Oq3f3?=
 =?us-ascii?Q?tZLDCGg2yXUzugCAmBvy9k9ZcM1gAWeHpWLw4Z13oXbFn+QtQcf5wwPlS38D?=
 =?us-ascii?Q?xKNfpPUV2pDrTlyATpNB2wHUwRh4R0q0zuzdfsK6WGryb3f0m9y/+HXgfaE9?=
 =?us-ascii?Q?harAhkmV/yOoMP7yB2Xy9DHCF5tFWJk13a2QcKR8BQiGvReIQ1E7YLC1SY6Q?=
 =?us-ascii?Q?JiS+mdGNzL6hWI9hENnu/pVZUx6TbzRokCYEmMQaTAcm4VdeGH7xr7OyV/RO?=
 =?us-ascii?Q?85w3E19z9zUGCfEYSLDCkspzbLBln1KaIlWuRYoLpzwRAp4B6p6E+QQLlKC+?=
 =?us-ascii?Q?MRCZs1qoZAWksVzS3i/1uKd41EY0INwtUX+Qk75k5gQqllrNDK9usRM+gh7z?=
 =?us-ascii?Q?GJPrvc3zKpUKqwbnYW8kOrIdDQHiiO2ulNqYvaqyFnuEhkKVpg5ZXqVxG7J6?=
 =?us-ascii?Q?ZbCR9/msK2W7wNhzvF3czZZCpWNYX59Jz0zIIT2c0t/OwzIiZQpnVO3SE3Q+?=
 =?us-ascii?Q?r1+50c9Br26keOE5PJvaWoQ+nM+cx7svuT/hmXlb1Fp2iNuweuycWT6UVAjZ?=
 =?us-ascii?Q?iM8n9StcTAiPUVWvARyx/UHWMDC1B7RJB525DOU/52tYu5KXOCwprKQPgSos?=
 =?us-ascii?Q?ByjZBIT0GvYSyGNVEFRi7RYiguFNoJYWaFDmU97T/tjI8z8SJ2VnoYd6yzxz?=
 =?us-ascii?Q?7WugT7uZbxUdGGkxA99vd01r4JfJlG/LaaS2WRnwm/VcxsLlGXhLMUB+HV6V?=
 =?us-ascii?Q?i/M1tNVRt/dBWJ9BoZy/9/zi?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8cb2f25b-f2e0-4fb2-6456-08d9420923bf
X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB5506.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Jul 2021 12:08:47.4009
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YLJEPTVR+auwbDsjjuFUtgSOaLhUfO4VvJS4uhlAem6TuXVgigxwLJCLmFVj2TqI
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5351
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Jul 07, 2021 at 05:33:35PM -0700, Raj, Ashok wrote:
> On Wed, Jul 07, 2021 at 08:58:22PM -0300, Jason Gunthorpe wrote:
> > On Wed, Jul 07, 2021 at 03:12:16PM -0700, Raj, Ashok wrote:
> > > Hi Thomas
> > > 
> > > On Wed, Jul 07, 2021 at 10:50:52AM +0200, Thomas Gleixner wrote:
> > > > Megha,
> > > > 
> > > > On Wed, Jul 07 2021 at 09:49, Megha Dey wrote:
> > > > > Per your suggestions during the last meeting, we wanted to confirm the 
> > > > > sequence to program the PASID into the IMS entries:
> > > > >
> > > > > 1. Add a PASID member to struct msi_desc (Add as part of a union. Other 
> > > > > source-id's such as Jason's vm-id can be added to it)
> > > > 
> > > > Yes. Though we also discussed storing the default PASID in struct device
> > > > to begin with which is then copied to the msi_desc entries during
> > > > allocation.
> > > 
> > > Using default PASID in struct device will work for sub-devices until the
> > > guest needs to enable ENQCMD support. Since the guest kernel can ask for an
> > > interrupt by specifying something in the descriptor submitted via ENQCMD.
> > > Using the PASID in struct device won't be sufficient.
> > 
> > Could you could store a pasid table in the struct device and index it
> > by vector?
> 
> Possibly... what ever Thomas things is clean. The device specific driver
> would have this already. So providing some call to get this filled in vs
> storing that in struct device. Someone close at heart to the driver model
> is best to comment :-)
> 
> IMS core owns the format of the entries right now vs device specific driver. 
> I suppose your use case requiring a vm_id might have a different format. 
> So this is yet another one the core needs to learn and adapt?

All entry format stuff is device specific, it shouldn't be in "core"
code.

It is is the same reason that the IRQ chip driver for IDXD should have
IDXD in the name, it is not a generic "IMS core" thing.

The question mark is probably the locking model, but if IDXD
guarentees the pasid table doesn't change while the irq is active then
maybe it works out well enough.

Associating a void * with the irq is also possibly reasonable, I'm not
sure which path makes the most sense.

Jason