Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 10 Feb 2022 19:45:08 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Matthew Rosato <mjrosato@linux.ibm.com>
Cc:     Niklas Schnelle <schnelle@linux.ibm.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        linux-s390@vger.kernel.org, cohuck@redhat.com,
        farman@linux.ibm.com, pmorel@linux.ibm.com,
        borntraeger@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com,
        gerald.schaefer@linux.ibm.com, agordeev@linux.ibm.com,
        frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com,
        vneethv@linux.ibm.com, oberpar@linux.ibm.com, freude@linux.ibm.com,
        thuth@redhat.com, pasic@linux.ibm.com, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3 24/30] vfio-pci/zdev: wire up group notifier
Message-ID: <20220210234508.GJ4160@nvidia.com>
References: <20220208104319.4861fb22.alex.williamson@redhat.com>
 <20220208185141.GH4160@nvidia.com>
 <20220208122624.43ad52ef.alex.williamson@redhat.com>
 <438d8b1e-e149-35f1-a8c9-ed338eb97430@linux.ibm.com>
 <20220208204041.GK4160@nvidia.com>
 <13cf51210d125d48a47d55d9c6a20c93f5a2b78b.camel@linux.ibm.com>
 <20220210130150.GF4160@nvidia.com>
 <fc5cce270dc01d46a6a42f2d268166a0a952fcb3.camel@linux.ibm.com>
 <20220210152305.GG4160@nvidia.com>
 <bcf04ad2-848b-de03-5610-d99e3b761b10@linux.ibm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <bcf04ad2-848b-de03-5610-d99e3b761b10@linux.ibm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?OegLJe+I0HpG6iw3mvF61Poi5ceVfIFVFjGCNP4HJ83RrEn0lIpPlQT/mvlY?=
 =?us-ascii?Q?ZjphmQgBTpCKc0dk2pelJ+B5ePZKQh2fggE7HCJ0paJK2EUkRXNNAhfp2GP3?=
 =?us-ascii?Q?9Ysl4QiaDBdoS+ugI8TOUMIrE++pvPUV2nl8A493nQ4oRNLlxeQJzvIg+vfQ?=
 =?us-ascii?Q?9u6tV5ZYjjPsKRgZYJrG9PWh3JdCXjtwRme/OuAqeki/+oHTZ53Xy3sUR82B?=
 =?us-ascii?Q?VCDRlAQpvhkceV0KafmCfowQAD/+7/R0shMMB9nya4GUQlhOTIBw11CCbAbx?=
 =?us-ascii?Q?ye+7f52ptXloUyjD1V7MBpZdHY/U43xioy07XEwo1UuNbRGsVn5d8Ee6nptA?=
 =?us-ascii?Q?Hbx2doJi05QkV95EP6adkR1vBj1e+yblLuQzu8lNEuYsCozxIMgbnlf7q1hl?=
 =?us-ascii?Q?UZUnYu24TINO2ifDOvblqw8smR7JIZCiN9ZcRQV6FOHaJBTcQGB0uM6k94Hm?=
 =?us-ascii?Q?Ud0kHh4tknmVFmVmyD8dwOmZbM+1h/TXjyEqt+2VYD3jJZ6WkQiLpoZEeWnd?=
 =?us-ascii?Q?kQHrdL5BQCisHGzhok+kvh5IubYa+LmF4XZaEA2CA1oDKTQoL70HZ0NHdeFa?=
 =?us-ascii?Q?sPyfrmh6Z3D8VdXnAYmvgxFoOYABT7dsUDZ6wXeVhdlLRAl7LheWTBLPYpib?=
 =?us-ascii?Q?Gg0CCkN+pACTrD1h+f+hyDfuMiuHeTxtU1dCyxXN/9wfyAv3V4sUZxgaPfcI?=
 =?us-ascii?Q?OBeSWB5uzsYwBbQ8wO0Esxr96vq5XM8ofUdNpVZNWyMElHs7pxeL68Evw7wD?=
 =?us-ascii?Q?G1DEEco6GDB/6Qjsm6zEtqK5kx5K8aht9ZgBoV5fFFAcDQHid3s+EoN8HFYt?=
 =?us-ascii?Q?JftG67IfZm6W1B9GzJJcucYKt8xWFXH9+ZqbBYYj8/atpDcesbp1Z8kRcUfV?=
 =?us-ascii?Q?4vT6a1lu2AYCqix+UCWoPuKlp9QWb+1LSIqjgWVcgBY1GRIk4SOsXstLIwZ+?=
 =?us-ascii?Q?e/nJDwiFJim+bmgi7BpmNlKMvVpx0LApOlcSd8LKHrj1rb9kYfxdKyR9vIdp?=
 =?us-ascii?Q?mR8dIcOzMzpRESmLUO0GIpPCNv4KJtbgcsCW5IUXvp1BZHV0PlNZQ2MlHtCA?=
 =?us-ascii?Q?nxpnu+uOI6wGj/gPE69LtofewadRXcWgk9hsMLhIPEHHJp3PZ2NgVxEOhiaP?=
 =?us-ascii?Q?+Vi6MRByKfgKt6e9oAGkXz7BCeQCdAUMICVRdfQpPeQW1+jfycWoQO3tEsrV?=
 =?us-ascii?Q?o0vx+ukpIRswd0W+oxo8gyqL4BjjzcOImiHmKDBipdOfyYoXdvsAWPub3UPY?=
 =?us-ascii?Q?AN20aMuill9yt78qjG/xbODg47Mj7TQyUx8BdlL/7yBRsLKBULMMDW83Zcw4?=
 =?us-ascii?Q?8G9robcElUVT82SVhOrCeIZmUxzwfjGIpfNI0KV1quAkSK7eAixMImrWheDh?=
 =?us-ascii?Q?lUKBvrN+ISDRosuJvamFUs/atJ3nb6zTDGJYoSe5+qqSPcI7OG+uCAG5ucxM?=
 =?us-ascii?Q?yZrFOWBD6LLBmcuv19upYrTMVeqpPkU6tpBxc66H+GKMa0yA0ykU4RSimcdI?=
 =?us-ascii?Q?tUQttrwiyRoLJTcLulTgZUEKbYQDOiADwJzyZYeir7UD13In8jthu9CR9MQp?=
 =?us-ascii?Q?61VAndOVS18OUmftGnU=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 87a6526c-871b-4fa8-7674-08d9ecef5faf
X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Feb 2022 23:45:09.8027
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 2LkcPVj45R8EmvqE0HeDBuHCUKFL2uG50X2zxyfuk+L2k8woVsDK90kk3qqm+Ryi
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1201MB0064
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Feb 10, 2022 at 01:59:35PM -0500, Matthew Rosato wrote:

> OK, looking into this, thanks for the pointer...  Sounds to me like we then
> want to make the determination upfront and then ensure the right iommu
> domain ops are registered for the device sometime before creation, based
> upon the usecase -- general userspace: s390_iommu_ops (existing), kvm:
> s390_kvm_iommu_domain (new).

Yes, that is the idea. I expect there will be many types of these
special iommu domains. eg Intel has talked about directly using the
KVM CPU page tabe as the IOMMU page table.
 
> > When the special creation flow is triggered you'd just create one of
> > these with the proper ops already setup. >
> > We are imagining a special ioctl to create these things and each IOMMU
> > HW driver can supply a unique implementation suited to their HW
> > design.
> 
> But I haven't connected the dots on this part -- At the end of the day for
> this 'special creation flow' I need the kvm + starting point of the guest
> table + format before we let the new s390_kvm_iommu_domain start doing
> automatic map/unmap during RPCIT intercept -- This initial setup has to come
> from a special ioctl as you say, but where do you see it living?  I could
> certainly roll my own via a KVM ioctl or whatever, but it sounds like you're
> also referring to a general-purpose ioctl to encompass each of the different
> unique implementations, with this s390 kvm approach being one.

So, the ioctl will need, as input, a kvm FD and an iommufd FD, and
additional IOMMU driver specific data (format, starting, etc).

The kvm supplies the context for the RPCIT to be captured in

The result is the creation of an iommu_domain inside iommufd, done by
some iommu_ops->alloc_domain_xxxx() driver callback

Which FD has the ioctl it is a bit of an aesthetic choice, but I
predict that iommufd makes the most sense since an object is being
created inside iommfd.

This flow is very similar to the 'userspace page table' flow others
are looking at, but has the extra twist that a KVM FD is needed to supply
the CPU page table.

It may overlap nicely with the intel direction I mentioned. It is just
ugly layering wise that KVM is getting shoved into platform code and
uapis all over the place, but I suppose that is unavoidable. And the
required loose coupling with the kvm module means all kinds of
symbol_gets'etc :(

Jason