Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 12 Jan 2023 13:27:41 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Matthew Rosato <mjrosato@linux.ibm.com>
Cc:     Sean Christopherson <seanjc@google.com>,
        alex.williamson@redhat.com, pbonzini@redhat.com, cohuck@redhat.com,
        farman@linux.ibm.com, pmorel@linux.ibm.com,
        borntraeger@linux.ibm.com, frankja@linux.ibm.com,
        imbrenda@linux.ibm.com, david@redhat.com, akrowiak@linux.ibm.com,
        jjherne@linux.ibm.com, pasic@linux.ibm.com,
        zhenyuw@linux.intel.com, zhi.a.wang@intel.com,
        linux-s390@vger.kernel.org, kvm@vger.kernel.org,
        intel-gvt-dev@lists.freedesktop.org,
        intel-gfx@lists.freedesktop.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH 1/2] KVM: async kvm_destroy_vm for vfio devices
Message-ID: <Y8BDDQi9W86UfszF@nvidia.com>
References: <20230109201037.33051-1-mjrosato@linux.ibm.com>
 <20230109201037.33051-2-mjrosato@linux.ibm.com>
 <Y78UCz5oeuntSQtK@google.com>
 <Y78Wk2/P5+gLMdpk@nvidia.com>
 <Y78hzsHiwaFpL60+@google.com>
 <Y8AA8r5MzKQIF8I7@nvidia.com>
 <f7c39317-92a4-520e-8e69-a8606cd40e9a@linux.ibm.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <f7c39317-92a4-520e-8e69-a8606cd40e9a@linux.ibm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?O6P/K9OMRkVNoyLk6BMATJ58EfNc6TAkKgpJJ5CG1dQYo1wuEYKORv5D2euR?=
 =?us-ascii?Q?z91zqL5R3xeua7W2UcimdehzehqBkUVWNAeCVLAMpCeklNV/bksDOQVrKZ1u?=
 =?us-ascii?Q?/hxIgKRX81cXgWR+BAooauqr5A2KboMrihaeVKn+2qjAZlIXaOf9G3llz0XX?=
 =?us-ascii?Q?ZrngCaL8G3e3e/voOvd96XCA5uRP0+9sbSXn0CqBpIm80HxKvRbAKJuHzmji?=
 =?us-ascii?Q?Yl0hFvzEHdK2xGp7JTAowmjn76T6d2DkpB1/dJapOO1Dzd+NmkATFRQQmSQ1?=
 =?us-ascii?Q?XAd574O6YrSxse39AUDtblpd5OdyGLl96BljkGIrw+6PfQMreJE1S2cNeQon?=
 =?us-ascii?Q?ckHzuWaAL/mv17lK+nxkPsnBvrbCcLnCFyk0oE8Nolbjb5nI7stexzHAwSj9?=
 =?us-ascii?Q?SpWcEv91uTh6/J+C2Nkxb1HEz8oHNiOXGejcKyYYT82dyQ4jKNHcGIIiVZE5?=
 =?us-ascii?Q?ZUOL9s16z8ESDfr794B1kNvxTTaE55hw+Rt/AiQFeb9UDkVRG5fs/YRPzNno?=
 =?us-ascii?Q?I26mWmXCFSmIOlOc46+D9I+SepIwjd6IryYF6YSypND0nQs/ibchUvvo92KS?=
 =?us-ascii?Q?zVdVh6pdV2asLRsyd+qf0H6j+HUL91roT7Mral3D5J0gTDiniqf4jXtxd3TJ?=
 =?us-ascii?Q?hj1a+6SVjhjj2TRzau3x3/KCPqfiYhjgPl7m9sOY6yazPGPhe23MoClNCD9d?=
 =?us-ascii?Q?i//lgIeFnzkSdvdQFAscTl8bHdGCo9gNmVTqICg+okVxm46+dVpF6WrGOzv5?=
 =?us-ascii?Q?e8YIBdf9R2WKFUr6OQ6WGKLxXjGIgvyFhqk+KMBvzZl/Cwh5sAeOfctPhNW1?=
 =?us-ascii?Q?Zd8a6pRyqfsHKf/mm2uS7V+8wJ/po/ui72CFhh8g1E9jS6jLEDK49fpF6ZfF?=
 =?us-ascii?Q?PBEDT3c4ZVUTOXZwQd0haCz9Z/HAa9a2C2Zjikn3nnbcNVCliVFWG0dRuV8z?=
 =?us-ascii?Q?/yFBjzCXu0TrldDxkG6gBCyls/EoFSQ9glx0T4utjv49NksyKXzy2jNzcWo6?=
 =?us-ascii?Q?7f4U+WgnOirSBMFr4G60LMEl2OgUTNCjTPYPGuPO2cWio0nfyqlfo6z7y39P?=
 =?us-ascii?Q?5NtXo/04tMea+Gzab94cSEPpvdsOEsLsxqApu7KIu94owy2HakamIdI7ONAd?=
 =?us-ascii?Q?97YZMjI6ZnUqalY0aK5cpG3+sGUJR5/+efJcAfhJvqYZPWDOp7en3dKHMb+K?=
 =?us-ascii?Q?t5mr2w9LtzL6+hhkBc1aC6bSfBDx7NJH+bYOCAzMCeB8bf45XBgaiOq514ac?=
 =?us-ascii?Q?jYaBRar0htmJ+Ku2mH92vySF78xD3sfx68r8LxTiKyFq6ROm0kohyiQ3FXUy?=
 =?us-ascii?Q?RbYAMuPRmJbadDg/zUy+AsxYZk44y7ou1cKsC2pm420jNU397LGDqU2u69DH?=
 =?us-ascii?Q?I2ItXrbH9nVFXYcE3oWxeX/BRM8llsUx8zlZ4v8z3Ppyj8Z/EJrdtfBk2YSp?=
 =?us-ascii?Q?nC9b9JnrGuicWS6iq0o2kT3vge731Yb2IOzLvYAPcfqvskBtDfBGZ8jI9YSe?=
 =?us-ascii?Q?ehBTeIPT9MEA08/une6/xj5W4x+T5oENHo79/F1ybmbBz/KN6mf4VLFME6wH?=
 =?us-ascii?Q?gjpU8vUmZuFZKVHpikF98Y73Fe52Oc24rAlx1A4B?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7f46d4fe-62cb-4b14-8df2-08daf4c24f6d
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jan 2023 17:27:42.2740
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ++ipFJ8JeXQ26INqwLyhOglearOHip6ePI6X/XAw1w/mNmErT2SxLOD7MchR2qzz
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB7463
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Jan 12, 2023 at 12:21:17PM -0500, Matthew Rosato wrote:

> So should I work up a v2 that does symbol gets for kvm_get_kvm_safe
> and kvm_put_kvm from vfio_main and drop kvm_put_kvm_async?  Or is
> the patch Yi is working on changing things such that will also
> address the deadlock issue?

I don't think Yi's part will help

> +361,22 @@ static int vfio_device_first_open(struct vfio_device
> *device, if (ret) goto err_module_put;
>  
> +       if (kvm && !vfio_kvm_get(kvm)) {

Do call it kvm_get_safe though

> +               ret = -ENOENT;
> +               goto err_unuse_iommu;
> +       }
>         device->kvm = kvm;
>         if (device->ops->open_device) {
>                 ret = device->ops->open_device(device);
>                 if (ret)
> -                       goto err_unuse_iommu;
> +                       goto err_put_kvm;
>         }
>         return 0;
>  
> -err_unuse_iommu:
> +err_put_kvm:
> +       vfio_put_kvm(kvm);
>         device->kvm = NULL;
> +err_unuse_iommu:
>         if (iommufd)
>                 vfio_iommufd_unbind(device);
>         else
> @@ -465,6 +471,9 @@ static int vfio_device_fops_release(struct inode *inode, struct file *filep)
>  
>         vfio_device_group_close(device);
>  
> +       if (device->open_count == 0 && device->group->kvm)
> +               vfio_kvm_put(device->group->kvm);
> +

No, you can't touch group->kvm without holding the group lock,
that is the whole point of the problem..

This has to be device->kvm

Jason