Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 13 Jan 2023 10:49:06 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Sean Christopherson <seanjc@google.com>
Cc:     Alex Williamson <alex.williamson@redhat.com>,
        Matthew Rosato <mjrosato@linux.ibm.com>, pbonzini@redhat.com,
        cohuck@redhat.com, farman@linux.ibm.com, pmorel@linux.ibm.com,
        borntraeger@linux.ibm.com, frankja@linux.ibm.com,
        imbrenda@linux.ibm.com, david@redhat.com, akrowiak@linux.ibm.com,
        jjherne@linux.ibm.com, pasic@linux.ibm.com,
        zhenyuw@linux.intel.com, zhi.a.wang@intel.com,
        linux-s390@vger.kernel.org, kvm@vger.kernel.org,
        intel-gvt-dev@lists.freedesktop.org,
        intel-gfx@lists.freedesktop.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2] vfio: fix potential deadlock on vfio group lock
Message-ID: <Y8FvYhpIRIiX22Op@nvidia.com>
References: <20230112203844.41179-1-mjrosato@linux.ibm.com>
 <20230112140517.6db5b346.alex.williamson@redhat.com>
 <bce7912a-f904-b5a3-d234-c3e2c42d9e54@linux.ibm.com>
 <Y8CX8YwT/T9v4U/D@google.com>
 <20230112175648.158dca5f.alex.williamson@redhat.com>
 <Y8C8WpkjgFelldTQ@google.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y8C8WpkjgFelldTQ@google.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?D3Eu+T/J4rp95pnTWZWBiqqFDBfY1p+lumNoKL9HsOVdy2UyIf1qoajsf/T+?=
 =?us-ascii?Q?0jE3KW38ubQm1b1cUEVwFAAp5JGhKgj3umEdw12sHlXdkdCxC1QDatsB4mwH?=
 =?us-ascii?Q?bmsPwkVikDj810mcOWvEUTx+inn7iI+MFn09+Qm5UqLunBpXJc+9WkORJGhp?=
 =?us-ascii?Q?UKue8MDg4bNd5N+FHsWXaVJiOnOkmYt4u+fu5e6qclHgBkQn2LIunhXJfsYf?=
 =?us-ascii?Q?wKJNL3AZ8A62++0emm0xit4j3VcBiUxw9078JQ0beTmKMwMPwwIsANnlhTPR?=
 =?us-ascii?Q?G+XIWcU9N5z2xrGXXvTaDwXMz47BLH9rGb3you6beM4q99gXHWhiIMbu5yoX?=
 =?us-ascii?Q?rUwMGXmx3xTm1CvT/t4i19f17swQfE/cYLwJeqVR+infhTgzwm0hPLjxIi6Z?=
 =?us-ascii?Q?gUCec1QiqOjclrLs/mCckZaWsSWcv+G+iru7wGxpjF1LKu8fQItV7bt5WnLi?=
 =?us-ascii?Q?rZooUzOzuhPKlIjpG4ikYADMTiqCtKruzZu4Xb8F4uEgTMrSTFJnD3efJ8k/?=
 =?us-ascii?Q?Hdtcez9AZjl+G9iazcprK8zjVVXg9jx/mZn6Fdy0ghktioL4bbn0m4maERYr?=
 =?us-ascii?Q?KoXkqvrcE41xBjGtdGebpmStvNjWYPLRUsjocS/nXmMNhEPTrKHzbph9/DId?=
 =?us-ascii?Q?COirjAYfEUbZ12ncuXFG1DRYgb2vafpd5OI9BtaxrEGbhmNYLdi/018N0OAl?=
 =?us-ascii?Q?6tAW3zLrdgd1f2xaUJahS6X5L9yVxckHlMGCd7Wz6TH8OIh26vFdfb6WuxVR?=
 =?us-ascii?Q?6AEiJVqOAD/QMlNq3RTlJKjwXNmV9RAvn+jxrPqNm64V09RxQ+rWM+kYFxc0?=
 =?us-ascii?Q?vkA3DXNSw1Lpf654jiSVuDSeIjfq2gBkj8QjXcFy0qPBFN+M+5BMslukBomP?=
 =?us-ascii?Q?8kCgN7R8ih+2is89zm3I9aDYwb5sGdfE6CuJ18xTRID4nfh1jRDZSZv5Vg7g?=
 =?us-ascii?Q?RR4BCtEXXeX5HEz4QmdDmz5HYgb3GVyY3EXqQwQbzrWGINtlm+BoHNGNvF1r?=
 =?us-ascii?Q?m+lkpIF5xDNfRXisoavZQcALorEYs3pvRZn/Hqt9lcEelutbqMR388kDzTx1?=
 =?us-ascii?Q?8/u9PeTinv2nYMruihCxR9iuvsqNxgvm1cdw7z/CQucAlPWeGV4Qcw1Lssoy?=
 =?us-ascii?Q?CUH6SDFO95O50dW3TrGRHjuoTbxhEWvP5WCZPhezy3CZ1wvan/OFaqFHMS7L?=
 =?us-ascii?Q?DVo/yaowoOjhX5w5F4rOYgdIERQURTxITD1CoNLRrOur/a3Bt86wnA9qHwgY?=
 =?us-ascii?Q?/q5eM5Lem9IwP+nq877A8p7GM8H9Cc+g8YjmNJVpScC8mbEazMHT2oM4Oqvo?=
 =?us-ascii?Q?mpSXdLcq/IJZjLi+u1uZnRhCjKXRhwYOlXQ/7HPJQWfuXqcbzmxdskm2lRtU?=
 =?us-ascii?Q?laNGFSrSKmAVfeq0dJeSOfBXLniopmQefKhjcOj64I6z61EywwnvrDXHhR6t?=
 =?us-ascii?Q?vGamSW/GbSDnBBHpesKCRwk7KCt92CFJqzs15FspL9xzJxU0paGThJZPpNna?=
 =?us-ascii?Q?657y0IcaEfTMr9d7fytmb0bprJpUDtqg6IJgfIqW1zgQQueAieKGEuOniJLQ?=
 =?us-ascii?Q?N8rjtvf8j/Yj3JH5QuYjDxRkKWSk+7vGbsDBJ8Zg?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c294d0d3-7fa7-44ba-0ed0-08daf57552b8
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Jan 2023 14:49:07.6150
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 2z1J7xWDyEUNrek56bdz+ZNFV5GQ8jZRwNjt8P2U51I65/3YKLIrvyO/7g/w7zTL
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6382
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Jan 13, 2023 at 02:05:14AM +0000, Sean Christopherson wrote:
> On Thu, Jan 12, 2023, Alex Williamson wrote:
> > On Thu, 12 Jan 2023 23:29:53 +0000
> > Sean Christopherson <seanjc@google.com> wrote:
> > 
> > > On Thu, Jan 12, 2023, Matthew Rosato wrote:
> > > > On 1/12/23 4:05 PM, Alex Williamson wrote:  
> > > > > On Thu, 12 Jan 2023 15:38:44 -0500
> > > > > Matthew Rosato <mjrosato@linux.ibm.com> wrote:  
> > > > >> @@ -344,6 +345,35 @@ static bool vfio_assert_device_open(struct vfio_device *device)
> > > > >>  	return !WARN_ON_ONCE(!READ_ONCE(device->open_count));
> > > > >>  }
> > > > >>  
> > > > >> +static bool vfio_kvm_get_kvm_safe(struct kvm *kvm)
> > > > >> +{
> > > > >> +	bool (*fn)(struct kvm *kvm);
> > > > >> +	bool ret;
> > > > >> +
> > > > >> +	fn = symbol_get(kvm_get_kvm_safe);
> > > > >> +	if (WARN_ON(!fn))  
> > > 
> > > In a related vein to Alex's comments about error handling, this should not WARN.
> > > WARNing during vfio_kvm_put_kvm() makes sense, but the "get" is somewhat blind.
> > 
> > It's not exactly blind though, we wouldn't have a kvm pointer if the
> > kvm-vfio device hadn't stuffed one into the group.  We only call into
> > here if we have a non-NULL pointer, so it wouldn't simply be that the
> > kvm module isn't available for this to fire, but more that we have an
> > API change to make the symbol no longer exist.  A WARN for that doesn't
> > seem unreasonable.  Thanks,
> 
> Hmm, I was thinking that it might be possible for kvm.ko to be on its way out,
> but barring use of force module unload, which breaks things left and right, kvm.ko
> can only be going if all VMs have been destroyed.

If we really care about these details then we should obtain both the
get_safe and put together, the put pointer should be stored in the
device and it should be symbol_put'd back once the kvm is put back and
it isn't needed any more.

This properly mimics how normal module stacking would work

Jason