Received: by 2002:a05:7412:37c9:b0:e2:908c:2ebd with SMTP id jz9csp450682rdb; Mon, 18 Sep 2023 23:30:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFnmB59ZNieyH9tLLIYTecdzMo7Vb0Ehe7leNVE5vDCuDnqlE+XyMuvg09SuJIv3WXueRGs X-Received: by 2002:a05:6358:722:b0:135:5ede:f352 with SMTP id e34-20020a056358072200b001355edef352mr12008394rwj.8.1695105029041; Mon, 18 Sep 2023 23:30:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695105028; cv=none; d=google.com; s=arc-20160816; b=L7Q1dt3H2MlKLzmm32xfHpcrl/Hq7r+5CpREtSSUSezcfcYAzGB5PGMLKZpDIUg9KE j4FLAt7kyId4FcR3xPZmM6EjFIaQGLOVlfWuPW9P/j7ytACzDZ5vMILOMhiCoLJEYI5G opy6d9z6+mkYUJeWtvBxixBdnbwREgMf4IYk+5sBbnWhMLzFTVCLIZiigktt7fgIRJ6Z 1HWbEPyhza7qK11a1Ds+KrsycYGx9D4HiR1ETikAtVgjtfqG0QrvLdhmJtoUF9MTqyia L1pqOcpCDBtBprM80c4lqKw5X5rxDC07589oi3K9mEx12MF5rhYTw1o+yBB9E0GKtO6j UH2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=CCJJzfg2EU4sdimz+f4BBAfZOdlEW4gvV76ebOvr24U=; fh=+i2Z+Bpp/5N3ZNaHkNF2Yd/3QhJpNwPGjPT1YqjPH7U=; b=C5d4v1gTawjQoUOA4+g8Sl3A6IAusoP9EmJw/jjgq0P10bqm+QJC7ExYWEVIJGTb0E uuKzPADH5iJCwxIB0QhqaXA75KBrB6AcUSb4Vm/X32sg22csxPPSA2OCnCcAL3TrAMIp Wb9NugfVHtJutUp4+sUnJDtdErCdFlJ+GGDyYJiYHvr4ijh2syrnnMbtaiuTp38+qX6V XSU9fMjisb6PRf5s4OIIQXoaQb/t4tmA+BOaRbDyrSVL0DCAonrQeABqUNbY5G6OOXND dAse6J+yucmvveSs5soi5BjQvATj3DTTs+R6X+s+XR+GsxtElLZ+lsfDIrkx6avJ2hw1 ildA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=ZoQvfRDm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id bt8-20020a632908000000b0057744d09d2fsi9116562pgb.18.2023.09.18.23.30.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Sep 2023 23:30:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=ZoQvfRDm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 7D17B819803A; Mon, 18 Sep 2023 17:02:28 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230319AbjISAC0 (ORCPT + 99 others); Mon, 18 Sep 2023 20:02:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50228 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230099AbjISACZ (ORCPT ); Mon, 18 Sep 2023 20:02:25 -0400 Received: from mail-qv1-xf34.google.com (mail-qv1-xf34.google.com [IPv6:2607:f8b0:4864:20::f34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB7ED99 for ; Mon, 18 Sep 2023 17:02:17 -0700 (PDT) Received: by mail-qv1-xf34.google.com with SMTP id 6a1803df08f44-6563fe128efso13154656d6.3 for ; Mon, 18 Sep 2023 17:02:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1695081737; x=1695686537; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=CCJJzfg2EU4sdimz+f4BBAfZOdlEW4gvV76ebOvr24U=; b=ZoQvfRDm7IP+/rJcKtrAiSuUPUkCcVIHgYVHgxh1JSLa+HGcGw16W/uGelOoXpuLoZ okU9Io9iOZUcobE8aVSRxDJ/0ud7kiBrWbceB/641Jbk+X3J9PzI2W6UVfFQwpATHMay yvpfLhTQZAElo/0yrSp40YIlWxJ/ajZYVtMQ1nrauy55c8EeQ1Q/6UBfECCj45GATJtn vu1diTLINJ40zYZy7wfhz15ByA4zC4vTl+ZXgk+txKZrLwIvMtVBTlRj0iQD9PJ+fH+V tLgMvjW2lpb1VpkAVITq9oq8LJl+tn2su1A9+9gqiMr4r2cepsf5PY/dUutJMYDU5ZJf KCow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695081737; x=1695686537; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=CCJJzfg2EU4sdimz+f4BBAfZOdlEW4gvV76ebOvr24U=; b=eU5F0IXXg2/T599LF6KErMNS6sd/n+GR/WmPd0jKSNiwG+/7z3kj6LbdYTsxlY5q8/ neXH8tv8TWQLd/QqiUgbjTUbMNu4d/b4Yr1+K4L2mMo52sjRvVqeXPnROMBvi4zHjJaL 5lr2EDpWh0GS5EJh8al/YeVXMfZY+veOdzi9MUstYtFdPpXEP1i2+SLjIWWBYZyNpMzH tG5q2grfTfnM60S4SMj6PDwcd0qpQDDkBqJZlsVSktqFKN1ATchi8tInao+4ppM9Lakr sPogwgQ90hWPBALrffkgkREOuixRYyJPVO5FymI/6zG6MMKUDFpUr/nkm1gxZMeMlDs7 Ji7g== X-Gm-Message-State: AOJu0YwAiQY+Aa8X3XI/lBYMIHNKKfbsaQeBHvr0PdMPi5meIsN365kl h5bR7xDB6Iu1WF+0erfezypTpw== X-Received: by 2002:a0c:db83:0:b0:657:5372:ac0d with SMTP id m3-20020a0cdb83000000b006575372ac0dmr5192825qvk.38.1695081736995; Mon, 18 Sep 2023 17:02:16 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-26-201.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.26.201]) by smtp.gmail.com with ESMTPSA id n18-20020a0ce492000000b0064713c8fab7sm600289qvl.59.2023.09.18.17.02.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Sep 2023 17:02:16 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1qiOBz-000834-UF; Mon, 18 Sep 2023 21:02:15 -0300 Date: Mon, 18 Sep 2023 21:02:15 -0300 From: Jason Gunthorpe To: Thomas Gleixner Cc: Shannon Nelson , alex.williamson@redhat.com, kevin.tian@intel.com, reinette.chatre@intel.com, kvm@vger.kernel.org, brett.creeley@amd.com, linux-kernel@vger.kernel.org, Leon Romanovsky Subject: Re: [PATCH vfio] vfio/pci: remove msi domain on msi disable Message-ID: <20230919000215.GQ13795@ziepe.ca> References: <20230914191406.54656-1-shannon.nelson@amd.com> <20230918141705.GE13795@ziepe.ca> <87led3xqye.ffs@tglx> <20230918233735.GP13795@ziepe.ca> <87a5tjxcva.ffs@tglx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87a5tjxcva.ffs@tglx> X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 18 Sep 2023 17:02:29 -0700 (PDT) On Tue, Sep 19, 2023 at 01:47:37AM +0200, Thomas Gleixner wrote: > On Mon, Sep 18 2023 at 20:37, Jason Gunthorpe wrote: > > On Mon, Sep 18, 2023 at 08:43:21PM +0200, Thomas Gleixner wrote: > >> On Mon, Sep 18 2023 at 11:17, Jason Gunthorpe wrote: > >> > On Thu, Sep 14, 2023 at 12:14:06PM -0700, Shannon Nelson wrote: > >> >> The new MSI dynamic allocation machinery is great for making the irq > >> >> management more flexible. It includes caching information about the > >> >> MSI domain which gets reused on each new open of a VFIO fd. However, > >> >> this causes an issue when the underlying hardware has flexible MSI-x > >> >> configurations, as a changed configuration doesn't get seen between > >> >> new opens, and is only refreshed between PCI unbind/bind cycles. > >> >> > >> >> In our device we can change the per-VF MSI-x resource allocation > >> >> without the need for rebooting or function reset. For example, > >> >> > >> >> 1. Initial power up and kernel boot: > >> >> # lspci -s 2e:00.1 -vv | grep MSI-X > >> >> Capabilities: [a0] MSI-X: Enable+ Count=8 Masked- > >> >> > >> >> 2. Device VF configuration change happens with no reset > >> > > >> > Is this an out of tree driver problem? > >> > > >> > The intree way to alter the MSI configuration is via > >> > sriov_set_msix_vec_count, and there is only one in-tree driver that > >> > uses it right now. > >> > >> Right, but that only addresses the driver specific issues. > > > > Sort of.. sriov_vf_msix_count_store() is intended to be the entry > > point for this and if the kernel grows places that cache the value or > > something then this function should flush those caches too. > > Sorry. What I wanted to say is that the driver callback is not the right > place to reload the MSI domains after the change. Oh, that isn't even what Shannon's patch does, it patched VFIO's main PCI driver - not a sriov_set_msix_vec_count() callback :( Shannon's scenario doesn't even use sriov_vf_msix_count_store() at all - the AMD device just randomly changes its MSI count whenever it likes. > > I suppose flushing happens implicitly because Shannon reports that > > things work fine if the driver is rebound. Since > > sriov_vf_msix_count_store() ensures there is no driver bound before > > proceeding it probe/unprobe must be flushing out everything? > > Correct. So sriov_set_msix_vec_count() could just do: > > ret = pdev->driver->sriov_set_msix_vec_count(vf_dev, val); > if (!ret) > teardown_msi_domain(pdev); > > Right? It subtly isn't needed, sriov_vf_msix_count_store() already requires no driver is associated with the device and this: int msi_setup_device_data(struct device *dev) { struct msi_device_data *md; int ret, i; if (dev->msi.data) return 0; md = devres_alloc(msi_device_data_release, sizeof(*md), GFP_KERNEL); if (!md) return -ENOMEM; Already ensured that msi_remove_device_irq_domain() was called via msi_device_data_release() triggering as part of the devm shutdown of the bound driver. So, the intree mechanism to change the MSI vector size works. The crazy mechanism where the device just changes its value without synchronizing to the OS does not. I don't think we need to try and fix that.. Jason