Received: by 2002:a05:7412:37c9:b0:e2:908c:2ebd with SMTP id jz9csp372879rdb; Mon, 18 Sep 2023 19:20:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGJMGnJBXplM1v3Wr860sly+oPVDxXAnl2ylJ91CuoQ2MxU7M3XPIc9BJ9TdYh4ytF6TJwN X-Received: by 2002:a05:6359:428b:b0:143:3845:e6fb with SMTP id kp11-20020a056359428b00b001433845e6fbmr4668955rwb.27.1695090038434; Mon, 18 Sep 2023 19:20:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695090038; cv=none; d=google.com; s=arc-20160816; b=piqHmV87C1Py6iBIAOTBruq1dXFn3tp/DRoYTdGFgc2jV1ctuebRaNNgKkiR8Gll5S 2x2Hs8EwU4LeA8EjGFus4JuxMOMFigkrdphwaAjMLqA+2jGWceyiPOqrIoMwyUfNkCiz B+PY2n8zT7lpa7E4NbpkhkM8dgNBQv0B9bG46yprW21/x+DKIMDmOTkayjqqx1eBfaps PUJRQuscrYKVPAj3GKx3pqzUyPYEH8sDccCUugdaoRcx4TbrssjLMp8qKd0eoowmlwrF sWOSlmPIvjSjiYWzYIc7AaJRcjO0GT50syQMqgeFBGEsV5k7aCkpJgbFUfBdMQvZQ1Ov 66Kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:message-id:date:references :in-reply-to:subject:cc:to:dkim-signature:dkim-signature:from; bh=/RQ+MZYYkIGZimhGMzDLvmhGo+fnFrbGeertT/ie31Q=; fh=YtAJvZ68JdGg/6+oi9glt1TYhfwgGAkKBdaoWeZOpBY=; b=eIVX/pV3d9j0/2oOerHcipRasUJPM1casbqyBkb7wWHExBMqfHDmLf2ZxcyZr1k1wG SGl2HxcciQHrA8qZrD4q3s962V6DEfObSYwvXjELgrNp15PwIjVY4KX7HZkRtijUBuZP OUCZ32PEmue0dKvzm27cBB7ENulBQhh8DEGalewHhHrvLZMDOE4TRWodt0ug+CQRwJ1n xWMns/XrG9PEjLJqPYrFxXqLPPl4cwJ5ZCm/5iEiO8tM6WFuogDz5RoXoCI3R+4BQ7bd MwSqmtiwaoOehwha24lVKcdtxMzYhUaoxDZZDU0e+Qn8IvRD3yEuP6Lc8ovVuqcyJ86u w4nQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=N+LJ52EZ; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Return-Path: Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id cr11-20020a056a000f0b00b0069069ac51a8si4251395pfb.22.2023.09.18.19.20.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Sep 2023 19:20:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=N+LJ52EZ; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 96123840FE30; Mon, 18 Sep 2023 11:43:40 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229821AbjIRSnf (ORCPT + 99 others); Mon, 18 Sep 2023 14:43:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44156 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229654AbjIRSna (ORCPT ); Mon, 18 Sep 2023 14:43:30 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 24886109; Mon, 18 Sep 2023 11:43:23 -0700 (PDT) From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1695062602; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=/RQ+MZYYkIGZimhGMzDLvmhGo+fnFrbGeertT/ie31Q=; b=N+LJ52EZv2tUrO6xCr1UMKdEay09OpaHz+XKZAUwKx/57FDDFx8OAhect7CtTr/dqVJWS2 uZ/WPiVG9kqesUZmj+cTOTM8S6DzkqisZXiEW2XgEem3AU8oocZK42RXLdTw8qOGdYEpaW cyEtXymmCN4Av7jfcT7X9ZzwwqRqPUuUVFFVDZJlvzCX6GHKfPZaF24wLquJEhyTjvTmt0 RYC+XKhbH5eNn+oILVnt+p/g55IPJ3WT9cIgaw7PeUYzD6JW5x1udq57LqM+gItsc7K6BV QDWaT7drUozGe5HURM9OPH7uJ6mZnRPovbmWw/nbaCHeYwEYtlfwiAnl/syijA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1695062602; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=/RQ+MZYYkIGZimhGMzDLvmhGo+fnFrbGeertT/ie31Q=; b=A8Nwf7ugDqcf469ra2fSnXBXHZu4t6IWlloE6yfNFc+izWgC5LSeVcCCclQDGkqyuXA+qR 4JCaiEkf2YXmKmDA== To: Jason Gunthorpe , Shannon Nelson Cc: alex.williamson@redhat.com, kevin.tian@intel.com, reinette.chatre@intel.com, kvm@vger.kernel.org, brett.creeley@amd.com, linux-kernel@vger.kernel.org, Leon Romanovsky Subject: Re: [PATCH vfio] vfio/pci: remove msi domain on msi disable In-Reply-To: <20230918141705.GE13795@ziepe.ca> References: <20230914191406.54656-1-shannon.nelson@amd.com> <20230918141705.GE13795@ziepe.ca> Date: Mon, 18 Sep 2023 20:43:21 +0200 Message-ID: <87led3xqye.ffs@tglx> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 18 Sep 2023 11:43:40 -0700 (PDT) On Mon, Sep 18 2023 at 11:17, Jason Gunthorpe wrote: > On Thu, Sep 14, 2023 at 12:14:06PM -0700, Shannon Nelson wrote: >> The new MSI dynamic allocation machinery is great for making the irq >> management more flexible. It includes caching information about the >> MSI domain which gets reused on each new open of a VFIO fd. However, >> this causes an issue when the underlying hardware has flexible MSI-x >> configurations, as a changed configuration doesn't get seen between >> new opens, and is only refreshed between PCI unbind/bind cycles. >> >> In our device we can change the per-VF MSI-x resource allocation >> without the need for rebooting or function reset. For example, >> >> 1. Initial power up and kernel boot: >> # lspci -s 2e:00.1 -vv | grep MSI-X >> Capabilities: [a0] MSI-X: Enable+ Count=8 Masked- >> >> 2. Device VF configuration change happens with no reset > > Is this an out of tree driver problem? > > The intree way to alter the MSI configuration is via > sriov_set_msix_vec_count, and there is only one in-tree driver that > uses it right now. Right, but that only addresses the driver specific issues. > If something is going wrong here it should be fixed in the > sriov_set_msix_vec_count() machinery, possibly in the pci core to > synchronize the msi_domain view of the world. Right, we should definitely not do that on a per driver basis. Thanks, tglx