Received: by 2002:a05:7412:37c9:b0:e2:908c:2ebd with SMTP id jz9csp447373rdb; Mon, 18 Sep 2023 23:20:20 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFjDivUXYpFP/xyQ5GkDNwdT5ejtBHHhPs5B1WGVTEXPC6Izz2jGIVVm8RxAL9orKqv5Nxz X-Received: by 2002:a05:6358:e49b:b0:140:e7b2:e482 with SMTP id by27-20020a056358e49b00b00140e7b2e482mr9868051rwb.5.1695104420102; Mon, 18 Sep 2023 23:20:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695104420; cv=none; d=google.com; s=arc-20160816; b=KdrBDNGVmBk+I/k8v3wjUSevilkKK4E0E7fo+plbzzoFpnKSAMhbRL7id87s71Wjdc bUFEuJ35avDjBf5rpBX9ArufWTOTvw3rR2ax8LSoZtWj4v+MFeYcJyQh/1CWSoycqfb/ /nee6i24mFiBas5ho4AkSdALF2RLV6T5U9frZqD0R2QP7qu2HriG5UAFWFamdlUQlUUf B3MTpu1oUwUKrRCoi+3ky9rZEn1IXl3+OtklFwcWk93rMmXYgmMqxffnqZr887SvCptM EKZ7naq9HEL38qkvW9N8BzMZAFm+gZZgphg1JGSqViG9H8yPppfCK0EvWS0jqCyPx3Uv 7bKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:message-id:date:references :in-reply-to:subject:cc:to:dkim-signature:dkim-signature:from; bh=zoPrsSaT29FlvfJy1a7+Tv5EMLFoI3yK58qV6pzrQU0=; fh=sv9TbEkfvjSw6GeHZfiJBrLlmVUXPWGhP8Y1ju690VY=; b=thxhJo+KnToOM2K/1uQ94E2eG+SGHBu0wFZpJpjFTCW0bXqZsFJsvpVLg4p7vt9wDB eUf7TtrLB4j55jX+wvd++T1MjGIXBp2Lspv7RSyb0BSPVbJqXJHBap3TaU+q7u2y32qu Gw6h9rNT7mf9Ru6TiRlru2Vsw7kpC1xXK34ZPRctH+LYILyTzV4Py/Pc0jzzeqYeMHbz OiMArkq167xlSWV/NQot8rg2vXE1S+pc4TqLHpSY7eNsNChzYO1Tk6TIfBKwBWzt35d0 4w3t6e0S8Xtez4LZkf6UaCg44T1l+rzWP0w4uYhZkYWSmEpu8fOFLiZ14aMZhnJs6u6A 3gfA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=PcNpWXyR; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Return-Path: Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id g186-20020a636bc3000000b00565550ba905si9113715pgc.622.2023.09.18.23.20.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Sep 2023 23:20:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=PcNpWXyR; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 50BCC8096A44; Mon, 18 Sep 2023 16:47:49 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230209AbjIRXrp (ORCPT + 99 others); Mon, 18 Sep 2023 19:47:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229436AbjIRXrp (ORCPT ); Mon, 18 Sep 2023 19:47:45 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 81D4890; Mon, 18 Sep 2023 16:47:39 -0700 (PDT) From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1695080858; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=zoPrsSaT29FlvfJy1a7+Tv5EMLFoI3yK58qV6pzrQU0=; b=PcNpWXyRLqtfgKxGwS3RjJmw8mKIoehQ28WqW/nSnu8PcULPs/9bgk/U5Wv8T2RFOmcxBw G8JOscsI6gyYAnfk9AkLfyYe8nrO/+UBS+yuHoW1XT++/o49JTHrhzz0Btuxgmi1UyRwCN ncvBBvqmaYf5L32IIvbTFSgXtQHp0+PaKpp+68E02RV6Cw/JsHgkNPcX0VdfLpdz3NWw9u 4FbcGo2QCEdxAwVIyWtT7sonOeNvMPN8UcJuE9kh749VsYOtfOnMecslg8gRTCKUAQiJEa xc97v9uQHYWx5vk2eB0ZbDxs4mv2BG1gPAkZmk/elPSZmWBgw0tJfHDaN4wRww== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1695080858; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=zoPrsSaT29FlvfJy1a7+Tv5EMLFoI3yK58qV6pzrQU0=; b=KrDZUpg4xWw9c5L6hoeA9PpnHq/wBo4OOWV+Y48eEgzi4zdat8V0DRpd+O/vVeOrXbzam6 rhwYgn4l9dm1yjCw== To: Jason Gunthorpe Cc: Shannon Nelson , alex.williamson@redhat.com, kevin.tian@intel.com, reinette.chatre@intel.com, kvm@vger.kernel.org, brett.creeley@amd.com, linux-kernel@vger.kernel.org, Leon Romanovsky Subject: Re: [PATCH vfio] vfio/pci: remove msi domain on msi disable In-Reply-To: <20230918233735.GP13795@ziepe.ca> References: <20230914191406.54656-1-shannon.nelson@amd.com> <20230918141705.GE13795@ziepe.ca> <87led3xqye.ffs@tglx> <20230918233735.GP13795@ziepe.ca> Date: Tue, 19 Sep 2023 01:47:37 +0200 Message-ID: <87a5tjxcva.ffs@tglx> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Mon, 18 Sep 2023 16:47:49 -0700 (PDT) On Mon, Sep 18 2023 at 20:37, Jason Gunthorpe wrote: > On Mon, Sep 18, 2023 at 08:43:21PM +0200, Thomas Gleixner wrote: >> On Mon, Sep 18 2023 at 11:17, Jason Gunthorpe wrote: >> > On Thu, Sep 14, 2023 at 12:14:06PM -0700, Shannon Nelson wrote: >> >> The new MSI dynamic allocation machinery is great for making the irq >> >> management more flexible. It includes caching information about the >> >> MSI domain which gets reused on each new open of a VFIO fd. However, >> >> this causes an issue when the underlying hardware has flexible MSI-x >> >> configurations, as a changed configuration doesn't get seen between >> >> new opens, and is only refreshed between PCI unbind/bind cycles. >> >> >> >> In our device we can change the per-VF MSI-x resource allocation >> >> without the need for rebooting or function reset. For example, >> >> >> >> 1. Initial power up and kernel boot: >> >> # lspci -s 2e:00.1 -vv | grep MSI-X >> >> Capabilities: [a0] MSI-X: Enable+ Count=8 Masked- >> >> >> >> 2. Device VF configuration change happens with no reset >> > >> > Is this an out of tree driver problem? >> > >> > The intree way to alter the MSI configuration is via >> > sriov_set_msix_vec_count, and there is only one in-tree driver that >> > uses it right now. >> >> Right, but that only addresses the driver specific issues. > > Sort of.. sriov_vf_msix_count_store() is intended to be the entry > point for this and if the kernel grows places that cache the value or > something then this function should flush those caches too. Sorry. What I wanted to say is that the driver callback is not the right place to reload the MSI domains after the change. > I suppose flushing happens implicitly because Shannon reports that > things work fine if the driver is rebound. Since > sriov_vf_msix_count_store() ensures there is no driver bound before > proceeding it probe/unprobe must be flushing out everything? Correct. So sriov_set_msix_vec_count() could just do: ret = pdev->driver->sriov_set_msix_vec_count(vf_dev, val); if (!ret) teardown_msi_domain(pdev); Right? Thanks, tglx