Received: by 2002:a05:7412:37c9:b0:e2:908c:2ebd with SMTP id jz9csp415611rdb; Mon, 18 Sep 2023 21:44:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFpIzaGOuO2hThD+X2W5rKUG5VkNTHNwCT3NFktcy09wKDx1kd7R626IR0MblWKXKM2BL/C X-Received: by 2002:a05:620a:1918:b0:773:d17c:661b with SMTP id bj24-20020a05620a191800b00773d17c661bmr5466004qkb.56.1695098671863; Mon, 18 Sep 2023 21:44:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695098671; cv=none; d=google.com; s=arc-20160816; b=vAIBS2ZUR1knHLIF7vf7OXmXYk/SGq98WiBWQXjT7d9h04UXqVXq5yzk1nHyqJQpJ8 ES3QYos1BgAqVT7FEUonJ5BwQjfxXSsREWVRH8hUKIqXuGAErQq9JepADyoC8AziTWFO dJWsrb6N7EXCRjwQTsXLgLovG5Wq+sGi2A2A6MVOb/EsQMYKcugYLGb29LPgA2b5zYUJ y6WiuhWB1Rn7uZimrfkpSmpqxdOhxyTgFdXKwaeeIrDxJJKzgKP65LGLENndAItlSWrE uSqoKia0Joknfyn+e2s93Su5ZzgpnvdO4YMOjgU8LI5jSVuGv3P76fEkUspKPUoOxpQY NDkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=WnuAQjafo4kC6lltDk9NFMHIoXmaRJB2ByVDtL0sp2g=; fh=+i2Z+Bpp/5N3ZNaHkNF2Yd/3QhJpNwPGjPT1YqjPH7U=; b=qYQmO3+q2xSHIsXhBW3X2fIokkoDhvBXCDQlJTrpexuYRmJEf/6SlHnYbiOy2JAcEh COuLmZIQ3vqV1X+5OpkCB+rfWz2zRZaOBE5PxTFSqaePuLHPb1pAfm3PYXtFMFSBwlNC R/L12GiJh92yfss6x1+EhAX2q/efLDgAmiIih4ZdDvmc9pfSVXOM2SImLqFTreE12yFm +R2nYoaikfnoBfSJzGTSbgjAupnYzwtILwfys+vRous/SYLa3lVilrFpAAboUYXoAB+X B6dhWoDtHkQ3YjQg6DHyD3jWDQ+CRtCW2L8J4WcIPslVqpODC+owtp09pCRYv72MzVhW L+VQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=AC8RRGIo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id l24-20020a637018000000b0056336fb99aasi785866pgc.850.2023.09.18.21.44.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Sep 2023 21:44:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=AC8RRGIo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id F3988823367C; Mon, 18 Sep 2023 16:37:44 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230221AbjIRXho (ORCPT + 99 others); Mon, 18 Sep 2023 19:37:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51938 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229436AbjIRXhm (ORCPT ); Mon, 18 Sep 2023 19:37:42 -0400 Received: from mail-qv1-xf2a.google.com (mail-qv1-xf2a.google.com [IPv6:2607:f8b0:4864:20::f2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5010890 for ; Mon, 18 Sep 2023 16:37:37 -0700 (PDT) Received: by mail-qv1-xf2a.google.com with SMTP id 6a1803df08f44-65627d49d24so18716876d6.2 for ; Mon, 18 Sep 2023 16:37:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1695080256; x=1695685056; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=WnuAQjafo4kC6lltDk9NFMHIoXmaRJB2ByVDtL0sp2g=; b=AC8RRGIoI88WRFlirO+mRdZr1/Hkw7YoBqdRGjxaYE4TOZ5VCj2yxVVUI8nkGDsnrV me3S42dW6uVZDkJtr0sZAgou2QZ2wf9n4vYYpbfqtRU9HLmqjruz76zbmZG8gfTEYx+r mvOFczvVon16xQLasFZZ+ae9JW8JcKnCN4pxUkvMToPb9iF+a9qbELk30daNcYAK3WEL t81HogPSRUajDAv4wsE19QwlhXg6r3B06+S6gZlSHL80OKa6Dv2PjFZxtrAsERk+qfWC /qKJ4SNn5PPJwxAddGEO7uwArYV3F4if2UtyldKmCV5FfBIHXQUnVmKaM6TTrdMjE5F8 x4Ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695080256; x=1695685056; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=WnuAQjafo4kC6lltDk9NFMHIoXmaRJB2ByVDtL0sp2g=; b=kNJ7TeEwlKpVweq1Mobydv1sDNEJE/YcmRunfw/UEo7aBpPDhxvPwDfSBrOSztqAGp rhuXlha7TPPTFr4woeJ6YosMNtDF1FRDdFdPPDqX9pEPgqL7kXCumr2PVOPQ10nIwmzk ZP0VleAd7cyMoFNiGroADdzBzOUUPLAAe4ss3qnBOd3ROmakzpP1DisvKurUxKglpUu4 x6dIXAg1NMwuzFyxcs7p3KRd/5JfqU6DX2m3gbThrz4WgTh+tg7q3nEJRRg/75Gqdb2C 1+RrdYf4TPlVEE3JkF7adC3IdazVhDbI4ofHbj4VVAQDPTRoYB5vL+9VVccHxbTasLlN z0bg== X-Gm-Message-State: AOJu0YwD0Eo6ybaR+MEOQ8CR7Of3iWHITxh/9a0HgmfaneoQkI72qomU m/WwB38+kCJtNIZl14ShARAy2w== X-Received: by 2002:a05:6214:92d:b0:656:2ccf:5aa1 with SMTP id dk13-20020a056214092d00b006562ccf5aa1mr7799072qvb.45.1695080256461; Mon, 18 Sep 2023 16:37:36 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-26-201.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.26.201]) by smtp.gmail.com with ESMTPSA id m19-20020a0cf193000000b0064f364f3584sm3823816qvl.97.2023.09.18.16.37.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Sep 2023 16:37:35 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1qiNo7-0007rC-FO; Mon, 18 Sep 2023 20:37:35 -0300 Date: Mon, 18 Sep 2023 20:37:35 -0300 From: Jason Gunthorpe To: Thomas Gleixner Cc: Shannon Nelson , alex.williamson@redhat.com, kevin.tian@intel.com, reinette.chatre@intel.com, kvm@vger.kernel.org, brett.creeley@amd.com, linux-kernel@vger.kernel.org, Leon Romanovsky Subject: Re: [PATCH vfio] vfio/pci: remove msi domain on msi disable Message-ID: <20230918233735.GP13795@ziepe.ca> References: <20230914191406.54656-1-shannon.nelson@amd.com> <20230918141705.GE13795@ziepe.ca> <87led3xqye.ffs@tglx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87led3xqye.ffs@tglx> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 18 Sep 2023 16:37:45 -0700 (PDT) On Mon, Sep 18, 2023 at 08:43:21PM +0200, Thomas Gleixner wrote: > On Mon, Sep 18 2023 at 11:17, Jason Gunthorpe wrote: > > On Thu, Sep 14, 2023 at 12:14:06PM -0700, Shannon Nelson wrote: > >> The new MSI dynamic allocation machinery is great for making the irq > >> management more flexible. It includes caching information about the > >> MSI domain which gets reused on each new open of a VFIO fd. However, > >> this causes an issue when the underlying hardware has flexible MSI-x > >> configurations, as a changed configuration doesn't get seen between > >> new opens, and is only refreshed between PCI unbind/bind cycles. > >> > >> In our device we can change the per-VF MSI-x resource allocation > >> without the need for rebooting or function reset. For example, > >> > >> 1. Initial power up and kernel boot: > >> # lspci -s 2e:00.1 -vv | grep MSI-X > >> Capabilities: [a0] MSI-X: Enable+ Count=8 Masked- > >> > >> 2. Device VF configuration change happens with no reset > > > > Is this an out of tree driver problem? > > > > The intree way to alter the MSI configuration is via > > sriov_set_msix_vec_count, and there is only one in-tree driver that > > uses it right now. > > Right, but that only addresses the driver specific issues. Sort of.. sriov_vf_msix_count_store() is intended to be the entry point for this and if the kernel grows places that cache the value or something then this function should flush those caches too. I suppose flushing happens implicitly because Shannon reports that things work fine if the driver is rebound. Since sriov_vf_msix_count_store() ensures there is no driver bound before proceeding it probe/unprobe must be flushing out everything? Jason