Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Thomas Gleixner <tglx@linutronix.de>
To:     "Tian\, Kevin" <kevin.tian@intel.com>,
        Alex Williamson <alex.williamson@redhat.com>
Cc:     Jason Gunthorpe <jgg@nvidia.com>,
        "Dey\, Megha" <megha.dey@intel.com>,
        "Raj\, Ashok" <ashok.raj@intel.com>,
        "Pan\, Jacob jun" <jacob.jun.pan@intel.com>,
        "Jiang\, Dave" <dave.jiang@intel.com>,
        "Liu\, Yi L" <yi.l.liu@intel.com>,
        "Lu\, Baolu" <baolu.lu@intel.com>,
        "Williams\, Dan J" <dan.j.williams@intel.com>,
        "Luck\, Tony" <tony.luck@intel.com>,
        "Kumar\, Sanjay K" <sanjay.k.kumar@intel.com>,
        LKML <linux-kernel@vger.kernel.org>, KVM <kvm@vger.kernel.org>,
        Kirti Wankhede <kwankhede@nvidia.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Marc Zyngier <maz@kernel.org>,
        Bjorn Helgaas <helgaas@kernel.org>
Subject: RE: Virtualizing MSI-X on IMS via VFIO
In-Reply-To: <87o8buuyfy.ffs@nanos.tec.linutronix.de>
References: <MWHPR11MB1886E14C57689A253D9B40C08C079@MWHPR11MB1886.namprd11.prod.outlook.com> <8735t7wazk.ffs@nanos.tec.linutronix.de> <20210624154434.11809b8f.alex.williamson@redhat.com> <BN9PR11MB5433063F826F5CEC93BCE0E38C069@BN9PR11MB5433.namprd11.prod.outlook.com> <87o8buuyfy.ffs@nanos.tec.linutronix.de>
Date:   Fri, 25 Jun 2021 23:19:42 +0200
Message-ID: <87czs9vdzl.ffs@nanos.tec.linutronix.de>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: bulk

On Fri, Jun 25 2021 at 10:43, Thomas Gleixner wrote:
> On Fri, Jun 25 2021 at 05:21, Kevin Tian wrote:
>>> From: Alex Williamson <alex.williamson@redhat.com>
>>> So caching/latching occurs on unmask for MSI-X, but I can't find
>>> similar statements for MSI.  If you have, please note them.  It's
>>> possible MSI is per interrupt.
>>
>> I checked PCI Local Bus Specification rev3.0. At that time MSI and
>> MSI-X were described/compared together in almost every paragraph 
>> in 6.8.3.4 (Per-vector Masking and Function Masking). The paragraph
>> that you cited is the last one in that section. It's a pity that MSI is
>> not clarified in this paragraph but it gives me the impression that 
>> MSI function is not permitted to cache address and data values. 
>> Later after MSI and MSI-X descriptions were split into separate 
>> sections in PCIe spec, this impression is definitely weakened a lot.
>>
>> If true, this even implies that software is free to change data/addr
>> when MSI is unmasked, which is sort of counter-intuitive to most
>> people.
>
> Yes, software is free to do that and it has to deal with the
> consequences. See arch/x86/kernel/apic/msi.c::msi_set_affinity().

Well, it's actually forced to do so when the MSI implementation does not
provide masking. If masking is available then it should be used also for
MSI as it prevents the nasty update problem where the hardware can
observe inconsistent state... Which x86 does correctly except for that
startup issue you spotted, but I'm not at all sure about the rest.

The startup issue is halfways trivial to fix I think, but there are
other issues with the PCI/MSI-X handling which I discovered while
staring at that code for a while. Still working on it.

Thanks,

        tglx