Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Date:   Wed, 1 Jun 2022 19:56:47 +0200
From:   Lukas Wunner <lukas@wunner.de>
To:     Ira Weiny <ira.weiny@intel.com>
Cc:     Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Christoph Hellwig <hch@infradead.org>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        Ben Widawsky <ben.widawsky@intel.com>,
        linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org,
        linux-pci@vger.kernel.org
Subject: Re: [PATCH V8 03/10] PCI: Create PCI library functions in support of
 DOE mailboxes.
Message-ID: <20220601175647.GA21509@wunner.de>
References: <20220414203237.2198665-1-ira.weiny@intel.com>
 <20220414203237.2198665-4-ira.weiny@intel.com>
 <20220530190657.GA14765@wunner.de>
 <20220531113350.0000421e@Huawei.com>
 <YpbWCYujYDEkMm1B@iweiny-desk3>
 <20220601071808.GA19924@wunner.de>
 <Ypee328j+l6ZdbUT@iweiny-desk3>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Ypee328j+l6ZdbUT@iweiny-desk3>
User-Agent: Mutt/1.10.1 (2018-07-13)
Precedence: bulk

On Wed, Jun 01, 2022 at 10:16:15AM -0700, Ira Weiny wrote:
> On Wed, Jun 01, 2022 at 09:18:08AM +0200, Lukas Wunner wrote:
> > You only need to re-check the Data Object Ready bit on the last-but-one
> > dword in case the function was reset concurrently.  Per sec. 6.30.2,
> > "An FLR to a Function must result in the aborting of any DOE transfer
> > in progress."
> 
> I think I disagree.  Even if we do that and an FLR comes before the last read
> the last read could be 0's.

PCIe r6.0, Table 7-316 says:

  "If there is no additional data object ready for transfer, the
   DOE instance must clear this bit after the entire data object has been
   transferred, as indicated by software writing to the DOE Read Data
   Mailbox Register after reading the final DW of the data object."

Remember that you *read* a dword from the mailbox and then acknowledge
reception to the mailbox by *writing* a dword to the mailbox.

So you check that the Data Object Ready bit is set before acknowledging
the final dword with a register write.  That's race-free.

(I realize me talking about the "last-but-one dword" above was quite
unclear, sorry about that.)

Thanks,

Lukas