Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 1 Jul 2022 15:22:38 -0700
From:   Ira Weiny <ira.weiny@intel.com>
To:     Jonathan Cameron <Jonathan.Cameron@huawei.com>
CC:     Dan Williams <dan.j.williams@intel.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        "Li, Ming" <ming4.li@intel.com>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Lukas Wunner <lukas@wunner.de>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        "Dave Jiang" <dave.jiang@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        <linux-kernel@vger.kernel.org>, <linux-cxl@vger.kernel.org>,
        <linux-pci@vger.kernel.org>
Subject: Re: [PATCH V12 3/9] PCI: Create PCIe library functions in support of
 DOE mailboxes.
Message-ID: <Yr9zroBeSLfWyp5U@iweiny-desk3>
References: <20220628041527.742333-1-ira.weiny@intel.com>
 <20220628041527.742333-4-ira.weiny@intel.com>
 <20220628151626.00001c0f@Huawei.com>
 <YrtGcNWcmzumj2gA@iweiny-desk3>
 <20220629150947.000012be@Huawei.com>
 <Yr0nysRl7x49me1K@iweiny-desk3>
 <20220630162540.00002910@Huawei.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20220630162540.00002910@Huawei.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?4EnkC/zDrik2PPQsXZFlGlZrk85fBXDospmRrtS7nlUlFlYXxcCMGxvFN9ZL?=
 =?us-ascii?Q?+qfPwRGf6DtwQNAKqkEC+2jBM4Ao78QVjGqvseDzrJ9sJucjJTa0cQXKDvTM?=
 =?us-ascii?Q?byaO8w02gx4tgUgOGISOSoKvX7hWfp/UIh8J4YcLBAcRHcNuLWQgUETxIPMt?=
 =?us-ascii?Q?fL1a6ZtYGhnGfPuUmNhofBd67HplTpwLNe2kTzx9D8DMJgmC976X+Nkw/2Za?=
 =?us-ascii?Q?sQSZw4mZ1E203kfMjL4FYorBQKx01Pr0gW3P3csEbotheSsnraoZulQ7EFmQ?=
 =?us-ascii?Q?8KVs5cRYh6qD1SZOWJxPwxqd0yNssOpfU8WD6oDMMAAH412zkjPZXczWWoDi?=
 =?us-ascii?Q?TZ8sTvI9KJ4pKPE0dkcMBcW2/UycYbefZkUAG7ELz8cOVkHVocw+O/Ckmr5V?=
 =?us-ascii?Q?8cMUixrIub1FiJGP+Nngd9c30h3LyoW8PmdXbrxmKjWJQypWfjKMTAmPkaOu?=
 =?us-ascii?Q?aOZJnpeLRNpGdgLWGFWhH9sRCHmNGUNWKlVuAhY8JjDI1d+wgmMG5DWJvk0n?=
 =?us-ascii?Q?3lMJBnVkZRRvG6VuTbu4EedhIsneiQb97d2VzYiGJ0WRPylGcqOBfMl8JGjh?=
 =?us-ascii?Q?GdahoSnaGm/EEFWel0UWXeliuuYdnTu0NXmS5NO19UFy0SVbCszB86GshKzK?=
 =?us-ascii?Q?KMuKodZNm0OXSN6oYWdW9qZ+J4d/FUKv74k2Iyk/GBbKGPqhsukE6yPg2dxK?=
 =?us-ascii?Q?alhgfnQDjKQnFlZNR/QT/w/FqDrFg9nsvbfUOw0iQJvy+ZH3o8F7Z+cEr/U5?=
 =?us-ascii?Q?qOww2M2MpP59U3LJuSIoT5WaVjxJY7KihzwEhbLyNFOcJk0U9wB0MVbKaFCI?=
 =?us-ascii?Q?sOrKSqs0Ng6kLXmnkZXAjukFt4rps6TH7aaMmOVuMXtTJQsEKDvIolgMDAQ3?=
 =?us-ascii?Q?LhOx5rag8kqk2UKTR3w8R5NZzKoIxWBEHrXJWHDy1xAUaHl1OoVvWoDrtE6C?=
 =?us-ascii?Q?in3DZitwz20jatAjpfyS6ciMu+mG04C74DXInuN/maZw1WcG6X7IMx9gQR2a?=
 =?us-ascii?Q?+e4jQvl4msnyp3rhqjXw9Pbh4GitHzBYkLTPgmoTIpQ7WR676VwWMYGfO09+?=
 =?us-ascii?Q?/z1oGvd3I1B49nIA+Mu7zkj7bN3J0WGOmAXnswiCmPsY7TwSbkZr8ZCv755O?=
 =?us-ascii?Q?rPigVU06LVqDj5XFeD4Z5UlkeKToh7coJx6JE0LbLDPwhvZGqknIpTg+1V8q?=
 =?us-ascii?Q?Ac2+UQtXiQHwBQ6+h1KephLOkSnpIcYHt6zXIKl79rH6eRpmssU5a03mgRQV?=
 =?us-ascii?Q?xCT2HbJ0BNCTk6TxFfRQJ9gSsRG0Dvm7FrfEme7/Xk84FW3SRhJE4xjV53G5?=
 =?us-ascii?Q?i+Xt3WpXEshqK3XUeCX+mVF2W72J6nELPN4lU8PootKzVQX2skgg8pJRKpS+?=
 =?us-ascii?Q?F5/2w9EktP7Sivs5BD/Z5O600yTbD2TvOVREYRIh532LUfZIe/5qfG0LUTV2?=
 =?us-ascii?Q?jSM+/DjU7Nujj/2EV/bfXOu4dTtQD3W+dyZ0zga60iKkt2bpF5SschNN0OMg?=
 =?us-ascii?Q?6nz0UiD6OYjGTLADr6/dJMVP/xhzhs/Td6pp7yGcFIhH8+Z7mVDLe8xnVTGQ?=
 =?us-ascii?Q?zQgkpKjn7oSdZedfudh4YAx+FC928gh+2lx4fU0n/V4RjhSm7PmN/REbBkut?=
 =?us-ascii?Q?LZ06CiT4RJlVciLm3I95pxyCN3dp6ePvCswGCJZEhHBh?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ad5f32d7-285c-4777-3c30-08da5bb03c82
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6311.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Jul 2022 22:22:51.5304
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 4CNm9sNl8DNURSu5jazSOdxJTb3rkVSvCb7/IVgxE0QAsRBHkcGVFA4dpeCF8xCDZXYVvHsWj5518ogX/6x5rg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR11MB3107
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Jun 30, 2022 at 04:25:40PM +0100, Jonathan Cameron wrote:
> On Wed, 29 Jun 2022 21:34:18 -0700
> Ira Weiny <ira.weiny@intel.com> wrote:
> 

[snip]

I've dropped the IRQ support and was polishing things up.  Without the IRQ I
don't think any 'arming' makes sense.

However, in working through the sequence again I think I found another problem.
I _think_...  :-/

> > 
> > But we are only going to see this if some other entity is using the mailbox
> > right?  And I don't think that is going to be common, is it?
> 
> BUSY on entry to doe_statemachine_work() is indeed only relevant if
> some other entity is trampling on us. It's best effort only.
> 
> BUSY during normal flow is the one I care about.
> In most cases it will go like (assuming we clear the int status in the handler as now)
> 
>              Send Object
> BUSY      ________|-----___________________
> PROC      ________|------------------______
> OBJ RDY   ___________________________-------
> Int Status______________-____________-_____

So I did not realize that BUSY could clear like this.  I thought the point of
BUSY was to indicate someone else had an exchange in flight.

What happens if another entity jumps in during the PROC time?  How does one
know that OBJ RDY is _our_ object ready and not someone else's?

For example 'entity' issues a send, we see busy clear and also start a
send.  But the device is still processing the send from 'entity':

            Send Object(entity) Send Object (Linux)
BUSY      ___|----_______________|---______________________________
PROC      ___|-----------------------------___|-----------------___
OBJ RDY   _________________________________-------______________---
Int Status________-__________________-_____-____________________-__

                                          ^^^
					  This...

... is _not_ Linux's object!?!?!?!

Can that happen?

If so this is entirely broken.  Even Polling OBJ RDY will break.  And worse yet
we will not even see BUSY being set in any 'abnormal' way.

> 
> where I've added PROC to mean the device is processing the data.
> Once it clears the input buffer on the device and hence the device can accept
> another protocol request BUSY will drop.  If device has some pipelining
> or runs multiple protocols in different threads, you can think of that busy
> period just being the time it needs to copy out the request to some protocol
> thread specific storage.

BUSY was not at all doing what I thought it did.  I'm now concerned that it is
completely broken WRT to other entities even without IRQs.  Frankly I'm
confused why pci_doe_send_req() even checks for busy because it is unlikely
that we will ever see it set.  For sure we won't from our side because the
workqueue is going to process one task at a time.

If Linux wanted to have multiple objects in flight I think we would need a much
more complex state machine than we had.  Maybe your original state machine
handled this.  If so, I apologize for missing this subtle point.

At this point I'm debating removing the check for BUSY as well because I don't
see the point.  (Other than maybe flagging some error to say that 'entity' may
be messing things up for us and bailing.)

Thoughts?
Ira

> 
> You won't see this in QEMU without extra hacks because we shorten the
> flow so that whole thing is instantaneous.
> 
> If those two interrupts per transfer occur well spread out they can result in
> your INT flag being set too often and some of the waits dropping through early.
> 
> It will 'work' I think though because you ultimately spin on Data object
> ready which won't be set until after the second interrupt.
>