Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 23 Nov 2022 09:35:37 -0800
From:   Ira Weiny <ira.weiny@intel.com>
To:     Jonathan Cameron <Jonathan.Cameron@huawei.com>
CC:     Dan Williams <dan.j.williams@intel.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Lukas Wunner <lukas@wunner.de>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        Gregory Price <gregory.price@memverge.com>,
        "Li, Ming" <ming4.li@intel.com>, <linux-pci@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <linux-cxl@vger.kernel.org>
Subject: Re: [PATCH V2 1/2] PCI/DOE: Remove the pci_doe_flush_mb() call
Message-ID: <Y35Z6fB8Sm4JcPdK@iweiny-desk3>
References: <20221122155324.1878416-1-ira.weiny@intel.com>
 <20221122155324.1878416-2-ira.weiny@intel.com>
 <20221122163426.0000467c@Huawei.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20221122163426.0000467c@Huawei.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?B7X5lQAFkhznxvI6GroITbNvfAZEwKgF6SGjSI59uflu3nOiBx0vxdHZm4Tq?=
 =?us-ascii?Q?odCl0txEdOevemOunzseZuD/0E3aDfK+Gviegwow1+KfUi69jXgV1W7YfvIm?=
 =?us-ascii?Q?w3EerAIp/QwOq+f92iAP3AHwqoDTsBgff+4rtKOStAky63nkSF+HM6TbjANE?=
 =?us-ascii?Q?RKdO3k+rer/+bUIV5aAZOKsMNC8gfsFxFkZmW12Z1dmy/jaUz7jThpA7rlim?=
 =?us-ascii?Q?TUin+ZuudZgu4Kh8VgBtDZEPNNqdzhv5iPaO3UtuOZRd9k3Ad5eJEurLnZEc?=
 =?us-ascii?Q?r3J6yrZCWj9kRvosK/1AkRu1hsQMpLN24eltNsX0hwYWtSH75cCag+fQuhXi?=
 =?us-ascii?Q?vY0OtvjqBgZIgS5kAmwlS0HRKDeOc4S7P0VEmQuF4h+LAQzS2LcmvdBozooC?=
 =?us-ascii?Q?82PCSotbJ9OX5qaC+4eNfjrJ7Gbqs51a+umgvDG1hd2vnJGXszilZyQoRhvX?=
 =?us-ascii?Q?2iLqgnJrdMXZxLMM55JZO44wHY/TUV1dsc3ZIDAyP47V96piCXpj411KExhH?=
 =?us-ascii?Q?KAKlxpqcIQowiiXj8GVlTmJzTPZZxOPQy1JFhfk1GrCPAUlWYoALfaglT9jc?=
 =?us-ascii?Q?bj3TPX5DiZN/SouGy3HAd+v/HMZsmCFUCn0WhkYdfoUTTTOca+n+iSuU+zKp?=
 =?us-ascii?Q?kdqhEZtGCZVXxtuEffUWgxUDK45phqpSRH3U2sbnbZ0AP/fCJtkri9v4kalZ?=
 =?us-ascii?Q?9nKN00+3Sw1dURBt3zRXaW4oHTHVM20buvWkpMKujf4byWxbWDQD7IwTUGHb?=
 =?us-ascii?Q?f1D64jnvP4hm0hoDEgno9p4ounZQszKaWO30WF7zYePyOBMqq1kaBsG/dAoW?=
 =?us-ascii?Q?7TdSUqPdlRCabnsRTgl6nZTNFOFjNqjjkbhVedZ5PPG07R0p8+rTNwHB4jsN?=
 =?us-ascii?Q?aWhRAeW2JfAmt1l/ppZPQEDrPffQnmMAmsz6h38Kot/J6N73f4VGGFguCIbm?=
 =?us-ascii?Q?mWvZpOlivK6p1bO6R/AyNWr1MB5zPtZ021CH89VLFVlhq41RQ/fcXEZfppDt?=
 =?us-ascii?Q?GZt1ucQRwrYgYxsnpQlPtFmhnWxaiy25hNr7ULPW+TdzTAdTeFme/RWLRXYy?=
 =?us-ascii?Q?KLm8YMfqpxpN1brkbos0eC8Q9CC20mCSPYTESpZhSuYjVNtBukf7srEk7xLU?=
 =?us-ascii?Q?gLiJl95cp+CW6e3IX/NqOjJKsiol1FEhGvz+ctLZ8shGsrltnWsA5+J0Xkno?=
 =?us-ascii?Q?KYIQY9ck+W7qlGtIAJGeYYpqXhvWuRqy5BnqPv4LKfeEDmH4CpNu1cFgRwFV?=
 =?us-ascii?Q?5s72GHIszSEePrmXYLBA5EQ5MQejCXgf7B+QKP8MgYVeIdN44JjyUz5E6S5E?=
 =?us-ascii?Q?N+nXk2t67Hae3VfoZcHUit7fM40zWup7a62Z9pMTWVgtfjMTLpq3ubpnpeG/?=
 =?us-ascii?Q?QeOdOD0l/JnGlVWRWZNQZZR/K9rmDZM2SixljXzNhJ9tJng/j0iuwFXtNWrI?=
 =?us-ascii?Q?+2P4KX8V5srmKUrcxM6wlc5a9tEBXL3o8q5Dfj2dI+aJaoNc7I7C+l6X88hc?=
 =?us-ascii?Q?3zSIL/Mv8SQ6MTDwyBBEcYYgW+JpEB19A2lPgCi0R8gMO1igqoqhozZFi+wP?=
 =?us-ascii?Q?MmyfQR1yJMAsXIA0EldTX1E+UX7jal15AhBoRW5c?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 8cb47da8-9624-4e5a-ae89-08dacd792535
X-MS-Exchange-CrossTenant-AuthSource: SA1PR11MB6733.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Nov 2022 17:35:42.7858
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: r15XHWEyRXBziik57OBdeUb3qexkPAtgjhT/60muxXrgDkQF4nUAzqrp2YDbZScjaf1gsWvWBNkslaqgTYaJfA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB6468
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Nov 22, 2022 at 04:34:26PM +0000, Jonathan Cameron wrote:
> On Tue, 22 Nov 2022 07:53:23 -0800
> ira.weiny@intel.com wrote:
> 
> > From: Ira Weiny <ira.weiny@intel.com>
> > 
> > Each struct doe_mb is managed as part of the PCI device.  They can't go
> > away as long as the PCI device exists.  pci_doe_flush_mb() was set up to
> > flush the workqueue and prevent any further submissions to the mailboxes
> > when the PCI device goes away.  Unfortunately, this was fundamentally
> > flawed.  There was no guarantee that a struct doe_mb remained after
> > pci_doe_flush_mb() returned.  Therefore, the doe_mb state could be
> > invalid when those threads waiting on the workqueue were flushed.
> > 
> > Fortunately the current code is safe because all callers make a
> > synchronous call to pci_doe_submit_task() and maintain a reference on the
> > PCI device.
> > 
> > For these reasons, pci_doe_flush_mb() will never be called while tasks
> > are being processed and there is no use for it.
> > 
> > Remove the dead code around pci_doe_flush_mb().
> > 
> > Signed-off-by: Ira Weiny <ira.weiny@intel.com>
> 
> Looks fine I think, though one question inline.
>  
> > ---
> >  drivers/pci/doe.c | 48 ++++-------------------------------------------
> >  1 file changed, 4 insertions(+), 44 deletions(-)
> > 
> > diff --git a/drivers/pci/doe.c b/drivers/pci/doe.c
> > index e402f05068a5..260313e9052e 100644
> > --- a/drivers/pci/doe.c
> > +++ b/drivers/pci/doe.c
> > @@ -24,10 +24,9 @@
> >  
> >  /* Timeout of 1 second from 6.30.2 Operation, PCI Spec r6.0 */
> >  #define PCI_DOE_TIMEOUT HZ
> > -#define PCI_DOE_POLL_INTERVAL	(PCI_DOE_TIMEOUT / 128)
> > +#define PCI_DOE_POLL_INTERVAL	8
> 
> Why this change?  

msleep_interruptible() takes a millisecond value and wait_event_timeout() takes
jiffies.  1/128 of a second is ~8ms.

While for most configs (HZ == 1000) the value does not change.  I don't believe
this would be true for all configs.  Thus a more explicit define.

I'll add a comment.

Ira

> 
> >  
> > -#define PCI_DOE_FLAG_CANCEL	0
> > -#define PCI_DOE_FLAG_DEAD	1
> > +#define PCI_DOE_FLAG_DEAD	0
> >  
> >  /**
> >   * struct pci_doe_mb - State for a single DOE mailbox
> > @@ -53,15 +52,6 @@ struct pci_doe_mb {
> >  	unsigned long flags;
> >  };
> >  
> > -static int pci_doe_wait(struct pci_doe_mb *doe_mb, unsigned long timeout)
> > -{
> > -	if (wait_event_timeout(doe_mb->wq,
> > -			       test_bit(PCI_DOE_FLAG_CANCEL, &doe_mb->flags),
> > -			       timeout))
> > -		return -EIO;
> > -	return 0;
> > -}
> > -
> >  static void pci_doe_write_ctrl(struct pci_doe_mb *doe_mb, u32 val)
> >  {
> >  	struct pci_dev *pdev = doe_mb->pdev;
> > @@ -82,12 +72,9 @@ static int pci_doe_abort(struct pci_doe_mb *doe_mb)
> >  	pci_doe_write_ctrl(doe_mb, PCI_DOE_CTRL_ABORT);
> >  
> >  	do {
> > -		int rc;
> >  		u32 val;
> >  
> > -		rc = pci_doe_wait(doe_mb, PCI_DOE_POLL_INTERVAL);
> > -		if (rc)
> > -			return rc;
> > +		msleep_interruptible(PCI_DOE_POLL_INTERVAL);
> >  		pci_read_config_dword(pdev, offset + PCI_DOE_STATUS, &val);
> >  
> >  		/* Abort success! */
> > @@ -278,11 +265,7 @@ static void doe_statemachine_work(struct work_struct *work)
> >  			signal_task_abort(task, -EIO);
> >  			return;
> >  		}
> > -		rc = pci_doe_wait(doe_mb, PCI_DOE_POLL_INTERVAL);
> > -		if (rc) {
> > -			signal_task_abort(task, rc);
> > -			return;
> > -		}
> > +		msleep_interruptible(PCI_DOE_POLL_INTERVAL);
> >  		goto retry_resp;
> >  	}
> >  
> > @@ -383,21 +366,6 @@ static void pci_doe_destroy_workqueue(void *mb)
> >  	destroy_workqueue(doe_mb->work_queue);
> >  }
> >  
> > -static void pci_doe_flush_mb(void *mb)
> > -{
> > -	struct pci_doe_mb *doe_mb = mb;
> > -
> > -	/* Stop all pending work items from starting */
> > -	set_bit(PCI_DOE_FLAG_DEAD, &doe_mb->flags);
> > -
> > -	/* Cancel an in progress work item, if necessary */
> > -	set_bit(PCI_DOE_FLAG_CANCEL, &doe_mb->flags);
> > -	wake_up(&doe_mb->wq);
> > -
> > -	/* Flush all work items */
> > -	flush_workqueue(doe_mb->work_queue);
> > -}
> > -
> >  /**
> >   * pcim_doe_create_mb() - Create a DOE mailbox object
> >   *
> > @@ -450,14 +418,6 @@ struct pci_doe_mb *pcim_doe_create_mb(struct pci_dev *pdev, u16 cap_offset)
> >  		return ERR_PTR(rc);
> >  	}
> >  
> > -	/*
> > -	 * The state machine and the mailbox should be in sync now;
> > -	 * Set up mailbox flush prior to using the mailbox to query protocols.
> > -	 */
> > -	rc = devm_add_action_or_reset(dev, pci_doe_flush_mb, doe_mb);
> > -	if (rc)
> > -		return ERR_PTR(rc);
> > -
> >  	rc = pci_doe_cache_protocols(doe_mb);
> >  	if (rc) {
> >  		pci_err(pdev, "[%x] failed to cache protocols : %d\n",
>