Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 18 Nov 2022 15:26:17 -0800
From:   Ira Weiny <ira.weiny@intel.com>
To:     Jonathan Cameron <Jonathan.Cameron@huawei.com>
CC:     Dan Williams <dan.j.williams@intel.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Alison Schofield <alison.schofield@intel.com>,
        "Vishal Verma" <vishal.l.verma@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        Davidlohr Bueso <dave@stgolabs.net>,
        <linux-kernel@vger.kernel.org>, <linux-cxl@vger.kernel.org>
Subject: Re: [PATCH 02/11] cxl/mem: Implement Get Event Records command
Message-ID: <Y3gUmSuR3OxUwkDm@iweiny-desk3>
References: <20221110185758.879472-1-ira.weiny@intel.com>
 <20221110185758.879472-3-ira.weiny@intel.com>
 <20221116151936.0000662f@Huawei.com>
 <Y3WEmMlLfPoYG1R5@iweiny-desk3>
 <20221117104337.00001a3f@Huawei.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20221117104337.00001a3f@Huawei.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?aqk/8dnkMziPVZvZkilumPmAFDzdVEZUp7rDmnUBQ9sUQJbx4ubVxim70aHU?=
 =?us-ascii?Q?tpV9aUc+ApT62ZEOQP0bigaXNeEOhztvAzE0or2ZGpCf9ye3niwGHBy6dB1U?=
 =?us-ascii?Q?fyreVvsYNMXgtv5UgykkxLawKNyatjCM1xDutASlgJ5bVFGX6uwHCxddzkQc?=
 =?us-ascii?Q?t3BxK9oDq97SXUBnxPhF8QGDb1njbNWJrprowgV6ePyd/CCxr7WxUhHeObfE?=
 =?us-ascii?Q?RTU1wblPpe8VdMpaMe1aDhXm1Lnd6VcR+zU++BpkE2CNkaTiinTF0au56/BN?=
 =?us-ascii?Q?CGCujPZaYnZxDZc1AQrClsP0B7UNRdrV5AOhQV4hDGKsoi0+pDd8BhTRfDwC?=
 =?us-ascii?Q?HK3OAoMeYgY+ocTD8qVP+IJzyI91o2xs91CCrdyvyogbJCP2ph5kgvNwOik3?=
 =?us-ascii?Q?HoOMqEoos9T6cGqcBQFKFr77pkwoUxuIi52uI+3WbkqBsad0xDfJcT1HUw40?=
 =?us-ascii?Q?oFmIf3fugSnZNarEdxss6QjS2WRK2cq3CVEtSC+Eq7UJK5Rvslm1H7FSTy9z?=
 =?us-ascii?Q?EyZ3JkgWA9BYI6VihItmFYc1cgLcGuWfDZ2qobEIF/E1JIiSiXHq9/U801ZO?=
 =?us-ascii?Q?g4GhWaqFwzJhyYIBsi3N5bbq+gk12R9xgrY5gvc+S2Bz842TOWeO0J+Lc8tC?=
 =?us-ascii?Q?aXI6z9tulCZfWazbTE2phxyU6sGvC7HAj0ppo8iC/bfsXo4l7uW7gfm2rG29?=
 =?us-ascii?Q?jdw8hI1+JIOdkUqNQ1bM9vNBYmMaN1vA/rlMnfANHSHsQWrYRp9Kq+SN+3We?=
 =?us-ascii?Q?LviCwG/HTmg1Q/perTGQTN+ytOdON2dxhEOcRhvzKTKBPn/0y8pIn5TCufBs?=
 =?us-ascii?Q?97YLJE8TKq3QQwUcy5FZS3LMHgZXO4zpp9MiYXMOs7p7ax2r/fGrblgR1sS0?=
 =?us-ascii?Q?RlWwBfxx0GgofU5SuolP94wuLgUGqIV2J9AVM6MQ8y64rnuAxy8OTvhJDv9E?=
 =?us-ascii?Q?5Dgd/47Vt67WY46T9xDH6s9f+1w/XJXm4S1x5Yda1pzNM3b+YYh1p3bfMljO?=
 =?us-ascii?Q?NwMhex36mLX0NgLgq7CCLgv4DJLwqm15/5151I/r02qx5utvam16t2XqeRNO?=
 =?us-ascii?Q?nLKjiXzRA5BnHwArjdzO59CrFYoy/ISGEtBeLJxLPQY6jS+nCIciuW4EDKwT?=
 =?us-ascii?Q?lxNP6dWOVIkGAHENJ0PbxrJ5NIkI6EGKBjDlqdrpVyEgw4y4G5gqI9wAKMwa?=
 =?us-ascii?Q?F+q2mtsG6RUa+cdDg9fY3vUUDeR7LGGmq94eiSOcX9Ey7+6Q/7BaelwCXwmS?=
 =?us-ascii?Q?j3H2WChK6Tu0twRIOZxwwWP3NtWuaMEVnRlRxKFckU37kw2apZbgvky4MnK3?=
 =?us-ascii?Q?J3K4U7yVC3jgqWKtory2ECpS8Bun7OYJgrF/+okE28Y6V52WILoYWkJXfWZ1?=
 =?us-ascii?Q?nC1te28xN3ckjI8NtsHG+USvSCiEElfzByELuGLsyVod8HdFvEGpO6dfzqJm?=
 =?us-ascii?Q?o3dtgnDHXhN/czp2W7UiAtHzLQpkEuyjvWBgWjIQ93aZ/odmx7Tig+ZM8pzW?=
 =?us-ascii?Q?2E21ChRPZr/6lpJtQcBbsJlSuiEKNpk3GNSr4vjDMaRwZ+W2uiS9S6ZwZVdm?=
 =?us-ascii?Q?jSmHar+kQAnBzaAga7EC139aJdoQ2MOVAtydkJKO?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 4d151f10-ba18-4161-6561-08dac9bc4dbd
X-MS-Exchange-CrossTenant-AuthSource: SA1PR11MB6733.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Nov 2022 23:26:22.3684
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: DYIml33voLQZ/qbe38Z36dM6hGmsJrYPE9sjAOeWyWsqDzN8LlEIfSjYLON962X31LRQzp93GYHMN9CvUnXUYA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB5887
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Nov 17, 2022 at 10:43:37AM +0000, Jonathan Cameron wrote:
> On Wed, 16 Nov 2022 16:47:20 -0800
> Ira Weiny <ira.weiny@intel.com> wrote:
> 
> 

[snip]

> > 
> > >   
> > > > +			int i;
> > > > +
> > > > +			for (i = 0; i < nr_rec; i++)
> > > > +				trace_cxl_generic_event(dev_name(cxlds->dev),
> > > > +							type,
> > > > +							&payload.record[i]);
> > > > +		}
> > > > +
> > > > +		if (trace_cxl_overflow_enabled() &&
> > > > +		    (payload.flags & CXL_GET_EVENT_FLAG_OVERFLOW))
> > > > +			trace_cxl_overflow(dev_name(cxlds->dev), type, &payload);
> > > > +
> > > > +	} while (pl_nr > CXL_GET_EVENT_NR_RECORDS ||  
> > > 
> > > Isn't pl_nr > CXL_GET_EVENT_NR_RECORDS a hardware bug? It's the number in returned
> > > payload not the total number.  
> > 
> > I don't think so.  The only value passed to the device is the _input_ payload
> > size.  The output payload size is not passed to the device and is not included
> > in the Get Event Records Input Payload.  (Table 8-49)
> > 
> > So my previous code was wrong.  Here is an example I think which is within the
> > spec but would result in the more records flag not being set.
> > 
> > 	Device log depth == 10
> > 	nr log entries == 7
> > 	nr log entries in 1MB ~= (1M - hdr size) / 128 ~= 8000
> > 
> > Device sets Output Payload.Event Record Count == 7 (which is < 8000).  Common
> > mailbox code truncates that to 3.  More Event Records == 0 because it sent all
> > 7 that it had.
> > 
> > This code will clear 3 and read again 2 more times.
> > 
> > Am I reading that wrong?
> 
> I think this is still wrong, but for a different reason. :)

I hope not...  :-/

> If we don't clear the records and more records is set, that means it didn't
> fit in the mailbox payload (potentially 1MB)  then the next read
> will return the next set of records from there.

That is not how I read the Get Event Records command:

From 8.2.9.2.2 Get Event Records

... "Devices shall return event records to the host in the temporal order the
device detected the events in. The event occurring the earliest in time, in the
specific event log, shall be returned first."

If item 3 below is earlier than 4 then it must be returned if we have not
cleared it.  At least that is how I read the above.  :-/

> 
> Taking this patch only, let's say the mailbox takes 4 records.
> Read 1: Records 0, 1, 2, 3 More set.
>    We handle 0, 1, 2
> Read 2: Records 4, 5, 6 More not set.
>    We handle 4, 5, 6
> 
> Record 3 is never handled.
> 
> If we add in clearing as happens later in the series,

I suppose I should squash the patches as this may not work without the
clearing.  :-/

> the current
> assumption is that if we clear some records a subsequent read will
> start again.  I'm not sure that is true. If it is spec reference needed.
> 
> So assumption is
> Read 1: Records 0, 1, 2, 3 More set
>   Clear 0, 1, 2
> Read 2: Records 3, 4, 5, 6
>   Clear 3, 4, 5 More not set, but catch it with the condition above.
> Read 3: 6 only
>   Clear 6
> 
> However, I think a valid implementation could do the following
> (imagine a ring buffer with a pointer to the 'next' record to read out and
>  each record has a 'valid' flag to deal with corner cases around
>  sequences such as read log once, start reading again and some
>  clears occur using handles obtained from first read - not that
>  case isn't ruled out by the spec as far as I can see).

I believe this is a violation because the next pointer can't be advanced until
the record is cleared.  Otherwise the device is not returning items in temporal
order based on what is in the log.

> 
> Read 1: Records 0, 1, 2, 3 More set.  'next' pointer points to record 4.
>   Clear 0, 1, 2
> Read 2: Records 4, 5, 6 More not set. 'next' pointer points to record 7.
>   Clear 4, 5, 6
> 
> Skipping record 3.
> 
> So I think we have to absorb the full mailbox payload each time to guarantee
> we don't skip events or process them out of order (which is what would happen
> if we relied on a retry loop - we aren't allowed to clear them out of
> order anyway 8.2.9.2.3 "Events shall be cleared in temporal order. The device
> shall verify the event record handles specified in the input payload are in
> temporal order. ... "). 
> Obviously that temporal order thing is only relevant if we get my second
> example occurring on real hardware.  I think the spec is vague enough
> to allow that implementation.  Would have been easy to specify this originally
> but it probably won't go in as errata so we need to cope with all the
> flexibility that is present.

:-(  Yea coulda, woulda, shoulda...  ;-)

> 
> What fun and oh for a parameter to control how many records are returned!

Yea.  But I really don't think there is a problem unless someone really take
liberty with the spec.  I think it boils down to how one interprets _when_ a
record is removed from the log.

If the record is removed when it is returned (as in your 'next' pointer
example) then why have a clear at all?  If my interpretation is correct then
the next available entry is the one which has not been cleared.  Therefore in
your example 'next' is not incremented until clear has been called.  I think
that implementation is also supported by the idea that records must be cleared
in temporal order.  Otherwise I think devices would get confused.

FWIW the qemu implementation is based on my interpretation ATM.

Ira

> 
> Jonathan
> 
> 
> > 
> > >   
> > > > +		 payload.flags & CXL_GET_EVENT_FLAG_MORE_RECORDS);
> > > > +}  
> > > 
> 
> > 
>