From:   "Chen, Mike Ximing" <mike.ximing.chen@intel.com>
To:     Stephen Hemminger <stephen@networkplumber.org>
CC:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "arnd@arndb.de" <arnd@arndb.de>,
        "gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
        "Williams, Dan J" <dan.j.williams@intel.com>,
        "pierre-louis.bossart@linux.intel.com" 
        <pierre-louis.bossart@linux.intel.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "davem@davemloft.net" <davem@davemloft.net>,
        "kuba@kernel.org" <kuba@kernel.org>
Subject: RE: [RFC PATCH v12 17/17] dlb: add basic sysfs interfaces
Thread-Topic: [RFC PATCH v12 17/17] dlb: add basic sysfs interfaces
Thread-Index: AQHX9jdp8zCLYq9nh0Woi5QXeJNDCaw9mkwAgABNtGA=
Date:   Wed, 22 Dec 2021 04:21:06 +0000
Message-ID: <CO1PR11MB5170DC5C16ADE17DA4C5AD0FD97D9@CO1PR11MB5170.namprd11.prod.outlook.com>
References: <20211221065047.290182-1-mike.ximing.chen@intel.com>
        <20211221065047.290182-18-mike.ximing.chen@intel.com>
 <20211221153458.51710479@hermes.local>
In-Reply-To: <20211221153458.51710479@hermes.local>
Accept-Language: en-US
Content-Language: en-US
dlp-version: 11.6.200.16
dlp-reaction: no-action
dlp-product: dlpe-windows
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?BX8qNY+VBOdRftM1JP1TpfG+ybUfWtgMqLD+Kq8rHg1aUUH91zWoDKneNWVI?=
 =?us-ascii?Q?hSqV9sQt6IiRYIJt7aPx0qbTr4DLCoi5mWxVzBijRXdOdpTcyze4irAo4bLl?=
 =?us-ascii?Q?xdcEpsgqCZgBAEHOTo/GCcDEi8RueD7LKl42HqNZqpG61lEBxq5ii4BB2Zzy?=
 =?us-ascii?Q?uHpLGgLPPemio+QJf9RbhBCFENemUhKfGy7iP/SWN7ok4hidbaRKOsnKO9t0?=
 =?us-ascii?Q?I6/jiXDsK2XVo0VrDImfzr4kn9D8dCLebG4V1f266IAbr6O35XFi/tEBusOF?=
 =?us-ascii?Q?RaLPjeGIF0O4ss4AH7ftJLR/GfPodkMW7DXpRXH9NOcYq6N+wDtLk8cSHUj8?=
 =?us-ascii?Q?u5dEPGLYFTyjIyDnn2FGJSAohiK0IxoO2Mp37wh6nRL0K/aUzLNftOepYuwv?=
 =?us-ascii?Q?9a6GC2XQlTs81YHQULcnBlFtSS9IgxW+C0E5lvjbgUqXJlwX42eU4VYUxTrZ?=
 =?us-ascii?Q?/HD9zWXql6UJbi4rsGPcAGkqJQucb1YOcVzHP4REPaIfG9yqdWLzo7iM/8j4?=
 =?us-ascii?Q?wcAt6mYrGdEMKGlvGaLM4/7lt2HIMnYyLXjerDNhAlbh81YzmpEx2dO9URdW?=
 =?us-ascii?Q?moHUgJV8Pu2mFfG7eSO+zYYx6++nQ2s4qAjcIDGlOnrq9A0RJW3daxefD7Mi?=
 =?us-ascii?Q?EHrgjUAGq2HceC5rluOSJ+3vjghMoukf/B2Hye4B8x/J83Be6be3GNo9OKsb?=
 =?us-ascii?Q?YWNmxRvY0yH3jJExJ7pt4l/2resGQ9YWaZ2avnw//UXnlBemJ7Oy/EN6oHfU?=
 =?us-ascii?Q?1XhRXeKS4qD11XtmCpSjNCH8PQ6WY5+xJLyjMt0+otnxZKlAD8O2+LQKWKBO?=
 =?us-ascii?Q?VNIVB0d+lXiiLzbbk12TJJ1E4kyQIoia5/BnG8bxUlSD2JN14NzJN8cVSJDd?=
 =?us-ascii?Q?I0jmLUgN+8VDhp8uFfw6nxJ3vxl/v86E+nu/tWkix2l2TO1WvMMEk7bCspcZ?=
 =?us-ascii?Q?GG1dx3l691hXN4aPDe7+uA7VpoG3clnbFzqKaNKXgEeAV2AtKV88LEoJyQiS?=
 =?us-ascii?Q?KgqKoaexZHu0ui/CYk5IksvL0f47CxK/E+e+i7AZqO+C85kw1wxhnEBjbD0L?=
 =?us-ascii?Q?b9ihigxP5gxziqJcCb79VBzZZxOUo5YwFB+oi/mzaDcvVQHGVJb9rf4qraAM?=
 =?us-ascii?Q?oI5raT8nzim47SwyPlwd7PPzHvpN1gPgfOrh/fQmye4WX+pgmYKRXcr+s3o1?=
 =?us-ascii?Q?nxtp+xLNYAPc6YcRNJhmzrW7MnqJ7xA4BR33gFweHxibpi2Kl7NAHfmKNvBA?=
 =?us-ascii?Q?LNMTzXKtf7wMhvznuTBJs+WvexjSXPHhI/2TrDnvMWAqr1NMczVnxrm/3QzW?=
 =?us-ascii?Q?33Ua3GR266C8pwfKX6KCdWqQlVRXrp/yujFWGMn/rLSxbhYeVjNUIO11wtsE?=
 =?us-ascii?Q?A3cUX5PF9TwgpG7VCZkiqSHJcvAN3LgQat4qRSyxrDLrgJEihUgQ3YEHtUVv?=
 =?us-ascii?Q?wbQs15k8OJh1uHP+/LrjpB4w+mNzWARpiGZXDF1NjJnur9EJCAXMbQJRkLFC?=
 =?us-ascii?Q?AaeS4PryMnITE7m1CP+lwc/WPAzm9EWP84cboxDRlyEoSdG5kUOy6NWgo/rC?=
 =?us-ascii?Q?Y8SieGVNr3tnCkJeD4zHTFFzcGMq82DUDPMBQDNyOS9eFHRum9Bwqz3Cf4zL?=
 =?us-ascii?Q?kIgXf9cy7qGnjSho/pnQH08rL8dBxiOVynp75sascFNjxcRBY6S7Iec8lrjd?=
 =?us-ascii?Q?XgIxtQ=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB5170.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 869a9a07-3c2f-4c2e-2b46-08d9c5027925
X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Dec 2021 04:21:06.0988
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: WlriGxS4IrLH0OtG3mYxIe5G54yiO3zg82ZGEE0GvQKV4Dpz8FurDKg1yYeOVFE+GIUMQBOZ4XUHzbDcUnsUzU40Upm3Ca1kk43Z1VWbwDI=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR11MB1743
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Tuesday, December 21, 2021 6:35 PM
> To: Chen, Mike Ximing <mike.ximing.chen@intel.com>
> Cc: linux-kernel@vger.kernel.org; arnd@arndb.de; gregkh@linuxfoundation.o=
rg; Williams, Dan J
> <dan.j.williams@intel.com>; pierre-louis.bossart@linux.intel.com; netdev@=
vger.kernel.org;
> davem@davemloft.net; kuba@kernel.org
> Subject: Re: [RFC PATCH v12 17/17] dlb: add basic sysfs interfaces
>=20
> On Tue, 21 Dec 2021 00:50:47 -0600
> Mike Ximing Chen <mike.ximing.chen@intel.com> wrote:
>=20
> > The dlb sysfs interfaces include files for reading the total and
> > available device resources, and reading the device ID and version. The
> > interfaces are used for device level configurations and resource
> > inquiries.
> >
> > Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
> > ---
> >  Documentation/ABI/testing/sysfs-driver-dlb | 116 ++++++++++++
> >  drivers/misc/dlb/dlb_args.h                |  34 ++++
> >  drivers/misc/dlb/dlb_main.c                |   5 +
> >  drivers/misc/dlb/dlb_main.h                |   3 +
> >  drivers/misc/dlb/dlb_pf_ops.c              | 195 +++++++++++++++++++++
> >  drivers/misc/dlb/dlb_resource.c            |  50 ++++++
> >  6 files changed, 403 insertions(+)
> >  create mode 100644 Documentation/ABI/testing/sysfs-driver-dlb
> >
> > diff --git a/Documentation/ABI/testing/sysfs-driver-dlb
> > b/Documentation/ABI/testing/sysfs-driver-dlb
> > new file mode 100644
> > index 000000000000..bf09ef6f8a3a
> > --- /dev/null
> > +++ b/Documentation/ABI/testing/sysfs-driver-dlb
> > @@ -0,0 +1,116 @@
> > +What:		/sys/bus/pci/devices/.../total_resources/num_atomic_inflights
> > +What:		/sys/bus/pci/devices/.../total_resources/num_dir_credits
> > +What:		/sys/bus/pci/devices/.../total_resources/num_dir_ports
> > +What:		/sys/bus/pci/devices/.../total_resources/num_hist_list_entries
> > +What:		/sys/bus/pci/devices/.../total_resources/num_ldb_credits
> > +What:		/sys/bus/pci/devices/.../total_resources/num_ldb_ports
> > +What:		/sys/bus/pci/devices/.../total_resources/num_cos0_ldb_ports
> > +What:		/sys/bus/pci/devices/.../total_resources/num_cos1_ldb_ports
> > +What:		/sys/bus/pci/devices/.../total_resources/num_cos2_ldb_ports
> > +What:		/sys/bus/pci/devices/.../total_resources/num_cos3_ldb_ports
> > +What:		/sys/bus/pci/devices/.../total_resources/num_ldb_queues
> > +What:		/sys/bus/pci/devices/.../total_resources/num_sched_domains
> > +Date:		Oct 15, 2021
> > +KernelVersion:	5.15
> > +Contact:	mike.ximing.chen@intel.com
> > +Description:
> > +		The total_resources subdirectory contains read-only files that
> > +		indicate the total number of resources in the device.
> > +
> > +		num_atomic_inflights:  Total number of atomic inflights in the
> > +				       device. Atomic inflights refers to the
> > +				       on-device storage used by the atomic
> > +				       scheduler.
> > +
> > +		num_dir_credits:       Total number of directed credits in the
> > +				       device.
> > +
> > +		num_dir_ports:	       Total number of directed ports (and
> > +				       queues) in the device.
> > +
> > +		num_hist_list_entries: Total number of history list entries in
> > +				       the device.
> > +
> > +		num_ldb_credits:       Total number of load-balanced credits in
> > +				       the device.
> > +
> > +		num_ldb_ports:	       Total number of load-balanced ports in
> > +				       the device.
> > +
> > +		num_cos<M>_ldb_ports:  Total number of load-balanced ports
> > +				       belonging to class-of-service M in the
> > +				       device.
> > +
> > +		num_ldb_queues:	       Total number of load-balanced queues in
> > +				       the device.
> > +
> > +		num_sched_domains:     Total number of scheduling domains in the
> > +				       device.
> > +
>=20
> Sysfs is only slightly better than /proc as an API.
> If it is just for testing than debugfs might be better.
>=20
Sysfs in our driver is not only for testing. It is used for the system conf=
iguration
at run time.

> Could this be done with a real netlink interface?
> Maybe as part of devlink?
Thanks for the suggestion. I will look into some sample implementations of
devlink/netlink. Our current plan is to stay with the configfs interface, a=
nd
find ways to resolve issues related to atomic update and resource reset at
tear-down time.

Mike=20