Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Bruce Fields <bfields@fieldses.org>
CC:     linux-rdma <linux-rdma@vger.kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH v1 0/6] Deprecate dprintk in svcrdma
Thread-Topic: [PATCH v1 0/6] Deprecate dprintk in svcrdma
Thread-Index: AQHXwEErGl+cqu1kpUWdnKCEhdLzOavRFhUAgAAOGgA=
Date:   Wed, 13 Oct 2021 16:49:55 +0000
Message-ID: <53AEBF77-7470-4B52-B69E-3CC515C3F393@oracle.com>
References: <163413628188.6408.17033105928649076434.stgit@bazille.1015granger.net>
 <20211013155926.GC6260@fieldses.org>
In-Reply-To: <20211013155926.GC6260@fieldses.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?NmO6EtlEjO/8B09e6JGD23tbiegNj+ovtEiFkECOTYPKXOf1C4xBF5hYnjZu?=
 =?us-ascii?Q?98djRjNMP1SsnYI2Ucl3gZagrEkG9g7MDv2+hZqRIa4+z1vvBc4iv1b9kpIn?=
 =?us-ascii?Q?7cVA7s0eO924d8xz/kOwAg7NuUEQd/vB23PuBtSee3inAoyy1Ht+J+j3jYqF?=
 =?us-ascii?Q?scRmDptdCNNwqWovpre9U7lChjqAq2pTwkFrDWpU861x+UxGqN6J3p1JgTQ/?=
 =?us-ascii?Q?NDlI2l2/E9HY/5daAfxpylXQ36mXHw90MjN16kqJOw2GCFsmQ8JUFsm7lHH3?=
 =?us-ascii?Q?dNkRwItI8HUuv6g1NxMy1jTL/4yrW9eRAwX8N8DGl1BCbmCZagyklNcI0mya?=
 =?us-ascii?Q?wYpMTs5uvp7ZOw57+vW5fwRBWO6YwfYkzo/qY+VXWbMkXQKRO9ppRFZkFeD5?=
 =?us-ascii?Q?s1R6hBmfY5BNWrGuFkwDp25J76svd9rCHr+d4Xqh933d3MTQ3jJDVvJTb8Xk?=
 =?us-ascii?Q?lZHQMeeHhz6vPukH8fhgi/5AvXCJXfdkTj1m02QLCdv3VRVGw8TF12+6ar0t?=
 =?us-ascii?Q?agguklEmjeYHtUGNMZb0I4yHqc0mqvedhnxqKMD356CCeYXpp1k5dEHH2jeB?=
 =?us-ascii?Q?TtZWIvVzsExrC++S9Zi4CM+U1pqRFV+f1u6GPhgEOHyedZrC/Xu0Ttrc7zPt?=
 =?us-ascii?Q?Z9zvGopHKfUP84n/h03Mo9qGtvW/4/ofooA6KWZxYeSvcFm9xqUF2s9BB9UD?=
 =?us-ascii?Q?6yEaTpWnNVPZ4WN+unzmEKr1LwVlAo+QpBF3jVsq+5huaP80v0l635eA54eY?=
 =?us-ascii?Q?Mo70LwWqEn9wnssOOAT3TZiUr7F3PcA08jflEYh/oqTOLDKIPkjfeSCXG5s2?=
 =?us-ascii?Q?UohCeZpM92CD5IX1M7lX6I0/jgk0ejGjBsjkgHt6Y57piT/S2x7Rf1w/e57C?=
 =?us-ascii?Q?01v343Pbvtrfkw4YcYBfSd9+2vGbhn/Jfa/GfaD1Q4Msz5nfZ44uSZZj2sOk?=
 =?us-ascii?Q?yypXd8ldMU1bnqa0ap1YCrqpLcUuaAq1IZF6nhE+zqn7ArrOUlCAJt70lo9D?=
 =?us-ascii?Q?Vm/mSJrQF/kIdvd0YZTXv1wetGOnmArEEWbsSs48bcvA1CHik47CiJYlr3bz?=
 =?us-ascii?Q?9eZfEl7bhoETFPTYWo1iTV8v81nI9wfRIJbXMdmyVoS4Mlj1CkVzQTxp1AnN?=
 =?us-ascii?Q?O0BLGlvcxPVoXJ//ZfuxemuVbiYq23Mf3qoapm181dtcRCAgRpWGzG32n/Fx?=
 =?us-ascii?Q?bQ8n5dLUQbgwRQxzhK0rYoEDYB1XiaTD045GrekcjwWiXTJ8h7SPgHVE+p0K?=
 =?us-ascii?Q?7eAbNWM29nFA6MW7JVG/rEmUx3DnwoeS6W5bvwkFeZSAEvZetTTudIT7bkrv?=
 =?us-ascii?Q?0neqjQRNpA+zsTkZdAaXWA58?=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="us-ascii"
Content-ID: <C4A6FFDDACB5D747A294E3EDE2D90F9F@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: be51252f-a637-4c11-f049-08d98e697c58
X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Oct 2021 16:49:55.8111
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 6QH+vgkKQ8Luk6lAGnJaSNAmCA/hfOdSTCvrXCPLJKb2shD2iipNkcpCTIrZCcIfML2CMfk0bBn9Cw0VAlXwVg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB4086
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10136 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0
 adultscore=0 bulkscore=0 mlxscore=0 spamscore=0 mlxlogscore=999
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2109230001 definitions=main-2110130104
X-Proofpoint-GUID: X_IXNXRzo05NvEm8tQGWqT0RyXWxPolK
X-Proofpoint-ORIG-GUID: X_IXNXRzo05NvEm8tQGWqT0RyXWxPolK
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Oct 13, 2021, at 11:59 AM, J. Bruce Fields <bfields@fieldses.org> wrot=
e:
>=20
> On Wed, Oct 13, 2021 at 10:46:49AM -0400, Chuck Lever wrote:
>> This patch series moves forward with the removal of dprintk in
>> SUNRPC in favor of tracepoints. This is the last step for the
>> svcrdma component.
>=20
> Makes sense to me.
>=20
> I would like some (very short) documentation, somewhere.  Partly just
> for my sake!  I'm not sure exactly what to recommend to bug reporters.
>=20
> I guess=20
>=20
> 	trace-cmd record -e 'sunrpc:*'
> 	trace-cmd report
>=20
> would be a rough substitute for "rpcdebug -m rpc -s all"?

It would, but tracepoints can be enabled one event
at a time. If you're looking for a direct replacement
for a specific rpcdebug invocation, it might be better
to examine the current sunrpc debug facilities and
provide specific command lines to mimic those.

"rpcdebug -vh" gives us:

rpc        xprt call debug nfs auth bind sched trans svcsock svcdsp misc ca=
che all
nfs        vfs dircache lookupcache pagecache proc xdr file root callback c=
lient mount fscache pnfs pnfs_ld state all
nfsd       sock fh export svc proc fileop auth repcache xdr lockd all
nlm        svc client clntlock svclock monitor clntsubs svcsubs hostcache x=
dr all


If tracepoints are named carefully, we can provide
specific command lines to enable them as groups. So,
for instance, I was thinking rpcdebug might display:

	trace-cmd list | grep svcrdma

to list tracepoints related to server side RDMA, or:

	trace-cmd list | grep svcsock

to show tracepoints related to server side sockets.
Then:

	trace-cmd record -e sunrpc:svcsock\*

enables just the socket-related trace events, which
coincidentally happens to line up with:

	rpcdebug -m rpc -s svcsock


> Do we have a couple examples of issues that could be diagnosed with
> tracepoints?

Anything you can do with dprintk you can do with trace
points. Plus because tracepoints are lower overhead, they
can be enabled and used in production environments,
unlike dprintk.

Also, tracepoints can trigger specific user space actions
when they fire. You could for example set up a tracepoint
in the RPC client that fires when a retransmit timeout
occurs, and it could trigger a script to start tcpdump.


> In the past I don't feel like I've ended up using dprintks
> all that much; somehow they're not usually where I need them.  But maybe
> that's just me.  And maybe as we put more thought into where tracepoints
> should be, they'll get more useful.

> Documentation/filesystems/nfs/, or the linux-nfs wiki, could be easy
> places to put it.  Though *something* in the man pages would be nice.
> At a minimum, a warning in rpcdebug(8) that we're gradually phasing out
> dprintks.

As I understood the conversation last week, SteveD and
DaveW volunteered to be responsible for changes to
rpcdebug?

So far we haven't had much documentation for dprintk. That
means we are starting more or less from scratch for
explaining observability in the NFS stacks. Free rein, and
all that.

--
Chuck Lever