2019-03-12 16:06:46

by Olga Kornievskaia

[permalink] [raw]
Subject: [PATCH 1/1] [SUNRPC] fix null pointer deref in tracepoints in back channel

From: Olga Kornievskaia <[email protected]>

Backchannel doesn't have the rq_task->tk_clientid pointer set.

Otherwise can lead to the following oops:
ocalhost login: [ 111.385319] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
[ 111.388073] #PF error: [normal kernel read fault]
[ 111.389452] PGD 80000000290d8067 P4D 80000000290d8067 PUD 75f25067 PMD 0
[ 111.391224] Oops: 0000 [#1] SMP PTI
[ 111.392151] CPU: 0 PID: 3533 Comm: NFSv4 callback Not tainted 5.0.0-rc7+ #1
[ 111.393787] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 111.396340] RIP: 0010:trace_event_raw_event_xprt_enq_xmit+0x6f/0xf0 [sunrpc]
[ 111.397974] Code: 00 00 00 48 89 ee 48 89 e7 e8 bd 0a 85 d7 48 85 c0 74 4a 41 0f b7 94 24 e0 00 00 00 48 89 e7 89 50 08 49 8b 94 24 a8 00 00 00 <8b> 52 04 89 50 0c 49 8b 94 24 c0 00 00 00 8b 92 a8 00 00 00 0f ca
[ 111.402215] RSP: 0018:ffffb98743263cf8 EFLAGS: 00010286
[ 111.403406] RAX: ffffa0890fc3bc88 RBX: 0000000000000003 RCX: 0000000000000000
[ 111.405057] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffb98743263cf8
[ 111.406656] RBP: ffffa0896f5368f0 R08: 0000000000000246 R09: 0000000000000000
[ 111.408437] R10: ffffe19b01c01500 R11: 0000000000000000 R12: ffffa08977d28a00
[ 111.410210] R13: 0000000000000004 R14: ffffa089315303f0 R15: ffffa08931530000
[ 111.411856] FS: 0000000000000000(0000) GS:ffffa0897bc00000(0000) knlGS:0000000000000000
[ 111.413699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 111.415068] CR2: 0000000000000004 CR3: 000000002ac90004 CR4: 00000000001606f0
[ 111.416745] Call Trace:
[ 111.417339] xprt_request_enqueue_transmit+0x2b6/0x4a0 [sunrpc]
[ 111.418709] ? rpc_task_need_encode+0x40/0x40 [sunrpc]
[ 111.419957] call_bc_transmit+0xd5/0x170 [sunrpc]
[ 111.421067] __rpc_execute+0x7e/0x3f0 [sunrpc]
[ 111.422177] rpc_run_bc_task+0x78/0xd0 [sunrpc]
[ 111.423212] bc_svc_process+0x281/0x340 [sunrpc]
[ 111.424325] nfs41_callback_svc+0x130/0x1c0 [nfsv4]
[ 111.425430] ? remove_wait_queue+0x60/0x60
[ 111.426398] kthread+0xf5/0x130
[ 111.427155] ? nfs_callback_authenticate+0x50/0x50 [nfsv4]
[ 111.428388] ? kthread_bind+0x10/0x10
[ 111.429270] ret_from_fork+0x1f/0x30

localhost login: [ 467.462259] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
[ 467.464411] #PF error: [normal kernel read fault]
[ 467.465445] PGD 80000000728c1067 P4D 80000000728c1067 PUD 728c0067 PMD 0
[ 467.466980] Oops: 0000 [#1] SMP PTI
[ 467.467759] CPU: 0 PID: 3517 Comm: NFSv4 callback Not tainted 5.0.0-rc7+ #1
[ 467.469393] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 467.471840] RIP: 0010:trace_event_raw_event_xprt_transmit+0x7c/0xf0 [sunrpc]
[ 467.473392] Code: f6 48 85 c0 74 4b 49 8b 94 24 98 00 00 00 48 89 e7 0f b7 92 e0 00 00 00 89 50 08 49 8b 94 24 98 00 00 00 48 8b 92 a8 00 00 00 <8b> 52 04 89 50 0c 41 8b 94 24 a8 00 00 00 0f ca 89 50 10 41 8b 94
[ 467.477605] RSP: 0018:ffffabe7434fbcd0 EFLAGS: 00010282
[ 467.478793] RAX: ffff99720fc3bce0 RBX: 0000000000000003 RCX: 0000000000000000
[ 467.480409] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffabe7434fbcd0
[ 467.482011] RBP: ffff99726f631948 R08: 0000000000000246 R09: 0000000000000000
[ 467.483591] R10: 0000000070000000 R11: 0000000000000000 R12: ffff997277dfcc00
[ 467.485226] R13: 0000000000000000 R14: 0000000000000000 R15: ffff99722fecdca8
[ 467.486830] FS: 0000000000000000(0000) GS:ffff99727bc00000(0000) knlGS:0000000000000000
[ 467.488596] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 467.489931] CR2: 0000000000000004 CR3: 00000000270e6006 CR4: 00000000001606f0
[ 467.491559] Call Trace:
[ 467.492128] xprt_transmit+0x303/0x3f0 [sunrpc]
[ 467.493143] ? rpc_task_need_encode+0x40/0x40 [sunrpc]
[ 467.494328] call_bc_transmit+0x49/0x170 [sunrpc]
[ 467.495379] __rpc_execute+0x7e/0x3f0 [sunrpc]
[ 467.496451] rpc_run_bc_task+0x78/0xd0 [sunrpc]
[ 467.497467] bc_svc_process+0x281/0x340 [sunrpc]
[ 467.498507] nfs41_callback_svc+0x130/0x1c0 [nfsv4]
[ 467.499751] ? remove_wait_queue+0x60/0x60
[ 467.500686] kthread+0xf5/0x130
[ 467.501438] ? nfs_callback_authenticate+0x50/0x50 [nfsv4]
[ 467.502640] ? kthread_bind+0x10/0x10
[ 467.503454] ret_from_fork+0x1f/0x30

Signed-off-by: Olga Kornievskaia <[email protected]>
---
include/trace/events/sunrpc.h | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/include/trace/events/sunrpc.h b/include/trace/events/sunrpc.h
index 8451f30..7e899e6 100644
--- a/include/trace/events/sunrpc.h
+++ b/include/trace/events/sunrpc.h
@@ -712,7 +712,8 @@

TP_fast_assign(
__entry->task_id = rqst->rq_task->tk_pid;
- __entry->client_id = rqst->rq_task->tk_client->cl_clid;
+ __entry->client_id = rqst->rq_task->tk_client ?
+ rqst->rq_task->tk_client->cl_clid : -1;
__entry->xid = be32_to_cpu(rqst->rq_xid);
__entry->seqno = rqst->rq_seqno;
__entry->status = status;
@@ -742,7 +743,8 @@

TP_fast_assign(
__entry->task_id = task->tk_pid;
- __entry->client_id = task->tk_client->cl_clid;
+ __entry->client_id = task->tk_client ?
+ task->tk_client->cl_clid : -1;
__entry->xid = be32_to_cpu(task->tk_rqstp->rq_xid);
__entry->seqno = task->tk_rqstp->rq_seqno;
__entry->stage = stage;
--
1.8.3.1



2019-03-12 20:12:30

by Trond Myklebust

[permalink] [raw]
Subject: Re: [PATCH 1/1] [SUNRPC] fix null pointer deref in tracepoints in back channel

On Tue, 2019-03-12 at 12:06 -0400, Olga Kornievskaia wrote:
> From: Olga Kornievskaia <[email protected]>
>
> Backchannel doesn't have the rq_task->tk_clientid pointer set.
>
> Otherwise can lead to the following oops:
> ocalhost login: [ 111.385319] BUG: unable to handle kernel NULL
> pointer dereference at 0000000000000004
> [ 111.388073] #PF error: [normal kernel read fault]
> [ 111.389452] PGD 80000000290d8067 P4D 80000000290d8067 PUD 75f25067
> PMD 0
> [ 111.391224] Oops: 0000 [#1] SMP PTI
> [ 111.392151] CPU: 0 PID: 3533 Comm: NFSv4 callback Not tainted
> 5.0.0-rc7+ #1
> [ 111.393787] Hardware name: VMware, Inc. VMware Virtual
> Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
> [ 111.396340] RIP:
> 0010:trace_event_raw_event_xprt_enq_xmit+0x6f/0xf0 [sunrpc]
> [ 111.397974] Code: 00 00 00 48 89 ee 48 89 e7 e8 bd 0a 85 d7 48 85
> c0 74 4a 41 0f b7 94 24 e0 00 00 00 48 89 e7 89 50 08 49 8b 94 24 a8
> 00 00 00 <8b> 52 04 89 50 0c 49 8b 94 24 c0 00 00 00 8b 92 a8 00 00
> 00 0f ca
> [ 111.402215] RSP: 0018:ffffb98743263cf8 EFLAGS: 00010286
> [ 111.403406] RAX: ffffa0890fc3bc88 RBX: 0000000000000003 RCX:
> 0000000000000000
> [ 111.405057] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> ffffb98743263cf8
> [ 111.406656] RBP: ffffa0896f5368f0 R08: 0000000000000246 R09:
> 0000000000000000
> [ 111.408437] R10: ffffe19b01c01500 R11: 0000000000000000 R12:
> ffffa08977d28a00
> [ 111.410210] R13: 0000000000000004 R14: ffffa089315303f0 R15:
> ffffa08931530000
> [ 111.411856] FS: 0000000000000000(0000) GS:ffffa0897bc00000(0000)
> knlGS:0000000000000000
> [ 111.413699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 111.415068] CR2: 0000000000000004 CR3: 000000002ac90004 CR4:
> 00000000001606f0
> [ 111.416745] Call Trace:
> [ 111.417339] xprt_request_enqueue_transmit+0x2b6/0x4a0 [sunrpc]
> [ 111.418709] ? rpc_task_need_encode+0x40/0x40 [sunrpc]
> [ 111.419957] call_bc_transmit+0xd5/0x170 [sunrpc]
> [ 111.421067] __rpc_execute+0x7e/0x3f0 [sunrpc]
> [ 111.422177] rpc_run_bc_task+0x78/0xd0 [sunrpc]
> [ 111.423212] bc_svc_process+0x281/0x340 [sunrpc]
> [ 111.424325] nfs41_callback_svc+0x130/0x1c0 [nfsv4]
> [ 111.425430] ? remove_wait_queue+0x60/0x60
> [ 111.426398] kthread+0xf5/0x130
> [ 111.427155] ? nfs_callback_authenticate+0x50/0x50 [nfsv4]
> [ 111.428388] ? kthread_bind+0x10/0x10
> [ 111.429270] ret_from_fork+0x1f/0x30
>
> localhost login: [ 467.462259] BUG: unable to handle kernel NULL
> pointer dereference at 0000000000000004
> [ 467.464411] #PF error: [normal kernel read fault]
> [ 467.465445] PGD 80000000728c1067 P4D 80000000728c1067 PUD 728c0067
> PMD 0
> [ 467.466980] Oops: 0000 [#1] SMP PTI
> [ 467.467759] CPU: 0 PID: 3517 Comm: NFSv4 callback Not tainted
> 5.0.0-rc7+ #1
> [ 467.469393] Hardware name: VMware, Inc. VMware Virtual
> Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
> [ 467.471840] RIP:
> 0010:trace_event_raw_event_xprt_transmit+0x7c/0xf0 [sunrpc]
> [ 467.473392] Code: f6 48 85 c0 74 4b 49 8b 94 24 98 00 00 00 48 89
> e7 0f b7 92 e0 00 00 00 89 50 08 49 8b 94 24 98 00 00 00 48 8b 92 a8
> 00 00 00 <8b> 52 04 89 50 0c 41 8b 94 24 a8 00 00 00 0f ca 89 50 10
> 41 8b 94
> [ 467.477605] RSP: 0018:ffffabe7434fbcd0 EFLAGS: 00010282
> [ 467.478793] RAX: ffff99720fc3bce0 RBX: 0000000000000003 RCX:
> 0000000000000000
> [ 467.480409] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> ffffabe7434fbcd0
> [ 467.482011] RBP: ffff99726f631948 R08: 0000000000000246 R09:
> 0000000000000000
> [ 467.483591] R10: 0000000070000000 R11: 0000000000000000 R12:
> ffff997277dfcc00
> [ 467.485226] R13: 0000000000000000 R14: 0000000000000000 R15:
> ffff99722fecdca8
> [ 467.486830] FS: 0000000000000000(0000) GS:ffff99727bc00000(0000)
> knlGS:0000000000000000
> [ 467.488596] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 467.489931] CR2: 0000000000000004 CR3: 00000000270e6006 CR4:
> 00000000001606f0
> [ 467.491559] Call Trace:
> [ 467.492128] xprt_transmit+0x303/0x3f0 [sunrpc]
> [ 467.493143] ? rpc_task_need_encode+0x40/0x40 [sunrpc]
> [ 467.494328] call_bc_transmit+0x49/0x170 [sunrpc]
> [ 467.495379] __rpc_execute+0x7e/0x3f0 [sunrpc]
> [ 467.496451] rpc_run_bc_task+0x78/0xd0 [sunrpc]
> [ 467.497467] bc_svc_process+0x281/0x340 [sunrpc]
> [ 467.498507] nfs41_callback_svc+0x130/0x1c0 [nfsv4]
> [ 467.499751] ? remove_wait_queue+0x60/0x60
> [ 467.500686] kthread+0xf5/0x130
> [ 467.501438] ? nfs_callback_authenticate+0x50/0x50 [nfsv4]
> [ 467.502640] ? kthread_bind+0x10/0x10
> [ 467.503454] ret_from_fork+0x1f/0x30
>
> Signed-off-by: Olga Kornievskaia <[email protected]>
> ---
> include/trace/events/sunrpc.h | 6 ++++--
> 1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/include/trace/events/sunrpc.h
> b/include/trace/events/sunrpc.h
> index 8451f30..7e899e6 100644
> --- a/include/trace/events/sunrpc.h
> +++ b/include/trace/events/sunrpc.h
> @@ -712,7 +712,8 @@
>
> TP_fast_assign(
> __entry->task_id = rqst->rq_task->tk_pid;
> - __entry->client_id = rqst->rq_task->tk_client->cl_clid;
> + __entry->client_id = rqst->rq_task->tk_client ?
> + rqst->rq_task->tk_client->cl_clid : -1;
> __entry->xid = be32_to_cpu(rqst->rq_xid);
> __entry->seqno = rqst->rq_seqno;
> __entry->status = status;
> @@ -742,7 +743,8 @@
>
> TP_fast_assign(
> __entry->task_id = task->tk_pid;
> - __entry->client_id = task->tk_client->cl_clid;
> + __entry->client_id = task->tk_client ?
> + task->tk_client->cl_clid : -1;
> __entry->xid = be32_to_cpu(task->tk_rqstp->rq_xid);
> __entry->seqno = task->tk_rqstp->rq_seqno;
> __entry->stage = stage;

Thanks Olga! Applied to linux-next...

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]