2017-01-30 20:18:35

by Olga Kornievskaia

[permalink] [raw]
Subject: upstream nfsd kernel oops in auth_rpcgss

Hi Bruce,

I ran into this oops in the nfsd (below) (4.10-rc3 kernel). To trigger
this I had a client (unsuccessfully) try to mount the server with krb5
where the server doesn=E2=80=99t have the rpcsec_gss_krb5 module built.

Below code seems to fix things:

diff --git a/net/sunrpc/auth_gss/svcauth_gss.c
b/net/sunrpc/auth_gss/svcauth_gss.c
index 886e9d38..7faffd8 100644
--- a/net/sunrpc/auth_gss/svcauth_gss.c
+++ b/net/sunrpc/auth_gss/svcauth_gss.c
@@ -1187,9 +1187,9 @@ static int gss_proxy_save_rsc(struct cache_detail *cd=
,
status =3D -EOPNOTSUPP;
/* get mech handle from OID */
gm =3D gss_mech_get_by_OID(&ud->mech_oid);
+ rsci.cred.cr_gss_mech =3D gm;
if (!gm)
goto out;
- rsci.cred.cr_gss_mech =3D gm;



status =3D -EINVAL;
/* mech-specific data: */

I guess the problem is because rsci.cred gets initialed from something
that=E2=80=99s passed in that doesn=E2=80=99t have it=E2=80=99s cr_gss_mech=
initialized to
NULL. So when gss_mech_get_by_OID() fails and it calls rsc_free() and
tries to free a bad pointer.


Don=E2=80=99t know if this is acceptable or it should be traced to where th=
e
uninitialized pointer is coming from.

If this approach is ok, I can re-send this as a patch.

[120408.542387] general protection fault: 0000 [#1] SMP
[120408.544273] Modules linked in: nfsd nfs_acl nfsv4 dns_resolver nfs
lockd auth_rpcgss grace fscache sunrpc fuse ip6t_rpfilter ip6t_REJECT
nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT
nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack
nf_conntrack snd_seq_midi snd_seq_midi_event ebtable_nat
ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_mangle
ip6table_security ip6table_raw ip6table_filter ip6_tables
iptable_mangle iptable_security iptable_raw rfcomm iptable_filter bnep
ip_tables coretemp crc32_pclmul crc32c_intel ghash_clmulni_intel btusb
btrtl btbcm btintel bluetooth rfkill snd_ens1371 snd_ac97_codec ppdev
ac97_bus snd_seq aesni_intel crypto_simd cryptd glue_helper uvcvideo
videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core
snd_pcm pcspkr serio_raw
[120408.558655] videodev snd_rawmidi snd_timer snd_seq_device snd
soundcore vmw_vmci shpchp i2c_piix4 parport_pc parport acpi_cpufreq
uinput xfs libcrc32c sr_mod sd_mod cdrom ata_generic pata_acpi vmwgfx
drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm
ata_piix e1000 ahci libahci libata mptspi scsi_transport_spi mptscsih
mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
[120408.565724] CPU: 0 PID: 3601 Comm: nfsd Not tainted 4.10.0-rc3+ #16
[120408.567037] Hardware name: VMware, Inc. VMware Virtual
Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[120408.569225] task: ffff8800776f95c0 task.stack: ffffc90003d58000
[120408.570483] RIP: 0010:gss_mech_put+0xb/0x20 [auth_rpcgss]
[120408.571614] RSP: 0018:ffffc90003d5bb78 EFLAGS: 00010206
[120408.572731] RAX: ffff880072ecc301 RBX: ffffc90003d5bba8 RCX:
00000000003d0040
[120408.574240] RDX: 00000000003d003f RSI: 0000000000000246 RDI:
4f432e4553554f48
[120408.575685] RBP: ffffc90003d5bb90 R08: 000000000001b2c0 R09:
ffffffffa0703e0c
[120408.577149] R10: ffff88007b61b2c0 R11: ffffea0001cbb300 R12:
ffff8800730c1500
[120408.578592] R13: 00000000ffffffa1 R14: ffff880073a94b40 R15:
ffff880074e0d600
[120408.580058] FS: 0000000000000000(0000) GS:ffff88007b600000(0000)
knlGS:0000000000000000
[120408.581708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[120408.582889] CR2: 00007fe3980140b8 CR3: 000000006f063000 CR4:
00000000001406f0
[120408.584395] Call Trace:
[120408.584946] ? rsc_free+0x55/0x90 [auth_rpcgss]
[120408.585901] gss_proxy_save_rsc+0xb2/0x2a0 [auth_rpcgss]
[120408.587017] svcauth_gss_proxy_init+0x3cc/0x520 [auth_rpcgss]
[120408.588257] ? __enqueue_entity+0x6c/0x70
[120408.589101] svcauth_gss_accept+0x391/0xb90 [auth_rpcgss]
[120408.590212] ? try_to_wake_up+0x4a/0x360
[120408.591036] ? wake_up_process+0x15/0x20
[120408.592093] ? svc_xprt_do_enqueue+0x12e/0x2d0 [sunrpc]
[120408.593177] svc_authenticate+0xe1/0x100 [sunrpc]
[120408.594168] svc_process_common+0x203/0x710 [sunrpc]
[120408.595220] svc_process+0x105/0x1c0 [sunrpc]
[120408.596278] nfsd+0xe9/0x160 [nfsd]
[120408.597060] kthread+0x101/0x140
[120408.597734] ? nfsd_destroy+0x60/0x60 [nfsd]
[120408.598626] ? kthread_park+0x90/0x90
[120408.599448] ret_from_fork+0x22/0x30
[120408.600232] Code: 0f 1f 44 00 00 55 48 89 e5 53 48 89 fb 48 8b 7f
10 e8 9a 0a 9f e0 48 89 d8 5b 5d c3 0f 1f 40 00 0f 1f 44 00 00 48 85
ff 74 0e 55 <48> 8b 7f 10 48 89 e5 e8 d9 f6 9e e0 5d f3 c3 66 0f 1f 44
00 00
[120408.604123] RIP: gss_mech_put+0xb/0x20 [auth_rpcgss] RSP: ffffc90003d5b=
b78
[120408.607012] ---[ end trace 473ba7f734debac4 ]---
[120408.608027] Kernel panic - not syncing: Fatal exception
[120408.609268] Kernel Offset: disabled
[120408.610027] ---[ end Kernel panic - not syncing: Fatal exception


2017-01-30 21:02:41

by J. Bruce Fields

[permalink] [raw]
Subject: Re: upstream nfsd kernel oops in auth_rpcgss

On Mon, Jan 30, 2017 at 03:15:36PM -0500, Olga Kornievskaia wrote:
> Hi Bruce,
>
> I ran into this oops in the nfsd (below) (4.10-rc3 kernel). To trigger this I had a client (unsuccessfully) try to mount the server with krb5 where the server doesn’t have the rpcsec_gss_krb5 module built.

Whoops, thanks for catching that.

> Below code seems to fix things:
>
> diff --git a/net/sunrpc/auth_gss/svcauth_gss.c b/net/sunrpc/auth_gss/svcauth_gss.c
> index 886e9d38..7faffd8 100644
> --- a/net/sunrpc/auth_gss/svcauth_gss.c
> +++ b/net/sunrpc/auth_gss/svcauth_gss.c
> @@ -1187,9 +1187,9 @@ static int gss_proxy_save_rsc(struct cache_detail *cd,
> status = -EOPNOTSUPP;
> /* get mech handle from OID */
> gm = gss_mech_get_by_OID(&ud->mech_oid);
> + rsci.cred.cr_gss_mech = gm;
> if (!gm)
> goto out;
> - rsci.cred.cr_gss_mech = gm;
>
> status = -EINVAL;
> /* mech-specific data: */
>
> I guess the problem is because rsci.cred gets initialed from something that’s passed in that doesn’t have it’s cr_gss_mech initialized to NULL. So when gss_mech_get_by_OID() fails and it calls rsc_free() and tries to free a bad pointer.

Hm, yes, just above there's:

rsci.cred = ud->creds;

and it does make me a little nervous if ud->creds isn't completely
initialized.

The only caller of gss_proxy_save_rsc is svcauth_gss_proxy_init:

status = gss_proxy_save_rsc(sn->rsc_cache, &ud, &handle);

ud there is on the stack, and initialized by:

memset(&ud, 0, sizeof(ud));

Hm, in gssp_accept_sec_context_upcall, there's:

data->creds = *(struct svc_cred *)value->data;

That looks really weird. I'm not sure what it should be.

--b.


>
> Don’t know if this is acceptable or it should be traced to where the uninitialized pointer is coming from.
>
> If this approach is ok, I can re-send this as a patch.
>
> [120408.542387] general protection fault: 0000 [#1] SMP
> [120408.544273] Modules linked in: nfsd nfs_acl nfsv4 dns_resolver nfs lockd auth_rpcgss grace fscache sunrpc fuse ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack snd_seq_midi snd_seq_midi_event ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_mangle iptable_security iptable_raw rfcomm iptable_filter bnep ip_tables coretemp crc32_pclmul crc32c_intel ghash_clmulni_intel btusb btrtl btbcm btintel bluetooth rfkill snd_ens1371 snd_ac97_codec ppdev ac97_bus snd_seq aesni_intel crypto_simd cryptd glue_helper uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core snd_pcm pcspkr serio_raw
> [120408.558655] videodev snd_rawmidi snd_timer snd_seq_device snd soundcore vmw_vmci shpchp i2c_piix4 parport_pc parport acpi_cpufreq uinput xfs libcrc32c sr_mod sd_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix e1000 ahci libahci libata mptspi scsi_transport_spi mptscsih mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
> [120408.565724] CPU: 0 PID: 3601 Comm: nfsd Not tainted 4.10.0-rc3+ #16
> [120408.567037] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
> [120408.569225] task: ffff8800776f95c0 task.stack: ffffc90003d58000
> [120408.570483] RIP: 0010:gss_mech_put+0xb/0x20 [auth_rpcgss]
> [120408.571614] RSP: 0018:ffffc90003d5bb78 EFLAGS: 00010206
> [120408.572731] RAX: ffff880072ecc301 RBX: ffffc90003d5bba8 RCX: 00000000003d0040
> [120408.574240] RDX: 00000000003d003f RSI: 0000000000000246 RDI: 4f432e4553554f48
> [120408.575685] RBP: ffffc90003d5bb90 R08: 000000000001b2c0 R09: ffffffffa0703e0c
> [120408.577149] R10: ffff88007b61b2c0 R11: ffffea0001cbb300 R12: ffff8800730c1500
> [120408.578592] R13: 00000000ffffffa1 R14: ffff880073a94b40 R15: ffff880074e0d600
> [120408.580058] FS: 0000000000000000(0000) GS:ffff88007b600000(0000) knlGS:0000000000000000
> [120408.581708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [120408.582889] CR2: 00007fe3980140b8 CR3: 000000006f063000 CR4: 00000000001406f0
> [120408.584395] Call Trace:
> [120408.584946] ? rsc_free+0x55/0x90 [auth_rpcgss]
> [120408.585901] gss_proxy_save_rsc+0xb2/0x2a0 [auth_rpcgss]
> [120408.587017] svcauth_gss_proxy_init+0x3cc/0x520 [auth_rpcgss]
> [120408.588257] ? __enqueue_entity+0x6c/0x70
> [120408.589101] svcauth_gss_accept+0x391/0xb90 [auth_rpcgss]
> [120408.590212] ? try_to_wake_up+0x4a/0x360
> [120408.591036] ? wake_up_process+0x15/0x20
> [120408.592093] ? svc_xprt_do_enqueue+0x12e/0x2d0 [sunrpc]
> [120408.593177] svc_authenticate+0xe1/0x100 [sunrpc]
> [120408.594168] svc_process_common+0x203/0x710 [sunrpc]
> [120408.595220] svc_process+0x105/0x1c0 [sunrpc]
> [120408.596278] nfsd+0xe9/0x160 [nfsd]
> [120408.597060] kthread+0x101/0x140
> [120408.597734] ? nfsd_destroy+0x60/0x60 [nfsd]
> [120408.598626] ? kthread_park+0x90/0x90
> [120408.599448] ret_from_fork+0x22/0x30
> [120408.600232] Code: 0f 1f 44 00 00 55 48 89 e5 53 48 89 fb 48 8b 7f 10 e8 9a 0a 9f e0 48 89 d8 5b 5d c3 0f 1f 40 00 0f 1f 44 00 00 48 85 ff 74 0e 55 <48> 8b 7f 10 48 89 e5 e8 d9 f6 9e e0 5d f3 c3 66 0f 1f 44 00 00
> [120408.604123] RIP: gss_mech_put+0xb/0x20 [auth_rpcgss] RSP: ffffc90003d5bb78
> [120408.607012] ---[ end trace 473ba7f734debac4 ]---
> [120408.608027] Kernel panic - not syncing: Fatal exception
> [120408.609268] Kernel Offset: disabled
> [120408.610027] ---[ end Kernel panic - not syncing: Fatal exception
>

2017-01-30 21:20:04

by Olga Kornievskaia

[permalink] [raw]
Subject: Re: upstream nfsd kernel oops in auth_rpcgss

On Mon, Jan 30, 2017 at 4:02 PM, J. Bruce Fields <[email protected]> wrote=
:
> On Mon, Jan 30, 2017 at 03:15:36PM -0500, Olga Kornievskaia wrote:
>> Hi Bruce,
>>
>> I ran into this oops in the nfsd (below) (4.10-rc3 kernel). To trigger t=
his I had a client (unsuccessfully) try to mount the server with krb5 where=
the server doesn=E2=80=99t have the rpcsec_gss_krb5 module built.
>
> Whoops, thanks for catching that.
>
>> Below code seems to fix things:
>>
>> diff --git a/net/sunrpc/auth_gss/svcauth_gss.c b/net/sunrpc/auth_gss/svc=
auth_gss.c
>> index 886e9d38..7faffd8 100644
>> --- a/net/sunrpc/auth_gss/svcauth_gss.c
>> +++ b/net/sunrpc/auth_gss/svcauth_gss.c
>> @@ -1187,9 +1187,9 @@ static int gss_proxy_save_rsc(struct cache_detail =
*cd,
>> status =3D -EOPNOTSUPP;
>> /* get mech handle from OID */
>> gm =3D gss_mech_get_by_OID(&ud->mech_oid);
>> + rsci.cred.cr_gss_mech =3D gm;
>> if (!gm)
>> goto out;
>> - rsci.cred.cr_gss_mech =3D gm;
>>
>> status =3D -EINVAL;
>> /* mech-specific data: */
>>
>> I guess the problem is because rsci.cred gets initialed from something t=
hat=E2=80=99s passed in that doesn=E2=80=99t have it=E2=80=99s cr_gss_mech =
initialized to NULL. So when gss_mech_get_by_OID() fails and it calls rsc_f=
ree() and tries to free a bad pointer.
>
> Hm, yes, just above there's:
>
> rsci.cred =3D ud->creds;
>
> and it does make me a little nervous if ud->creds isn't completely
> initialized.
>
> The only caller of gss_proxy_save_rsc is svcauth_gss_proxy_init:
>
> status =3D gss_proxy_save_rsc(sn->rsc_cache, &ud, &handle);
>
> ud there is on the stack, and initialized by:
>
> memset(&ud, 0, sizeof(ud));
>
> Hm, in gssp_accept_sec_context_upcall, there's:
>
> data->creds =3D *(struct svc_cred *)value->data;
>
> That looks really weird. I'm not sure what it should be.

Isn't this stuff coming from the proxy upcall? How likely would it be
that it contains a pointer to something that points to the kernel
module? I used to know the svcgss upcall but that was before the
proxy. So I have no idea...


>> Don=E2=80=99t know if this is acceptable or it should be traced to where=
the uninitialized pointer is coming from.
>>
>> If this approach is ok, I can re-send this as a patch.
>>
>> [120408.542387] general protection fault: 0000 [#1] SMP
>> [120408.544273] Modules linked in: nfsd nfs_acl nfsv4 dns_resolver nfs l=
ockd auth_rpcgss grace fscache sunrpc fuse ip6t_rpfilter ip6t_REJECT nf_rej=
ect_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT nf_reject_ipv4 nf_conn=
track_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack snd_seq_midi snd_seq_mi=
di_event ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables =
ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables i=
ptable_mangle iptable_security iptable_raw rfcomm iptable_filter bnep ip_ta=
bles coretemp crc32_pclmul crc32c_intel ghash_clmulni_intel btusb btrtl btb=
cm btintel bluetooth rfkill snd_ens1371 snd_ac97_codec ppdev ac97_bus snd_s=
eq aesni_intel crypto_simd cryptd glue_helper uvcvideo videobuf2_vmalloc vi=
deobuf2_memops videobuf2_v4l2 videobuf2_core snd_pcm pcspkr serio_raw
>> [120408.558655] videodev snd_rawmidi snd_timer snd_seq_device snd sound=
core vmw_vmci shpchp i2c_piix4 parport_pc parport acpi_cpufreq uinput xfs l=
ibcrc32c sr_mod sd_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sy=
scopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix e1000 ahci lib=
ahci libata mptspi scsi_transport_spi mptscsih mptbase i2c_core dm_mirror d=
m_region_hash dm_log dm_mod
>> [120408.565724] CPU: 0 PID: 3601 Comm: nfsd Not tainted 4.10.0-rc3+ #16
>> [120408.567037] Hardware name: VMware, Inc. VMware Virtual Platform/440B=
X Desktop Reference Platform, BIOS 6.00 07/02/2015
>> [120408.569225] task: ffff8800776f95c0 task.stack: ffffc90003d58000
>> [120408.570483] RIP: 0010:gss_mech_put+0xb/0x20 [auth_rpcgss]
>> [120408.571614] RSP: 0018:ffffc90003d5bb78 EFLAGS: 00010206
>> [120408.572731] RAX: ffff880072ecc301 RBX: ffffc90003d5bba8 RCX: 0000000=
0003d0040
>> [120408.574240] RDX: 00000000003d003f RSI: 0000000000000246 RDI: 4f432e4=
553554f48
>> [120408.575685] RBP: ffffc90003d5bb90 R08: 000000000001b2c0 R09: fffffff=
fa0703e0c
>> [120408.577149] R10: ffff88007b61b2c0 R11: ffffea0001cbb300 R12: ffff880=
0730c1500
>> [120408.578592] R13: 00000000ffffffa1 R14: ffff880073a94b40 R15: ffff880=
074e0d600
>> [120408.580058] FS: 0000000000000000(0000) GS:ffff88007b600000(0000) kn=
lGS:0000000000000000
>> [120408.581708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [120408.582889] CR2: 00007fe3980140b8 CR3: 000000006f063000 CR4: 0000000=
0001406f0
>> [120408.584395] Call Trace:
>> [120408.584946] ? rsc_free+0x55/0x90 [auth_rpcgss]
>> [120408.585901] gss_proxy_save_rsc+0xb2/0x2a0 [auth_rpcgss]
>> [120408.587017] svcauth_gss_proxy_init+0x3cc/0x520 [auth_rpcgss]
>> [120408.588257] ? __enqueue_entity+0x6c/0x70
>> [120408.589101] svcauth_gss_accept+0x391/0xb90 [auth_rpcgss]
>> [120408.590212] ? try_to_wake_up+0x4a/0x360
>> [120408.591036] ? wake_up_process+0x15/0x20
>> [120408.592093] ? svc_xprt_do_enqueue+0x12e/0x2d0 [sunrpc]
>> [120408.593177] svc_authenticate+0xe1/0x100 [sunrpc]
>> [120408.594168] svc_process_common+0x203/0x710 [sunrpc]
>> [120408.595220] svc_process+0x105/0x1c0 [sunrpc]
>> [120408.596278] nfsd+0xe9/0x160 [nfsd]
>> [120408.597060] kthread+0x101/0x140
>> [120408.597734] ? nfsd_destroy+0x60/0x60 [nfsd]
>> [120408.598626] ? kthread_park+0x90/0x90
>> [120408.599448] ret_from_fork+0x22/0x30
>> [120408.600232] Code: 0f 1f 44 00 00 55 48 89 e5 53 48 89 fb 48 8b 7f 10=
e8 9a 0a 9f e0 48 89 d8 5b 5d c3 0f 1f 40 00 0f 1f 44 00 00 48 85 ff 74 0e=
55 <48> 8b 7f 10 48 89 e5 e8 d9 f6 9e e0 5d f3 c3 66 0f 1f 44 00 00
>> [120408.604123] RIP: gss_mech_put+0xb/0x20 [auth_rpcgss] RSP: ffffc90003=
d5bb78
>> [120408.607012] ---[ end trace 473ba7f734debac4 ]---
>> [120408.608027] Kernel panic - not syncing: Fatal exception
>> [120408.609268] Kernel Offset: disabled
>> [120408.610027] ---[ end Kernel panic - not syncing: Fatal exception
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2017-01-30 21:32:40

by J. Bruce Fields

[permalink] [raw]
Subject: Re: upstream nfsd kernel oops in auth_rpcgss

On Mon, Jan 30, 2017 at 04:02:37PM -0500, J. Bruce Fields wrote:
> On Mon, Jan 30, 2017 at 03:15:36PM -0500, Olga Kornievskaia wrote:
> > Hi Bruce,
> >
> > I ran into this oops in the nfsd (below) (4.10-rc3 kernel). To trigger this I had a client (unsuccessfully) try to mount the server with krb5 where the server doesn’t have the rpcsec_gss_krb5 module built.
>
> Whoops, thanks for catching that.
>
> > Below code seems to fix things:
> >
> > diff --git a/net/sunrpc/auth_gss/svcauth_gss.c b/net/sunrpc/auth_gss/svcauth_gss.c
> > index 886e9d38..7faffd8 100644
> > --- a/net/sunrpc/auth_gss/svcauth_gss.c
> > +++ b/net/sunrpc/auth_gss/svcauth_gss.c
> > @@ -1187,9 +1187,9 @@ static int gss_proxy_save_rsc(struct cache_detail *cd,
> > status = -EOPNOTSUPP;
> > /* get mech handle from OID */
> > gm = gss_mech_get_by_OID(&ud->mech_oid);
> > + rsci.cred.cr_gss_mech = gm;
> > if (!gm)
> > goto out;
> > - rsci.cred.cr_gss_mech = gm;
> >
> > status = -EINVAL;
> > /* mech-specific data: */
> >
> > I guess the problem is because rsci.cred gets initialed from something that’s passed in that doesn’t have it’s cr_gss_mech initialized to NULL. So when gss_mech_get_by_OID() fails and it calls rsc_free() and tries to free a bad pointer.
>
> Hm, yes, just above there's:
>
> rsci.cred = ud->creds;
>
> and it does make me a little nervous if ud->creds isn't completely
> initialized.
>
> The only caller of gss_proxy_save_rsc is svcauth_gss_proxy_init:
>
> status = gss_proxy_save_rsc(sn->rsc_cache, &ud, &handle);
>
> ud there is on the stack, and initialized by:
>
> memset(&ud, 0, sizeof(ud));
>
> Hm, in gssp_accept_sec_context_upcall, there's:
>
> data->creds = *(struct svc_cred *)value->data;
>
> That looks really weird. I'm not sure what it should be.

Does this do it?

--b.

diff --git a/net/sunrpc/auth_gss/gss_rpc_xdr.c b/net/sunrpc/auth_gss/gss_rpc_xdr.c
index dc6fb79a361f..25d9a9cf7b66 100644
--- a/net/sunrpc/auth_gss/gss_rpc_xdr.c
+++ b/net/sunrpc/auth_gss/gss_rpc_xdr.c
@@ -260,7 +260,7 @@ static int gssx_dec_option_array(struct xdr_stream *xdr,
if (!oa->data)
return -ENOMEM;

- creds = kmalloc(sizeof(struct svc_cred), GFP_KERNEL);
+ creds = kzalloc(sizeof(struct svc_cred), GFP_KERNEL);
if (!creds) {
kfree(oa->data);
return -ENOMEM;

2017-01-30 21:36:51

by Olga Kornievskaia

[permalink] [raw]
Subject: Re: upstream nfsd kernel oops in auth_rpcgss

On Mon, Jan 30, 2017 at 4:23 PM, J. Bruce Fields <[email protected]> wrote=
:
> On Mon, Jan 30, 2017 at 04:02:37PM -0500, J. Bruce Fields wrote:
>> On Mon, Jan 30, 2017 at 03:15:36PM -0500, Olga Kornievskaia wrote:
>> > Hi Bruce,
>> >
>> > I ran into this oops in the nfsd (below) (4.10-rc3 kernel). To trigger=
this I had a client (unsuccessfully) try to mount the server with krb5 whe=
re the server doesn=E2=80=99t have the rpcsec_gss_krb5 module built.
>>
>> Whoops, thanks for catching that.
>>
>> > Below code seems to fix things:
>> >
>> > diff --git a/net/sunrpc/auth_gss/svcauth_gss.c b/net/sunrpc/auth_gss/s=
vcauth_gss.c
>> > index 886e9d38..7faffd8 100644
>> > --- a/net/sunrpc/auth_gss/svcauth_gss.c
>> > +++ b/net/sunrpc/auth_gss/svcauth_gss.c
>> > @@ -1187,9 +1187,9 @@ static int gss_proxy_save_rsc(struct cache_detai=
l *cd,
>> > status =3D -EOPNOTSUPP;
>> > /* get mech handle from OID */
>> > gm =3D gss_mech_get_by_OID(&ud->mech_oid);
>> > + rsci.cred.cr_gss_mech =3D gm;
>> > if (!gm)
>> > goto out;
>> > - rsci.cred.cr_gss_mech =3D gm;
>> >
>> > status =3D -EINVAL;
>> > /* mech-specific data: */
>> >
>> > I guess the problem is because rsci.cred gets initialed from something=
that=E2=80=99s passed in that doesn=E2=80=99t have it=E2=80=99s cr_gss_mec=
h initialized to NULL. So when gss_mech_get_by_OID() fails and it calls rsc=
_free() and tries to free a bad pointer.
>>
>> Hm, yes, just above there's:
>>
>> rsci.cred =3D ud->creds;
>>
>> and it does make me a little nervous if ud->creds isn't completely
>> initialized.
>>
>> The only caller of gss_proxy_save_rsc is svcauth_gss_proxy_init:
>>
>> status =3D gss_proxy_save_rsc(sn->rsc_cache, &ud, &handle);
>>
>> ud there is on the stack, and initialized by:
>>
>> memset(&ud, 0, sizeof(ud));
>>
>> Hm, in gssp_accept_sec_context_upcall, there's:
>>
>> data->creds =3D *(struct svc_cred *)value->data;
>>
>> That looks really weird. I'm not sure what it should be.
>
> Does this do it?
>
> --b.
>
> diff --git a/net/sunrpc/auth_gss/gss_rpc_xdr.c b/net/sunrpc/auth_gss/gss_=
rpc_xdr.c
> index dc6fb79a361f..25d9a9cf7b66 100644
> --- a/net/sunrpc/auth_gss/gss_rpc_xdr.c
> +++ b/net/sunrpc/auth_gss/gss_rpc_xdr.c
> @@ -260,7 +260,7 @@ static int gssx_dec_option_array(struct xdr_stream *x=
dr,
> if (!oa->data)
> return -ENOMEM;
>
> - creds =3D kmalloc(sizeof(struct svc_cred), GFP_KERNEL);
> + creds =3D kzalloc(sizeof(struct svc_cred), GFP_KERNEL);
> if (!creds) {
> kfree(oa->data);
> return -ENOMEM;

Yes, it does!

> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2017-01-31 17:23:44

by J. Bruce Fields

[permalink] [raw]
Subject: Re: upstream nfsd kernel oops in auth_rpcgss

On Mon, Jan 30, 2017 at 04:36:50PM -0500, Olga Kornievskaia wrote:
> On Mon, Jan 30, 2017 at 4:23 PM, J. Bruce Fields <[email protected]> wrote:
> > Does this do it?
> >
> > diff --git a/net/sunrpc/auth_gss/gss_rpc_xdr.c b/net/sunrpc/auth_gss/gss_rpc_xdr.c
> > index dc6fb79a361f..25d9a9cf7b66 100644
> > --- a/net/sunrpc/auth_gss/gss_rpc_xdr.c
> > +++ b/net/sunrpc/auth_gss/gss_rpc_xdr.c
> > @@ -260,7 +260,7 @@ static int gssx_dec_option_array(struct xdr_stream *xdr,
> > if (!oa->data)
> > return -ENOMEM;
> >
> > - creds = kmalloc(sizeof(struct svc_cred), GFP_KERNEL);
> > + creds = kzalloc(sizeof(struct svc_cred), GFP_KERNEL);
> > if (!creds) {
> > kfree(oa->data);
> > return -ENOMEM;
>
> Yes, it does!

Applying for 4.10 as follows. Thanks!

--b.

commit 154f2f7cade0
Author: J. Bruce Fields <[email protected]>
Date: Tue Jan 31 11:37:50 2017 -0500

svcrpc: fix oops in absence of krb5 module

Olga Kornievskaia says: "I ran into this oops in the nfsd (below)
(4.10-rc3 kernel). To trigger this I had a client (unsuccessfully) try
to mount the server with krb5 where the server doesn't have the
rpcsec_gss_krb5 module built."

The problem is that rsci.cred is copied from a svc_cred structure that
gss_proxy didn't properly initialize. Fix that.

[120408.542387] general protection fault: 0000 [#1] SMP
...
[120408.565724] CPU: 0 PID: 3601 Comm: nfsd Not tainted 4.10.0-rc3+ #16
[120408.567037] Hardware name: VMware, Inc. VMware Virtual =
Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[120408.569225] task: ffff8800776f95c0 task.stack: ffffc90003d58000
[120408.570483] RIP: 0010:gss_mech_put+0xb/0x20 [auth_rpcgss]
...
[120408.584946] ? rsc_free+0x55/0x90 [auth_rpcgss]
[120408.585901] gss_proxy_save_rsc+0xb2/0x2a0 [auth_rpcgss]
[120408.587017] svcauth_gss_proxy_init+0x3cc/0x520 [auth_rpcgss]
[120408.588257] ? __enqueue_entity+0x6c/0x70
[120408.589101] svcauth_gss_accept+0x391/0xb90 [auth_rpcgss]
[120408.590212] ? try_to_wake_up+0x4a/0x360
[120408.591036] ? wake_up_process+0x15/0x20
[120408.592093] ? svc_xprt_do_enqueue+0x12e/0x2d0 [sunrpc]
[120408.593177] svc_authenticate+0xe1/0x100 [sunrpc]
[120408.594168] svc_process_common+0x203/0x710 [sunrpc]
[120408.595220] svc_process+0x105/0x1c0 [sunrpc]
[120408.596278] nfsd+0xe9/0x160 [nfsd]
[120408.597060] kthread+0x101/0x140
[120408.597734] ? nfsd_destroy+0x60/0x60 [nfsd]
[120408.598626] ? kthread_park+0x90/0x90
[120408.599448] ret_from_fork+0x22/0x30

Fixes: 1d658336b05f "SUNRPC: Add RPC based upcall mechanism for RPCGSS auth"
Cc: [email protected]
Cc: Simo Sorce <[email protected]>
Reported-by: Olga Kornievskaia <[email protected]>
Tested-by: Olga Kornievskaia <[email protected]>
Signed-off-by: J. Bruce Fields <[email protected]>

diff --git a/net/sunrpc/auth_gss/gss_rpc_xdr.c b/net/sunrpc/auth_gss/gss_rpc_xdr.c
index dc6fb79a361f..25d9a9cf7b66 100644
--- a/net/sunrpc/auth_gss/gss_rpc_xdr.c
+++ b/net/sunrpc/auth_gss/gss_rpc_xdr.c
@@ -260,7 +260,7 @@ static int gssx_dec_option_array(struct xdr_stream *xdr,
if (!oa->data)
return -ENOMEM;

- creds = kmalloc(sizeof(struct svc_cred), GFP_KERNEL);
+ creds = kzalloc(sizeof(struct svc_cred), GFP_KERNEL);
if (!creds) {
kfree(oa->data);
return -ENOMEM;