As of 3b5cf20c "sunrpc: Use skcipher and ahash/shash", I get a NULL
dereference in krb5_encrypt on an NFS server when a client attempts to
mount using krb5. I haven't tried to figure out what's going on beyond
that....
--b.
[ 97.781559] IP: [<ffffffffa011d108>] krb5_encrypt+0x138/0x1f0 [rpcsec_gss_krb5]
[ 97.782100] PGD 0
[ 97.782257] Oops: 0000 [#1] PREEMPT SMP
[ 97.782646] Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
[ 97.783709] CPU: 0 PID: 4415 Comm: nfsd Not tainted 4.6.0-rc1-00029-gc05c2ec #489
[ 97.784015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.2-20150714_191134- 04/01/2014
[ 97.784015] task: ffff8800783b85c0 ti: ffff8800783bc000 task.ti: ffff8800783bc000
[ 97.784015] RIP: 0010:[<ffffffffa011d108>] [<ffffffffa011d108>] krb5_encrypt+0x138/0x1f0 [rpcsec_gss_krb5]
[ 97.784015] RSP: 0018:ffff8800783bf898 EFLAGS: 00010282
[ 97.784015] RAX: 0000000000000246 RBX: 1ffff1000f077f13 RCX: 0000000000000000
[ 97.784015] RDX: 00000000000000a0 RSI: ffffea0001af0502 RDI: ffff8800783bf898
[ 97.784015] RBP: ffff8800783bf950 R08: 0000000000000010 R09: ffff8800783bf908
[ 97.784015] R10: 0000000000000028 R11: ffff88007a53c000 R12: ffff8800783bf8f8
[ 97.784015] R13: ffff8800783bf898 R14: ffff8800783bf908 R15: ffff88007b2600a0
[ 97.784015] FS: 0000000000000000(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
[ 97.784015] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 97.784015] CR2: 0000000000000226 CR3: 0000000002006000 CR4: 00000000000406f0
[ 97.784015] Stack:
[ 97.784015] ffff880000000010 ffff8800783bf8f8 ffff8800783bf908 ffff8800783bf908
[ 97.784015] 0000000000000246 00000000024000c0 0000000000000000 0000000000000000
[ 97.784015] 0000000000000246 ffff880000000000 ffff88007c22e700 00000010783bf900
[ 97.784015] Call Trace:
[ 97.784015] [<ffffffffa011e93f>] krb5_derive_key+0x27f/0x360 [rpcsec_gss_krb5]
[ 97.784015] [<ffffffffa011aa02>] gss_import_sec_context_kerberos+0x852/0xd50 [rpcsec_gss_krb5]
[ 97.784015] [<ffffffffa00a12ad>] gss_import_sec_context+0x7d/0xa0 [auth_rpcgss]
[ 97.784015] [<ffffffffa00a211f>] gss_proxy_save_rsc+0x19f/0x230 [auth_rpcgss]
[ 97.784015] [<ffffffffa00a2853>] svcauth_gss_proxy_init+0x4d3/0x630 [auth_rpcgss]
[ 97.784015] [<ffffffffa00a2385>] ? svcauth_gss_proxy_init+0x5/0x630 [auth_rpcgss]
[ 97.784015] [<ffffffffa00a4874>] svcauth_gss_accept+0x584/0xd40 [auth_rpcgss]
[ 97.784015] [<ffffffffa00a47a4>] ? svcauth_gss_accept+0x4b4/0xd40 [auth_rpcgss]
[ 97.784015] [<ffffffffa0019257>] svc_authenticate+0xf7/0x100 [sunrpc]
[ 97.784015] [<ffffffffa001514d>] svc_process_common+0x1ed/0x630 [sunrpc]
[ 97.784015] [<ffffffffa00156c5>] svc_process+0x135/0x390 [sunrpc]
[ 97.784015] [<ffffffffa00b4811>] nfsd+0x181/0x280 [nfsd]
[ 97.784015] [<ffffffffa00b4695>] ? nfsd+0x5/0x280 [nfsd]
[ 97.784015] [<ffffffffa00b4690>] ? nfsd_destroy+0x190/0x190 [nfsd]
[ 97.784015] [<ffffffff810a186f>] kthread+0xef/0x110
[ 97.784015] [<ffffffff81ae00c2>] ret_from_fork+0x22/0x40
[ 97.784015] [<ffffffff810a1780>] ? kthread_create_on_node+0x200/0x200
[ 97.784015] Code: 38 00 00 00 00 00 00 00 c7 04 dd 48 00 00 00 00 00 00 00 44 89 04 dd 00 00 00 00 4c 89 34 dd 10 00 00 00 4c 89 34 dd 18 00 00 00 <ff> 50 e0 41 89 c4 48 8b 04 dd 40 00 00 00 4c 89 ef 8b 70 f4 48
[ 97.784015] RIP [<ffffffffa011d108>] krb5_encrypt+0x138/0x1f0 [rpcsec_gss_krb5]
[ 97.784015] RSP <ffff8800783bf898>
[ 97.784015] CR2: 0000000000000226
On Sat, Apr 02, 2016 at 08:13:19PM -0400, J. Bruce Fields wrote:
> As of 3b5cf20c "sunrpc: Use skcipher and ahash/shash", I get a NULL
> dereference in krb5_encrypt on an NFS server when a client attempts to
> mount using krb5. I haven't tried to figure out what's going on beyond
> that....
Oops, looks like I missed a couple of set_tfm calls.
---8<--
Subject: sunrpc: Add missing skcipher_request_set_tfm calls
A couple of skcipher_request users were missing set_tfm calls which
leads to a crash when they are called.
Fixes: 3b5cf20cf439 ("sunrpc: Use skcipher and ahash/shash")
Reported-by: J. Bruce Fields <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>
diff --git a/net/sunrpc/auth_gss/gss_krb5_crypto.c b/net/sunrpc/auth_gss/gss_krb5_crypto.c
index d94a8e1..ccc59aa 100644
--- a/net/sunrpc/auth_gss/gss_krb5_crypto.c
+++ b/net/sunrpc/auth_gss/gss_krb5_crypto.c
@@ -78,6 +78,7 @@ krb5_encrypt(
memcpy(out, in, length);
sg_init_one(sg, out, length);
+ skcipher_request_set_tfm(req, tfm);
skcipher_request_set_callback(req, 0, NULL, NULL);
skcipher_request_set_crypt(req, sg, sg, length, local_iv);
@@ -115,6 +116,7 @@ krb5_decrypt(
memcpy(out, in, length);
sg_init_one(sg, out, length);
+ skcipher_request_set_tfm(req, tfm);
skcipher_request_set_callback(req, 0, NULL, NULL);
skcipher_request_set_crypt(req, sg, sg, length, local_iv);
--
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Sun, Apr 03, 2016 at 09:09:52AM +0800, Herbert Xu wrote:
> On Sat, Apr 02, 2016 at 08:13:19PM -0400, J. Bruce Fields wrote:
> > As of 3b5cf20c "sunrpc: Use skcipher and ahash/shash", I get a NULL
> > dereference in krb5_encrypt on an NFS server when a client attempts to
> > mount using krb5. I haven't tried to figure out what's going on beyond
> > that....
>
> Oops, looks like I missed a couple of set_tfm calls.
Thanks. It's getting further now, but appears to be freezing later.
Possibly unrelated. I'm travelling, and it'll be Monday or Wednesday
till I can take another look.
--b.
>
> ---8<--
> Subject: sunrpc: Add missing skcipher_request_set_tfm calls
>
> A couple of skcipher_request users were missing set_tfm calls which
> leads to a crash when they are called.
>
> Fixes: 3b5cf20cf439 ("sunrpc: Use skcipher and ahash/shash")
> Reported-by: J. Bruce Fields <[email protected]>
> Signed-off-by: Herbert Xu <[email protected]>
>
> diff --git a/net/sunrpc/auth_gss/gss_krb5_crypto.c b/net/sunrpc/auth_gss/gss_krb5_crypto.c
> index d94a8e1..ccc59aa 100644
> --- a/net/sunrpc/auth_gss/gss_krb5_crypto.c
> +++ b/net/sunrpc/auth_gss/gss_krb5_crypto.c
> @@ -78,6 +78,7 @@ krb5_encrypt(
> memcpy(out, in, length);
> sg_init_one(sg, out, length);
>
> + skcipher_request_set_tfm(req, tfm);
> skcipher_request_set_callback(req, 0, NULL, NULL);
> skcipher_request_set_crypt(req, sg, sg, length, local_iv);
>
> @@ -115,6 +116,7 @@ krb5_decrypt(
> memcpy(out, in, length);
> sg_init_one(sg, out, length);
>
> + skcipher_request_set_tfm(req, tfm);
> skcipher_request_set_callback(req, 0, NULL, NULL);
> skcipher_request_set_crypt(req, sg, sg, length, local_iv);
>
> --
> Email: Herbert Xu <[email protected]>
> Home Page: http://gondor.apana.org.au/~herbert/
> PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Sat, Apr 02, 2016 at 11:59:00PM -0400, J. Bruce Fields wrote:
>
> Thanks. It's getting further now, but appears to be freezing later.
> Possibly unrelated. I'm travelling, and it'll be Monday or Wednesday
> till I can take another look.
Thanks for the update. I've found another bug in the hash conversion
that causes memory corruption which may lead to your hang.
Here's a patch with the previous fix plus the new hash fixes.
---8<---
The skcpiher/shash conversion introduced a number of bugs in the
sunrpc code:
1) Missing calls to skcipher_request_set_tfm lead to crashes.
2) The allocation size of shash_desc is too small which leads to
memory corruption.
Fixes: 3b5cf20cf439 ("sunrpc: Use skcipher and ahash/shash")
Reported-by: J. Bruce Fields <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>
diff --git a/net/sunrpc/auth_gss/gss_krb5_crypto.c b/net/sunrpc/auth_gss/gss_krb5_crypto.c
index d94a8e1..da26455 100644
--- a/net/sunrpc/auth_gss/gss_krb5_crypto.c
+++ b/net/sunrpc/auth_gss/gss_krb5_crypto.c
@@ -78,6 +78,7 @@ krb5_encrypt(
memcpy(out, in, length);
sg_init_one(sg, out, length);
+ skcipher_request_set_tfm(req, tfm);
skcipher_request_set_callback(req, 0, NULL, NULL);
skcipher_request_set_crypt(req, sg, sg, length, local_iv);
@@ -115,6 +116,7 @@ krb5_decrypt(
memcpy(out, in, length);
sg_init_one(sg, out, length);
+ skcipher_request_set_tfm(req, tfm);
skcipher_request_set_callback(req, 0, NULL, NULL);
skcipher_request_set_crypt(req, sg, sg, length, local_iv);
@@ -946,7 +948,8 @@ krb5_rc4_setup_seq_key(struct krb5_ctx *kctx, struct crypto_skcipher *cipher,
return PTR_ERR(hmac);
}
- desc = kmalloc(sizeof(*desc), GFP_KERNEL);
+ desc = kmalloc(sizeof(*desc) + crypto_shash_descsize(hmac),
+ GFP_KERNEL);
if (!desc) {
dprintk("%s: failed to allocate shash descriptor for '%s'\n",
__func__, kctx->gk5e->cksum_name);
@@ -1012,7 +1015,8 @@ krb5_rc4_setup_enc_key(struct krb5_ctx *kctx, struct crypto_skcipher *cipher,
return PTR_ERR(hmac);
}
- desc = kmalloc(sizeof(*desc), GFP_KERNEL);
+ desc = kmalloc(sizeof(*desc) + crypto_shash_descsize(hmac),
+ GFP_KERNEL);
if (!desc) {
dprintk("%s: failed to allocate shash descriptor for '%s'\n",
__func__, kctx->gk5e->cksum_name);
diff --git a/net/sunrpc/auth_gss/gss_krb5_mech.c b/net/sunrpc/auth_gss/gss_krb5_mech.c
index 71341cc..6542749 100644
--- a/net/sunrpc/auth_gss/gss_krb5_mech.c
+++ b/net/sunrpc/auth_gss/gss_krb5_mech.c
@@ -451,7 +451,8 @@ context_derive_keys_rc4(struct krb5_ctx *ctx)
goto out_err_free_hmac;
- desc = kmalloc(sizeof(*desc), GFP_KERNEL);
+ desc = kmalloc(sizeof(*desc) + crypto_shash_descsize(hmac),
+ GFP_KERNEL);
if (!desc) {
dprintk("%s: failed to allocate hash descriptor for '%s'\n",
__func__, ctx->gk5e->cksum_name);
--
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Sun, Apr 03, 2016 at 12:37:15PM +0800, Herbert Xu wrote:
> On Sat, Apr 02, 2016 at 11:59:00PM -0400, J. Bruce Fields wrote:
> >
> > Thanks. It's getting further now, but appears to be freezing later.
> > Possibly unrelated. I'm travelling, and it'll be Monday or Wednesday
> > till I can take another look.
>
> Thanks for the update. I've found another bug in the hash conversion
> that causes memory corruption which may lead to your hang.
>
> Here's a patch with the previous fix plus the new hash fixes.
OK, I did get a chance to run this, and so far it looks good--it got
faszter than the last time, anyway. Thanks!
For some reason, the original didn't appear to get cc'd to the linux-nfs
list. Or did it, and I missed it? I do get lazy sometimes, but in
general something like this I'd at least grab and run some tests on.
Especially if there's a git tree I can grab, then it just takes me a
minute to kick off.
--b.
>
> ---8<---
> The skcpiher/shash conversion introduced a number of bugs in the
> sunrpc code:
>
> 1) Missing calls to skcipher_request_set_tfm lead to crashes.
> 2) The allocation size of shash_desc is too small which leads to
> memory corruption.
>
> Fixes: 3b5cf20cf439 ("sunrpc: Use skcipher and ahash/shash")
> Reported-by: J. Bruce Fields <[email protected]>
> Signed-off-by: Herbert Xu <[email protected]>
>
> diff --git a/net/sunrpc/auth_gss/gss_krb5_crypto.c b/net/sunrpc/auth_gss/gss_krb5_crypto.c
> index d94a8e1..da26455 100644
> --- a/net/sunrpc/auth_gss/gss_krb5_crypto.c
> +++ b/net/sunrpc/auth_gss/gss_krb5_crypto.c
> @@ -78,6 +78,7 @@ krb5_encrypt(
> memcpy(out, in, length);
> sg_init_one(sg, out, length);
>
> + skcipher_request_set_tfm(req, tfm);
> skcipher_request_set_callback(req, 0, NULL, NULL);
> skcipher_request_set_crypt(req, sg, sg, length, local_iv);
>
> @@ -115,6 +116,7 @@ krb5_decrypt(
> memcpy(out, in, length);
> sg_init_one(sg, out, length);
>
> + skcipher_request_set_tfm(req, tfm);
> skcipher_request_set_callback(req, 0, NULL, NULL);
> skcipher_request_set_crypt(req, sg, sg, length, local_iv);
>
> @@ -946,7 +948,8 @@ krb5_rc4_setup_seq_key(struct krb5_ctx *kctx, struct crypto_skcipher *cipher,
> return PTR_ERR(hmac);
> }
>
> - desc = kmalloc(sizeof(*desc), GFP_KERNEL);
> + desc = kmalloc(sizeof(*desc) + crypto_shash_descsize(hmac),
> + GFP_KERNEL);
> if (!desc) {
> dprintk("%s: failed to allocate shash descriptor for '%s'\n",
> __func__, kctx->gk5e->cksum_name);
> @@ -1012,7 +1015,8 @@ krb5_rc4_setup_enc_key(struct krb5_ctx *kctx, struct crypto_skcipher *cipher,
> return PTR_ERR(hmac);
> }
>
> - desc = kmalloc(sizeof(*desc), GFP_KERNEL);
> + desc = kmalloc(sizeof(*desc) + crypto_shash_descsize(hmac),
> + GFP_KERNEL);
> if (!desc) {
> dprintk("%s: failed to allocate shash descriptor for '%s'\n",
> __func__, kctx->gk5e->cksum_name);
> diff --git a/net/sunrpc/auth_gss/gss_krb5_mech.c b/net/sunrpc/auth_gss/gss_krb5_mech.c
> index 71341cc..6542749 100644
> --- a/net/sunrpc/auth_gss/gss_krb5_mech.c
> +++ b/net/sunrpc/auth_gss/gss_krb5_mech.c
> @@ -451,7 +451,8 @@ context_derive_keys_rc4(struct krb5_ctx *ctx)
> goto out_err_free_hmac;
>
>
> - desc = kmalloc(sizeof(*desc), GFP_KERNEL);
> + desc = kmalloc(sizeof(*desc) + crypto_shash_descsize(hmac),
> + GFP_KERNEL);
> if (!desc) {
> dprintk("%s: failed to allocate hash descriptor for '%s'\n",
> __func__, ctx->gk5e->cksum_name);
> --
> Email: Herbert Xu <[email protected]>
> Home Page: http://gondor.apana.org.au/~herbert/
> PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Sun, Apr 03, 2016 at 06:15:43PM -0400, J. Bruce Fields wrote:
>
> OK, I did get a chance to run this, and so far it looks good--it got
> faszter than the last time, anyway. Thanks!
Thanks!
> For some reason, the original didn't appear to get cc'd to the linux-nfs
> list. Or did it, and I missed it? I do get lazy sometimes, but in
> general something like this I'd at least grab and run some tests on.
> Especially if there's a git tree I can grab, then it just takes me a
> minute to kick off.
I'm pretty sure it did get to linux-nfs, or at least the archive :)
https://www.spinics.net/lists/linux-nfs/msg56240.html
Cheers,
--
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Mon, Apr 04, 2016 at 09:22:02AM +0800, Herbert Xu wrote:
> On Sun, Apr 03, 2016 at 06:15:43PM -0400, J. Bruce Fields wrote:
> > For some reason, the original didn't appear to get cc'd to the linux-nfs
> > list. Or did it, and I missed it? I do get lazy sometimes, but in
> > general something like this I'd at least grab and run some tests on.
> > Especially if there's a git tree I can grab, then it just takes me a
> > minute to kick off.
>
> I'm pretty sure it did get to linux-nfs, or at least the archive :)
>
> https://www.spinics.net/lists/linux-nfs/msg56240.html
D'oh. I was probably just lame, then. Thanks for the fix. Feel free
to add my tested-by: if you want.
Hm, now I'm seeing list corruption in the rpc code on callbacks....
That's almost certainly unrelated to this, though.
--b.