Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751360Ab2E1XoC (ORCPT ); Mon, 28 May 2012 19:44:02 -0400 Received: from mx2.netapp.com ([216.240.18.37]:55921 "EHLO mx2.netapp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750990Ab2E1Xn7 (ORCPT ); Mon, 28 May 2012 19:43:59 -0400 X-IronPort-AV: E=Sophos;i="4.75,674,1330934400"; d="scan'208";a="651514468" From: "Myklebust, Trond" To: Stanislav Kinsbursky CC: Dave Jones , "bfields@fieldses.org" , "linux-nfs@vger.kernel.org" , Linux Kernel Subject: Re: 3.4. sunrpc oops during shutdown Thread-Topic: 3.4. sunrpc oops during shutdown Thread-Index: AQHNN3wvtXMDx9rJ3kyqfagGDPb51pbaoqcAgABSZwCAAAbFAIAFYfSA Date: Mon, 28 May 2012 23:43:40 +0000 Message-ID: <1338248617.3057.5.camel@lade.trondhjem.org> References: <20120521171411.GA15562@redhat.com> <1337623475.9583.13.camel@lade.trondhjem.org> <4FBF3EF2.4090804@parallels.com> <1337951250.2535.5.camel@lade.trondhjem.org> <4FBF89C0.2030805@parallels.com> In-Reply-To: <4FBF89C0.2030805@parallels.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.104.60.115] Content-Type: text/plain; charset="utf-8" Content-ID: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by nfs id q4SNidlM023687 Content-Length: 5964 Lines: 117 On Fri, 2012-05-25 at 17:31 +0400, Stanislav Kinsbursky wrote: > On 25.05.2012 17:07, Myklebust, Trond wrote: > > On Fri, 2012-05-25 at 12:12 +0400, Stanislav Kinsbursky wrote: > >> On 21.05.2012 22:03, Myklebust, Trond wrote: > >>> On Mon, 2012-05-21 at 13:14 -0400, Dave Jones wrote: > >>>> Tried to shutdown a machine, got this, and a bunch of hung processes. > >>>> There was one NFS mount mounted at the time. > >>>> > >>>> Dave > >>>> > >>>> BUG: unable to handle kernel NULL pointer dereference at 0000000000000028 > >>>> IP: [] svc_destroy+0x1f/0x140 [sunrpc] > >>>> PGD 1434c4067 PUD 144964067 PMD 0 > >>>> Oops: 0000 [#1] PREEMPT SMP > >>>> CPU 4 > >>>> Modules linked in: ip6table_filter(-) ip6_tables nfsd nfs fscache auth_rpcgss nfs_acl lockd ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 > >>>> > >>>> Pid: 6946, comm: ntpd Not tainted 3.4.0+ #13 > >>>> RIP: 0010:[] [] svc_destroy+0x1f/0x140 [sunrpc] > >>>> RSP: 0018:ffff880143c65c48 EFLAGS: 00010286 > >>>> RAX: 0000000000000000 RBX: ffff880142cd41a0 RCX: 0000000000000006 > >>>> RDX: 0000000000000040 RSI: ffff880143105028 RDI: ffff880142cd41a0 > >>>> RBP: ffff880143c65c58 R08: 0000000000000000 R09: 0000000000000001 > >>>> R10: 0000000000000000 R11: 0000000000000000 R12: ffff88013bc5a148 > >>>> R13: ffff880140981658 R14: ffff880142cd41a0 R15: ffff880146c88000 > >>>> FS: 00007fdc0382a740(0000) GS:ffff880149400000(0000) knlGS:0000000000000000 > >>>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > >>>> CR2: 0000000000000028 CR3: 0000000036cbb000 CR4: 00000000001407e0 > >>>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > >>>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > >>>> Process ntpd (pid: 6946, threadinfo ffff880143c64000, task ffff880143104940) > >>>> Stack: > >>>> ffff880140981660 ffff88013bc5a148 ffff880143c65c88 ffffffffa01193a6 > >>>> 0000000000000000 ffff88013e566020 ffff88013e565f28 ffff880146ee6ac0 > >>>> ffff880143c65ca8 ffffffffa024f403 ffff880143c65ca8 ffff880143d3a4f8 > >>>> Call Trace: > >>>> [] svc_exit_thread+0xa6/0xb0 [sunrpc] > >>>> [] nfs_callback_down+0x53/0x90 [nfs] > >>>> [] nfs_free_client+0xfe/0x120 [nfs] > >>>> [] nfs_put_client+0x29f/0x420 [nfs] > >>>> [] ? nfs_put_client+0x1a0/0x420 [nfs] > >>>> [] nfs_free_server+0x16f/0x2e0 [nfs] > >>>> [] ? nfs_free_server+0x23/0x2e0 [nfs] > >>>> [] nfs4_kill_super+0x3c/0x50 [nfs] > >>>> [] deactivate_locked_super+0x3c/0xa0 > >>>> [] deactivate_super+0x4e/0x70 > >>>> [] mntput_no_expire+0xb4/0x100 > >>>> [] mntput+0x26/0x40 > >>>> [] release_mounts+0x77/0x90 > >>>> [] put_mnt_ns+0x66/0x80 > >>>> [] free_nsproxy+0x1f/0xb0 > >>>> [] switch_task_namespaces+0x5e/0x70 > >>>> [] exit_task_namespaces+0x10/0x20 > >>>> [] do_exit+0x4ee/0xb80 > >>>> [] ? retint_swapgs+0xe/0x13 > >>>> [] do_group_exit+0x4f/0xc0 > >>>> [] sys_exit_group+0x17/0x20 > >>>> [] system_call_fastpath+0x16/0x1b > >>>> Code: 48 8b 5d f0 4c 8b 65 f8 c9 c3 66 90 55 48 89 e5 41 54 53 66 66 66 66 90 65 48 8b 04 25 80 ba 00 00 48 8b 80 50 05 00 00 48 89 fb<4c> 8b 60 28 8b 47 58 85 c0 0f 84 ec 00 00 00 83 e8 01 85 c0 89 > >>> > >>> Aside from the fact that the current net_namespace is not guaranteed to > >>> exist when we are called from free_nsproxy, svc_destroy() looks > >>> seriously broken: > >> > >> Trond, looks like you are mistaken here. > >> Any process holds references to all namespaces it belong to (copy_net_ns() > >> increase usage counter). And network namespace is released after mount namespace > >> in free_nsproxy. > > > > That doesn't help you though. switch_task_namespaces will have already > > set current->nsproxy to NULL, which is why we Oops when we try to read > > current->nsproxy->net_ns in svc_exit_thread(). > > > >>> > >>> * On the one hand it is trying to free struct svc_serv (and > >>> presumably all structures owned by struct svc_serv). > >>> * On the other hand, it tries to pass a parameter to > >>> svc_close_net() saying "please don't free structures on my > >>> sv_tempsocks, or sv_permsocks list unless they match this net > >>> namespace". > >>> > >> > >> I've sent patches, which moves svc_shutdown_net() from svc_destroy() ("SUNRPC: > >> separate per-net data creation from service"). > >> with this patch set it's assumed, that per-net resources will be created or > >> released prior to service creation and destruction. > > > > Are those patches appropriate for inclusion in the stable kernel series > > so that we can fix 3.4? > > > > Yes. But unfortunately, this won't be enough. > "NFS: callback threads containerization" patch set is required as well. > > A a bugfix, I can suggest "SUNRPC: separate per-net data creation from service" > patch set + pass hard-coded "init_net" for NFS callback shutdown routines > (instead of current->nsproxy->net_ns). This should work. Hi Stanislav, My question is why should svc_destroy() care about net namespaces at all? Once an application is calling svc_destroy(), it is trying to close down the entire service. It really should not matter to which net namespace a particular socket belongs: they _all_ need to be destroyed. Cheers, Trond -- Trond Myklebust Linux NFS client maintainer NetApp Trond.Myklebust@netapp.com www.netapp.com ????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay?ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m????????????I?