On Tue, 2012-10-23 at 13:57 -0400, Trond Myklebust wrote:
> On Tue, 2012-10-23 at 17:44 +0000, Myklebust, Trond wrote:
> > You can't hold a spinlock while sleeping. Both mutex_lock() and nsm_create() can definitely sleep.
> >
> > The correct way to do this is to grab the spinlock and recheck the value of ln->nsm_users inside the 'if (!IS_ERR())' condition. If it is still zero, bump it and set ln->nsm_clnt, otherwise bump it, get the existing ln->nsm_clnt and call rpc_shutdown_clnt() on the redundant nsm client after dropping the spinlock.
> >
> > Cheers
> > Trond
>
> Can you please check if the following patch fixes the issue?
>
> Cheers
> Trond
>
Meh... This one gets rid of the 100% redundant mutex...
8<-----------------------------------------------------------
>From 4187c816a15df12544ebcfa6b961fce96458e244 Mon Sep 17 00:00:00 2001
From: Trond Myklebust <[email protected]>
Date: Tue, 23 Oct 2012 13:51:58 -0400
Subject: [PATCH] LOCKD: fix races in nsm_client_get
Commit e9406db20fecbfcab646bad157b4cfdc7cadddfb (lockd: per-net
NSM client creation and destruction helpers introduced) contains
a nasty race on initialisation of the per-net NSM client because
it doesn't check whether or not the client is set after grabbing
the nsm_create_mutex.
Reported-by: Nix <[email protected]>
Signed-off-by: Trond Myklebust <[email protected]>
Cc: [email protected]
---
fs/lockd/mon.c | 43 ++++++++++++++++++++++++++-----------------
1 file changed, 26 insertions(+), 17 deletions(-)
diff --git a/fs/lockd/mon.c b/fs/lockd/mon.c
index e4fb3ba..fe69560 100644
--- a/fs/lockd/mon.c
+++ b/fs/lockd/mon.c
@@ -85,29 +85,38 @@ static struct rpc_clnt *nsm_create(struct net *net)
return rpc_create(&args);
}
+static struct rpc_clnt *nsm_client_set(struct lockd_net *ln,
+ struct rpc_clnt *clnt)
+{
+ spin_lock(&ln->nsm_clnt_lock);
+ if (ln->nsm_users == 0) {
+ if (clnt == NULL)
+ goto out;
+ ln->nsm_clnt = clnt;
+ }
+ clnt = ln->nsm_clnt;
+ ln->nsm_users++;
+out:
+ spin_unlock(&ln->nsm_clnt_lock);
+ return clnt;
+}
+
static struct rpc_clnt *nsm_client_get(struct net *net)
{
- static DEFINE_MUTEX(nsm_create_mutex);
- struct rpc_clnt *clnt;
+ struct rpc_clnt *clnt, *new;
struct lockd_net *ln = net_generic(net, lockd_net_id);
- spin_lock(&ln->nsm_clnt_lock);
- if (ln->nsm_users) {
- ln->nsm_users++;
- clnt = ln->nsm_clnt;
- spin_unlock(&ln->nsm_clnt_lock);
+ clnt = nsm_client_set(ln, NULL);
+ if (clnt != NULL)
goto out;
- }
- spin_unlock(&ln->nsm_clnt_lock);
- mutex_lock(&nsm_create_mutex);
- clnt = nsm_create(net);
- if (!IS_ERR(clnt)) {
- ln->nsm_clnt = clnt;
- smp_wmb();
- ln->nsm_users = 1;
- }
- mutex_unlock(&nsm_create_mutex);
+ clnt = new = nsm_create(net);
+ if (IS_ERR(clnt))
+ goto out;
+
+ clnt = nsm_client_set(ln, new);
+ if (clnt != new)
+ rpc_shutdown_client(new);
out:
return clnt;
}
--
1.7.11.7
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
http://www.netapp.com
????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay? ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m???? ????????I?
On 23 Oct 2012, Trond Myklebust outgrape:
> On Tue, 2012-10-23 at 13:57 -0400, Trond Myklebust wrote:
>> On Tue, 2012-10-23 at 17:44 +0000, Myklebust, Trond wrote:
>> > You can't hold a spinlock while sleeping. Both mutex_lock() and nsm_create() can definitely sleep.
>> >
>> > The correct way to do this is to grab the spinlock and recheck the value of ln->nsm_users inside the 'if (!IS_ERR())' condition. If it is still zero, bump it and set ln->nsm_clnt, otherwise bump it, get the existing ln->nsm_clnt and call rpc_shutdown_clnt() on the redundant nsm client after dropping the spinlock.
>> >
>> > Cheers
>> > Trond
>>
>> Can you please check if the following patch fixes the issue?
>>
>> Cheers
>> Trond
>>
> Meh... This one gets rid of the 100% redundant mutex...
No help, I'm afraid:
[ 894.005699] ------------[ cut here ]------------
[ 894.005929] kernel BUG at fs/lockd/mon.c:159!
[ 894.006156] invalid opcode: 0000 [#1] SMP
[ 894.006451] Modules linked in: firewire_ohci firewire_core [last unloaded: microcode]
[ 894.007005] CPU 1
[ 894.007050] Pid: 1035, comm: lockd Not tainted 3.6.3-dirty #1 empty empty/S7010
[ 894.007669] RIP: 0010:[<ffffffff8120fbbc>] [<ffffffff8120fbbc>] nsm_mon_unmon+0x64/0x98
[ 894.008126] RSP: 0018:ffff880620a23ce0 EFLAGS: 00010246
[ 894.008355] RAX: ffff880620a23ce8 RBX: 0000000000000000 RCX: 0000000000000000
[ 894.008591] RDX: ffff880620a23d58 RSI: 0000000000000002 RDI: ffff880620a23d30
[ 894.008827] RBP: ffff880620a23d40 R08: 0000000000000000 R09: ffffea00188e4f00
[ 894.009063] R10: ffffffff814d032f R11: 0000000000000020 R12: 0000000000000000
[ 894.009300] R13: ffff88061f067e40 R14: ffff88061f067ee8 R15: ffff88062393dc00
[ 894.009537] FS: 0000000000000000(0000) GS:ffff88063fc40000(0000) knlGS:0000000000000000
[ 894.009956] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 894.010187] CR2: 00007f056a9a6ff0 CR3: 0000000001a0b000 CR4: 00000000000027e0
[ 894.010422] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 894.010659] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 894.010896] Process lockd (pid: 1035, threadinfo ffff880620a22000, task ffff8806208b5900)
[ 894.011310] Stack:
[ 894.011528] 0000000000000010 ffff8806102d3db1 00000003000186b5 ffffffff00000010
[ 894.012083] ffff8806102d3dc1 000000000000008c 0000000000000000 ffff880620a23ce8
[ 894.012637] ffff880620a23d58 0000000000000000 ffff88061f067ee8 ffff8806102d3d00
[ 894.013190] Call Trace:
[ 894.013413] [<ffffffff8120ff07>] nsm_monitor+0x123/0x17e
[ 894.013645] [<ffffffff81211b72>] nlm4svc_retrieve_args+0x62/0xd7
[ 894.013879] [<ffffffff81211f71>] nlm4svc_proc_lock+0x3c/0xb5
[ 894.014112] [<ffffffff812116a3>] ? nlm4svc_decode_lockargs+0x47/0xb2
[ 894.014349] [<ffffffff814d89fa>] svc_process+0x3bf/0x6a1
[ 894.014581] [<ffffffff8120d5f0>] lockd+0x127/0x164
[ 894.014810] [<ffffffff8120d4c9>] ? set_grace_period+0x8a/0x8a
[ 894.015046] [<ffffffff8107bcbc>] kthread+0x8b/0x93
[ 894.015277] [<ffffffff81501334>] kernel_thread_helper+0x4/0x10
[ 894.015511] [<ffffffff8107bc31>] ? kthread_worker_fn+0xe1/0xe1
[ 894.015744] [<ffffffff81501330>] ? gs_change+0xb/0xb
[ 894.015972] Code: b8 10 00 00 00 48 89 45 c0 48 8d 81 8c 00 00 00 b9 08 00 00 00 48 89 45 c8 89 d8 f3 ab 48 8d 45 a8 48 89 55 e0 48 89 45 d8 75 02 <0f> 0b 89 f6 48 c7 02 00 00 00 00 4c 89 c7 48 6b f6 38 ba 00 04
[ 894.018895] RIP [<ffffffff8120fbbc>] nsm_mon_unmon+0x64/0x98
[ 894.019163] RSP <ffff880620a23ce0>
[ 894.019401] ---[ end trace b8ef5cb81bec72c8 ]---
Slightly different timing, but still boom.
This patch fixes two problems:
1) Removes races on NSM creation.
2) Fixes silly misprint on NSM client destruction (usage counter was checked
for non-zero value instead of zero).
Signed-off-by: Stanislav Kinsbursky <[email protected]>
---
fs/lockd/mon.c | 35 +++++++++++++++++++++++------------
1 files changed, 23 insertions(+), 12 deletions(-)
diff --git a/fs/lockd/mon.c b/fs/lockd/mon.c
index e4fb3ba..e3e59f6 100644
--- a/fs/lockd/mon.c
+++ b/fs/lockd/mon.c
@@ -85,30 +85,41 @@ static struct rpc_clnt *nsm_create(struct net *net)
return rpc_create(&args);
}
-static struct rpc_clnt *nsm_client_get(struct net *net)
+static struct rpc_clnt *nsm_get_client(struct net *net)
{
- static DEFINE_MUTEX(nsm_create_mutex);
- struct rpc_clnt *clnt;
+ struct rpc_clnt *clnt = NULL;
struct lockd_net *ln = net_generic(net, lockd_net_id);
spin_lock(&ln->nsm_clnt_lock);
if (ln->nsm_users) {
ln->nsm_users++;
clnt = ln->nsm_clnt;
- spin_unlock(&ln->nsm_clnt_lock);
- goto out;
}
spin_unlock(&ln->nsm_clnt_lock);
+ return clnt;
+}
+
+static struct rpc_clnt *nsm_client_get(struct net *net)
+{
+ static DEFINE_MUTEX(nsm_create_mutex);
+ struct rpc_clnt *clnt;
+ struct lockd_net *ln = net_generic(net, lockd_net_id);
+
+ clnt = nsm_get_client(net);
+ if (clnt)
+ return clnt;
mutex_lock(&nsm_create_mutex);
- clnt = nsm_create(net);
- if (!IS_ERR(clnt)) {
- ln->nsm_clnt = clnt;
- smp_wmb();
- ln->nsm_users = 1;
+ clnt = nsm_get_client(net);
+ if (clnt == NULL) {
+ clnt = nsm_create(net);
+ if (!IS_ERR(clnt)) {
+ ln->nsm_clnt = clnt;
+ smp_wmb();
+ ln->nsm_users = 1;
+ }
}
mutex_unlock(&nsm_create_mutex);
-out:
return clnt;
}
@@ -120,7 +131,7 @@ static void nsm_client_put(struct net *net)
spin_lock(&ln->nsm_clnt_lock);
if (ln->nsm_users) {
- if (--ln->nsm_users)
+ if (--ln->nsm_users == 0)
ln->nsm_clnt = NULL;
shutdown = !ln->nsm_users;
}
Hi,
the patch metioned in https://lkml.org/lkml/2012/10/24/175 seems to fix
the 3.6.3 oops (while 3.6.2 works fine) at 16-cores opteron server.
please queue this path for 3.6.$next.
BR,
Paweł.
[173788.113576] ------------[ cut here ]------------
[173788.133439] hrtimer: interrupt took 11004406 ns
[173788.157195] kernel BUG at fs/lockd/mon.c:150!
[173788.179641] invalid opcode: 0000 [#1] SMP
[173788.202033] Modules linked in: nfsv4 fuse nfsv3 nfs fscache nfsd auth_rpcgss nfs_acl lockd sunrpc ipmi_si ipmi_devintf ipmi_msghandler sch_sfq iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_filter xt_TCPMSS xt_tcpudp iptable_mangle ip_tables ip6table_filter ip6_tables x_tables quota_v2 quota_tree ext4 crc16 jbd2 raid10 raid0 dm_mod uvesafb autofs4 dummy ide_cd_mod cdrom ata_generic pata_acpi pata_atiixp sp5100_tco ide_pci_generic igb ptp pps_core psmouse k10temp mgag200 serio_raw dca pcspkr ttm powernow_k8 drm_kms_helper drm mperf freq_table kvm_amd evdev joydev i2c_piix4 kvm i2c_algo_bit hid_generic syscopyarea sysfillrect sysimgblt hwmon microcode atiixp amd64_edac_mod edac_core i2c_core ide_core processor edac_mce_amd button ext3 mbcache jbd sd_mod crc_t10dif raid1 md_mod
[173788.378811] ahci libahci libata scsi_mod usbhid hid ohci_hcd ehci_hcd usbcore usb_common
[173788.416270] CPU 2
[173788.416648] Pid: 1383, comm: lockd Not tainted 3.6.3 #11 Supermicro H8DGU/H8DGU
[173788.493500] RIP: 0010:[<ffffffffa04e64c0>] [<ffffffffa04e64c0>] nsm_mon_unmon+0x90/0xa0 [lockd]
[173788.529520] RSP: 0000:ffff8808093cdd00 EFLAGS: 00010246
[173788.565141] RAX: ffff8808093cdd28 RBX: ffff880ba2353200 RCX: 0000000000000000
[173788.601765] RDX: ffff8808093cdd68 RSI: 0000000000000002 RDI: ffff880ba2353200
[173788.638672] RBP: ffff8808093cdd50 R08: 00000000000168a0 R09: 000000000000ffff
[173788.675546] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880407db6c00
[173788.712500] R13: 0000000000000000 R14: ffff8808093cde28 R15: ffff8808093cde20
[173788.749767] FS: 00007f105fe73780(0000) GS:ffff88040fc80000(0000) knlGS:00000000f6663700
[173788.788015] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[173788.826367] CR2: 0000000000bce580 CR3: 000000044b252000 CR4: 00000000000007e0
[173788.865560] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[173788.904753] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[173788.943652] Process lockd (pid: 1383, threadinfo ffff8808093cc000, task ffff880808db3020)
[173788.983327] Stack:
[173789.022719] ffff8808093cdd60 ffffffffa04ae9e4 ffff8808093cdd28 ffff8808093cdd68
[173789.063923] 0000000000000000 ffff880ba23532b1 00000003000186b5 0000000400000010
[173789.105657] ffff880ba23532c1 000000000000008c ffff8808093cdd90 ffffffffa04e6821
[173789.148126] Call Trace:
[173789.190527] [<ffffffffa04ae9e4>] ? sunrpc_cache_lookup+0x74/0x2f0 [sunrpc]
[173789.233864] [<ffffffffa04e6821>] nsm_monitor+0xd1/0x1b0 [lockd]
[173789.277890] [<ffffffffa04e8d18>] nlm4svc_retrieve_args+0xa8/0xf0 [lockd]
[173789.322014] [<ffffffffa04e90c2>] nlm4svc_proc_lock+0x52/0xe0 [lockd]
[173789.366333] [<ffffffffa04e86c9>] ? nlm4svc_decode_lockargs+0x49/0xc0 [lockd]
[173789.411109] [<ffffffffa04a48d7>] svc_process+0x707/0x7a0 [sunrpc]
[173789.456179] [<ffffffffa04e3825>] lockd+0xa5/0x1b0 [lockd]
[173789.500017] [<ffffffffa04e3780>] ? set_grace_period+0xa0/0xa0 [lockd]
[173789.543446] [<ffffffff810726ce>] kthread+0x8e/0xa0
[173789.585890] [<ffffffff814af784>] kernel_thread_helper+0x4/0x10
[173789.628042] [<ffffffff81072640>] ? kthread_freezable_should_stop+0x70/0x70
[173789.670892] [<ffffffff814af780>] ? gs_change+0x13/0x13
[173789.713913] Code: 00 00 00 48 c1 e6 06 ba 00 04 00 00 48 29 c6 48 03 71 38 48 89 75 b8 48 8d 75 b8 e8 1b 3c fb ff 31 d2 85 c0 0f 4e d0 c9 89 d0 c3 <0f> 0b 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 fe b9
[173789.806212] RIP [<ffffffffa04e64c0>] nsm_mon_unmon+0x90/0xa0 [lockd]
[173789.851690] RSP <ffff8808093cdd00>
[173789.897665] ---[ end trace c8774e11cc39ecc3 ]---
On Wed, Oct 31, 2012 at 06:27:36PM +0100, Paweł Sikora wrote:
> Hi,
>
> the patch metioned in https://lkml.org/lkml/2012/10/24/175 seems to fix
> the 3.6.3 oops (while 3.6.2 works fine) at 16-cores opteron server.
> please queue this path for 3.6.$next.
Is it in Linus's tree already? If so, what is the git commit id?
thanks,
greg k-h
On Wednesday 31 of October 2012 10:49:46 Greg KH wrote:
> On Wed, Oct 31, 2012 at 06:27:36PM +0100, Paweł Sikora wrote:
> > Hi,
> >
> > the patch metioned in https://lkml.org/lkml/2012/10/24/175 seems to fix
> > the 3.6.3 oops (while 3.6.2 works fine) at 16-cores opteron server.
> > please queue this path for 3.6.$next.
>
> Is it in Linus's tree already? If so, what is the git commit id?
the mainstream contains some lock deamon fixes already:
* e498daa LOCKD: Clear ln->nsm_clnt only when ln->nsm_users is zero
* a4ee8d9 LOCKD: fix races in nsm_client_get
but i don't know where is the right fix. Stanislav, could you put some light on this?
BR,
Paweł.
Hi,
Greg KH wrote:
> On Wed, Oct 31, 2012 at 06:27:36PM +0100, Paweł Sikora wrote:
>> the patch metioned in https://lkml.org/lkml/2012/10/24/175 seems to fix
>> the 3.6.3 oops (while 3.6.2 works fine) at 16-cores opteron server.
>> please queue this path for 3.6.$next.
>
> Is it in Linus's tree already? If so, what is the git commit id?
One of
a4ee8d978e47 LOCKD: fix races in nsm_client_get
e498daa81295 LOCKD: Clear ln->nsm_clnt only when ln->nsm_users is zero
both of which were included in v3.6.5.
Hope that helps,
Jonathan
> -----Original Message-----
> From: [email protected] [mailto:linux-kernel-
> [email protected]] On Behalf Of Pawel Sikora
> Sent: Wednesday, October 31, 2012 2:03 PM
> To: Greg KH
> Cc: [email protected]; [email protected]; linux-
> [email protected]; [email protected]; [email protected]
> Subject: Re: [PATCH] lockd: fix races in per-net NSM client handling
>
> On Wednesday 31 of October 2012 10:49:46 Greg KH wrote:
> > On Wed, Oct 31, 2012 at 06:27:36PM +0100, Paweł Sikora wrote:
> > > Hi,
> > >
> > > the patch metioned in https://lkml.org/lkml/2012/10/24/175 seems to
> > > fix the 3.6.3 oops (while 3.6.2 works fine) at 16-cores opteron server.
> > > please queue this path for 3.6.$next.
> >
> > Is it in Linus's tree already? If so, what is the git commit id?
>
> the mainstream contains some lock deamon fixes already:
>
> * e498daa LOCKD: Clear ln->nsm_clnt only when ln->nsm_users is zero
> * a4ee8d9 LOCKD: fix races in nsm_client_get
>
> but i don't know where is the right fix. Stanislav, could you put some light on
> this?
The above 2 patches (which are already in 3.6.5) replace Stanislav's patch, which will not be merged upstream.
Trond
????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay? ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m???? ????????I?
On Wed, Oct 31, 2012 at 11:05:51AM -0700, Jonathan Nieder wrote:
> Hi,
>
> Greg KH wrote:
> > On Wed, Oct 31, 2012 at 06:27:36PM +0100, Paweł Sikora wrote:
>
> >> the patch metioned in https://lkml.org/lkml/2012/10/24/175 seems to fix
> >> the 3.6.3 oops (while 3.6.2 works fine) at 16-cores opteron server.
> >> please queue this path for 3.6.$next.
> >
> > Is it in Linus's tree already? If so, what is the git commit id?
>
> One of
>
> a4ee8d978e47 LOCKD: fix races in nsm_client_get
> e498daa81295 LOCKD: Clear ln->nsm_clnt only when ln->nsm_users is zero
>
> both of which were included in v3.6.5.
Ok, Paweł, does 3.6.5 work properly for you?
thanks,
greg k-h
On Wednesday 31 of October 2012 11:22:06 Greg KH wrote:
> On Wed, Oct 31, 2012 at 11:05:51AM -0700, Jonathan Nieder wrote:
> > Hi,
> >
> > Greg KH wrote:
> > > On Wed, Oct 31, 2012 at 06:27:36PM +0100, Paweł Sikora wrote:
> >
> > >> the patch metioned in https://lkml.org/lkml/2012/10/24/175 seems to fix
> > >> the 3.6.3 oops (while 3.6.2 works fine) at 16-cores opteron server.
> > >> please queue this path for 3.6.$next.
> > >
> > > Is it in Linus's tree already? If so, what is the git commit id?
> >
> > One of
> >
> > a4ee8d978e47 LOCKD: fix races in nsm_client_get
> > e498daa81295 LOCKD: Clear ln->nsm_clnt only when ln->nsm_users is zero
> >
> > both of which were included in v3.6.5.
>
> Ok, Paweł, does 3.6.5 work properly for you?
~12h uptime with full cpu/nfs load and all servers with 3.6.5 seem to work stable.
On Thu, Nov 01, 2012 at 07:54:21AM +0100, Paweł Sikora wrote:
> On Wednesday 31 of October 2012 11:22:06 Greg KH wrote:
> > On Wed, Oct 31, 2012 at 11:05:51AM -0700, Jonathan Nieder wrote:
> > > Hi,
> > >
> > > Greg KH wrote:
> > > > On Wed, Oct 31, 2012 at 06:27:36PM +0100, Paweł Sikora wrote:
> > >
> > > >> the patch metioned in https://lkml.org/lkml/2012/10/24/175 seems to fix
> > > >> the 3.6.3 oops (while 3.6.2 works fine) at 16-cores opteron server.
> > > >> please queue this path for 3.6.$next.
> > > >
> > > > Is it in Linus's tree already? If so, what is the git commit id?
> > >
> > > One of
> > >
> > > a4ee8d978e47 LOCKD: fix races in nsm_client_get
> > > e498daa81295 LOCKD: Clear ln->nsm_clnt only when ln->nsm_users is zero
> > >
> > > both of which were included in v3.6.5.
> >
> > Ok, Paweł, does 3.6.5 work properly for you?
>
> ~12h uptime with full cpu/nfs load and all servers with 3.6.5 seem to work stable.
Wonderful, thanks for testing.
greg k-h