From: Tao Ma <[email protected]>
In fa0d7e3, we use rcu free inode instead of freeing the inode
directly. It causes a problem when we rmmod immediately after
we umount the volume[1].
So we need to call synchronize_rcu after we kill_sb so that
the inode is freed before we do rmmod. The idea is inspired
by Chris Mason[2]. I tested with ext4 by umount+rmmod and it
doesn't show any error by now.
1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2
Cc: Nick Piggin <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Chris Mason <[email protected]>
Cc: Boaz Harrosh <[email protected]>
Signed-off-by: Tao Ma <[email protected]>
---
fs/super.c | 7 +++++++
1 files changed, 7 insertions(+), 0 deletions(-)
diff --git a/fs/super.c b/fs/super.c
index 74e149e..315bce9 100644
--- a/fs/super.c
+++ b/fs/super.c
@@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)
struct file_system_type *fs = s->s_type;
if (atomic_dec_and_test(&s->s_active)) {
fs->kill_sb(s);
+ /*
+ * We need to synchronize rcu here so that
+ * the delayed rcu inode free can be executed
+ * before we put_super.
+ * https://bugzilla.kernel.org/show_bug.cgi?id=27652
+ */
+ synchronize_rcu();
put_filesystem(fs);
put_super(s);
} else {
--
1.6.3.GIT
On 02/05/2011 11:01 AM, Tao Ma wrote:
> From: Tao Ma <[email protected]>
>
> In fa0d7e3, we use rcu free inode instead of freeing the inode
> directly. It causes a problem when we rmmod immediately after
> we umount the volume[1].
>
> So we need to call synchronize_rcu after we kill_sb so that
> the inode is freed before we do rmmod. The idea is inspired
> by Chris Mason[2]. I tested with ext4 by umount+rmmod and it
> doesn't show any error by now.
>
> 1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
> 2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2
>
> Cc: Nick Piggin <[email protected]>
> Cc: Al Viro <[email protected]>
> Cc: Chris Mason <[email protected]>
> Cc: Boaz Harrosh <[email protected]>
> Signed-off-by: Tao Ma <[email protected]>
> ---
> fs/super.c | 7 +++++++
> 1 files changed, 7 insertions(+), 0 deletions(-)
>
> diff --git a/fs/super.c b/fs/super.c
> index 74e149e..315bce9 100644
> --- a/fs/super.c
> +++ b/fs/super.c
> @@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)
> struct file_system_type *fs = s->s_type;
> if (atomic_dec_and_test(&s->s_active)) {
> fs->kill_sb(s);
> + /*
> + * We need to synchronize rcu here so that
> + * the delayed rcu inode free can be executed
> + * before we put_super.
> + * https://bugzilla.kernel.org/show_bug.cgi?id=27652
> + */
> + synchronize_rcu();
> put_filesystem(fs);
> put_super(s);
> } else {
Sorry for not testing sooner.
The above does not work I still get the exact same crash!!
Looking at the code for synchronize_rcu() it looks like it might not be
enough. It looks like all it does is a memory barrier. But we need
something that will actually pump these pending releases.
(I might be way off here)
BTW after I get the Warning from the kmem_cache_destroy:
slab error in kmem_cache_destroy(): cache `exofs_inode_cache': Can't free all objects
Call Trace:
754efe08: [<6007e9a6>] kmem_cache_destroy+0x82/0xca
754efe38: [<7a9296ba>] exit_exofs+0x1a/0x1c [exofs]
754efe48: [<60054c10>] sys_delete_module+0x1b9/0x217
754efee8: [<60014d60>] handle_syscall+0x58/0x70
754eff08: [<60024163>] userspace+0x2dd/0x38a
754effc8: [<600126af>] fork_handler+0x62/0x69
I also get a Kernel crash. I suspect it's when finally these
free_rcu come and the module (and kmem_cache) are no longer there.
What to do? Nick?
Boaz
On Mon, 07 Feb 2011 15:14:40 +0200, Boaz Harrosh <[email protected]> wrote:
> On 02/05/2011 11:01 AM, Tao Ma wrote:
> > From: Tao Ma <[email protected]>
> >
> > In fa0d7e3, we use rcu free inode instead of freeing the inode
> > directly. It causes a problem when we rmmod immediately after
> > we umount the volume[1].
> >
> > So we need to call synchronize_rcu after we kill_sb so that
> > the inode is freed before we do rmmod. The idea is inspired
> > by Chris Mason[2]. I tested with ext4 by umount+rmmod and it
> > doesn't show any error by now.
> >
> > 1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
> > 2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2
> >
> > Cc: Nick Piggin <[email protected]>
> > Cc: Al Viro <[email protected]>
> > Cc: Chris Mason <[email protected]>
> > Cc: Boaz Harrosh <[email protected]>
> > Signed-off-by: Tao Ma <[email protected]>
> > ---
> > fs/super.c | 7 +++++++
> > 1 files changed, 7 insertions(+), 0 deletions(-)
> >
> > diff --git a/fs/super.c b/fs/super.c
> > index 74e149e..315bce9 100644
> > --- a/fs/super.c
> > +++ b/fs/super.c
> > @@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)
> > struct file_system_type *fs = s->s_type;
> > if (atomic_dec_and_test(&s->s_active)) {
> > fs->kill_sb(s);
> > + /*
> > + * We need to synchronize rcu here so that
> > + * the delayed rcu inode free can be executed
> > + * before we put_super.
> > + * https://bugzilla.kernel.org/show_bug.cgi?id=27652
> > + */
> > + synchronize_rcu();
> > put_filesystem(fs);
> > put_super(s);
> > } else {
>
>
> Sorry for not testing sooner.
>
> The above does not work I still get the exact same crash!!
>
> Looking at the code for synchronize_rcu() it looks like it might not be
> enough. It looks like all it does is a memory barrier. But we need
> something that will actually pump these pending releases.
> (I might be way off here)
>
> BTW after I get the Warning from the kmem_cache_destroy:
> slab error in kmem_cache_destroy(): cache `exofs_inode_cache': Can't free all objects
> Call Trace:
> 754efe08: [<6007e9a6>] kmem_cache_destroy+0x82/0xca
> 754efe38: [<7a9296ba>] exit_exofs+0x1a/0x1c [exofs]
> 754efe48: [<60054c10>] sys_delete_module+0x1b9/0x217
> 754efee8: [<60014d60>] handle_syscall+0x58/0x70
> 754eff08: [<60024163>] userspace+0x2dd/0x38a
> 754effc8: [<600126af>] fork_handler+0x62/0x69
>
>
> I also get a Kernel crash. I suspect it's when finally these
> free_rcu come and the module (and kmem_cache) are no longer there.
>
> What to do? Nick?
http://lwn.net/Articles/217484/ explains how to wait for rcu callback to finish
-aneesh
On 02/08/2011 06:57 PM, Aneesh Kumar K. V wrote:
> On Mon, 07 Feb 2011 15:14:40 +0200, Boaz Harrosh <[email protected]> wrote:
>> On 02/05/2011 11:01 AM, Tao Ma wrote:
>>> From: Tao Ma <[email protected]>
>>>
>>> In fa0d7e3, we use rcu free inode instead of freeing the inode
>>> directly. It causes a problem when we rmmod immediately after
>>> we umount the volume[1].
>>>
>>> So we need to call synchronize_rcu after we kill_sb so that
>>> the inode is freed before we do rmmod. The idea is inspired
>>> by Chris Mason[2]. I tested with ext4 by umount+rmmod and it
>>> doesn't show any error by now.
>>>
>>> 1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
>>> 2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2
>>>
>>> Cc: Nick Piggin <[email protected]>
>>> Cc: Al Viro <[email protected]>
>>> Cc: Chris Mason <[email protected]>
>>> Cc: Boaz Harrosh <[email protected]>
>>> Signed-off-by: Tao Ma <[email protected]>
>>> ---
>>> fs/super.c | 7 +++++++
>>> 1 files changed, 7 insertions(+), 0 deletions(-)
>>>
>>> diff --git a/fs/super.c b/fs/super.c
>>> index 74e149e..315bce9 100644
>>> --- a/fs/super.c
>>> +++ b/fs/super.c
>>> @@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)
>>> struct file_system_type *fs = s->s_type;
>>> if (atomic_dec_and_test(&s->s_active)) {
>>> fs->kill_sb(s);
>>> + /*
>>> + * We need to synchronize rcu here so that
>>> + * the delayed rcu inode free can be executed
>>> + * before we put_super.
>>> + * https://bugzilla.kernel.org/show_bug.cgi?id=27652
>>> + */
>>> + synchronize_rcu();
>>> put_filesystem(fs);
>>> put_super(s);
>>> } else {
>>
>>
<>
>
> http://lwn.net/Articles/217484/ explains how to wait for rcu callback to finish
>
> -aneesh
Yes thanks Aneesh, rcu_barrier does the trick
---
From: Boaz Harrosh <[email protected]>
In fa0d7e3, we use rcu free inode instead of freeing the inode
directly. It causes a problem when we rmmod immediately after
we umount the volume[1].
So we need to call rcu_barrier after we kill_sb so that
the inode is freed before we do rmmod. The idea is inspired
by Aneesh Kumar. rcu_barrier will wait for all callbacks
to end before preceding. The original patch was done by
Tao Ma, but synchronize_rcu() is not enough here.
1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2
Cc: Nick Piggin <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Chris Mason <[email protected]>
Cc: Tao Ma <[email protected]>
Signed-off-by: Boaz Harrosh <[email protected]>
---
git diff --stat -p -M fs/super.c
fs/super.c | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/fs/super.c b/fs/super.c
index 74e149e..5fd4ec9 100644
--- a/fs/super.c
+++ b/fs/super.c
@@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)
struct file_system_type *fs = s->s_type;
if (atomic_dec_and_test(&s->s_active)) {
fs->kill_sb(s);
+ /*
+ * We need to synchronize rcu here so that
+ * the delayed rcu inode free can be executed
+ * before we put_super.
+ * https://bugzilla.kernel.org/show_bug.cgi?id=27652
+ */
+ rcu_barrier();
put_filesystem(fs);
put_super(s);
} else {
On 02/09/2011 01:25 AM, Boaz Harrosh wrote:
> On 02/08/2011 06:57 PM, Aneesh Kumar K. V wrote:
>
>> On Mon, 07 Feb 2011 15:14:40 +0200, Boaz Harrosh<[email protected]> wrote:
>>
>>> On 02/05/2011 11:01 AM, Tao Ma wrote:
>>>
>>>> From: Tao Ma<[email protected]>
>>>>
>>>> In fa0d7e3, we use rcu free inode instead of freeing the inode
>>>> directly. It causes a problem when we rmmod immediately after
>>>> we umount the volume[1].
>>>>
>>>> So we need to call synchronize_rcu after we kill_sb so that
>>>> the inode is freed before we do rmmod. The idea is inspired
>>>> by Chris Mason[2]. I tested with ext4 by umount+rmmod and it
>>>> doesn't show any error by now.
>>>>
>>>> 1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
>>>> 2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2
>>>>
>>>> Cc: Nick Piggin<[email protected]>
>>>> Cc: Al Viro<[email protected]>
>>>> Cc: Chris Mason<[email protected]>
>>>> Cc: Boaz Harrosh<[email protected]>
>>>> Signed-off-by: Tao Ma<[email protected]>
>>>> ---
>>>> fs/super.c | 7 +++++++
>>>> 1 files changed, 7 insertions(+), 0 deletions(-)
>>>>
>>>> diff --git a/fs/super.c b/fs/super.c
>>>> index 74e149e..315bce9 100644
>>>> --- a/fs/super.c
>>>> +++ b/fs/super.c
>>>> @@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)
>>>> struct file_system_type *fs = s->s_type;
>>>> if (atomic_dec_and_test(&s->s_active)) {
>>>> fs->kill_sb(s);
>>>> + /*
>>>> + * We need to synchronize rcu here so that
>>>> + * the delayed rcu inode free can be executed
>>>> + * before we put_super.
>>>> + * https://bugzilla.kernel.org/show_bug.cgi?id=27652
>>>> + */
>>>> + synchronize_rcu();
>>>> put_filesystem(fs);
>>>> put_super(s);
>>>> } else {
>>>>
>>>
>>>
> <>
>
>> http://lwn.net/Articles/217484/ explains how to wait for rcu callback to finish
>>
>> -aneesh
>>
> Yes thanks Aneesh, rcu_barrier does the trick
> ---
> From: Boaz Harrosh<[email protected]>
>
> In fa0d7e3, we use rcu free inode instead of freeing the inode
> directly. It causes a problem when we rmmod immediately after
> we umount the volume[1].
>
> So we need to call rcu_barrier after we kill_sb so that
> the inode is freed before we do rmmod. The idea is inspired
> by Aneesh Kumar. rcu_barrier will wait for all callbacks
> to end before preceding. The original patch was done by
> Tao Ma, but synchronize_rcu() is not enough here.
>
> 1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
> 2. http://marc.info/?l=linux-fsdevel&m=129684698713709&w=2
>
> Cc: Nick Piggin<[email protected]>
> Cc: Al Viro<[email protected]>
> Cc: Chris Mason<[email protected]>
> Cc: Tao Ma<[email protected]>
> Signed-off-by: Boaz Harrosh<[email protected]>
>
It works now in my ext4 test box. Thanks for your work.
Tested-by: Tao Ma <[email protected]>
> ---
> git diff --stat -p -M fs/super.c
> fs/super.c | 1 +
> 1 files changed, 1 insertions(+), 0 deletions(-)
>
> diff --git a/fs/super.c b/fs/super.c
> index 74e149e..5fd4ec9 100644
> --- a/fs/super.c
> +++ b/fs/super.c
> @@ -177,6 +177,13 @@ void deactivate_locked_super(struct super_block *s)
> struct file_system_type *fs = s->s_type;
> if (atomic_dec_and_test(&s->s_active)) {
> fs->kill_sb(s);
> + /*
> + * We need to synchronize rcu here so that
> + * the delayed rcu inode free can be executed
> + * before we put_super.
> + * https://bugzilla.kernel.org/show_bug.cgi?id=27652
> + */
> + rcu_barrier();
> put_filesystem(fs);
> put_super(s);
> } else {
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> + /*
> + * We need to synchronize rcu here so that
> + * the delayed rcu inode free can be executed
> + * before we put_super.
> + * https://bugzilla.kernel.org/show_bug.cgi?id=27652
> + */
URLs in comments are not useful, descriptions of the issues in comments
should be complete enough to understand the issue. I think the comment
without the url is enough, though.
In fa0d7e3, we use rcu free inode instead of freeing the inode
directly. It causes a crash when we rmmod immediately after
we umount the volume[1].
So we need to call rcu_barrier after we kill_sb so that
the inode is freed before we do rmmod. The idea is inspired
by Aneesh Kumar. rcu_barrier will wait for all callbacks
to end before preceding. The original patch was done by
Tao Ma, but synchronize_rcu() is not enough here.
1. http://marc.info/?l=linux-fsdevel&m=129680863330185&w=2
Cc: Nick Piggin <[email protected]>
Cc: Al Viro <[email protected]>
Cc: Chris Mason <[email protected]>
Tested-by: Tao Ma <[email protected]>
Signed-off-by: Boaz Harrosh <[email protected]>
---
fs/super.c | 5 +++++
1 files changed, 5 insertions(+), 0 deletions(-)
diff --git a/fs/super.c b/fs/super.c
index 74e149e..7e9dd4c 100644
--- a/fs/super.c
+++ b/fs/super.c
@@ -177,6 +177,11 @@ void deactivate_locked_super(struct super_block *s)
struct file_system_type *fs = s->s_type;
if (atomic_dec_and_test(&s->s_active)) {
fs->kill_sb(s);
+ /*
+ * We need to call rcu_barrier so all the delayed rcu free
+ * inodes are flushed before we release the fs module.
+ */
+ rcu_barrier();
put_filesystem(fs);
put_super(s);
} else {
--
1.7.2.3