Introduce __kfree_rcu() for kfree_rcu()
It like __kfree_rcu() in slab.c.
We can calculate the object poiter from a poiter inside this
object in slub.c, so we can use it for rcu callback and free
the object.
Signed-off-by: Lai Jiangshan <[email protected]>
---
diff --git a/mm/slub.c b/mm/slub.c
index 0280eee..b438ec5 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2756,6 +2756,30 @@ void kfree(const void *x)
}
EXPORT_SYMBOL(kfree);
+static void kfree_rcu_callback(struct rcu_head *rcu)
+{
+ void *portion = rcu;
+ struct page *page = virt_to_head_page(portion);
+
+ if (unlikely(!PageSlab(page))) {
+ BUG_ON(!PageCompound(page));
+ put_page(page);
+ } else {
+ struct kmem_cache *s = page->slab;
+ void *start = page_address(page);
+ unsigned int index = (portion - start) / s->size;
+ void *object = start + s->size * index;
+
+ slab_free(s, page, object, _RET_IP_);
+ }
+}
+
+void __kfree_rcu(const void *x, struct rcu_head *rcu)
+{
+ call_rcu(rcu, kfree_rcu_callback);
+}
+EXPORT_SYMBOL(__kfree_rcu);
+
/*
* kmem_cache_shrink removes empty slabs from the partial lists and sorts
* the remaining slabs by the number of items in use. The slabs with the
On Tue, 3 Mar 2009, Lai Jiangshan wrote:
> +
> +void __kfree_rcu(const void *x, struct rcu_head *rcu)
> +{
> + call_rcu(rcu, kfree_rcu_callback);
> +}
> +EXPORT_SYMBOL(__kfree_rcu);
> +
The parameter x is unused why require it? I see that SLOB needs it. Cant
you do a similar trick as in SLUB just calculating the start address of
the object from the rcu address?
Christoph Lameter wrote:
> On Tue, 3 Mar 2009, Lai Jiangshan wrote:
>
>> +
>> +void __kfree_rcu(const void *x, struct rcu_head *rcu)
>> +{
>> + call_rcu(rcu, kfree_rcu_callback);
>> +}
>> +EXPORT_SYMBOL(__kfree_rcu);
>> +
>
> The parameter x is unused why require it? I see that SLOB needs it. Cant
> you do a similar trick as in SLUB just calculating the start address of
> the object from the rcu address?
>
The first parameter of __kfree_rcu() is unused in slab,slub,slqb.
(I used it before, for CONFIG_SLUB_DEBUG=y ...etc. But I found
these debugging code is needless)
Object sizes of the memory objects in slob's kmalloc-slab are various.
We hardly calculate the start address of the object from the rcu address.
Lai
On Thu, 5 Mar 2009, Lai Jiangshan wrote:
> The first parameter of __kfree_rcu() is unused in slab,slub,slqb.
> (I used it before, for CONFIG_SLUB_DEBUG=y ...etc. But I found
> these debugging code is needless)
>
> Object sizes of the memory objects in slob's kmalloc-slab are various.
> We hardly calculate the start address of the object from the rcu address.
But you can start with the descriptor at the beginning of the page and
scan until you find the object.
On Thu, Mar 05, 2009 at 09:39:03AM -0500, Christoph Lameter wrote:
> On Thu, 5 Mar 2009, Lai Jiangshan wrote:
>
> > The first parameter of __kfree_rcu() is unused in slab,slub,slqb.
> > (I used it before, for CONFIG_SLUB_DEBUG=y ...etc. But I found
> > these debugging code is needless)
> >
> > Object sizes of the memory objects in slob's kmalloc-slab are various.
> > We hardly calculate the start address of the object from the rcu address.
>
> But you can start with the descriptor at the beginning of the page and
> scan until you find the object.
Another approach would be to keep a per-page bitmap of the objects
that are awaiting a grace period.
Not sure whether or not it is worthwhile, but it does reduce the
per-object overhead to two bits (one for the objects awaiting the
current grace period, the other for objects that will await the next
grace period).
Thanx, Paul
On Tue, 2009-03-03 at 21:44 +0800, Lai Jiangshan wrote:
> Introduce __kfree_rcu() for kfree_rcu()
>
> It like __kfree_rcu() in slab.c.
> We can calculate the object poiter from a poiter inside this
> object in slub.c, so we can use it for rcu callback and free
> the object.
>
> Signed-off-by: Lai Jiangshan <[email protected]>
> ---
> diff --git a/mm/slub.c b/mm/slub.c
> index 0280eee..b438ec5 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -2756,6 +2756,30 @@ void kfree(const void *x)
> }
> EXPORT_SYMBOL(kfree);
>
> +static void kfree_rcu_callback(struct rcu_head *rcu)
> +{
> + void *portion = rcu;
> + struct page *page = virt_to_head_page(portion);
> +
> + if (unlikely(!PageSlab(page))) {
> + BUG_ON(!PageCompound(page));
I'd prefer you did a WARN_ON() to increase the likelihood that people
will be able to copy-paste the oops for a bug report.
> + put_page(page);
> + } else {
> + struct kmem_cache *s = page->slab;
> + void *start = page_address(page);
> + unsigned int index = (portion - start) / s->size;
> + void *object = start + s->size * index;
> +
Again, I think the above would be more readable if you split the
variable initialization from the declaration.
> + slab_free(s, page, object, _RET_IP_);
> + }
> +}
> +
> +void __kfree_rcu(const void *x, struct rcu_head *rcu)
> +{
> + call_rcu(rcu, kfree_rcu_callback);
> +}
> +EXPORT_SYMBOL(__kfree_rcu);
> +
> /*
> * kmem_cache_shrink removes empty slabs from the partial lists and sorts
> * the remaining slabs by the number of items in use. The slabs with the
>
>
>
>
>
>
>
>
>
>