Hello,
These patches improve kfree_rcu() to support:
- kfree_rcu() headless usage.
- both vmalloc() and slab free'ing support using array of pointers.
- simpler debugobjects handling.
It applies on rcu/dev branch as of March 29th.
Testing with rcuperf shows following changes. The memory footprint reduces and
batches go slightly up. This is assumed an acceptable change.
with all patches:
Total time taken by all kfree'ers: 27312964461 ns, loops: 20000, batches: 3120, memory footprint: 211MB
Total time taken by all kfree'ers: 26773272309 ns, loops: 20000, batches: 3084, memory footprint: 208M
without:
Total time taken by all kfree'ers: 25711621811 ns, loops: 20000, batches: 2814, memory footprint: 230MB
Total time taken by all kfree'ers: 25775800546 ns, loops: 20000, batches: 2755, memory footprint: 230MB
These have been pushed to the git tree at:
git://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git (branch rcu/kfree)
cgit view:
https://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git/log/?h=rcu/kfree
thanks,
- Joel
Joel Fernandes (Google) (5):
rcu/tree: Simplify debug_objects handling
rcu/tree: Clarify emergency path comment better
rcu/tree: Remove extra next variable in kfree worker function
rcu/tree: Simplify is_vmalloc_addr expression
rcu/tree: Make kvfree_rcu() tolerate any alignment
Uladzislau Rezki (Sony) (13):
mm/list_lru.c: Rename kvfree_rcu() to local variant
rcu: Introduce kvfree_rcu() interface
rcu: Rename rcu_invoke_kfree_callback/rcu_kfree_callback
rcu: Rename __is_kfree_rcu_offset() macro
rcu: Rename kfree_call_rcu() to the kvfree_call_rcu().
mm/list_lru.c: Remove kvfree_rcu_local() function
rcu/tree: Simplify KFREE_BULK_MAX_ENTR macro
rcu/tree: Maintain separate array for vmalloc ptrs
rcu/tree: Introduce expedited_drain flag
rcu/tree: Support reclaim for head-less object
rcu/tiny: Move kvfree_call_rcu() out of header
rcu/tiny: Support reclaim for head-less object
rcu: Support headless variant in the kvfree_rcu()
include/linux/rcupdate.h | 53 ++++++-
include/linux/rcutiny.h | 6 +-
include/linux/rcutree.h | 2 +-
include/trace/events/rcu.h | 8 +-
kernel/rcu/tiny.c | 168 +++++++++++++++++++-
kernel/rcu/tree.c | 315 ++++++++++++++++++++++++++-----------
mm/list_lru.c | 11 +-
7 files changed, 443 insertions(+), 120 deletions(-)
--
2.26.0.rc2.310.g2932bb562d-goog
From: "Uladzislau Rezki (Sony)" <[email protected]>
Rename __is_kfree_rcu_offset to __is_kvfree_rcu_offset.
All RCU paths use kvfree() now instead of kfree(), thus
rename it.
Signed-off-by: Uladzislau Rezki (Sony) <[email protected]>
Reviewed-by: Joel Fernandes (Google) <[email protected]>
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
include/linux/rcupdate.h | 6 +++---
kernel/rcu/tiny.c | 2 +-
kernel/rcu/tree.c | 4 ++--
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 8b7128d0860e2..c6f6a195cb1cd 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -823,16 +823,16 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
/*
* Does the specified offset indicate that the corresponding rcu_head
- * structure can be handled by kfree_rcu()?
+ * structure can be handled by kvfree_rcu()?
*/
-#define __is_kfree_rcu_offset(offset) ((offset) < 4096)
+#define __is_kvfree_rcu_offset(offset) ((offset) < 4096)
/*
* Helper macro for kfree_rcu() to prevent argument-expansion eyestrain.
*/
#define __kfree_rcu(head, offset) \
do { \
- BUILD_BUG_ON(!__is_kfree_rcu_offset(offset)); \
+ BUILD_BUG_ON(!__is_kvfree_rcu_offset(offset)); \
kfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \
} while (0)
diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index 3dd8e6e207b09..aa897c3f2e92c 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -85,7 +85,7 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head)
unsigned long offset = (unsigned long)head->func;
rcu_lock_acquire(&rcu_callback_map);
- if (__is_kfree_rcu_offset(offset)) {
+ if (__is_kvfree_rcu_offset(offset)) {
trace_rcu_invoke_kvfree_callback("", head, offset);
kvfree((void *)head - offset);
rcu_lock_release(&rcu_callback_map);
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 88b744ce896c0..1209945a34bfd 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2743,7 +2743,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func)
return; // Enqueued onto ->nocb_bypass, so just leave.
// If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock.
rcu_segcblist_enqueue(&rdp->cblist, head);
- if (__is_kfree_rcu_offset((unsigned long)func))
+ if (__is_kvfree_rcu_offset((unsigned long)func))
trace_rcu_kvfree_callback(rcu_state.name, head,
(unsigned long)func,
rcu_segcblist_n_cbs(&rdp->cblist));
@@ -2937,7 +2937,7 @@ static void kfree_rcu_work(struct work_struct *work)
rcu_lock_acquire(&rcu_callback_map);
trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
- if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset)))
+ if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset)))
kvfree((void *)head - offset);
rcu_lock_release(&rcu_callback_map);
--
2.26.0.rc2.310.g2932bb562d-goog
From: "Uladzislau Rezki (Sony)" <[email protected]>
Update the kvfree_call_rcu() with head-less support, it
means an object without any rcu_head structure can be
reclaimed after GP.
To store pointers there are two chain-arrays maintained
one for SLAB and another one is for vmalloc. Both types
of objects(head-less variant and regular one) are placed
there based on the type.
It can be that maintaining of arrays becomes impossible
due to high memory pressure. For such reason there is an
emergency path. In that case objects with rcu_head inside
are just queued building one way list. Later on that list
is drained.
As for head-less variant. Such objects do not have any
rcu_head helper inside. Thus it is dynamically attached.
As a result an object consists of back-pointer and regular
rcu_head. It implies that emergency path can detect such
object type, therefore they are tagged. So a back-pointer
could be freed as well as dynamically attached wrapper.
Even though such approach requires dynamic memory it needs
only sizeof(unsigned long *) + sizeof(struct rcu_head) bytes,
thus SLAB is used to obtain it. Finally if attaching of the
rcu_head and queuing get failed, the current context has
to follow might_sleep() annotation, thus below steps could
be applied:
a) wait until a grace period has elapsed;
b) direct inlining of the kvfree() call.
Reviewed-by: Joel Fernandes (Google) <[email protected]>
Signed-off-by: Uladzislau Rezki (Sony) <[email protected]>
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
kernel/rcu/tree.c | 93 +++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 85 insertions(+), 8 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 3b94526f490cb..204292378101b 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2959,19 +2959,34 @@ static void kfree_rcu_work(struct work_struct *work)
* when we could not allocate a bulk array.
*
* Under that condition an object is queued to the
- * list instead.
+ * list instead. Please note that head-less objects
+ * have dynamically attached rcu_head, so they also
+ * contain a back-pointer that has to be freed.
*/
for (; head; head = next) {
unsigned long offset = (unsigned long)head->func;
- void *ptr = (void *)head - offset;
+ bool headless;
+ void *ptr;
next = head->next;
+
+ /* We tag the headless object, if so adjust offset. */
+ headless = (((unsigned long) head - offset) & BIT(0));
+ if (headless)
+ offset -= 1;
+
+ ptr = (void *) head - offset;
debug_rcu_head_unqueue((struct rcu_head *)ptr);
+
rcu_lock_acquire(&rcu_callback_map);
trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
- if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset)))
+ if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset))) {
+ if (headless)
+ kvfree((void *) *((unsigned long *) ptr));
+
kvfree(ptr);
+ }
rcu_lock_release(&rcu_callback_map);
cond_resched_tasks_rcu_qs();
@@ -3120,6 +3135,24 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
return true;
}
+static inline struct rcu_head *attach_rcu_head_to_object(void *obj)
+{
+ unsigned long *ptr;
+
+ ptr = kmalloc(sizeof(unsigned long *) +
+ sizeof(struct rcu_head), GFP_NOWAIT | __GFP_NOWARN);
+
+ if (!ptr)
+ ptr = kmalloc(sizeof(unsigned long *) +
+ sizeof(struct rcu_head), GFP_ATOMIC | __GFP_NOWARN);
+
+ if (!ptr)
+ return NULL;
+
+ ptr[0] = (unsigned long) obj;
+ return ((struct rcu_head *) ++ptr);
+}
+
/*
* Queue a request for lazy invocation of appropriate free routine after a
* grace period. Please note there are three paths are maintained, two are the
@@ -3138,20 +3171,37 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
unsigned long flags;
struct kfree_rcu_cpu *krcp;
bool expedited_drain = false;
+ bool success;
void *ptr;
+ if (head) {
+ ptr = (void *) head - (unsigned long) func;
+ } else {
+ /*
+ * Please note there is a limitation for the head-less
+ * variant, that is why there is a clear rule for such
+ * objects:
+ *
+ * use it from might_sleep() context only. For other
+ * places please embed an rcu_head to your structures.
+ */
+ might_sleep();
+ ptr = (unsigned long *) func;
+ }
+
local_irq_save(flags); // For safely calling this_cpu_ptr().
krcp = this_cpu_ptr(&krc);
if (krcp->initialized)
spin_lock(&krcp->lock);
- ptr = (void *)head - (unsigned long)func;
-
// Queue the object but don't yet schedule the batch.
if (debug_rcu_head_queue(ptr)) {
// Probable double kfree_rcu(), just leak.
WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n",
__func__, head);
+
+ /* Mark as success and leave. */
+ success = true;
goto unlock_return;
}
@@ -3159,7 +3209,22 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
* Under high memory pressure GFP_NOWAIT can fail,
* in that case the emergency path is maintained.
*/
- if (!kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr)) {
+ success = kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr);
+ if (!success) {
+ /* Is headless object? */
+ if (head == NULL) {
+ head = attach_rcu_head_to_object(ptr);
+ if (head == NULL)
+ goto unlock_return;
+
+ /*
+ * Tag the headless object. Such objects have a back-pointer
+ * to the original allocated memory, that has to be freed as
+ * well as dynamically attached wrapper/head.
+ */
+ func = (rcu_callback_t) (sizeof(unsigned long *) + 1);
+ }
+
head->func = func;
head->next = krcp->head;
krcp->head = head;
@@ -3171,15 +3236,15 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
* appropriate free calls.
*/
expedited_drain = true;
+ success = true;
}
WRITE_ONCE(krcp->count, krcp->count + 1);
// Set timer to drain after KFREE_DRAIN_JIFFIES.
if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING &&
- !krcp->monitor_todo) {
+ !krcp->monitor_todo) {
krcp->monitor_todo = true;
-
schedule_delayed_work(&krcp->monitor_work,
expedited_drain ? 0 : KFREE_DRAIN_JIFFIES);
}
@@ -3188,6 +3253,18 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
if (krcp->initialized)
spin_unlock(&krcp->lock);
local_irq_restore(flags);
+
+ /*
+ * High memory pressure, so inline kvfree() after
+ * synchronize_rcu(). We can do it from might_sleep()
+ * context only, so the current CPU can pass the QS
+ * state.
+ */
+ if (!success) {
+ debug_rcu_head_unqueue(ptr);
+ synchronize_rcu();
+ kvfree(ptr);
+ }
}
EXPORT_SYMBOL_GPL(kvfree_call_rcu);
--
2.26.0.rc2.310.g2932bb562d-goog
From: "Uladzislau Rezki (Sony)" <[email protected]>
Make a kvfree_call_rcu() function to support head-less
freeing. Same as for tree-RCU, for such purpose we store
pointers in array. SLAB and vmalloc ptrs. are mixed and
coexist together.
Under high memory pressure it can be that maintaining of
arrays becomes impossible. Objects with an rcu_head are
released via call_rcu(). When it comes to the head-less
variant, the kvfree() call is directly inlined, i.e. we
do the same as for tree-RCU:
a) wait until a grace period has elapsed;
b) direct inlining of the kvfree() call.
Thus the current context has to follow might_sleep()
annotation. Also please note that for tiny-RCU any
call of synchronize_rcu() is actually a quiescent
state, therefore (a) does nothing.
Reviewed-by: Joel Fernandes (Google) <[email protected]>
Signed-off-by: Uladzislau Rezki (Sony) <[email protected]>
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
kernel/rcu/tiny.c | 157 +++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 156 insertions(+), 1 deletion(-)
diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index 508c82faa45c3..b1c31a935db93 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -40,6 +40,29 @@ static struct rcu_ctrlblk rcu_ctrlblk = {
.curtail = &rcu_ctrlblk.rcucblist,
};
+/* Can be common with tree-RCU. */
+#define KVFREE_DRAIN_JIFFIES (HZ / 50)
+
+/* Can be common with tree-RCU. */
+struct kvfree_rcu_bulk_data {
+ unsigned long nr_records;
+ struct kvfree_rcu_bulk_data *next;
+ void *records[];
+};
+
+/* Can be common with tree-RCU. */
+#define KVFREE_BULK_MAX_ENTR \
+ ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *))
+
+static struct kvfree_rcu_bulk_data *kvhead;
+static struct kvfree_rcu_bulk_data *kvhead_free;
+static struct kvfree_rcu_bulk_data *kvcache;
+
+static DEFINE_STATIC_KEY_FALSE(rcu_init_done);
+static struct delayed_work monitor_work;
+static struct rcu_work rcu_work;
+static bool monitor_todo;
+
void rcu_barrier(void)
{
wait_rcu_gp(call_rcu);
@@ -177,9 +200,137 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func)
}
EXPORT_SYMBOL_GPL(call_rcu);
+static inline bool
+kvfree_call_rcu_add_ptr_to_bulk(void *ptr)
+{
+ struct kvfree_rcu_bulk_data *bnode;
+
+ if (!kvhead || kvhead->nr_records == KVFREE_BULK_MAX_ENTR) {
+ bnode = xchg(&kvcache, NULL);
+ if (!bnode)
+ bnode = (struct kvfree_rcu_bulk_data *)
+ __get_free_page(GFP_NOWAIT | __GFP_NOWARN);
+
+ if (unlikely(!bnode))
+ return false;
+
+ /* Initialize the new block. */
+ bnode->nr_records = 0;
+ bnode->next = kvhead;
+
+ /* Attach it to the bvhead. */
+ kvhead = bnode;
+ }
+
+ /* Done. */
+ kvhead->records[kvhead->nr_records++] = ptr;
+ return true;
+}
+
+static void
+kvfree_rcu_work(struct work_struct *work)
+{
+ struct kvfree_rcu_bulk_data *kvhead_tofree, *next;
+ unsigned long flags;
+ int i;
+
+ local_irq_save(flags);
+ kvhead_tofree = kvhead_free;
+ kvhead_free = NULL;
+ local_irq_restore(flags);
+
+ /* Reclaim process. */
+ for (; kvhead_tofree; kvhead_tofree = next) {
+ next = kvhead_tofree->next;
+
+ for (i = 0; i < kvhead_tofree->nr_records; i++) {
+ debug_rcu_head_unqueue((struct rcu_head *)
+ kvhead_tofree->records[i]);
+ kvfree(kvhead_tofree->records[i]);
+ }
+
+ if (cmpxchg(&kvcache, NULL, kvhead_tofree))
+ free_page((unsigned long) kvhead_tofree);
+ }
+}
+
+static inline bool
+queue_kvfree_rcu_work(void)
+{
+ /* Check if the free channel is available. */
+ if (kvhead_free)
+ return false;
+
+ kvhead_free = kvhead;
+ kvhead = NULL;
+
+ /*
+ * Queue the job for memory reclaim after GP.
+ */
+ queue_rcu_work(system_wq, &rcu_work);
+ return true;
+}
+
+static void kvfree_rcu_monitor(struct work_struct *work)
+{
+ unsigned long flags;
+ bool queued;
+
+ local_irq_save(flags);
+ queued = queue_kvfree_rcu_work();
+ if (queued)
+ /* Success. */
+ monitor_todo = false;
+ local_irq_restore(flags);
+
+ /*
+ * If previous RCU reclaim process is still in progress,
+ * schedule the work one more time to try again later.
+ */
+ if (monitor_todo)
+ schedule_delayed_work(&monitor_work,
+ KVFREE_DRAIN_JIFFIES);
+}
+
void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
{
- call_rcu(head, func);
+ unsigned long flags;
+ bool success;
+ void *ptr;
+
+ if (head) {
+ ptr = (void *) head - (unsigned long) func;
+ } else {
+ might_sleep();
+ ptr = (void *) func;
+ }
+
+ if (debug_rcu_head_queue(ptr)) {
+ /* Probable double free, just leak. */
+ WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n",
+ __func__, head);
+ return;
+ }
+
+ local_irq_save(flags);
+ success = kvfree_call_rcu_add_ptr_to_bulk(ptr);
+ if (static_branch_likely(&rcu_init_done)) {
+ if (success && !monitor_todo) {
+ monitor_todo = true;
+ schedule_delayed_work(&monitor_work,
+ KVFREE_DRAIN_JIFFIES);
+ }
+ }
+ local_irq_restore(flags);
+
+ if (!success) {
+ if (!head) {
+ synchronize_rcu();
+ kvfree(ptr);
+ } else {
+ call_rcu(head, func);
+ }
+ }
}
EXPORT_SYMBOL_GPL(kvfree_call_rcu);
@@ -188,4 +339,8 @@ void __init rcu_init(void)
open_softirq(RCU_SOFTIRQ, rcu_process_callbacks);
rcu_early_boot_tests();
srcu_init();
+
+ INIT_DELAYED_WORK(&monitor_work, kvfree_rcu_monitor);
+ INIT_RCU_WORK(&rcu_work, kvfree_rcu_work);
+ static_branch_enable(&rcu_init_done);
}
--
2.26.0.rc2.310.g2932bb562d-goog
Handle cases where the the object being kvfree_rcu()'d is not aligned by
2-byte boundaries.
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
kernel/rcu/tree.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 311d216c7faa7..d6536374d12a9 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2827,6 +2827,9 @@ struct kvfree_rcu_bulk_data {
#define KVFREE_BULK_MAX_ENTR \
((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *))
+/* Encoding the offset of a fake rcu_head to indicate the head is a wrapper. */
+#define RCU_HEADLESS_KFREE BIT(31)
+
/**
* struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests
* @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period
@@ -2970,9 +2973,9 @@ static void kfree_rcu_work(struct work_struct *work)
next = head->next;
/* We tag the headless object, if so adjust offset. */
- headless = (((unsigned long) head - offset) & BIT(0));
+ headless = !!(offset & RCU_HEADLESS_KFREE);
if (headless)
- offset -= 1;
+ offset &= ~(RCU_HEADLESS_KFREE);
ptr = (void *) head - offset;
debug_rcu_head_unqueue((struct rcu_head *)ptr);
@@ -3221,7 +3224,7 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
* to the original allocated memory, that has to be freed as
* well as dynamically attached wrapper/head.
*/
- func = (rcu_callback_t) (sizeof(unsigned long *) + 1);
+ func = (rcu_callback_t)(sizeof(unsigned long *) | RCU_HEADLESS_KFREE);
}
head->func = func;
--
2.26.0.rc2.310.g2932bb562d-goog
No change in code, small refactor.
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
kernel/rcu/tree.c | 11 +++++------
1 file changed, 5 insertions(+), 6 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 204292378101b..56c9e102a901d 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2889,8 +2889,7 @@ debug_rcu_bhead_unqueue(struct kvfree_rcu_bulk_data *bhead)
static void kfree_rcu_work(struct work_struct *work)
{
unsigned long flags;
- struct kvfree_rcu_bulk_data *bkhead, *bknext;
- struct kvfree_rcu_bulk_data *bvhead, *bvnext;
+ struct kvfree_rcu_bulk_data *bkhead, *bvhead, *bnext;
struct rcu_head *head, *next;
struct kfree_rcu_cpu *krcp;
struct kfree_rcu_cpu_work *krwp;
@@ -2915,8 +2914,8 @@ static void kfree_rcu_work(struct work_struct *work)
spin_unlock_irqrestore(&krcp->lock, flags);
/* kmalloc()/kfree() channel. */
- for (; bkhead; bkhead = bknext) {
- bknext = bkhead->next;
+ for (; bkhead; bkhead = bnext) {
+ bnext = bkhead->next;
debug_rcu_bhead_unqueue(bkhead);
@@ -2934,8 +2933,8 @@ static void kfree_rcu_work(struct work_struct *work)
}
/* vmalloc()/vfree() channel. */
- for (; bvhead; bvhead = bvnext) {
- bvnext = bvhead->next;
+ for (; bvhead; bvhead = bnext) {
+ bnext = bvhead->next;
debug_rcu_bhead_unqueue(bvhead);
--
2.26.0.rc2.310.g2932bb562d-goog
From: "Uladzislau Rezki (Sony)" <[email protected]>
Make it possible to pass one or two arguments to the
kvfree_rcu() macro what corresponds to either headless
case or not, so it becomes a bit versatile.
As a result we obtain two ways of using that macro,
below are two examples:
a) kvfree_rcu(ptr, rhf);
struct X {
struct rcu_head rhf;
unsigned char data[100];
};
void *ptr = kvmalloc(sizeof(struct X), GFP_KERNEL);
if (ptr)
kvfree_rcu(ptr, rhf);
b) kvfree_rcu(ptr);
void *ptr = kvmalloc(some_bytes, GFP_KERNEL);
if (ptr)
kvfree_rcu(ptr);
Last one, we name it headless variant, only needs one
argument, means it does not require any rcu_head to be
present within the type of ptr.
There is a restriction the (b) context has to fall into
might_sleep() annotation. To check that, please activate
the CONFIG_DEBUG_ATOMIC_SLEEP option in your kernel.
Reviewed-by: Joel Fernandes (Google) <[email protected]>
Signed-off-by: Uladzislau Rezki (Sony) <[email protected]>
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
include/linux/rcupdate.h | 38 ++++++++++++++++++++++++++++++++++----
1 file changed, 34 insertions(+), 4 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index edb6eeba49f83..7d04bbeeeef14 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -872,12 +872,42 @@ do { \
/**
* kvfree_rcu() - kvfree an object after a grace period.
- * @ptr: pointer to kvfree
- * @rhf: the name of the struct rcu_head within the type of @ptr.
*
- * Same as kfree_rcu(), just simple alias.
+ * This macro consists of one or two arguments and it is
+ * based on whether an object is head-less or not. If it
+ * has a head then a semantic stays the same as it used
+ * to be before:
+ *
+ * kvfree_rcu(ptr, rhf);
+ *
+ * where @ptr is a pointer to kvfree(), @rhf is the name
+ * of the rcu_head structure within the type of @ptr.
+ *
+ * When it comes to head-less variant, only one argument
+ * is passed and that is just a pointer which has to be
+ * freed after a grace period. Therefore the semantic is
+ *
+ * kvfree_rcu(ptr);
+ *
+ * where @ptr is a pointer to kvfree().
+ *
+ * Please note, head-less way of freeing is permitted to
+ * use from a context that has to follow might_sleep()
+ * annotation. Otherwise, please switch and embed the
+ * rcu_head structure within the type of @ptr.
*/
-#define kvfree_rcu(ptr, rhf) kfree_rcu(ptr, rhf)
+#define kvfree_rcu(...) KVFREE_GET_MACRO(__VA_ARGS__, \
+ kvfree_rcu_arg_2, kvfree_rcu_arg_1)(__VA_ARGS__)
+
+#define KVFREE_GET_MACRO(_1, _2, NAME, ...) NAME
+#define kvfree_rcu_arg_2(ptr, rhf) kfree_rcu(ptr, rhf)
+#define kvfree_rcu_arg_1(ptr) \
+do { \
+ typeof(ptr) ___p = (ptr); \
+ \
+ if (___p) \
+ kvfree_call_rcu(NULL, (rcu_callback_t) (___p)); \
+} while (0)
/*
* Place this after a lock-acquisition primitive to guarantee that
--
2.26.0.rc2.310.g2932bb562d-goog
No code change, small refactor.
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
kernel/rcu/tree.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 56c9e102a901d..311d216c7faa7 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3104,7 +3104,7 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
return false;
lockdep_assert_held(&krcp->lock);
- idx = !is_vmalloc_addr(ptr) ? 0:1;
+ idx = !!is_vmalloc_addr(ptr);
/* Check if a new block is required. */
if (!krcp->bkvhead[idx] ||
--
2.26.0.rc2.310.g2932bb562d-goog
From: "Uladzislau Rezki (Sony)" <[email protected]>
Move inlined kvfree_call_rcu() function out of the
header file. This step is a preparation for head-lees
support.
Reviewed-by: Joel Fernandes (Google) <[email protected]>
Signed-off-by: Uladzislau Rezki (Sony) <[email protected]>
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
include/linux/rcutiny.h | 6 +-----
kernel/rcu/tiny.c | 6 ++++++
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index 5ba0bcb231976..19a780afa444a 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -34,11 +34,7 @@ static inline void synchronize_rcu_expedited(void)
synchronize_rcu();
}
-static inline void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
-{
- call_rcu(head, func);
-}
-
+void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func);
void rcu_qs(void);
static inline void rcu_softirq_qs(void)
diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index aa897c3f2e92c..508c82faa45c3 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -177,6 +177,12 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func)
}
EXPORT_SYMBOL_GPL(call_rcu);
+void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
+{
+ call_rcu(head, func);
+}
+EXPORT_SYMBOL_GPL(kvfree_call_rcu);
+
void __init rcu_init(void)
{
open_softirq(RCU_SOFTIRQ, rcu_process_callbacks);
--
2.26.0.rc2.310.g2932bb562d-goog
From: "Uladzislau Rezki (Sony)" <[email protected]>
It is used and set to true when the bulk array can not
be maintained, it happens under low memory condition
and memory pressure.
In that case the drain work is scheduled right away and
not after KFREE_DRAIN_JIFFIES. It tends to speed up the
reclaim path. On the other hand, there is no data showing
the difference yet.
Reviewed-by: Joel Fernandes (Google) <[email protected]>
Signed-off-by: Uladzislau Rezki (Sony) <[email protected]>
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
kernel/rcu/tree.c | 20 ++++++++++++++++----
1 file changed, 16 insertions(+), 4 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 8fbc8450284db..3b94526f490cb 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3128,14 +3128,16 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
* due to memory pressure.
*
* Each kvfree_call_rcu() request is added to a batch. The batch will be drained
- * every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch will
- * be free'd in workqueue context. This allows us to: batch requests together to
- * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load.
+ * every KFREE_DRAIN_JIFFIES number of jiffies or can be scheduled right away if
+ * a low memory is detected. All the objects in the batch will be free'd in
+ * workqueue context. This allows us to: batch requests together to reduce the
+ * number of grace periods during heavy kfree_rcu()/kvfree_rcu() load.
*/
void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
{
unsigned long flags;
struct kfree_rcu_cpu *krcp;
+ bool expedited_drain = false;
void *ptr;
local_irq_save(flags); // For safely calling this_cpu_ptr().
@@ -3161,6 +3163,14 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
head->func = func;
head->next = krcp->head;
krcp->head = head;
+
+ /*
+ * There was an issue to place the pointer directly
+ * into array, due to memory pressure. Initiate an
+ * expedited drain to accelerate lazy invocation of
+ * appropriate free calls.
+ */
+ expedited_drain = true;
}
WRITE_ONCE(krcp->count, krcp->count + 1);
@@ -3169,7 +3179,9 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING &&
!krcp->monitor_todo) {
krcp->monitor_todo = true;
- schedule_delayed_work(&krcp->monitor_work, KFREE_DRAIN_JIFFIES);
+
+ schedule_delayed_work(&krcp->monitor_work,
+ expedited_drain ? 0 : KFREE_DRAIN_JIFFIES);
}
unlock_return:
--
2.26.0.rc2.310.g2932bb562d-goog
In order to prepare for future changes for headless RCU support, make the
debug_objects handling in kfree_rcu use the final 'pointer' value of the
object, instead of depending on the head.
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
kernel/rcu/tree.c | 30 +++++++++++++-----------------
1 file changed, 13 insertions(+), 17 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 3fb19ea039912..95d1f5e20d5ec 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2825,7 +2825,6 @@ struct kfree_rcu_bulk_data {
unsigned long nr_records;
void *records[KFREE_BULK_MAX_ENTR];
struct kfree_rcu_bulk_data *next;
- struct rcu_head *head_free_debug;
};
/**
@@ -2875,11 +2874,11 @@ struct kfree_rcu_cpu {
static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc);
static __always_inline void
-debug_rcu_head_unqueue_bulk(struct rcu_head *head)
+debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead)
{
#ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD
- for (; head; head = head->next)
- debug_rcu_head_unqueue(head);
+ for (int i = 0; i < bhead->nr_records; i++)
+ debug_rcu_head_unqueue((struct rcu_head *)(bhead->records[i]));
#endif
}
@@ -2909,7 +2908,7 @@ static void kfree_rcu_work(struct work_struct *work)
for (; bhead; bhead = bnext) {
bnext = bhead->next;
- debug_rcu_head_unqueue_bulk(bhead->head_free_debug);
+ debug_rcu_bhead_unqueue(bhead);
rcu_lock_acquire(&rcu_callback_map);
trace_rcu_invoke_kfree_bulk_callback(rcu_state.name,
@@ -2931,14 +2930,15 @@ static void kfree_rcu_work(struct work_struct *work)
*/
for (; head; head = next) {
unsigned long offset = (unsigned long)head->func;
+ void *ptr = (void *)head - offset;
next = head->next;
- debug_rcu_head_unqueue(head);
+ debug_rcu_head_unqueue((struct rcu_head *)ptr);
rcu_lock_acquire(&rcu_callback_map);
trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset)))
- kvfree((void *)head - offset);
+ kvfree(ptr);
rcu_lock_release(&rcu_callback_map);
cond_resched_tasks_rcu_qs();
@@ -3062,18 +3062,11 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp,
/* Initialize the new block. */
bnode->nr_records = 0;
bnode->next = krcp->bhead;
- bnode->head_free_debug = NULL;
/* Attach it to the head. */
krcp->bhead = bnode;
}
-#ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD
- head->func = func;
- head->next = krcp->bhead->head_free_debug;
- krcp->bhead->head_free_debug = head;
-#endif
-
/* Finally insert. */
krcp->bhead->records[krcp->bhead->nr_records++] =
(void *) head - (unsigned long) func;
@@ -3097,14 +3090,17 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
{
unsigned long flags;
struct kfree_rcu_cpu *krcp;
+ void *ptr;
local_irq_save(flags); // For safely calling this_cpu_ptr().
krcp = this_cpu_ptr(&krc);
if (krcp->initialized)
spin_lock(&krcp->lock);
+ ptr = (void *)head - (unsigned long)func;
+
// Queue the object but don't yet schedule the batch.
- if (debug_rcu_head_queue(head)) {
+ if (debug_rcu_head_queue(ptr)) {
// Probable double kfree_rcu(), just leak.
WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n",
__func__, head);
@@ -3121,8 +3117,8 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
* Under high memory pressure GFP_NOWAIT can fail,
* in that case the emergency path is maintained.
*/
- if (is_vmalloc_addr((void *) head - (unsigned long) func) ||
- !kfree_call_rcu_add_ptr_to_bulk(krcp, head, func)) {
+ if (is_vmalloc_addr(ptr) ||
+ !kfree_call_rcu_add_ptr_to_bulk(krcp, head, func)) {
head->func = func;
head->next = krcp->head;
krcp->head = head;
--
2.26.0.rc2.310.g2932bb562d-goog
From: "Uladzislau Rezki (Sony)" <[email protected]>
The reason is, it is capable of freeing vmalloc()
memory now.
Do the same with __kfree_rcu() macro, it becomes
__kvfree_rcu(), the reason is the same as pointed
above.
Signed-off-by: Uladzislau Rezki (Sony) <[email protected]>
Reviewed-by: Joel Fernandes (Google) <[email protected]>
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
include/linux/rcupdate.h | 8 ++++----
include/linux/rcutiny.h | 2 +-
include/linux/rcutree.h | 2 +-
kernel/rcu/tree.c | 8 ++++----
4 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index c6f6a195cb1cd..edb6eeba49f83 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -830,10 +830,10 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
/*
* Helper macro for kfree_rcu() to prevent argument-expansion eyestrain.
*/
-#define __kfree_rcu(head, offset) \
+#define __kvfree_rcu(head, offset) \
do { \
BUILD_BUG_ON(!__is_kvfree_rcu_offset(offset)); \
- kfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \
+ kvfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \
} while (0)
/**
@@ -852,7 +852,7 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
* Because the functions are not allowed in the low-order 4096 bytes of
* kernel virtual memory, offsets up to 4095 bytes can be accommodated.
* If the offset is larger than 4095 bytes, a compile-time error will
- * be generated in __kfree_rcu(). If this error is triggered, you can
+ * be generated in __kvfree_rcu(). If this error is triggered, you can
* either fall back to use of call_rcu() or rearrange the structure to
* position the rcu_head structure into the first 4096 bytes.
*
@@ -867,7 +867,7 @@ do { \
typeof (ptr) ___p = (ptr); \
\
if (___p) \
- __kfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \
+ __kvfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \
} while (0)
/**
diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index d77e11186afd1..5ba0bcb231976 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -34,7 +34,7 @@ static inline void synchronize_rcu_expedited(void)
synchronize_rcu();
}
-static inline void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
+static inline void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
{
call_rcu(head, func);
}
diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
index 45f3f66bb04df..3a7829d69fef8 100644
--- a/include/linux/rcutree.h
+++ b/include/linux/rcutree.h
@@ -33,7 +33,7 @@ static inline void rcu_virt_note_context_switch(int cpu)
}
void synchronize_rcu_expedited(void);
-void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func);
+void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func);
void rcu_barrier(void);
bool rcu_eqs_special_set(int cpu);
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 1209945a34bfd..3fb19ea039912 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3082,18 +3082,18 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp,
}
/*
- * Queue a request for lazy invocation of kfree_bulk()/kfree() after a grace
+ * Queue a request for lazy invocation of kfree_bulk()/kvfree() after a grace
* period. Please note there are two paths are maintained, one is the main one
* that uses kfree_bulk() interface and second one is emergency one, that is
* used only when the main path can not be maintained temporary, due to memory
* pressure.
*
- * Each kfree_call_rcu() request is added to a batch. The batch will be drained
+ * Each kvfree_call_rcu() request is added to a batch. The batch will be drained
* every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch will
* be free'd in workqueue context. This allows us to: batch requests together to
* reduce the number of grace periods during heavy kfree_rcu() load.
*/
-void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
+void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
{
unsigned long flags;
struct kfree_rcu_cpu *krcp;
@@ -3142,7 +3142,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
spin_unlock(&krcp->lock);
local_irq_restore(flags);
}
-EXPORT_SYMBOL_GPL(kfree_call_rcu);
+EXPORT_SYMBOL_GPL(kvfree_call_rcu);
static unsigned long
kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
--
2.26.0.rc2.310.g2932bb562d-goog
From: "Uladzislau Rezki (Sony)" <[email protected]>
To do so we use an array of common kvfree_rcu_bulk_data
structure. It consists of two elements, index number 0
corresponds to SLAB ptrs., whereas vmalloc pointers can
be accessed by using index number 1.
The reason of not mixing pointers is to have an easy way
to to distinguish them.
It is also the preparation patch for head-less objects
support. When an object is head-less we can not queue
it into any list, instead a pointer is placed directly
into an array.
Reviewed-by: Joel Fernandes (Google) <[email protected]>
Signed-off-by: Uladzislau Rezki (Sony) <[email protected]>
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
kernel/rcu/tree.c | 179 ++++++++++++++++++++++++++++------------------
1 file changed, 108 insertions(+), 71 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index cfe456e68c644..8fbc8450284db 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2808,38 +2808,36 @@ EXPORT_SYMBOL_GPL(call_rcu);
#define KFREE_N_BATCHES 2
/**
- * struct kfree_rcu_bulk_data - single block to store kfree_rcu() pointers
+ * struct kvfree_rcu_bulk_data - single block to store kvfree() pointers
* @nr_records: Number of active pointers in the array
- * @records: Array of the kfree_rcu() pointers
* @next: Next bulk object in the block chain
- * @head_free_debug: For debug, when CONFIG_DEBUG_OBJECTS_RCU_HEAD is set
+ * @records: Array of the SLAB pointers
*/
-struct kfree_rcu_bulk_data {
+struct kvfree_rcu_bulk_data {
unsigned long nr_records;
- void *records[KFREE_BULK_MAX_ENTR];
- struct kfree_rcu_bulk_data *next;
+ struct kvfree_rcu_bulk_data *next;
+ void *records[];
};
/*
* This macro defines how many entries the "records" array
* will contain. It is based on the fact that the size of
- * kfree_rcu_bulk_data structure becomes exactly one page.
+ * kvfree_rcu_bulk_data become exactly one page.
*/
-#define KFREE_BULK_MAX_ENTR \
- ((PAGE_SIZE - sizeof(struct kfree_rcu_bulk_data)) / sizeof(void *))
+#define KVFREE_BULK_MAX_ENTR \
+ ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *))
/**
* struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests
* @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period
* @head_free: List of kfree_rcu() objects waiting for a grace period
- * @bhead_free: Bulk-List of kfree_rcu() objects waiting for a grace period
+ * @bkvhead_free: Bulk-List of kfree_rcu() objects waiting for a grace period
* @krcp: Pointer to @kfree_rcu_cpu structure
*/
-
struct kfree_rcu_cpu_work {
struct rcu_work rcu_work;
struct rcu_head *head_free;
- struct kfree_rcu_bulk_data *bhead_free;
+ struct kvfree_rcu_bulk_data *bkvhead_free[2];
struct kfree_rcu_cpu *krcp;
};
@@ -2861,8 +2859,9 @@ struct kfree_rcu_cpu_work {
*/
struct kfree_rcu_cpu {
struct rcu_head *head;
- struct kfree_rcu_bulk_data *bhead;
- struct kfree_rcu_bulk_data *bcached;
+ struct kvfree_rcu_bulk_data *bkvhead[2];
+ struct kvfree_rcu_bulk_data *bkvcache[2];
+
struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES];
spinlock_t lock;
struct delayed_work monitor_work;
@@ -2875,7 +2874,7 @@ struct kfree_rcu_cpu {
static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc);
static __always_inline void
-debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead)
+debug_rcu_bhead_unqueue(struct kvfree_rcu_bulk_data *bhead)
{
#ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD
for (int i = 0; i < bhead->nr_records; i++)
@@ -2890,45 +2889,77 @@ debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead)
static void kfree_rcu_work(struct work_struct *work)
{
unsigned long flags;
+ struct kvfree_rcu_bulk_data *bkhead, *bknext;
+ struct kvfree_rcu_bulk_data *bvhead, *bvnext;
struct rcu_head *head, *next;
- struct kfree_rcu_bulk_data *bhead, *bnext;
struct kfree_rcu_cpu *krcp;
struct kfree_rcu_cpu_work *krwp;
+ int i;
krwp = container_of(to_rcu_work(work),
- struct kfree_rcu_cpu_work, rcu_work);
+ struct kfree_rcu_cpu_work, rcu_work);
+
krcp = krwp->krcp;
spin_lock_irqsave(&krcp->lock, flags);
+ /* Channel 1. */
+ bkhead = krwp->bkvhead_free[0];
+ krwp->bkvhead_free[0] = NULL;
+
+ /* Channel 2. */
+ bvhead = krwp->bkvhead_free[1];
+ krwp->bkvhead_free[1] = NULL;
+
+ /* Channel 3. */
head = krwp->head_free;
krwp->head_free = NULL;
- bhead = krwp->bhead_free;
- krwp->bhead_free = NULL;
spin_unlock_irqrestore(&krcp->lock, flags);
- /* "bhead" is now private, so traverse locklessly. */
- for (; bhead; bhead = bnext) {
- bnext = bhead->next;
+ /* kmalloc()/kfree() channel. */
+ for (; bkhead; bkhead = bknext) {
+ bknext = bkhead->next;
- debug_rcu_bhead_unqueue(bhead);
+ debug_rcu_bhead_unqueue(bkhead);
rcu_lock_acquire(&rcu_callback_map);
trace_rcu_invoke_kfree_bulk_callback(rcu_state.name,
- bhead->nr_records, bhead->records);
+ bkhead->nr_records, bkhead->records);
+
+ kfree_bulk(bkhead->nr_records, bkhead->records);
+ rcu_lock_release(&rcu_callback_map);
+
+ if (cmpxchg(&krcp->bkvcache[0], NULL, bkhead))
+ free_page((unsigned long) bkhead);
+
+ cond_resched_tasks_rcu_qs();
+ }
+
+ /* vmalloc()/vfree() channel. */
+ for (; bvhead; bvhead = bvnext) {
+ bvnext = bvhead->next;
+
+ debug_rcu_bhead_unqueue(bvhead);
- kfree_bulk(bhead->nr_records, bhead->records);
+ rcu_lock_acquire(&rcu_callback_map);
+ for (i = 0; i < bvhead->nr_records; i++) {
+ trace_rcu_invoke_kvfree_callback(rcu_state.name,
+ (struct rcu_head *) bvhead->records[i], 0);
+ vfree(bvhead->records[i]);
+ }
rcu_lock_release(&rcu_callback_map);
- if (cmpxchg(&krcp->bcached, NULL, bhead))
- free_page((unsigned long) bhead);
+ if (cmpxchg(&krcp->bkvcache[1], NULL, bvhead))
+ free_page((unsigned long) bvhead);
cond_resched_tasks_rcu_qs();
}
/*
- * We can end up here either with 1) vmalloc() pointers or 2) were low
- * on memory and could not allocate a bulk array. It can happen under
- * low memory condition when an allocation gets failed, so the "bulk"
- * path can not be temporarly used.
+ * This path covers emergency case only due to high
+ * memory pressure also means low memory condition,
+ * when we could not allocate a bulk array.
+ *
+ * Under that condition an object is queued to the
+ * list instead.
*/
for (; head; head = next) {
unsigned long offset = (unsigned long)head->func;
@@ -2965,21 +2996,34 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp)
krwp = &(krcp->krw_arr[i]);
/*
- * Try to detach bhead or head and attach it over any
+ * Try to detach bkvhead or head and attach it over any
* available corresponding free channel. It can be that
* a previous RCU batch is in progress, it means that
* immediately to queue another one is not possible so
* return false to tell caller to retry.
*/
- if ((krcp->bhead && !krwp->bhead_free) ||
+ if ((krcp->bkvhead[0] && !krwp->bkvhead_free[0]) ||
+ (krcp->bkvhead[1] && !krwp->bkvhead_free[1]) ||
(krcp->head && !krwp->head_free)) {
- /* Channel 1. */
- if (!krwp->bhead_free) {
- krwp->bhead_free = krcp->bhead;
- krcp->bhead = NULL;
+ /*
+ * Channel 1 corresponds to SLAB ptrs.
+ */
+ if (!krwp->bkvhead_free[0]) {
+ krwp->bkvhead_free[0] = krcp->bkvhead[0];
+ krcp->bkvhead[0] = NULL;
+ }
+
+ /*
+ * Channel 2 corresponds to vmalloc ptrs.
+ */
+ if (!krwp->bkvhead_free[1]) {
+ krwp->bkvhead_free[1] = krcp->bkvhead[1];
+ krcp->bkvhead[1] = NULL;
}
- /* Channel 2. */
+ /*
+ * Channel 3 corresponds to emergency path.
+ */
if (!krwp->head_free) {
krwp->head_free = krcp->head;
krcp->head = NULL;
@@ -2988,10 +3032,11 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp)
WRITE_ONCE(krcp->count, 0);
/*
- * One work is per one batch, so there are two "free channels",
- * "bhead_free" and "head_free" the batch can handle. It can be
- * that the work is in the pending state when two channels have
- * been detached following each other, one by one.
+ * One work is per one batch, so there are three
+ * "free channels", the batch can handle. It can
+ * be that the work is in the pending state when
+ * channels have been detached following by each
+ * other.
*/
queue_rcu_work(system_wq, &krwp->rcu_work);
queued = true;
@@ -3036,26 +3081,25 @@ static void kfree_rcu_monitor(struct work_struct *work)
}
static inline bool
-kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp,
- struct rcu_head *head, rcu_callback_t func)
+kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
{
- struct kfree_rcu_bulk_data *bnode;
+ struct kvfree_rcu_bulk_data *bnode;
+ int idx;
if (unlikely(!krcp->initialized))
return false;
lockdep_assert_held(&krcp->lock);
+ idx = !is_vmalloc_addr(ptr) ? 0:1;
/* Check if a new block is required. */
- if (!krcp->bhead ||
- krcp->bhead->nr_records == KFREE_BULK_MAX_ENTR) {
- bnode = xchg(&krcp->bcached, NULL);
- if (!bnode) {
- WARN_ON_ONCE(sizeof(struct kfree_rcu_bulk_data) > PAGE_SIZE);
-
- bnode = (struct kfree_rcu_bulk_data *)
+ if (!krcp->bkvhead[idx] ||
+ krcp->bkvhead[idx]->nr_records ==
+ KVFREE_BULK_MAX_ENTR) {
+ bnode = xchg(&krcp->bkvcache[idx], NULL);
+ if (!bnode)
+ bnode = (struct kvfree_rcu_bulk_data *)
__get_free_page(GFP_NOWAIT | __GFP_NOWARN);
- }
/* Switch to emergency path. */
if (unlikely(!bnode))
@@ -3063,30 +3107,30 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp,
/* Initialize the new block. */
bnode->nr_records = 0;
- bnode->next = krcp->bhead;
+ bnode->next = krcp->bkvhead[idx];
/* Attach it to the head. */
- krcp->bhead = bnode;
+ krcp->bkvhead[idx] = bnode;
}
/* Finally insert. */
- krcp->bhead->records[krcp->bhead->nr_records++] =
- (void *) head - (unsigned long) func;
+ krcp->bkvhead[idx]->records
+ [krcp->bkvhead[idx]->nr_records++] = ptr;
return true;
}
/*
- * Queue a request for lazy invocation of kfree_bulk()/kvfree() after a grace
- * period. Please note there are two paths are maintained, one is the main one
- * that uses kfree_bulk() interface and second one is emergency one, that is
- * used only when the main path can not be maintained temporary, due to memory
- * pressure.
+ * Queue a request for lazy invocation of appropriate free routine after a
+ * grace period. Please note there are three paths are maintained, two are the
+ * main ones that use array of pointers interface and third one is emergency
+ * one, that is used only when the main path can not be maintained temporary,
+ * due to memory pressure.
*
* Each kvfree_call_rcu() request is added to a batch. The batch will be drained
* every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch will
* be free'd in workqueue context. This allows us to: batch requests together to
- * reduce the number of grace periods during heavy kfree_rcu() load.
+ * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load.
*/
void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
{
@@ -3110,17 +3154,10 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
}
/*
- * We do not queue vmalloc pointers into array,
- * instead they are just queued to the list. We
- * do it because of:
- * a) to distinguish kmalloc()/vmalloc() ptrs;
- * b) there is no vmalloc_bulk() interface.
- *
* Under high memory pressure GFP_NOWAIT can fail,
* in that case the emergency path is maintained.
*/
- if (is_vmalloc_addr(ptr) ||
- !kfree_call_rcu_add_ptr_to_bulk(krcp, head, func)) {
+ if (!kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr)) {
head->func = func;
head->next = krcp->head;
krcp->head = head;
--
2.26.0.rc2.310.g2932bb562d-goog
From: "Uladzislau Rezki (Sony)" <[email protected]>
Rename kvfree_rcu() function to the kvfree_rcu_local()
one. The aim is to introduce the public API that would
conflict with this one.
Signed-off-by: Uladzislau Rezki (Sony) <[email protected]>
Reviewed-by: Joel Fernandes (Google) <[email protected]>
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
mm/list_lru.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/mm/list_lru.c b/mm/list_lru.c
index 0f1f6b06b7f36..386424688f805 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -383,14 +383,14 @@ static void memcg_destroy_list_lru_node(struct list_lru_node *nlru)
struct list_lru_memcg *memcg_lrus;
/*
* This is called when shrinker has already been unregistered,
- * and nobody can use it. So, there is no need to use kvfree_rcu().
+ * and nobody can use it. So, there is no need to use kvfree_rcu_local().
*/
memcg_lrus = rcu_dereference_protected(nlru->memcg_lrus, true);
__memcg_destroy_list_lru_node(memcg_lrus, 0, memcg_nr_cache_ids);
kvfree(memcg_lrus);
}
-static void kvfree_rcu(struct rcu_head *head)
+static void kvfree_rcu_local(struct rcu_head *head)
{
struct list_lru_memcg *mlru;
@@ -429,7 +429,7 @@ static int memcg_update_list_lru_node(struct list_lru_node *nlru,
rcu_assign_pointer(nlru->memcg_lrus, new);
spin_unlock_irq(&nlru->lock);
- call_rcu(&old->rcu, kvfree_rcu);
+ call_rcu(&old->rcu, kvfree_rcu_local);
return 0;
}
--
2.26.0.rc2.310.g2932bb562d-goog
From: "Uladzislau Rezki (Sony)" <[email protected]>
We can simplify KFREE_BULK_MAX_ENTR macro and get rid of
magic numbers which were used to make the structure to be
exactly one page.
Suggested-by: Boqun Feng <[email protected]>
Reviewed-by: Joel Fernandes (Google) <[email protected]>
Signed-off-by: Uladzislau Rezki (Sony) <[email protected]>
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
kernel/rcu/tree.c | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 8dfa4b32e4d00..cfe456e68c644 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2807,13 +2807,6 @@ EXPORT_SYMBOL_GPL(call_rcu);
#define KFREE_DRAIN_JIFFIES (HZ / 50)
#define KFREE_N_BATCHES 2
-/*
- * This macro defines how many entries the "records" array
- * will contain. It is based on the fact that the size of
- * kfree_rcu_bulk_data structure becomes exactly one page.
- */
-#define KFREE_BULK_MAX_ENTR ((PAGE_SIZE / sizeof(void *)) - 3)
-
/**
* struct kfree_rcu_bulk_data - single block to store kfree_rcu() pointers
* @nr_records: Number of active pointers in the array
@@ -2827,6 +2820,14 @@ struct kfree_rcu_bulk_data {
struct kfree_rcu_bulk_data *next;
};
+/*
+ * This macro defines how many entries the "records" array
+ * will contain. It is based on the fact that the size of
+ * kfree_rcu_bulk_data structure becomes exactly one page.
+ */
+#define KFREE_BULK_MAX_ENTR \
+ ((PAGE_SIZE - sizeof(struct kfree_rcu_bulk_data)) / sizeof(void *))
+
/**
* struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests
* @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period
--
2.26.0.rc2.310.g2932bb562d-goog
From: "Uladzislau Rezki (Sony)" <[email protected]>
Rename rcu_invoke_kfree_callback to rcu_invoke_kvfree_callback.
Do the same with second trace event, that is rcu_kfree_callback,
it becomes rcu_kvfree_callback. The reason is to be aligned with
kvfree notation.
Signed-off-by: Uladzislau Rezki (Sony) <[email protected]>
Reviewed-by: Joel Fernandes (Google) <[email protected]>
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
include/trace/events/rcu.h | 8 ++++----
kernel/rcu/tiny.c | 2 +-
kernel/rcu/tree.c | 4 ++--
3 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h
index f9a7811148e2a..0ee93d0b1daa8 100644
--- a/include/trace/events/rcu.h
+++ b/include/trace/events/rcu.h
@@ -506,13 +506,13 @@ TRACE_EVENT_RCU(rcu_callback,
/*
* Tracepoint for the registration of a single RCU callback of the special
- * kfree() form. The first argument is the RCU type, the second argument
+ * kvfree() form. The first argument is the RCU type, the second argument
* is a pointer to the RCU callback, the third argument is the offset
* of the callback within the enclosing RCU-protected data structure,
* the fourth argument is the number of lazy callbacks queued, and the
* fifth argument is the total number of callbacks queued.
*/
-TRACE_EVENT_RCU(rcu_kfree_callback,
+TRACE_EVENT_RCU(rcu_kvfree_callback,
TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset,
long qlen),
@@ -596,12 +596,12 @@ TRACE_EVENT_RCU(rcu_invoke_callback,
/*
* Tracepoint for the invocation of a single RCU callback of the special
- * kfree() form. The first argument is the RCU flavor, the second
+ * kvfree() form. The first argument is the RCU flavor, the second
* argument is a pointer to the RCU callback, and the third argument
* is the offset of the callback within the enclosing RCU-protected
* data structure.
*/
-TRACE_EVENT_RCU(rcu_invoke_kfree_callback,
+TRACE_EVENT_RCU(rcu_invoke_kvfree_callback,
TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset),
diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index 4b99f7b88beec..3dd8e6e207b09 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -86,7 +86,7 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head)
rcu_lock_acquire(&rcu_callback_map);
if (__is_kfree_rcu_offset(offset)) {
- trace_rcu_invoke_kfree_callback("", head, offset);
+ trace_rcu_invoke_kvfree_callback("", head, offset);
kvfree((void *)head - offset);
rcu_lock_release(&rcu_callback_map);
return true;
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 2d10c50621c38..88b744ce896c0 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2744,7 +2744,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func)
// If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock.
rcu_segcblist_enqueue(&rdp->cblist, head);
if (__is_kfree_rcu_offset((unsigned long)func))
- trace_rcu_kfree_callback(rcu_state.name, head,
+ trace_rcu_kvfree_callback(rcu_state.name, head,
(unsigned long)func,
rcu_segcblist_n_cbs(&rdp->cblist));
else
@@ -2935,7 +2935,7 @@ static void kfree_rcu_work(struct work_struct *work)
next = head->next;
debug_rcu_head_unqueue(head);
rcu_lock_acquire(&rcu_callback_map);
- trace_rcu_invoke_kfree_callback(rcu_state.name, head, offset);
+ trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset)))
kvfree((void *)head - offset);
--
2.26.0.rc2.310.g2932bb562d-goog
From: "Uladzislau Rezki (Sony)" <[email protected]>
kvfree_rcu() can deal with an allocated memory that is obtained
via kvmalloc(). It can return two types of allocated memory or
"pointers", one can belong to regular SLAB allocator and another
one can be vmalloc one. It depends on requested size and memory
pressure.
Based on that, two streams are split, thus if a pointer belongs
to vmalloc allocator it is queued to the list, otherwise SLAB
one is queued into "bulk array" for further processing.
The main reason of such splitting is:
a) to distinguish kmalloc()/vmalloc() ptrs;
b) there is no vmalloc_bulk() interface.
As of now we have list_lru.c user that needs such interface,
also there will be new comers. Apart of that it is preparation
to have a head-less variant later.
Signed-off-by: Uladzislau Rezki (Sony) <[email protected]>
Reviewed-by: Joel Fernandes (Google) <[email protected]>
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
include/linux/rcupdate.h | 9 +++++++++
kernel/rcu/tiny.c | 3 ++-
kernel/rcu/tree.c | 17 ++++++++++++-----
3 files changed, 23 insertions(+), 6 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 3598bbb5ff407..8b7128d0860e2 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -870,6 +870,15 @@ do { \
__kfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \
} while (0)
+/**
+ * kvfree_rcu() - kvfree an object after a grace period.
+ * @ptr: pointer to kvfree
+ * @rhf: the name of the struct rcu_head within the type of @ptr.
+ *
+ * Same as kfree_rcu(), just simple alias.
+ */
+#define kvfree_rcu(ptr, rhf) kfree_rcu(ptr, rhf)
+
/*
* Place this after a lock-acquisition primitive to guarantee that
* an UNLOCK+LOCK pair acts as a full barrier. This guarantee applies
diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index dd572ce7c7479..4b99f7b88beec 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -23,6 +23,7 @@
#include <linux/cpu.h>
#include <linux/prefetch.h>
#include <linux/slab.h>
+#include <linux/mm.h>
#include "rcu.h"
@@ -86,7 +87,7 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head)
rcu_lock_acquire(&rcu_callback_map);
if (__is_kfree_rcu_offset(offset)) {
trace_rcu_invoke_kfree_callback("", head, offset);
- kfree((void *)head - offset);
+ kvfree((void *)head - offset);
rcu_lock_release(&rcu_callback_map);
return true;
}
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 4eb424eb44acb..2d10c50621c38 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2925,9 +2925,9 @@ static void kfree_rcu_work(struct work_struct *work)
}
/*
- * Emergency case only. It can happen under low memory
- * condition when an allocation gets failed, so the "bulk"
- * path can not be temporary maintained.
+ * vmalloc() pointers end up here also emergency case. It can
+ * happen under low memory condition when an allocation gets
+ * failed, so the "bulk" path can not be temporary maintained.
*/
for (; head; head = next) {
unsigned long offset = (unsigned long)head->func;
@@ -2938,7 +2938,7 @@ static void kfree_rcu_work(struct work_struct *work)
trace_rcu_invoke_kfree_callback(rcu_state.name, head, offset);
if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset)))
- kfree((void *)head - offset);
+ kvfree((void *)head - offset);
rcu_lock_release(&rcu_callback_map);
cond_resched_tasks_rcu_qs();
@@ -3112,10 +3112,17 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
}
/*
+ * We do not queue vmalloc pointers into array,
+ * instead they are just queued to the list. We
+ * do it because of:
+ * a) to distinguish kmalloc()/vmalloc() ptrs;
+ * b) there is no vmalloc_bulk() interface.
+ *
* Under high memory pressure GFP_NOWAIT can fail,
* in that case the emergency path is maintained.
*/
- if (unlikely(!kfree_call_rcu_add_ptr_to_bulk(krcp, head, func))) {
+ if (is_vmalloc_addr((void *) head - (unsigned long) func) ||
+ !kfree_call_rcu_add_ptr_to_bulk(krcp, head, func)) {
head->func = func;
head->next = krcp->head;
krcp->head = head;
--
2.26.0.rc2.310.g2932bb562d-goog
Clarify emergency path comment better in kfree_rcu().
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
kernel/rcu/tree.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 95d1f5e20d5ec..8dfa4b32e4d00 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2924,9 +2924,10 @@ static void kfree_rcu_work(struct work_struct *work)
}
/*
- * vmalloc() pointers end up here also emergency case. It can
- * happen under low memory condition when an allocation gets
- * failed, so the "bulk" path can not be temporary maintained.
+ * We can end up here either with 1) vmalloc() pointers or 2) were low
+ * on memory and could not allocate a bulk array. It can happen under
+ * low memory condition when an allocation gets failed, so the "bulk"
+ * path can not be temporarly used.
*/
for (; head; head = next) {
unsigned long offset = (unsigned long)head->func;
--
2.26.0.rc2.310.g2932bb562d-goog
From: "Uladzislau Rezki (Sony)" <[email protected]>
Since there is newly introduced kvfree_rcu() API, there is no need in
queuing and using call_rcu() to kvfree() an object after the GP.
Remove kvfree_rcu_local() function and replace call_rcu() by new
kvfree_rcu() API that does the same but in more efficient way.
Signed-off-by: Uladzislau Rezki (Sony) <[email protected]>
Reviewed-by: Joel Fernandes (Google) <[email protected]>
Signed-off-by: Joel Fernandes (Google) <[email protected]>
---
mm/list_lru.c | 13 +++----------
1 file changed, 3 insertions(+), 10 deletions(-)
diff --git a/mm/list_lru.c b/mm/list_lru.c
index 386424688f805..69becdb224080 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -12,6 +12,7 @@
#include <linux/slab.h>
#include <linux/mutex.h>
#include <linux/memcontrol.h>
+#include <linux/rcupdate.h>
#include "slab.h"
#ifdef CONFIG_MEMCG_KMEM
@@ -383,21 +384,13 @@ static void memcg_destroy_list_lru_node(struct list_lru_node *nlru)
struct list_lru_memcg *memcg_lrus;
/*
* This is called when shrinker has already been unregistered,
- * and nobody can use it. So, there is no need to use kvfree_rcu_local().
+ * and nobody can use it. So, there is no need to use kvfree_rcu().
*/
memcg_lrus = rcu_dereference_protected(nlru->memcg_lrus, true);
__memcg_destroy_list_lru_node(memcg_lrus, 0, memcg_nr_cache_ids);
kvfree(memcg_lrus);
}
-static void kvfree_rcu_local(struct rcu_head *head)
-{
- struct list_lru_memcg *mlru;
-
- mlru = container_of(head, struct list_lru_memcg, rcu);
- kvfree(mlru);
-}
-
static int memcg_update_list_lru_node(struct list_lru_node *nlru,
int old_size, int new_size)
{
@@ -429,7 +422,7 @@ static int memcg_update_list_lru_node(struct list_lru_node *nlru,
rcu_assign_pointer(nlru->memcg_lrus, new);
spin_unlock_irq(&nlru->lock);
- call_rcu(&old->rcu, kvfree_rcu_local);
+ kvfree_rcu(old, rcu);
return 0;
}
--
2.26.0.rc2.310.g2932bb562d-goog
Hi "Joel,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on rcu/dev]
[also build test ERROR on rcu/rcu/next next-20200327]
[cannot apply to linus/master linux/master v5.6]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Joel-Fernandes-Google/kfree_rcu-improvements-for-rcu-dev/20200330-113719
base: https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
config: powerpc-defconfig (attached as .config)
compiler: powerpc64-linux-gcc (GCC) 9.3.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.3.0 make.cross ARCH=powerpc
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <[email protected]>
All errors (new ones prefixed by >>):
kernel/rcu/tree.c: In function 'kfree_rcu_work':
>> kernel/rcu/tree.c:2946:4: error: implicit declaration of function 'vfree'; did you mean 'kfree'? [-Werror=implicit-function-declaration]
2946 | vfree(bvhead->records[i]);
| ^~~~~
| kfree
cc1: some warnings being treated as errors
vim +2946 kernel/rcu/tree.c
2884
2885 /*
2886 * This function is invoked in workqueue context after a grace period.
2887 * It frees all the objects queued on ->bhead_free or ->head_free.
2888 */
2889 static void kfree_rcu_work(struct work_struct *work)
2890 {
2891 unsigned long flags;
2892 struct kvfree_rcu_bulk_data *bkhead, *bknext;
2893 struct kvfree_rcu_bulk_data *bvhead, *bvnext;
2894 struct rcu_head *head, *next;
2895 struct kfree_rcu_cpu *krcp;
2896 struct kfree_rcu_cpu_work *krwp;
2897 int i;
2898
2899 krwp = container_of(to_rcu_work(work),
2900 struct kfree_rcu_cpu_work, rcu_work);
2901
2902 krcp = krwp->krcp;
2903 spin_lock_irqsave(&krcp->lock, flags);
2904 /* Channel 1. */
2905 bkhead = krwp->bkvhead_free[0];
2906 krwp->bkvhead_free[0] = NULL;
2907
2908 /* Channel 2. */
2909 bvhead = krwp->bkvhead_free[1];
2910 krwp->bkvhead_free[1] = NULL;
2911
2912 /* Channel 3. */
2913 head = krwp->head_free;
2914 krwp->head_free = NULL;
2915 spin_unlock_irqrestore(&krcp->lock, flags);
2916
2917 /* kmalloc()/kfree() channel. */
2918 for (; bkhead; bkhead = bknext) {
2919 bknext = bkhead->next;
2920
2921 debug_rcu_bhead_unqueue(bkhead);
2922
2923 rcu_lock_acquire(&rcu_callback_map);
2924 trace_rcu_invoke_kfree_bulk_callback(rcu_state.name,
2925 bkhead->nr_records, bkhead->records);
2926
2927 kfree_bulk(bkhead->nr_records, bkhead->records);
2928 rcu_lock_release(&rcu_callback_map);
2929
2930 if (cmpxchg(&krcp->bkvcache[0], NULL, bkhead))
2931 free_page((unsigned long) bkhead);
2932
2933 cond_resched_tasks_rcu_qs();
2934 }
2935
2936 /* vmalloc()/vfree() channel. */
2937 for (; bvhead; bvhead = bvnext) {
2938 bvnext = bvhead->next;
2939
2940 debug_rcu_bhead_unqueue(bvhead);
2941
2942 rcu_lock_acquire(&rcu_callback_map);
2943 for (i = 0; i < bvhead->nr_records; i++) {
2944 trace_rcu_invoke_kvfree_callback(rcu_state.name,
2945 (struct rcu_head *) bvhead->records[i], 0);
> 2946 vfree(bvhead->records[i]);
2947 }
2948 rcu_lock_release(&rcu_callback_map);
2949
2950 if (cmpxchg(&krcp->bkvcache[1], NULL, bvhead))
2951 free_page((unsigned long) bvhead);
2952
2953 cond_resched_tasks_rcu_qs();
2954 }
2955
2956 /*
2957 * This path covers emergency case only due to high
2958 * memory pressure also means low memory condition,
2959 * when we could not allocate a bulk array.
2960 *
2961 * Under that condition an object is queued to the
2962 * list instead.
2963 */
2964 for (; head; head = next) {
2965 unsigned long offset = (unsigned long)head->func;
2966 void *ptr = (void *)head - offset;
2967
2968 next = head->next;
2969 debug_rcu_head_unqueue((struct rcu_head *)ptr);
2970 rcu_lock_acquire(&rcu_callback_map);
2971 trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
2972
2973 if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset)))
2974 kvfree(ptr);
2975
2976 rcu_lock_release(&rcu_callback_map);
2977 cond_resched_tasks_rcu_qs();
2978 }
2979 }
2980
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/[email protected]
Hi "Joel,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on rcu/dev]
[also build test ERROR on rcu/rcu/next next-20200327]
[cannot apply to linus/master linux/master v5.6]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Joel-Fernandes-Google/kfree_rcu-improvements-for-rcu-dev/20200330-113719
base: https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
config: i386-allyesconfig (attached as .config)
compiler: gcc-7 (Debian 7.5.0-5) 7.5.0
reproduce:
# save the attached .config to linux build tree
make ARCH=i386
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <[email protected]>
All errors (new ones prefixed by >>):
kernel/rcu/tree.c: In function 'debug_rcu_bhead_unqueue':
>> kernel/rcu/tree.c:2880:2: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
for (int i = 0; i < bhead->nr_records; i++)
^~~
kernel/rcu/tree.c:2880:2: note: use option -std=c99, -std=gnu99, -std=c11 or -std=gnu11 to compile your code
vim +/for +2880 kernel/rcu/tree.c
2875
2876 static __always_inline void
2877 debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead)
2878 {
2879 #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD
> 2880 for (int i = 0; i < bhead->nr_records; i++)
2881 debug_rcu_head_unqueue((struct rcu_head *)(bhead->records[i]));
2882 #endif
2883 }
2884
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/[email protected]
Hi "Joel,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on rcu/dev]
[also build test ERROR on rcu/rcu/next next-20200327]
[cannot apply to linus/master linux/master v5.6]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Joel-Fernandes-Google/kfree_rcu-improvements-for-rcu-dev/20200330-113719
base: https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
config: mips-randconfig-a001-20200330 (attached as .config)
compiler: mips64el-linux-gcc (GCC) 5.5.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=5.5.0 make.cross ARCH=mips
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <[email protected]>
All errors (new ones prefixed by >>):
kernel/rcu/tree.c: In function 'kfree_rcu_work':
>> kernel/rcu/tree.c:2946:4: error: implicit declaration of function 'vfree' [-Werror=implicit-function-declaration]
vfree(bvhead->records[i]);
^
cc1: some warnings being treated as errors
vim +/vfree +2946 kernel/rcu/tree.c
2884
2885 /*
2886 * This function is invoked in workqueue context after a grace period.
2887 * It frees all the objects queued on ->bhead_free or ->head_free.
2888 */
2889 static void kfree_rcu_work(struct work_struct *work)
2890 {
2891 unsigned long flags;
2892 struct kvfree_rcu_bulk_data *bkhead, *bknext;
2893 struct kvfree_rcu_bulk_data *bvhead, *bvnext;
2894 struct rcu_head *head, *next;
2895 struct kfree_rcu_cpu *krcp;
2896 struct kfree_rcu_cpu_work *krwp;
2897 int i;
2898
2899 krwp = container_of(to_rcu_work(work),
2900 struct kfree_rcu_cpu_work, rcu_work);
2901
2902 krcp = krwp->krcp;
2903 spin_lock_irqsave(&krcp->lock, flags);
2904 /* Channel 1. */
2905 bkhead = krwp->bkvhead_free[0];
2906 krwp->bkvhead_free[0] = NULL;
2907
2908 /* Channel 2. */
2909 bvhead = krwp->bkvhead_free[1];
2910 krwp->bkvhead_free[1] = NULL;
2911
2912 /* Channel 3. */
2913 head = krwp->head_free;
2914 krwp->head_free = NULL;
2915 spin_unlock_irqrestore(&krcp->lock, flags);
2916
2917 /* kmalloc()/kfree() channel. */
2918 for (; bkhead; bkhead = bknext) {
2919 bknext = bkhead->next;
2920
2921 debug_rcu_bhead_unqueue(bkhead);
2922
2923 rcu_lock_acquire(&rcu_callback_map);
2924 trace_rcu_invoke_kfree_bulk_callback(rcu_state.name,
2925 bkhead->nr_records, bkhead->records);
2926
2927 kfree_bulk(bkhead->nr_records, bkhead->records);
2928 rcu_lock_release(&rcu_callback_map);
2929
2930 if (cmpxchg(&krcp->bkvcache[0], NULL, bkhead))
2931 free_page((unsigned long) bkhead);
2932
2933 cond_resched_tasks_rcu_qs();
2934 }
2935
2936 /* vmalloc()/vfree() channel. */
2937 for (; bvhead; bvhead = bvnext) {
2938 bvnext = bvhead->next;
2939
2940 debug_rcu_bhead_unqueue(bvhead);
2941
2942 rcu_lock_acquire(&rcu_callback_map);
2943 for (i = 0; i < bvhead->nr_records; i++) {
2944 trace_rcu_invoke_kvfree_callback(rcu_state.name,
2945 (struct rcu_head *) bvhead->records[i], 0);
> 2946 vfree(bvhead->records[i]);
2947 }
2948 rcu_lock_release(&rcu_callback_map);
2949
2950 if (cmpxchg(&krcp->bkvcache[1], NULL, bvhead))
2951 free_page((unsigned long) bvhead);
2952
2953 cond_resched_tasks_rcu_qs();
2954 }
2955
2956 /*
2957 * This path covers emergency case only due to high
2958 * memory pressure also means low memory condition,
2959 * when we could not allocate a bulk array.
2960 *
2961 * Under that condition an object is queued to the
2962 * list instead.
2963 */
2964 for (; head; head = next) {
2965 unsigned long offset = (unsigned long)head->func;
2966 void *ptr = (void *)head - offset;
2967
2968 next = head->next;
2969 debug_rcu_head_unqueue((struct rcu_head *)ptr);
2970 rcu_lock_acquire(&rcu_callback_map);
2971 trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
2972
2973 if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset)))
2974 kvfree(ptr);
2975
2976 rcu_lock_release(&rcu_callback_map);
2977 cond_resched_tasks_rcu_qs();
2978 }
2979 }
2980
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/[email protected]
Hello, Joel.
Sent out the patch fixing build error.
--
Vlad Rezki
> Hi "Joel,
>
> Thank you for the patch! Yet something to improve:
>
> [auto build test ERROR on rcu/dev]
> [also build test ERROR on rcu/rcu/next next-20200327]
> [cannot apply to linus/master linux/master v5.6]
> [if your patch is applied to the wrong git tree, please drop us a note to help
> improve the system. BTW, we also suggest to use '--base' option to specify the
> base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
>
> url: https://github.com/0day-ci/linux/commits/Joel-Fernandes-Google/kfree_rcu-improvements-for-rcu-dev/20200330-113719
> base: https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
> config: mips-randconfig-a001-20200330 (attached as .config)
> compiler: mips64el-linux-gcc (GCC) 5.5.0
> reproduce:
> wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
> chmod +x ~/bin/make.cross
> # save the attached .config to linux build tree
> GCC_VERSION=5.5.0 make.cross ARCH=mips
>
> If you fix the issue, kindly add following tag
> Reported-by: kbuild test robot <[email protected]>
>
> All errors (new ones prefixed by >>):
>
> kernel/rcu/tree.c: In function 'kfree_rcu_work':
> >> kernel/rcu/tree.c:2946:4: error: implicit declaration of function 'vfree' [-Werror=implicit-function-declaration]
> vfree(bvhead->records[i]);
> ^
> cc1: some warnings being treated as errors
>
> vim +/vfree +2946 kernel/rcu/tree.c
>
> 2884
> 2885 /*
> 2886 * This function is invoked in workqueue context after a grace period.
> 2887 * It frees all the objects queued on ->bhead_free or ->head_free.
> 2888 */
> 2889 static void kfree_rcu_work(struct work_struct *work)
> 2890 {
> 2891 unsigned long flags;
> 2892 struct kvfree_rcu_bulk_data *bkhead, *bknext;
> 2893 struct kvfree_rcu_bulk_data *bvhead, *bvnext;
> 2894 struct rcu_head *head, *next;
> 2895 struct kfree_rcu_cpu *krcp;
> 2896 struct kfree_rcu_cpu_work *krwp;
> 2897 int i;
> 2898
> 2899 krwp = container_of(to_rcu_work(work),
> 2900 struct kfree_rcu_cpu_work, rcu_work);
> 2901
> 2902 krcp = krwp->krcp;
> 2903 spin_lock_irqsave(&krcp->lock, flags);
> 2904 /* Channel 1. */
> 2905 bkhead = krwp->bkvhead_free[0];
> 2906 krwp->bkvhead_free[0] = NULL;
> 2907
> 2908 /* Channel 2. */
> 2909 bvhead = krwp->bkvhead_free[1];
> 2910 krwp->bkvhead_free[1] = NULL;
> 2911
> 2912 /* Channel 3. */
> 2913 head = krwp->head_free;
> 2914 krwp->head_free = NULL;
> 2915 spin_unlock_irqrestore(&krcp->lock, flags);
> 2916
> 2917 /* kmalloc()/kfree() channel. */
> 2918 for (; bkhead; bkhead = bknext) {
> 2919 bknext = bkhead->next;
> 2920
> 2921 debug_rcu_bhead_unqueue(bkhead);
> 2922
> 2923 rcu_lock_acquire(&rcu_callback_map);
> 2924 trace_rcu_invoke_kfree_bulk_callback(rcu_state.name,
> 2925 bkhead->nr_records, bkhead->records);
> 2926
> 2927 kfree_bulk(bkhead->nr_records, bkhead->records);
> 2928 rcu_lock_release(&rcu_callback_map);
> 2929
> 2930 if (cmpxchg(&krcp->bkvcache[0], NULL, bkhead))
> 2931 free_page((unsigned long) bkhead);
> 2932
> 2933 cond_resched_tasks_rcu_qs();
> 2934 }
> 2935
> 2936 /* vmalloc()/vfree() channel. */
> 2937 for (; bvhead; bvhead = bvnext) {
> 2938 bvnext = bvhead->next;
> 2939
> 2940 debug_rcu_bhead_unqueue(bvhead);
> 2941
> 2942 rcu_lock_acquire(&rcu_callback_map);
> 2943 for (i = 0; i < bvhead->nr_records; i++) {
> 2944 trace_rcu_invoke_kvfree_callback(rcu_state.name,
> 2945 (struct rcu_head *) bvhead->records[i], 0);
> > 2946 vfree(bvhead->records[i]);
> 2947 }
> 2948 rcu_lock_release(&rcu_callback_map);
> 2949
> 2950 if (cmpxchg(&krcp->bkvcache[1], NULL, bvhead))
> 2951 free_page((unsigned long) bvhead);
> 2952
> 2953 cond_resched_tasks_rcu_qs();
> 2954 }
> 2955
> 2956 /*
> 2957 * This path covers emergency case only due to high
> 2958 * memory pressure also means low memory condition,
> 2959 * when we could not allocate a bulk array.
> 2960 *
> 2961 * Under that condition an object is queued to the
> 2962 * list instead.
> 2963 */
> 2964 for (; head; head = next) {
> 2965 unsigned long offset = (unsigned long)head->func;
> 2966 void *ptr = (void *)head - offset;
> 2967
> 2968 next = head->next;
> 2969 debug_rcu_head_unqueue((struct rcu_head *)ptr);
> 2970 rcu_lock_acquire(&rcu_callback_map);
> 2971 trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
> 2972
> 2973 if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset)))
> 2974 kvfree(ptr);
> 2975
> 2976 rcu_lock_release(&rcu_callback_map);
> 2977 cond_resched_tasks_rcu_qs();
> 2978 }
> 2979 }
> 2980
>
> ---
> 0-DAY CI Kernel Test Service, Intel Corporation
> https://lists.01.org/hyperkitty/list/[email protected]
On Mon, Mar 30, 2020 at 05:29:51PM +0200, Uladzislau Rezki wrote:
> Hello, Joel.
>
> Sent out the patch fixing build error.
... where? It didn't get cc'd to linux-mm?
On Mon, Mar 30, 2020 at 08:31:49AM -0700, Matthew Wilcox wrote:
> On Mon, Mar 30, 2020 at 05:29:51PM +0200, Uladzislau Rezki wrote:
> > Hello, Joel.
> >
> > Sent out the patch fixing build error.
>
> ... where? It didn't get cc'd to linux-mm?
The kbuild test robot complained. Prior than the build error, the
patch didn't seem all that relevant to linux-mm. ;-)
Thanx, Paul
On Mon, Mar 30, 2020 at 08:37:02AM -0700, Paul E. McKenney wrote:
> On Mon, Mar 30, 2020 at 08:31:49AM -0700, Matthew Wilcox wrote:
> > On Mon, Mar 30, 2020 at 05:29:51PM +0200, Uladzislau Rezki wrote:
> > > Hello, Joel.
> > >
> > > Sent out the patch fixing build error.
> >
> > ... where? It didn't get cc'd to linux-mm?
>
> The kbuild test robot complained. Prior than the build error, the
> patch didn't seem all that relevant to linux-mm. ;-)
I asked the preprocessor to tell me why I didn't hit this in my tree. Seems
it because vmalloc.h is included in my tree through the following includes.
./include/linux/nmi.h
./arch/x86/include/asm/nmi.h
./arch/x86/include/asm/io.h
./include/asm-generic/io.h
./include/linux/vmalloc.h
Such paths may not exist in kbuild robot's tree, so I will apply Vlad's patch to
fix this and push it to my rcu/kfree branch.
thanks,
- Joel
On Mon, Mar 30, 2020 at 01:16:06PM -0400, Joel Fernandes wrote:
> On Mon, Mar 30, 2020 at 08:37:02AM -0700, Paul E. McKenney wrote:
> > On Mon, Mar 30, 2020 at 08:31:49AM -0700, Matthew Wilcox wrote:
> > > On Mon, Mar 30, 2020 at 05:29:51PM +0200, Uladzislau Rezki wrote:
> > > > Hello, Joel.
> > > >
> > > > Sent out the patch fixing build error.
> > >
> > > ... where? It didn't get cc'd to linux-mm?
> >
> > The kbuild test robot complained. Prior than the build error, the
> > patch didn't seem all that relevant to linux-mm. ;-)
>
> I asked the preprocessor to tell me why I didn't hit this in my tree. Seems
> it because vmalloc.h is included in my tree through the following includes.
>
Same to me, i did not manage to hit that build error.
--
Vlad Rezki
On Mon, Mar 30, 2020 at 07:43:38PM +0200, Uladzislau Rezki wrote:
> On Mon, Mar 30, 2020 at 01:16:06PM -0400, Joel Fernandes wrote:
> > On Mon, Mar 30, 2020 at 08:37:02AM -0700, Paul E. McKenney wrote:
> > > On Mon, Mar 30, 2020 at 08:31:49AM -0700, Matthew Wilcox wrote:
> > > > On Mon, Mar 30, 2020 at 05:29:51PM +0200, Uladzislau Rezki wrote:
> > > > > Hello, Joel.
> > > > >
> > > > > Sent out the patch fixing build error.
> > > >
> > > > ... where? It didn't get cc'd to linux-mm?
> > >
> > > The kbuild test robot complained. Prior than the build error, the
> > > patch didn't seem all that relevant to linux-mm. ;-)
> >
> > I asked the preprocessor to tell me why I didn't hit this in my tree. Seems
> > it because vmalloc.h is included in my tree through the following includes.
> >
> Same to me, i did not manage to hit that build error.
This is a common occurrence for me. The kbuild test robot can be very
helpful for this sort of thing. ;-)
Thanx, Paul
On Mon, Mar 30, 2020 at 03:00:18PM +0800, kbuild test robot wrote:
> Hi "Joel,
>
> Thank you for the patch! Yet something to improve:
>
> [auto build test ERROR on rcu/dev]
> [also build test ERROR on rcu/rcu/next next-20200327]
> [cannot apply to linus/master linux/master v5.6]
> [if your patch is applied to the wrong git tree, please drop us a note to help
> improve the system. BTW, we also suggest to use '--base' option to specify the
> base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
Should have run allmodconfig, this is a slightly older patch and did not see
a problem earlier in prior testing.
Will make the trivial fix now and update my tree, thanks!
- Joel