2022-04-22 21:04:06

by John Ogness

[permalink] [raw]
Subject: [PATCH printk v3 00/15] printk/for-next

This is v3 of a series to implement a kthread for each registered
console. v2 is here [0]. The kthreads locklessly retrieve the
records from the printk ringbuffer and also do not cause any lock
contention between each other. This allows consoles to run at full
speed. For example, a netconsole is able to dump records much
faster than a serial or vt console. Also, during normal operation,
printk() callers are completely decoupled from console printing.

There are situations where kthread printing is not sufficient. For
example, during panic situations, where the kthreads may not get a
chance to schedule. In such cases, the current method of attempting
to print directly within the printk() caller context is used. New
functions printk_prefer_direct_enter() and
printk_prefer_direct_exit() are made available to mark areas of the
kernel where direct printing is preferred. (These should only be
areas that do not occur during normal operation.)

This series also introduces pr_flush(): a might_sleep() function
that will block until all active printing threads have caught up
to the latest record at the time of the pr_flush() call. This
function is useful, for example, to wait until pending records
are flushed to consoles before suspending.

Note that this series does *not* increase the reliability of console
printing. Rather it focuses on the non-interference aspect of
printk() by decoupling printk() callers from printing (during normal
operation). Nonetheless, the reliability aspect should not worsen
due to this series.

John Ogness

[0] https://lore.kernel.org/lkml/[email protected]

Changes since v2:

- Threaded printers no longer care about preferred direct printing.
As with v1, they will print whenever they are not blocked.

- Provide a separate patch to fix a missing memory barrier in
wake_up_klogd() and add memory barrier comments to all
appropriate @log_wait usage sites.

- Provide a separate patch to wake all waiters.

- Provide a separate patch to wake waiters for deferred console
output and add comments explaining why.

- Introduce console_lock_single_hold() and
console_unlock_single_release() to acquire @console_sem and lock a
single threaded printer. This allows console start/stop and
console unregistration with synchronized con->flags and without
disturbing other threaded printers.

- Introduce __console_is_usable() to avoid some redundance between
threaded and direct printing code.

- Do not create a printer thread if con->write() is not set. (I do
not understand why we even allow registration if con->write() is
not set. The checks were added in 2.1.31 for no obvious reason.)

- Only allow handovers between console_trylock() contexts. A
console_lock() context cannot handover the console_lock to a
console_trylock() context because the blocked kthreads would need
to be unblocked via mutex.

- console_flush_all() returns true only if at least one console is
usable and all messages to all usable consoles were printed.
Otherwise it returns false.

- Remove redundant panic check in console_unlock().

- Rename printk_console_msg() to con_printk() and use syntax similar
to dev_printk(). (I did not name it console_printk() because there
already exists a symbol with that name.)

- Remove blocked check in register_console() since it is always
true.

- In unregister_console(), stop the kthread after the console has
been removed from the list. Use the per-console mutex for
synchronized kthread stopping.

- Use the console_lock for synchronized activation of the fallback
permanent direct printing mode.

- Use the same checks in printer_should_wake() as in
printk_kthread_func() to avoid infinite loop danger.

- Rename PRINTK_PENDING_OUTPUT flag to PRINTK_PENDING_DIRECT_OUTPUT.

- Expand commit messages relating to memory barriers, kthreads, and
the usage of the per-console mutex.

John Ogness (15):
printk: rename cpulock functions
printk: cpu sync always disable interrupts
printk: add missing memory barrier to wake_up_klogd()
printk: wake up all waiters
printk: wake waiters for safe and NMI contexts
printk: get caller_id/timestamp after migration disable
printk: call boot_delay_msec() in printk_delay()
printk: add con_printk() macro for console details
printk: refactor and rework printing logic
printk: move buffer definitions into console_emit_next_record() caller
printk: add pr_flush()
printk: add functions to prefer direct printing
printk: add kthread console printers
printk: extend console_lock for proper kthread support
printk: remove @console_locked

drivers/tty/sysrq.c | 2 +
include/linux/console.h | 19 +
include/linux/printk.h | 82 ++-
kernel/hung_task.c | 11 +-
kernel/panic.c | 4 +
kernel/printk/printk.c | 1197 +++++++++++++++++++++++++++++----------
kernel/rcu/tree_stall.h | 2 +
kernel/reboot.c | 14 +-
kernel/watchdog.c | 4 +
kernel/watchdog_hld.c | 4 +
lib/dump_stack.c | 4 +-
lib/nmi_backtrace.c | 4 +-
12 files changed, 1021 insertions(+), 326 deletions(-)


base-commit: 84d7df104dbab9c3dda8f2c5b46f9a6fc256fe02
--
2.30.2


2022-04-22 21:31:18

by John Ogness

[permalink] [raw]
Subject: [PATCH printk v3 06/15] printk: get caller_id/timestamp after migration disable

Currently the local CPU timestamp and caller_id for the record are
collected while migration is enabled. Since this information is
CPU-specific, it should be collected with migration disabled.

Migration is disabled immediately after collecting this information
anyway, so just move the information collection to after the
migration disabling.

Signed-off-by: John Ogness <[email protected]>
Reviewed-by: Sergey Senozhatsky <[email protected]>
Reviewed-by: Petr Mladek <[email protected]>
---
kernel/printk/printk.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 7bb148a1debb..82ad3d3d0d4a 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -2063,7 +2063,7 @@ static inline void printk_delay(void)
static inline u32 printk_caller_id(void)
{
return in_task() ? task_pid_nr(current) :
- 0x80000000 + raw_smp_processor_id();
+ 0x80000000 + smp_processor_id();
}

/**
@@ -2145,7 +2145,6 @@ int vprintk_store(int facility, int level,
const struct dev_printk_info *dev_info,
const char *fmt, va_list args)
{
- const u32 caller_id = printk_caller_id();
struct prb_reserved_entry e;
enum printk_info_flags flags = 0;
struct printk_record r;
@@ -2155,10 +2154,14 @@ int vprintk_store(int facility, int level,
u8 *recursion_ptr;
u16 reserve_size;
va_list args2;
+ u32 caller_id;
u16 text_len;
int ret = 0;
u64 ts_nsec;

+ if (!printk_enter_irqsave(recursion_ptr, irqflags))
+ return 0;
+
/*
* Since the duration of printk() can vary depending on the message
* and state of the ringbuffer, grab the timestamp now so that it is
@@ -2167,8 +2170,7 @@ int vprintk_store(int facility, int level,
*/
ts_nsec = local_clock();

- if (!printk_enter_irqsave(recursion_ptr, irqflags))
- return 0;
+ caller_id = printk_caller_id();

/*
* The sprintf needs to come first since the syslog prefix might be
--
2.30.2

2022-04-22 22:08:01

by John Ogness

[permalink] [raw]
Subject: [PATCH printk v3 13/15] printk: add kthread console printers

Create a kthread for each console to perform console printing. During
normal operation (@system_state == SYSTEM_RUNNING), the kthread
printers are responsible for all printing on their respective
consoles.

During non-normal operation, console printing is done as it has been:
within the context of the printk caller or within irqwork triggered
by the printk caller, referred to as direct printing.

Since threaded console printers are responsible for all printing
during normal operation, this also includes messages generated via
deferred printk calls. If direct printing is in effect during a
deferred printk call, the queued irqwork will perform the direct
printing. To make it clear that this is the only time that the
irqwork will perform direct printing, rename the flag
PRINTK_PENDING_OUTPUT to PRINTK_PENDING_DIRECT_OUTPUT.

Threaded console printers synchronize against each other and against
console lockers by taking the console lock for each message that is
printed.

Note that the kthread printers do not care about direct printing.
They will always try to print if new records are available. They can
be blocked by direct printing, but will be woken again once direct
printing is finished.

Console unregistration is a bit tricky because the associated
kthread printer cannot be stopped while the console lock is held.
A policy is implemented that states: whichever task clears
con->thread (under the console lock) is responsible for stopping
the kthread. unregister_console() will clear con->thread while
the console lock is held and then stop the kthread after releasing
the console lock.

For consoles that have implemented the exit() callback, the kthread
is stopped before exit() is called.

Signed-off-by: John Ogness <[email protected]>
---
include/linux/console.h | 2 +
kernel/printk/printk.c | 303 +++++++++++++++++++++++++++++++++++++---
2 files changed, 283 insertions(+), 22 deletions(-)

diff --git a/include/linux/console.h b/include/linux/console.h
index 8c1686e2c233..9a251e70c090 100644
--- a/include/linux/console.h
+++ b/include/linux/console.h
@@ -153,6 +153,8 @@ struct console {
uint ospeed;
u64 seq;
unsigned long dropped;
+ struct task_struct *thread;
+
void *data;
struct console *next;
};
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index ed7f738261cc..7e6f1f19abe5 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -361,6 +361,13 @@ static int console_msg_format = MSG_FORMAT_DEFAULT;
/* syslog_lock protects syslog_* variables and write access to clear_seq. */
static DEFINE_MUTEX(syslog_lock);

+/*
+ * A flag to signify if printk_activate_kthreads() has already started the
+ * kthread printers. If true, any later registered consoles must start their
+ * own kthread directly. The flag is write protected by the console_lock.
+ */
+static bool printk_kthreads_available;
+
#ifdef CONFIG_PRINTK
static atomic_t printk_prefer_direct = ATOMIC_INIT(0);

@@ -390,6 +397,14 @@ void printk_prefer_direct_exit(void)
WARN_ON(atomic_dec_if_positive(&printk_prefer_direct) < 0);
}

+static inline bool allow_direct_printing(void)
+{
+ return (!printk_kthreads_available ||
+ system_state > SYSTEM_RUNNING ||
+ oops_in_progress ||
+ atomic_read(&printk_prefer_direct));
+}
+
DECLARE_WAIT_QUEUE_HEAD(log_wait);
/* All 3 protected by @syslog_lock. */
/* the next printk record to read by syslog(READ) or /proc/kmsg */
@@ -2280,10 +2295,10 @@ asmlinkage int vprintk_emit(int facility, int level,
printed_len = vprintk_store(facility, level, dev_info, fmt, args);

/* If called from the scheduler, we can not call up(). */
- if (!in_sched) {
+ if (!in_sched && allow_direct_printing()) {
/*
* The caller may be holding system-critical or
- * timing-sensitive locks. Disable preemption during
+ * timing-sensitive locks. Disable preemption during direct
* printing of all remaining records to all consoles so that
* this context can return as soon as possible. Hopefully
* another printk() caller will take over the printing.
@@ -2326,6 +2341,8 @@ EXPORT_SYMBOL(_printk);

static bool __pr_flush(struct console *con, int timeout_ms, bool reset_on_progress);

+static void printk_start_kthread(struct console *con);
+
#else /* CONFIG_PRINTK */

#define CONSOLE_LOG_MAX 0
@@ -2359,6 +2376,8 @@ static void call_console_driver(struct console *con, const char *text, size_t le
}
static bool suppress_message_printing(int level) { return false; }
static bool __pr_flush(struct console *con, int timeout_ms, bool reset_on_progress) { return true; }
+static void printk_start_kthread(struct console *con) { }
+static bool allow_direct_printing(void) { return true; }

#endif /* CONFIG_PRINTK */

@@ -2559,6 +2578,13 @@ void resume_console(void)
down_console_sem();
console_suspended = 0;
console_unlock();
+
+ /*
+ * While suspended, new records may have been added to the
+ * ringbuffer. Wake up the kthread printers to print them.
+ */
+ wake_up_klogd();
+
pr_flush(1000, true);
}

@@ -2577,6 +2603,9 @@ static int console_cpu_notify(unsigned int cpu)
/* If trylock fails, someone else is doing the printing */
if (console_trylock())
console_unlock();
+
+ /* Wake kthread printers. Some may have become usable. */
+ wake_up_klogd();
}
return 0;
}
@@ -2648,18 +2677,9 @@ static bool abandon_console_lock_in_panic(void)
return atomic_read(&panic_cpu) != raw_smp_processor_id();
}

-/*
- * Check if the given console is currently capable and allowed to print
- * records.
- *
- * Requires the console_lock.
- */
-static inline bool console_is_usable(struct console *con)
+static inline bool __console_is_usable(short flags)
{
- if (!(con->flags & CON_ENABLED))
- return false;
-
- if (!con->write)
+ if (!(flags & CON_ENABLED))
return false;

/*
@@ -2668,12 +2688,26 @@ static inline bool console_is_usable(struct console *con)
* cope (CON_ANYTIME) don't call them until this CPU is officially up.
*/
if (!cpu_online(raw_smp_processor_id()) &&
- !(con->flags & CON_ANYTIME))
+ !(flags & CON_ANYTIME))
return false;

return true;
}

+/*
+ * Check if the given console is currently capable and allowed to print
+ * records.
+ *
+ * Requires the console_lock.
+ */
+static inline bool console_is_usable(struct console *con)
+{
+ if (!con->write)
+ return false;
+
+ return __console_is_usable(con->flags);
+}
+
static void __console_unlock(void)
{
console_locked = 0;
@@ -2786,8 +2820,8 @@ static bool console_emit_next_record(struct console *con, char *text, char *ext_
* were flushed to all usable consoles. A returned false informs the caller
* that everything was not flushed (either there were no usable consoles or
* another context has taken over printing or it is a panic situation and this
- * is not the panic CPU). Regardless the reason, the caller should assume it
- * is not useful to immediately try again.
+ * is not the panic CPU or direct printing is not preferred). Regardless the
+ * reason, the caller should assume it is not useful to immediately try again.
*
* Requires the console_lock.
*/
@@ -2804,6 +2838,10 @@ static bool console_flush_all(bool do_cond_resched, u64 *next_seq, bool *handove
*handover = false;

do {
+ /* Let the kthread printers do the work if they can. */
+ if (!allow_direct_printing())
+ return false;
+
any_progress = false;

for_each_console(con) {
@@ -3018,6 +3056,10 @@ void console_start(struct console *console)
console_lock();
console->flags |= CON_ENABLED;
console_unlock();
+
+ /* Wake the newly enabled kthread printer. */
+ wake_up_klogd();
+
__pr_flush(console, 1000, true);
}
EXPORT_SYMBOL(console_start);
@@ -3218,6 +3260,8 @@ void register_console(struct console *newcon)
nr_ext_console_drivers++;

newcon->dropped = 0;
+ newcon->thread = NULL;
+
if (newcon->flags & CON_PRINTBUFFER) {
/* Get a consistent copy of @syslog_seq. */
mutex_lock(&syslog_lock);
@@ -3227,6 +3271,10 @@ void register_console(struct console *newcon)
/* Begin with next message. */
newcon->seq = prb_next_seq(prb);
}
+
+ if (printk_kthreads_available)
+ printk_start_kthread(newcon);
+
console_unlock();
console_sysfs_notify();

@@ -3253,6 +3301,7 @@ EXPORT_SYMBOL(register_console);

int unregister_console(struct console *console)
{
+ struct task_struct *thd;
struct console *con;
int res;

@@ -3293,7 +3342,20 @@ int unregister_console(struct console *console)
console_drivers->flags |= CON_CONSDEV;

console->flags &= ~CON_ENABLED;
+
+ /*
+ * console->thread can only be cleared under the console lock. But
+ * stopping the thread must be done without the console lock. The
+ * task that clears @thread is the task that stops the kthread.
+ */
+ thd = console->thread;
+ console->thread = NULL;
+
console_unlock();
+
+ if (thd)
+ kthread_stop(thd);
+
console_sysfs_notify();

if (console->exit)
@@ -3389,6 +3451,20 @@ static int __init printk_late_init(void)
}
late_initcall(printk_late_init);

+static int __init printk_activate_kthreads(void)
+{
+ struct console *con;
+
+ console_lock();
+ printk_kthreads_available = true;
+ for_each_console(con)
+ printk_start_kthread(con);
+ console_unlock();
+
+ return 0;
+}
+early_initcall(printk_activate_kthreads);
+
#if defined CONFIG_PRINTK
/* If @con is specified, only wait for that console. Otherwise wait for all. */
static bool __pr_flush(struct console *con, int timeout_ms, bool reset_on_progress)
@@ -3463,11 +3539,180 @@ bool pr_flush(int timeout_ms, bool reset_on_progress)
}
EXPORT_SYMBOL(pr_flush);

+static void __printk_fallback_preferred_direct(void)
+{
+ printk_prefer_direct_enter();
+ pr_err("falling back to preferred direct printing\n");
+ printk_kthreads_available = false;
+}
+
+/*
+ * Enter preferred direct printing, but never exit. Mark console threads as
+ * unavailable. The system is then forever in preferred direct printing and
+ * any printing threads will exit.
+ *
+ * Must *not* be called under console_lock. Use
+ * __printk_fallback_preferred_direct() if already holding console_lock.
+ */
+static void printk_fallback_preferred_direct(void)
+{
+ console_lock();
+ __printk_fallback_preferred_direct();
+ console_unlock();
+}
+
+static bool printer_should_wake(struct console *con, u64 seq)
+{
+ short flags;
+
+ if (kthread_should_stop() || !printk_kthreads_available)
+ return true;
+
+ if (console_suspended)
+ return false;
+
+ /*
+ * This is an unsafe read from con->flags, but a false positive is
+ * not a problem. Worst case it would allow the printer to wake up
+ * although it is disabled. But the printer will notice that when
+ * attempting to print and instead go back to sleep.
+ */
+ flags = data_race(READ_ONCE(con->flags));
+
+ if (!__console_is_usable(flags))
+ return false;
+
+ return prb_read_valid(prb, seq, NULL);
+}
+
+static int printk_kthread_func(void *data)
+{
+ struct console *con = data;
+ char *dropped_text = NULL;
+ char *ext_text = NULL;
+ bool handover;
+ u64 seq = 0;
+ char *text;
+ int error;
+
+ text = kmalloc(CONSOLE_LOG_MAX, GFP_KERNEL);
+ if (!text) {
+ con_printk(KERN_ERR, con, "failed to allocate text buffer\n");
+ printk_fallback_preferred_direct();
+ goto out;
+ }
+
+ if (con->flags & CON_EXTENDED) {
+ ext_text = kmalloc(CONSOLE_EXT_LOG_MAX, GFP_KERNEL);
+ if (!ext_text) {
+ con_printk(KERN_ERR, con, "failed to allocate ext_text buffer\n");
+ printk_fallback_preferred_direct();
+ goto out;
+ }
+ } else {
+ dropped_text = kmalloc(DROPPED_TEXT_MAX, GFP_KERNEL);
+ if (!dropped_text) {
+ con_printk(KERN_ERR, con, "failed to allocate dropped_text buffer\n");
+ printk_fallback_preferred_direct();
+ goto out;
+ }
+ }
+
+ con_printk(KERN_INFO, con, "printing thread started\n");
+
+ for (;;) {
+ /*
+ * Guarantee this task is visible on the waitqueue before
+ * checking the wake condition.
+ *
+ * The full memory barrier within set_current_state() of
+ * prepare_to_wait_event() pairs with the full memory barrier
+ * within wq_has_sleeper().
+ *
+ * This pairs with __wake_up_klogd:A.
+ */
+ error = wait_event_interruptible(log_wait,
+ printer_should_wake(con, seq)); /* LMM(printk_kthread_func:A) */
+
+ if (kthread_should_stop() || !printk_kthreads_available)
+ break;
+
+ if (error)
+ continue;
+
+ console_lock();
+
+ if (console_suspended) {
+ up_console_sem();
+ continue;
+ }
+
+ if (!console_is_usable(con)) {
+ __console_unlock();
+ continue;
+ }
+
+ /*
+ * Even though the printk kthread is always preemptible, it is
+ * still not allowed to call cond_resched() from within
+ * console drivers. The task may become non-preemptible in the
+ * console driver call chain. For example, vt_console_print()
+ * takes a spinlock and then can call into fbcon_redraw(),
+ * which can conditionally invoke cond_resched().
+ */
+ console_may_schedule = 0;
+ console_emit_next_record(con, text, ext_text, dropped_text, &handover);
+ if (handover)
+ continue;
+
+ seq = con->seq;
+
+ __console_unlock();
+ }
+
+ con_printk(KERN_INFO, con, "printing thread stopped\n");
+out:
+ kfree(dropped_text);
+ kfree(ext_text);
+ kfree(text);
+
+ console_lock();
+ /*
+ * If this kthread is being stopped by another task, con->thread will
+ * already be NULL. That is fine. The important thing is that it is
+ * NULL after the kthread exits.
+ */
+ con->thread = NULL;
+ console_unlock();
+
+ return 0;
+}
+
+/* Must be called under console_lock. */
+static void printk_start_kthread(struct console *con)
+{
+ /*
+ * Do not start a kthread if there is no write() callback. The
+ * kthreads assume the write() callback exists.
+ */
+ if (!con->write)
+ return;
+
+ con->thread = kthread_run(printk_kthread_func, con,
+ "pr/%s%d", con->name, con->index);
+ if (IS_ERR(con->thread)) {
+ con->thread = NULL;
+ con_printk(KERN_ERR, con, "unable to start printing thread\n");
+ __printk_fallback_preferred_direct();
+ return;
+ }
+}
+
/*
* Delayed printk version, for scheduler-internal messages:
*/
-#define PRINTK_PENDING_WAKEUP 0x01
-#define PRINTK_PENDING_OUTPUT 0x02
+#define PRINTK_PENDING_WAKEUP 0x01
+#define PRINTK_PENDING_DIRECT_OUTPUT 0x02

static DEFINE_PER_CPU(int, printk_pending);

@@ -3475,10 +3720,14 @@ static void wake_up_klogd_work_func(struct irq_work *irq_work)
{
int pending = this_cpu_xchg(printk_pending, 0);

- if (pending & PRINTK_PENDING_OUTPUT) {
+ if (pending & PRINTK_PENDING_DIRECT_OUTPUT) {
+ printk_prefer_direct_enter();
+
/* If trylock fails, someone else is doing the printing */
if (console_trylock())
console_unlock();
+
+ printk_prefer_direct_exit();
}

if (pending & PRINTK_PENDING_WAKEUP)
@@ -3503,10 +3752,11 @@ static void __wake_up_klogd(int val)
* prepare_to_wait_event(), which is called after ___wait_event() adds
* the waiter but before it has checked the wait condition.
*
- * This pairs with devkmsg_read:A and syslog_print:A.
+ * This pairs with devkmsg_read:A, syslog_print:A, and
+ * printk_kthread_func:A.
*/
if (wq_has_sleeper(&log_wait) || /* LMM(__wake_up_klogd:A) */
- (val & PRINTK_PENDING_OUTPUT)) {
+ (val & PRINTK_PENDING_DIRECT_OUTPUT)) {
this_cpu_or(printk_pending, val);
irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
}
@@ -3524,7 +3774,16 @@ void defer_console_output(void)
* New messages may have been added directly to the ringbuffer
* using vprintk_store(), so wake any waiters as well.
*/
- __wake_up_klogd(PRINTK_PENDING_WAKEUP | PRINTK_PENDING_OUTPUT);
+ int val = PRINTK_PENDING_WAKEUP;
+
+ /*
+ * If console deferring was called with preferred direct printing,
+ * make the irqwork perform the direct printing.
+ */
+ if (atomic_read(&printk_prefer_direct))
+ val |= PRINTK_PENDING_DIRECT_OUTPUT;
+
+ __wake_up_klogd(val);
}

void printk_trigger_flush(void)
--
2.30.2