LinuxLists.cc - [RFC/PATCH] Documentation of kernel messages

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

Hi Dave,

On Wed, 2007-06-13 at 09:37 -0700, Dave Hansen wrote:

[snip]

> I'm not sure we want to make Linux more like z/* in this regard. :)
>
> The problem with your proposal is that every time a new message in the
> kernel is created or modified, you need somebody to go update that
> documentation.

You only have to document printks, which are using the KMSG macros.
Since there are tons of self-explaining printks, most of them do not
have to use them.

If you change the meaning of a KMSG printk and you don't update the
documentation, it is the same thing like changing a kernel API function
and forgetting to update the kernel-doc comment.

> It's going to get out-of-sync very fast if this isn't
> done, and you either need to convince and teach each and every kernel
> contributor to follow your lead, or have a team of highly trained code
> monkeys to watch git-commits and resubmit documentation for every diff
> that touches a printk.

It is a matter of discipline. If a device driver maintainer decides to
document some messages using the KMSG macros, he has to take care that
the documentation is up-to-date. Since the printks and the descriptions
are maintained together in the kernel code, I think, that's doable.

And the checker tool can help to keep the descriptions up-to-date, since
it produces warnings, if you use C=1 during the kernel build.

> I think it's a great idea to go and update some of these obtuse kernel
> messages with more understandable wording. It might even be nice to
> update the Documentation/ about what kinds of messages are good or bad
> to have. But, I just don't think this is viable to convince everybody
> to do this.

Again, not everybody has to use KMSG printks.

> Do you have a list of printks that are causing your
> customers trouble?

There are quite a lot of messages in our s390 device drivers which
should be documented. Especially for character device drivers (e.g.
tape), where the user has direct interaction with the device.

One Example from the tape device driver: "Another host has reserved the
tape device"

We should document, how to find out the host, which reserved the tape
and what has to be done on the other host in order to release the tape
etc.

> Does every printk need one of these? What do we do with the existing
> printks?

Again: Most printks don't need documentation. We can leave them as they
are.

> Who will convert them over? How do we enforce the rules if
> new printks must have documentation written for them? Who writes the
> documentation if the printk author does not?

The maintainer of the component (e.g. device driver) is responsible for
the KMSG printks. If he accepts a patch, which modifies KMSG printks, he
has to take care, that the patch also updates the KMSG_DOC part.

> If we have a catalog of kernel messages, it obviously needs to be
> versioned, but does that mean that we need to keep it in the kernel
> tree?

For each kernel a separate message catalog can be automatically created.
This message catalog fits exactly to one kernel and should be installed
together with the kernel.

Michael

2007-06-13 17:49:57

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Wed, 2007-06-13 at 21:16 +0400, Alexey Dobriyan wrote:
> On Wed, Jun 13, 2007 at 05:06:57PM +0200, holzheu wrote:
> > -CHECK = sparse
> > -
> > -CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ -Wbitwise $(CF)
> > +CHECK = scripts/kmsg-doc check
> > +CHECKFLAGS =
>
> Ick. Don't touch those checking kernel with sparse.

Of course we don't want to have that in the vanilla kernel. I just
wanted to show, how the checker tool has to be used.

Michael

2007-06-13 17:50:22

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Wed, Jun 13, 2007 at 05:06:57PM +0200, holzheu wrote:
> Current prototype implementation:
> =================================
>
> The structure of a kernel message is: <component>.<msg number>: <msg>
>
> * component: Name of the kernel or driver component e.g. "pci", "ide",
> etc.
> * msg number: Within the component unique number of a kernel message.
> * msg: printk message
>
> New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be
> used in printk. These macros have as parameter the message number and
> are using a per c-file defined macro KMSG_COMPONENT.
>
> Example: Define message 2 in component "kmsgtest":
>
> #define KMSG_COMPONENT "kmsgtest"
>
> void f(void)
> {
> printk(KMSG_ERR(1) "device %x not online\n", devno);
> }

Ick, why are you ignoring what we have already with dev_printk() and
friends? We are just finally getting developers to use that, I think it
will be almost impossible to get people to change to something else,
especially one that isn't even as "correct" as what dev_printk() offers
you today, will be quite hard.

So, why not use what we already have and work off of it?

thanks,

greg k-h

2007-06-13 18:09:50

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Wednesday 13 June 2007 12:50:55 [email protected] wrote:
> > In general we think, that also for Linux it is a good thing to have
> > documentation for the most important kernel/driver messages. Even
> > kernel hackers not always are aware of the meaning of kernel messages
> > for components, which they don't know in detail. Most of the messages
> > are self explaining but sometimes you get something like "Clocksource
> > tsc unstable (delta = 7304132729 ns)" and you wonder if your system is
> > going to explode.

Isn't this what the severity levels are for? Not just filtering messages out
but telling you whether or not it's important?

> This is probably best addressed by cleaning up the actual messages so
> they're a bit more informative.

Seconded.

The point of a diagnostic message is to convey information. Deep in the
bowels of chipsets there are diagnostic messages that get spit out as
hexcodes that some program needs to interpret, but that's why we have tools
like http://kernel.org/pub/linux/kernel/people/davej/tools/parsemce.c and
such.

If the diagnostic messages need a talmud, I see this is as more of a problem
with said diagnostic messages, rather than excuse to add another layer.

> > New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be
> > used in printk. These macros have as parameter the message number and
> > are using a per c-file defined macro KMSG_COMPONENT.
>
> Gaak. *NO*.

If you're going to tweak printk, please make the darn thing take an integer
first argument rather than a string first argument, so we can use clever
macros to remove them at compile time rather than having them compiled in but
not displaying.

This is one of those things buried down on my todo list. I think it can be
converted incrementally

Going back to the original patch here, there are exactly two examples of what
this brave new infrastructure would be used for:

> +/**
> + * message
> + * @0: Device number of device
> + * @1: Status of device
> + *
> + * Description:
> + * Information message about the status of our virtual msgtest device. The
> + * following values for the status parameter are available.
> + *
> + * 0 - Device is offline
> + *
> + * 1 - Device is online
> + *
> + * 2 - Device is broken
> + *
> + * User Response:
> + * If device is broken, replace it or fix it.
> + */
> +
> +KMSG_DOC(kmsgtest, 1, "device %x has status %i");

This is just a thought: why not have the actual kprintf say "Device number %x
is %s" where %s is "online", "offline", or "broken"?

I.E. have the kernel output a user readable message in the first place.

> +/**
> + * message
> + * @0: Device number of device.
> + *
> + * Description:
> + * An operation has been performed on the msgtest device, but the device
> has + * not been set online. Therefore the operation failed.
> + *
> + * User Response:
> + * Operator should set device online. Issue "chccwdev -e <device number>".
> + */
> +
> +KMSG_DOC(kmsgtest, 2, "device %x not online");

kprintf(LEVEL "device %x not online, try 'chccwdev -e %x'", devnum, devnum)

Rob

2007-06-13 18:14:58

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

Hi Valdis,

On Wed, 2007-06-13 at 12:50 -0400, [email protected] wrote:
> On Wed, 13 Jun 2007 17:06:57 +0200, holzheu said:
> > They are used to that, because all other operating systems on that
> > platform like z/OS, z/VM or z/VSE have message catalogs with detailed
> > descriptions about the semantics of the messages.
>
> 25 years ago, I did OS/MVT and OS/VS1 for a living, so I know *all* about
> the infamous "What does IEF507E mean again?"...

:-)

> > In general we think, that also for Linux it is a good thing to have
> > documentation for the most important kernel/driver messages. Even
> > kernel hackers not always are aware of the meaning of kernel messages
> > for components, which they don't know in detail. Most of the messages
> > are self explaining but sometimes you get something like "Clocksource
> > tsc unstable (delta = 7304132729 ns)" and you wonder if your system is
> > going to explode.
>
> This is probably best addressed by cleaning up the actual messages so they're
> a bit more informative.

Of course that would be good, too.

But I think, that we sometimes have the dilemma, that we want to keep
the printks short, but also want to provide as much information as
possible.

If the information is to big for the printk itself, because you would
need 10 lines to explain what happened, wouldn't it be good to have a
place where to put that information?

> > New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be
> > used in printk. These macros have as parameter the message number and
> > are using a per c-file defined macro KMSG_COMPONENT.
>
> Gaak. *NO*.
>
> The *only* reason that the MVS and VM message catalogs worked at all is
> because each component had a message repository that went across *all* the
> source files - the instant you saw IEFnnns, you knew that IEF covered the
> job scheduler, nnn was a *unique* number, and s was a Severe/Warning/Info
> flag. IGG was always data management, and so on. This breaks horribly if
> you have 2 C files that define subtly different KMSG_COMPONENT values (or
> even worse, 2 or more duplicates).
>
> [/usr/src/linux-2.6.22-rc4-mm2] find . -name '*.c' | wc -l
> 9959
> [/usr/src/linux-2.6.22-rc4-mm2] find . -name '*.h' | wc -l
> 9933
> [/usr/src/linux-2.6.22-rc4-mm2] find . -type d | wc -l
> 1736
>
> You plan to maintain message uniqueness how?
> [/usr/src/linux-2.6.22-rc4-mm2]1 find . -name '*.c' | sed -r 's?.*/([^/]*)?\1?' | sort | uniq -c | sort -nr | head
> 105 setup.c
> 90 irq.c
> 66 time.c
> 58 init.c
> 50 inode.c
> 39 io.c
> 38 pci.c
> 37 file.c
> 32 signal.c
> 32 ptrace.c
>
> Looks like you're going to have to embed a lot of the path in that KMSG_COMPONENT
> to make it unique - and you want to keep that message under 80 or so chars total.
>

For each kernel component, like a device driver, we could have one
KMSG_COMPONENT (e.g. "acpi", "pci", etc). Within that component the
message ids have to be unique. A tool could check, if messages are
unique within the kernel sources.

We could use something like a Documentation/kmsg-components file with a
list of all component names using KMSG printks.

> > /**
> > * message
> > * @0: device number of device.
> > *
> > * Description:
> > * An operation has been performed on the msgtest device, but the
> > * device has not been set online. Therefore the operation failed
>
> If you don't understand 'Device /dev/foo offline', this description
> doesn't help any. And that's true for *most* of the kernel messages
> already - if you don't understand the message already, a paragraph
> explanation isn't going to help much. Consider the average OOPS
> message, which contains stuff like 'EIP=0x..'. Telling the user that
> EIP means Execution Instruction Pointer isn't likely to help - if they
> knew what the pointer *did*, they'd probably already know EIP.

I agree with you, that most of the kernel messages do not need further
documentation. But I am convinced, that there are plenty of printks,
where additional documentation would be helpful.

> > *
> > * User Response:
> > * Operator should set device online.
> > * Issue "chccwdev -e <device number>".
>
> And this is where the weakness of this scheme *really* hits. I've actually run
> into cases where an operator followed the listed "Operator Response" for a
> "device offline", and issued a 'VARY 0C0,ONLINE'. And then we got a flood of
> I/O errors because the previous shift downed the device because it was having
> issues. The response the operator *should* have done is "assign a different
> tape drive, like, oh maybe the operational ones at 0C1 through 0C4"...

I can understand your frustration here. But that's a general problem
with documentation. You never can foresee everything.

But should this mean, that we shouldn't document anything?

Michael

2007-06-13 18:16:31

by Andrew Morton

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

> On Wed, 13 Jun 2007 17:06:57 +0200 holzheu <[email protected]> wrote:
> Greetings,
>
> The operation of a Linux system sometimes requires to decode the
> meaning of a specific kernel message, e.g. an error message of a
> driver. Especially on our mainframe zSeries platform system
> administrators want to have descriptions for Linux kernel messages.
> They are used to that, because all other operating systems on that
> platform like z/OS, z/VM or z/VSE have message catalogs with detailed
> descriptions about the semantics of the messages.
>
> In general we think, that also for Linux it is a good thing to have
> documentation for the most important kernel/driver messages. Even
> kernel hackers not always are aware of the meaning of kernel messages
> for components, which they don't know in detail. Most of the messages
> are self explaining but sometimes you get something like "Clocksource
> tsc unstable (delta = 7304132729 ns)" and you wonder if your system is
> going to explode.
>
> Unfortunately currently there is no general infrastructure in the Linux
> kernel for the documentation of messages. I worked on a proposal, how
> that could be implemented in an easy way using the already existing
> kernel-doc infrastructure and using printk. The proposal is as follows
>
> 1. We use message identifiers in order to map messages to message
> descriptions. A message identifier consists out of a component name and
> within that component unique message number.
>
> 2. Messages and message descriptions are maintained together in the
> kernel sources in order to keep them up to date. Messages catalog are
> generated automatically for exactly one kernel level.
>
> 3. A special tool checks, if messages are not documented or if there
> are message descriptions without corresponding messages.
>
> 4. We use the already existing kernel-doc tool to generate an online
> message catalog or e.g. a pdf for offline documentation.
>
> Current prototype implementation:
> =================================
>
> The structure of a kernel message is: <component>.<msg number>: <msg>
>
> * component: Name of the kernel or driver component e.g. "pci", "ide",
> etc.
> * msg number: Within the component unique number of a kernel message.
> * msg: printk message
>
> New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be
> used in printk. These macros have as parameter the message number and
> are using a per c-file defined macro KMSG_COMPONENT.
>
> Example: Define message 2 in component "kmsgtest":
>
> #define KMSG_COMPONENT "kmsgtest"
>
> void f(void)
> {
> printk(KMSG_ERR(1) "device %x not online\n", devno);
> }
>
> The output of that kernel message would be:
> "kmsgtest.1: device 4711 not online"
>
> The messages have to be documented within the C source file
> in the following way:
>
> /**
> * message
> * @0: device number of device.
> *
> * Description:
> * An operation has been performed on the msgtest device, but the
> * device has not been set online. Therefore the operation failed
> *
> * User Response:
> * Operator should set device online.
> * Issue "chccwdev -e <device number>".
> *
> */
>
> KMSG_DOC(kmsgtest, 2, "Device %x not online");
>
> I created a patch for the kernel-doc tool so it can be used to generate
> a catalog of all kernel messages:
>
> >> kernel-doc -man kmsgtest.c > kmsgtest.2
> >> man ./kmsgtest.2
>
> # Kernel API(9) Linux Messages Kernel API(9)
> #
> # MESSAGE:
> # kmsgtest.2: "device %x not online"
> #
> # Parameter:
> # 1 Device number of device.
> #
> # Description:
> # An operation has been performed on the msgtest device, but
> # the device has not been set online. Therefore the operation failed.
> #
> # User Response:
> # Operator should set device online.
> # Issue "chccwdev -e <device number>".
> #
> # May 2007 kmsgtest.2 Kernel API(9)
>
> A nice thing would be to include the online kernel message catalog in
> the kernel rpm. One possibility for that would be to have one man page
> per message. If an operator finds the message kmsgtest.2 in
> var/log/messages and wants to know what the message means, he simply
> issues "man kmsgtest.2" and gets the description.
>
> To ensure that all messages are documented and there are no message
> descriptions without corresponding messages, a checker tool is
> provided. To enable message checking, in the toplevel Makefile the
> following has to be added:
>
> CHECK = scripts/kmsg_check.pl check
>
> To enable message checking during kernel build, the "C" option has
> to be used:
>
> >> make modules C=1
> CHK include/linux/version.h
> CHK include/linux/utsrelease.h
> CHECK drivers/kmsgtest/kmsgtest.c
> drivers/kmsgtest/kmsgtest.c: Missing description for: kmsgtest.1
> drivers/kmsgtest/kmsgtest.c: Description without message for: kmsgtest.3
>
> Please note, that the patch for the kernel-doc and kmsg-doc tools
> is just a prototype and is neither complete nor perfect.
>
> Michael
>
> Acked-by: Martin Schwidefsky <[email protected]>
> Acked-by: Heiko Carstens <[email protected]>
> Signed-off-by: Michael Holzheu <[email protected]>

Your proposal is similar to one I made to some Japanese developers
earlier this year. I was more modest, proposing that we

- add an enhanced printk

xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

- An externally managed database will provide translations, diagnostics,
remedial actions, etc, keyed off the msgid string

- Format and management of the msgid hasn't been clearly identified yet
as far as I know. But all we really need is that each ID be unique,
kernel-wide. We develop a simple tool which can check the whole tree
for duplicated msgids.

- From a practical day-to-day POV what this means is that a person
from the kernel-messages team will prepare a patch for your driver
which adds the msgids and you the driver author should take care not
to break the tagging.

This is deliberately designed to be minimum-impact upon general
developers. We want a system in place where driver/fs/etc developers
can concentrate on what they do, and kernel-message developers can
as independently as possibe take care of what _they_ do.

- A project has started up and there is a mailing list (cc'ed here) and
meetings are happening at the LF collaboration summit this week.

Please see https://lists.linux-foundation.org/mailman/listinfo/lf_kernel_messages
and don't miss the next conference call ;)

(argh. The lf_kernel_messages archives are subscriber-only and it could be that
only members can post to it. Oh well. Please subscribe, and review the archives)

>
> Makefile | 5
> drivers/Makefile | 1
> drivers/kmsgtest/Makefile | 5
> drivers/kmsgtest/kmsgtest.c | 59 +++++++++++
> include/linux/kmsg.h | 32 ++++++
> scripts/kernel-doc | 116 +++++++++++++++++++++-
> scripts/kmsg-doc | 231 ++++++++++++++++++++++++++++++++++++++++++++
> 7 files changed, 445 insertions(+), 4 deletions(-)
>
> diff -Naur linux-2.6.21/Makefile linux-2.6.21-kmsg/Makefile
> --- linux-2.6.21/Makefile 2007-04-26 05:08:32.000000000 +0200
> +++ linux-2.6.21-kmsg/Makefile 2007-06-05 15:17:51.000000000 +0200
> @@ -293,9 +293,8 @@
> DEPMOD = /sbin/depmod
> KALLSYMS = scripts/kallsyms
> PERL = perl
> -CHECK = sparse
> -
> -CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ -Wbitwise $(CF)
> +CHECK = scripts/kmsg-doc check
> +CHECKFLAGS =
> MODFLAGS = -DMODULE
> CFLAGS_MODULE = $(MODFLAGS)
> AFLAGS_MODULE = $(MODFLAGS)
> diff -Naur linux-2.6.21/drivers/Makefile linux-2.6.21-kmsg/drivers/Makefile
> --- linux-2.6.21/drivers/Makefile 2007-04-26 05:08:32.000000000 +0200
> +++ linux-2.6.21-kmsg/drivers/Makefile 2007-06-05 15:17:51.000000000 +0200
> @@ -8,6 +8,7 @@
> obj-$(CONFIG_PCI) += pci/
> obj-$(CONFIG_PARISC) += parisc/
> obj-$(CONFIG_RAPIDIO) += rapidio/
> +obj-m += kmsgtest/
> obj-y += video/
> obj-$(CONFIG_ACPI) += acpi/
> # PnP must come after ACPI since it will eventually need to check if acpi
> diff -Naur linux-2.6.21/drivers/kmsgtest/Makefile linux-2.6.21-kmsg/drivers/kmsgtest/Makefile
> --- linux-2.6.21/drivers/kmsgtest/Makefile 1970-01-01 01:00:00.000000000 +0100
> +++ linux-2.6.21-kmsg/drivers/kmsgtest/Makefile 2007-06-05 15:17:51.000000000 +0200
> @@ -0,0 +1,5 @@
> +#
> +# Makefile for kernel message test module
> +#
> +
> +obj-m += kmsgtest.o
> diff -Naur linux-2.6.21/drivers/kmsgtest/kmsgtest.c linux-2.6.21-kmsg/drivers/kmsgtest/kmsgtest.c
> --- linux-2.6.21/drivers/kmsgtest/kmsgtest.c 1970-01-01 01:00:00.000000000 +0100
> +++ linux-2.6.21-kmsg/drivers/kmsgtest/kmsgtest.c 2007-06-05 15:17:51.000000000 +0200
> @@ -0,0 +1,59 @@
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/kmsg.h>
> +
> +static int devno = 0x4711;
> +static int status = 1;
> +
> +#define KMSG_COMPONENT "kmsgtest"
> +
> +static int __init kmsgtest_init(void)
> +{
> + printk(KMSG_INFO(1) "device %x has status %i\n", devno, status);
> + printk(KMSG_ERR(2) "device %x not online\n", devno);
> +
> + return 0;
> +}
> +
> +static void __exit kmsgtest_exit(void)
> +{
> + printk("kmsgtest module exit\n");
> +}
> +
> +module_init(kmsgtest_init);
> +module_exit(kmsgtest_exit);
> +
> +/**
> + * message
> + * @0: Device number of device
> + * @1: Status of device
> + *
> + * Description:
> + * Information message about the status of our virtual msgtest device. The
> + * following values for the status parameter are available.
> + *
> + * 0 - Device is offline
> + *
> + * 1 - Device is online
> + *
> + * 2 - Device is broken
> + *
> + * User Response:
> + * If device is broken, replace it or fix it.
> + */
> +
> +KMSG_DOC(kmsgtest, 1, "device %x has status %i");
> +
> +/**
> + * message
> + * @0: Device number of device.
> + *
> + * Description:
> + * An operation has been performed on the msgtest device, but the device has
> + * not been set online. Therefore the operation failed.
> + *
> + * User Response:
> + * Operator should set device online. Issue "chccwdev -e <device number>".
> + */
> +
> +KMSG_DOC(kmsgtest, 2, "device %x not online");
> diff -Naur linux-2.6.21/include/linux/kmsg.h linux-2.6.21-kmsg/include/linux/kmsg.h
> --- linux-2.6.21/include/linux/kmsg.h 1970-01-01 01:00:00.000000000 +0100
> +++ linux-2.6.21-kmsg/include/linux/kmsg.h 2007-06-05 15:17:51.000000000 +0200
> @@ -0,0 +1,32 @@
> +#ifndef _LINUX_KMSG_H
> +#define _LINUX_KMSG_H
> +
> +#ifdef __KMSG_CHECKER
> +
> +#define KMSG_EMERG(num) __KMSG_CHECK(EMERG, num)
> +#define KMSG_ALERT(num) __KMSG_CHECK(ALERT, num)
> +#define KMSG_CRIT(num) __KMSG_CHECK(CRIT, num)
> +#define KMSG_ERR(num) __KMSG_CHECK(ERR, num)
> +#define KMSG_WARNING(num) __KMSG_CHECK(WARNING, num)
> +#define KMSG_NOTICE(num) __KMSG_CHECK(NOTICE, num)
> +#define KMSG_INFO(num) __KMSG_CHECK(INFO, num)
> +#define KMSG_DEBUG(num) __KMSG_CHECK(DEBUG, num)
> +
> +#define KMSG_DOC(comp, num, str) __KMSG_DOC(comp, num, str)
> +
> +#else
> +
> +#define KMSG_EMERG(num) KERN_EMERG KMSG_COMPONENT "." #num ": "
> +#define KMSG_ALERT(num) KERN_ALERT KMSG_COMPONENT "." #num ": "
> +#define KMSG_CRIT(num) KERN_CRIT KMSG_COMPONENT "." #num ": "
> +#define KMSG_ERR(num) KERN_ERR KMSG_COMPONENT "." #num ": "
> +#define KMSG_WARNING(num) KERN_WARNING KMSG_COMPONENT "." #num ": "
> +#define KMSG_NOTICE(num) KERN_NOTICE KMSG_COMPONENT "." #num ": "
> +#define KMSG_INFO(num) KERN_INFO KMSG_COMPONENT "." #num ": "
> +#define KMSG_DEBUG(num) KERN_DEBUG KMSG_COMPONENT "." #num ": "
> +
> +#define KMSG_DOC(comp, num, str)
> +
> +#endif /* __KMSG_CHECKER */
> +
> +#endif /* _LINUX_KMSG_H */
> diff -Naur linux-2.6.21/scripts/kernel-doc linux-2.6.21-kmsg/scripts/kernel-doc
> --- linux-2.6.21/scripts/kernel-doc 2007-04-26 05:08:32.000000000 +0200
> +++ linux-2.6.21-kmsg/scripts/kernel-doc 2007-06-05 15:17:51.000000000 +0200
> @@ -256,7 +256,7 @@
> my $in_doc_sect;
>
> #declaration types: can be
> -# 'function', 'struct', 'union', 'enum', 'typedef'
> +# 'function', 'struct', 'union', 'enum', 'typedef', 'message'
> my $decl_type;
>
> my $doc_special = "\@\%\$\&";
> @@ -1163,6 +1163,55 @@
> output_section_text(@_);
> }
>
> +##
> +# output message in text
> +sub output_message_text(%) {
> + my %args = %{$_[0]};
> + my ($parameter);
> +
> + print $args{'id'}.": \"".$args{'message'}."\"\n\n";
> +
> + print "Parameters:\n\n";
> + foreach $parameter (@{$args{'parameterlist'}}) {
> + ($parameter =~ /^#/) && next;
> + my $parameter_name = $parameter;
> + $parameter_name =~ s/\[.*//;
> + ($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
> + print "$parameter\n\t";
> + print $args{'parameterdescs'}{$parameter_name}."\n";
> + }
> + output_section_text(@_);
> +}
> +
> +##
> +# output message in man
> +sub output_message_man(%) {
> + my %args = %{$_[0]};
> + my ($parameter, $section);
> +
> + print ".TH \"$args{'module'}\" 9 \"".$args{'id'}."\" \"$man_date\" \"Linux Messages\" LINUX\n";
> +
> + print ".SH MESSAGE\n";
> + print $args{'id'}.": "."\"".$args{'message'}."\"\n";
> +
> + print ".SH Parameters\n";
> + foreach $parameter (@{$args{'parameterlist'}}) {
> + ($parameter =~ /^#/) && next;
> +
> + my $parameter_name = $parameter;
> + $parameter_name =~ s/\[.*//;
> +
> + ($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
> + print ".IP \"".$parameter."\" 12\n";
> + output_highlight($args{'parameterdescs'}{$parameter_name});
> + }
> + foreach $section (@{$args{'sectionlist'}}) {
> + print ".SH \"$section\"\n";
> + output_highlight($args{'sections'}{$section});
> + print "\n";
> + }
> +}
> +
> #output sections in text
> sub output_section_text(%) {
> my %args = %{$_[0]};
> @@ -1607,6 +1656,69 @@
> });
> }
>
> +sub create_parameterlist_msg($$) {
> + my $args = shift;
> + my $file = shift;
> + my $splitter = "%";
> + my $type;
> + my $param = 0;
> + my $first = 1;
> +
> + # temporarily replace commas
> + while ($args =~ /($[^$,]+),/) {
> + $args =~ s/($[^$,]+),/$1#/g;
> + }
> +
> + foreach my $arg (split($splitter, $args)) {
> + if ($first) {
> + $first = 0;
> + next;
> + }
> + $type = substr($arg,0,1);
> +
> + # XXX introduce better type checking
> +
> + push_parameter($param, $type, $file);
> + $param += 1;
> + }
> +}
> +
> +##
> +# takes a message prototype and the name of the current file being
> +# processed and spits out all the details stored in the global
> +# arrays/hashes.
> +sub dump_message($$) {
> + my $x = shift;
> + my $file = shift;
> +
> + if ($x =~/(KMSG_DOC)$\s*(\w+)\s*\,\s*(\w+)\s*\,\s*\"(.*)\"$/) {
> + $declaration_name = "$2.$3";
> + my $members = $4;
> + # strip comments:
> + $members =~ s/\/\*.*?\*\///gos;
> +
> + create_parameterlist_msg($members, $file);
> +
> + output_declaration($declaration_name,
> + 'message',
> + {'message' => $members,
> + 'id' => $declaration_name,
> + 'module' => $modulename,
> + 'parameterlist' => \@parameterlist,
> + 'parameterdescs' => \%parameterdescs,
> + 'parametertypes' => \%parametertypes,
> + 'sectionlist' => \@sectionlist,
> + 'sections' => \%sections,
> + 'purpose' => $declaration_purpose,
> + 'type' => $decl_type
> + });
> + }
> + else {
> + print STDERR "Error(${file}:$.): Cannot parse message!\n";
> + ++$errors;
> + }
> +}
> +
> sub process_file($);
>
> # Read the file that maps relative names to absolute names for
> @@ -1782,6 +1894,8 @@
> $decl_type = 'enum';
> } elsif ($identifier =~ m/^typedef/) {
> $decl_type = 'typedef';
> + } elsif ($identifier =~ m/^message/) {
> + $decl_type = 'message';
> } else {
> $decl_type = 'function';
> }
> diff -Naur linux-2.6.21/scripts/kmsg-doc linux-2.6.21-kmsg/scripts/kmsg-doc
> --- linux-2.6.21/scripts/kmsg-doc 1970-01-01 01:00:00.000000000 +0100
> +++ linux-2.6.21-kmsg/scripts/kmsg-doc 2007-06-05 15:17:51.000000000 +0200
> @@ -0,0 +1,231 @@
> +#!/usr/bin/perl
> +#
> +# Tool to check kernel messages
> +#
> +# Can be used in toplevel Linux Makefile in the following way:
> +#
> +# CHECK = scripts/kmsg-doc check
> +# CHECKFLAGS =
> +#
> +# Note: This is just a prototype and neither perfect nor complete!
> +#
> +# Copyright (C) IBM Corp. 2007
> +# Author(s): Michael Holzheu <[email protected]>
> +#
> +
> +sub create_message($$$$$)
> +{
> + my ($sev, $component, $number, $text, $params) = @_;
> +
> + $text =~ s/\\n//; # remove trailing newline character
> + $message_id = "$component.$number";
> + $messages{$message_count}->{'ID'} = $message_id;
> + $messages{$message_count}->{'COMP'} = $component;
> + $messages{$message_count}->{'NR'} = $number;
> + $messages{$message_count}->{'MSG'} = $text;
> + $messages{$message_count}->{'SEV'} = $sev;
> +
> + @parms = split(/[\s]*,[\s]*/,$params);
> + $parm_count = 0;
> + foreach $parm (@parms) {
> + if (!($parm eq "")) {
> + $messages{$message_count}->{'PARM_NAME'}->{$parm_count} = $parm;
> + $parm_count += 1;
> + }
> + }
> + $messages{$message_count}->{'PARM_COUNT'} = $parm_count;
> + $message_count += 1;
> +}
> +
> +sub get_msgs($)
> +{
> + my ($filename)=@_;
> +
> + $message_count = 0;
> + open(FD, $filename);
> + my @lines=<FD>;
> + foreach $line (@lines) {
> + if ($line =~ /\s*printk$[\s]*__KMSG_CHECK\((.*)\,[\s]*(.*)$[\s]*"(.*)"[\s]*(.*)[\s]*\);/) {
> + create_message($1, $component, $2, $3, $4);
> + }
> + }
> +}
> +
> +sub get_descriptions($)
> +{
> + my ($filename)=@_;
> + my $desc_start;
> +
> + $description_count = 0;
> + $desc_start = 0;
> + open(FD, $filename);
> + my @lines=<FD>;
> + foreach $line (@lines) {
> + if ($line =~ /#define [\s]*KMSG_COMPONENT [\s]*"(.*)"/) {
> + $component = $1;
> + }
> + if ($line =~ /\s*\/\*\*$/) {
> + $msg_start = 1;
> + $parm_count = 0;
> + next;
> + }
> + if (($msg_start == 1) && ($line =~ / \* message/)) {
> + $desc_start = 1;
> + next;
> + }
> + if ($line =~ / \*\//) {
> + $desc_start = 0;
> + next;
> + }
> + if ($desc_start == 1) {
> + $descriptions{$description_count}->{'DESC'} .= "$line";
> + next;
> + }
> + if ($line =~
> + /\s*KMSG_DOC$\s*(.*)\s*\,\s*(.*)\s*\,\s*\"(.*)\"\s*$;/) {
> + my $param_count = 0;
> + my $first = 1;
> + my $type;
> +
> + $descriptions{$description_count}->{'ID'} = "$1\.$2";
> + $descriptions{$description_count}->{'COMP'} = "$1";
> + $descriptions{$description_count}->{'NR'} = "$2";
> + $descriptions{$description_count}->{'MSG'} = "$3";
> + foreach my $arg (split("%", $3)) {
> + if ($first) {
> + $first = 0;
> + next;
> + }
> + $type = substr($arg, 0, 1);
> + $descriptions{$description_count}->{'PARM_TYPE'}->{$param_count} = $type;
> + $param_count += 1;
> + }
> + $descriptions{$description_count}->{'PARM_COUNT'} = $param_count;
> + $description_count += 1;
> + }
> + }
> +}
> +
> +sub print_messages()
> +{
> + for ($i = 0; $i < $message_count; $i++) {
> + print "MESSAGE: $messages{$i}->{'ID'}\n";
> + }
> +}
> +
> +sub print_descriptions($)
> +{
> + my ($message_id)=@_;
> +
> + for ($i = 0; $i < $description_count; $i++) {
> + if (($descriptions{$i}->{'ID'} eq $message_id) || $message_id eq "all") {
> + print "==============================================================================\n";
> + print "\[$descriptions{$i}->{'COMP'}\.$descriptions{$i}->{'NR'}\] $descriptions{$i}->{'MSG'}\n";
> + print "\n";
> + print "Parameters:\n";
> + for ($j = 0; $j < $descriptions{$i}->{'PARM_COUNT'}; $j++) {
> + print " $descriptions{$i}->{'PARM_TYPE'}->{$j}: $descriptions{$i}->{'PARM_DESC'}->{$j}\n";
> + }
> + print "\n";
> + print "Description:\n";
> + print "$descriptions{$i}->{'DESC'}\n";
> + print "==============================================================================\n";
> + }
> + }
> +}
> +
> +sub check_messages($)
> +{
> + my ($filename)=@_;
> +
> + for ($i = 0; $i < $message_count; $i++) {
> + $found = 0;
> + for ($j = 0; $j < $description_count; $j++) {
> + if ($messages{$i}->{'ID'} eq $descriptions{$j}->{'ID'}) {
> + $found = 1;
> + last;
> + }
> + }
> + if (!$found) {
> + print STDERR "$filename: Missing description for: $messages{$i}->{'ID'}\n";
> + }
> + }
> + for ($i = 0; $i < $description_count; $i++) {
> + $found = 0;
> + for ($j = 0; $j < $message_count; $j++) {
> + if ($messages{$j}->{'ID'} eq $descriptions{$i}->{'ID'}) {
> + $found = 1;
> + last;
> + }
> + }
> + if (!$found) {
> + print STDERR "$filename: Description without message for: $descriptions{$i}->{'ID'}\n";
> + }
> + }
> +}
> +
> +sub print_templates()
> +{
> +
> + for ($i = 0; $i < $message_count; $i++) {
> + $found = 0;
> + for ($j = 0; $j < $description_count; $j++) {
> + if ($messages{$i}->{'ID'} eq $descriptions{$j}->{'ID'}) {
> + $found = 1;
> + }
> + }
> + if (!$found) {
> + print "/**\n";
> + print " * message\n";
> + for ($k = 0; $k < $messages{$i}->{'PARM_COUNT'}; $k++) {
> + print " * \@$k: $messages{$i}->{'PARM_NAME'}->{$k}\n";
> + }
> + print " *\n";
> + print " * Description:\n";
> + print " *\n";
> + print " * User Response:\n";
> + print " */\n";
> + print "\n";
> + print "KMSG_DOC($messages{$i}->{'COMP'}, $messages{$i}->{'NR'}, \"$messages{$i}->{'MSG'}\");\n"
> + }
> + }
> +}
> +
> +sub usage()
> +{
> + print "USAGE: kmsg_tool print | check <file>\n";
> + exit 1;
> +}
> +
> +$option = shift;
> +
> +if ($option eq "check") {
> + $gcc_options = "-E -D __KMSG_CHECKER ";
> + do {
> + $filename = $tmp;
> + $tmp = shift;
> + $tmp =~ s/$/\\\(/;
> + $tmp =~ s/$/\\\)/;
> + $gcc_options .= " $tmp";
> + } while (!($tmp eq ""));
> +
> + $gcc_options =~ s/-Wbitwise//; # XXX hack to remove -Wbitwise CHECKFLAG
> + $gcc_options =~ s/-D__STDC__//; # XXX hack to remove -D__STDC__
> + $tmp_file = "$filename.msg";
> + system("gcc $gcc_options > $tmp_file");
> + get_descriptions($filename);
> + get_msgs($tmp_file);
> + check_messages($filename);
> + print_templates();
> + system("rm $tmp_file");
> +} elsif ($option eq "print") {
> + $filename = shift;
> + do {
> + print STDERR "Processing: $filename\n";
> + get_descriptions($filename);
> + print_descriptions("all");
> + $filename = shift;
> + } while (!($filename eq ""));
> +} else {
> + usage();
> +}
>

2007-06-13 18:21:42

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

Hi Greg,

> Ick, why are you ignoring what we have already with dev_printk() and
> friends? We are just finally getting developers to use that, I think it
> will be almost impossible to get people to change to something else,
> especially one that isn't even as "correct" as what dev_printk() offers
> you today, will be quite hard.
>
> So, why not use what we already have and work off of it?

dev_printk() and friends are great, since they already define something
like KMSG_COMPONENT: The driver name.

But they unfortunately do not solve our problem. We need an identifier
for a documented message in order to find the right description for a
message.

We could add new dev_printk macros for KMSG printks:

#define dev_printk_kmsg(kmsg, dev, format, arg...) \
printk(kmsg "%s: " format , (dev)->bus_id , ## arg)
#define dev_err_kmsg(dev, nr, format, arg...) \
dev_printk_kmsg(KMSG_ERR(nr) , dev , format , ## arg)

E.g. hub.c:
===========

#define KMSG_COMPONENT "hub"

dev_err_kmsg(&udev->dev, 1,
"can't read configurations, error %d\n", err);

Which would result in:

hub.1 5-0:1.0: can't read configuration error 4711

Michael

2007-06-13 18:23:17

by David Miller

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

I think my general response to something like this, if it goes
in, would be to stop emitting useful kernel log messages
in the code I write because having to document it too on top
of that is just too much extra work to be worthwhile.

2007-06-13 18:30:30

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Wed, Jun 13, 2007 at 08:18:00PM +0200, holzheu wrote:
> Hi Greg,
>
> > Ick, why are you ignoring what we have already with dev_printk() and
> > friends? We are just finally getting developers to use that, I think it
> > will be almost impossible to get people to change to something else,
> > especially one that isn't even as "correct" as what dev_printk() offers
> > you today, will be quite hard.
> >
> > So, why not use what we already have and work off of it?
>
> dev_printk() and friends are great, since they already define something
> like KMSG_COMPONENT: The driver name.

They provide way more than that, they also provide the explicit device
that is being discussed, as well as other things depending on the
device.

So if you are going to do this, please use the already-in-place macros
to hook into, don't try to get the driver authors to pick up something
new and different, as it's going to be _very_ difficult, trust me...

thanks,

greg k-h

2007-06-13 18:31:37

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Wed, 2007-06-13 at 11:23 -0700, David Miller wrote:
> I think my general response to something like this, if it goes
> in, would be to stop emitting useful kernel log messages
> in the code I write because having to document it too on top
> of that is just too much extra work to be worthwhile.

Nobody will force you to document your messages. As I said, there is no
need to document self-explaining printks.

Michael

2007-06-13 18:49:29

by Dave Hansen

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Wed, 2007-06-13 at 11:32 -0700, Greg KH wrote:
>
> > dev_printk() and friends are great, since they already define
> something
> > like KMSG_COMPONENT: The driver name.
>
> They provide way more than that, they also provide the explicit device
> that is being discussed, as well as other things depending on the
> device.

Instead of keeping a list of kernel message ids somewhere, can you base
this off the dev_printk() format argument?

You could keep a list of those somewhere, extracted from the kernel
source, and correlate documentation with the _format_ instead of an
arbitrarily assigned message number.

-- Dave

2007-06-13 18:49:53

by Kok, Auke

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

Greg KH wrote:
> On Wed, Jun 13, 2007 at 08:18:00PM +0200, holzheu wrote:
>> Hi Greg,
>>
>>> Ick, why are you ignoring what we have already with dev_printk() and
>>> friends? We are just finally getting developers to use that, I think it
>>> will be almost impossible to get people to change to something else,
>>> especially one that isn't even as "correct" as what dev_printk() offers
>>> you today, will be quite hard.
>>>
>>> So, why not use what we already have and work off of it?
>> dev_printk() and friends are great, since they already define something
>> like KMSG_COMPONENT: The driver name.
>
> They provide way more than that, they also provide the explicit device
> that is being discussed, as well as other things depending on the
> device.
>
> So if you are going to do this, please use the already-in-place macros
> to hook into, don't try to get the driver authors to pick up something
> new and different, as it's going to be _very_ difficult, trust me...

absolutely... I'm currently trying to convince everyone to get a generic
netdevice-centric printk macro (dev_printk does not print the interface name,
which the netdevice can't store locally since it can change) in and even that is
a pain :)

Auke

2007-06-13 19:19:59

by Joe Perches

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Wed, 2007-06-13 at 20:18 +0200, holzheu wrote:
> But they unfortunately do not solve our problem. We need an identifier
> for a documented message in order to find the right description for a
> message.

I believe it better to simply add __FILE__ & __LINE__ to the
macro rather than some other externally specified unique
identifier that adds developer overhead and easily gets stale.

If you go overboard, you could do something like:

#define dev_dbg(dev, format, arg...) \
dev_printk(KERN_DEBUG, dev, DBG_FMT format DBG_ARG, ## arg)

#define dev_err(dev, format, arg...) \
dev_printk(KERN_ERR, dev, PR_FMT format PR_ARG, ## arg)

etc...

and use config variables to control what additional info is printk'ed.

Go overboard without a life preserver and add __TIMESTAMP__ too.

/*
*/
#if defined DBG_FMT
#undef DBG_FMT
#endif
#if defined DBG_ARG
#undef DBG_ARG
#endif

#if defined DBG_FILE && defined DBG_FUNCTION && defined DBG_LINE
#define DBG_FMT "(%s:%s:%u) "
#define DBG_ARG , __FILE__ , __FUNCTION__ , __LINE__
#elif defined DBG_FILE && defined DBG_FUNCTION && !defined DBG_LINE
#define DBG_FMT "(%s:%s) "
#define DBG_ARG , __FILE__ , __FUNCTION__
#elif defined DBG_FILE && !defined DBG_FUNCTION && defined DBG_LINE
#define DBG_FMT "(%s:%u) "
#define DBG_ARG , __FILE__ , __LINE__
#elif defined DBG_FILE && !defined DBG_FUNCTION && !defined DBG_LINE
#define DBG_FMT "(%s) "
#define DBG_ARG , __FILE__
#elif !defined DBG_FILE && defined DBG_FUNCTION && defined DBG_LINE
#define DBG_FMT "(%s:%u) "
#define DBG_ARG , __FUNCTION__ , __LINE__
#elif !defined DBG_FILE && defined DBG_FUNCTION && !defined DBG_LINE
#define DBG_FMT "(%s) "
#define DBG_ARG , __FUNCTION__
#elif !defined DBG_FILE && !defined DBG_FUNCTION && defined DBG_LINE
#define DBG_FMT "(%u) "
#define DBG_ARG , __LINE__
#else
#define DBG_FMT
#define DBG_ARG
#endif

/*
*/

#if defined PR_FMT
#undef PR_FMT
#endif
#if defined PR_ARG
#undef PR_ARG
#endif

#if defined PR_FILE && defined PR_FUNCTION && defined PR_LINE
#define PR_FMT "(%s:%s:%u) "
#define PR_ARG , __FILE__ , __FUNCTION__ , __LINE__
#elif defined PR_FILE && defined PR_FUNCTION && !defined PR_LINE
#define PR_FMT "(%s:%s) "
#define PR_ARG , __FILE__ , __FUNCTION__
#elif defined PR_FILE && !defined PR_FUNCTION && defined PR_LINE
#define PR_FMT "(%s:%u) "
#define PR_ARG , __FILE__ , __LINE__
#elif defined PR_FILE && !defined PR_FUNCTION && !defined PR_LINE
#define PR_FMT "(%s) "
#define PR_ARG , __FILE__
#elif !defined PR_FILE && defined PR_FUNCTION && defined PR_LINE
#define PR_FMT "(%s:%u) "
#define PR_ARG , __FUNCTION__ , __LINE__
#elif !defined PR_FILE && defined PR_FUNCTION && !defined PR_LINE
#define PR_FMT "(%s) "
#define PR_ARG , __FUNCTION__
#elif !defined PR_FILE && !defined PR_FUNCTION && defined PR_LINE
#define PR_FMT "(%u) "
#define PR_ARG , __LINE__
#else
#define PR_FMT
#define PR_ARG
#endif

2007-06-14 02:55:29

by Valdis Klētnieks

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Wed, 13 Jun 2007 12:04:56 PDT, Joe Perches said:

> I believe it better to simply add __FILE__ & __LINE__ to the
> macro rather than some other externally specified unique
> identifier that adds developer overhead and easily gets stale.

There's been plenty of times I've wished for that. Now if we just found a way
to do something sane for drivers/net/wireless/mac80211/iwlwifi/base.c ;)

Of course, I'm probably atypical - for me, kernel messages come in two forms,
the kind that are immediately obvious at first reading, and the kind that
you end up having to look at the code and wonder WTF? it was doing inside
that never-should-happen-on-this-hardware while loop in the first place... :)

Attachments:

(No filename) (226.00 B)

2007-06-14 07:06:41

by H. Peter Anvin

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

[email protected] wrote:
> On Wed, 13 Jun 2007 12:04:56 PDT, Joe Perches said:
>
>> I believe it better to simply add __FILE__ & __LINE__ to the
>> macro rather than some other externally specified unique
>> identifier that adds developer overhead and easily gets stale.
>
> There's been plenty of times I've wished for that. Now if we just found a way
> to do something sane for drivers/net/wireless/mac80211/iwlwifi/base.c ;)
>

Why don't you just redefine the KERN_* macros which is used by almost
every printk in the system to contain said __FILE__ and __LINE__, then?

-hpa

2007-06-14 08:16:07

by Martin Schwidefsky

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Wed, 2007-06-13 at 11:32 -0700, Greg KH wrote:
> > > So, why not use what we already have and work off of it?
> >
> > dev_printk() and friends are great, since they already define
> something
> > like KMSG_COMPONENT: The driver name.
>
> They provide way more than that, they also provide the explicit device
> that is being discussed, as well as other things depending on the
> device.

dev_printk() and friends provide additional information for a printk
that is related to a device. Not every printk is about a device, so I
think Michaels proposal is orthorgonal to dev_printk. We definitly
should have a dev_printk variant that uses the kmsg documentation tag
AND we should have a normal printk variant as well that uses the kmsg
tagging.

> So if you are going to do this, please use the already-in-place macros
> to hook into, don't try to get the driver authors to pick up something
> new and different, as it's going to be _very_ difficult, trust me...

There is no doubt that it will be difficult to get a larger part of the
developers use the kmsg scheme (e.g. see the reaction of Dave M.). We do
not have the illusion that we can replace every single message in the
kernel, nor do we think that it would make sense to do so. What we would
do for s390 is to check each message in drivers/s390 and think hard if
the message should be 1) removed, 2) replaced with a {dev_}printk_kmsg
or 3) left as it is. Fortunatly for us there are not too many drivers
for s390 ..

--
blue skies,
Martin.

"Reality continues to ruin my life." - Calvin.

2007-06-14 08:29:59

by Martin Schwidefsky

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Wed, 2007-06-13 at 11:15 -0700, Andrew Morton wrote:
> Your proposal is similar to one I made to some Japanese developers
> earlier this year.

Interesting. Our requirement comes from Japan as well.

> I was more modest, proposing that we
>
> - add an enhanced printk
>
> xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

Some kind of id is always required to be able to identify the message.
The task is to make the tagging as simple as possible.

> - An externally managed database will provide translations, diagnostics,
> remedial actions, etc, keyed off the msgid string

There I have my doubts if this can work. If you keep an external
database with the additional information about the messages the kernel
code and the database will get out of sync. Thats why we advocate the
kernel-doc style approach: the message catalogue will always be in sync
because it is delivered with the kernel.

> - Format and management of the msgid hasn't been clearly identified yet
> as far as I know. But all we really need is that each ID be unique,
> kernel-wide. We develop a simple tool which can check the whole tree
> for duplicated msgids.

This is a paint point with Michaels approach as well. The KMSG_COMPONENT
defines are rather ugly. A clever way how to generate kernel unique IDs
would be nice. Intervention required..

> - From a practical day-to-day POV what this means is that a person
> from the kernel-messages team will prepare a patch for your driver
> which adds the msgids and you the driver author should take care not
> to break the tagging.
>
> This is deliberately designed to be minimum-impact upon general
> developers. We want a system in place where driver/fs/etc developers
> can concentrate on what they do, and kernel-message developers can
> as independently as possibe take care of what _they_ do.

With the KMSG approach all that needs to be done is to replace the
KERN_xxx with KMSG_xxx(<number) and to add a comment at the end of the
file.

> - A project has started up and there is a mailing list (cc'ed here) and
> meetings are happening at the LF collaboration summit this week.
>
> Please see https://lists.linux-foundation.org/mailman/listinfo/lf_kernel_messages
> and don't miss the next conference call ;)
>
> (argh. The lf_kernel_messages archives are subscriber-only and it could be that
> only members can post to it. Oh well. Please subscribe, and review the archives)

Very interesting. We did not know about this project. Thanks for the
hint.

--
blue skies,
Martin.

"Reality continues to ruin my life." - Calvin.

2007-06-14 08:47:17

by Krzysztof Halasa

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

Hi,

holzheu <[email protected]> writes:

> If the information is to big for the printk itself, because you would
> need 10 lines to explain what happened, wouldn't it be good to have a
> place where to put that information?

I think if a message really has to be 10 lines long to be clear then
it should just be 10 lines long. The keyword is "really".

Example: oops.
--
Krzysztof Halasa

2007-06-14 09:41:38

by Jan Kara

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

<snip>

> Your proposal is similar to one I made to some Japanese developers
> earlier this year. I was more modest, proposing that we
>
> - add an enhanced printk
>
> xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
Maybe a stupid idea but why do we want to assign these numbers by hand?
I can imagine it could introduce collisions when merging tons of patches
with new messages... Wouldn't it be better to compute say, 8-byte hash
from the message and use it as it's identifier? We could do this
automagically at compile time. I know it also has it's problems - you
fix a spelling and the message gets a different id and you have to
update translation/documentation catalogue but maybe that could be
solved too...

Honza

2007-06-14 10:42:39

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Thu, 2007-06-14 at 11:41 +0200, Jan Kara wrote:
> <snip>
>
> > Your proposal is similar to one I made to some Japanese developers
> > earlier this year. I was more modest, proposing that we
> >
> > - add an enhanced printk
> >
> > xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
> Maybe a stupid idea but why do we want to assign these numbers by hand?
> I can imagine it could introduce collisions when merging tons of patches
> with new messages... Wouldn't it be better to compute say, 8-byte hash
> from the message and use it as it's identifier? We could do this
> automagically at compile time.

Of course automatically generated message numbers would be great and
something like:

hub.4a5bcd77: Detected some problem.

looks acceptable for me.

We could generate the hash using the format string of the printk. Since
we specify the format string also in KMSG_DOC, the hash for the
KMSG_DOC and the printk should match and we have the required link
between printk and description.

So technically that's probably doable.

Problems are:

* hashes are not unique
* We need an additional preprocessor step
* The might be people, who find 8 character hash values ugly in printks

The big advantage is, that we do not need to maintain message numbers.

> I know it also has it's problems - you
> fix a spelling and the message gets a different id and you have to
> update translation/documentation catalogue but maybe that could be
> solved too...

Since in our approach the message catalog is created automatically for
exactly one kernel and the message catalog belongs therefore to exactly
one kernel, I think the problem of changing error numbers is not too
big.

Michael

2007-06-14 11:51:28

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Thu, 2007-06-14 at 12:38 +0200, holzheu wrote:
> On Thu, 2007-06-14 at 11:41 +0200, Jan Kara wrote:
> > <snip>
> >
> > > Your proposal is similar to one I made to some Japanese developers
> > > earlier this year. I was more modest, proposing that we
> > >
> > > - add an enhanced printk
> > >
> > > xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
> > Maybe a stupid idea but why do we want to assign these numbers by hand?
> > I can imagine it could introduce collisions when merging tons of patches
> > with new messages... Wouldn't it be better to compute say, 8-byte hash
> > from the message and use it as it's identifier? We could do this
> > automagically at compile time.
>
> Of course automatically generated message numbers would be great and
> something like:
>
> hub.4a5bcd77: Detected some problem.

Sorry, I first read: 8 characters not 8 bytes.

Indeed, "hub.d41d8cd98f00b204: Detected some problem." ... does not look
like very beautiful.

But maybe also 4 bytes would be enough, since the hash only has to be
unique within one component e.g. "hub".

Michael

2007-06-14 12:13:46

by Jan Kara

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Thu 14-06-07 13:47:41, holzheu wrote:
> On Thu, 2007-06-14 at 12:38 +0200, holzheu wrote:
> > On Thu, 2007-06-14 at 11:41 +0200, Jan Kara wrote:
> > > <snip>
> > >
> > > > Your proposal is similar to one I made to some Japanese developers
> > > > earlier this year. I was more modest, proposing that we
> > > >
> > > > - add an enhanced printk
> > > >
> > > > xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
> > > Maybe a stupid idea but why do we want to assign these numbers by hand?
> > > I can imagine it could introduce collisions when merging tons of patches
> > > with new messages... Wouldn't it be better to compute say, 8-byte hash
> > > from the message and use it as it's identifier? We could do this
> > > automagically at compile time.
> >
> > Of course automatically generated message numbers would be great and
> > something like:
> >
> > hub.4a5bcd77: Detected some problem.
>
> Sorry, I first read: 8 characters not 8 bytes.
>
> Indeed, "hub.d41d8cd98f00b204: Detected some problem." ... does not look
> like very beautiful.
>
> But maybe also 4 bytes would be enough, since the hash only has to be
> unique within one component e.g. "hub".
It depends how large components you expect. For example for 10000
messages there is already 1% probability of collision so it means sooner or
later we are going to hit it... For 1000 messages the probability is
roughly 0.1% which is still not so small that I'd be comfortable with it.
On the other hand we could use something like gperf to generate perfect
hashing function and then we don't have to worry about colisions.

Honza
--
Jan Kara <[email protected]>
SuSE CR Labs

2007-06-14 15:26:12

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Thu, 2007-06-14 at 14:26 +0200, Jan Kara wrote:

<snip>

> > But maybe also 4 bytes would be enough, since the hash only has to be
> > unique within one component e.g. "hub".
> It depends how large components you expect. For example for 10000
> messages there is already 1% probability of collision so it means sooner or
> later we are going to hit it... For 1000 messages the probability is
> roughly 0.1% which is still not so small that I'd be comfortable with it.

I wouldn't expect, that a "normal" component like a device driver will
have more than 100 documented messages.

Collisions are ugly, but will not hurt much. The operator will see for a
message two possible descriptions. Normally it should be possible to
figure out manually which description is the right one.

Michael

2007-06-15 08:49:35

by Jan Engelhardt

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Jun 13 2007 12:50, [email protected] wrote:
>> In general we think, that also for Linux it is a good thing to have
>> documentation for the most important kernel/driver messages. Even
>> kernel hackers not always are aware of the meaning of kernel messages
>> for components, which they don't know in detail. Most of the messages
>> are self explaining but sometimes you get something like "Clocksource
>> tsc unstable (delta = 7304132729 ns)" and you wonder if your system is
>> going to explode.

I suppose no. If it is going to explode, it will tell you :-)

/* fs/super.c */
printk("VFS: Busy inodes after unmount of %s. "
"Self-destruct in 5 seconds. Have a nice day...\n",
sb->s_id);

>This is probably best addressed by cleaning up the actual messages so they're
>a bit more informative.

Jan
--

2007-06-15 08:52:23

by Jan Engelhardt

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Jun 13 2007 12:04, Joe Perches wrote:
>On Wed, 2007-06-13 at 20:18 +0200, holzheu wrote:
>> But they unfortunately do not solve our problem. We need an identifier
>> for a documented message in order to find the right description for a
>> message.
>
>I believe it better to simply add __FILE__ & __LINE__ to the

Erk, no. That bloats the kernel to the max. Especially since kernel
files are compiled with a rather longish path.

Jan
--

2007-06-15 12:40:54

by Pavel Machek

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

Hi!

> > Your proposal is similar to one I made to some Japanese developers
> > earlier this year. I was more modest, proposing that we
> >
> > - add an enhanced printk
> >
> > xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
> Maybe a stupid idea but why do we want to assign these numbers by hand?
> I can imagine it could introduce collisions when merging tons of patches
> with new messages... Wouldn't it be better to compute say, 8-byte hash
> from the message and use it as it's identifier? We could do this

Why hashes? Just key it off full message, and treat it like normal
translation problem.

'lp%d: on fire' -> 'your printer failed basic tests, you should check
cabling'.

--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

2007-06-15 18:43:04

by Gerrit Huizenga

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Thu, 14 Jun 2007 12:38:53 +0200, holzheu wrote:
> On Thu, 2007-06-14 at 11:41 +0200, Jan Kara wrote:
> > <snip>
> >
> > > Your proposal is similar to one I made to some Japanese developers
> > > earlier this year. I was more modest, proposing that we
> > >
> > > - add an enhanced printk
> > >
> > > xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
> > Maybe a stupid idea but why do we want to assign these numbers by hand?
> > I can imagine it could introduce collisions when merging tons of patches
> > with new messages... Wouldn't it be better to compute say, 8-byte hash
> > from the message and use it as it's identifier? We could do this
> > automagically at compile time.
>
> Of course automatically generated message numbers would be great and
> something like:
>
> hub.4a5bcd77: Detected some problem.
>
> looks acceptable for me.
>
> We could generate the hash using the format string of the printk. Since
> we specify the format string also in KMSG_DOC, the hash for the
> KMSG_DOC and the printk should match and we have the required link
> between printk and description.
>
> So technically that's probably doable.
>
> Problems are:
>
> * hashes are not unique
> * We need an additional preprocessor step
> * The might be people, who find 8 character hash values ugly in printks
>
> The big advantage is, that we do not need to maintain message numbers.
>
> > I know it also has it's problems - you
> > fix a spelling and the message gets a different id and you have to
> > update translation/documentation catalogue but maybe that could be
> > solved too...
>
> Since in our approach the message catalog is created automatically for
> exactly one kernel and the message catalog belongs therefore to exactly
> one kernel, I think the problem of changing error numbers is not too
> big.

We just had a meeting with the Japanese and several other participants
from the vendor and community side and came up with a potential proposal
that is similar to many things discussed here. It has the benefit that
it seems implementable and low/no overhead on most kernel developers.

The basic proposal is to use a tool, run by the kernel Makefile to
extract kernel messages from either the kernel source or the .i files
(both have advantages, although I prefer the source to the .i file
since it 1) gets all messages and 2) is probably a little quicker with
less impact to the standard kernel make.

These messages would be stored in a file in the source tree, e.g.
usr/src/linux/Translations/English. As each message is added to that
file, we calculate, say, an MD5 sum of the printk (dev_printk, sdev_printk,
etc.) string, and the text file ultimately contains:

MD5 Checksum of text; the printk text itself, the File name, the line number.

The checksum is run over just the printk. We definitely would not include
the line number since the line number is too volatile. Including the
file name in the hash *might* help disambiguate the hash a bit better in
the case of duplicates, but there was some debate that duplicates might
be better handled in other ways.

Andrew mentioned a mechanism for adding a subsystem tag or other tag
which helps disambiguate the message, either in the message file or in
the end user documentation (e.g. the Message Pedia/mPedia that the Japanese
have already created with ~350 messages, and a total of ~700 targetted
by the end of the year).

That tag could be appended to the beginning of the printk, to the end of
the printk, or even in a formatted comment at the end of the printk that
the tool could extract.

Then, the translations could be managed by anyone outside of the normal/
core kernel community, by simply creating a translation file, e.g.
usr/src/linux/Translations/Japanese, which contained the MD5 sum, the
translated message, the file name and line number (the last two redundent
perhaps but informational, and automatically generated if possible).

The files in the Translations directory could be uesd as the unique
keys for an external database (such as the Message Pedia, vendors or
distributions help pages, etc.) to help look up and explain root cause
of a problem. The key property here is that the MD5 sum becomes the
key to all database entries to look up that key.

Further, yet another kernel config option could allow distros to output
the calculated MD5 sum to be printed, much like we do with timestamps
today.

End result is that these in-kernel message catalogs for translation are
updated automatically (mostly no kernel developer changes needed) and
the translations can be maintained by anyone who is interested.

On the topic of MD5 collisions, using a disambiguating tag would be a
simple addition for the few cases where that happens, the tool could
be educated to use that tag in the calculation of the MD5 sum, and we
have a 98% solution which impacts <1% of the kernel developers.

Folks present for this discussion included Ted T'so, James Bottomley,
several of the key Japenese folks interested in using this for debugging,
and reps from several vendors who would find this sort of info useful
for their folks supporting Linux in the field.

Comments?

gerrit

2007-06-15 18:51:48

by Randy Dunlap

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

Gerrit Huizenga wrote:
> On Thu, 14 Jun 2007 12:38:53 +0200, holzheu wrote:
>> On Thu, 2007-06-14 at 11:41 +0200, Jan Kara wrote:
>>> <snip>
>>>
>>>> Your proposal is similar to one I made to some Japanese developers
>>>> earlier this year. I was more modest, proposing that we
>>>>
>>>> - add an enhanced printk
>>>>
>>>> xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
>>> Maybe a stupid idea but why do we want to assign these numbers by hand?
>>> I can imagine it could introduce collisions when merging tons of patches
>>> with new messages... Wouldn't it be better to compute say, 8-byte hash
>>> from the message and use it as it's identifier? We could do this
>>> automagically at compile time.
>> Of course automatically generated message numbers would be great and
>> something like:
>>
>> hub.4a5bcd77: Detected some problem.
>>
>> looks acceptable for me.
>>
>> We could generate the hash using the format string of the printk. Since
>> we specify the format string also in KMSG_DOC, the hash for the
>> KMSG_DOC and the printk should match and we have the required link
>> between printk and description.
>>
>> So technically that's probably doable.
>>
>> Problems are:
>>
>> * hashes are not unique
>> * We need an additional preprocessor step
>> * The might be people, who find 8 character hash values ugly in printks
>>
>> The big advantage is, that we do not need to maintain message numbers.
>>
>>> I know it also has it's problems - you
>>> fix a spelling and the message gets a different id and you have to
>>> update translation/documentation catalogue but maybe that could be
>>> solved too...
>> Since in our approach the message catalog is created automatically for
>> exactly one kernel and the message catalog belongs therefore to exactly
>> one kernel, I think the problem of changing error numbers is not too
>> big.
>
> We just had a meeting with the Japanese and several other participants
> from the vendor and community side and came up with a potential proposal
> that is similar to many things discussed here. It has the benefit that
> it seems implementable and low/no overhead on most kernel developers.
>
> The basic proposal is to use a tool, run by the kernel Makefile to
> extract kernel messages from either the kernel source or the .i files
> (both have advantages, although I prefer the source to the .i file
> since it 1) gets all messages and 2) is probably a little quicker with
> less impact to the standard kernel make.
>
> These messages would be stored in a file in the source tree, e.g.
> usr/src/linux/Translations/English. As each message is added to that
> file, we calculate, say, an MD5 sum of the printk (dev_printk, sdev_printk,
> etc.) string, and the text file ultimately contains:
>
> MD5 Checksum of text; the printk text itself, the File name, the line number.
>
> The checksum is run over just the printk. We definitely would not include
> the line number since the line number is too volatile. Including the
> file name in the hash *might* help disambiguate the hash a bit better in
> the case of duplicates, but there was some debate that duplicates might
> be better handled in other ways.
>
> Andrew mentioned a mechanism for adding a subsystem tag or other tag
> which helps disambiguate the message, either in the message file or in
> the end user documentation (e.g. the Message Pedia/mPedia that the Japanese
> have already created with ~350 messages, and a total of ~700 targetted
> by the end of the year).
>
> That tag could be appended to the beginning of the printk, to the end of
> the printk, or even in a formatted comment at the end of the printk that
> the tool could extract.
>
> Then, the translations could be managed by anyone outside of the normal/
> core kernel community, by simply creating a translation file, e.g.
> usr/src/linux/Translations/Japanese, which contained the MD5 sum, the
> translated message, the file name and line number (the last two redundent
> perhaps but informational, and automatically generated if possible).
>
> The files in the Translations directory could be uesd as the unique
> keys for an external database (such as the Message Pedia, vendors or
> distributions help pages, etc.) to help look up and explain root cause
> of a problem. The key property here is that the MD5 sum becomes the
> key to all database entries to look up that key.
>
> Further, yet another kernel config option could allow distros to output
> the calculated MD5 sum to be printed, much like we do with timestamps
> today.
>
> End result is that these in-kernel message catalogs for translation are
> updated automatically (mostly no kernel developer changes needed) and
> the translations can be maintained by anyone who is interested.
>
> On the topic of MD5 collisions, using a disambiguating tag would be a
> simple addition for the few cases where that happens, the tool could
> be educated to use that tag in the calculation of the MD5 sum, and we
> have a 98% solution which impacts <1% of the kernel developers.
>
> Folks present for this discussion included Ted T'so, James Bottomley,
> several of the key Japenese folks interested in using this for debugging,
> and reps from several vendors who would find this sort of info useful
> for their folks supporting Linux in the field.
>
> Comments?

For those of us who have not been in on these meetings, I think that
some serious justifications are needed. The last paragraph began to
go in that direction, but it needs to be more detailed and convincing.
And "for debugging" doesn't cut it IMO.

--
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

2007-06-15 20:01:48

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Fri, Jun 15, 2007 at 11:51:51AM -0700, Randy Dunlap wrote:
> Gerrit Huizenga wrote:
> > On Thu, 14 Jun 2007 12:38:53 +0200, holzheu wrote:
> >> On Thu, 2007-06-14 at 11:41 +0200, Jan Kara wrote:
> >>> <snip>
> >>>
> >>>> Your proposal is similar to one I made to some Japanese developers
> >>>> earlier this year. I was more modest, proposing that we
> >>>>
> >>>> - add an enhanced printk
> >>>>
> >>>> xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
> >>> Maybe a stupid idea but why do we want to assign these numbers by hand?
> >>> I can imagine it could introduce collisions when merging tons of patches
> >>> with new messages... Wouldn't it be better to compute say, 8-byte hash
> >>> from the message and use it as it's identifier? We could do this
> >>> automagically at compile time.
> >> Of course automatically generated message numbers would be great and
> >> something like:
> >>
> >> hub.4a5bcd77: Detected some problem.
> >>
> >> looks acceptable for me.
> >>
> >> We could generate the hash using the format string of the printk. Since
> >> we specify the format string also in KMSG_DOC, the hash for the
> >> KMSG_DOC and the printk should match and we have the required link
> >> between printk and description.
> >>
> >> So technically that's probably doable.
> >>
> >> Problems are:
> >>
> >> * hashes are not unique
> >> * We need an additional preprocessor step
> >> * The might be people, who find 8 character hash values ugly in printks
> >>
> >> The big advantage is, that we do not need to maintain message numbers.
> >>
> >>> I know it also has it's problems - you
> >>> fix a spelling and the message gets a different id and you have to
> >>> update translation/documentation catalogue but maybe that could be
> >>> solved too...
> >> Since in our approach the message catalog is created automatically for
> >> exactly one kernel and the message catalog belongs therefore to exactly
> >> one kernel, I think the problem of changing error numbers is not too
> >> big.
> > We just had a meeting with the Japanese and several other participants
> > from the vendor and community side and came up with a potential proposal
> > that is similar to many things discussed here. It has the benefit that
> > it seems implementable and low/no overhead on most kernel developers.
> > The basic proposal is to use a tool, run by the kernel Makefile to
> > extract kernel messages from either the kernel source or the .i files
> > (both have advantages, although I prefer the source to the .i file
> > since it 1) gets all messages and 2) is probably a little quicker with
> > less impact to the standard kernel make.
> > These messages would be stored in a file in the source tree, e.g.
> > usr/src/linux/Translations/English. As each message is added to that
> > file, we calculate, say, an MD5 sum of the printk (dev_printk, sdev_printk,
> > etc.) string, and the text file ultimately contains:
> > MD5 Checksum of text; the printk text itself, the File name, the line
> > number.
> > The checksum is run over just the printk. We definitely would not include
> > the line number since the line number is too volatile. Including the
> > file name in the hash *might* help disambiguate the hash a bit better in
> > the case of duplicates, but there was some debate that duplicates might
> > be better handled in other ways.
> > Andrew mentioned a mechanism for adding a subsystem tag or other tag
> > which helps disambiguate the message, either in the message file or in
> > the end user documentation (e.g. the Message Pedia/mPedia that the Japanese
> > have already created with ~350 messages, and a total of ~700 targetted
> > by the end of the year).
> > That tag could be appended to the beginning of the printk, to the end of
> > the printk, or even in a formatted comment at the end of the printk that
> > the tool could extract.
> > Then, the translations could be managed by anyone outside of the normal/
> > core kernel community, by simply creating a translation file, e.g.
> > usr/src/linux/Translations/Japanese, which contained the MD5 sum, the
> > translated message, the file name and line number (the last two redundent
> > perhaps but informational, and automatically generated if possible).
> > The files in the Translations directory could be uesd as the unique
> > keys for an external database (such as the Message Pedia, vendors or
> > distributions help pages, etc.) to help look up and explain root cause
> > of a problem. The key property here is that the MD5 sum becomes the
> > key to all database entries to look up that key.
> > Further, yet another kernel config option could allow distros to output
> > the calculated MD5 sum to be printed, much like we do with timestamps
> > today.
> > End result is that these in-kernel message catalogs for translation are
> > updated automatically (mostly no kernel developer changes needed) and
> > the translations can be maintained by anyone who is interested.
> > On the topic of MD5 collisions, using a disambiguating tag would be a
> > simple addition for the few cases where that happens, the tool could
> > be educated to use that tag in the calculation of the MD5 sum, and we
> > have a 98% solution which impacts <1% of the kernel developers.
> > Folks present for this discussion included Ted T'so, James Bottomley,
> > several of the key Japenese folks interested in using this for debugging,
> > and reps from several vendors who would find this sort of info useful
> > for their folks supporting Linux in the field.
> > Comments?
>
> For those of us who have not been in on these meetings, I think that
> some serious justifications are needed. The last paragraph began to
> go in that direction, but it needs to be more detailed and convincing.
> And "for debugging" doesn't cut it IMO.

Full "justification" has never really been clear outside of the need to
try to duplicate what other Unixes have had for their support
departments in the past. And yes, I've seen and sat through the
presentation by the Japanese group many different times, and talked to
the individual people who want this.

And you know, that's fine. If anyone wants to go off and do such a
project like this, in a way that is almost entirely self-contained with
no affect on the development process or any kernel developer, I have no
objection to it at all.

And I think the proposal that Andrew made meets that requirement, so
let's wait and see what actual code comes out of this and judge it on
that merit at that time.

thanks,

greg k-h

2007-06-15 20:56:18

by Gerrit Huizenga

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Fri, 15 Jun 2007 11:51:51 PDT, Randy Dunlap wrote:
>
> For those of us who have not been in on these meetings, I think that
> some serious justifications are needed. The last paragraph began to
> go in that direction, but it needs to be more detailed and convincing.
> And "for debugging" doesn't cut it IMO.
>
> --
> ~Randy

Fair point, Randy.

This came up this time because people in Japan supporting Linux customers
are not always familiar with Linux kernel internals and thus debugging
problems based on a terse, English language message is just painful. Also,
any given message doesn't really provide any input about what *really*
failed, what should be done about it, how to correct this issue, or what
parts to pull out of your machine and replace with properly working parts.

So, the Message Pedia is basically a Japanese language wiki today where
people doing support can annotate the messages with real life experience on
what to *do* about the error.

Part of the problem is that the error messages when extracted don't
stand alone and don't easily allow tracing back to the real source. When
the problem is handed from the second line support folks deeper into the
support or development organization, the message has to be found in the
source tree in some non-ambiguous fashion.

The other problem that comes up in other, non-English locales is that
debugging your Linux box when you are a non-native English speaker is
just plain impossible. Providing a means for getting localized kernel
messages allows more end users to at least reasonably debug their own
machine. Most every user level project today allows localization directly
but the kernel has never had any reasonable mechanism for localisation.

Now, the proposal here doesn't make the kernel "as good as" user level
application, but it would at least provide a searchable key to help
people get localized guidance on problems.

I'm sure there are other benefits that most can see based on just simple
debuggability. But the Japanese goal is really to construct a database
of debugging info & context based on their experiences.

And, I'm sure that others can speak for themselves. I'm mostly playing
scribe on this one - we abandoned our error and event subsystem work years
ago, even though this was one of the biggest weaknesses we saw in customer
support 3-5 years ago, simply because our changes were still viewed as
too invasive for mainline kernel. These proposed changes help a lot, while
remaining almost completely invisible to most developers.

gerrit

2007-06-18 12:59:52

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

Hi Gerrit,

The common thing of your and our approach is, that we need an ID to
identify a message either by:

* Using hashes + component name (maybe)
* Or using hand-made message numbers + component name. Numbers have
to be unique within the kernel component

I think the main difference of your approach is, that documentation will
be maintained outside the kernel. This has advantages, but also
disadvantages:

(+) Developers have less work to do :-)
(+) Kernel sources and patches have less lines of code, since message
descriptions are not contained in the kernel sources.
(+) You can support multiple languages

(-) Hard to keep printks and descriptions up-to-date. If using hashes,
each typo fix in a printk breaks the link to your documentation.
(-) Since the developer knows the meaning of a printk best, he probably
would be the right person to write the description.
(-) For every Distribution or vanilla kernel you probably need separate
external message databases.

I like the fact, that every printk gets an ID in your approach and you
do not need additional printk macros.

Maybe both approaches are not mutually exclusive. If developers would
like to document their messages within the kernel sources, they still
could use kernel-doc and our KMSG_DOC macro do do that.

I admit, that I am not sure about the the best solution here.

Michael

2007-06-18 13:11:16

by Sam Ravnborg

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Fri, Jun 15, 2007 at 11:42:15AM -0700, Gerrit Huizenga wrote:
>
> Andrew mentioned a mechanism for adding a subsystem tag or other tag
> which helps disambiguate the message, either in the message file or in
> the end user documentation (e.g. the Message Pedia/mPedia that the Japanese
> have already created with ~350 messages, and a total of ~700 targetted
> by the end of the year).
>
> That tag could be appended to the beginning of the printk, to the end of
> the printk, or even in a formatted comment at the end of the printk that
> the tool could extract.

The best way to maintain such a subsystem tag most be supported by
kbuild. So say we have the subsystem "scsi" then we can set this subsystem
tag in drivers/scsi/Makefile and let drivers/scsi/arm/Makefile inherit
this vaule doing nothing.

Addign the subsystem tag to the printk could than be done using the
C preprocessor and we are all fine.

If we go by the kernel-doc way to document kernel messages (the few that
actually needs additional explanation) then we need to teach kernel-doc
to collect the same subsystem tag but that should be doable.

Sam

2007-06-18 13:16:18

by Arjan van de Ven

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Mon, 2007-06-18 at 14:55 +0200, holzheu wrote:
> Hi Gerrit,
>
> The common thing of your and our approach is, that we need an ID to
> identify a message either by:

Maybe I am missing something big, but why is an ID needed?
The message IS the ID right? That's the only thing that is robust
against code moving about....

(And yes I am aware that very occasionally there might be a duplicate
message that REALLY is different, but I suspect that that is so
incredibly rare that it's just simpler to reword one of the two in such
case)

2007-06-18 13:19:55

by Jan Kara

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Mon 18-06-07 06:12:54, Arjan van de Ven wrote:
> On Mon, 2007-06-18 at 14:55 +0200, holzheu wrote:
> > Hi Gerrit,
> >
> > The common thing of your and our approach is, that we need an ID to
> > identify a message either by:
>
>
> Maybe I am missing something big, but why is an ID needed?
> The message IS the ID right? That's the only thing that is robust
> against code moving about....
I think the problem is messages contain device numbers, and all similar
kind of data. So what you'd really like is to use >format string< as ID.
But that is not easily accessible for the user, so you'd have to print it
with each message. And that's ... ugly IMHO.

> (And yes I am aware that very occasionally there might be a duplicate
> message that REALLY is different, but I suspect that that is so
> incredibly rare that it's just simpler to reword one of the two in such
> case)

Honza
--
Jan Kara <[email protected]>
SuSE CR Labs

2007-06-18 13:46:29

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

Hi Pavel,

On Fri, 2007-06-15 at 12:40 +0000, Pavel Machek wrote:
> Hi!
>
> > > Your proposal is similar to one I made to some Japanese developers
> > > earlier this year. I was more modest, proposing that we
> > >
> > > - add an enhanced printk
> > >
> > > xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
> > Maybe a stupid idea but why do we want to assign these numbers by hand?
> > I can imagine it could introduce collisions when merging tons of patches
> > with new messages... Wouldn't it be better to compute say, 8-byte hash
> > from the message and use it as it's identifier? We could do this
>
> Why hashes? Just key it off full message, and treat it like normal
> translation problem.
>
> 'lp%d: on fire' -> 'your printer failed basic tests, you should check
> cabling'.

Hmmm... What about extracting all format strings from the KMSG_DOC
macros and storing them in a file. E.g.:

printk("lp%d: on fire");

/**
* message
*
* Description:
* Your printer failed basic tests, you should check cabling
...

KMSG_DOC(lp, "lp%d: on fire");

Some build mechanism creates a file with all format strings of the
KMSG_DOC macros:

E.g.: cat kmsgs.txt

# MSG: "lp%d: on fire"
# COMPONENT: lp
# DESC: Your printer failed basic tests, you should check cabling

Then we provide a tool e.g. "kmsg", which gets as input the actual
kernel message and searches the message database for matching format
strings:

>> kmsg "lp0: on fire"
>> Your printer ...

Of course, we can get duplicate hits, too.

The advantage of that approach is, that we don't need additional message
identifiers like hashes.

Michael

2007-06-18 13:57:53

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Mon, 2007-06-18 at 06:12 -0700, Arjan van de Ven wrote:
> On Mon, 2007-06-18 at 14:55 +0200, holzheu wrote:
> > Hi Gerrit,
> >
> > The common thing of your and our approach is, that we need an ID to
> > identify a message either by:
>
>
> Maybe I am missing something big, but why is an ID needed?
> The message IS the ID right? That's the only thing that is robust
> against code moving about....

Yes. As already discussed with Pavel, it is one option to use the format
string of the message as message ID. The disadvantage compared to
message IDs like hashes is, that format strings might be even less
unique than hashes and it's probably less convenient for searching by
operators.

An operator has to issue either:

>> info lp.4711
or
>> info "lp0: on fire"

Michael

2007-06-18 14:02:32

by Pavel Machek

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

Hi!

> > 'lp%d: on fire' -> 'your printer failed basic tests, you should check
> > cabling'.
>
> Hmmm... What about extracting all format strings from the KMSG_DOC
> macros and storing them in a file. E.g.:
>
> printk("lp%d: on fire");
>
> /**
> * message
> *
> * Description:
> * Your printer failed basic tests, you should check cabling
> ...
>
> KMSG_DOC(lp, "lp%d: on fire");

Yes, that was the idea. But we have already tools and standard format
for this (gettext), we should just use that.
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

2007-06-18 15:41:18

by Randy Dunlap

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Mon, 18 Jun 2007 15:11:57 +0200 Sam Ravnborg wrote:

> On Fri, Jun 15, 2007 at 11:42:15AM -0700, Gerrit Huizenga wrote:
> >
> > Andrew mentioned a mechanism for adding a subsystem tag or other tag
> > which helps disambiguate the message, either in the message file or in
> > the end user documentation (e.g. the Message Pedia/mPedia that the Japanese
> > have already created with ~350 messages, and a total of ~700 targetted
> > by the end of the year).
> >
> > That tag could be appended to the beginning of the printk, to the end of
> > the printk, or even in a formatted comment at the end of the printk that
> > the tool could extract.
>
> The best way to maintain such a subsystem tag most be supported by
> kbuild. So say we have the subsystem "scsi" then we can set this subsystem
> tag in drivers/scsi/Makefile and let drivers/scsi/arm/Makefile inherit
> this vaule doing nothing.
>
> Addign the subsystem tag to the printk could than be done using the
> C preprocessor and we are all fine.
>
> If we go by the kernel-doc way to document kernel messages (the few that
> actually needs additional explanation) then we need to teach kernel-doc
> to collect the same subsystem tag but that should be doable.

Actually I would prefer that such a tool be a clone of
scripts/kernel-doc and use a (slightly?) modified tag instead of
the kernel-doc "/**" tag if this route is used at all.

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

2007-06-19 00:11:26

by Tim Bird

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

Gerrit Huizenga wrote:
> Further, yet another kernel config option could allow distros to output
> the calculated MD5 sum to be printed, much like we do with timestamps
> today.

> Comments?

Would the compiled-in text then also become replaceable?
Or is the MD5 sum output expected to be in addition to
the regular English message?

If message replacement at compile-time is supported, this
could allow for creating "short" versions of the messages,
which could have a beneficial impact on kernel size.

Right now, it is possible to completely disable printks
and re-claim about 100K of memory. However, in some
embedded configurations, even if you are space-constrained
it's desirable to retain some of the printks, for in-field
debugging. Thus not very many embedded developers
disable printks completely, even though the option has
been supported for a while. (That, and many aren't caught
up to the kernel version where it was introduced (2.6.10) :-)

But compressed messages (shortened text through abbreviations,
or just outputting the ID alone, etc.) could save SOME
of the space, in trade for less readability. Heck, just
removing all vowels would probably save 10k, and not
hurt readability that much.

Finally, for testing, it's handy to also
have automatic translation generators.
At a former company I worked for, they had translators
that would output:
* all messages shortened by 20%
* all messages lengthened by 20%
* every message converted to pig-latin

These were mostly used for testing if the strings broke
screen real-estate constraints (which don't apply to
kernel messages). But the automatic translators
would sometimes catch messages with weird attributes.

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

2007-06-19 01:56:17

by Arjan van de Ven

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Mon, 2007-06-18 at 15:53 +0200, holzheu wrote:
> On Mon, 2007-06-18 at 06:12 -0700, Arjan van de Ven wrote:
> > On Mon, 2007-06-18 at 14:55 +0200, holzheu wrote:
> > > Hi Gerrit,
> > >
> > > The common thing of your and our approach is, that we need an ID to
> > > identify a message either by:
> >
> >
> > Maybe I am missing something big, but why is an ID needed?
> > The message IS the ID right? That's the only thing that is robust
> > against code moving about....
>
> Yes. As already discussed with Pavel, it is one option to use the format
> string of the message as message ID. The disadvantage compared to
> message IDs like hashes is, that format strings might be even less
> unique than hashes

if the hash comes from the string in the first place I have a hard time
believing that.

> and it's probably less convenient for searching by
> operators.

I'm not convinced of that...

>
> An operator has to issue either:
>
> >> info lp.4711
> or
> >> info "lp0: on fire"

well.... surely the messages are caught by some userspace program,
right? (like syslog).. that can do the lookup and make it all
conveniently lookup-able and cross-referencable etc etc....

--
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

2007-06-19 03:52:46

by Gerrit Huizenga

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Mon, 18 Jun 2007 17:13:19 PDT, Tim Bird wrote:
> Gerrit Huizenga wrote:
> > Further, yet another kernel config option could allow distros to output
> > the calculated MD5 sum to be printed, much like we do with timestamps
> > today.
>
> > Comments?
>
> Would the compiled-in text then also become replaceable?
> Or is the MD5 sum output expected to be in addition to
> the regular English message?

The MD5 sum would be calculated at print time, no proposal in
here to replace the in-kernel text. And, I'm not sure that replacing
it with an MD5 sum would every be significantly shorter, ergo
I don't think this helps your problem.

The methods of post-processing to "shrink" the kernel text here
seem too ugly to ponder. And I just pondered a few that made my
head hurt (sort the MD5 sums, re-insert the number of the MD5 sum
as an integer instead of the original text, and, beyond being gross,
what do you do with the formatting info about the args?).

> If message replacement at compile-time is supported, this
> could allow for creating "short" versions of the messages,
> which could have a beneficial impact on kernel size.
>
> Right now, it is possible to completely disable printks
> and re-claim about 100K of memory. However, in some
> embedded configurations, even if you are space-constrained
> it's desirable to retain some of the printks, for in-field
> debugging. Thus not very many embedded developers
> disable printks completely, even though the option has
> been supported for a while. (That, and many aren't caught
> up to the kernel version where it was introduced (2.6.10) :-)

Which ones matter? Odds are you could use the KERNEL_LOGLEVEL or
such to mark those, then compile out the rest.

> But compressed messages (shortened text through abbreviations,
> or just outputting the ID alone, etc.) could save SOME
> of the space, in trade for less readability. Heck, just
> removing all vowels would probably save 10k, and not
> hurt readability that much.

Ick. Are you serious? How about running them through a valley
girl filter and then converting to high school text messaging?

> Finally, for testing, it's handy to also
> have automatic translation generators.
> At a former company I worked for, they had translators
> that would output:
> * all messages shortened by 20%
> * all messages lengthened by 20%
> * every message converted to pig-latin

Double yucko.

> These were mostly used for testing if the strings broke
> screen real-estate constraints (which don't apply to
> kernel messages). But the automatic translators
> would sometimes catch messages with weird attributes.

I don't think people are worried about the correctness of
the messages and message formats - much of that can actually
be detected by simple tools. The goal here was to at least
allow people that support operating systems and for whom
English (and English collation sequences) is not a primary
language do some initial system diagosis.

I think compiling out the messages of something below some LOGLEVEL
is a lot more practical.

gerrit

2007-06-19 06:08:19

by Kunai, Takashi

[permalink] [raw]

Subject: Re: [Lf_kernel_messages] Re: [RFC/PATCH] Documentation of kernel messages

Hi Michael,

I initiated this discussion in The Linux Foundation Summit@Google Campus
last week from the viewpoint of requirements by Japanese vendors/users.

As for a file to store message descriptions, I perfectly agree with your
analysis and we also noticed some (+) and (-) you pointed out in the
meeting last week. Our conclusion was to have a file outside of the
kernel source codes to lessen developers burdens and to encourage better
documentations of descriptions (as Andrew mentioned in his initial message).

Takashi Kunai,
The Linux Foundation Japan

holzheu wrote:
> Hi Gerrit,
>
> The common thing of your and our approach is, that we need an ID to
> identify a message either by:
>
> * Using hashes + component name (maybe)
> * Or using hand-made message numbers + component name. Numbers have
> to be unique within the kernel component
>
> I think the main difference of your approach is, that documentation will
> be maintained outside the kernel. This has advantages, but also
> disadvantages:
>
> (+) Developers have less work to do :-)
> (+) Kernel sources and patches have less lines of code, since message
> descriptions are not contained in the kernel sources.
> (+) You can support multiple languages
>
> (-) Hard to keep printks and descriptions up-to-date. If using hashes,
> each typo fix in a printk breaks the link to your documentation.
> (-) Since the developer knows the meaning of a printk best, he probably
> would be the right person to write the description.
> (-) For every Distribution or vanilla kernel you probably need separate
> external message databases.
>
> I like the fact, that every printk gets an ID in your approach and you
> do not need additional printk macros.
>
> Maybe both approaches are not mutually exclusive. If developers would
> like to document their messages within the kernel sources, they still
> could use kernel-doc and our KMSG_DOC macro do do that.
>
> I admit, that I am not sure about the the best solution here.
>
> Michael
>
>
> _______________________________________________
> Lf_kernel_messages mailing list
> [email protected]
> https://lists.linux-foundation.org/mailman/listinfo/lf_kernel_messages
>
>

--
Kunai, Takashi
The Linux Foundation Japan (New since '07/2/5)
Shibuya Mark City West 22nd Floor
1-12-1 Dogenzaka,Shibuya-ku,Tokyo,Japan
zip150-0043 tel+81-3-4360-5493

2007-06-19 08:55:27

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Mon, 2007-06-18 at 18:36 -0700, Arjan van de Ven wrote:
> On Mon, 2007-06-18 at 15:53 +0200, holzheu wrote:
> > On Mon, 2007-06-18 at 06:12 -0700, Arjan van de Ven wrote:
> > > On Mon, 2007-06-18 at 14:55 +0200, holzheu wrote:
> > > > Hi Gerrit,
> > > >
> > > > The common thing of your and our approach is, that we need an ID to
> > > > identify a message either by:
> > >
> > >
> > > Maybe I am missing something big, but why is an ID needed?
> > > The message IS the ID right? That's the only thing that is robust
> > > against code moving about....
> >
> > Yes. As already discussed with Pavel, it is one option to use the format
> > string of the message as message ID. The disadvantage compared to
> > message IDs like hashes is, that format strings might be even less
> > unique than hashes
>
> if the hash comes from the string in the first place I have a hard time
> believing that.
>

Just think of all messages containing %s, which can expand to every
possible string.

Michael

2007-06-19 11:35:27

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Mon, 2007-06-18 at 18:36 -0700, Arjan van de Ven wrote:

<snip>

> well.... surely the messages are caught by some userspace program,
> right? (like syslog).. that can do the lookup and make it all
> conveniently lookup-able and cross-referencable etc etc....

Ok, I agree. Maybe that's really a good idea!

So what about the following proposal:

* Use printk format strings as message IDs
* Use kernel-doc (or similar mechanism) for message documentation
* Use Subsystem tag/component ID for each printk. This might be
optional, but would make sense to get messages more unique.
* Implement a new tool (kmsg) to allow operators
to access kernel message descriptions in a convenient way.

The kmsg tool will read the dmsg buffer together with the list of
documented printks, which has been created in the kernel build using
the format strings defined in the KMSG_DOC macros. It matches the known
format strings to the dmesg messages and informs the operator, for
which message he can get a description.

Something like:

>> kmsg
Linux version 2.6.9-42.0.3.ELsmp...
....
[<ID>] lp0 on fire

>> kmsg <ID>
Desciption: your printer failed basic tests, you should check cabling

The <ID> is generated by the kmsg tool and could be e.g. the hashes for
the matching format strings.

Michael

2007-06-19 19:28:36

by Arjan van de Ven

[permalink] [raw]

Subject: Re: [RFC/PATCH] Documentation of kernel messages

On Tue, 2007-06-19 at 10:51 +0200, holzheu wrote:
> On Mon, 2007-06-18 at 18:36 -0700, Arjan van de Ven wrote:
> > On Mon, 2007-06-18 at 15:53 +0200, holzheu wrote:
> > > On Mon, 2007-06-18 at 06:12 -0700, Arjan van de Ven wrote:
> > > > On Mon, 2007-06-18 at 14:55 +0200, holzheu wrote:
> > > > > Hi Gerrit,
> > > > >
> > > > > The common thing of your and our approach is, that we need an ID to
> > > > > identify a message either by:
> > > >
> > > >
> > > > Maybe I am missing something big, but why is an ID needed?
> > > > The message IS the ID right? That's the only thing that is robust
> > > > against code moving about....
> > >
> > > Yes. As already discussed with Pavel, it is one option to use the format
> > > string of the message as message ID. The disadvantage compared to
> > > message IDs like hashes is, that format strings might be even less
> > > unique than hashes
> >
> > if the hash comes from the string in the first place I have a hard time
> > believing that.
> >
>
> Just think of all messages containing %s, which can expand to every
> possible string.

of course you hash on the string before you do the format stuff....

--
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

2007-06-25 13:48:08

[permalink] [raw]

Subject: Documentation of kernel messages (Summary)

Hi all,

Any idea, how to proceed with this topic? Do you think that any of the
suggested solutions for documentation / translation of kernel messages
will have a chance to be included in the kernel?

I tried to summarize the outcome of this discussion...

There are two main issues:

* Translate messages
* Document messages

For both, we have to give messages identifiers in order to match them
to the corresponding documentation / translation. Three possible
solutions have been suggested:

1. Use message numbers maintained by driver owners
2. Automatically create hashes for printks
3. Use printk format string as message identifier

Regarding the question where to put documentation and translations we
have two options:

* In the kernel tree
* Somewhere outside the kernel tree

Another issue of the discussion was the usage of the component name for
printks.

Useful comments:
================
Andrew Morton:
Suggested an alternative approach using a new printk function with
message ID parameter. Documentation / Translation should be done
outside the kernel.

Gerrit Huizenga:
Suggested an alternative approach using hashes as message IDs.

Greg Kroah-Hartman:
He pointed out, that we need a solution which integrates dev_xxx()
macros.

Randy Dunlop:
He doesn't want to have message documentation included in the
kernel-doc tool. We should create new tool for kernel messages.

Pavel Machek:
Suggested to use gettext for message documentation / translation.

Useful links:
=============
There already exists a mailing list for the kernel messages topic:
https://lists.linux-foundation.org/mailman/listinfo/lf_kernel_messages

There was an LWN article summarizing some aspects of the discussion:
http://lwn.net/Articles/238948/

Pros and Cons of the different Message ID methods:
==================================================
Message numbers maintained by driver owners:
+ Unique IDs
+ Can be kept stable between different kernel versions
+ Efficient way of matching printk to Documentation or translation
- Difficult to maintain

printk hashes:
+ Easy to maintain
+ Efficient way of matching printk to Documentation or translation
- Additional preprocessor step needed
- No unique IDs
- Ugly in messages, since hashes can have many characters
- Volatile, since typo fixes change hash values

Format strings:
+ No change needed for printks
+ Matching of printk to corresponding documentation or translation is
not as efficient as using hashes
- No unique IDs

Pros and Cons of external documentation (vs. internal documentation):
====================================================================
+ Developers have less work to do
+ Kernel sources and patches have less lines of code, since message
descriptions are not contained in the kernel sources.
+ You can support multiple languages

- Hard to keep printks and descriptions up-to-date. If using hashes,
each typo fix in a printk breaks the link to your documentation.
- Since the developer knows the meaning of a printk best, he probably
would be the right person to write the description.
- For every Distribution or vanilla kernel you probably need separate
external message databases.

Pros and Cons of using component names in printks:
==================================================
+ Makes it easier to identify source of message
+ Makes message more unique (especially, when using Hashes or format
strings as message IDs)
- Changes of printks necessary

Implementation issues (depending on what solution we implement):
===============================================================
* Preprocessor tool for hashes
* Check tool for messages/message description
* Tool to extract kernel documentation from kernel tree
* Sysadmin tool to query message descriptions
* New kernel macros for Messages to support Component names / message
numbers

2007-06-25 15:44:55

[permalink] [raw]

Subject: Re: Documentation of kernel messages (Summary)

On Monday 25 June 2007 09:48:41 Michael Holzheu wrote:
> Hi all,
>
> Any idea, how to proceed with this topic? Do you think that any of the
> suggested solutions for documentation / translation of kernel messages
> will have a chance to be included in the kernel?

Personally? No to the second question, which renders the first "do it
yourself outside of the tree".

Just a guess, and I don't speak for anyone else here, but I think most of us
are waiting to see how long it takes you to lose interest.

> I tried to summarize the outcome of this discussion...
>
> There are two main issues:
>
> * Translate messages
> * Document messages

I'm somewhat sympathetic to translation (although I tend to think of it as a
distro's job, and still wonder why IBM is pushing this directly rather than
trying to bounce it off Red Hat or somebody). But you could put a sed filter
around dmesg to give you swahili output entirely in userspace. Have you
shown the simple approach to be insufficient yet?

Is "document messages" the same problem as "translate messages", or are these
two different problems conflated together?

I find "document messages" to be a horrible idea conceptually, because I think
the messages should _be_ documentation. If the message is unclear it should
be clarified. "There's too much garbage on the floor, we should laminate it
so we can't smell it anymore." Er, no...

> Three possible solutions have been suggested:
>
> 1. Use message numbers maintained by driver owners
> 2. Automatically create hashes for printks
> 3. Use printk format string as message identifier

#2 has the advantage that you can do it without buy-in from the community, and
without touching the code. (Of course it doesn't necessarily help you if the
printk string is "%s:%s".)

But I'm still convinced you could tackle over 90% of the problem from
userspace, without touching the kernel.

> Pros and Cons of the different Message ID methods:
> ==================================================
> Message numbers maintained by driver owners:
> + Unique IDs
> + Can be kept stable between different kernel versions
> + Efficient way of matching printk to Documentation or translation
> - Difficult to maintain

Will at best only ever be done for some drivers, not all. But that's probably
inescapable no matter what your approach is.

> printk hashes:
> - Additional preprocessor step needed

Only for you guys. Those of us building our own kernels and NOT translating
them or producing "the big book of explanations of kernel messages for IBM
customers who don't understand them" shouldn't have to run this preprocessor
step.

> Pros and Cons of external documentation (vs. internal documentation):
> ====================================================================
> + Developers have less work to do
> + Kernel sources and patches have less lines of code, since message
> descriptions are not contained in the kernel sources.
> + You can support multiple languages
>
> - Hard to keep printks and descriptions up-to-date. If using hashes,
> each typo fix in a printk breaks the link to your documentation.
> - Since the developer knows the meaning of a printk best, he probably
> would be the right person to write the description.

Imposing additional requirements on developers isn't going to work unless you
reject patches from developers who don't follow your procedures and submit
the appropriately filled-out forms with each patch for processing by the
central bureaucracy.

Signed-off-by works because A) Linus enforces it, B) The overhead's very low,
C) attribution is already highly good thing within our community values, D)
it annoys SCO.

The javadoc function comments in the source don't have complete coverage and
probably never will (although they're getting better). The premise of doing
something similar for messages is that a translating dmesg in userspace can't
necessarily get complete coverage, so you need a solution modeled on
something that doesn't have complete covereage years after its
introduction...

> - For every Distribution or vanilla kernel you probably need separate
> external message databases.

"If you're resigned to doing that anyway you don't actually need him to lie
there all the time, do you?" - Ford Prefect

If you need an external message database what's wrong with doing it entirely
in userspace via a sed-based translating dmesg?

Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.

2007-06-25 18:05:56

by Sam Ravnborg

[permalink] [raw]

Subject: Re: Documentation of kernel messages (Summary)

> I find "document messages" to be a horrible idea conceptually, because I think
> the messages should _be_ documentation. If the message is unclear it should
> be clarified. "There's too much garbage on the floor, we should laminate it
> so we can't smell it anymore." Er, no...

"Document message" is for the cases where the message itself does not convey
enough info.
No need to add more text when the initial message is good enough.

Sam

2007-06-27 15:10:54

[permalink] [raw]

Subject: Re: Documentation of kernel messages (Summary)

On Mon, 2007-06-25 at 11:44 -0400, Rob Landley wrote:
> On Monday 25 June 2007 09:48:41 Michael Holzheu wrote:
> > Hi all,
> >
> > Any idea, how to proceed with this topic? Do you think that any of the
> > suggested solutions for documentation / translation of kernel messages
> > will have a chance to be included in the kernel?
>
> Personally? No to the second question, which renders the first "do it
> yourself outside of the tree".

If that is the opinion of the majority here, fine. If there is no hard
rule on how to define printk macros, one option for us would be to
define some new s390 specific printk macros for our device drivers.
Similar to hundreds of other driver specific printk macros in the
kernel.

> Just a guess, and I don't speak for anyone else here, but I think most of us
> are waiting to see how long it takes you to lose interest.

:-) Work is not always fun. Sometimes it is just a duty.

Michael

2007-08-03 18:06:21

by Randy Dunlap

[permalink] [raw]

Subject: Re: Documentation of kernel messages (Summary)

On Mon, 16 Jul 2007 15:53:06 -0400 Rob Landley wrote:

> On Sunday 15 July 2007 12:28:06 pm Randy Dunlap wrote:
> > On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
> > > On Friday 13 July 2007 11:54:41 pm Randy Dunlap wrote:
> > > > > If there's interest, I can push some patches to clean up
> > > > > Documentation by moving files into subdirectories, but
> > > > > Documentation's not well-suited to link out to the web. (You need
> > > > > html for that, and it's text.)
> > > >
> > > > I think that you should start moving Documentation/ files around
> > > > and adding subdirectories -- if you are pretty sure of where they
> > > > should go (i.e., they won't likely be moved again later on).
> > >
> > > You mean like these?
> > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
> > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html
> >
> > Yes. Have any of these been merged?
>
> Not that I know of. They seemed to meet with resounding indifference, so I
> went on to other things...

You need to be persistent. Please re-submit this.

and since it's just file & dir moves and done via git, it should go to
Linus. However, he may not want it right now since the primary
merge window for 2.6.23 has closed (although he has been merging
some other doc updates recently).

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

2007-08-03 19:34:05

[permalink] [raw]

Subject: Re: Documentation of kernel messages (Summary)

On Friday 03 August 2007 1:11:55 pm Randy Dunlap wrote:
> On Mon, 16 Jul 2007 15:53:06 -0400 Rob Landley wrote:
> > On Sunday 15 July 2007 12:28:06 pm Randy Dunlap wrote:
> > > On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
> > > > On Friday 13 July 2007 11:54:41 pm Randy Dunlap wrote:
> > > > > > If there's interest, I can push some patches to clean up
> > > > > > Documentation by moving files into subdirectories, but
> > > > > > Documentation's not well-suited to link out to the web. (You
> > > > > > need html for that, and it's text.)
> > > > >
> > > > > I think that you should start moving Documentation/ files around
> > > > > and adding subdirectories -- if you are pretty sure of where they
> > > > > should go (i.e., they won't likely be moved again later on).
> > > >
> > > > You mean like these?
> > > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
> > > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html
> > >
> > > Yes. Have any of these been merged?
> >
> > Not that I know of. They seemed to meet with resounding indifference, so
> > I went on to other things...
>
> You need to be persistent. Please re-submit this.

Greg KH thinks it's a good idea to add language directories to the top level
of Documentation, and there are slightly more than two of those:
http://www.translatorscafe.com/cafe/Help.asp?HelpID=59

Since you think it's worth doing, I'll resubmit the work I've already done
here, but I no longer think trying to shovel Documentation into some
semblance of order against the will of people like Greg is a good use of
time.

These days I'm trying to create an html index that links into Documentation in
a coherent order (with categories and everything), and using automated tools
to detect files that aren't linked to, or links that point to a file that
isn't there anymore. This is obviously still a work in progress, but I think
it's a better approach.

> and since it's just file & dir moves and done via git, it should go to
> Linus. However, he may not want it right now since the primary
> merge window for 2.6.23 has closed (although he has been merging
> some other doc updates recently).

If moving text files from one location to another in the Documentation
directory breaks the build or causes bugs in the kernel, we have bigger
problems. :)

Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.

2007-08-04 03:49:55

[permalink] [raw]

Subject: Re: Documentation of kernel messages (Summary)

On Fri, Aug 03, 2007 at 03:32:04PM -0400, Rob Landley wrote:
> On Friday 03 August 2007 1:11:55 pm Randy Dunlap wrote:
> > On Mon, 16 Jul 2007 15:53:06 -0400 Rob Landley wrote:
> > > On Sunday 15 July 2007 12:28:06 pm Randy Dunlap wrote:
> > > > On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
> > > > > On Friday 13 July 2007 11:54:41 pm Randy Dunlap wrote:
> > > > > > > If there's interest, I can push some patches to clean up
> > > > > > > Documentation by moving files into subdirectories, but
> > > > > > > Documentation's not well-suited to link out to the web. (You
> > > > > > > need html for that, and it's text.)
> > > > > >
> > > > > > I think that you should start moving Documentation/ files around
> > > > > > and adding subdirectories -- if you are pretty sure of where they
> > > > > > should go (i.e., they won't likely be moved again later on).
> > > > >
> > > > > You mean like these?
> > > > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
> > > > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html
> > > >
> > > > Yes. Have any of these been merged?
> > >
> > > Not that I know of. They seemed to meet with resounding indifference, so
> > > I went on to other things...
> >
> > You need to be persistent. Please re-submit this.
>
> Greg KH thinks it's a good idea to add language directories to the top level
> of Documentation, and there are slightly more than two of those:
> http://www.translatorscafe.com/cafe/Help.asp?HelpID=59
>
> Since you think it's worth doing, I'll resubmit the work I've already done
> here, but I no longer think trying to shovel Documentation into some
> semblance of order against the will of people like Greg is a good use of
> time.

My "will" is only that some of these documents be translated _and_ put
in the main kernel source tree. I really have no opinion on where they
need to go within that directory. If you have a better place for them,
please let me know.

thanks,

greg k-h

2007-08-04 03:50:24

[permalink] [raw]

Subject: Re: Documentation of kernel messages (Summary)

On Fri, Aug 03, 2007 at 03:32:04PM -0400, Rob Landley wrote:
>
> These days I'm trying to create an html index that links into Documentation in
> a coherent order (with categories and everything), and using automated tools
> to detect files that aren't linked to, or links that point to a file that
> isn't there anymore. This is obviously still a work in progress, but I think
> it's a better approach.

Better than cleaning up what we have in the kernel source tree? Why not
work on that first, then the "automated" type stuff would be much easier
to do later, right?

thanks,

greg k-h

2007-08-04 18:54:33

[permalink] [raw]

Subject: Re: Documentation of kernel messages (Summary)

On Friday 03 August 2007 10:52:06 pm Greg KH wrote:
> On Fri, Aug 03, 2007 at 03:32:04PM -0400, Rob Landley wrote:
> > These days I'm trying to create an html index that links into
> > Documentation in a coherent order (with categories and everything), and
> > using automated tools to detect files that aren't linked to, or links
> > that point to a file that isn't there anymore. This is obviously still a
> > work in progress, but I think it's a better approach.
>
> Better than cleaning up what we have in the kernel source tree?

Yes, I just said that.

> Why not work on that first, then the "automated" type stuff would be much
> easier to do later, right?

How does an automated 404 checker that identifies files nothing's linking to
get easier if the source files are in a different order? They could be
alphabetical in one big directory and that would work the same.

Moving Documentation around is pointless churn that does nothing to prevent
you from adding language directories to the top level on a whim, as if there
were only two instead of (as I pointed out last message), several dozen.
(While Randy Dunlap's poking me to resubmit my patch to group the
architecture documentation into an architecture subdirectory, you're adding
language directories to the top level instead of in their own subdirectory.)

Documentation doesn't even cross-link to the output of make htmldocs (which
has its own structure imposed on it due to being extracted from the kernel
sources). The kernel tarball has _two_ documentation sources that don't
significantly cross-reference each other, and Rusty just submitted "make
Preparation!" for lguest that's totally unrelated to either of them (and
starts from a README file buried in the source code). None of this links to
the menuconfig help entries, and the only reference in Documentation/
to "make help" is in Documentation/kbuild/makefiles.txt which explains that
its purpose is to list the available architecture targets (something it does
not, in my experience, actually do).

The idea that the kernel Documentation directory is the master repository of
kernel documentation is an unworkable fantasy. The Documentation directory
cannot index for all the kernel documentation resources out on the web,
because it's in text not html. Documentation was created on the assumption
that it would contain all interesting resources, as text files, but that
doesn't match reality. Documentation is merely one resource among many, and
to link _out_ you need HTML.

Some of the resources out there are organized chronologically, such as the
Linux Journal archives http://www.linuxjournal.com/xstatic/magazine/archives,
the Ottawa Linux Symposium papers (http://kernel.org/doc/ols), or the
kernel-traffic archives
http://www.kernel-traffic.org/kernel-traffic/archives.html. Some have their
own indexes, such as the Linux Weekly News kernel articles
http://lwn.net/Kernel/Index/ and the Linux Device Drivers book
http://lwn.net/Kernel/LDD3/. Some are random bits picked out of developer
blogs, found with google, reasonably coherent articles on wikipedia. Some
are huge self-contained lumps on specific topics such as Mel Gorman's mm
documentation or lots of the embedded stuff.

So no matter what reorganization I do to Documentation, its structure (or lack
thereof) is incidental to coherently indexing most of the kernel
documentation that's out there. (Right now the best index is "google" but
that's only useful if you know what questions to ask. But getting up-to-date
versions of Documentation and the output of make htmldocs on the web lets
google find it. (Last year, back when I was still working on BusyBox, I did
a google trawl for ext2 documentation to replace the horrible mke2fs busybox
used to have, and the first three pages of hits did _NOT_ include a copy of
Documentation/filesystems/ext2.txt. Ironically, if you google for "ext2
documentation" today the sixth hit is the unfinished ext2 documentation I'd
just started to write before I found Documentation/filesystems/ext2.txt.)

Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.

2007-08-04 19:02:36

[permalink] [raw]

Subject: Re: Documentation of kernel messages (Summary)

On Friday 03 August 2007 10:50:38 pm Greg KH wrote:
> On Fri, Aug 03, 2007 at 03:32:04PM -0400, Rob Landley wrote:
> > On Friday 03 August 2007 1:11:55 pm Randy Dunlap wrote:
> > > On Mon, 16 Jul 2007 15:53:06 -0400 Rob Landley wrote:
> > > > On Sunday 15 July 2007 12:28:06 pm Randy Dunlap wrote:
> > > > > On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
> > > > > > On Friday 13 July 2007 11:54:41 pm Randy Dunlap wrote:
> > > > > > > > If there's interest, I can push some patches to clean up
> > > > > > > > Documentation by moving files into subdirectories, but
> > > > > > > > Documentation's not well-suited to link out to the web. (You
> > > > > > > > need html for that, and it's text.)
> > > > > > >
> > > > > > > I think that you should start moving Documentation/ files
> > > > > > > around and adding subdirectories -- if you are pretty sure of
> > > > > > > where they should go (i.e., they won't likely be moved again
> > > > > > > later on).
> > > > > >
> > > > > > You mean like these?
> > > > > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
> > > > > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html
> > > > >
> > > > > Yes. Have any of these been merged?
> > > >
> > > > Not that I know of. They seemed to meet with resounding
> > > > indifference, so I went on to other things...
> > >
> > > You need to be persistent. Please re-submit this.
> >
> > Greg KH thinks it's a good idea to add language directories to the top
> > level of Documentation, and there are slightly more than two of those:
> > http://www.translatorscafe.com/cafe/Help.asp?HelpID=59
> >
> > Since you think it's worth doing, I'll resubmit the work I've already
> > done here, but I no longer think trying to shovel Documentation into some
> > semblance of order against the will of people like Greg is a good use of
> > time.
>
> My "will" is only that some of these documents be translated _and_ put
> in the main kernel source tree. I really have no opinion on where they
> need to go within that directory. If you have a better place for them,
> please let me know.

A common subdirectory. I'd probably put it in "Documentation/translated" with
a README at the top level of that explaining what it's for.

I can push a patch to move them, but reorganizing Documentation doesn't turn
it into an index that can link out resources outside itself. If reorganizing
it to be less of a compost heap is a separate goal in and of itself, I can do
that...

Rob
--
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson.

2007-08-04 20:11:12

by Stefan Richter

[permalink] [raw]

Subject: Re: Documentation of kernel messages (Summary)

Rob Landley wrote:
> Documentation is merely one resource among many, and
> to link _out_ you need HTML.

Do you suggest HTML files in linux/Documentation? I think we can follow
URIs and other references in plaintext files just fine.

(People who want clickable links in plaintext files can use plaintext
viewers which implement this, or utilities or browser plugins which
follow URLs in the clipboard.)
--
Stefan Richter
-=====-=-=== =--- --=--
http://arcgraph.de/sr/

2007-08-06 15:36:32