LinuxLists.cc - patch for common networking error messages

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Janice M Girouard <[email protected]>
Date: Mon, 16 Jun 2003 15:30:22 -0500

EMSG_NET_LINK_UP "%s: state change: link up, %d Mbps, %s-duplex\n"

Should indicate flow control state too.

EMSG_NET_START_QUEUE "%s: performance event: (re)starting netdev queue\n"
EMSG_NET_STOP_QUEUE "%s: performance event: stopping netdev queue\n"

Oh _ABSOLUTELY NOT_, you're not printing a message
for normal events like this. Especially those that are
going to occur on highly loaded systems.

2003-06-16 20:39:52

by Andi Kleen

[permalink] [raw]

Subject: Re: patch for common networking error messages

On Mon, Jun 16, 2003 at 01:38:41PM -0700, David S. Miller wrote:
> From: Janice M Girouard <[email protected]>
> Date: Mon, 16 Jun 2003 15:30:22 -0500
>
> EMSG_NET_LINK_UP "%s: state change: link up, %d Mbps, %s-duplex\n"
>
> Should indicate flow control state too.

It would be actually useful to wrap these in real functions.

Why? It will make supporting netconsole easier which has to be careful
to never recurse in the network driver.

-Andi

2003-06-16 20:42:07

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Andi Kleen <[email protected]>
Date: Mon, 16 Jun 2003 22:53:42 +0200

Why? It will make supporting netconsole easier which has to be careful
to never recurse in the network driver.

printk can check this

2003-06-16 20:48:25

[permalink] [raw]

Subject: Re: patch for common networking error messages

I agree that it's not desirable to introduce a bunch of messages that we
aren't already logging. I didn't show the netif_msg prefix because I
was trying to focus the patch on the common messages, but you would
normally proceed a message with:

if netif_msg_link()
printk("some text to indicate the link is up/down")

The netif_msg_link test would normally filter out what messages should
be logged.

Or, just leave out the message call. I added one or two messages to the
tg3 and e1000 drivers to demonstrate where you might use these common
messages... just to show that various drivers could use the text.
Actually using the specific message would be completely up to the
developer.

Jaince

David S. Miller wrote:

> From: Janice M Girouard <[email protected]>
> Date: Mon, 16 Jun 2003 15:30:22 -0500
>
> EMSG_NET_LINK_UP "%s: state change: link up, %d Mbps, %s-duplex\n"
>
>Should indicate flow control state too.
>
> EMSG_NET_START_QUEUE "%s: performance event: (re)starting netdev queue\n"
> EMSG_NET_STOP_QUEUE "%s: performance event: stopping netdev queue\n"
>
>Oh _ABSOLUTELY NOT_, you're not printing a message
>for normal events like this. Especially those that are
>going to occur on highly loaded systems.
>
>
>

2003-06-16 22:00:49

[permalink] [raw]

Subject: Re: patch for common networking error messages

Janice M Girouard wrote:

> Below is a patch that demonstrates standard messages for ethernet device
> drivers. I would like your feedback on the concept of standard network
> messages, and any suggestions for messages to include.

Very useful! I'd like to see a short note on this in Documentation/
networking..Or perhaps if there is already a RAS best practices
kind of doc or something, add to that? (sorry, havent checked)
But it would be handy for people who wanted to contribute
patches for other drivers.

Essentially, things like some guidelines on classifying some
of those messages, when creating new messages. eg when is
something a state change and when is it a performance event?
I notice some slight ambiguity in your defs..(sorry, very
minor nitpick :)).

I'd certainly like to see messages from the driver when the
card enters/leaves promiscuous mode, as an example of things
we'd like to add...

thanks,
Nivedita

> The intent of the standard message change is to:
> 1) Ensure key events are communicated to user space in a predictable
> way, enabling automated diagnostic systems or error log analysis,
> 2) Reduce the number of puzzling messages that are logged -- in this
> case, by replacing them with standard messages, and/or
> 3) Identify the device (or driver name) that is responsible for the error.

2003-06-16 22:03:37

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Nivedita Singhvi <[email protected]>
Date: Mon, 16 Jun 2003 15:13:05 -0700

I'd certainly like to see messages from the driver when the
card enters/leaves promiscuous mode,

egrep "promiscuous mode" net/core/dev.c | grep printk

2003-06-16 22:12:42

by Andrew Morton

[permalink] [raw]

Subject: Re: patch for common networking error messages

"David S. Miller" <[email protected]> wrote:
>
> From: Andi Kleen <[email protected]>
> Date: Mon, 16 Jun 2003 22:53:42 +0200
>
> Why? It will make supporting netconsole easier which has to be careful
> to never recurse in the network driver.
>
> printk can check this

Actually it already does, to cover the case where an interrupt handler calls
printk while process-context code is performing a printk.

The nested printk will squirt the message into the log buffer and the
"outer" code will display it.

2003-06-16 22:15:47

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: "David S. Miller" <[email protected]>
Date : 06/16/2003 05:13 PM

egrep "promiscuous mode" net/core/dev.c | grep printk

I noticed when I performed the grep, the printk shows:
printk(KERN_INFO "device %s %s promiscuous mode\n"

For the sake of consistency and automatic error log analysis, it might be
nice to standardize on a message closer to:
printk(KERN_INFO "%s: %s promiscuous mode\n"

It's somewhat common, but not universal to start the error message with the
device name followed by a colon.

Janice

2003-06-16 22:18:16

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Janice Girouard <[email protected]>
Date: Mon, 16 Jun 2003 17:29:15 -0500

For the sake of consistency and automatic error log analysis, it might be

And all the scripts checking for the existing messages
in log files? Screw them, right?

This whole idea is starting to leave a very bad taste in
my mouth...

2003-06-16 22:32:28

[permalink] [raw]

Subject: Re: patch for common networking error messages

David S. Miller wrote:
> From: Janice Girouard <[email protected]>
> Date: Mon, 16 Jun 2003 17:29:15 -0500
>
> For the sake of consistency and automatic error log analysis, it might be
>
> And all the scripts checking for the existing messages
> in log files? Screw them, right?

Are you saying we never get to change any current
log messages ever again on accnt of the scripts that are
monitoring for those precise words? Hope not :)

I'd agree a lot of thought (and agreement :))has to go
into this before changing minor nits and stuff, and not
causing too much disruption..Evolution, as opposed to
revolution ;). I would hope that most wouldnt need changing..

thanks,
Nivedita

2003-06-16 22:34:36

[permalink] [raw]

Subject: Re: patch for common networking error messages

Janice M Girouard wrote:
> I agree that it's not desirable to introduce a bunch of messages that we
> aren't already logging. I didn't show the netif_msg prefix because I
> was trying to focus the patch on the common messages, but you would
> normally proceed a message with:
>
> if netif_msg_link()
> printk("some text to indicate the link is up/down")
>
> The netif_msg_link test would normally filter out what messages should
> be logged.

There are several issues at play here.

1) In general, I think you're approaching the logging from the wrong
angle. Start with netif_msg_xxx/NETIF_MSG_xxx first, and figure out the
logging API for those cases. These cover the majority of common cases,
and most are not specific to hardware at all. Starting at the driver
level and trying to move driver-specific messages into the upper layers
is the wrong direction, I feel.

2) If we are going to do major surgery on messages, make them more
computer-parseable at the same time. Human readable, since it must
above-all-else be kernel hacker readable, ... but computer parseable.

Here is an example. DISCLAIMER: No doubt there is a better format, it
is merely for illustration.
"%s: performance event: scatter/gather I/O disabled\n"
becomes
"dev=%s evt=perf sgio=disabled\n"

Basically a key-value format. Resist the urge to use numeric response
codes. For stuff like this, I think both Linus and the typical human
brain prefer English words to numeric response codes. This suggested
output is not unlike some arch's show-processor-state sysrq output.

3) _Somebody_ needs to do some "ground pounding", and figure out what
info sysadmins and users want to see. Event logging in general, so far,
seems to me more like a management checklist item than a real user
need... but I am quite willing to be proved wrong. Until we get
feedback along these lines, I tend to resist changes like this in
general. My initial read of your attached patch was that it was a long
of source churn, and I couldn't fathom what any user would gain from it all.

Jeff

There are a whole bunch of netif_msg_xxx and corresponding NETIF_MSG_xxx
bits. I don't see much need to change that I think getting the logging
API right for those would be an important first step.

Jeff

P.S. It is important to note the bits are laid out in increasing verbosity.

2003-06-16 22:39:27

[permalink] [raw]

Subject: Re: patch for common networking error messages

David S. Miller wrote:
> From: Nivedita Singhvi <[email protected]>
> Date: Mon, 16 Jun 2003 15:13:05 -0700
>
> I'd certainly like to see messages from the driver when the
> card enters/leaves promiscuous mode,
>
> egrep "promiscuous mode" net/core/dev.c | grep printk

Yeah, but dev_mc_upload() doesnt return any status ;).
(For those of us who distrust hw (Sorry Scott! :))).

But it was just my example, I assure you. I'm not
holding up a flag in the wind on this particular nit.
I do see positives in the feature as a whole though.
It would be a shame to get grounded for minor things..

thanks,
Nivedita

2003-06-16 22:37:09

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: David S. Miller <[email protected]>
Date:06/16/2003 05:27 PM

And all the scripts checking for the existing messages in log files?
Screw them, right?

That's a good point. One possible suggestion would be to submit more than
one stdmsgs.h files. One a legacy file, and one that is more consistent
from message to message.. shooting for a gradual migration.

Ultimately, I think standard messages would greatly support/simplify
scripts, especially between the myriad of ethernet drivers. Each one
reports the data slightly differently, so you're error log analysis needs
to recognize 100 or so ways of being told that the link just went down.

Janice

"David S. Miller"
<[email protected] To: Janice Girouard/Austin/IBM@IBMUS
> cc: Daniel Stekloff/Beaverton/IBM@IBMUS,
[email protected], [email protected],
06/16/2003 05:27 [email protected], Larry Kessler/Beaverton/IBM@IBMUS,
PM [email protected], [email protected],
[email protected]
Subject: Re: patch for common networking error messages

From: Janice Girouard <[email protected]>
Date: Mon, 16 Jun 2003 17:29:15 -0500

For the sake of consistency and automatic error log analysis, it might
be

And all the scripts checking for the existing messages
in log files? Screw them, right?

This whole idea is starting to leave a very bad taste in
my mouth...

2003-06-16 22:43:25

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Nivedita Singhvi <[email protected]>
Date: Mon, 16 Jun 2003 15:45:20 -0700

[ I removed this [email protected] from the CC:, it bounces... ]

I'd agree a lot of thought (and agreement :))has to go
into this before changing minor nits and stuff, and not
causing too much disruption..Evolution, as opposed to
revolution ;). I would hope that most wouldnt need changing..

There would be absolutely ZERO disruption if you guys would use you
brains and implement what you're actually trying to achieve, a system
event logging mechanism.

We have a message queueing mechanism using sockets, called netlink,
and you can make whatever actions in the kernel you think should be
monitored go and stuff messages into this system event netlink socket.

Then, you don't have to standardize a bunch of absolutely silly
strings (I mean, the concept is so incredibly stupid), you get events
that are in a precisely defined format going over this netlink socket.

Then whoever in userspace reads out the messages can interpret them
however the fuck it wants to. It is then trivial to parse the
messages and filter them. Furthermore, you could even transmit such
messages over a network connection to a remote logging server as-is.

And hey, look, for network links going up and down we have the hooks
already. Funny that...

2003-06-16 22:41:07

[permalink] [raw]

Subject: Re: patch for common networking error messages

Jeff Garzik wrote:
> 3) _Somebody_ needs to do some "ground pounding", and figure out what
> info sysadmins and users want to see. Event logging in general, so far,
> seems to me more like a management checklist item than a real user
> need... but I am quite willing to be proved wrong. Until we get
> feedback along these lines, I tend to resist changes like this in
> general. My initial read of your attached patch was that it was a long
> of source churn, and I couldn't fathom what any user would gain from it

make that "a lot of"

> There are a whole bunch of netif_msg_xxx and corresponding NETIF_MSG_xxx
> bits. I don't see much need to change that I think getting the logging
> API right for those would be an important first step.
>
> Jeff

arg :) I should fire my editor.

Jeff

2003-06-16 22:46:10

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Janice Girouard <[email protected]>
Date: Mon, 16 Jun 2003 17:50:08 -0500

One possible suggestion would be to submit more than one stdmsgs.h
files. One a legacy file, and one that is more consistent from
message to message.. shooting for a gradual migration.

Let me know when you're back on planet earth ok?

Standardizing strings is an absolutely FRUITLESS exercise.

If you want events, standardize events and push them over
a queueing based communications channel to userspace, namely
using netlink sockets.

2003-06-16 22:47:57

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Nivedita Singhvi <[email protected]>
Date: Mon, 16 Jun 2003 15:50:34 -0700

I do see positives in the feature as a whole though.

Would you design a network protocol this way? By passing
strings like "open connection please", "sure no problem"
back and forth between server and client?

Of course not.

So why are we even remotely considering the standardization
of _STRINGS_ for event reporting?

2003-06-16 23:03:25

by Donald Becker

[permalink] [raw]

Subject: Re: patch for common networking error messages

On Mon, 16 Jun 2003, Nivedita Singhvi wrote:
> Janice M Girouard wrote:
>
> > Below is a patch that demonstrates standard messages for ethernet device
> > drivers. I would like your feedback on the concept of standard network
> > messages, and any suggestions for messages to include.

There are many things that can go wrong, and most of them are very
dependent on the NIC's architecture. A few minutes of thought would
lead to the understanding that a enforced-common list of error/problem
messages is equivalent to a set of error counters. We have had that
since the initial Linux network hardware layer design. (Note: BSD of
that era had only a single "error" counter. )

> Essentially, things like some guidelines on classifying some
> of those messages, when creating new messages. eg when is
> something a state change and when is it a performance event?

The NETIF_MSG_* enable bits classify the status message types.

> I'd certainly like to see messages from the driver when the
> card enters/leaves promiscuous mode, as an example of things
> we'd like to add...

Most Ethernet drivers should already do this, right around the message
/* Unconditionally log net taps. */

> > 2) Reduce the number of puzzling messages that are logged -- in this
> > case, by replacing them with standard messages, and/or
> > 3) Identify the device (or driver name) that is responsible for the error.

The (physical) interface name, dev->name, should prefix every error
message. This is a recommendation, not enforced, so not every driver
does it.
[[[
Comment: Status/error messages are not an area to demonstrate
creativity, but all too frequently it becomes one.
]]]

--
Donald Becker [email protected]
Scyld Computing Corporation http://www.scyld.com
914 Bay Ridge Road, Suite 220 Scyld Beowulf cluster system
Annapolis MD 21403 410-990-9993

2003-06-16 23:54:14

[permalink] [raw]

Subject: Re: patch for common networking error messages

David S. Miller wrote:

> There would be absolutely ZERO disruption if you guys would use you
> brains and implement what you're actually trying to achieve, a system
> event logging mechanism.

> We have a message queueing mechanism using sockets, called netlink,
> and you can make whatever actions in the kernel you think should be
> monitored go and stuff messages into this system event netlink socket.

I should clarify here that I was speaking strictly for my lonesome sorry
self :), and have no knowledge of what the state of the various
RAS projects currently are, and the approaches they are trying..
For all I know, they may be currently trying precisely that..

Janice's patch is the first I've seen in this area (Luckily,
most of the time they keep me in a cave :) :)), and I do
appreciate *something* being done in this area, it seemed a
good start and really, I dont care how its implemented, I'll
leave that to the folks who have spent longer than the
8 mins I currently have on it..

> Then, you don't have to standardize a bunch of absolutely silly
> strings (I mean, the concept is so incredibly stupid), you get events
> that are in a precisely defined format going over this netlink socket.

Well, right now, thats all we have, right? Silly strings? But
thats not really my position, which is more like:
Whatever! Whatever! Somebody! Make it so! :) :).

> Then whoever in userspace reads out the messages can interpret them
> however the fuck it wants to. It is then trivial to parse the
> messages and filter them. Furthermore, you could even transmit such
> messages over a network connection to a remote logging server as-is.
>
> And hey, look, for network links going up and down we have the hooks
> already. Funny that...

OK, that is a good idea.. :)

thanks,
Nivedita

2003-06-17 00:31:44

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: David S. Miller" <[email protected]>
Date: Mon, 16 Jun 2003 17:50:08 -0500

If you want events, standardize events and push them over
a queueing based communications channel to userspace, namely
using netlink sockets.

It sounds like you are proposing a new family for the netlink
subsystem. I've included a list of the families I see in 2.5.70
the end of this note. I'm trying to understand which family handles
events such as link state changes or device initialization failure?
It sounds like you're proposing somethink equivalent to
NETLINK_TCP_DIAG. Is that right?

There seems to be some overlap between netlink and netdev notifier
events. Netdev notifier reports many key events. One event I don't
see reported is any indication that device initialization failed.
#define NETDEV_UP 0x0001
#define NETDEV_DOWN 0x0002
#define NETDEV_REBOOT 0x0003
#define NETDEV_CHANGE 0x0004
#define NETDEV_REGISTER 0x0005
#define NETDEV_UNREGISTER 0x0006
#define NETDEV_CHANGEMTU 0x0007
#define NETDEV_CHANGEADDR 0x0008
#define NETDEV_GOING_DOWN 0x0009
#define NETDEV_CHANGENAME 0x000A

One issue with netdev, is that it doesn't seem to allow
for the flexibility of the information passed that netlink has.
For example, when you issue a NETDEV_CHANGE, it's not clear
from the event what the specific change was.

==============================
NETLINK 2.5.70

netlink.h

#define NETLINK_ROUTE 0 /* Routing/device hook */
#define NETLINK_SKIP 1 /* Reserved for ENskip */
#define NETLINK_USERSOCK 2 /* Reserved for user mode socket protocols */
#define NETLINK_FIREWALL 3 /* Firewalling hook */
#define NETLINK_TCPDIAG 4 /* TCP socket monitoring */
#define NETLINK_NFLOG 5 /* netfilter/iptables ULOG */
#define NETLINK_XFRM 6 /* ipsec */
#define NETLINK_ARPD 8
#define NETLINK_ROUTE6 11 /* af_inet6 route comm channel */
#define NETLINK_IP6_FW 13
#define NETLINK_DNRTMSG 14 /* DECnet routing messages */
#define NETLINK_TAPBASE 16 /* 16 to 31 are ethertap */

Descriptions for many of these events can be found at
http://www.europe.redhat.com/documentation/man-pages/man7/netlink.7.php3

NETLINK_ROUTE
Receives routing updates and may be used
to modify the IPv4 routing table (see
NETLINK_FIREWALL
Receives packets sent by the IPv4 firewall code.
NETLINK_ARPD
For managing the arp table in user space.
NETLINK_ROUTE6
Receives and sends IPv6 routing table updates.
NETLINK_IP6_FW
to receive packets that failed the IPv6 firewall
checks (currently not implemented).
NETLINK_TAPBASE...NETLINK_TAPBASE+15
are the instances of the ethertap device. Ethertap
is a pseudo network tunnel device that allows an
ethernet driver to be simulated from user space.
NETLINK_SKIP
Reserved for ENskip.
NETLINK_USERSOCK
is reserved for future user space protocols.

2003-06-17 01:10:17

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Janice Girouard <[email protected]>
Date: Mon, 16 Jun 2003 19:44:22 -0500

It sounds like you are proposing a new family for the netlink
subsystem.

Exactly, you have to create this.

2003-06-17 02:01:59

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Janice Girouard <[email protected]>
Date: Mon, 16 Jun 2003 19:44:22 -0500

It sounds like you are proposing a new family for the netlink
subsystem.

From: "David S. Miller" <[email protected]>

Date:06/16/2003 08:19 PM

Exactly, you have to create this.

Okay. That solves the issue of events generated in a plethora of formats
for the same event. Any suggestions on what should be included in this new
family? I can present a patch to suggest a starting point. However, it
would be great to hear from everyone that has any initial thoughts.

One question that comes to mind, since there is some overlap with netdev
notifier events, should we include those events in the new family? I can
envision a couple of approaches:

1) keep the two interfaces (netdev notifier and netlink), with separate end
users in mind and duplicate the events to each interface. Possibly
thinking about migrating to just one interface over time. Applications
would then just receive one set of events.

2) keep the two interfaces, with no duplication of messages, clarifying the
uses for the two interfaces. An application would then register, and
obtain events from the two separate mechanisms.

p.s. thanks for all the input so far.

2003-06-17 04:21:03

by Valdis Klētnieks

[permalink] [raw]

Subject: Re: patch for common networking error messages

On Mon, 16 Jun 2003 21:12:50 CDT, Janice Girouard said:

> Okay. That solves the issue of events generated in a plethora of formats
> for the same event. Any suggestions on what should be included in this new
> family? I can present a patch to suggest a starting point. However, it
> would be great to hear from everyone that has any initial thoughts.

Well, at the risk of torquing off any SCO supporters, I'd suggest a quick
peek over at the general design of the AIX errpt/trace systems - in both
cases, data comes out of the kernel in a formatted binary stream, and then
a 'template' file is used to drive the parsing of the formatted data.

Quite slick overall, and nicely extensible - you add a new kernel subsystem
that has more trace points, you just tack its templates onto the end of
the format file and you're good to go....

(And SCO can't even claim that's their code - it's pretty obvious the
parentage of the AIX errpt/trace logging is the OS/VS1 and MVS SMF logging
features :)

Attachments:

(No filename) (226.00 B)

2003-06-17 06:56:21

by Andi Kleen

[permalink] [raw]

Subject: Re: patch for common networking error messages

> Actually it already does, to cover the case where an interrupt handler calls
> printk while process-context code is performing a printk.

I don't think it'll work. Both printk and release_console_sem take the logbuf_lock,
which will deadlock if the same CPU already holds it.

It would need to use the usual linux recursive lock hack.

-Andi

2003-06-17 07:05:58

by Andrew Morton

[permalink] [raw]

Subject: Re: patch for common networking error messages

Andi Kleen <[email protected]> wrote:
>
> > Actually it already does, to cover the case where an interrupt handler calls
> > printk while process-context code is performing a printk.
>
> I don't think it'll work. Both printk and release_console_sem take the logbuf_lock,
> which will deadlock if the same CPU already holds it.

Look more closely.

logbuf_lock is only held to protect the logbuf contents and its indices.
And to pin down the current console_sem holder to reliably ensure that
he'll print the text which the nested printer just placed in the buffer.

We do not call the console drivers while holding logbuf_lock. console_sem
is held across the console driver call.

2003-06-17 14:22:34

by Mr. James W. Laferriere

[permalink] [raw]

Subject: Re: patch for common networking error messages

Hello Dave , I know your time is constrained . BUT DAMN IT there
were other questions ask in the (partially quoted) email you are
replying to . Some of those answers would be nice to know .
RSVP , JimL

On Mon, 16 Jun 2003, David S. Miller wrote:
> From: Janice Girouard <[email protected]>
> Date: Mon, 16 Jun 2003 19:44:22 -0500
> It sounds like you are proposing a new family for the netlink
> subsystem.

> Exactly, you have to create this.

--
+------------------------------------------------------------------+
| James W. Laferriere | System Techniques | Give me VMS |
| Network Engineer | P.O. Box 854 | Give me Linux |
| [email protected] | Coudersport PA 16915 | only on AXP |
+------------------------------------------------------------------+

2003-06-17 15:55:38

by Stephen Hemminger

[permalink] [raw]

Subject: Re: patch for common networking error messages

Binary interface's will never cut it. Read the hotplug thread to see how Linus
said, he will never add a binary event daemon interface.

That said, there is an oppurtunity for to provide a ascii interface (similar to
/sbin/hotplug) decodes the data from the rtnetlink interface in a standardized format.

Then it would be easy to write things like perl monitoring scripts that do things
like:
perl phone-me-if-network-dies.pl < /proc/net/events

Don't flame me about the choce of name. /proc/net/events is not the right name
to use for such an interface since adding more to /proc is probably not desired.

2003-06-17 16:00:08

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Stephen Hemminger <[email protected]>
Date: Tue, 17 Jun 2003 09:08:59 -0700

Read the hotplug thread to see how Linus
said, he will never add a binary event daemon interface.

Funny, rtnetlink is exactly this and it is in the tree :-)

Every networking configuration event is transmitted over
rtnetlink sockets to all listeners, in a fixed binary
format.

What Linus doesn't want is this for configuration events,
ie. things like "device appears" not "my ethernet burped"

2003-06-17 18:32:53

[permalink] [raw]

Subject: Re: patch for common networking error messages

David S. Miller wrote:
> From: Stephen Hemminger <[email protected]>
> Date: Tue, 17 Jun 2003 09:08:59 -0700
>
> Read the hotplug thread to see how Linus
> said, he will never add a binary event daemon interface.
>
> Funny, rtnetlink is exactly this and it is in the tree :-)

...and it's been in the tree for quite a while too. It's a shame people
aren't taking advantage of it's obvious utility...

Jeff

2003-06-17 18:55:14

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Jeff Gazik <[email protected]>
Date: Tue, 17 Jun 2003 01:46

...and it's been in the tree for quite a while too. It's a shame
people aren't taking advantage of it's obvious utility...

I'd like to hear what others believe belong in this new netlink family.
The two events that come to mind for this family (or netdev notified if
that's more appropriate) are:
1) device initialization failures,
2) events that drive load balancing software. Right now if we need to
throttle the card, we don't send events up to indicate we have reached
capacity.

Possibly the first might belong in the netdev notified family, and the
second in the netlink family, since you might want to present more than
a two state (success/failure.. or just failure in this case) result.
-

2003-06-17 19:09:27

[permalink] [raw]

Subject: Re: patch for common networking error messages

Janice M Girouard wrote:
> 2) events that drive load balancing software. Right now if we need to
> throttle the card, we don't send events up to indicate we have reached
> capacity.

Question related to this item specifically :)

Do you want to individually send 4000 - 16000 (or more) TX stop / start
events per second to userspace? :) At some point Heisenburg defeats
low latency :)

If not (and I hope not), perhaps also look into the net stack statistics
already kept (or add more sampling stats if necessary), and instead
trigger events based on sampling those statistics.

Jeff

2003-06-17 19:35:42

[permalink] [raw]

Subject: Re: patch for common networking error messages

Jeff Garzik wrote:

Do you want to individually send 4000 - 16000 (or more) TX stop /
start events per second to userspace? :) At some point Heisenburg
defeats low latency :)

How about looking at 1000 byte packet transmit example. A gigabit
adapter would send 125,000 packets per second. I'm thinking that most
of the time, you will have enough available buffers in the adapter that
you don't start to see the adapter buffers completely fill up. Are you
saying that 3.2% - 12.8% of the time in this case you're disabling the
tcp/ip stack because the transmit buffers on your card are completely
full? Perhaps with zero copy enabled, but the tcp/ip cpu load alone
will throttle your ability to fill the adapter buffers up.

What does your own experience indicate for gigabit adapter cards?

I could see the buffers backing up for 10/100 cards. So that case
favors your point. I'm still thinking that it's a sign someone should
be buying a 2nd card and ramping up their network capability. But I can
see your point.

2003-06-17 19:41:22

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Janice M Girouard <[email protected]>
Date: Tue, 17 Jun 2003 14:46:56 -0500

I could see the buffers backing up for 10/100 cards. So that case
favors your point. I'm still thinking that it's a sign someone should
be buying a 2nd card and ramping up their network capability. But I can
see your point.

And when we have 1GHZ memory busses and 10GHz cpus tomorrow,
what does this say for 1gbit and 10gbit cards?

You want to define a machine as having too much "work" or not, yet you
only want to consider one metric to do so. Such schemes are
fundamentally flawed.

2003-06-17 20:13:08

[permalink] [raw]

Subject: Re: patch for common networking error messages

From David S. Miller:

And when we have 1GHZ memory busses and 10GHz cpus tomorrow,
what does this say for 1gbit and 10gbit cards?

Such schemes are fundamentally flawed.

Bottom line.. I was asking for input, and I received it. It's valid
to say... look at the statistics. I really like the concept of driving
events through netlink, but querying statistics works.

p.s. It's been my experience that the memory system is the main
bottleneck when trying to support a heavy network load. When the 10
Gigabit card emerges, and it's here today, the memory system will be
pressed to support it, especially if you're not using zerocopy and
you're thinking of using more than one card. Perhaps if RDMA is
capabilities are added to Linux, then things might be different.

So.. when do you think RDMA will show up on Linx?

2003-06-17 20:18:00

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Janice M Girouard <[email protected]>
Date: Tue, 17 Jun 2003 15:24:20 -0500

Perhaps if RDMA is capabilities are added to Linux, then things
might be different.

So.. when do you think RDMA will show up on Linx?

RDMA is total junk.

On RX, clever RX buffer management is what we need.

On TX zerocopy + TSO is more than sufficient and we have
that today.

2003-06-17 20:27:28

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: David S. Miller" <[email protected]>
Date: 06/17/2003 03:27 PM

On RX, clever RX buffer management is what we need.

What RX buffer management are you proposing? I'm having a hard time
understanding how you'll get rid of the copy without support from the
card.

2003-06-17 20:32:44

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Janice Girouard <[email protected]>
Date: Tue, 17 Jun 2003 15:40:48 -0500

From: David S. Miller" <[email protected]>
Date: 06/17/2003 03:27 PM

On RX, clever RX buffer management is what we need.

What RX buffer management are you proposing? I'm having a hard time
understanding how you'll get rid of the copy without support from the
card.

Sigh... someone write store email down somewhere for the next time
someone asks about this.

The "one true way (tm)" works like this:

1) Chip has a "flow cache", LRU based, managed like routing caches
in many production router implementations. Difference is
that it merely does flow watching.

Flow entries are keyed on saddr/daddr/sport/dport. Flow misses
kill the oldest entry, and replace it with the new one.

Entries are only created in response to full sized data
packets.

2) The receive buffering is segmented into small (256 byte) and
PAGE sized buffers. IP/TCP/whatever headers (determined using
a simply programmable header parser logic, so you can do things
like RPC etc. headers for NFS) are put into the "small" buffers,
data portions for matching flows get accumulated into the PAGE
sized buffers.

It is implied that the card's flow cache keeps track of the
pointers into page it is currently trying to fill for that
flow.

So the first time you see a flow, you add a entry and grab a page
buffer and stick the data part into the page buffer and the
TCP/IP/etc. headers into a "small" buffer. You defer a configurable
amount of time waiting for more TCP data packets (a packet train)
to accumulate more into the PAGE buffer for that flow.

Such receive buffers are presented to the stack as a linked list
of packets, with some indicator that together their data parts are
filling a page.

Things like "sys_receivefile()" and NFS flip these things into the
filesystem page cache.

I'm surprised this isn't evident to more people...

2003-06-17 20:44:27

[permalink] [raw]

Subject: Re: patch for common networking error messages

Did I understand:

> 1) Chip has a "flow cache", LRU based, managed like routing caches

You need the chip to support your technique. Are the vendors picking up on
this? I still don't see how this gets rid of the copy_to_user space once
you've gathered the buffers. How do you feed the user buffer addresses to
the card? You must have something equivalent to the queue pair management
supported in RDMA. What technique are you using? Is it proprietary?

"David S. Miller"
<[email protected] To: Janice Girouard/Austin/IBM@IBMUS
> cc: [email protected], [email protected],
[email protected], Daniel Stekloff/Beaverton/IBM@IBMUS,
06/17/2003 03:42 Larry Kessler/Beaverton/IBM@IBMUS,
PM [email protected], [email protected],
[email protected]
Subject: Re: patch for common networking error messages

From: Janice Girouard <[email protected]>
Date: Tue, 17 Jun 2003 15:40:48 -0500

From: David S. Miller" <[email protected]>
Date: 06/17/2003 03:27 PM

On RX, clever RX buffer management is what we need.

What RX buffer management are you proposing? I'm having a hard time
understanding how you'll get rid of the copy without support from the
card.

Sigh... someone write store email down somewhere for the next time
someone asks about this.

The "one true way (tm)" works like this:

1) Chip has a "flow cache", LRU based, managed like routing caches
in many production router implementations. Difference is
that it merely does flow watching.

Flow entries are keyed on saddr/daddr/sport/dport. Flow misses
kill the oldest entry, and replace it with the new one.

Entries are only created in response to full sized data
packets.

2) The receive buffering is segmented into small (256 byte) and
PAGE sized buffers. IP/TCP/whatever headers (determined using
a simply programmable header parser logic, so you can do things
like RPC etc. headers for NFS) are put into the "small" buffers,
data portions for matching flows get accumulated into the PAGE
sized buffers.

It is implied that the card's flow cache keeps track of the
pointers into page it is currently trying to fill for that
flow.

So the first time you see a flow, you add a entry and grab a page
buffer and stick the data part into the page buffer and the
TCP/IP/etc. headers into a "small" buffer. You defer a configurable
amount of time waiting for more TCP data packets (a packet train)
to accumulate more into the PAGE buffer for that flow.

Such receive buffers are presented to the stack as a linked list
of packets, with some indicator that together their data parts are
filling a page.

Things like "sys_receivefile()" and NFS flip these things into the
filesystem page cache.

I'm surprised this isn't evident to more people...

2003-06-17 21:05:20

[permalink] [raw]

Subject: Re: patch for common networking error messages

From: Janice Girouard <[email protected]>
Date: Tue, 17 Jun 2003 15:57:59 -0500

Did I understand:

> 1) Chip has a "flow cache", LRU based, managed like routing caches

You need the chip to support your technique.

No shit Sherlock.

But it should be noted that the idea can be fully verified in software
by adding the scheme into the RX processing of some existing ethernet
driver.

Are the vendors picking up on this?

If they're going to ignore my ideas, that's not my problem.

I still don't see how this gets rid of the copy_to_user space once
you've gathered the buffers. How do you feed the user buffer addresses to
the card?

You flip the pages into userspace, ie. you replace the page the user
currently has with the one the networking buffer is using.

What technique are you using? Is it proprietary?

ROFL!

2003-06-21 12:25:19

by Alan

[permalink] [raw]

Subject: Re: patch for common networking error messages

On Llu, 2003-06-16 at 23:55, David S. Miller wrote:
> Let me know when you're back on planet earth ok?
>
> Standardizing strings is an absolutely FRUITLESS exercise.

Standardising strings is a real help for end users, but its not the way
to approach logging issues I agree.

2003-06-21 14:13:44

by Jamal Hadi

[permalink] [raw]

Subject: Re: patch for common networking error messages

On Sat, 21 Jun 2003, Alan Cox wrote:

> On Llu, 2003-06-16 at 23:55, David S. Miller wrote:
> > Let me know when you're back on planet earth ok?
> >
> > Standardizing strings is an absolutely FRUITLESS exercise.
>
> Standardising strings is a real help for end users, but its not the way
> to approach logging issues I agree.

now that xml is the holy grail ive seen people actually
preach xml strings as encoding for protocols ;-> The arguement
i have seen put forward is that strings are easier to read
for users than binary encoding ;-> Therefore they can debug problems.
There maybe cases where this may be valid[1] - the only problem is
a lot of loonies will think this is the next sliced bread.

what about all that bandwidth stoopid xml consumes?
"bandwidth? Who has a problem with bandwidth?;->
what about all that involved processiong of stoopid xml?
"cpu? who has CPU problems?"
Intel has a 10Gige NIC, a 2Mhz cpu, adn 4G DDR Ram for your hungry
applications.
Its a conspiracy i tell ya ;->

cheers,
jamal

[1] For people who use expect for example to send string commands
to a remote system to configure things, when expect (simple req-resp)
becomes too simple you may need something more sophisticated.
They are already sending strings across tcp probably.
Infact a IETF working group has been formed to standardixe this.
http://www.ietf.org/html.charters/netconf-charter.html
theres a draft at :
http://www.ietf.org/internet-drafts/draft-enns-xmlconf-spec-00.txt

The only unfortunate side effect to this is you will see a lot
idjots putting XML in protocols from now on just because.

2003-06-23 00:38:04