LinuxLists.cc - [GIT]: Networking

[permalink] [raw]

Subject: Re: [GIT]: Networking

On Sun, 20 Jul 2008, David Miller wrote:
>
> Hello Linus. This is the main networking merge for 2.6.27

Grr. And I quote:

Security table (IP_NF_SECURITY) [Y/n/?] (NEW) ?

This option adds a `security' table to iptables, for use
with Mandatory Access Control (MAC) policy.

If unsure, say N.

why the heck does this new config option apparently default to 'Y'? It's a
new option, so no old users can need it, and the docs even say you should
say 'N' unless you know what you're doing.

(Same issue with the IPv6 version).

Don't do this.

Linus

2008-07-21 01:03:20

[permalink] [raw]

Subject: Re: [GIT]: Networking

From: Linus Torvalds <[email protected]>
Date: Sun, 20 Jul 2008 17:54:04 -0700 (PDT)

> On Sun, 20 Jul 2008, David Miller wrote:
> >
> > Hello Linus. This is the main networking merge for 2.6.27
>
> Grr. And I quote:
>
> Security table (IP_NF_SECURITY) [Y/n/?] (NEW) ?
>
> This option adds a `security' table to iptables, for use
> with Mandatory Access Control (MAC) policy.
>
> If unsure, say N.
>
> why the heck does this new config option apparently default to 'Y'? It's a
> new option, so no old users can need it, and the docs even say you should
> say 'N' unless you know what you're doing.
>
> (Same issue with the IPv6 version).
>
> Don't do this.

James/Patrick please fix this.

Thanks.

2008-07-21 01:08:22

[permalink] [raw]

Subject: Re: [GIT]: Networking

This warning also seems to be new:

net/ipv6/addrconf.c: In function ?addrconf_add_linklocal?:
net/ipv6/addrconf.c:2318: warning: unused variable ?net?

and looking at the code it's apparently because I'm not an optimistic
enough dad.

But hey, if you had three pre-teenage girls, you might not be all that
optimistic either. So I think that's reasonable.

Problem seems to have been introduced by 53b7997f ("ipv6 netns: Make
several "global" sysctl variables namespace aware")

Linus

2008-07-21 01:12:21

[permalink] [raw]

Subject: Re: [GIT]: Networking

> > > This is the main networking merge for 2.6.27

Looks like multiqueue changes broken netconsole (it remains silent).

Also, I had to enable CONFIG_NET_SCHED=y, otherwise it wouldn't link:

net/built-in.o: In function `dev_queue_xmit':
(.text+0x11210): undefined reference to `qdisc_calculate_pkt_len'
net/built-in.o: In function `__qdisc_destroy':
sch_generic.c:(.text+0x1ddcd): undefined reference to `qdisc_put_stab'

And finally, boot process hangs after TCP cubic is initialized.

This is net-2.6 tree (175f9c1bba9b825d22b142d183c9e175488b260c)
(tainted with conntracking-in-net-ns changes, though).

2008-07-21 01:14:51

[permalink] [raw]

Subject: Re: [GIT]: Networking

From: Alexey Dobriyan <[email protected]>
Date: Mon, 21 Jul 2008 05:09:52 +0400

> > > > This is the main networking merge for 2.6.27
>
> Looks like multiqueue changes broken netconsole (it remains silent).

Can you dig a little bit deeper into this one? It ought to
work fine.

Thanks.

> Also, I had to enable CONFIG_NET_SCHED=y, otherwise it wouldn't link:

I just pushed Linus a fix for that, Stephen Rothwell noticed it
too.

> And finally, boot process hangs after TCP cubic is initialized.

That very much should not happen. Can you turn on initcall
tracing so we can see where that is happening exactly?

2008-07-21 01:17:19

[permalink] [raw]

Subject: Re: [GIT]: Networking

From: Linus Torvalds <[email protected]>
Date: Sun, 20 Jul 2008 18:07:28 -0700 (PDT)

> This warning also seems to be new:
>
> net/ipv6/addrconf.c: In function ?addrconf_add_linklocal?:
> net/ipv6/addrconf.c:2318: warning: unused variable ?net?
>
> and looking at the code it's apparently because I'm not an optimistic
> enough dad.
>
> But hey, if you had three pre-teenage girls, you might not be all that
> optimistic either. So I think that's reasonable.
>
> Problem seems to have been introduced by 53b7997f ("ipv6 netns: Make
> several "global" sysctl variables namespace aware")

Let's just expand the thing in the one spot it gets
referenced in.

Please apply, thanks.

ipv6: Fix warning in addrconf code.

Reported by Linus.

Signed-off-by: David S. Miller <[email protected]>

diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
index 580ae50..9f4fcce 100644
--- a/net/ipv6/addrconf.c
+++ b/net/ipv6/addrconf.c
@@ -2315,12 +2315,11 @@ static void init_loopback(struct net_device *dev)
static void addrconf_add_linklocal(struct inet6_dev *idev, struct in6_addr *addr)
{
struct inet6_ifaddr * ifp;
- struct net *net = dev_net(idev->dev);
u32 addr_flags = IFA_F_PERMANENT;

#ifdef CONFIG_IPV6_OPTIMISTIC_DAD
if (idev->cnf.optimistic_dad &&
- !net->ipv6.devconf_all->forwarding)
+ !dev_net(idev->dev)->ipv6.devconf_all->forwarding)
addr_flags |= IFA_F_OPTIMISTIC;
#endif

????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay?ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m????????????I?

2008-07-21 01:21:35

[permalink] [raw]

Subject: Re: [GIT]: Networking

David Miller wrote:
> From: Linus Torvalds <[email protected]>
> Date: Sun, 20 Jul 2008 17:54:04 -0700 (PDT)
>
>
>> Grr. And I quote:
>>
>> Security table (IP_NF_SECURITY) [Y/n/?] (NEW) ?
>>
>> This option adds a `security' table to iptables, for use
>> with Mandatory Access Control (MAC) policy.
>>
>> If unsure, say N.
>>
>> why the heck does this new config option apparently default to 'Y'? It's a
>> new option, so no old users can need it, and the docs even say you should
>> say 'N' unless you know what you're doing.
>>
>> (Same issue with the IPv6 version).
>>
>> Don't do this.
>>
>
> James/Patrick please fix this.
>

This is only the NETFILTER_ADVANCED=n default (for SECURITY=y).
The netfilter defaults for NETFILTER_ADVANCED=n should be m/y for
things that are needed by mainstream distributions for normal
usage.

I'm not sure how this is going to be used, James?

2008-07-21 01:24:50

[permalink] [raw]

Subject: Re: [GIT]: Networking

On Sun, Jul 20, 2008 at 06:14:40PM -0700, David Miller wrote:
> > And finally, boot process hangs after TCP cubic is initialized.
>
> That very much should not happen. Can you turn on initcall
> tracing so we can see where that is happening exactly?

Before that BUG_ON in __netif_schedule() happens (and quickly scrolls
off). But after netconsole ups device (which is ATL1 if it matters).

2008-07-21 02:43:26

[permalink] [raw]

Subject: Re: [GIT]: Networking

OK, the problem is really this buglet:

BUG_ON(q == &noop_qdisc);
__netif_schedule
netif_tx_wake_queue
netif_wake_queue
atl1_check_link
atl1_up or atlx_link_chg_task
run_workqueue

2008-07-21 02:49:16

[permalink] [raw]

Subject: Re: [GIT]: Networking

From: Alexey Dobriyan <[email protected]>
Date: Mon, 21 Jul 2008 06:40:56 +0400

> OK, the problem is really this buglet:
>
> BUG_ON(q == &noop_qdisc);
> __netif_schedule
> netif_tx_wake_queue
> netif_wake_queue
> atl1_check_link
> atl1_up or atlx_link_chg_task
> run_workqueue

Thanks for the backtrace I'll work on fixing this.

2008-07-21 05:11:22

[permalink] [raw]

Subject: Re: [GIT]: Networking

From: David Miller <[email protected]>
Date: Sun, 20 Jul 2008 19:48:56 -0700 (PDT)

> From: Alexey Dobriyan <[email protected]>
> Date: Mon, 21 Jul 2008 06:40:56 +0400
>
> > OK, the problem is really this buglet:
> >
> > BUG_ON(q == &noop_qdisc);
> > __netif_schedule
> > netif_tx_wake_queue
> > netif_wake_queue
> > atl1_check_link
> > atl1_up or atlx_link_chg_task
> > run_workqueue
>
> Thanks for the backtrace I'll work on fixing this.

[ Jeff and co., this is basically the kind of patch I want
to see eventually for the Intel drivers too... ]

Alexey, please try this patch:

atl1: Do not wake queue before queue has been started.

Based upon a bug report by Alexey Dobriyan.

Packet flow during link state events should not be done by
waking and stopping the TX queue anyways, that is handled
transparently by netif_carrier_{on,off}().

So, remove the netif_{wake,stop}_queue() calls in the link
check code, and add the necessary netif_start_queue() call
to atl1_up().

Signed-off-by: David S. Miller <[email protected]>

diff --git a/drivers/net/atlx/atl1.c b/drivers/net/atlx/atl1.c
index 3e22e78..f12e3d1 100644
--- a/drivers/net/atlx/atl1.c
+++ b/drivers/net/atlx/atl1.c
@@ -1308,7 +1308,6 @@ static u32 atl1_check_link(struct atl1_adapter *adapter)
dev_info(&adapter->pdev->dev, "link is down\n");
adapter->link_speed = SPEED_0;
netif_carrier_off(netdev);
- netif_stop_queue(netdev);
}
return 0;
}
@@ -1358,7 +1357,6 @@ static u32 atl1_check_link(struct atl1_adapter *adapter)
if (!netif_carrier_ok(netdev)) {
/* Link down -> Up */
netif_carrier_on(netdev);
- netif_wake_queue(netdev);
}
return 0;
}
@@ -2627,6 +2625,7 @@ static s32 atl1_up(struct atl1_adapter *adapter)
mod_timer(&adapter->watchdog_timer, jiffies);
atlx_irq_enable(adapter);
atl1_check_link(adapter);
+ netif_start_queue(netdev);
return 0;

err_up:

2008-07-21 08:37:27

[permalink] [raw]

Subject: iwlwifi: fix build bug in "iwlwifi: fix LED stall"

hi David,

> iwlwifi: fix LED stall

please find a small build fix below.

Ingo

-------------->
commit c61b0199e779caf2dcfdb6e83439c1fdf9f20209
Author: Ingo Molnar <[email protected]>
Date: Mon Jul 21 10:33:42 2008 +0200

iwlwifi: fix build bug in "iwlwifi: fix LED stall"

-tip testing found the following build failure:

drivers/net/wireless/iwlwifi/iwl-led.c: In function ‘iwl_led_brightness_set’:
drivers/net/wireless/iwlwifi/iwl-led.c:198: error: ‘led_type_str’ undeclared (first use in this function)
drivers/net/wireless/iwlwifi/iwl-led.c:198: error: (Each undeclared identifier is reported only once
drivers/net/wireless/iwlwifi/iwl-led.c:198: error: for each function it appears in.)

Triggered if this driver is built with !CONFIG_IWLWIFI_DEBUG. Introduced
by commit 0eee61273.

The best fix is to make led_type_str available as a zero-size symbol and to
only add members to the array if CONFIG_IWLWIFI_DEBUG is set. This way
there's no overhead in the debugging case and we have type checking in the
IWL_DEBUG_LED() macro as well.

Signed-off-by: Ingo Molnar <[email protected]>
---
drivers/net/wireless/iwlwifi/iwl-led.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/wireless/iwlwifi/iwl-led.c b/drivers/net/wireless/iwlwifi/iwl-led.c
index 899d7a2..d7129f7 100644
--- a/drivers/net/wireless/iwlwifi/iwl-led.c
+++ b/drivers/net/wireless/iwlwifi/iwl-led.c
@@ -44,15 +44,15 @@
#include "iwl-io.h"
#include "iwl-helpers.h"

-#ifdef CONFIG_IWLWIFI_DEBUG
static const char *led_type_str[] = {
+#ifdef CONFIG_IWLWIFI_DEBUG
__stringify(IWL_LED_TRG_TX),
__stringify(IWL_LED_TRG_RX),
__stringify(IWL_LED_TRG_ASSOC),
__stringify(IWL_LED_TRG_RADIO),
NULL
-};
#endif /* CONFIG_IWLWIFI_DEBUG */
+};

static const struct {

2008-07-21 09:48:31

by Alexander Beregalov

[permalink] [raw]

Subject: Re: [GIT]: Networking

Hi David!

There is another problem on sparc64 and happymeal ethernet card.

when tring to up interface:

kernel BUG at net/core/dev.c:1328

ip(1090): Kernel bad sw trap 5

__netif_schedule
happy_meal_set_multicast
__dev_set_rx_mode
dev_set_rx_mode
dev_open
dev_change_flags
do_setlink
rtnl_newlink
rtnetlink_rcv_msg
netlink_rcv_skb

2008-07-21 10:03:16

[permalink] [raw]

Subject: RE: iwlwifi: fix build bug in "iwlwifi: fix LED stall"

>-----Original Message-----
>From: Ingo Molnar [mailto:[email protected]]
>Sent: Monday, July 21, 2008 11:37 AM
>To: David Miller
>Cc: [email protected]; [email protected]; Winkler,
Tomas;
>John W. Linville
>Subject: iwlwifi: fix build bug in "iwlwifi: fix LED stall"
>
>
>hi David,
>
>> iwlwifi: fix LED stall
>
>please find a small build fix below.
>
There are already patches that fix this. John has to pick them up from
the mail.
Thanks
Tomas

---------------------------------------------------------------------
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

2008-07-21 10:16:54

by Ben Hutchings

[permalink] [raw]

Subject: Re: [GIT]: Networking

Alexander Beregalov wrote:
> Hi David!
>
> There is another problem on sparc64 and happymeal ethernet card.
>
> when tring to up interface:
>
> kernel BUG at net/core/dev.c:1328
[...]

This is yet another driver calling netif_wake_queue() during dev_open(),
when there is no real qdisc present. (And yes, sfc is another of those
drivers - I will post a patch after internal review.)

Ben.

--
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.

2008-07-21 10:54:03

[permalink] [raw]

Subject: Re: iwlwifi: fix build bug in "iwlwifi: fix LED stall"

* Winkler, Tomas <[email protected]> wrote:

>
>
> >-----Original Message-----
> >From: Ingo Molnar [mailto:[email protected]]
> >Sent: Monday, July 21, 2008 11:37 AM
> >To: David Miller
> >Cc: [email protected]; [email protected]; Winkler,
> Tomas;
> >John W. Linville
> >Subject: iwlwifi: fix build bug in "iwlwifi: fix LED stall"
> >
> >
> >hi David,
> >
> >> iwlwifi: fix LED stall
> >
> >please find a small build fix below.
> >
>
> There are already patches that fix this. John has to pick them up from
> the mail. Thanks Tomas

could you post the fix in this thread too please?

Ingo

2008-07-21 11:29:38

[permalink] [raw]

Subject: Re: [GIT]: Networking

Patrick McHardy wrote:
>>> Security table (IP_NF_SECURITY) [Y/n/?] (NEW) ?
...
> This is only the NETFILTER_ADVANCED=n default (for SECURITY=y).
> The netfilter defaults for NETFILTER_ADVANCED=n should be m/y for
> things that are needed by mainstream distributions for normal
> usage.

Don't ask what mainstream distributors would choose, ask how "make
oldconfig" or "make silentoldconfig" should behave. They should not add
features that weren't there before and are not necessary to continue
using all features of the old configuration.
--
Stefan Richter
-=====-==--- -=== =-=-=
http://arcgraph.de/sr/

2008-07-21 11:46:56

by James Morris

[permalink] [raw]

Subject: Re: [GIT]: Networking

On Mon, 21 Jul 2008, Patrick McHardy wrote:

> > James/Patrick please fix this.
> >
>
> This is only the NETFILTER_ADVANCED=n default (for SECURITY=y).
> The netfilter defaults for NETFILTER_ADVANCED=n should be m/y for
> things that are needed by mainstream distributions for normal
> usage.
>
> I'm not sure how this is going to be used, James?

I think the idea now is that everything new is N by default, but the
intention is to have this enabled in Fedora/RHEL.

Patrick, would you please fix this up? The only dev box I have access to
at the moment doesn't boot with recent git (I think it's the macbook2
issue).

- James
--
James Morris
<[email protected]>

2008-07-21 11:59:46

[permalink] [raw]

Subject: Re: [GIT]: Networking

On Sun, Jul 20, 2008 at 10:11:06PM -0700, David Miller wrote:
> From: David Miller <[email protected]>
> Date: Sun, 20 Jul 2008 19:48:56 -0700 (PDT)
>
> > From: Alexey Dobriyan <[email protected]>
> > Date: Mon, 21 Jul 2008 06:40:56 +0400
> >
> > > OK, the problem is really this buglet:
> > >
> > > BUG_ON(q == &noop_qdisc);
> > > __netif_schedule
> > > netif_tx_wake_queue
> > > netif_wake_queue
> > > atl1_check_link
> > > atl1_up or atlx_link_chg_task
> > > run_workqueue
> >
> > Thanks for the backtrace I'll work on fixing this.
>
> [ Jeff and co., this is basically the kind of patch I want
> to see eventually for the Intel drivers too... ]
>
> Alexey, please try this patch:
>
> atl1: Do not wake queue before queue has been started.

Patch helps and networking is back with and without using netconsole,
thank you!

Also wan to say, 8139too driver survived multiqueue TX changes fine.

2008-07-21 12:06:48

[permalink] [raw]

Subject: Re: [GIT]: Networking

James Morris wrote:
> On Mon, 21 Jul 2008, Patrick McHardy wrote:
>
>
>> This is only the NETFILTER_ADVANCED=n default (for SECURITY=y).
>> The netfilter defaults for NETFILTER_ADVANCED=n should be m/y for
>> things that are needed by mainstream distributions for normal
>> usage.
>>
>> I'm not sure how this is going to be used, James?
>>
>
> I think the idea now is that everything new is N by default, but the
> intention is to have this enabled in Fedora/RHEL.

Well, this option (NETFILTER_ADVANCED) was introduced specifically
so Linus doesn't have to go through and enable all the netfilter
options manually :)

The idea was that NETFILTER_ADVANCED=n enables everything needed
by mainstream distributions and hides the rest. We can certainly
change the default for this option, but that makes NETFILTER_ADVANCED
pretty much useless.

> Patrick, would you please fix this up? The only dev box I have access to
> at the moment doesn't boot with recent git (I think it's the macbook2
> issue).

Sure. I'd like to hear whether Linus still wants this changed though.

2008-07-21 12:12:54

[permalink] [raw]

Subject: [PATCH] iwlwifi: RS small compile warnings without CONFIG_IWLWIFI_DEBUG

From: Denis V. Lunev <[email protected]>

iwl-agn-rs.c: In function 'rs_clear':
iwl-agn-rs.c:2405: warning: unused variable 'priv

Signed-off-by: Denis V. Lunev <[email protected]>
Signed-off-by: Tomas Winkler <[email protected]>
---
drivers/net/wireless/iwlwifi/iwl-agn-rs.c | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/drivers/net/wireless/iwlwifi/iwl-agn-rs.c b/drivers/net/wireless/iwlwifi/iwl-agn-rs.c
index 6a4b229..7ddd07d 100644
--- a/drivers/net/wireless/iwlwifi/iwl-agn-rs.c
+++ b/drivers/net/wireless/iwlwifi/iwl-agn-rs.c
@@ -2396,6 +2396,7 @@ static void rs_free(void *priv_rate)

static void rs_clear(void *priv_rate)
{
+#ifdef CONFIG_IWLWIFI_DEBUG
struct iwl_priv *priv = (struct iwl_priv *) priv_rate;

IWL_DEBUG_RATE("enter\n");
@@ -2403,6 +2404,7 @@ static void rs_clear(void *priv_rate)
/* TODO - add rate scale state reset */

IWL_DEBUG_RATE("leave\n");
+#endif /* CONFIG_IWLWIFI_DEBUG */
}

static void rs_free_sta(void *priv_rate, void *priv_sta)
--
1.5.4.1

---------------------------------------------------------------------
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

2008-07-21 12:13:15

[permalink] [raw]

Subject: [PATCH] iwlwifi: small compile warnings without CONFIG_IWLWIFI_DEBUG

From: Denis V. Lunev <[email protected]>

CC [M] drivers/net/wireless/iwlwifi/iwl-scan.o
drivers/net/wireless/iwlwifi/iwl-scan.c: In function 'iwl_rx_scan_complete_notif':
drivers/net/wireless/iwlwifi/iwl-scan.c:274: warning: unused variable 'scan_notif'

Signed-off-by: Denis V. Lunev <[email protected]>
Acked-by: Tomas Winkler <[email protected]>
---
drivers/net/wireless/iwlwifi/iwl-scan.c | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/drivers/net/wireless/iwlwifi/iwl-scan.c b/drivers/net/wireless/iwlwifi/iwl-scan.c
index 847690b..0033232 100644
--- a/drivers/net/wireless/iwlwifi/iwl-scan.c
+++ b/drivers/net/wireless/iwlwifi/iwl-scan.c
@@ -270,6 +270,7 @@ static void iwl_rx_scan_results_notif(struct iwl_priv *priv,
static void iwl_rx_scan_complete_notif(struct iwl_priv *priv,
struct iwl_rx_mem_buffer *rxb)
{
+#ifdef CONFIG_IWLWIFI_DEBUG
struct iwl_rx_packet *pkt = (struct iwl_rx_packet *)rxb->skb->data;
struct iwl_scancomplete_notification *scan_notif = (void *)pkt->u.raw;

@@ -277,6 +278,7 @@ static void iwl_rx_scan_complete_notif(struct iwl_priv *priv,
scan_notif->scanned_channels,
scan_notif->tsf_low,
scan_notif->tsf_high, scan_notif->status);
+#endif

/* The HW is no longer scanning */
clear_bit(STATUS_SCAN_HW, &priv->status);
--
1.5.4.1

---------------------------------------------------------------------
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

2008-07-21 12:13:30

[permalink] [raw]

Subject: [PATCH] iwlwifi: compilation error when CONFIG_IWLWIFI_DEBUG is not set

From: Denis V. Lunev <[email protected]>

CC [M] drivers/net/wireless/iwlwifi/iwl-rfkill.o
drivers/net/wireless/iwlwifi/iwl-led.c: In function 'iwl_led_brightness_set':
drivers/net/wireless/iwlwifi/iwl-led.c:198: error: 'led_type_str' undeclared (first use in this function)
drivers/net/wireless/iwlwifi/iwl-led.c:198: error: (Each undeclared identifier is reported only once
drivers/net/wireless/iwlwifi/iwl-led.c:198: error: for each function it appears in.)

The problem is that led_type_str is defined under CONFIG_IWLWIFI_DEBUG
while IWL_DEBUG is a static inline function in this case. Replace it
with macro.

Signed-off-by: Denis V. Lunev <[email protected]>
Signed-off-by: Tomas Winkler <[email protected]>
---
drivers/net/wireless/iwlwifi/iwl-debug.h | 8 ++------
1 files changed, 2 insertions(+), 6 deletions(-)

diff --git a/drivers/net/wireless/iwlwifi/iwl-debug.h b/drivers/net/wireless/iwlwifi/iwl-debug.h
index 097a72f..b4ffd33 100644
--- a/drivers/net/wireless/iwlwifi/iwl-debug.h
+++ b/drivers/net/wireless/iwlwifi/iwl-debug.h
@@ -68,12 +68,8 @@ void iwl_dbgfs_unregister(struct iwl_priv *priv);
#endif

#else
-static inline void IWL_DEBUG(int level, const char *fmt, ...)
-{
-}
-static inline void IWL_DEBUG_LIMIT(int level, const char *fmt, ...)
-{
-}
+#define IWL_DEBUG(level, fmt, args...)
+#define IWL_DEBUG_LIMIT(level, fmt, args...)
#endif /* CONFIG_IWLWIFI_DEBUG */

--
1.5.4.1

---------------------------------------------------------------------
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

2008-07-21 13:31:22

[permalink] [raw]

Subject: [crash, bisected] Kernel BUG at ffffffff8079afb1 (__netif_schedule())

David,

-tip testing on latest -git (v2.6.26-5253-g14b395e) triggered the
following boot crash on a Core2Duo 64-bit testsystem:

ADDRCONF(NETDEV_UP): eth0: link is not ready
eth0: Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
------------[ cut here ]------------
Kernel BUG at ffffffff8079afb1 [verbose debug info unavailable]
invalid opcode: 0000 [1] SMP
CPU 0
Pid: 7, comm: events/0 Not tainted 2.6.26-rc8 #21302
RIP: 0010:[<ffffffff8079afb1>] [<ffffffff8079afb1>] __netif_schedule+0xd/0x64
RSP: 0018:ffff81003fa4be30 EFLAGS: 00010246
RAX: 00000000ffffffff RBX: ffff81003e9f49f0 RCX: ffffffff80c38fe0
RDX: ffff81003e9e7940 RSI: ffffffff80c3fdc0 RDI: ffffffff80c3fdc0
RBP: ffff81003e9f49f0 R08: 0000000000001607 R09: ffff810003b1c380
R10: 0000000000000005 R11: 00000100ffffffff R12: 0000000000000000
R13: ffff81003e9f49f0 R14: ffff81003e9f4000 R15: ffff81003e9f46c0
FS: 0000000000000000(0000) GS:ffffffff80c7c000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000de7ef0 CR3: 000000003e5f1000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process events/0 (pid: 7, threadinfo ffff81003fa4a000, task ffff81003fa20c30)
Stack: ffff81003e9f49f0 ffffffff805083a4 ffffffff80d19840 ffff81003e9f4a08
ffff81003e9e7880 0000000000000000 0000000000000000 0000000000000202
000000003f9f5f10 ffff81003fa06a40 ffffffff80507f5c ffff81003fa06a48
Call Trace:
[<ffffffff805083a4>] ? e1000_watchdog_task+0x448/0x635
[<ffffffff80507f5c>] ? e1000_watchdog_task+0x0/0x635
[<ffffffff8023e11b>] ? run_workqueue+0x80/0x112
[<ffffffff8023e9f6>] ? worker_thread+0xd9/0xe8
[<ffffffff802410af>] ? autoremove_wake_function+0x0/0x2e
[<ffffffff8023e91d>] ? worker_thread+0x0/0xe8
[<ffffffff80240f58>] ? kthread+0x47/0x73
[<ffffffff8022d557>] ? schedule_tail+0x28/0x5d
[<ffffffff8020c288>] ? child_rip+0xa/0x12
[<ffffffff80240f11>] ? kthread+0x0/0x73
[<ffffffff8020c27e>] ? child_rip+0x0/0x12

Code: c2 48 8b 42 30 48 89 06 48 89 72 30 e8 30 ab a9 ff 48 89 df 57 9d 66 0f 1f 44 00 00 5b c3 48 81 ff c0 fd c3 80 53 48 89 fe 75 04 <0f> 0b eb fe f0 0f ba 6f 30 01 19 c0 85 c0 75 45 9c 58 66 0f 1f
RIP [<ffffffff8079afb1>] __netif_schedule+0xd/0x64
RSP <ffff81003fa4be30>
Kernel panic - not syncing: Fatal exception

i've bisected it back to:

| 37437bb2e1ae8af470dfcd5b4ff454110894ccaf is first bad commit
| commit 37437bb2e1ae8af470dfcd5b4ff454110894ccaf
| Author: David S. Miller <[email protected]>
| Date: Wed Jul 16 02:15:04 2008 -0700
|
| pkt_sched: Schedule qdiscs instead of netdev_queue.

bisection log:

# bad: [14b395e1] Merge branch 'for-2.6.27' of git://linux-nfs.org/~
# good: [bce7f795] Linux 2.6.26
# good: [cadc7236] Merge branch 'bkl-removal' into next
# bad: [a0c80b8d] pkt_sched: Make default qdisc nonshared-multiqueue
# good: [30902dc4] ax25: Fix std timer socket destroy handling.
# good: [fbd8f13a] net-sched: sch_htb: move hash and sibling list rem
# good: [83aa2e9b] netlabel: return msg overflow error from netlbl_ci
# good: [0388b002] icmp: add struct net argument to icmp_out_count
# good: [ca12a1ac] inet: prepare net on the stack for NET accounting
# good: [8f0f2227] net: Implement simple sw TX hashing.
# bad: [17715e68] pkt_sched: Use per-queue locking in shutdown_sched
# good: [e2627c85] pkt_sched: Make QDISC_RUNNING a qdisc state.
# bad: [37437bbf] pkt_sched: Schedule qdiscs instead of netdev_queue
# good: [7698b4ff] pkt_sched: Add and use qdisc_root() and qdisc_root

config and crashlog:

http://redhat.com/~mingo/misc/config-Mon_Jul_21_13_59_54_CEST_2008.bad
http://redhat.com/~mingo/misc/crash-Mon_Jul_21_13_59_54_CEST_2008.log

[ Note: the bootlog says 2.6.26-rc8 - that's because bisection dived
back to when you cut that devel tree of yours. ]

As the bug is reproducible i can test patches, etc. Let me know if you
need more info than this.

Thanks,

Ingo

2008-07-21 13:45:37

[permalink] [raw]

Subject: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

* Ingo Molnar <[email protected]> wrote:

> David,
>
> -tip testing on latest -git (v2.6.26-5253-g14b395e) triggered the
> following boot crash on a Core2Duo 64-bit testsystem:
>
> ADDRCONF(NETDEV_UP): eth0: link is not ready
> eth0: Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> ------------[ cut here ]------------
> Kernel BUG at ffffffff8079afb1 [verbose debug info unavailable]
> invalid opcode: 0000 [1] SMP
> CPU 0
> Pid: 7, comm: events/0 Not tainted 2.6.26-rc8 #21302
> RIP: 0010:[<ffffffff8079afb1>] [<ffffffff8079afb1>] __netif_schedule+0xd/0x64

note, my tests have also triggered another boot crash on the same
system, using the same config:

PM: Removing info for No Bus:phy0
mac80211_hwsim: ieee80211_register_hw failed (-2)
BUG: unable to handle kernel NULL pointer dereference at 0000000000000370
IP: [<ffffffff808da9f1>] rollback_registered+0x2a/0xd6
PGD 0
Oops: 0000 [1] SMP
CPU 1
Pid: 1, comm: swapper Not tainted 2.6.26-tip-00013-g6de15c6-dirty #21290
RIP: 0010:[<ffffffff808da9f1>] [<ffffffff808da9f1>] rollback_registered+0x2a/0xd6
RSP: 0018:ffff88003f83fe00 EFLAGS: 00010212
RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffff88003d4baed8
RDX: ffffffff80979f1d RSI: 0000000000000046 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffff80d6f4a0 R09: ffff880004576800
R10: 0000000000000000 R11: ffffffff80406afe R12: 0000000000000000
R13: ffff88003d4bb9a0 R14: 0000000000000000 R15: 0000000000000008
FS: 0000000000000000(0000) GS:ffff88003f829160(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000370 CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 1, threadinfo ffff88003f83e000, task ffff88003f824000)
Stack: 0000000000000000 ffffffff808daacf ffff88003d4ba2c0 ffffffff8097e1da
ffff88003d4bb9a0 ffffffff8060eb76 00000000fffffffe ffff88003d4ba2c0
ffff88003d4bb9e0 ffffffff811be87a ffff88003f83fea0 ffffffff8024e672
Call Trace:
[<ffffffff808daacf>] unregister_netdevice+0x32/0x77
[<ffffffff8097e1da>] ieee80211_unregister_hw+0x35/0xd4
[<ffffffff8060eb76>] mac80211_hwsim_free+0x1d/0x6a
[<ffffffff811be87a>] init_mac80211_hwsim+0x2df/0x2f0
[<ffffffff8024e672>] getnstimeofday+0x38/0x95
[<ffffffff8024c76a>] ktime_get_ts+0x21/0x49
[<ffffffff811be59b>] init_mac80211_hwsim+0x0/0x2f0
[<ffffffff8020a042>] do_one_initcall+0x42/0x13b
[<ffffffff80247105>] __queue_work+0x23/0x33
[<ffffffff811a09e0>] kernel_init+0x203/0x271
[<ffffffff80234e73>] schedule_tail+0x28/0x60
[<ffffffff80211079>] child_rip+0xa/0x11
[<ffffffff811a07dd>] kernel_init+0x0/0x271
[<ffffffff8021106f>] child_rip+0x0/0x11

Code: c3 53 48 89 fb e8 38 78 00 00 85 c0 75 1d ba ce 0e 00 00 48 c7 c6 b5 e9 d4 80 48 c7 c7 4f 85 ca 80 e8 f3 f1 95 ff e8 aa 7c 93 ff <83> bb 70 03 00 00 00 75 15 48 89 da 48 89 de 48 c7 c7 35 eb d4
RIP [<ffffffff808da9f1>] rollback_registered+0x2a/0xd6
RSP <ffff88003f83fe00>
CR2: 0000000000000370
Kernel panic - not syncing: Fatal exception
Rebooting in 1 seconds..Press any key to enter the menu

this crash led to the bisection result i posted in the previous mail.
This could be a dual bug and one of the crashes masks the other one.

Maybe the __netif_schedule bug is already fixed and when i tried to
bisect the rollback_registered crash i ran into it as bisection went
back into networking history?

Same config as before:

http://redhat.com/~mingo/misc/config-Mon_Jul_21_13_59_54_CEST_2008.bad

Full crashlog:

http://redhat.com/~mingo/misc/crash-Mon_Jul_21_13_59_52_CEST_2008.log

If the __netif_schedule() bug is already fixed by a later commit then i
could attempt to bisect this other crash as well, given an sha1 that i
could cherry-pick into each bisection point.

Thanks,

Ingo

2008-07-21 13:50:56

[permalink] [raw]

Subject: Re: [GIT]: Networking

David,

* David Miller <[email protected]> wrote:

> e1000: remove PCI Express device IDs

networking in -git (v2.6.26-5253-g14b395e, dead e1000 interface)
silently broke on two testboxes of mine earlier today, and i've bisected
[*] it back to:

| d03157babed7424f5391af43200593768ce69c9a is first bad commit
| commit d03157babed7424f5391af43200593768ce69c9a
| Author: Auke Kok <[email protected]>
| Date: Sun Jun 22 15:21:29 2008 -0700
|
| e1000: remove PCI Express device IDs
|
| We do not want to prolong the situation much longer that e1000
| and e1000e support these devices at the same time. As a result,
| take out the bandage that was added for the interim period
| and remove all the PCI Express device IDs from e1000.

i have migrated these testboxes to e1000e. (i migrated a third one
already but forgot about these two and i was stupid enough to do a
bisection suspecting some new bug.)

I'm wondering, couldnt we warn about the removed pci express support via
the kernel log? Some simple printk. (Perhaps also with a default-off
.config option that shuts up this warning for users who intentionally
boot E1000=y without e1000e support.)

Thanks,

Ingo

[*] the bisection log:

# bad: [14b395e1] Merge branch 'for-2.6.27' of git://linux-nfs.org/~
# good: [bce7f795] Linux 2.6.26
# good: [cadc7236] Merge branch 'bkl-removal' into next
# bad: [a0c80b8d] pkt_sched: Make default qdisc nonshared-multiqueue
# good: [30902dc4] ax25: Fix std timer socket destroy handling.
# bad: [fbd8f13a] net-sched: sch_htb: move hash and sibling list rem
# good: [40af48c1] rt2x00: kill URB for all TX queues during disable_
# bad: [4977929a] iwlwifi: control 11n capabilities through module p
# bad: [28f49d89] Merge branch 'master' of master.kernel.org:/pub/sc
# bad: [0caa116d] net: sh_eth: Fix compile error sh_eth
# good: [485ca22c] DM9000: Re-unite menuconfig entries for DM9000 dri
# good: [177db6f0] ixgbe: add LRO support
# bad: [6e4f6f6f] e1000e: make ioport free
# bad: [d03157ba] e1000: remove PCI Express device IDs

2008-07-21 14:16:22

[permalink] [raw]

Subject: Re: [GIT]: Networking

Ingo Molnar wrote:
> I'm wondering, couldnt we warn about the removed pci express support via
> the kernel log? Some simple printk. (Perhaps also with a default-off
> .config option that shuts up this warning for users who intentionally
> boot E1000=y without e1000e support.)

e1000_init_module() already has two printk()s. A third unconditional
one won't hurt a lot. The first two could also be joined.
--
Stefan Richter
-=====-==--- -=== =-=-=
http://arcgraph.de/sr/

2008-07-21 14:31:12

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

* Ingo Molnar <[email protected]> wrote:

> Pid: 1, comm: swapper Not tainted 2.6.26-tip-00013-g6de15c6-dirty #21290

some more information: find below the same crash with vanilla
linus/master and no extra patches. The crash site is:

(gdb) list *0xffffffff808be0c2
0xffffffff808be0c2 is in rollback_registered (net/core/dev.c:3793).
3788 {
3789 BUG_ON(dev_boot_phase);
3790 ASSERT_RTNL();
3791
3792 /* Some devices call without registering for initialization unwind. */
3793 if (dev->reg_state == NETREG_UNINITIALIZED) {
3794 printk(KERN_DEBUG "unregister_netdevice: device %s/%p never "
3795 "was registered\n", dev->name, dev);
3796
3797 WARN_ON(1);
(gdb)

Thanks,

Ingo

----------------------------------->
Linux version 2.6.26-05253-g14b395e (mingo@dione) (gcc version 4.2.3) #21308 SMP Mon Jul 21 16:14:51 CEST 2008
Command line: root=/dev/sda1 earlyprintk=vga console=ttyS0,115200 console=tty 5 profile=0 debug initcall_debug apic=debug apic=verbose ignore_loglevel sysrq_always_enabled pci=nomsi
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000003ed94000 (usable)
BIOS-e820: 000000003ed94000 - 000000003ee4e000 (ACPI NVS)
BIOS-e820: 000000003ee4e000 - 000000003fea2000 (usable)
BIOS-e820: 000000003fea2000 - 000000003fee9000 (ACPI NVS)
BIOS-e820: 000000003fee9000 - 000000003feed000 (usable)
BIOS-e820: 000000003feed000 - 000000003feff000 (ACPI data)
BIOS-e820: 000000003feff000 - 000000003ff00000 (usable)
KERNEL supported cpus:
Intel GenuineIntel
AMD AuthenticAMD
Centaur CentaurHauls
console [earlyvga0] enabled
debug: ignoring loglevel setting.
last_pfn = 0x3ff00 max_arch_pfn = 0x3ffffffff
init_memory_mapping
0000000000 - 003fe00000 page 2M
003fe00000 - 003ff00000 page 4k
kernel direct mapping tables up to 3ff00000 @ 8000-b000
last_map_addr: 3ff00000 end: 3ff00000
ACPI: RSDP 000FE020, 0014 (r0 INTEL )
ACPI: RSDT 3FEFDE48, 0050 (r1 INTEL D975XBX 4B9 MSFT 1000013)
ACPI: FACP 3FEFCF10, 0074 (r1 INTEL D975XBX 4B9 MSFT 1000013)
ACPI: DSDT 3FEF8010, 3E70 (r1 INTEL D975XBX 4B9 MSFT 1000013)
ACPI: FACS 3FEDFC40, 0040
ACPI: APIC 3FEFCE10, 0078 (r1 INTEL D975XBX 4B9 MSFT 1000013)
ACPI: WDDT 3FEF7F90, 0040 (r1 INTEL D975XBX 4B9 MSFT 1000013)
ACPI: MCFG 3FEF7F10, 003C (r1 INTEL D975XBX 4B9 MSFT 1000013)
ACPI: ASF! 3FEFCD10, 00A6 (r32 INTEL D975XBX 4B9 MSFT 1000013)
ACPI: HPET 3FEF7E90, 0038 (r1 INTEL D975XBX 4B9 MSFT 1000013)
ACPI: SSDT 3FEFDC10, 01BC (r1 INTEL CpuPm 4B9 MSFT 1000013)
ACPI: SSDT 3FEFDA10, 01B7 (r1 INTEL Cpu0Ist 4B9 MSFT 1000013)
ACPI: SSDT 3FEFD810, 01B7 (r1 INTEL Cpu1Ist 4B9 MSFT 1000013)
ACPI: SSDT 3FEFD610, 01B7 (r1 INTEL Cpu2Ist 4B9 MSFT 1000013)
ACPI: SSDT 3FEFD410, 01B7 (r1 INTEL Cpu3Ist 4B9 MSFT 1000013)
Entering add_active_range(0, 0x0, 0x9f) 0 entries of 25600 used
Entering add_active_range(0, 0x100, 0x3ed94) 1 entries of 25600 used
Entering add_active_range(0, 0x3ee4e, 0x3fea2) 2 entries of 25600 used
Entering add_active_range(0, 0x3fee9, 0x3feed) 3 entries of 25600 used
Entering add_active_range(0, 0x3feff, 0x3ff00) 4 entries of 25600 used
(5 early reservations) ==> bootmem
#0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
#1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000]
#2 [0000200000 - 0001484b34] TEXT DATA BSS ==> [0000200000 - 0001484b34]
#3 [000009fc00 - 0000100000] BIOS reserved ==> [000009fc00 - 0000100000]
#4 [0000008000 - 0000009000] PGTABLE ==> [0000008000 - 0000009000]
Zone PFN ranges:
DMA 0x00000000 -> 0x00001000
DMA32 0x00001000 -> 0x00100000
Normal 0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[5] active PFN ranges
0: 0x00000000 -> 0x0000009f
0: 0x00000100 -> 0x0003ed94
0: 0x0003ee4e -> 0x0003fea2
0: 0x0003fee9 -> 0x0003feed
0: 0x0003feff -> 0x0003ff00
On node 0 totalpages: 261516
DMA zone: 56 pages used for memmap
DMA zone: 100 pages reserved
DMA zone: 3843 pages, LIFO batch:0
DMA32 zone: 3525 pages used for memmap
DMA32 zone: 253992 pages, LIFO batch:31
Normal zone: 0 pages used for memmap
Movable zone: 0 pages used for memmap
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x82] disabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x83] disabled)
ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x02] dfl dfl lint[0x1])
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 0, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Setting APIC routing to flat
ACPI: HPET id: 0x8086a201 base: 0xfed00000
SMP: Allowing 4 CPUs, 2 hotplug CPUs
mapped APIC to ffffffffff5fc000 ( fee00000)
mapped IOAPIC to ffffffffff5fb000 (00000000fec00000)
Allocating PCI resources starting at 40000000 (gap: 3ff00000:c0100000)
PERCPU: Allocating 40256 bytes of per cpu data
NR_CPUS: 4096, nr_cpu_ids: 4, nr_node_ids 512
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 257835
Kernel command line: root=/dev/sda1 earlyprintk=vga console=ttyS0,115200 console=tty 5 profile=0 debug initcall_debug apic=debug apic=verbose ignore_loglevel sysrq_always_enabled pci=nomsi
kernel profiling enabled (shift: 0)
debug: sysrq always enabled.
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
TSC calibrated against PIT
Detected 2933.406 MHz processor.
Console: colour VGA+ 80x25
console handover: boot [earlyvga0] -> real [tty0]
console [ttyS0] enabled
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Checking aperture...
No AGP bridge found
Memory: 978020k/1047552k available (8071k kernel code, 67476k reserved, 6822k data, 540k init)
CPA: page pool initialized 1 of 1 pages preallocated
hpet clockevent registered
Calibrating delay loop (skipped), value calculated using timer frequency.. <6>5866.81 BogoMIPS (lpj=11733624)
Security Framework initialized
SELinux: Initializing.
SELinux: Starting in permissive mode
Mount-cache hash table entries: 256
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 4096K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
CPU0: Thermal monitoring enabled (TM2)
using mwait in idle threads.
ACPI: Core revision 20080609
enabled ExtINT on CPU#0
ENABLING IO-APIC IRQs
init IO_APIC IRQs
IO-APIC (apicid-pin) 2-0 not connected.
IOAPIC[0]: Set routing entry (2-1 -> 0x31 -> IRQ 1 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-2 -> 0x30 -> IRQ 0 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-3 -> 0x33 -> IRQ 3 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-4 -> 0x34 -> IRQ 4 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-5 -> 0x35 -> IRQ 5 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-6 -> 0x36 -> IRQ 6 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-7 -> 0x37 -> IRQ 7 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-8 -> 0x38 -> IRQ 8 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-9 -> 0x39 -> IRQ 9 Mode:1 Active:0)
IOAPIC[0]: Set routing entry (2-10 -> 0x3a -> IRQ 10 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-11 -> 0x3b -> IRQ 11 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-12 -> 0x3c -> IRQ 12 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-13 -> 0x3d -> IRQ 13 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-14 -> 0x3e -> IRQ 14 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-15 -> 0x3f -> IRQ 15 Mode:0 Active:0)
IO-APIC (apicid-pin) 2-16, 2-17, 2-18, 2-19, 2-20, 2-21, 2-22, 2-23 not connected.
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
CPU0: Intel(R) Core(TM)2 CPU E6800 @ 2.93GHz stepping 05
Using local APIC timer interrupts.
APIC timer calibration result 16667083
Detected 16.667 MHz APIC timer.
Booting processor 1/1 ip 6000
Initializing CPU#1
masked ExtINT on CPU#1
Calibrating delay using timer specific routine.. <6>5866.88 BogoMIPS (lpj=11733771)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 4096K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
CPU1: Thermal monitoring enabled (TM2)
CPU1: Intel(R) Core(TM)2 CPU E6800 @ 2.93GHz stepping 05
checking TSC synchronization [CPU#0 -> CPU#1]: passed.
Brought up 2 CPUs
Total of 2 processors activated (11733.69 BogoMIPS).
calling init_cpufreq_transition_notifier_list+0x0/0x1b
initcall init_cpufreq_transition_notifier_list+0x0/0x1b returned 0 after 0 msecs
calling net_ns_init+0x0/0x133
net_namespace: 1120 bytes
initcall net_ns_init+0x0/0x133 returned 0 after 0 msecs
calling cpufreq_tsc+0x0/0x16
initcall cpufreq_tsc+0x0/0x16 returned 0 after 0 msecs
calling init_smp_flush+0x0/0x51
initcall init_smp_flush+0x0/0x51 returned 0 after 0 msecs
calling print_banner+0x0/0xf
Booting paravirtualized kernel on bare hardware
initcall print_banner+0x0/0xf returned 0 after 3 msecs
calling sysctl_init+0x0/0x32
initcall sysctl_init+0x0/0x32 returned 0 after 0 msecs
calling ksysfs_init+0x0/0xb9
initcall ksysfs_init+0x0/0xb9 returned 0 after 0 msecs
calling init_jiffies_clocksource+0x0/0xc
initcall init_jiffies_clocksource+0x0/0xc returned 0 after 0 msecs
calling pm_init+0x0/0x35
initcall pm_init+0x0/0x35 returned 0 after 0 msecs
calling filelock_init+0x0/0x2e
initcall filelock_init+0x0/0x2e returned 0 after 0 msecs
calling init_script_binfmt+0x0/0xc
initcall init_script_binfmt+0x0/0xc returned 0 after 0 msecs
calling init_elf_binfmt+0x0/0xc
initcall init_elf_binfmt+0x0/0xc returned 0 after 0 msecs
calling init_compat_elf_binfmt+0x0/0xc
initcall init_compat_elf_binfmt+0x0/0xc returned 0 after 0 msecs
calling debugfs_init+0x0/0x47
initcall debugfs_init+0x0/0x47 returned 0 after 0 msecs
calling securityfs_init+0x0/0x47
initcall securityfs_init+0x0/0x47 returned 0 after 0 msecs
calling random32_init+0x0/0x5f
initcall random32_init+0x0/0x5f returned 0 after 0 msecs
calling cpufreq_core_init+0x0/0x84
initcall cpufreq_core_init+0x0/0x84 returned 0 after 0 msecs
calling cpuidle_init+0x0/0x36
initcall cpuidle_init+0x0/0x36 returned 0 after 0 msecs
calling virtio_init+0x0/0x26
initcall virtio_init+0x0/0x26 returned 0 after 0 msecs
calling sock_init+0x0/0x5e
initcall sock_init+0x0/0x5e returned 0 after 0 msecs
calling netpoll_init+0x0/0x2c
initcall netpoll_init+0x0/0x2c returned 0 after 0 msecs
calling netlink_proto_init+0x0/0x13e
NET: Registered protocol family 16
initcall netlink_proto_init+0x0/0x13e returned 0 after 3 msecs
calling bdi_class_init+0x0/0x3d
initcall bdi_class_init+0x0/0x3d returned 0 after 0 msecs
calling kobject_uevent_init+0x0/0x45
initcall kobject_uevent_init+0x0/0x45 returned 0 after 0 msecs
calling pcibus_class_init+0x0/0xc
initcall pcibus_class_init+0x0/0xc returned 0 after 0 msecs
calling pci_driver_init+0x0/0xc
initcall pci_driver_init+0x0/0xc returned 0 after 0 msecs
calling lcd_class_init+0x0/0x49
initcall lcd_class_init+0x0/0x49 returned 0 after 0 msecs
calling backlight_class_init+0x0/0x4a
initcall backlight_class_init+0x0/0x4a returned 0 after 0 msecs
calling dock_init+0x0/0x5c
No dock devices found.
initcall dock_init+0x0/0x5c returned 0 after 0 msecs
calling tty_class_init+0x0/0x2a
initcall tty_class_init+0x0/0x2a returned 0 after 0 msecs
calling vtconsole_class_init+0x0/0xaf
initcall vtconsole_class_init+0x0/0xaf returned 0 after 0 msecs
calling enable_pci_io_ecs+0x0/0x2e
initcall enable_pci_io_ecs+0x0/0x2e returned 0 after 0 msecs
calling early_fill_mp_bus_info+0x0/0x763
initcall early_fill_mp_bus_info+0x0/0x763 returned 0 after 0 msecs
calling arch_kdebugfs_init+0x0/0x3
initcall arch_kdebugfs_init+0x0/0x3 returned 0 after 0 msecs
calling ffh_cstate_init+0x0/0x31
initcall ffh_cstate_init+0x0/0x31 returned 0 after 0 msecs
calling acpi_pci_init+0x0/0x3b
ACPI: bus type pci registered
initcall acpi_pci_init+0x0/0x3b returned 0 after 0 msecs
calling init_acpi_device_notify+0x0/0x4b
initcall init_acpi_device_notify+0x0/0x4b returned 0 after 0 msecs
calling pci_arch_init+0x0/0x44
PCI: MCFG configuration 0: base f0000000 segment 0 buses 0 - 127
PCI: Not using MMCONFIG.
PCI: Using configuration type 1 for base access
initcall pci_arch_init+0x0/0x44 returned 0 after 11 msecs
calling topology_init+0x0/0x31
initcall topology_init+0x0/0x31 returned 0 after 0 msecs
calling param_sysfs_init+0x0/0x1c1
initcall param_sysfs_init+0x0/0x1c1 returned 0 after 11 msecs
calling pm_sysrq_init+0x0/0x19
initcall pm_sysrq_init+0x0/0x19 returned 0 after 0 msecs
calling readahead_init+0x0/0x2e
initcall readahead_init+0x0/0x2e returned 0 after 3 msecs
calling init_bio+0x0/0xc0
initcall init_bio+0x0/0xc0 returned 0 after 0 msecs
calling integrity_init+0x0/0x35
initcall integrity_init+0x0/0x35 returned 0 after 0 msecs
calling blk_settings_init+0x0/0x25
initcall blk_settings_init+0x0/0x25 returned 0 after 0 msecs
calling blk_ioc_init+0x0/0x2a
initcall blk_ioc_init+0x0/0x2a returned 0 after 0 msecs
calling genhd_device_init+0x0/0x36
initcall genhd_device_init+0x0/0x36 returned 0 after 0 msecs
calling blk_dev_integrity_init+0x0/0x2a
initcall blk_dev_integrity_init+0x0/0x2a returned 0 after 0 msecs
calling pci_slot_init+0x0/0x45
initcall pci_slot_init+0x0/0x45 returned 0 after 0 msecs
calling acpi_init+0x0/0x21b
ACPI: EC: Look up EC in DSDT
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
PCI: MCFG configuration 0: base f0000000 segment 0 buses 0 - 127
PCI: BIOS Bug: MCFG area at f0000000 is not reserved in ACPI motherboard resources
PCI: Not using MMCONFIG.
initcall acpi_init+0x0/0x21b returned 0 after 26 msecs
calling acpi_scan_init+0x0/0x113
initcall acpi_scan_init+0x0/0x113 returned 0 after 3 msecs
calling acpi_ec_init+0x0/0x61
initcall acpi_ec_init+0x0/0x61 returned 0 after 0 msecs
calling acpi_pci_root_init+0x0/0x28
ACPI: PCI Root Bridge [PCI0] (0000:00)
pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
pci 0000:00:01.0: PME# disabled
pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1b.0: PME# disabled
pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.0: PME# disabled
pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.4: PME# disabled
pci 0000:00:1c.5: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.5: PME# disabled
pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
pci 0000:00:1d.7: PME# disabled
pci 0000:00:1f.2: PME# supported from D3hot
pci 0000:00:1f.2: PME# disabled
pci 0000:01:00.0: supports D1
pci 0000:01:00.0: supports D2
pci 0000:01:00.1: supports D1
pci 0000:01:00.1: supports D2
pci 0000:02:00.0: supports D1
pci 0000:02:00.0: supports D2
pci 0000:02:00.0: PME# supported from D0 D1 D2
pci 0000:02:00.0: PME# disabled
pci 0000:04:00.0: PME# supported from D0 D3hot D3cold
pci 0000:04:00.0: PME# disabled
pci 0000:05:04.0: supports D1
pci 0000:05:04.0: supports D2
pci 0000:05:04.0: PME# supported from D0 D1 D2 D3hot
pci 0000:05:04.0: PME# disabled
pci 0000:05:05.0: supports D1
pci 0000:05:05.0: supports D2
pci 0000:00:1e.0: transparent bridge
bus 00 -> node 0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P32_._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX4._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX5._PRT]
initcall acpi_pci_root_init+0x0/0x28 returned 0 after 110 msecs
calling acpi_pci_link_init+0x0/0x48
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 7 9 10 *11 12)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 7 9 *10 11 12)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 7 9 *10 11 12)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 7 *9 10 11 12)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 7 9 10 11 12) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 7 9 10 11 12) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 7 *9 10 11 12)
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 7 9 10 *11 12)
initcall acpi_pci_link_init+0x0/0x48 returned 0 after 49 msecs
calling acpi_power_init+0x0/0x77
initcall acpi_power_init+0x0/0x77 returned 0 after 0 msecs
calling acpi_system_init+0x0/0x264
initcall acpi_system_init+0x0/0x264 returned 0 after 0 msecs
calling pnp_init+0x0/0x1f
Linux Plug and Play Support v0.97 (c) Adam Belay
initcall pnp_init+0x0/0x1f returned 0 after 3 msecs
calling pnpacpi_init+0x0/0x8c
pnp: PnP ACPI init
ACPI: bus type pnp registered
IOAPIC[0]: Set routing entry (2-8 -> 0x38 -> IRQ 8 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-13 -> 0x3d -> IRQ 13 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-6 -> 0x36 -> IRQ 6 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-7 -> 0x37 -> IRQ 7 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-12 -> 0x3c -> IRQ 12 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-1 -> 0x31 -> IRQ 1 Mode:0 Active:0)
IOAPIC[0]: Set routing entry (2-4 -> 0x34 -> IRQ 4 Mode:0 Active:0)
pnp: PnP ACPI: found 13 devices
ACPI: ACPI bus type pnp unregistered
initcall pnpacpi_init+0x0/0x8c returned 0 after 38 msecs
calling misc_init+0x0/0x8e
initcall misc_init+0x0/0x8e returned 0 after 0 msecs
calling cn_init+0x0/0xee
initcall cn_init+0x0/0xee returned 0 after 0 msecs
calling tifm_init+0x0/0x6d
initcall tifm_init+0x0/0x6d returned 0 after 0 msecs
calling phy_init+0x0/0x27
initcall phy_init+0x0/0x27 returned 0 after 0 msecs
calling init_dvbdev+0x0/0xaf
initcall init_dvbdev+0x0/0xaf returned 0 after 0 msecs
calling init_scsi+0x0/0x72
SCSI subsystem initialized
initcall init_scsi+0x0/0x72 returned 0 after 3 msecs
calling ata_init+0x0/0x32a
libata version 3.00 loaded.
initcall ata_init+0x0/0x32a returned 0 after 3 msecs
calling spi_init+0x0/0x73
initcall spi_init+0x0/0x73 returned 0 after 0 msecs
calling init_pcmcia_cs+0x0/0x27
initcall init_pcmcia_cs+0x0/0x27 returned 0 after 0 msecs
calling usb_init+0x0/0x104
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
initcall usb_init+0x0/0x104 returned 0 after 11 msecs
calling serio_init+0x0/0x8b
initcall serio_init+0x0/0x8b returned 0 after 0 msecs
calling gameport_init+0x0/0x8b
initcall gameport_init+0x0/0x8b returned 0 after 0 msecs
calling input_init+0x0/0xfc
initcall input_init+0x0/0xfc returned 0 after 0 msecs
calling i2c_init+0x0/0x55
initcall i2c_init+0x0/0x55 returned 0 after 0 msecs
calling power_supply_class_init+0x0/0x34
initcall power_supply_class_init+0x0/0x34 returned 0 after 0 msecs
calling thermal_init+0x0/0x2f
initcall thermal_init+0x0/0x2f returned 0 after 0 msecs
calling mmc_init+0x0/0x65
initcall mmc_init+0x0/0x65 returned 0 after 0 msecs
calling leds_init+0x0/0x2a
initcall leds_init+0x0/0x2a returned 0 after 0 msecs
calling dma_bus_init+0x0/0x2b
initcall dma_bus_init+0x0/0x2b returned 0 after 0 msecs
calling ac97_bus_init+0x0/0xc
initcall ac97_bus_init+0x0/0xc returned 0 after 0 msecs
calling pci_subsys_init+0x0/0x116
PCI: Using ACPI for IRQ routing
number of MP IRQ sources: 15.
number of IO-APIC #2 registers: 24.
testing the IO APIC.......................

IO APIC #2......
.... register #00: 00000000
....... : physical APIC id: 00
.... register #01: 00170020
....... : max redirection entries: 0017
....... : PRQ implemented: 0
....... : IO APIC version: 0020
.... register #02: 00170020
....... : arbitration: 00
.... IRQ redirection table:
NR Dst Mask Trig IRR Pol Stat Dmod Deli Vect:
00 000 1 0 0 0 0 0 0 00
01 003 0 0 0 0 0 1 1 31
02 003 0 0 0 0 0 1 1 30
03 003 0 0 0 0 0 1 1 33
04 003 0 0 0 0 0 1 1 34
05 003 0 0 0 0 0 1 1 35
06 003 0 0 0 0 0 1 1 36
07 003 0 0 0 0 0 1 1 37
08 003 0 0 0 0 0 1 1 38
09 003 0 1 0 0 0 1 1 39
0a 003 0 0 0 0 0 1 1 3A
0b 003 0 0 0 0 0 1 1 3B
0c 003 0 0 0 0 0 1 1 3C
0d 003 0 0 0 0 0 1 1 3D
0e 003 0 0 0 0 0 1 1 3E
0f 003 0 0 0 0 0 1 1 3F
10 000 1 0 0 0 0 0 0 00
11 000 1 0 0 0 0 0 0 00
12 000 1 0 0 0 0 0 0 00
13 000 1 0 0 0 0 0 0 00
14 000 1 0 0 0 0 0 0 00
15 000 1 0 0 0 0 0 0 00
16 000 1 0 0 0 0 0 0 00
17 000 1 0 0 0 0 0 0 00
IRQ to pin mappings:
IRQ0 -> 0:2
IRQ1 -> 0:1
IRQ3 -> 0:3
IRQ4 -> 0:4
IRQ5 -> 0:5
IRQ6 -> 0:6
IRQ7 -> 0:7
IRQ8 -> 0:8
IRQ9 -> 0:9
IRQ10 -> 0:10
IRQ11 -> 0:11
IRQ12 -> 0:12
IRQ13 -> 0:13
IRQ14 -> 0:14
IRQ15 -> 0:15
.................................... done.
initcall pci_subsys_init+0x0/0x116 returned 0 after 164 msecs
calling proto_init+0x0/0x2e
initcall proto_init+0x0/0x2e returned 0 after 0 msecs
calling net_dev_init+0x0/0x158
initcall net_dev_init+0x0/0x158 returned 0 after 0 msecs
calling neigh_init+0x0/0x71
initcall neigh_init+0x0/0x71 returned 0 after 0 msecs
calling fib_rules_init+0x0/0x9c
initcall fib_rules_init+0x0/0x9c returned 0 after 0 msecs
calling genl_init+0x0/0xce
initcall genl_init+0x0/0xce returned 0 after 15 msecs
calling cipso_v4_init+0x0/0x67
initcall cipso_v4_init+0x0/0x67 returned 0 after 0 msecs
calling wireless_nlevent_init+0x0/0x2c
initcall wireless_nlevent_init+0x0/0x2c returned 0 after 0 msecs
calling cfg80211_init+0x0/0x55
initcall cfg80211_init+0x0/0x55 returned 0 after 11 msecs
calling ieee80211_init+0x0/0xd
initcall ieee80211_init+0x0/0xd returned 0 after 0 msecs
calling netlbl_init+0x0/0x83
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
initcall netlbl_init+0x0/0x83 returned 0 after 11 msecs
calling sysctl_init+0x0/0x2f
initcall sysctl_init+0x0/0x2f returned 0 after 0 msecs
calling pci_iommu_init+0x0/0x12
PCI-GART: No AMD northbridge found.
initcall pci_iommu_init+0x0/0x12 returned 0 after 3 msecs
calling hpet_late_init+0x0/0xfc
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
hpet0: 3 64-bit timers, 14318180 Hz
initcall hpet_late_init+0x0/0xfc returned 0 after 7 msecs
calling clocksource_done_booting+0x0/0xd
initcall clocksource_done_booting+0x0/0xd<7>Switched to high resolution mode on CPU 0
Switched to high resolution mode on CPU 1
returned 0 after 0 msecs
calling tracer_alloc_buffers+0x0/0x4ec
tracer: 1286 pages allocated for 65536 entries of 80 bytes
actual entries 65586
initcall tracer_alloc_buffers+0x0/0x4ec returned 0 after 8 msecs
calling init_pipe_fs+0x0/0x42
initcall init_pipe_fs+0x0/0x42 returned 0 after 0 msecs
calling init_mnt_writers+0x0/0x58
initcall init_mnt_writers+0x0/0x58 returned 0 after 0 msecs
calling anon_inode_init+0x0/0x10b
initcall anon_inode_init+0x0/0x10b returned 0 after 0 msecs
calling acpi_event_init+0x0/0x52
initcall acpi_event_init+0x0/0x52 returned 0 after 10 msecs
calling pnp_system_init+0x0/0xc
system 00:01: iomem range 0xf0000000-0xf3ffffff has been reserved
system 00:01: iomem range 0xfed13000-0xfed13fff has been reserved
system 00:01: iomem range 0xfed14000-0xfed17fff has been reserved
system 00:01: iomem range 0xfed18000-0xfed18fff has been reserved
system 00:01: iomem range 0xfed19000-0xfed19fff has been reserved
system 00:01: iomem range 0xfed1c000-0xfed1ffff has been reserved
system 00:01: iomem range 0xfed20000-0xfed3ffff has been reserved
system 00:01: iomem range 0xfed45000-0xfed99fff has been reserved
system 00:01: iomem range 0xc0000-0xdffff has been reserved
system 00:01: iomem range 0xe0000-0xfffff could not be reserved
system 00:06: ioport range 0x500-0x53f has been reserved
system 00:06: ioport range 0x400-0x47f has been reserved
system 00:06: ioport range 0x680-0x6ff has been reserved
initcall pnp_system_init+0x0/0xc returned 0 after 70 msecs
calling chr_dev_init+0x0/0x9a
initcall chr_dev_init+0x0/0x9a returned 0 after 0 msecs
calling firmware_class_init+0x0/0x68
initcall firmware_class_init+0x0/0x68 returned 0 after 0 msecs
calling loopback_init+0x0/0xc
initcall loopback_init+0x0/0xc returned 0 after 0 msecs
calling cpufreq_gov_performance_init+0x0/0xc
initcall cpufreq_gov_performance_init+0x0/0xc returned 0 after 0 msecs
calling cpufreq_gov_dbs_init+0x0/0x47
initcall cpufreq_gov_dbs_init+0x0/0x47 returned 0 after 0 msecs
calling ssb_modinit+0x0/0x4b
initcall ssb_modinit+0x0/0x4b returned 0 after 0 msecs
calling pcibios_assign_resources+0x0/0x86
pci 0000:00:01.0: PCI bridge, secondary bus 0000:01
pci 0000:00:01.0: IO window: 0x3000-0x3fff
pci 0000:00:01.0: MEM window: 0x50300000-0x503fffff
pci 0000:00:01.0: PREFETCH window: 0x00000040000000-0x0000004fffffff
pci 0000:00:1c.0: PCI bridge, secondary bus 0000:02
pci 0000:00:1c.0: IO window: disabled
pci 0000:00:1c.0: MEM window: 0x50200000-0x502fffff
pci 0000:00:1c.0: PREFETCH window: disabled
pci 0000:00:1c.4: PCI bridge, secondary bus 0000:03
pci 0000:00:1c.4: IO window: disabled
pci 0000:00:1c.4: MEM window: disabled
pci 0000:00:1c.4: PREFETCH window: disabled
pci 0000:00:1c.5: PCI bridge, secondary bus 0000:04
pci 0000:00:1c.5: IO window: 0x2000-0x2fff
pci 0000:00:1c.5: MEM window: 0x50100000-0x501fffff
pci 0000:00:1c.5: PREFETCH window: disabled
pci 0000:00:1e.0: PCI bridge, secondary bus 0000:05
pci 0000:00:1e.0: IO window: 0x1000-0x1fff
pci 0000:00:1e.0: MEM window: 0x50000000-0x500fffff
pci 0000:00:1e.0: PREFETCH window: disabled
IOAPIC[0]: Set routing entry (2-16 -> 0x49 -> IRQ 16 Mode:1 Active:1)
pci 0000:00:01.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:00:01.0 to 64
IOAPIC[0]: Set routing entry (2-17 -> 0x51 -> IRQ 17 Mode:1 Active:1)
pci 0000:00:1c.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
PCI: Setting latency timer of device 0000:00:1c.0 to 64
pci 0000:00:1c.4: PCI INT A -> GSI 17 (level, low) -> IRQ 17
PCI: Setting latency timer of device 0000:00:1c.4 to 64
pci 0000:00:1c.5: PCI INT B -> GSI 16 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:00:1c.5 to 64
PCI: Setting latency timer of device 0000:00:1e.0 to 64
initcall pcibios_assign_resources+0x0/0x86 returned 0 after 147 msecs
calling inet_init+0x0/0x1df
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
initcall inet_init+0x0/0x1df returned 0 after 75 msecs
calling af_unix_init+0x0/0x4b
NET: Registered protocol family 1
initcall af_unix_init+0x0/0x4b returned 0 after 3 msecs
calling default_rootfs+0x0/0x61
initcall default_rootfs+0x0/0x61 returned 0 after 0 msecs
calling i8259A_init_sysfs+0x0/0x23
initcall i8259A_init_sysfs+0x0/0x23 returned 0 after 0 msecs
calling vsyscall_init+0x0/0x6c
initcall vsyscall_init+0x0/0x6c returned 0 after 0 msecs
calling sbf_init+0x0/0xd5
initcall sbf_init+0x0/0xd5 returned 0 after 0 msecs
calling i8237A_init_sysfs+0x0/0x23
initcall i8237A_init_sysfs+0x0/0x23 returned 0 after 0 msecs
calling add_rtc_cmos+0x0/0x1d
initcall add_rtc_cmos+0x0/0x1d returned 0 after 0 msecs
calling cache_sysfs_init+0x0/0x55
initcall cache_sysfs_init+0x0/0x55 returned 0 after 0 msecs
calling mce_init_device+0x0/0x7f
initcall mce_init_device+0x0/0x7f returned 0 after 0 msecs
calling periodic_mcheck_init+0x0/0x3f
initcall periodic_mcheck_init+0x0/0x3f returned 0 after 0 msecs
calling thermal_throttle_init_device+0x0/0x7b
initcall thermal_throttle_init_device+0x0/0x7b returned 0 after 0 msecs
calling microcode_init+0x0/0xaf
IA-32 Microcode Update Driver: v1.14a <[email protected]>
initcall microcode_init+0x0/0xaf returned 0 after 5 msecs
calling init_lapic_sysfs+0x0/0x2f
initcall init_lapic_sysfs+0x0/0x2f returned 0 after 0 msecs
calling ioapic_init_sysfs+0x0/0x99
initcall ioapic_init_sysfs+0x0/0x99 returned 0 after 0 msecs
calling uv_ptc_init+0x0/0x75
initcall uv_ptc_init+0x0/0x75 returned 0 after 0 msecs
calling uv_bau_init+0x0/0x5d0
initcall uv_bau_init+0x0/0x5d0 returned 0 after 0 msecs
calling audit_classes_init+0x0/0xaf
initcall audit_classes_init+0x0/0xaf returned 0 after 0 msecs
calling aes_init+0x0/0xc
initcall aes_init+0x0/0xc returned 0 after 0 msecs
calling init+0x0/0xc
initcall init+0x0/0xc returned 0 after 0 msecs
calling init_vdso_vars+0x0/0x224
initcall init_vdso_vars+0x0/0x224 returned 0 after 0 msecs
calling ia32_binfmt_init+0x0/0x14
initcall ia32_binfmt_init+0x0/0x14 returned 0 after 0 msecs
calling sysenter_setup+0x0/0x2ac
initcall sysenter_setup+0x0/0x2ac returned 0 after 0 msecs
calling init_aout_binfmt+0x0/0xc
initcall init_aout_binfmt+0x0/0xc returned 0 after 0 msecs
calling create_proc_profile+0x0/0x319
initcall create_proc_profile+0x0/0x319 returned 0 after 0 msecs
calling ioresources_init+0x0/0x3c
initcall ioresources_init+0x0/0x3c returned 0 after 0 msecs
calling uid_cache_init+0x0/0x6f
initcall uid_cache_init+0x0/0x6f returned 0 after 0 msecs
calling init_posix_timers+0x0/0xa6
initcall init_posix_timers+0x0/0xa6 returned 0 after 0 msecs
calling init_posix_cpu_timers+0x0/0xc6
initcall init_posix_cpu_timers+0x0/0xc6 returned 0 after 0 msecs
calling nsproxy_cache_init+0x0/0x2d
initcall nsproxy_cache_init+0x0/0x2d returned 0 after 0 msecs
calling timekeeping_init_device+0x0/0x23
initcall timekeeping_init_device+0x0/0x23 returned 0 after 0 msecs
calling init_clocksource_sysfs+0x0/0x51
initcall init_clocksource_sysfs+0x0/0x51 returned 0 after 0 msecs
calling init_timer_list_procfs+0x0/0x2c
initcall init_timer_list_procfs+0x0/0x2c returned 0 after 0 msecs
calling futex_init+0x0/0x67
initcall futex_init+0x0/0x67 returned 0 after 0 msecs
calling proc_dma_init+0x0/0x22
initcall proc_dma_init+0x0/0x22 returned 0 after 0 msecs
calling kallsyms_init+0x0/0x25
initcall kallsyms_init+0x0/0x25 returned 0 after 0 msecs
calling pid_namespaces_init+0x0/0x2d
initcall pid_namespaces_init+0x0/0x2d returned 0 after 0 msecs
calling audit_init+0x0/0x126
audit: initializing netlink socket (disabled)
type=2000 audit(1216650963.392:1): initialized
initcall audit_init+0x0/0x126 returned 0 after 8 msecs
calling relay_init+0x0/0x14
initcall relay_init+0x0/0x14 returned 0 after 0 msecs
calling utsname_sysctl_init+0x0/0x14
initcall utsname_sysctl_init+0x0/0x14 returned 0 after 0 msecs
calling init_stack_trace+0x0/0xc
Testing tracer sysprof: PASSED
initcall init_stack_trace+0x0/0xc returned 0 after 98 msecs
calling init_per_zone_pages_min+0x0/0x45
initcall init_per_zone_pages_min+0x0/0x45 returned 0 after 0 msecs
calling pdflush_init+0x0/0x13
initcall pdflush_init+0x0/0x13 returned 0 after 0 msecs
calling kswapd_init+0x0/0x5e
initcall kswapd_init+0x0/0x5e returned 0 after 0 msecs
calling setup_vmstat+0x0/0x42
initcall setup_vmstat+0x0/0x42 returned 0 after 0 msecs
calling init_tmpfs+0x0/0x3d
initcall init_tmpfs+0x0/0x3d returned 0 after 0 msecs
calling fasync_init+0x0/0x2a
initcall fasync_init+0x0/0x2a returned 0 after 0 msecs
calling aio_setup+0x0/0x6e
initcall aio_setup+0x0/0x6e returned 0 after 0 msecs
calling inotify_setup+0x0/0xd
initcall inotify_setup+0x0/0xd returned 0 after 0 msecs
calling inotify_user_setup+0x0/0xb8
initcall inotify_user_setup+0x0/0xb8 returned 0 after 0 msecs
calling init_sys32_ioctl+0x0/0x85
initcall init_sys32_ioctl+0x0/0x85 returned 0 after 0 msecs
calling init_mbcache+0x0/0x14
initcall init_mbcache+0x0/0x14 returned 0 after 0 msecs
calling dquot_init+0x0/0xde
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
initcall dquot_init+0x0/0xde returned 0 after 7 msecs
calling init_v2_quota_format+0x0/0xc
initcall init_v2_quota_format+0x0/0xc returned 0 after 0 msecs
calling dnotify_init+0x0/0x2a
initcall dnotify_init+0x0/0x2a returned 0 after 0 msecs
calling configfs_init+0x0/0xa9
initcall configfs_init+0x0/0xa9 returned 0 after 0 msecs
calling init_devpts_fs+0x0/0x35
initcall init_devpts_fs+0x0/0x35 returned 0 after 0 msecs
calling init_ext3_fs+0x0/0x60
initcall init_ext3_fs+0x0/0x60 returned 0 after 0 msecs
calling journal_init+0x0/0xc5
initcall journal_init+0x0/0xc5 returned 0 after 0 msecs
calling init_ext2_fs+0x0/0x60
initcall init_ext2_fs+0x0/0x60 returned 0 after 0 msecs
calling init_ramfs_fs+0x0/0xc
initcall init_ramfs_fs+0x0/0xc returned 0 after 0 msecs
calling init_fat_fs+0x0/0x45
initcall init_fat_fs+0x0/0x45 returned 0 after 0 msecs
calling init_msdos_fs+0x0/0xc
initcall init_msdos_fs+0x0/0xc returned 0 after 0 msecs
calling init_vfat_fs+0x0/0xc
initcall init_vfat_fs+0x0/0xc returned 0 after 0 msecs
calling init_iso9660_fs+0x0/0x50
initcall init_iso9660_fs+0x0/0x50 returned 0 after 0 msecs
calling init_hfsplus_fs+0x0/0x53
initcall init_hfsplus_fs+0x0/0x53 returned 0 after 0 msecs
calling init_hfs_fs+0x0/0x53
initcall init_hfs_fs+0x0/0x53 returned 0 after 0 msecs
calling vxfs_init+0x0/0x4f
initcall vxfs_init+0x0/0x4f returned 0 after 0 msecs
calling init_nls_cp437+0x0/0xc
initcall init_nls_cp437+0x0/0xc returned 0 after 0 msecs
calling init_nls_cp775+0x0/0xc
initcall init_nls_cp775+0x0/0xc returned 0 after 0 msecs
calling init_nls_cp850+0x0/0xc
initcall init_nls_cp850+0x0/0xc returned 0 after 0 msecs
calling init_nls_cp852+0x0/0xc
initcall init_nls_cp852+0x0/0xc returned 0 after 0 msecs
calling init_nls_cp855+0x0/0xc
initcall init_nls_cp855+0x0/0xc returned 0 after 0 msecs
calling init_nls_cp857+0x0/0xc
initcall init_nls_cp857+0x0/0xc returned 0 after 0 msecs
calling init_nls_cp860+0x0/0xc
initcall init_nls_cp860+0x0/0xc returned 0 after 0 msecs
calling init_nls_cp863+0x0/0xc
initcall init_nls_cp863+0x0/0xc returned 0 after 0 msecs
calling init_nls_cp874+0x0/0xc
initcall init_nls_cp874+0x0/0xc returned 0 after 0 msecs
calling init_nls_cp932+0x0/0xc
initcall init_nls_cp932+0x0/0xc returned 0 after 0 msecs
calling init_nls_euc_jp+0x0/0x4a
initcall init_nls_euc_jp+0x0/0x4a returned 0 after 0 msecs
calling init_nls_cp936+0x0/0xc
initcall init_nls_cp936+0x0/0xc returned 0 after 0 msecs
calling init_nls_ascii+0x0/0xc
initcall init_nls_ascii+0x0/0xc returned 0 after 0 msecs
calling init_nls_iso8859_1+0x0/0xc
initcall init_nls_iso8859_1+0x0/0xc returned 0 after 0 msecs
calling init_nls_iso8859_5+0x0/0xc
initcall init_nls_iso8859_5+0x0/0xc returned 0 after 0 msecs
calling init_nls_iso8859_13+0x0/0xc
initcall init_nls_iso8859_13+0x0/0xc returned 0 after 0 msecs
calling init_nls_iso8859_14+0x0/0xc
initcall init_nls_iso8859_14+0x0/0xc returned 0 after 0 msecs
calling init_nls_koi8_r+0x0/0xc
initcall init_nls_koi8_r+0x0/0xc returned 0 after 0 msecs
calling init_nls_utf8+0x0/0x1f
initcall init_nls_utf8+0x0/0x1f returned 0 after 0 msecs
calling init_sysv_fs+0x0/0x45
initcall init_sysv_fs+0x0/0x45 returned 0 after 0 msecs
calling init_hpfs_fs+0x0/0x50
initcall init_hpfs_fs+0x0/0x50 returned 0 after 0 msecs
calling init_ntfs_fs+0x0/0x1c7
NTFS driver 2.1.29 [Flags: R/O].
initcall init_ntfs_fs+0x0/0x1c7 returned 0 after 2 msecs
calling init_qnx4_fs+0x0/0x62
QNX4 filesystem 0.2.3 registered.
initcall init_qnx4_fs+0x0/0x62 returned 0 after 3 msecs
calling init_adfs_fs+0x0/0x50
initcall init_adfs_fs+0x0/0x50 returned 0 after 0 msecs
calling fuse_init+0x0/0x11b
fuse init (API version 7.9)
initcall fuse_init+0x0/0x11b returned 0 after 2 msecs
calling init_jfs_fs+0x0/0x1db
JFS: nTxBlock = 7645, nTxLock = 61161
initcall init_jfs_fs+0x0/0x1db returned 0 after 8 msecs
calling init_befs_fs+0x0/0x71
BeFS version: 0.9.3
initcall init_befs_fs+0x0/0x71 returned 0 after 1 msecs
calling ocfs2_init+0x0/0x28b
OCFS2 1.5.0
initcall ocfs2_init+0x0/0x28b returned 0 after 1 msecs
calling ocfs2_stack_glue_init+0x0/0x8b
initcall ocfs2_stack_glue_init+0x0/0x8b returned 0 after 0 msecs
calling init_o2nm+0x0/0x90
OCFS2 Node Manager 1.5.0
initcall init_o2nm+0x0/0x90 returned 0 after 2 msecs
calling init_gfs2_fs+0x0/0x16e
GFS2 (built Jul 21 2008 16:13:18) installed
initcall init_gfs2_fs+0x0/0x16e returned 0 after 5 msecs
calling init_mqueue_fs+0x0/0xbc
initcall init_mqueue_fs+0x0/0xbc returned 0 after 0 msecs
calling key_proc_init+0x0/0x59
initcall key_proc_init+0x0/0x59 returned 0 after 0 msecs
calling selinux_nf_ip_init+0x0/0x4c
SELinux: Registering netfilter hooks
initcall selinux_nf_ip_init+0x0/0x4c returned 0 after 3 msecs
calling init_sel_fs+0x0/0x5e
initcall init_sel_fs+0x0/0x5e returned 0 after 0 msecs
calling selnl_init+0x0/0x4d
initcall selnl_init+0x0/0x4d returned 0 after 0 msecs
calling sel_netif_init+0x0/0x66
initcall sel_netif_init+0x0/0x66 returned 0 after 0 msecs
calling sel_netnode_init+0x0/0x78
initcall sel_netnode_init+0x0/0x78 returned 0 after 0 msecs
calling sel_netport_init+0x0/0x78
initcall sel_netport_init+0x0/0x78 returned 0 after 0 msecs
calling aurule_init+0x0/0x37
initcall aurule_init+0x0/0x37 returned 0 after 0 msecs
calling crypto_algapi_init+0x0/0xd
initcall crypto_algapi_init+0x0/0xd returned 0 after 0 msecs
calling blkcipher_module_init+0x0/0x20
initcall blkcipher_module_init+0x0/0x20 returned 0 after 0 msecs
calling seqiv_module_init+0x0/0xc
initcall seqiv_module_init+0x0/0xc returned 0 after 0 msecs
calling cryptomgr_init+0x0/0xc
initcall cryptomgr_init+0x0/0xc returned 0 after 0 msecs
calling hmac_module_init+0x0/0xc
initcall hmac_module_init+0x0/0xc returned 0 after 0 msecs
calling md5_mod_init+0x0/0xc
initcall md5_mod_init+0x0/0xc returned 0 after 0 msecs
calling rmd128_mod_init+0x0/0xc
initcall rmd128_mod_init+0x0/0xc returned 0 after 0 msecs
calling rmd160_mod_init+0x0/0xc
initcall rmd160_mod_init+0x0/0xc returned 0 after 0 msecs
calling rmd256_mod_init+0x0/0xc
initcall rmd256_mod_init+0x0/0xc returned 0 after 0 msecs
calling rmd320_mod_init+0x0/0xc
initcall rmd320_mod_init+0x0/0xc returned 0 after 0 msecs
calling sha1_generic_mod_init+0x0/0xc
initcall sha1_generic_mod_init+0x0/0xc returned 0 after 0 msecs
calling sha256_generic_mod_init+0x0/0x35
initcall sha256_generic_mod_init+0x0/0x35 returned 0 after 0 msecs
calling sha512_generic_mod_init+0x0/0x35
initcall sha512_generic_mod_init+0x0/0x35 returned 0 after 0 msecs
calling crypto_ecb_module_init+0x0/0xc
initcall crypto_ecb_module_init+0x0/0xc returned 0 after 0 msecs
calling crypto_cbc_module_init+0x0/0xc
initcall crypto_cbc_module_init+0x0/0xc returned 0 after 0 msecs
calling crypto_module_init+0x0/0xc
initcall crypto_module_init+0x0/0xc returned 0 after 0 msecs
calling crypto_ctr_module_init+0x0/0x35
initcall crypto_ctr_module_init+0x0/0x35 returned 0 after 0 msecs
calling crypto_ccm_module_init+0x0/0x53
initcall crypto_ccm_module_init+0x0/0x53 returned 0 after 0 msecs
calling cryptd_init+0x0/0xa8
initcall cryptd_init+0x0/0xa8 returned 0 after 0 msecs
calling des_generic_mod_init+0x0/0x35
initcall des_generic_mod_init+0x0/0x35 returned 0 after 0 msecs
calling fcrypt_mod_init+0x0/0xc
initcall fcrypt_mod_init+0x0/0xc returned 0 after 0 msecs
calling serpent_mod_init+0x0/0x35
initcall serpent_mod_init+0x0/0x35 returned 0 after 0 msecs
calling aes_init+0x0/0x2c5
initcall aes_init+0x0/0x2c5 returned 0 after 0 msecs
calling cast5_mod_init+0x0/0xc
initcall cast5_mod_init+0x0/0xc returned 0 after 0 msecs
calling arc4_init+0x0/0xc
initcall arc4_init+0x0/0xc returned 0 after 0 msecs
calling khazad_mod_init+0x0/0xc
initcall khazad_mod_init+0x0/0xc returned 0 after 0 msecs
calling seed_init+0x0/0xc
initcall seed_init+0x0/0xc returned 0 after 0 msecs
calling salsa20_generic_mod_init+0x0/0xc
initcall salsa20_generic_mod_init+0x0/0xc returned 0 after 0 msecs
calling deflate_mod_init+0x0/0xc
initcall deflate_mod_init+0x0/0xc returned 0 after 0 msecs
calling crc32c_mod_init+0x0/0x35
initcall crc32c_mod_init+0x0/0x35 returned 0 after 0 msecs
calling crypto_authenc_module_init+0x0/0xc
initcall crypto_authenc_module_init+0x0/0xc returned 0 after 0 msecs
calling bsg_init+0x0/0x11a
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
initcall bsg_init+0x0/0x11a returned 0 after 5 msecs
calling noop_init+0x0/0x14
io scheduler noop registered (default)
initcall noop_init+0x0/0x14 returned 0 after 3 msecs
calling percpu_counter_startup+0x0/0x14
initcall percpu_counter_startup+0x0/0x14 returned 0 after 0 msecs
calling pci_init+0x0/0x2b
pci 0000:01:00.0: Boot video device
initcall pci_init+0x0/0x2b returned 0 after 3 msecs
calling pci_proc_init+0x0/0x60
initcall pci_proc_init+0x0/0x60 returned 0 after 0 msecs
calling ltv350qv_init+0x0/0xc
initcall ltv350qv_init+0x0/0xc returned 0 after 0 msecs
calling corgibl_init+0x0/0xc
initcall corgibl_init+0x0/0xc returned 0 after 0 msecs
calling acpi_reserve_resources+0x0/0xeb
initcall acpi_reserve_resources+0x0/0xeb returned 0 after 0 msecs
calling acpi_ac_init+0x0/0x28
initcall acpi_ac_init+0x0/0x28 returned 0 after 0 msecs
calling acpi_battery_init+0x0/0x28
initcall acpi_battery_init+0x0/0x28 returned 0 after 0 msecs
calling acpi_button_init+0x0/0x5e
input: Power Button (FF) as /class/input/input0
ACPI: Power Button (FF) [PWRF]
input: Sleep Button (CM) as /class/input/input1
ACPI: Sleep Button (CM) [SLPB]
initcall acpi_button_init+0x0/0x5e returned 0 after 14 msecs
calling acpi_fan_init+0x0/0x5e
initcall acpi_fan_init+0x0/0x5e returned 0 after 0 msecs
calling irqrouter_init_sysfs+0x0/0x37
initcall irqrouter_init_sysfs+0x0/0x37 returned 0 after 0 msecs
calling acpi_processor_init+0x0/0xf6
ACPI: ACPI0007:00 is registered as cooling_device0
ACPI: ACPI0007:01 is registered as cooling_device1
initcall acpi_processor_init+0x0/0xf6 returned 0 after 9 msecs
calling acpi_container_init+0x0/0x43
initcall acpi_container_init+0x0/0x43 returned 0 after 1 msecs
calling toshiba_acpi_init+0x0/0x17d
initcall toshiba_acpi_init+0x0/0x17d returned -19 after 0 msecs
calling acpi_smb_hc_init+0x0/0x18
initcall acpi_smb_hc_init+0x0/0x18 returned 0 after 0 msecs
calling acpi_sbs_init+0x0/0x28
initcall acpi_sbs_init+0x0/0x28 returned 0 after 0 msecs
calling rand_initialize+0x0/0x2c
initcall rand_initialize+0x0/0x2c returned 0 after 0 msecs
calling tty_init+0x0/0x1c5
initcall tty_init+0x0/0x1c5 returned 0 after 25 msecs
calling pty_init+0x0/0x46e
initcall pty_init+0x0/0x46e returned 0 after 25 msecs
calling raw_init+0x0/0xca
initcall raw_init+0x0/0xca returned 0 after 0 msecs
calling r3964_init+0x0/0x3a
r3964: Philips r3964 Driver $Revision: 1.10 $
initcall r3964_init+0x0/0x3a returned 0 after 3 msecs
calling applicom_init+0x0/0x4cc
Applicom driver: $Id: ac.c,v 1.30 2000/03/22 16:03:57 dwmw2 Exp $
ac.o: No PCI boards found.
ac.o: For an ISA board you must supply memory and irq parameters.
initcall applicom_init+0x0/0x4cc returned -6 after 13 msecs
initcall applicom_init+0x0/0x4cc returned with error code -6
calling rtc_init+0x0/0xa6
Real Time Clock Driver v1.12ac
initcall rtc_init+0x0/0xa6 returned 0 after 2 msecs
calling hpet_init+0x0/0x60
initcall hpet_init+0x0/0x60 returned 0 after 0 msecs
calling nvram_init+0x0/0x80
Non-volatile memory driver v1.2
initcall nvram_init+0x0/0x80 returned 0 after 2 msecs
calling i8k_init+0x0/0x154
initcall i8k_init+0x0/0x154 returned -19 after 0 msecs
calling mod_init+0x0/0x1da
initcall mod_init+0x0/0x1da returned -19 after 0 msecs
calling ppdev_init+0x0/0xa7
ppdev: user-space parallel port driver
initcall ppdev_init+0x0/0xa7 returned 0 after 3 msecs
calling tlclk_init+0x0/0x1ae
telclk_interrup = 0xf non-mcpbl0010 hw.
initcall tlclk_init+0x0/0x1ae returned -6 after 3 msecs
initcall tlclk_init+0x0/0x1ae returned with error code -6
calling agp_init+0x0/0x26
Linux agpgart interface v0.103
initcall agp_init+0x0/0x26 returned 0 after 2 msecs
calling agp_intel_init+0x0/0x24
initcall agp_intel_init+0x0/0x24 returned 0 after 0 msecs
calling agp_via_init+0x0/0x24
initcall agp_via_init+0x0/0x24 returned 0 after 0 msecs
calling init_atmel+0x0/0x17c
initcall init_atmel+0x0/0x17c returned -19 after 0 msecs
calling drm_core_init+0x0/0xef
[drm] Initialized drm 1.1.0 20060810
initcall drm_core_init+0x0/0xef returned 0 after 3 msecs
calling tdfx_init+0x0/0xc
initcall tdfx_init+0x0/0xc returned 0 after 0 msecs
calling radeon_init+0x0/0x18
pci 0000:01:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:01:00.0 to 64
[drm] Initialized radeon 1.29.0 20080528 on minor 0
initcall radeon_init+0x0/0x18 returned 0 after 15 msecs
calling i810_init+0x0/0x18
initcall i810_init+0x0/0x18 returned 0 after 0 msecs
calling i915_init+0x0/0x18
initcall i915_init+0x0/0x18 returned 0 after 0 msecs
calling sis_init+0x0/0x18
initcall sis_init+0x0/0x18 returned 0 after 0 msecs
calling via_init+0x0/0x22
initcall via_init+0x0/0x22 returned 0 after 0 msecs
calling serial8250_init+0x0/0x122
Serial: 8250/16550 driver4 ports, IRQ sharing disabled
�serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
initcall serial8250_init+0x0/0x122 returned 0 after 251 msecs
calling jsm_init_module+0x0/0x3e
initcall jsm_init_module+0x0/0x3e returned 0 after 0 msecs
calling parport_default_proc_register+0x0/0x1b
initcall parport_default_proc_register+0x0/0x1b returned 0 after 0 msecs
calling parport_pc_init+0x0/0x338
parport_pc 00:08: reported by Plug and Play ACPI
parport0: PC-style at 0x378 (0x778), irq 7 [PCSPP(,...)]
initcall parport_pc_init+0x0/0x338 returned 0 after 10 msecs
calling parport_ax88796_init+0x0/0xc
initcall parport_ax88796_init+0x0/0xc returned 0 after 0 msecs
calling topology_sysfs_init+0x0/0x41
initcall topology_sysfs_init+0x0/0x41 returned 0 after 0 msecs
calling loop_init+0x0/0x196
loop: module loaded
initcall loop_init+0x0/0x196 returned 0 after 2 msecs
calling cpqarray_init+0x0/0x251
Compaq SMART2 Driver (v 2.6.0)
initcall cpqarray_init+0x0/0x251 returned 0 after 2 msecs
calling cciss_init+0x0/0x28
HP CISS Driver (v 3.6.20)
initcall cciss_init+0x0/0x28 returned 0 after 2 msecs
calling ub_init+0x0/0x6a
usbcore: registered new interface driver ub
initcall ub_init+0x0/0x6a returned 0 after 3 msecs
calling pasic3_base_init+0x0/0x13
initcall pasic3_base_init+0x0/0x13 returned -19 after 0 msecs
calling e1000_init_module+0x0/0x61
e1000e: Intel(R) PRO/1000 Network Driver - 0.3.3.3-k2
e1000e: Copyright (c) 1999-2008 Intel Corporation.
e1000e 0000:04:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
PCI: Setting latency timer of device 0000:04:00.0 to 64
eth0: (PCI Express:2.5GB/s:Width x1) 00:16:76:ab:6e:84
eth0: Intel(R) PRO/1000 Network Connection
eth0: MAC: 2, PHY: 2, PBA No: ffffff-0ff
initcall e1000_init_module+0x0/0x61 returned 0 after 141 msecs
calling ixgb_init_module+0x0/0x4b
Intel(R) PRO/10GbE Network Driver - version 1.0.135-k2-NAPI
Copyright (c) 1999-2008 Intel Corporation.
initcall ixgb_init_module+0x0/0x4b returned 0 after 8 msecs
calling ipg_init_module+0x0/0x15
initcall ipg_init_module+0x0/0x15 returned 0 after 0 msecs
calling cxgb3_init_module+0x0/0x20
initcall cxgb3_init_module+0x0/0x20 returned 0 after 0 msecs
calling vcan_init_module+0x0/0x36
vcan: Virtual CAN interface driver
initcall vcan_init_module+0x0/0x36 returned 0 after 3 msecs
calling atl1_init_module+0x0/0x15
initcall atl1_init_module+0x0/0x15 returned 0 after 0 msecs
calling plip_init+0x0/0x5d
NET3 PLIP version 2.4-parport [email protected]
plip0: Parallel port at 0x378, using IRQ 7.
initcall plip_init+0x0/0x5d returned 0 after 8 msecs
calling gem_init+0x0/0x15
initcall gem_init+0x0/0x15 returned 0 after 0 msecs
calling vortex_init+0x0/0xa9
initcall vortex_init+0x0/0xa9 returned 0 after 0 msecs
calling ne2k_pci_init+0x0/0x15
initcall ne2k_pci_init+0x0/0x15 returned 0 after 0 msecs
calling e100_init_module+0x0/0x5c
e100: Intel(R) PRO/100 Network Driver, 3.5.23-k4-NAPI
e100: Copyright(c) 1999-2006 Intel Corporation
initcall e100_init_module+0x0/0x5c returned 0 after 8 msecs
calling tlan_probe+0x0/0xda
ThunderLAN driver v1.15
TLAN: 0 devices installed, PCI: 0 EISA: 0
initcall tlan_probe+0x0/0xda returned -19 after 5 msecs
calling epic_init+0x0/0x15
initcall epic_init+0x0/0x15 returned 0 after 0 msecs
calling sis190_init_module+0x0/0x15
initcall sis190_init_module+0x0/0x15 returned 0 after 0 msecs
calling r6040_init+0x0/0x15
initcall r6040_init+0x0/0x15 returned 0 after 0 msecs
calling yellowfin_init+0x0/0x15
initcall yellowfin_init+0x0/0x15 returned 0 after 0 msecs
calling natsemi_init_mod+0x0/0x15
initcall natsemi_init_mod+0x0/0x15 returned 0 after 0 msecs
calling ns83820_init+0x0/0x28
ns83820.c: National Semiconductor DP83820 10/100/1000 driver.
initcall ns83820_init+0x0/0x28 returned 0 after 5 msecs
calling tg3_init+0x0/0x15
initcall tg3_init+0x0/0x15 returned 0 after 0 msecs
calling bnx2_init+0x0/0x15
initcall bnx2_init+0x0/0x15 returned 0 after 0 msecs
calling bnx2x_init+0x0/0x15
initcall bnx2x_init+0x0/0x15 returned 0 after 0 msecs
calling skge_init_module+0x0/0x15
initcall skge_init_module+0x0/0x15 returned 0 after 0 msecs
calling sky2_init_module+0x0/0x49
initcall sky2_init_module+0x0/0x49 returned 0 after 0 msecs
calling rhine_init+0x0/0x39
initcall rhine_init+0x0/0x39 returned 0 after 0 msecs
calling starfire_init+0x0/0x15
initcall starfire_init+0x0/0x15 returned 0 after 0 msecs
calling marvell_init+0x0/0x5e
initcall marvell_init+0x0/0x5e returned 0 after 0 msecs
calling cicada_init+0x0/0x35
initcall cicada_init+0x0/0x35 returned 0 after 0 msecs
calling lxt_init+0x0/0x35
initcall lxt_init+0x0/0x35 returned 0 after 0 msecs
calling qs6612_init+0x0/0xc
initcall qs6612_init+0x0/0xc returned 0 after 0 msecs
calling ip175c_init+0x0/0xc
initcall ip175c_init+0x0/0xc returned 0 after 0 msecs
calling fixed_mdio_bus_init+0x0/0x9e
Fixed MDIO Bus: probed
initcall fixed_mdio_bus_init+0x0/0x9e returned 0 after 2 msecs
calling sundance_init+0x0/0x15
initcall sundance_init+0x0/0x15 returned 0 after 0 msecs
calling hamachi_init+0x0/0x15
initcall hamachi_init+0x0/0x15 returned 0 after 0 msecs
calling net_olddevs_init+0x0/0x95
initcall net_olddevs_init+0x0/0x95 returned 0 after 0 msecs
calling b44_init+0x0/0x59
initcall b44_init+0x0/0x59 returned 0 after 0 msecs
calling init_nic+0x0/0x15
initcall init_nic+0x0/0x15 returned 0 after 0 msecs
calling ql3xxx_init_module+0x0/0x15
initcall ql3xxx_init_module+0x0/0x15 returned 0 after 0 msecs
calling dummy_init_module+0x0/0xb2
initcall dummy_init_module+0x0/0xb2 returned 0 after 0 msecs
calling macvlan_init_module+0x0/0x49
initcall macvlan_init_module+0x0/0x49 returned 0 after 0 msecs
calling dfx_init+0x0/0x15
initcall dfx_init+0x0/0x15 returned 0 after 0 msecs
calling rtl8139_init_module+0x0/0x15
initcall rtl8139_init_module+0x0/0x15 returned 0 after 0 msecs
calling atp_init_module+0x0/0x93
atp.c:v1.09=ac 2002/10/01 Donald Becker <[email protected]>
initcall atp_init_module+0x0/0x93 returned -19 after 5 msecs
calling eql_init_module+0x0/0x5b
Equalizer2002: Simon Janes ([email protected]) and David S. Miller ([email protected])
initcall eql_init_module+0x0/0x5b returned 0 after 7 msecs
calling tun_init+0x0/0x96
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
initcall tun_init+0x0/0x96 returned 0 after 8 msecs
calling rio_init+0x0/0x15
initcall rio_init+0x0/0x15 returned 0 after 0 msecs
calling s2io_starter+0x0/0x15
initcall s2io_starter+0x0/0x15 returned 0 after 0 msecs
calling xl_pci_init+0x0/0x15
initcall xl_pci_init+0x0/0x15 returned 0 after 0 msecs
calling init_dlci+0x0/0x35
DLCI driver v0.35, 4 Jan 1997, [email protected].
initcall init_dlci+0x0/0x35 returned 0 after 4 msecs
calling usb_rtl8150_init+0x0/0x28
rtl8150: rtl8150 based usb-ethernet driver v0.6.2 (2004/08/27)
usbcore: registered new interface driver rtl8150
initcall usb_rtl8150_init+0x0/0x28 returned 0 after 9 msecs
calling cdc_init+0x0/0x15
usbcore: registered new interface driver cdc_ether
initcall cdc_init+0x0/0x15 returned 0 after 4 msecs
calling dm9601_init+0x0/0x15
usbcore: registered new interface driver dm9601
initcall dm9601_init+0x0/0x15 returned 0 after 4 msecs
calling usbnet_init+0x0/0x15
usbcore: registered new interface driver gl620a
initcall usbnet_init+0x0/0x15 returned 0 after 4 msecs
calling plusb_init+0x0/0x15
usbcore: registered new interface driver plusb
initcall plusb_init+0x0/0x15 returned 0 after 4 msecs
calling rndis_init+0x0/0x15
usbcore: registered new interface driver rndis_host
initcall rndis_init+0x0/0x15 returned 0 after 4 msecs
calling cdc_subset_init+0x0/0x15
usbcore: registered new interface driver cdc_subset
initcall cdc_subset_init+0x0/0x15 returned 0 after 4 msecs
calling mcs7830_init+0x0/0x15
usbcore: registered new interface driver MOSCHIP usb-ethernet driver
initcall mcs7830_init+0x0/0x15 returned 0 after 5 msecs
calling usbnet_init+0x0/0x2b
initcall usbnet_init+0x0/0x2b returned 0 after 0 msecs
calling ipw2100_init+0x0/0x65
ipw2100: Intel(R) PRO/Wireless 2100 Network Driver, git-1.2.2
ipw2100: Copyright(c) 2003-2006 Intel Corporation
initcall ipw2100_init+0x0/0x65 returned 0 after 9 msecs
calling ipw_init+0x0/0x7f
ipw2200: Intel(R) PRO/Wireless 2200/2915 Network Driver, 1.2.2kd
ipw2200: Copyright(c) 2003-2006 Intel Corporation
initcall ipw_init+0x0/0x7f returned 0 after 9 msecs
calling init_orinoco+0x0/0x1d
orinoco 0.15 (David Gibson <[email protected]>, Pavel Roskin <[email protected]>, et al)
initcall init_orinoco+0x0/0x1d returned 0 after 8 msecs
calling init_hermes+0x0/0x3
initcall init_hermes+0x0/0x3 returned 0 after 0 msecs
calling orinoco_plx_init+0x0/0x2f
orinoco_plx 0.15 (Pavel Roskin <[email protected]>, David Gibson <[email protected]>, Daniel Barlow <[email protected]>)
initcall orinoco_plx_init+0x0/0x2f returned 0 after 10 msecs
calling orinoco_nortel_init+0x0/0x2f
orinoco_nortel 0.15 (Tobias Hoffmann & Christoph Jungegger <[email protected]>)
initcall orinoco_nortel_init+0x0/0x2f returned 0 after 6 msecs
calling airo_init_module+0x0/0xe3
airo(): Probing for PCI adapters
airo(): Finished probing for PCI adapters
initcall airo_init_module+0x0/0xe3 returned 0 after 6 msecs
calling prism54_module_init+0x0/0x36
Loaded prism54 driver, version 1.2
initcall prism54_module_init+0x0/0x36 returned 0 after 3 msecs
calling b43_init+0x0/0x42
Broadcom 43xx driver loaded [ Features: PL, Firmware-ID: FW13 ]
initcall b43_init+0x0/0x42 returned 0 after 5 msecs
calling usb_init+0x0/0xad
zd1211rw usb_init()
usbcore: registered new interface driver zd1211rw
zd1211rw initialized
initcall usb_init+0x0/0xad returned 0 after 8 msecs
calling rndis_wlan_init+0x0/0x15
usbcore: registered new interface driver rndis_wlan
initcall rndis_wlan_init+0x0/0x15 returned 0 after 4 msecs
calling zd1201_init+0x0/0x15
usbcore: registered new interface driver zd1201
initcall zd1201_init+0x0/0x15 returned 0 after 4 msecs
calling lbs_init_module+0x0/0x38
initcall lbs_init_module+0x0/0x38 returned 0 after 0 msecs
calling if_sdio_init_module+0x0/0x2d
libertas_sdio: Libertas SDIO driver
libertas_sdio: Copyright Pierre Ossman
initcall if_sdio_init_module+0x0/0x2d returned 0 after 6 msecs
calling rtl8180_init+0x0/0x15
initcall rtl8180_init+0x0/0x15 returned 0 after 0 msecs
calling rtl8187_init+0x0/0x15
usbcore: registered new interface driver rtl8187
initcall rtl8187_init+0x0/0x15 returned 0 after 4 msecs
calling iwl4965_init+0x0/0x6c
iwl4965: Intel(R) Wireless WiFi Link 4965AGN driver for Linux, 1.3.27kd
iwl4965: Copyright(c) 2003-2008 Intel Corporation
initcall iwl4965_init+0x0/0x6c returned 0 after 10 msecs
calling init_mac80211_hwsim+0x0/0x2f0
mac80211_hwsim: Initializing radio 0
phy0: Failed to select rate control algorithm
phy0: Failed to initialize rate control algorithm
mac80211_hwsim: ieee80211_register_hw failed (-2)
BUG: unable to handle kernel NULL pointer dereference at 0000000000000370
IP: [<ffffffff808be0c2>] rollback_registered+0x37/0xfb
PGD 0
Oops: 0000 [1] SMP
CPU 1
Pid: 1, comm: swapper Not tainted 2.6.26-05253-g14b395e #21308
RIP: 0010:[<ffffffff808be0c2>] [<ffffffff808be0c2>] rollback_registered+0x37/0xfb
RSP: 0018:ffff88003f83fe00 EFLAGS: 00010212
RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffff88003d078ed8
RDX: ffffffff8095de3d RSI: 0000000000000046 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8800042facc0
R10: 0000000000000000 R11: ffffffff804021ae R12: 0000000000000000
R13: ffff88003d0799a0 R14: 0000000000000000 R15: 0000000000000008
FS: 0000000000000000(0000) GS:ffff88003f829160(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000370 CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 1, threadinfo ffff88003f83e000, task ffff88003f824000)
Stack: 0000000000000000 ffffffff808be1b8 ffff88003d0782c0 ffffffff809621ab
ffff88003d0799a0 ffffffff805f1e5d 00000000fffffffe ffff88003d0782c0
ffff88003d0799e0 ffffffff811e83e4 ffff88003f83fea0 ffffffff80248f12
Call Trace:
[<ffffffff808be1b8>] unregister_netdevice+0x32/0x71
[<ffffffff809621ab>] ieee80211_unregister_hw+0x35/0xd4
[<ffffffff805f1e5d>] mac80211_hwsim_free+0x1d/0x6a
[<ffffffff811e83e4>] init_mac80211_hwsim+0x2df/0x2f0
[<ffffffff80248f12>] getnstimeofday+0x38/0x95
[<ffffffff80246f48>] ktime_get_ts+0x21/0x49
[<ffffffff811e8105>] init_mac80211_hwsim+0x0/0x2f0
[<ffffffff811cb8c9>] kernel_init+0x143/0x295
[<ffffffff80209b79>] __switch_to+0xb6/0x3bb
[<ffffffff8020c399>] child_rip+0xa/0x11
[<ffffffff811cb786>] kernel_init+0x0/0x295
[<ffffffff8020c38f>] child_rip+0x0/0x11

Code: 04 0f 0b eb fe e8 e3 78 00 00 85 c0 75 1d ba ce 0e 00 00 48 c7 c6 b8 e1 d8 80 48 c7 c7 3b 6c ce 80 e8 24 65 97 ff e8 0e f8 94 ff <8b> 83 70 03 00 00 85 c0 75 24 48 89 de 48 89 da 48 c7 c7 e4 e4
RIP [<ffffffff808be0c2>] rollback_registered+0x37/0xfb
RSP <ffff88003f83fe00>
CR2: 0000000000000370
---[ end trace 4b01b4c1a97dbebf ]---
Kernel panic - not syncing: Attempted to kill init!

2008-07-21 15:05:23

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

* Ingo Molnar <[email protected]> wrote:

> > Pid: 1, comm: swapper Not tainted 2.6.26-tip-00013-g6de15c6-dirty #21290
>
> some more information: find below the same crash with vanilla
> linus/master and no extra patches. The crash site is:

a 32-bit testbox just triggered the same crash too:

calling init_mac80211_hwsim+0x0/0x310
mac80211_hwsim: Initializing radio 0
phy0: Failed to select rate control algorithm
phy0: Failed to initialize rate control algorithm
mac80211_hwsim: ieee80211_register_hw failed (-2)
BUG: unable to handle kernel NULL pointer dereference at 00000298
IP: [<c06efb98>] rollback_registered+0x28/0x120
*pdpt = 0000000000bc9001 *pde = 0000000000000000
Oops: 0000 [#1] PREEMPT SMP

and that system has no wireless so i guess it's just some unregister
inbalance kind of init/deinit buglet.

Ingo

2008-07-21 15:07:38

[permalink] [raw]

Subject: Re: [crash, bisected] Kernel BUG at ffffffff8079afb1 (__netif_schedule())

From: Ingo Molnar <[email protected]>
Date: Mon, 21 Jul 2008 15:30:59 +0200

> -tip testing on latest -git (v2.6.26-5253-g14b395e) triggered the
> following boot crash on a Core2Duo 64-bit testsystem:
>
> ADDRCONF(NETDEV_UP): eth0: link is not ready
> eth0: Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

Should be fixed by:

e1000: resolve tx multiqueue bug

With the recent changes to tx mutiqueue, e1000 was not calling
netif_start_queue() before calling netif_wake_queue().
This causes an oops during loading of the driver.

(Based on commit d55b53fff0c2ddb639dca04c3f5a0854f292d982
("igb/ixgbe/e1000e: resolve tx multiqueue bug").)

Signed-off-by: Ben Hutchings <[email protected]>
Acked-by: Jeff Kirsher <[email protected]>
Signed-off-by: David S. Miller <[email protected]>

diff --git a/drivers/net/e1000/e1000_main.c b/drivers/net/e1000/e1000_main.c
index f8df8bd..cf12b05 100644
--- a/drivers/net/e1000/e1000_main.c
+++ b/drivers/net/e1000/e1000_main.c
@@ -1472,6 +1472,8 @@ e1000_open(struct net_device *netdev)

e1000_irq_enable(adapter);

+ netif_start_queue(netdev);
+
/* fire a link status change interrupt to start the watchdog */
E1000_WRITE_REG(&adapter->hw, ICS, E1000_ICS_LSC);

2008-07-21 15:11:17

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

From: Ingo Molnar <[email protected]>
Date: Mon, 21 Jul 2008 16:30:23 +0200

>
> * Ingo Molnar <[email protected]> wrote:
>
> > Pid: 1, comm: swapper Not tainted 2.6.26-tip-00013-g6de15c6-dirty #21290
>
> some more information: find below the same crash with vanilla
> linus/master and no extra patches. The crash site is:

Johannes/wireless-folks, can you take a look at this?

Thanks.

>
> (gdb) list *0xffffffff808be0c2
> 0xffffffff808be0c2 is in rollback_registered (net/core/dev.c:3793).
> 3788 {
> 3789 BUG_ON(dev_boot_phase);
> 3790 ASSERT_RTNL();
> 3791
> 3792 /* Some devices call without registering for initialization unwind. */
> 3793 if (dev->reg_state == NETREG_UNINITIALIZED) {
> 3794 printk(KERN_DEBUG "unregister_netdevice: device %s/%p never "
> 3795 "was registered\n", dev->name, dev);
> 3796
> 3797 WARN_ON(1);
> (gdb)
>
> Thanks,
>
> Ingo
>
> ----------------------------------->
> Linux version 2.6.26-05253-g14b395e (mingo@dione) (gcc version 4.2.3) #21308 SMP Mon Jul 21 16:14:51 CEST 2008
> Command line: root=/dev/sda1 earlyprintk=vga console=ttyS0,115200 console=tty 5 profile=0 debug initcall_debug apic=debug apic=verbose ignore_loglevel sysrq_always_enabled pci=nomsi
> BIOS-provided physical RAM map:
> BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
> BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
> BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
> BIOS-e820: 0000000000100000 - 000000003ed94000 (usable)
> BIOS-e820: 000000003ed94000 - 000000003ee4e000 (ACPI NVS)
> BIOS-e820: 000000003ee4e000 - 000000003fea2000 (usable)
> BIOS-e820: 000000003fea2000 - 000000003fee9000 (ACPI NVS)
> BIOS-e820: 000000003fee9000 - 000000003feed000 (usable)
> BIOS-e820: 000000003feed000 - 000000003feff000 (ACPI data)
> BIOS-e820: 000000003feff000 - 000000003ff00000 (usable)
> KERNEL supported cpus:
> Intel GenuineIntel
> AMD AuthenticAMD
> Centaur CentaurHauls
> console [earlyvga0] enabled
> debug: ignoring loglevel setting.
> last_pfn = 0x3ff00 max_arch_pfn = 0x3ffffffff
> init_memory_mapping
> 0000000000 - 003fe00000 page 2M
> 003fe00000 - 003ff00000 page 4k
> kernel direct mapping tables up to 3ff00000 @ 8000-b000
> last_map_addr: 3ff00000 end: 3ff00000
> ACPI: RSDP 000FE020, 0014 (r0 INTEL )
> ACPI: RSDT 3FEFDE48, 0050 (r1 INTEL D975XBX 4B9 MSFT 1000013)
> ACPI: FACP 3FEFCF10, 0074 (r1 INTEL D975XBX 4B9 MSFT 1000013)
> ACPI: DSDT 3FEF8010, 3E70 (r1 INTEL D975XBX 4B9 MSFT 1000013)
> ACPI: FACS 3FEDFC40, 0040
> ACPI: APIC 3FEFCE10, 0078 (r1 INTEL D975XBX 4B9 MSFT 1000013)
> ACPI: WDDT 3FEF7F90, 0040 (r1 INTEL D975XBX 4B9 MSFT 1000013)
> ACPI: MCFG 3FEF7F10, 003C (r1 INTEL D975XBX 4B9 MSFT 1000013)
> ACPI: ASF! 3FEFCD10, 00A6 (r32 INTEL D975XBX 4B9 MSFT 1000013)
> ACPI: HPET 3FEF7E90, 0038 (r1 INTEL D975XBX 4B9 MSFT 1000013)
> ACPI: SSDT 3FEFDC10, 01BC (r1 INTEL CpuPm 4B9 MSFT 1000013)
> ACPI: SSDT 3FEFDA10, 01B7 (r1 INTEL Cpu0Ist 4B9 MSFT 1000013)
> ACPI: SSDT 3FEFD810, 01B7 (r1 INTEL Cpu1Ist 4B9 MSFT 1000013)
> ACPI: SSDT 3FEFD610, 01B7 (r1 INTEL Cpu2Ist 4B9 MSFT 1000013)
> ACPI: SSDT 3FEFD410, 01B7 (r1 INTEL Cpu3Ist 4B9 MSFT 1000013)
> Entering add_active_range(0, 0x0, 0x9f) 0 entries of 25600 used
> Entering add_active_range(0, 0x100, 0x3ed94) 1 entries of 25600 used
> Entering add_active_range(0, 0x3ee4e, 0x3fea2) 2 entries of 25600 used
> Entering add_active_range(0, 0x3fee9, 0x3feed) 3 entries of 25600 used
> Entering add_active_range(0, 0x3feff, 0x3ff00) 4 entries of 25600 used
> (5 early reservations) ==> bootmem
> #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
> #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000]
> #2 [0000200000 - 0001484b34] TEXT DATA BSS ==> [0000200000 - 0001484b34]
> #3 [000009fc00 - 0000100000] BIOS reserved ==> [000009fc00 - 0000100000]
> #4 [0000008000 - 0000009000] PGTABLE ==> [0000008000 - 0000009000]
> Zone PFN ranges:
> DMA 0x00000000 -> 0x00001000
> DMA32 0x00001000 -> 0x00100000
> Normal 0x00100000 -> 0x00100000
> Movable zone start PFN for each node
> early_node_map[5] active PFN ranges
> 0: 0x00000000 -> 0x0000009f
> 0: 0x00000100 -> 0x0003ed94
> 0: 0x0003ee4e -> 0x0003fea2
> 0: 0x0003fee9 -> 0x0003feed
> 0: 0x0003feff -> 0x0003ff00
> On node 0 totalpages: 261516
> DMA zone: 56 pages used for memmap
> DMA zone: 100 pages reserved
> DMA zone: 3843 pages, LIFO batch:0
> DMA32 zone: 3525 pages used for memmap
> DMA32 zone: 253992 pages, LIFO batch:31
> Normal zone: 0 pages used for memmap
> Movable zone: 0 pages used for memmap
> ACPI: Local APIC address 0xfee00000
> ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
> ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
> ACPI: LAPIC (acpi_id[0x03] lapic_id[0x82] disabled)
> ACPI: LAPIC (acpi_id[0x04] lapic_id[0x83] disabled)
> ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1])
> ACPI: LAPIC_NMI (acpi_id[0x02] dfl dfl lint[0x1])
> ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
> IOAPIC[0]: apic_id 2, version 0, address 0xfec00000, GSI 0-23
> ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> ACPI: IRQ0 used by override.
> ACPI: IRQ2 used by override.
> ACPI: IRQ9 used by override.
> Setting APIC routing to flat
> ACPI: HPET id: 0x8086a201 base: 0xfed00000
> SMP: Allowing 4 CPUs, 2 hotplug CPUs
> mapped APIC to ffffffffff5fc000 ( fee00000)
> mapped IOAPIC to ffffffffff5fb000 (00000000fec00000)
> Allocating PCI resources starting at 40000000 (gap: 3ff00000:c0100000)
> PERCPU: Allocating 40256 bytes of per cpu data
> NR_CPUS: 4096, nr_cpu_ids: 4, nr_node_ids 512
> Built 1 zonelists in Zone order, mobility grouping on. Total pages: 257835
> Kernel command line: root=/dev/sda1 earlyprintk=vga console=ttyS0,115200 console=tty 5 profile=0 debug initcall_debug apic=debug apic=verbose ignore_loglevel sysrq_always_enabled pci=nomsi
> kernel profiling enabled (shift: 0)
> debug: sysrq always enabled.
> Initializing CPU#0
> PID hash table entries: 4096 (order: 12, 32768 bytes)
> TSC calibrated against PIT
> Detected 2933.406 MHz processor.
> Console: colour VGA+ 80x25
> console handover: boot [earlyvga0] -> real [tty0]
> console [ttyS0] enabled
> Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
> Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
> Checking aperture...
> No AGP bridge found
> Memory: 978020k/1047552k available (8071k kernel code, 67476k reserved, 6822k data, 540k init)
> CPA: page pool initialized 1 of 1 pages preallocated
> hpet clockevent registered
> Calibrating delay loop (skipped), value calculated using timer frequency.. <6>5866.81 BogoMIPS (lpj=11733624)
> Security Framework initialized
> SELinux: Initializing.
> SELinux: Starting in permissive mode
> Mount-cache hash table entries: 256
> CPU: L1 I cache: 32K, L1 D cache: 32K
> CPU: L2 cache: 4096K
> CPU: Physical Processor ID: 0
> CPU: Processor Core ID: 0
> CPU0: Thermal monitoring enabled (TM2)
> using mwait in idle threads.
> ACPI: Core revision 20080609
> enabled ExtINT on CPU#0
> ENABLING IO-APIC IRQs
> init IO_APIC IRQs
> IO-APIC (apicid-pin) 2-0 not connected.
> IOAPIC[0]: Set routing entry (2-1 -> 0x31 -> IRQ 1 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-2 -> 0x30 -> IRQ 0 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-3 -> 0x33 -> IRQ 3 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-4 -> 0x34 -> IRQ 4 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-5 -> 0x35 -> IRQ 5 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-6 -> 0x36 -> IRQ 6 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-7 -> 0x37 -> IRQ 7 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-8 -> 0x38 -> IRQ 8 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-9 -> 0x39 -> IRQ 9 Mode:1 Active:0)
> IOAPIC[0]: Set routing entry (2-10 -> 0x3a -> IRQ 10 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-11 -> 0x3b -> IRQ 11 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-12 -> 0x3c -> IRQ 12 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-13 -> 0x3d -> IRQ 13 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-14 -> 0x3e -> IRQ 14 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-15 -> 0x3f -> IRQ 15 Mode:0 Active:0)
> IO-APIC (apicid-pin) 2-16, 2-17, 2-18, 2-19, 2-20, 2-21, 2-22, 2-23 not connected.
> ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
> CPU0: Intel(R) Core(TM)2 CPU E6800 @ 2.93GHz stepping 05
> Using local APIC timer interrupts.
> APIC timer calibration result 16667083
> Detected 16.667 MHz APIC timer.
> Booting processor 1/1 ip 6000
> Initializing CPU#1
> masked ExtINT on CPU#1
> Calibrating delay using timer specific routine.. <6>5866.88 BogoMIPS (lpj=11733771)
> CPU: L1 I cache: 32K, L1 D cache: 32K
> CPU: L2 cache: 4096K
> CPU: Physical Processor ID: 0
> CPU: Processor Core ID: 1
> CPU1: Thermal monitoring enabled (TM2)
> CPU1: Intel(R) Core(TM)2 CPU E6800 @ 2.93GHz stepping 05
> checking TSC synchronization [CPU#0 -> CPU#1]: passed.
> Brought up 2 CPUs
> Total of 2 processors activated (11733.69 BogoMIPS).
> calling init_cpufreq_transition_notifier_list+0x0/0x1b
> initcall init_cpufreq_transition_notifier_list+0x0/0x1b returned 0 after 0 msecs
> calling net_ns_init+0x0/0x133
> net_namespace: 1120 bytes
> initcall net_ns_init+0x0/0x133 returned 0 after 0 msecs
> calling cpufreq_tsc+0x0/0x16
> initcall cpufreq_tsc+0x0/0x16 returned 0 after 0 msecs
> calling init_smp_flush+0x0/0x51
> initcall init_smp_flush+0x0/0x51 returned 0 after 0 msecs
> calling print_banner+0x0/0xf
> Booting paravirtualized kernel on bare hardware
> initcall print_banner+0x0/0xf returned 0 after 3 msecs
> calling sysctl_init+0x0/0x32
> initcall sysctl_init+0x0/0x32 returned 0 after 0 msecs
> calling ksysfs_init+0x0/0xb9
> initcall ksysfs_init+0x0/0xb9 returned 0 after 0 msecs
> calling init_jiffies_clocksource+0x0/0xc
> initcall init_jiffies_clocksource+0x0/0xc returned 0 after 0 msecs
> calling pm_init+0x0/0x35
> initcall pm_init+0x0/0x35 returned 0 after 0 msecs
> calling filelock_init+0x0/0x2e
> initcall filelock_init+0x0/0x2e returned 0 after 0 msecs
> calling init_script_binfmt+0x0/0xc
> initcall init_script_binfmt+0x0/0xc returned 0 after 0 msecs
> calling init_elf_binfmt+0x0/0xc
> initcall init_elf_binfmt+0x0/0xc returned 0 after 0 msecs
> calling init_compat_elf_binfmt+0x0/0xc
> initcall init_compat_elf_binfmt+0x0/0xc returned 0 after 0 msecs
> calling debugfs_init+0x0/0x47
> initcall debugfs_init+0x0/0x47 returned 0 after 0 msecs
> calling securityfs_init+0x0/0x47
> initcall securityfs_init+0x0/0x47 returned 0 after 0 msecs
> calling random32_init+0x0/0x5f
> initcall random32_init+0x0/0x5f returned 0 after 0 msecs
> calling cpufreq_core_init+0x0/0x84
> initcall cpufreq_core_init+0x0/0x84 returned 0 after 0 msecs
> calling cpuidle_init+0x0/0x36
> initcall cpuidle_init+0x0/0x36 returned 0 after 0 msecs
> calling virtio_init+0x0/0x26
> initcall virtio_init+0x0/0x26 returned 0 after 0 msecs
> calling sock_init+0x0/0x5e
> initcall sock_init+0x0/0x5e returned 0 after 0 msecs
> calling netpoll_init+0x0/0x2c
> initcall netpoll_init+0x0/0x2c returned 0 after 0 msecs
> calling netlink_proto_init+0x0/0x13e
> NET: Registered protocol family 16
> initcall netlink_proto_init+0x0/0x13e returned 0 after 3 msecs
> calling bdi_class_init+0x0/0x3d
> initcall bdi_class_init+0x0/0x3d returned 0 after 0 msecs
> calling kobject_uevent_init+0x0/0x45
> initcall kobject_uevent_init+0x0/0x45 returned 0 after 0 msecs
> calling pcibus_class_init+0x0/0xc
> initcall pcibus_class_init+0x0/0xc returned 0 after 0 msecs
> calling pci_driver_init+0x0/0xc
> initcall pci_driver_init+0x0/0xc returned 0 after 0 msecs
> calling lcd_class_init+0x0/0x49
> initcall lcd_class_init+0x0/0x49 returned 0 after 0 msecs
> calling backlight_class_init+0x0/0x4a
> initcall backlight_class_init+0x0/0x4a returned 0 after 0 msecs
> calling dock_init+0x0/0x5c
> No dock devices found.
> initcall dock_init+0x0/0x5c returned 0 after 0 msecs
> calling tty_class_init+0x0/0x2a
> initcall tty_class_init+0x0/0x2a returned 0 after 0 msecs
> calling vtconsole_class_init+0x0/0xaf
> initcall vtconsole_class_init+0x0/0xaf returned 0 after 0 msecs
> calling enable_pci_io_ecs+0x0/0x2e
> initcall enable_pci_io_ecs+0x0/0x2e returned 0 after 0 msecs
> calling early_fill_mp_bus_info+0x0/0x763
> initcall early_fill_mp_bus_info+0x0/0x763 returned 0 after 0 msecs
> calling arch_kdebugfs_init+0x0/0x3
> initcall arch_kdebugfs_init+0x0/0x3 returned 0 after 0 msecs
> calling ffh_cstate_init+0x0/0x31
> initcall ffh_cstate_init+0x0/0x31 returned 0 after 0 msecs
> calling acpi_pci_init+0x0/0x3b
> ACPI: bus type pci registered
> initcall acpi_pci_init+0x0/0x3b returned 0 after 0 msecs
> calling init_acpi_device_notify+0x0/0x4b
> initcall init_acpi_device_notify+0x0/0x4b returned 0 after 0 msecs
> calling pci_arch_init+0x0/0x44
> PCI: MCFG configuration 0: base f0000000 segment 0 buses 0 - 127
> PCI: Not using MMCONFIG.
> PCI: Using configuration type 1 for base access
> initcall pci_arch_init+0x0/0x44 returned 0 after 11 msecs
> calling topology_init+0x0/0x31
> initcall topology_init+0x0/0x31 returned 0 after 0 msecs
> calling param_sysfs_init+0x0/0x1c1
> initcall param_sysfs_init+0x0/0x1c1 returned 0 after 11 msecs
> calling pm_sysrq_init+0x0/0x19
> initcall pm_sysrq_init+0x0/0x19 returned 0 after 0 msecs
> calling readahead_init+0x0/0x2e
> initcall readahead_init+0x0/0x2e returned 0 after 3 msecs
> calling init_bio+0x0/0xc0
> initcall init_bio+0x0/0xc0 returned 0 after 0 msecs
> calling integrity_init+0x0/0x35
> initcall integrity_init+0x0/0x35 returned 0 after 0 msecs
> calling blk_settings_init+0x0/0x25
> initcall blk_settings_init+0x0/0x25 returned 0 after 0 msecs
> calling blk_ioc_init+0x0/0x2a
> initcall blk_ioc_init+0x0/0x2a returned 0 after 0 msecs
> calling genhd_device_init+0x0/0x36
> initcall genhd_device_init+0x0/0x36 returned 0 after 0 msecs
> calling blk_dev_integrity_init+0x0/0x2a
> initcall blk_dev_integrity_init+0x0/0x2a returned 0 after 0 msecs
> calling pci_slot_init+0x0/0x45
> initcall pci_slot_init+0x0/0x45 returned 0 after 0 msecs
> calling acpi_init+0x0/0x21b
> ACPI: EC: Look up EC in DSDT
> ACPI: Interpreter enabled
> ACPI: (supports S0 S5)
> ACPI: Using IOAPIC for interrupt routing
> PCI: MCFG configuration 0: base f0000000 segment 0 buses 0 - 127
> PCI: BIOS Bug: MCFG area at f0000000 is not reserved in ACPI motherboard resources
> PCI: Not using MMCONFIG.
> initcall acpi_init+0x0/0x21b returned 0 after 26 msecs
> calling acpi_scan_init+0x0/0x113
> initcall acpi_scan_init+0x0/0x113 returned 0 after 3 msecs
> calling acpi_ec_init+0x0/0x61
> initcall acpi_ec_init+0x0/0x61 returned 0 after 0 msecs
> calling acpi_pci_root_init+0x0/0x28
> ACPI: PCI Root Bridge [PCI0] (0000:00)
> pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
> pci 0000:00:01.0: PME# disabled
> pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold
> pci 0000:00:1b.0: PME# disabled
> pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
> pci 0000:00:1c.0: PME# disabled
> pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold
> pci 0000:00:1c.4: PME# disabled
> pci 0000:00:1c.5: PME# supported from D0 D3hot D3cold
> pci 0000:00:1c.5: PME# disabled
> pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
> pci 0000:00:1d.7: PME# disabled
> pci 0000:00:1f.2: PME# supported from D3hot
> pci 0000:00:1f.2: PME# disabled
> pci 0000:01:00.0: supports D1
> pci 0000:01:00.0: supports D2
> pci 0000:01:00.1: supports D1
> pci 0000:01:00.1: supports D2
> pci 0000:02:00.0: supports D1
> pci 0000:02:00.0: supports D2
> pci 0000:02:00.0: PME# supported from D0 D1 D2
> pci 0000:02:00.0: PME# disabled
> pci 0000:04:00.0: PME# supported from D0 D3hot D3cold
> pci 0000:04:00.0: PME# disabled
> pci 0000:05:04.0: supports D1
> pci 0000:05:04.0: supports D2
> pci 0000:05:04.0: PME# supported from D0 D1 D2 D3hot
> pci 0000:05:04.0: PME# disabled
> pci 0000:05:05.0: supports D1
> pci 0000:05:05.0: supports D2
> pci 0000:00:1e.0: transparent bridge
> bus 00 -> node 0
> ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
> ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P32_._PRT]
> ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX0._PRT]
> ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX4._PRT]
> ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX5._PRT]
> initcall acpi_pci_root_init+0x0/0x28 returned 0 after 110 msecs
> calling acpi_pci_link_init+0x0/0x48
> ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 7 9 10 *11 12)
> ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 7 9 *10 11 12)
> ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 7 9 *10 11 12)
> ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 7 *9 10 11 12)
> ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 7 9 10 11 12) *0, disabled.
> ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 7 9 10 11 12) *0, disabled.
> ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 7 *9 10 11 12)
> ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 7 9 10 *11 12)
> initcall acpi_pci_link_init+0x0/0x48 returned 0 after 49 msecs
> calling acpi_power_init+0x0/0x77
> initcall acpi_power_init+0x0/0x77 returned 0 after 0 msecs
> calling acpi_system_init+0x0/0x264
> initcall acpi_system_init+0x0/0x264 returned 0 after 0 msecs
> calling pnp_init+0x0/0x1f
> Linux Plug and Play Support v0.97 (c) Adam Belay
> initcall pnp_init+0x0/0x1f returned 0 after 3 msecs
> calling pnpacpi_init+0x0/0x8c
> pnp: PnP ACPI init
> ACPI: bus type pnp registered
> IOAPIC[0]: Set routing entry (2-8 -> 0x38 -> IRQ 8 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-13 -> 0x3d -> IRQ 13 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-6 -> 0x36 -> IRQ 6 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-7 -> 0x37 -> IRQ 7 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-12 -> 0x3c -> IRQ 12 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-1 -> 0x31 -> IRQ 1 Mode:0 Active:0)
> IOAPIC[0]: Set routing entry (2-4 -> 0x34 -> IRQ 4 Mode:0 Active:0)
> pnp: PnP ACPI: found 13 devices
> ACPI: ACPI bus type pnp unregistered
> initcall pnpacpi_init+0x0/0x8c returned 0 after 38 msecs
> calling misc_init+0x0/0x8e
> initcall misc_init+0x0/0x8e returned 0 after 0 msecs
> calling cn_init+0x0/0xee
> initcall cn_init+0x0/0xee returned 0 after 0 msecs
> calling tifm_init+0x0/0x6d
> initcall tifm_init+0x0/0x6d returned 0 after 0 msecs
> calling phy_init+0x0/0x27
> initcall phy_init+0x0/0x27 returned 0 after 0 msecs
> calling init_dvbdev+0x0/0xaf
> initcall init_dvbdev+0x0/0xaf returned 0 after 0 msecs
> calling init_scsi+0x0/0x72
> SCSI subsystem initialized
> initcall init_scsi+0x0/0x72 returned 0 after 3 msecs
> calling ata_init+0x0/0x32a
> libata version 3.00 loaded.
> initcall ata_init+0x0/0x32a returned 0 after 3 msecs
> calling spi_init+0x0/0x73
> initcall spi_init+0x0/0x73 returned 0 after 0 msecs
> calling init_pcmcia_cs+0x0/0x27
> initcall init_pcmcia_cs+0x0/0x27 returned 0 after 0 msecs
> calling usb_init+0x0/0x104
> usbcore: registered new interface driver usbfs
> usbcore: registered new interface driver hub
> usbcore: registered new device driver usb
> initcall usb_init+0x0/0x104 returned 0 after 11 msecs
> calling serio_init+0x0/0x8b
> initcall serio_init+0x0/0x8b returned 0 after 0 msecs
> calling gameport_init+0x0/0x8b
> initcall gameport_init+0x0/0x8b returned 0 after 0 msecs
> calling input_init+0x0/0xfc
> initcall input_init+0x0/0xfc returned 0 after 0 msecs
> calling i2c_init+0x0/0x55
> initcall i2c_init+0x0/0x55 returned 0 after 0 msecs
> calling power_supply_class_init+0x0/0x34
> initcall power_supply_class_init+0x0/0x34 returned 0 after 0 msecs
> calling thermal_init+0x0/0x2f
> initcall thermal_init+0x0/0x2f returned 0 after 0 msecs
> calling mmc_init+0x0/0x65
> initcall mmc_init+0x0/0x65 returned 0 after 0 msecs
> calling leds_init+0x0/0x2a
> initcall leds_init+0x0/0x2a returned 0 after 0 msecs
> calling dma_bus_init+0x0/0x2b
> initcall dma_bus_init+0x0/0x2b returned 0 after 0 msecs
> calling ac97_bus_init+0x0/0xc
> initcall ac97_bus_init+0x0/0xc returned 0 after 0 msecs
> calling pci_subsys_init+0x0/0x116
> PCI: Using ACPI for IRQ routing
> number of MP IRQ sources: 15.
> number of IO-APIC #2 registers: 24.
> testing the IO APIC.......................
>
> IO APIC #2......
> .... register #00: 00000000
> ....... : physical APIC id: 00
> .... register #01: 00170020
> ....... : max redirection entries: 0017
> ....... : PRQ implemented: 0
> ....... : IO APIC version: 0020
> .... register #02: 00170020
> ....... : arbitration: 00
> .... IRQ redirection table:
> NR Dst Mask Trig IRR Pol Stat Dmod Deli Vect:
> 00 000 1 0 0 0 0 0 0 00
> 01 003 0 0 0 0 0 1 1 31
> 02 003 0 0 0 0 0 1 1 30
> 03 003 0 0 0 0 0 1 1 33
> 04 003 0 0 0 0 0 1 1 34
> 05 003 0 0 0 0 0 1 1 35
> 06 003 0 0 0 0 0 1 1 36
> 07 003 0 0 0 0 0 1 1 37
> 08 003 0 0 0 0 0 1 1 38
> 09 003 0 1 0 0 0 1 1 39
> 0a 003 0 0 0 0 0 1 1 3A
> 0b 003 0 0 0 0 0 1 1 3B
> 0c 003 0 0 0 0 0 1 1 3C
> 0d 003 0 0 0 0 0 1 1 3D
> 0e 003 0 0 0 0 0 1 1 3E
> 0f 003 0 0 0 0 0 1 1 3F
> 10 000 1 0 0 0 0 0 0 00
> 11 000 1 0 0 0 0 0 0 00
> 12 000 1 0 0 0 0 0 0 00
> 13 000 1 0 0 0 0 0 0 00
> 14 000 1 0 0 0 0 0 0 00
> 15 000 1 0 0 0 0 0 0 00
> 16 000 1 0 0 0 0 0 0 00
> 17 000 1 0 0 0 0 0 0 00
> IRQ to pin mappings:
> IRQ0 -> 0:2
> IRQ1 -> 0:1
> IRQ3 -> 0:3
> IRQ4 -> 0:4
> IRQ5 -> 0:5
> IRQ6 -> 0:6
> IRQ7 -> 0:7
> IRQ8 -> 0:8
> IRQ9 -> 0:9
> IRQ10 -> 0:10
> IRQ11 -> 0:11
> IRQ12 -> 0:12
> IRQ13 -> 0:13
> IRQ14 -> 0:14
> IRQ15 -> 0:15
> .................................... done.
> initcall pci_subsys_init+0x0/0x116 returned 0 after 164 msecs
> calling proto_init+0x0/0x2e
> initcall proto_init+0x0/0x2e returned 0 after 0 msecs
> calling net_dev_init+0x0/0x158
> initcall net_dev_init+0x0/0x158 returned 0 after 0 msecs
> calling neigh_init+0x0/0x71
> initcall neigh_init+0x0/0x71 returned 0 after 0 msecs
> calling fib_rules_init+0x0/0x9c
> initcall fib_rules_init+0x0/0x9c returned 0 after 0 msecs
> calling genl_init+0x0/0xce
> initcall genl_init+0x0/0xce returned 0 after 15 msecs
> calling cipso_v4_init+0x0/0x67
> initcall cipso_v4_init+0x0/0x67 returned 0 after 0 msecs
> calling wireless_nlevent_init+0x0/0x2c
> initcall wireless_nlevent_init+0x0/0x2c returned 0 after 0 msecs
> calling cfg80211_init+0x0/0x55
> initcall cfg80211_init+0x0/0x55 returned 0 after 11 msecs
> calling ieee80211_init+0x0/0xd
> initcall ieee80211_init+0x0/0xd returned 0 after 0 msecs
> calling netlbl_init+0x0/0x83
> NetLabel: Initializing
> NetLabel: domain hash size = 128
> NetLabel: protocols = UNLABELED CIPSOv4
> NetLabel: unlabeled traffic allowed by default
> initcall netlbl_init+0x0/0x83 returned 0 after 11 msecs
> calling sysctl_init+0x0/0x2f
> initcall sysctl_init+0x0/0x2f returned 0 after 0 msecs
> calling pci_iommu_init+0x0/0x12
> PCI-GART: No AMD northbridge found.
> initcall pci_iommu_init+0x0/0x12 returned 0 after 3 msecs
> calling hpet_late_init+0x0/0xfc
> hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
> hpet0: 3 64-bit timers, 14318180 Hz
> initcall hpet_late_init+0x0/0xfc returned 0 after 7 msecs
> calling clocksource_done_booting+0x0/0xd
> initcall clocksource_done_booting+0x0/0xd<7>Switched to high resolution mode on CPU 0
> Switched to high resolution mode on CPU 1
> returned 0 after 0 msecs
> calling tracer_alloc_buffers+0x0/0x4ec
> tracer: 1286 pages allocated for 65536 entries of 80 bytes
> actual entries 65586
> initcall tracer_alloc_buffers+0x0/0x4ec returned 0 after 8 msecs
> calling init_pipe_fs+0x0/0x42
> initcall init_pipe_fs+0x0/0x42 returned 0 after 0 msecs
> calling init_mnt_writers+0x0/0x58
> initcall init_mnt_writers+0x0/0x58 returned 0 after 0 msecs
> calling anon_inode_init+0x0/0x10b
> initcall anon_inode_init+0x0/0x10b returned 0 after 0 msecs
> calling acpi_event_init+0x0/0x52
> initcall acpi_event_init+0x0/0x52 returned 0 after 10 msecs
> calling pnp_system_init+0x0/0xc
> system 00:01: iomem range 0xf0000000-0xf3ffffff has been reserved
> system 00:01: iomem range 0xfed13000-0xfed13fff has been reserved
> system 00:01: iomem range 0xfed14000-0xfed17fff has been reserved
> system 00:01: iomem range 0xfed18000-0xfed18fff has been reserved
> system 00:01: iomem range 0xfed19000-0xfed19fff has been reserved
> system 00:01: iomem range 0xfed1c000-0xfed1ffff has been reserved
> system 00:01: iomem range 0xfed20000-0xfed3ffff has been reserved
> system 00:01: iomem range 0xfed45000-0xfed99fff has been reserved
> system 00:01: iomem range 0xc0000-0xdffff has been reserved
> system 00:01: iomem range 0xe0000-0xfffff could not be reserved
> system 00:06: ioport range 0x500-0x53f has been reserved
> system 00:06: ioport range 0x400-0x47f has been reserved
> system 00:06: ioport range 0x680-0x6ff has been reserved
> initcall pnp_system_init+0x0/0xc returned 0 after 70 msecs
> calling chr_dev_init+0x0/0x9a
> initcall chr_dev_init+0x0/0x9a returned 0 after 0 msecs
> calling firmware_class_init+0x0/0x68
> initcall firmware_class_init+0x0/0x68 returned 0 after 0 msecs
> calling loopback_init+0x0/0xc
> initcall loopback_init+0x0/0xc returned 0 after 0 msecs
> calling cpufreq_gov_performance_init+0x0/0xc
> initcall cpufreq_gov_performance_init+0x0/0xc returned 0 after 0 msecs
> calling cpufreq_gov_dbs_init+0x0/0x47
> initcall cpufreq_gov_dbs_init+0x0/0x47 returned 0 after 0 msecs
> calling ssb_modinit+0x0/0x4b
> initcall ssb_modinit+0x0/0x4b returned 0 after 0 msecs
> calling pcibios_assign_resources+0x0/0x86
> pci 0000:00:01.0: PCI bridge, secondary bus 0000:01
> pci 0000:00:01.0: IO window: 0x3000-0x3fff
> pci 0000:00:01.0: MEM window: 0x50300000-0x503fffff
> pci 0000:00:01.0: PREFETCH window: 0x00000040000000-0x0000004fffffff
> pci 0000:00:1c.0: PCI bridge, secondary bus 0000:02
> pci 0000:00:1c.0: IO window: disabled
> pci 0000:00:1c.0: MEM window: 0x50200000-0x502fffff
> pci 0000:00:1c.0: PREFETCH window: disabled
> pci 0000:00:1c.4: PCI bridge, secondary bus 0000:03
> pci 0000:00:1c.4: IO window: disabled
> pci 0000:00:1c.4: MEM window: disabled
> pci 0000:00:1c.4: PREFETCH window: disabled
> pci 0000:00:1c.5: PCI bridge, secondary bus 0000:04
> pci 0000:00:1c.5: IO window: 0x2000-0x2fff
> pci 0000:00:1c.5: MEM window: 0x50100000-0x501fffff
> pci 0000:00:1c.5: PREFETCH window: disabled
> pci 0000:00:1e.0: PCI bridge, secondary bus 0000:05
> pci 0000:00:1e.0: IO window: 0x1000-0x1fff
> pci 0000:00:1e.0: MEM window: 0x50000000-0x500fffff
> pci 0000:00:1e.0: PREFETCH window: disabled
> IOAPIC[0]: Set routing entry (2-16 -> 0x49 -> IRQ 16 Mode:1 Active:1)
> pci 0000:00:01.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> PCI: Setting latency timer of device 0000:00:01.0 to 64
> IOAPIC[0]: Set routing entry (2-17 -> 0x51 -> IRQ 17 Mode:1 Active:1)
> pci 0000:00:1c.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
> PCI: Setting latency timer of device 0000:00:1c.0 to 64
> pci 0000:00:1c.4: PCI INT A -> GSI 17 (level, low) -> IRQ 17
> PCI: Setting latency timer of device 0000:00:1c.4 to 64
> pci 0000:00:1c.5: PCI INT B -> GSI 16 (level, low) -> IRQ 16
> PCI: Setting latency timer of device 0000:00:1c.5 to 64
> PCI: Setting latency timer of device 0000:00:1e.0 to 64
> initcall pcibios_assign_resources+0x0/0x86 returned 0 after 147 msecs
> calling inet_init+0x0/0x1df
> NET: Registered protocol family 2
> IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
> TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
> TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
> TCP: Hash tables configured (established 131072 bind 65536)
> TCP reno registered
> initcall inet_init+0x0/0x1df returned 0 after 75 msecs
> calling af_unix_init+0x0/0x4b
> NET: Registered protocol family 1
> initcall af_unix_init+0x0/0x4b returned 0 after 3 msecs
> calling default_rootfs+0x0/0x61
> initcall default_rootfs+0x0/0x61 returned 0 after 0 msecs
> calling i8259A_init_sysfs+0x0/0x23
> initcall i8259A_init_sysfs+0x0/0x23 returned 0 after 0 msecs
> calling vsyscall_init+0x0/0x6c
> initcall vsyscall_init+0x0/0x6c returned 0 after 0 msecs
> calling sbf_init+0x0/0xd5
> initcall sbf_init+0x0/0xd5 returned 0 after 0 msecs
> calling i8237A_init_sysfs+0x0/0x23
> initcall i8237A_init_sysfs+0x0/0x23 returned 0 after 0 msecs
> calling add_rtc_cmos+0x0/0x1d
> initcall add_rtc_cmos+0x0/0x1d returned 0 after 0 msecs
> calling cache_sysfs_init+0x0/0x55
> initcall cache_sysfs_init+0x0/0x55 returned 0 after 0 msecs
> calling mce_init_device+0x0/0x7f
> initcall mce_init_device+0x0/0x7f returned 0 after 0 msecs
> calling periodic_mcheck_init+0x0/0x3f
> initcall periodic_mcheck_init+0x0/0x3f returned 0 after 0 msecs
> calling thermal_throttle_init_device+0x0/0x7b
> initcall thermal_throttle_init_device+0x0/0x7b returned 0 after 0 msecs
> calling microcode_init+0x0/0xaf
> IA-32 Microcode Update Driver: v1.14a <[email protected]>
> initcall microcode_init+0x0/0xaf returned 0 after 5 msecs
> calling init_lapic_sysfs+0x0/0x2f
> initcall init_lapic_sysfs+0x0/0x2f returned 0 after 0 msecs
> calling ioapic_init_sysfs+0x0/0x99
> initcall ioapic_init_sysfs+0x0/0x99 returned 0 after 0 msecs
> calling uv_ptc_init+0x0/0x75
> initcall uv_ptc_init+0x0/0x75 returned 0 after 0 msecs
> calling uv_bau_init+0x0/0x5d0
> initcall uv_bau_init+0x0/0x5d0 returned 0 after 0 msecs
> calling audit_classes_init+0x0/0xaf
> initcall audit_classes_init+0x0/0xaf returned 0 after 0 msecs
> calling aes_init+0x0/0xc
> initcall aes_init+0x0/0xc returned 0 after 0 msecs
> calling init+0x0/0xc
> initcall init+0x0/0xc returned 0 after 0 msecs
> calling init_vdso_vars+0x0/0x224
> initcall init_vdso_vars+0x0/0x224 returned 0 after 0 msecs
> calling ia32_binfmt_init+0x0/0x14
> initcall ia32_binfmt_init+0x0/0x14 returned 0 after 0 msecs
> calling sysenter_setup+0x0/0x2ac
> initcall sysenter_setup+0x0/0x2ac returned 0 after 0 msecs
> calling init_aout_binfmt+0x0/0xc
> initcall init_aout_binfmt+0x0/0xc returned 0 after 0 msecs
> calling create_proc_profile+0x0/0x319
> initcall create_proc_profile+0x0/0x319 returned 0 after 0 msecs
> calling ioresources_init+0x0/0x3c
> initcall ioresources_init+0x0/0x3c returned 0 after 0 msecs
> calling uid_cache_init+0x0/0x6f
> initcall uid_cache_init+0x0/0x6f returned 0 after 0 msecs
> calling init_posix_timers+0x0/0xa6
> initcall init_posix_timers+0x0/0xa6 returned 0 after 0 msecs
> calling init_posix_cpu_timers+0x0/0xc6
> initcall init_posix_cpu_timers+0x0/0xc6 returned 0 after 0 msecs
> calling nsproxy_cache_init+0x0/0x2d
> initcall nsproxy_cache_init+0x0/0x2d returned 0 after 0 msecs
> calling timekeeping_init_device+0x0/0x23
> initcall timekeeping_init_device+0x0/0x23 returned 0 after 0 msecs
> calling init_clocksource_sysfs+0x0/0x51
> initcall init_clocksource_sysfs+0x0/0x51 returned 0 after 0 msecs
> calling init_timer_list_procfs+0x0/0x2c
> initcall init_timer_list_procfs+0x0/0x2c returned 0 after 0 msecs
> calling futex_init+0x0/0x67
> initcall futex_init+0x0/0x67 returned 0 after 0 msecs
> calling proc_dma_init+0x0/0x22
> initcall proc_dma_init+0x0/0x22 returned 0 after 0 msecs
> calling kallsyms_init+0x0/0x25
> initcall kallsyms_init+0x0/0x25 returned 0 after 0 msecs
> calling pid_namespaces_init+0x0/0x2d
> initcall pid_namespaces_init+0x0/0x2d returned 0 after 0 msecs
> calling audit_init+0x0/0x126
> audit: initializing netlink socket (disabled)
> type=2000 audit(1216650963.392:1): initialized
> initcall audit_init+0x0/0x126 returned 0 after 8 msecs
> calling relay_init+0x0/0x14
> initcall relay_init+0x0/0x14 returned 0 after 0 msecs
> calling utsname_sysctl_init+0x0/0x14
> initcall utsname_sysctl_init+0x0/0x14 returned 0 after 0 msecs
> calling init_stack_trace+0x0/0xc
> Testing tracer sysprof: PASSED
> initcall init_stack_trace+0x0/0xc returned 0 after 98 msecs
> calling init_per_zone_pages_min+0x0/0x45
> initcall init_per_zone_pages_min+0x0/0x45 returned 0 after 0 msecs
> calling pdflush_init+0x0/0x13
> initcall pdflush_init+0x0/0x13 returned 0 after 0 msecs
> calling kswapd_init+0x0/0x5e
> initcall kswapd_init+0x0/0x5e returned 0 after 0 msecs
> calling setup_vmstat+0x0/0x42
> initcall setup_vmstat+0x0/0x42 returned 0 after 0 msecs
> calling init_tmpfs+0x0/0x3d
> initcall init_tmpfs+0x0/0x3d returned 0 after 0 msecs
> calling fasync_init+0x0/0x2a
> initcall fasync_init+0x0/0x2a returned 0 after 0 msecs
> calling aio_setup+0x0/0x6e
> initcall aio_setup+0x0/0x6e returned 0 after 0 msecs
> calling inotify_setup+0x0/0xd
> initcall inotify_setup+0x0/0xd returned 0 after 0 msecs
> calling inotify_user_setup+0x0/0xb8
> initcall inotify_user_setup+0x0/0xb8 returned 0 after 0 msecs
> calling init_sys32_ioctl+0x0/0x85
> initcall init_sys32_ioctl+0x0/0x85 returned 0 after 0 msecs
> calling init_mbcache+0x0/0x14
> initcall init_mbcache+0x0/0x14 returned 0 after 0 msecs
> calling dquot_init+0x0/0xde
> VFS: Disk quotas dquot_6.5.1
> Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
> initcall dquot_init+0x0/0xde returned 0 after 7 msecs
> calling init_v2_quota_format+0x0/0xc
> initcall init_v2_quota_format+0x0/0xc returned 0 after 0 msecs
> calling dnotify_init+0x0/0x2a
> initcall dnotify_init+0x0/0x2a returned 0 after 0 msecs
> calling configfs_init+0x0/0xa9
> initcall configfs_init+0x0/0xa9 returned 0 after 0 msecs
> calling init_devpts_fs+0x0/0x35
> initcall init_devpts_fs+0x0/0x35 returned 0 after 0 msecs
> calling init_ext3_fs+0x0/0x60
> initcall init_ext3_fs+0x0/0x60 returned 0 after 0 msecs
> calling journal_init+0x0/0xc5
> initcall journal_init+0x0/0xc5 returned 0 after 0 msecs
> calling init_ext2_fs+0x0/0x60
> initcall init_ext2_fs+0x0/0x60 returned 0 after 0 msecs
> calling init_ramfs_fs+0x0/0xc
> initcall init_ramfs_fs+0x0/0xc returned 0 after 0 msecs
> calling init_fat_fs+0x0/0x45
> initcall init_fat_fs+0x0/0x45 returned 0 after 0 msecs
> calling init_msdos_fs+0x0/0xc
> initcall init_msdos_fs+0x0/0xc returned 0 after 0 msecs
> calling init_vfat_fs+0x0/0xc
> initcall init_vfat_fs+0x0/0xc returned 0 after 0 msecs
> calling init_iso9660_fs+0x0/0x50
> initcall init_iso9660_fs+0x0/0x50 returned 0 after 0 msecs
> calling init_hfsplus_fs+0x0/0x53
> initcall init_hfsplus_fs+0x0/0x53 returned 0 after 0 msecs
> calling init_hfs_fs+0x0/0x53
> initcall init_hfs_fs+0x0/0x53 returned 0 after 0 msecs
> calling vxfs_init+0x0/0x4f
> initcall vxfs_init+0x0/0x4f returned 0 after 0 msecs
> calling init_nls_cp437+0x0/0xc
> initcall init_nls_cp437+0x0/0xc returned 0 after 0 msecs
> calling init_nls_cp775+0x0/0xc
> initcall init_nls_cp775+0x0/0xc returned 0 after 0 msecs
> calling init_nls_cp850+0x0/0xc
> initcall init_nls_cp850+0x0/0xc returned 0 after 0 msecs
> calling init_nls_cp852+0x0/0xc
> initcall init_nls_cp852+0x0/0xc returned 0 after 0 msecs
> calling init_nls_cp855+0x0/0xc
> initcall init_nls_cp855+0x0/0xc returned 0 after 0 msecs
> calling init_nls_cp857+0x0/0xc
> initcall init_nls_cp857+0x0/0xc returned 0 after 0 msecs
> calling init_nls_cp860+0x0/0xc
> initcall init_nls_cp860+0x0/0xc returned 0 after 0 msecs
> calling init_nls_cp863+0x0/0xc
> initcall init_nls_cp863+0x0/0xc returned 0 after 0 msecs
> calling init_nls_cp874+0x0/0xc
> initcall init_nls_cp874+0x0/0xc returned 0 after 0 msecs
> calling init_nls_cp932+0x0/0xc
> initcall init_nls_cp932+0x0/0xc returned 0 after 0 msecs
> calling init_nls_euc_jp+0x0/0x4a
> initcall init_nls_euc_jp+0x0/0x4a returned 0 after 0 msecs
> calling init_nls_cp936+0x0/0xc
> initcall init_nls_cp936+0x0/0xc returned 0 after 0 msecs
> calling init_nls_ascii+0x0/0xc
> initcall init_nls_ascii+0x0/0xc returned 0 after 0 msecs
> calling init_nls_iso8859_1+0x0/0xc
> initcall init_nls_iso8859_1+0x0/0xc returned 0 after 0 msecs
> calling init_nls_iso8859_5+0x0/0xc
> initcall init_nls_iso8859_5+0x0/0xc returned 0 after 0 msecs
> calling init_nls_iso8859_13+0x0/0xc
> initcall init_nls_iso8859_13+0x0/0xc returned 0 after 0 msecs
> calling init_nls_iso8859_14+0x0/0xc
> initcall init_nls_iso8859_14+0x0/0xc returned 0 after 0 msecs
> calling init_nls_koi8_r+0x0/0xc
> initcall init_nls_koi8_r+0x0/0xc returned 0 after 0 msecs
> calling init_nls_utf8+0x0/0x1f
> initcall init_nls_utf8+0x0/0x1f returned 0 after 0 msecs
> calling init_sysv_fs+0x0/0x45
> initcall init_sysv_fs+0x0/0x45 returned 0 after 0 msecs
> calling init_hpfs_fs+0x0/0x50
> initcall init_hpfs_fs+0x0/0x50 returned 0 after 0 msecs
> calling init_ntfs_fs+0x0/0x1c7
> NTFS driver 2.1.29 [Flags: R/O].
> initcall init_ntfs_fs+0x0/0x1c7 returned 0 after 2 msecs
> calling init_qnx4_fs+0x0/0x62
> QNX4 filesystem 0.2.3 registered.
> initcall init_qnx4_fs+0x0/0x62 returned 0 after 3 msecs
> calling init_adfs_fs+0x0/0x50
> initcall init_adfs_fs+0x0/0x50 returned 0 after 0 msecs
> calling fuse_init+0x0/0x11b
> fuse init (API version 7.9)
> initcall fuse_init+0x0/0x11b returned 0 after 2 msecs
> calling init_jfs_fs+0x0/0x1db
> JFS: nTxBlock = 7645, nTxLock = 61161
> initcall init_jfs_fs+0x0/0x1db returned 0 after 8 msecs
> calling init_befs_fs+0x0/0x71
> BeFS version: 0.9.3
> initcall init_befs_fs+0x0/0x71 returned 0 after 1 msecs
> calling ocfs2_init+0x0/0x28b
> OCFS2 1.5.0
> initcall ocfs2_init+0x0/0x28b returned 0 after 1 msecs
> calling ocfs2_stack_glue_init+0x0/0x8b
> initcall ocfs2_stack_glue_init+0x0/0x8b returned 0 after 0 msecs
> calling init_o2nm+0x0/0x90
> OCFS2 Node Manager 1.5.0
> initcall init_o2nm+0x0/0x90 returned 0 after 2 msecs
> calling init_gfs2_fs+0x0/0x16e
> GFS2 (built Jul 21 2008 16:13:18) installed
> initcall init_gfs2_fs+0x0/0x16e returned 0 after 5 msecs
> calling init_mqueue_fs+0x0/0xbc
> initcall init_mqueue_fs+0x0/0xbc returned 0 after 0 msecs
> calling key_proc_init+0x0/0x59
> initcall key_proc_init+0x0/0x59 returned 0 after 0 msecs
> calling selinux_nf_ip_init+0x0/0x4c
> SELinux: Registering netfilter hooks
> initcall selinux_nf_ip_init+0x0/0x4c returned 0 after 3 msecs
> calling init_sel_fs+0x0/0x5e
> initcall init_sel_fs+0x0/0x5e returned 0 after 0 msecs
> calling selnl_init+0x0/0x4d
> initcall selnl_init+0x0/0x4d returned 0 after 0 msecs
> calling sel_netif_init+0x0/0x66
> initcall sel_netif_init+0x0/0x66 returned 0 after 0 msecs
> calling sel_netnode_init+0x0/0x78
> initcall sel_netnode_init+0x0/0x78 returned 0 after 0 msecs
> calling sel_netport_init+0x0/0x78
> initcall sel_netport_init+0x0/0x78 returned 0 after 0 msecs
> calling aurule_init+0x0/0x37
> initcall aurule_init+0x0/0x37 returned 0 after 0 msecs
> calling crypto_algapi_init+0x0/0xd
> initcall crypto_algapi_init+0x0/0xd returned 0 after 0 msecs
> calling blkcipher_module_init+0x0/0x20
> initcall blkcipher_module_init+0x0/0x20 returned 0 after 0 msecs
> calling seqiv_module_init+0x0/0xc
> initcall seqiv_module_init+0x0/0xc returned 0 after 0 msecs
> calling cryptomgr_init+0x0/0xc
> initcall cryptomgr_init+0x0/0xc returned 0 after 0 msecs
> calling hmac_module_init+0x0/0xc
> initcall hmac_module_init+0x0/0xc returned 0 after 0 msecs
> calling md5_mod_init+0x0/0xc
> initcall md5_mod_init+0x0/0xc returned 0 after 0 msecs
> calling rmd128_mod_init+0x0/0xc
> initcall rmd128_mod_init+0x0/0xc returned 0 after 0 msecs
> calling rmd160_mod_init+0x0/0xc
> initcall rmd160_mod_init+0x0/0xc returned 0 after 0 msecs
> calling rmd256_mod_init+0x0/0xc
> initcall rmd256_mod_init+0x0/0xc returned 0 after 0 msecs
> calling rmd320_mod_init+0x0/0xc
> initcall rmd320_mod_init+0x0/0xc returned 0 after 0 msecs
> calling sha1_generic_mod_init+0x0/0xc
> initcall sha1_generic_mod_init+0x0/0xc returned 0 after 0 msecs
> calling sha256_generic_mod_init+0x0/0x35
> initcall sha256_generic_mod_init+0x0/0x35 returned 0 after 0 msecs
> calling sha512_generic_mod_init+0x0/0x35
> initcall sha512_generic_mod_init+0x0/0x35 returned 0 after 0 msecs
> calling crypto_ecb_module_init+0x0/0xc
> initcall crypto_ecb_module_init+0x0/0xc returned 0 after 0 msecs
> calling crypto_cbc_module_init+0x0/0xc
> initcall crypto_cbc_module_init+0x0/0xc returned 0 after 0 msecs
> calling crypto_module_init+0x0/0xc
> initcall crypto_module_init+0x0/0xc returned 0 after 0 msecs
> calling crypto_ctr_module_init+0x0/0x35
> initcall crypto_ctr_module_init+0x0/0x35 returned 0 after 0 msecs
> calling crypto_ccm_module_init+0x0/0x53
> initcall crypto_ccm_module_init+0x0/0x53 returned 0 after 0 msecs
> calling cryptd_init+0x0/0xa8
> initcall cryptd_init+0x0/0xa8 returned 0 after 0 msecs
> calling des_generic_mod_init+0x0/0x35
> initcall des_generic_mod_init+0x0/0x35 returned 0 after 0 msecs
> calling fcrypt_mod_init+0x0/0xc
> initcall fcrypt_mod_init+0x0/0xc returned 0 after 0 msecs
> calling serpent_mod_init+0x0/0x35
> initcall serpent_mod_init+0x0/0x35 returned 0 after 0 msecs
> calling aes_init+0x0/0x2c5
> initcall aes_init+0x0/0x2c5 returned 0 after 0 msecs
> calling cast5_mod_init+0x0/0xc
> initcall cast5_mod_init+0x0/0xc returned 0 after 0 msecs
> calling arc4_init+0x0/0xc
> initcall arc4_init+0x0/0xc returned 0 after 0 msecs
> calling khazad_mod_init+0x0/0xc
> initcall khazad_mod_init+0x0/0xc returned 0 after 0 msecs
> calling seed_init+0x0/0xc
> initcall seed_init+0x0/0xc returned 0 after 0 msecs
> calling salsa20_generic_mod_init+0x0/0xc
> initcall salsa20_generic_mod_init+0x0/0xc returned 0 after 0 msecs
> calling deflate_mod_init+0x0/0xc
> initcall deflate_mod_init+0x0/0xc returned 0 after 0 msecs
> calling crc32c_mod_init+0x0/0x35
> initcall crc32c_mod_init+0x0/0x35 returned 0 after 0 msecs
> calling crypto_authenc_module_init+0x0/0xc
> initcall crypto_authenc_module_init+0x0/0xc returned 0 after 0 msecs
> calling bsg_init+0x0/0x11a
> Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
> initcall bsg_init+0x0/0x11a returned 0 after 5 msecs
> calling noop_init+0x0/0x14
> io scheduler noop registered (default)
> initcall noop_init+0x0/0x14 returned 0 after 3 msecs
> calling percpu_counter_startup+0x0/0x14
> initcall percpu_counter_startup+0x0/0x14 returned 0 after 0 msecs
> calling pci_init+0x0/0x2b
> pci 0000:01:00.0: Boot video device
> initcall pci_init+0x0/0x2b returned 0 after 3 msecs
> calling pci_proc_init+0x0/0x60
> initcall pci_proc_init+0x0/0x60 returned 0 after 0 msecs
> calling ltv350qv_init+0x0/0xc
> initcall ltv350qv_init+0x0/0xc returned 0 after 0 msecs
> calling corgibl_init+0x0/0xc
> initcall corgibl_init+0x0/0xc returned 0 after 0 msecs
> calling acpi_reserve_resources+0x0/0xeb
> initcall acpi_reserve_resources+0x0/0xeb returned 0 after 0 msecs
> calling acpi_ac_init+0x0/0x28
> initcall acpi_ac_init+0x0/0x28 returned 0 after 0 msecs
> calling acpi_battery_init+0x0/0x28
> initcall acpi_battery_init+0x0/0x28 returned 0 after 0 msecs
> calling acpi_button_init+0x0/0x5e
> input: Power Button (FF) as /class/input/input0
> ACPI: Power Button (FF) [PWRF]
> input: Sleep Button (CM) as /class/input/input1
> ACPI: Sleep Button (CM) [SLPB]
> initcall acpi_button_init+0x0/0x5e returned 0 after 14 msecs
> calling acpi_fan_init+0x0/0x5e
> initcall acpi_fan_init+0x0/0x5e returned 0 after 0 msecs
> calling irqrouter_init_sysfs+0x0/0x37
> initcall irqrouter_init_sysfs+0x0/0x37 returned 0 after 0 msecs
> calling acpi_processor_init+0x0/0xf6
> ACPI: ACPI0007:00 is registered as cooling_device0
> ACPI: ACPI0007:01 is registered as cooling_device1
> initcall acpi_processor_init+0x0/0xf6 returned 0 after 9 msecs
> calling acpi_container_init+0x0/0x43
> initcall acpi_container_init+0x0/0x43 returned 0 after 1 msecs
> calling toshiba_acpi_init+0x0/0x17d
> initcall toshiba_acpi_init+0x0/0x17d returned -19 after 0 msecs
> calling acpi_smb_hc_init+0x0/0x18
> initcall acpi_smb_hc_init+0x0/0x18 returned 0 after 0 msecs
> calling acpi_sbs_init+0x0/0x28
> initcall acpi_sbs_init+0x0/0x28 returned 0 after 0 msecs
> calling rand_initialize+0x0/0x2c
> initcall rand_initialize+0x0/0x2c returned 0 after 0 msecs
> calling tty_init+0x0/0x1c5
> initcall tty_init+0x0/0x1c5 returned 0 after 25 msecs
> calling pty_init+0x0/0x46e
> initcall pty_init+0x0/0x46e returned 0 after 25 msecs
> calling raw_init+0x0/0xca
> initcall raw_init+0x0/0xca returned 0 after 0 msecs
> calling r3964_init+0x0/0x3a
> r3964: Philips r3964 Driver $Revision: 1.10 $
> initcall r3964_init+0x0/0x3a returned 0 after 3 msecs
> calling applicom_init+0x0/0x4cc
> Applicom driver: $Id: ac.c,v 1.30 2000/03/22 16:03:57 dwmw2 Exp $
> ac.o: No PCI boards found.
> ac.o: For an ISA board you must supply memory and irq parameters.
> initcall applicom_init+0x0/0x4cc returned -6 after 13 msecs
> initcall applicom_init+0x0/0x4cc returned with error code -6
> calling rtc_init+0x0/0xa6
> Real Time Clock Driver v1.12ac
> initcall rtc_init+0x0/0xa6 returned 0 after 2 msecs
> calling hpet_init+0x0/0x60
> initcall hpet_init+0x0/0x60 returned 0 after 0 msecs
> calling nvram_init+0x0/0x80
> Non-volatile memory driver v1.2
> initcall nvram_init+0x0/0x80 returned 0 after 2 msecs
> calling i8k_init+0x0/0x154
> initcall i8k_init+0x0/0x154 returned -19 after 0 msecs
> calling mod_init+0x0/0x1da
> initcall mod_init+0x0/0x1da returned -19 after 0 msecs
> calling ppdev_init+0x0/0xa7
> ppdev: user-space parallel port driver
> initcall ppdev_init+0x0/0xa7 returned 0 after 3 msecs
> calling tlclk_init+0x0/0x1ae
> telclk_interrup = 0xf non-mcpbl0010 hw.
> initcall tlclk_init+0x0/0x1ae returned -6 after 3 msecs
> initcall tlclk_init+0x0/0x1ae returned with error code -6
> calling agp_init+0x0/0x26
> Linux agpgart interface v0.103
> initcall agp_init+0x0/0x26 returned 0 after 2 msecs
> calling agp_intel_init+0x0/0x24
> initcall agp_intel_init+0x0/0x24 returned 0 after 0 msecs
> calling agp_via_init+0x0/0x24
> initcall agp_via_init+0x0/0x24 returned 0 after 0 msecs
> calling init_atmel+0x0/0x17c
> initcall init_atmel+0x0/0x17c returned -19 after 0 msecs
> calling drm_core_init+0x0/0xef
> [drm] Initialized drm 1.1.0 20060810
> initcall drm_core_init+0x0/0xef returned 0 after 3 msecs
> calling tdfx_init+0x0/0xc
> initcall tdfx_init+0x0/0xc returned 0 after 0 msecs
> calling radeon_init+0x0/0x18
> pci 0000:01:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> PCI: Setting latency timer of device 0000:01:00.0 to 64
> [drm] Initialized radeon 1.29.0 20080528 on minor 0
> initcall radeon_init+0x0/0x18 returned 0 after 15 msecs
> calling i810_init+0x0/0x18
> initcall i810_init+0x0/0x18 returned 0 after 0 msecs
> calling i915_init+0x0/0x18
> initcall i915_init+0x0/0x18 returned 0 after 0 msecs
> calling sis_init+0x0/0x18
> initcall sis_init+0x0/0x18 returned 0 after 0 msecs
> calling via_init+0x0/0x22
> initcall via_init+0x0/0x22 returned 0 after 0 msecs
> calling serial8250_init+0x0/0x122
> Serial: 8250/16550 driver4 ports, IRQ sharing disabled
> �serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
> initcall serial8250_init+0x0/0x122 returned 0 after 251 msecs
> calling jsm_init_module+0x0/0x3e
> initcall jsm_init_module+0x0/0x3e returned 0 after 0 msecs
> calling parport_default_proc_register+0x0/0x1b
> initcall parport_default_proc_register+0x0/0x1b returned 0 after 0 msecs
> calling parport_pc_init+0x0/0x338
> parport_pc 00:08: reported by Plug and Play ACPI
> parport0: PC-style at 0x378 (0x778), irq 7 [PCSPP(,...)]
> initcall parport_pc_init+0x0/0x338 returned 0 after 10 msecs
> calling parport_ax88796_init+0x0/0xc
> initcall parport_ax88796_init+0x0/0xc returned 0 after 0 msecs
> calling topology_sysfs_init+0x0/0x41
> initcall topology_sysfs_init+0x0/0x41 returned 0 after 0 msecs
> calling loop_init+0x0/0x196
> loop: module loaded
> initcall loop_init+0x0/0x196 returned 0 after 2 msecs
> calling cpqarray_init+0x0/0x251
> Compaq SMART2 Driver (v 2.6.0)
> initcall cpqarray_init+0x0/0x251 returned 0 after 2 msecs
> calling cciss_init+0x0/0x28
> HP CISS Driver (v 3.6.20)
> initcall cciss_init+0x0/0x28 returned 0 after 2 msecs
> calling ub_init+0x0/0x6a
> usbcore: registered new interface driver ub
> initcall ub_init+0x0/0x6a returned 0 after 3 msecs
> calling pasic3_base_init+0x0/0x13
> initcall pasic3_base_init+0x0/0x13 returned -19 after 0 msecs
> calling e1000_init_module+0x0/0x61
> e1000e: Intel(R) PRO/1000 Network Driver - 0.3.3.3-k2
> e1000e: Copyright (c) 1999-2008 Intel Corporation.
> e1000e 0000:04:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
> PCI: Setting latency timer of device 0000:04:00.0 to 64
> eth0: (PCI Express:2.5GB/s:Width x1) 00:16:76:ab:6e:84
> eth0: Intel(R) PRO/1000 Network Connection
> eth0: MAC: 2, PHY: 2, PBA No: ffffff-0ff
> initcall e1000_init_module+0x0/0x61 returned 0 after 141 msecs
> calling ixgb_init_module+0x0/0x4b
> Intel(R) PRO/10GbE Network Driver - version 1.0.135-k2-NAPI
> Copyright (c) 1999-2008 Intel Corporation.
> initcall ixgb_init_module+0x0/0x4b returned 0 after 8 msecs
> calling ipg_init_module+0x0/0x15
> initcall ipg_init_module+0x0/0x15 returned 0 after 0 msecs
> calling cxgb3_init_module+0x0/0x20
> initcall cxgb3_init_module+0x0/0x20 returned 0 after 0 msecs
> calling vcan_init_module+0x0/0x36
> vcan: Virtual CAN interface driver
> initcall vcan_init_module+0x0/0x36 returned 0 after 3 msecs
> calling atl1_init_module+0x0/0x15
> initcall atl1_init_module+0x0/0x15 returned 0 after 0 msecs
> calling plip_init+0x0/0x5d
> NET3 PLIP version 2.4-parport [email protected]
> plip0: Parallel port at 0x378, using IRQ 7.
> initcall plip_init+0x0/0x5d returned 0 after 8 msecs
> calling gem_init+0x0/0x15
> initcall gem_init+0x0/0x15 returned 0 after 0 msecs
> calling vortex_init+0x0/0xa9
> initcall vortex_init+0x0/0xa9 returned 0 after 0 msecs
> calling ne2k_pci_init+0x0/0x15
> initcall ne2k_pci_init+0x0/0x15 returned 0 after 0 msecs
> calling e100_init_module+0x0/0x5c
> e100: Intel(R) PRO/100 Network Driver, 3.5.23-k4-NAPI
> e100: Copyright(c) 1999-2006 Intel Corporation
> initcall e100_init_module+0x0/0x5c returned 0 after 8 msecs
> calling tlan_probe+0x0/0xda
> ThunderLAN driver v1.15
> TLAN: 0 devices installed, PCI: 0 EISA: 0
> initcall tlan_probe+0x0/0xda returned -19 after 5 msecs
> calling epic_init+0x0/0x15
> initcall epic_init+0x0/0x15 returned 0 after 0 msecs
> calling sis190_init_module+0x0/0x15
> initcall sis190_init_module+0x0/0x15 returned 0 after 0 msecs
> calling r6040_init+0x0/0x15
> initcall r6040_init+0x0/0x15 returned 0 after 0 msecs
> calling yellowfin_init+0x0/0x15
> initcall yellowfin_init+0x0/0x15 returned 0 after 0 msecs
> calling natsemi_init_mod+0x0/0x15
> initcall natsemi_init_mod+0x0/0x15 returned 0 after 0 msecs
> calling ns83820_init+0x0/0x28
> ns83820.c: National Semiconductor DP83820 10/100/1000 driver.
> initcall ns83820_init+0x0/0x28 returned 0 after 5 msecs
> calling tg3_init+0x0/0x15
> initcall tg3_init+0x0/0x15 returned 0 after 0 msecs
> calling bnx2_init+0x0/0x15
> initcall bnx2_init+0x0/0x15 returned 0 after 0 msecs
> calling bnx2x_init+0x0/0x15
> initcall bnx2x_init+0x0/0x15 returned 0 after 0 msecs
> calling skge_init_module+0x0/0x15
> initcall skge_init_module+0x0/0x15 returned 0 after 0 msecs
> calling sky2_init_module+0x0/0x49
> initcall sky2_init_module+0x0/0x49 returned 0 after 0 msecs
> calling rhine_init+0x0/0x39
> initcall rhine_init+0x0/0x39 returned 0 after 0 msecs
> calling starfire_init+0x0/0x15
> initcall starfire_init+0x0/0x15 returned 0 after 0 msecs
> calling marvell_init+0x0/0x5e
> initcall marvell_init+0x0/0x5e returned 0 after 0 msecs
> calling cicada_init+0x0/0x35
> initcall cicada_init+0x0/0x35 returned 0 after 0 msecs
> calling lxt_init+0x0/0x35
> initcall lxt_init+0x0/0x35 returned 0 after 0 msecs
> calling qs6612_init+0x0/0xc
> initcall qs6612_init+0x0/0xc returned 0 after 0 msecs
> calling ip175c_init+0x0/0xc
> initcall ip175c_init+0x0/0xc returned 0 after 0 msecs
> calling fixed_mdio_bus_init+0x0/0x9e
> Fixed MDIO Bus: probed
> initcall fixed_mdio_bus_init+0x0/0x9e returned 0 after 2 msecs
> calling sundance_init+0x0/0x15
> initcall sundance_init+0x0/0x15 returned 0 after 0 msecs
> calling hamachi_init+0x0/0x15
> initcall hamachi_init+0x0/0x15 returned 0 after 0 msecs
> calling net_olddevs_init+0x0/0x95
> initcall net_olddevs_init+0x0/0x95 returned 0 after 0 msecs
> calling b44_init+0x0/0x59
> initcall b44_init+0x0/0x59 returned 0 after 0 msecs
> calling init_nic+0x0/0x15
> initcall init_nic+0x0/0x15 returned 0 after 0 msecs
> calling ql3xxx_init_module+0x0/0x15
> initcall ql3xxx_init_module+0x0/0x15 returned 0 after 0 msecs
> calling dummy_init_module+0x0/0xb2
> initcall dummy_init_module+0x0/0xb2 returned 0 after 0 msecs
> calling macvlan_init_module+0x0/0x49
> initcall macvlan_init_module+0x0/0x49 returned 0 after 0 msecs
> calling dfx_init+0x0/0x15
> initcall dfx_init+0x0/0x15 returned 0 after 0 msecs
> calling rtl8139_init_module+0x0/0x15
> initcall rtl8139_init_module+0x0/0x15 returned 0 after 0 msecs
> calling atp_init_module+0x0/0x93
> atp.c:v1.09=ac 2002/10/01 Donald Becker <[email protected]>
> initcall atp_init_module+0x0/0x93 returned -19 after 5 msecs
> calling eql_init_module+0x0/0x5b
> Equalizer2002: Simon Janes ([email protected]) and David S. Miller ([email protected])
> initcall eql_init_module+0x0/0x5b returned 0 after 7 msecs
> calling tun_init+0x0/0x96
> tun: Universal TUN/TAP device driver, 1.6
> tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
> initcall tun_init+0x0/0x96 returned 0 after 8 msecs
> calling rio_init+0x0/0x15
> initcall rio_init+0x0/0x15 returned 0 after 0 msecs
> calling s2io_starter+0x0/0x15
> initcall s2io_starter+0x0/0x15 returned 0 after 0 msecs
> calling xl_pci_init+0x0/0x15
> initcall xl_pci_init+0x0/0x15 returned 0 after 0 msecs
> calling init_dlci+0x0/0x35
> DLCI driver v0.35, 4 Jan 1997, [email protected].
> initcall init_dlci+0x0/0x35 returned 0 after 4 msecs
> calling usb_rtl8150_init+0x0/0x28
> rtl8150: rtl8150 based usb-ethernet driver v0.6.2 (2004/08/27)
> usbcore: registered new interface driver rtl8150
> initcall usb_rtl8150_init+0x0/0x28 returned 0 after 9 msecs
> calling cdc_init+0x0/0x15
> usbcore: registered new interface driver cdc_ether
> initcall cdc_init+0x0/0x15 returned 0 after 4 msecs
> calling dm9601_init+0x0/0x15
> usbcore: registered new interface driver dm9601
> initcall dm9601_init+0x0/0x15 returned 0 after 4 msecs
> calling usbnet_init+0x0/0x15
> usbcore: registered new interface driver gl620a
> initcall usbnet_init+0x0/0x15 returned 0 after 4 msecs
> calling plusb_init+0x0/0x15
> usbcore: registered new interface driver plusb
> initcall plusb_init+0x0/0x15 returned 0 after 4 msecs
> calling rndis_init+0x0/0x15
> usbcore: registered new interface driver rndis_host
> initcall rndis_init+0x0/0x15 returned 0 after 4 msecs
> calling cdc_subset_init+0x0/0x15
> usbcore: registered new interface driver cdc_subset
> initcall cdc_subset_init+0x0/0x15 returned 0 after 4 msecs
> calling mcs7830_init+0x0/0x15
> usbcore: registered new interface driver MOSCHIP usb-ethernet driver
> initcall mcs7830_init+0x0/0x15 returned 0 after 5 msecs
> calling usbnet_init+0x0/0x2b
> initcall usbnet_init+0x0/0x2b returned 0 after 0 msecs
> calling ipw2100_init+0x0/0x65
> ipw2100: Intel(R) PRO/Wireless 2100 Network Driver, git-1.2.2
> ipw2100: Copyright(c) 2003-2006 Intel Corporation
> initcall ipw2100_init+0x0/0x65 returned 0 after 9 msecs
> calling ipw_init+0x0/0x7f
> ipw2200: Intel(R) PRO/Wireless 2200/2915 Network Driver, 1.2.2kd
> ipw2200: Copyright(c) 2003-2006 Intel Corporation
> initcall ipw_init+0x0/0x7f returned 0 after 9 msecs
> calling init_orinoco+0x0/0x1d
> orinoco 0.15 (David Gibson <[email protected]>, Pavel Roskin <[email protected]>, et al)
> initcall init_orinoco+0x0/0x1d returned 0 after 8 msecs
> calling init_hermes+0x0/0x3
> initcall init_hermes+0x0/0x3 returned 0 after 0 msecs
> calling orinoco_plx_init+0x0/0x2f
> orinoco_plx 0.15 (Pavel Roskin <[email protected]>, David Gibson <[email protected]>, Daniel Barlow <[email protected]>)
> initcall orinoco_plx_init+0x0/0x2f returned 0 after 10 msecs
> calling orinoco_nortel_init+0x0/0x2f
> orinoco_nortel 0.15 (Tobias Hoffmann & Christoph Jungegger <[email protected]>)
> initcall orinoco_nortel_init+0x0/0x2f returned 0 after 6 msecs
> calling airo_init_module+0x0/0xe3
> airo(): Probing for PCI adapters
> airo(): Finished probing for PCI adapters
> initcall airo_init_module+0x0/0xe3 returned 0 after 6 msecs
> calling prism54_module_init+0x0/0x36
> Loaded prism54 driver, version 1.2
> initcall prism54_module_init+0x0/0x36 returned 0 after 3 msecs
> calling b43_init+0x0/0x42
> Broadcom 43xx driver loaded [ Features: PL, Firmware-ID: FW13 ]
> initcall b43_init+0x0/0x42 returned 0 after 5 msecs
> calling usb_init+0x0/0xad
> zd1211rw usb_init()
> usbcore: registered new interface driver zd1211rw
> zd1211rw initialized
> initcall usb_init+0x0/0xad returned 0 after 8 msecs
> calling rndis_wlan_init+0x0/0x15
> usbcore: registered new interface driver rndis_wlan
> initcall rndis_wlan_init+0x0/0x15 returned 0 after 4 msecs
> calling zd1201_init+0x0/0x15
> usbcore: registered new interface driver zd1201
> initcall zd1201_init+0x0/0x15 returned 0 after 4 msecs
> calling lbs_init_module+0x0/0x38
> initcall lbs_init_module+0x0/0x38 returned 0 after 0 msecs
> calling if_sdio_init_module+0x0/0x2d
> libertas_sdio: Libertas SDIO driver
> libertas_sdio: Copyright Pierre Ossman
> initcall if_sdio_init_module+0x0/0x2d returned 0 after 6 msecs
> calling rtl8180_init+0x0/0x15
> initcall rtl8180_init+0x0/0x15 returned 0 after 0 msecs
> calling rtl8187_init+0x0/0x15
> usbcore: registered new interface driver rtl8187
> initcall rtl8187_init+0x0/0x15 returned 0 after 4 msecs
> calling iwl4965_init+0x0/0x6c
> iwl4965: Intel(R) Wireless WiFi Link 4965AGN driver for Linux, 1.3.27kd
> iwl4965: Copyright(c) 2003-2008 Intel Corporation
> initcall iwl4965_init+0x0/0x6c returned 0 after 10 msecs
> calling init_mac80211_hwsim+0x0/0x2f0
> mac80211_hwsim: Initializing radio 0
> phy0: Failed to select rate control algorithm
> phy0: Failed to initialize rate control algorithm
> mac80211_hwsim: ieee80211_register_hw failed (-2)
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000370
> IP: [<ffffffff808be0c2>] rollback_registered+0x37/0xfb
> PGD 0
> Oops: 0000 [1] SMP
> CPU 1
> Pid: 1, comm: swapper Not tainted 2.6.26-05253-g14b395e #21308
> RIP: 0010:[<ffffffff808be0c2>] [<ffffffff808be0c2>] rollback_registered+0x37/0xfb
> RSP: 0018:ffff88003f83fe00 EFLAGS: 00010212
> RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffff88003d078ed8
> RDX: ffffffff8095de3d RSI: 0000000000000046 RDI: 0000000000000000
> RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8800042facc0
> R10: 0000000000000000 R11: ffffffff804021ae R12: 0000000000000000
> R13: ffff88003d0799a0 R14: 0000000000000000 R15: 0000000000000008
> FS: 0000000000000000(0000) GS:ffff88003f829160(0000) knlGS:0000000000000000
> CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 0000000000000370 CR3: 0000000000201000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process swapper (pid: 1, threadinfo ffff88003f83e000, task ffff88003f824000)
> Stack: 0000000000000000 ffffffff808be1b8 ffff88003d0782c0 ffffffff809621ab
> ffff88003d0799a0 ffffffff805f1e5d 00000000fffffffe ffff88003d0782c0
> ffff88003d0799e0 ffffffff811e83e4 ffff88003f83fea0 ffffffff80248f12
> Call Trace:
> [<ffffffff808be1b8>] unregister_netdevice+0x32/0x71
> [<ffffffff809621ab>] ieee80211_unregister_hw+0x35/0xd4
> [<ffffffff805f1e5d>] mac80211_hwsim_free+0x1d/0x6a
> [<ffffffff811e83e4>] init_mac80211_hwsim+0x2df/0x2f0
> [<ffffffff80248f12>] getnstimeofday+0x38/0x95
> [<ffffffff80246f48>] ktime_get_ts+0x21/0x49
> [<ffffffff811e8105>] init_mac80211_hwsim+0x0/0x2f0
> [<ffffffff811cb8c9>] kernel_init+0x143/0x295
> [<ffffffff80209b79>] __switch_to+0xb6/0x3bb
> [<ffffffff8020c399>] child_rip+0xa/0x11
> [<ffffffff811cb786>] kernel_init+0x0/0x295
> [<ffffffff8020c38f>] child_rip+0x0/0x11
>
>
> Code: 04 0f 0b eb fe e8 e3 78 00 00 85 c0 75 1d ba ce 0e 00 00 48 c7 c6 b8 e1 d8 80 48 c7 c7 3b 6c ce 80 e8 24 65 97 ff e8 0e f8 94 ff <8b> 83 70 03 00 00 85 c0 75 24 48 89 de 48 89 da 48 c7 c7 e4 e4
> RIP [<ffffffff808be0c2>] rollback_registered+0x37/0xfb
> RSP <ffff88003f83fe00>
> CR2: 0000000000000370
> ---[ end trace 4b01b4c1a97dbebf ]---
> Kernel panic - not syncing: Attempted to kill init!
????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay?ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m????????????I?

2008-07-21 15:24:32

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

From: Ingo Molnar <[email protected]>
Date: Mon, 21 Jul 2008 17:04:48 +0200

[ Adding linux-wireless CC, again, Ingo please retain it for
followups, thanks! ]

> * Ingo Molnar <[email protected]> wrote:
>
> > > Pid: 1, comm: swapper Not tainted 2.6.26-tip-00013-g6de15c6-dirty #21290
> >
> > some more information: find below the same crash with vanilla
> > linus/master and no extra patches. The crash site is:
>
> a 32-bit testbox just triggered the same crash too:
>
> calling init_mac80211_hwsim+0x0/0x310
> mac80211_hwsim: Initializing radio 0
> phy0: Failed to select rate control algorithm
> phy0: Failed to initialize rate control algorithm
> mac80211_hwsim: ieee80211_register_hw failed (-2)
> BUG: unable to handle kernel NULL pointer dereference at 00000298
> IP: [<c06efb98>] rollback_registered+0x28/0x120
> *pdpt = 0000000000bc9001 *pde = 0000000000000000
> Oops: 0000 [#1] PREEMPT SMP
>
> and that system has no wireless so i guess it's just some unregister
> inbalance kind of init/deinit buglet.
>
> Ingo
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2008-07-21 15:28:16

[permalink] [raw]

Subject: Re: [GIT]: Networking

From: Alexey Dobriyan <[email protected]>
Date: Mon, 21 Jul 2008 15:57:42 +0400

> On Sun, Jul 20, 2008 at 10:11:06PM -0700, David Miller wrote:
> > Alexey, please try this patch:
> >
> > atl1: Do not wake queue before queue has been started.
>
> Patch helps and networking is back with and without using netconsole,
> thank you!
>
> Also wan to say, 8139too driver survived multiqueue TX changes fine.

Thank you very much for testing, I'll push this fix.

2008-07-21 15:35:52

[permalink] [raw]

Subject: Re: [GIT]: Networking

From: Ben Hutchings <[email protected]>
Date: Mon, 21 Jul 2008 11:16:27 +0100

> Alexander Beregalov wrote:
> > Hi David!
> >
> > There is another problem on sparc64 and happymeal ethernet card.
> >
> > when tring to up interface:
> >
> > kernel BUG at net/core/dev.c:1328
> [...]
>
> This is yet another driver calling netif_wake_queue() during dev_open(),
> when there is no real qdisc present. (And yes, sfc is another of those
> drivers - I will post a patch after internal review.)

Yep, what idiot wrote this driver? ;-)

Alexander please try this patch:

sunhme: Remove stop/wake TX queue calls in set-multicast-list handler.

Based upon a bug report by Alexander Beregalov and commentary
from Ben Hutchings.

These are totally unnecessary, in particular because this
driver's ->hard_start_xmit() handler takes the same driver
spinlock that the set-multicast-list handler uses.

Signed-off-by: David S. Miller <[email protected]>

diff --git a/drivers/net/sunhme.c b/drivers/net/sunhme.c
index 1aa425b..b79d5f0 100644
--- a/drivers/net/sunhme.c
+++ b/drivers/net/sunhme.c
@@ -2377,8 +2377,6 @@ static void happy_meal_set_multicast(struct net_device *dev)

spin_lock_irq(&hp->happy_lock);

- netif_stop_queue(dev);
-
if ((dev->flags & IFF_ALLMULTI) || (dev->mc_count > 64)) {
hme_write32(hp, bregs + BMAC_HTABLE0, 0xffff);
hme_write32(hp, bregs + BMAC_HTABLE1, 0xffff);
@@ -2410,8 +2408,6 @@ static void happy_meal_set_multicast(struct net_device *dev)
hme_write32(hp, bregs + BMAC_HTABLE3, hash_table[3]);
}

- netif_wake_queue(dev);
-
spin_unlock_irq(&hp->happy_lock);
}

2008-07-21 16:04:55

by Alexander Beregalov

[permalink] [raw]

Subject: Re: [GIT]: Networking

2008/7/21 David Miller <[email protected]>:
> From: Ben Hutchings <[email protected]>
> Date: Mon, 21 Jul 2008 11:16:27 +0100
>
>> Alexander Beregalov wrote:
>> > Hi David!
>> >
>> > There is another problem on sparc64 and happymeal ethernet card.
>> >
>> > when tring to up interface:
>> >
>> > kernel BUG at net/core/dev.c:1328
>> [...]
>>
>> This is yet another driver calling netif_wake_queue() during dev_open(),
>> when there is no real qdisc present. (And yes, sfc is another of those
>> drivers - I will post a patch after internal review.)
>
> Yep, what idiot wrote this driver? ;-)
>
> Alexander please try this patch:
It works, thanks.

2008-07-21 16:50:11

[permalink] [raw]

Subject: Re: [GIT]: Networking

On Sun, 20 Jul 2008, David Miller wrote:
>
> Alexey, please try this patch:
>
> atl1: Do not wake queue before queue has been started.

David, can we please make this all a bit less fragile?

There are _millions_ of network drivers, and these changes seem to have
broken not just common drivers, but also drivers that "work" seem to now
have broken suspend/resume.

There's at least one report of suspend apparently oopsing now, and I
assume it's basically the same thing - it was bisected down to that same
37437bb2e1ae8af470dfcd5b4ff454110894ccaf commit ("pkt_sched: Schedule
qdiscs instead of netdev_queue.")

Why is it so unnecessarily fragile to begin with? Especially for stuff
that happens at bootup or suspend, doing a BUG_ON() is _particularly_
painful, because a dead machine means that you cannot get any logs or
anything else.

So wouldn't it be *much* better to do something like the appended, and at
least try to limp on, and maybe have a system that people can get logs
out of?

Linus

---
net/core/dev.c | 3 ++-
1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/net/core/dev.c b/net/core/dev.c
index 2eed17b..43ab4f5 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -1325,7 +1325,8 @@ static void dev_queue_xmit_nit(struct sk_buff *skb, struct net_device *dev)

void __netif_schedule(struct Qdisc *q)
{
- BUG_ON(q == &noop_qdisc);
+ if (WARN_ON_ONCE(q == &noop_qdisc))
+ return;

if (!test_and_set_bit(__QDISC_STATE_SCHED, &q->state)) {
struct softnet_data *sd;
---

2008-07-21 16:53:41

[permalink] [raw]

Subject: Re: [GIT]: Networking

From: Linus Torvalds <[email protected]>
Date: Mon, 21 Jul 2008 09:49:49 -0700 (PDT)

> Why is it so unnecessarily fragile to begin with? Especially for stuff
> that happens at bootup or suspend, doing a BUG_ON() is _particularly_
> painful, because a dead machine means that you cannot get any logs or
> anything else.

I agree, the BUG_ON() was the wrong way to go.

> So wouldn't it be *much* better to do something like the appended, and at
> least try to limp on, and maybe have a system that people can get logs
> out of?

Agreed, I'll apply this. Thanks.

> diff --git a/net/core/dev.c b/net/core/dev.c
> index 2eed17b..43ab4f5 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -1325,7 +1325,8 @@ static void dev_queue_xmit_nit(struct sk_buff *skb, struct net_device *dev)
>
> void __netif_schedule(struct Qdisc *q)
> {
> - BUG_ON(q == &noop_qdisc);
> + if (WARN_ON_ONCE(q == &noop_qdisc))
> + return;
>
> if (!test_and_set_bit(__QDISC_STATE_SCHED, &q->state)) {
> struct softnet_data *sd;
> ---

2008-07-21 17:29:08

[permalink] [raw]

Subject: Re: [GIT]: Networking

From: Patrick McHardy <[email protected]>
Date: Mon, 21 Jul 2008 14:05:57 +0200

> The idea was that NETFILTER_ADVANCED=n enables everything needed
> by mainstream distributions and hides the rest. We can certainly
> change the default for this option, but that makes NETFILTER_ADVANCED
> pretty much useless.

A new feature cannot possibly be used by existing distributions. I
think that's the main gripe.

Distributions themselves will enable the feature no matter what we
mark it's default as.

And once the feature thus becomes pervasive we can re-adjust the
default.

2008-07-21 17:40:46

[permalink] [raw]

Subject: Re: [GIT]: Networking

On Mon, 21 Jul 2008, David Miller wrote:

> From: Patrick McHardy <[email protected]>
> Date: Mon, 21 Jul 2008 14:05:57 +0200
>
> > The idea was that NETFILTER_ADVANCED=n enables everything needed
> > by mainstream distributions and hides the rest. We can certainly
> > change the default for this option, but that makes NETFILTER_ADVANCED
> > pretty much useless.
>
> A new feature cannot possibly be used by existing distributions. I
> think that's the main gripe.

Well, if the feature really is going to be something that a _normal_
netfilter config needs, then it should indeed be turned on.

However, nothing in the docs imply that at all. Can you explain? Why
should IP_NF_SECURITY be on, and why should a default netfilter table
enable it? And if it should, WHY THE HELL IS IT DOCUMENTED THAT YOU SHOULD
SAY 'N'?

Patrick, see my original report:

> Grr. And I quote:
>
> Security table (IP_NF_SECURITY) [Y/n/?] (NEW) ?
>
> This option adds a `security' table to iptables, for use
> with Mandatory Access Control (MAC) policy.
>
> If unsure, say N.

That option as it stands now MAKES NO SENSE. Either you should say 'Y'
(and you should explain _why_), or you should say 'N' (as documented) and
it should damn well default to 'N' too!

Linus

2008-07-21 18:19:18

by Ian Schram

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

I was looking at this out of interest, but I'm in no way familiar with the code.

Looks to me that the error handling code in mac80211_hwsim is awkward. Which
leads to it calling ieee80211_unregister_hw even when ieee80211_register_hw failed.

The function has a for loop where it generates all simulated radios. when something
fails, the error handling will call mac80211_hwsim_free which frees all simulated radios
who's pointer isn't zero. However the information stored is insufficient to determine
whether or not the call to ieee80211_register_hw succeeded or not for a specific radio.
The included patch makes init_mac80211_hwsim clean up the current simulated radio,
and then calls into mac80211_hwsim_free to clean up all the radios that did succeed.

This however doesn't explain why the rate control registration failed.. build tested this,
but had some problems reproducing the original problem.

signed-off-by: Ian Schram <[email protected]>
--- a/mac80211_hwsim.c 2008-07-21 18:48:38.000000000 +0200
+++ b/mac80211_hwsim.c 2008-07-21 19:31:44.000000000 +0200
@@ -364,8 +364,7 @@ static void mac80211_hwsim_free(void)
struct mac80211_hwsim_data *data;
data = hwsim_radios[i]->priv;
ieee80211_unregister_hw(hwsim_radios[i]);
- if (!IS_ERR(data->dev))
- device_unregister(data->dev);
+ device_unregister(data->dev);
ieee80211_free_hw(hwsim_radios[i]);
}
}
@@ -437,7 +436,7 @@ static int __init init_mac80211_hwsim(vo
"mac80211_hwsim: device_create_drvdata "
"failed (%ld)\n", PTR_ERR(data->dev));
err = -ENOMEM;
- goto failed;
+ goto failed_drvdata;
}
data->dev->driver = &mac80211_hwsim_driver;

@@ -461,7 +460,7 @@ static int __init init_mac80211_hwsim(vo
if (err < 0) {
printk(KERN_DEBUG "mac80211_hwsim: "
"ieee80211_register_hw failed (%d)\n", err);
- goto failed;
+ goto failed_hw;
}

printk(KERN_DEBUG "%s: hwaddr %s registered\n",
@@ -479,9 +478,9 @@ static int __init init_mac80211_hwsim(vo
rtnl_lock();

err = dev_alloc_name(hwsim_mon, hwsim_mon->name);
- if (err < 0) {
+ if (err < 0)
goto failed_mon;
- }
+

err = register_netdevice(hwsim_mon);
if (err < 0)
@@ -494,7 +493,14 @@ static int __init init_mac80211_hwsim(vo
failed_mon:
rtnl_unlock();
free_netdev(hwsim_mon);
+ mac80211_hwsim_free();
+ return err;

+failed_hw:
+ device_unregister(data->dev);
+failed_drvdata:
+ ieee80211_free_hw(hw);
+ hwsim_radios[i] = 0;
failed:
mac80211_hwsim_free();
return err;

2008-07-21 18:23:54

[permalink] [raw]

Subject: [crash] kernel BUG at net/core/dev.c:1328!

another one:

[ 24.426510] bus: 'platform': add device i8042
[ 24.430389] PM: Adding info for platform:i8042
[ 24.434799] ohci1394: fw-host0: IntEvent: 00030010
[ 24.434799] ohci1394: fw-host0: irq_handler: Bus reset requested
[ 24.434799] ohci1394: fw-host0: Cancel request received
[ 24.434799] ohci1394: fw-host0: Got RQPkt interrupt status=0x00008409
[ 24.434799] ohci1394: fw-host0: SelfID interrupt received (phyid 0, root)
[ 24.434799] ohci1394: fw-host0: SelfID packet 0x807f8c56 received
[ 24.434799] ieee1394: Including SelfID 0x568c7f80
[ 24.434799] ohci1394: fw-host0: SelfID for this node is 0x807f8c56
[ 24.434799] ohci1394: fw-host0: SelfID complete
[ 24.434799] ohci1394: fw-host0: PhyReqFilter=ffffffffffffffff
[ 24.434799] ieee1394: selfid_complete called with successful SelfID stage ... irm_id: 0xFFC0 node_id: 0xFFC0
[ 24.434799] ieee1394: NodeMgr: Processing reset for host 0
[ 24.434799] ------------[ cut here ]------------
[ 24.434799] kernel BUG at net/core/dev.c:1328!
[ 24.434799] invalid opcode: 0000 [1] SMP
[ 24.434799] CPU 0
[ 24.434799] Pid: 6, comm: events/0 Not tainted 2.6.26-tip-05790-gcad008f-dirty #13278
[ 24.434799] RIP: 0010:[<ffffffff809f5a9d>] [<ffffffff809f5a9d>] __netif_schedule+0x9d/0xb0
[ 24.434799] RSP: 0018:ffffffff812f6de8 EFLAGS: 00010046
[ 24.434799] RAX: 00000000ffffffff RBX: ffffffff81021d80 RCX: 0000000000000000
[ 24.434799] RDX: ffff88003c894930 RSI: 0000000000000000 RDI: ffffffff81021d80
[ 24.434799] RBP: ffffffff812f6df8 R08: 0000000000000000 R09: 0000000000000001
[ 24.434799] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88003c884000
[ 24.434799] R13: 0000000000000086 R14: ffff88003c8959d8 R15: ffff88003c886410
[ 24.434799] FS: 0000000000000000(0000) GS:ffffffff8115af80(0000) knlGS:0000000000000000
[ 24.434799] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[ 24.434799] CR2: 0000000000000000 CR3: 0000000000201000 CR4: 00000000000006e0
[ 24.434799] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 24.434799] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 24.434799] Process events/0 (pid: 6, threadinfo ffff88003f676000, task ffff88003f674180)
[ 24.434799] Stack: ffff88003c8958c0 ffff88003c884000 ffffffff812f6e48 ffffffff8088b66a
[ 24.434799] ffff88003c8959e0 ffff88003c895000 0000000000010000 ffffffff80fe7040
[ 24.434799] ffff88003c884000 0000000000000086 0000000000010000 ffff88003c886410
[ 24.434799] Call Trace:
[ 24.434799] <IRQ> [<ffffffff8088b66a>] ether1394_host_reset+0x10a/0x120
[ 24.434799] [<ffffffff80871a3e>] highlevel_host_reset+0x3e/0x80
[ 24.434799] [<ffffffff8086febc>] hpsb_selfid_complete+0x25c/0x330
[ 24.434799] [<ffffffff8087e22b>] ohci_irq_handler+0x82b/0xbc0
[ 24.434799] [<ffffffff80276e4b>] handle_IRQ_event+0x3b/0x70
[ 24.434799] [<ffffffff80278de9>] handle_level_irq+0xa9/0x130
[ 24.434799] [<ffffffff80210d98>] do_IRQ+0xc8/0x1b0
[ 24.434799] [<ffffffff8020ce13>] ret_from_intr+0x0/0x2e
[ 24.434799] <EOI> [<ffffffff80ba981f>] ? _spin_unlock_irqrestore+0x5f/0x70
[ 24.434799] [<ffffffff80879c16>] ? set_phy_reg+0x86/0xd0
[ 24.434799] [<ffffffff80871040>] ? delayed_reset_bus+0x0/0xf0
[ 24.434799] [<ffffffff8087a403>] ? ohci_devctl+0x153/0x240
[ 24.434799] [<ffffffff8086e7a2>] ? hpsb_reset_bus+0x22/0x30
[ 24.434799] [<ffffffff808710ec>] ? delayed_reset_bus+0xac/0xf0
[ 24.434799] [<ffffffff802562f9>] ? run_workqueue+0x179/0x220
[ 24.434799] [<ffffffff80256f04>] ? worker_thread+0xa4/0x110
[ 24.434799] [<ffffffff8025a320>] ? autoremove_wake_function+0x0/0x40
[ 24.434799] [<ffffffff80256e60>] ? worker_thread+0x0/0x110
[ 24.434799] [<ffffffff80259eed>] ? kthread+0x4d/0x80
[ 24.434799] [<ffffffff8020dca9>] ? child_rip+0xa/0x11
[ 24.434799] [<ffffffff8020cf3e>] ? restore_args+0x0/0x30
[ 24.434799] [<ffffffff80259ea0>] ? kthread+0x0/0x80
[ 24.434799] [<ffffffff8020dc9f>] ? child_rip+0x0/0x11
[ 24.434799]
[ 24.434799]
[ 24.434799] Code: 15 4c 89 e7 57 9d 66 0f 1f 44 00 00 e8 bd f1 86 ff 5b 41 5c c9 c3 e8 13 16 87 ff 4c 89 e7 57 9d 66 0f 1f 44 00 00 5b 41 5c c9 c3 <0f> 0b eb fe 66 66 66 90 66 66 66 90 66 66 66 90 66 66 90 55 48
[ 24.434799] RIP [<ffffffff809f5a9d>] __netif_schedule+0x9d/0xb0
[ 24.434799] RSP <ffffffff812f6de8>
[ 24.434799] Kernel panic - not syncing: Fatal exception in interrupt
[ 24.434799] Pid: 6, comm: events/0 Tainted: G D 2.6.26-tip-05790-gcad008f-dirty #13278

with this config:

http://redhat.com/~mingo/misc/config-Mon_Jul_21_20_05_45_CEST_2008.bad

(since this seems to have impacted ieee1394, Stefan Cc:-ed)

Should i spend time on bisecting this, or is this known already?

Ingo

2008-07-21 18:36:44

[permalink] [raw]

Subject: Re: [crash] kernel BUG at net/core/dev.c:1328!

On Mon, 21 Jul 2008, Ingo Molnar wrote:
>
> [ 24.434799] kernel BUG at net/core/dev.c:1328!

Ok, it is now a WARN_ON_ONCE() in my tree (which I _just_ pushed out).

So it's going to cause irritating messages (once), but the machine should
hopefully work.

> Should i spend time on bisecting this, or is this known already?

It's going to bisect down to the same commit you already bisected once,
it's the networking code that changed some of the rules, so various
network drivers that didn't follow the expected rules are now unhappy.

Maybe the network drivers are few enough that it will get fixed, or maybe
the WARN_ON_ONCE() will just be removed and the rule not reinforced.

I personally suspect the latter, since it seems to happen with just about
_any_ random network driver, including the common and well-maintained ones
(ie the Gods only help us for the truly odd/random cases)

Linus

2008-07-21 18:46:50

[permalink] [raw]

Subject: Re: [crash] kernel BUG at net/core/dev.c:1328!

* Linus Torvalds <[email protected]> wrote:

> On Mon, 21 Jul 2008, Ingo Molnar wrote:
> >
> > [ 24.434799] kernel BUG at net/core/dev.c:1328!
>
> Ok, it is now a WARN_ON_ONCE() in my tree (which I _just_ pushed out).
>
> So it's going to cause irritating messages (once), but the machine should
> hopefully work.
>
> > Should i spend time on bisecting this, or is this known already?
>
> It's going to bisect down to the same commit you already bisected
> once, it's the networking code that changed some of the rules, so
> various network drivers that didn't follow the expected rules are now
> unhappy.
>
> Maybe the network drivers are few enough that it will get fixed, or
> maybe the WARN_ON_ONCE() will just be removed and the rule not
> reinforced.

ok, have updated the testboxes to your latest push.

Btw., otherwise the big networking pull held up pretty well on a healthy
range of testboxes i have, it looked a lot scarier to me in the morning
than it turned out to be during the day. A couple of hundred tests
passed already and no indication of any runtime fragility so far. Boot
crashes/warnings can be annoying and hard to get a proper log of but
once the log is available they are normally quite easy for developers to
act upon.

Ingo

2008-07-21 19:01:09

[permalink] [raw]

Subject: Re: [crash] kernel BUG at net/core/dev.c:1328!

From: Linus Torvalds <[email protected]>
Date: Mon, 21 Jul 2008 11:35:21 -0700 (PDT)

> Maybe the network drivers are few enough that it will get fixed, or
> maybe the WARN_ON_ONCE() will just be removed and the rule not
> reinforced.
>
> I personally suspect the latter, since it seems to happen with just
> about _any_ random network driver, including the common and
> well-maintained ones (ie the Gods only help us for the truly
> odd/random cases)

Yes, we'll see how this plays out.

Ian Schram just posted a patch for the NULL pointer derfer in
wireless Ingo reported, so we'll see if that bug will be fixed
now as well.

2008-07-21 19:07:27

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

* Ian Schram <[email protected]> wrote:

> I was looking at this out of interest, but I'm in no way familiar with
> the code.

thanks Ian for the patch, i'll test it.

Note that it was whitespace damaged, find below a tidied up version of
the patch that i've applied to tip/out-of-tree.

Ingo

----------------------->
commit 2f77dd3a3b5c3a27298fa0a09d8703c09c633fc6
Author: Ian Schram <[email protected]>
Date: Mon Jul 21 20:18:25 2008 +0200

mac80211_hwsim.c: fix: BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

I was looking at this out of interest, but I'm in no way familiar with
the code.

Looks to me that the error handling code in mac80211_hwsim is awkward.
Which leads to it calling ieee80211_unregister_hw even when
ieee80211_register_hw failed.

The function has a for loop where it generates all simulated radios.
when something fails, the error handling will call mac80211_hwsim_free
which frees all simulated radios who's pointer isn't zero. However the
information stored is insufficient to determine whether or not the call
to ieee80211_register_hw succeeded or not for a specific radio. The
included patch makes init_mac80211_hwsim clean up the current simulated
radio, and then calls into mac80211_hwsim_free to clean up all the
radios that did succeed.

This however doesn't explain why the rate control registration failed..
build tested this, but had some problems reproducing the original
problem.

Signed-off-by: Ian Schram <[email protected]>
Signed-off-by: Ingo Molnar <[email protected]>
---
drivers/net/wireless/mac80211_hwsim.c | 18 ++++++++++++------
1 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/drivers/net/wireless/mac80211_hwsim.c b/drivers/net/wireless/mac80211_hwsim.c
index 913dc9f..5816230 100644
--- a/drivers/net/wireless/mac80211_hwsim.c
+++ b/drivers/net/wireless/mac80211_hwsim.c
@@ -364,8 +364,7 @@ static void mac80211_hwsim_free(void)
struct mac80211_hwsim_data *data;
data = hwsim_radios[i]->priv;
ieee80211_unregister_hw(hwsim_radios[i]);
- if (!IS_ERR(data->dev))
- device_unregister(data->dev);
+ device_unregister(data->dev);
ieee80211_free_hw(hwsim_radios[i]);
}
}
@@ -437,7 +436,7 @@ static int __init init_mac80211_hwsim(void)
"mac80211_hwsim: device_create_drvdata "
"failed (%ld)\n", PTR_ERR(data->dev));
err = -ENOMEM;
- goto failed;
+ goto failed_drvdata;
}
data->dev->driver = &mac80211_hwsim_driver;

@@ -461,7 +460,7 @@ static int __init init_mac80211_hwsim(void)
if (err < 0) {
printk(KERN_DEBUG "mac80211_hwsim: "
"ieee80211_register_hw failed (%d)\n", err);
- goto failed;
+ goto failed_hw;
}

printk(KERN_DEBUG "%s: hwaddr %s registered\n",
@@ -479,9 +478,9 @@ static int __init init_mac80211_hwsim(void)
rtnl_lock();

err = dev_alloc_name(hwsim_mon, hwsim_mon->name);
- if (err < 0) {
+ if (err < 0)
goto failed_mon;
- }
+

err = register_netdevice(hwsim_mon);
if (err < 0)
@@ -494,7 +493,14 @@ static int __init init_mac80211_hwsim(void)
failed_mon:
rtnl_unlock();
free_netdev(hwsim_mon);
+ mac80211_hwsim_free();
+ return err;

+failed_hw:
+ device_unregister(data->dev);
+failed_drvdata:
+ ieee80211_free_hw(hw);
+ hwsim_radios[i] = 0;
failed:
mac80211_hwsim_free();
return err;

2008-07-21 19:15:34

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

Ingo Molnar wrote:
> * Ian Schram <[email protected]> wrote:
>
>> I was looking at this out of interest, but I'm in no way familiar with
>> the code.
>
> thanks Ian for the patch, i'll test it.
>
> Note that it was whitespace damaged, find below a tidied up version of
> the patch that i've applied to tip/out-of-tree.
>
> Ingo

This patch may be needed to fix error handling in the hw_sim code, but I get the
crash even with that code disabled. I'm currently bisecting to find the culprit.

Larry

2008-07-21 19:21:13

[permalink] [raw]

Subject: Re: [crash] kernel BUG at net/core/dev.c:1328!

David Miller wrote:
> From: Linus Torvalds <[email protected]>
> Date: Mon, 21 Jul 2008 11:35:21 -0700 (PDT)
>
>> Maybe the network drivers are few enough that it will get fixed, or
>> maybe the WARN_ON_ONCE() will just be removed and the rule not
>> reinforced.
>>
>> I personally suspect the latter, since it seems to happen with just
>> about _any_ random network driver, including the common and
>> well-maintained ones (ie the Gods only help us for the truly
>> odd/random cases)
>
> Yes, we'll see how this plays out.
>
> Ian Schram just posted a patch for the NULL pointer derfer in
> wireless Ingo reported, so we'll see if that bug will be fixed
> now as well.

In the meantime: Is there perhaps something obviously wrong with
drivers/ieee1394/eth1394.c's netdevice initialization? We do it in
ether1394_add_host(), and shortly thereafter the crashing
ether1394_host_reset() is called. So we have essentially

(add host)
dev = alloc_netdev(...);
initialize various members in dev...
register_netdev(dev);

(host reset)
netif_stop_queue(dev);
discard some stale 1394 stuff if there were some...
netif_wake_queue(dev); <-- crashes in __netif_schedule(dev);

--
Stefan Richter
-=====-==--- -=== =-=-=
http://arcgraph.de/sr/

2008-07-21 19:31:18

[permalink] [raw]

Subject: Re: [crash] kernel BUG at net/core/dev.c:1328!

* Ingo Molnar <[email protected]> wrote:

> ok, have updated the testboxes to your latest push.

and indeed i got the new warning:

[ 24.460170] ieee1394: selfid_complete called with successful SelfID stage ... irm_id: 0xFFC0 node_id: 0xFFC0
[ 24.460170] ieee1394: NodeMgr: Processing reset for host 0
[ 24.460170] ------------[ cut here ]------------
[ 24.460170] WARNING: at net/core/dev.c:1330 __netif_schedule+0xc6/0xe0()
[ 24.460170] Pid: 6, comm: events/0 Not tainted 2.6.26-tip-05989-g5172ce5-dirty #13279
[ 24.460170]
[ 24.460170] Call Trace:
[ 24.460170] <IRQ> [<ffffffff8024461a>] warn_on_slowpath+0x5a/0x90
[ 24.460170] [<ffffffff80264c4d>] ? trace_hardirqs_off+0xd/0x10
[ 24.460170] [<ffffffff80ba98ca>] ? _spin_unlock_irqrestore+0x4a/0x70
[ 24.460170] [<ffffffff8088b501>] ? ether1394_reset_priv+0x121/0x140
[ 24.460170] [<ffffffff809f5db6>] __netif_schedule+0xc6/0xe0
[ 24.460170] [<ffffffff8088b62a>] ether1394_host_reset+0x10a/0x120
[ 24.460170] [<ffffffff808719fe>] highlevel_host_reset+0x3e/0x80
[ 24.460170] [<ffffffff8086fe7c>] hpsb_selfid_complete+0x25c/0x330
[ 24.460170] [<ffffffff8087e1eb>] ohci_irq_handler+0x82b/0xbc0
[ 24.460170] [<ffffffff80276e4b>] handle_IRQ_event+0x3b/0x70
[ 24.460170] [<ffffffff80278de9>] handle_level_irq+0xa9/0x130
[ 24.460170] [<ffffffff80210d98>] do_IRQ+0xc8/0x1b0
[ 24.460170] [<ffffffff8020ce13>] ret_from_intr+0x0/0x2e
[ 24.460170] <EOI> [<ffffffff80ba98df>] ? _spin_unlock_irqrestore+0x5f/0x70
[ 24.460170] [<ffffffff80879bd6>] ? set_phy_reg+0x86/0xd0
[ 24.460170] [<ffffffff80871000>] ? delayed_reset_bus+0x0/0xf0
[ 24.460170] [<ffffffff8087a3c3>] ? ohci_devctl+0x153/0x240
[ 24.460170] [<ffffffff8086e762>] ? hpsb_reset_bus+0x22/0x30
[ 24.460170] [<ffffffff808710ac>] ? delayed_reset_bus+0xac/0xf0
[ 24.460170] [<ffffffff802562f9>] ? run_workqueue+0x179/0x220
[ 24.460170] [<ffffffff80256f04>] ? worker_thread+0xa4/0x110
[ 24.460170] [<ffffffff8025a320>] ? autoremove_wake_function+0x0/0x40
[ 24.460170] [<ffffffff80256e60>] ? worker_thread+0x0/0x110
[ 24.460170] [<ffffffff80259eed>] ? kthread+0x4d/0x80
[ 24.460170] [<ffffffff8020dca9>] ? child_rip+0xa/0x11
[ 24.460170] [<ffffffff8020cf3e>] ? restore_args+0x0/0x30
[ 24.460170] [<ffffffff80259ea0>] ? kthread+0x0/0x80
[ 24.460170] [<ffffffff8020dc9f>] ? child_rip+0x0/0x11
[ 24.460170]
[ 24.460170] ---[ end trace 7b0ee76e032c9e07 ]---
[ 24.717334] ohci1394: fw-host0: Single packet rcv'd
[ 24.717334] ohci1394: fw-host0: Got phy packet ctx=0 ... discarded
[ 24.728466] Registering platform device 'i8042'. Parent at platform
[ 24.735133] device: 'i8042': device_add
[ 24.741452] bus: 'platform': add device i8042
[ 24.745331] PM: Adding info for platform:i8042

but otherwise the box has booted up just fine. (and even ohci1394 seemed
to continue fine after that warning)

Ingo

2008-07-21 19:35:53

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

* Larry Finger <[email protected]> wrote:

> Ingo Molnar wrote:
>> * Ian Schram <[email protected]> wrote:
>>
>>> I was looking at this out of interest, but I'm in no way familiar
>>> with the code.
>>
>> thanks Ian for the patch, i'll test it.
>>
>> Note that it was whitespace damaged, find below a tidied up version of
>> the patch that i've applied to tip/out-of-tree.
>>
>> Ingo
>
> This patch may be needed to fix error handling in the hw_sim code, but
> I get the crash even with that code disabled. I'm currently bisecting
> to find the culprit.

ok. I just reactivated CONFIG_MAC80211_HWSIM, applied Ian's fix and the
crash went away:

calling iwl4965_init+0x0/0x6c
iwl4965: Intel(R) Wireless WiFi Link 4965AGN driver for Linux, 1.3.27kd
iwl4965: Copyright(c) 2003-2008 Intel Corporation
initcall iwl4965_init+0x0/0x6c returned 0 after 10 msecs
calling init_mac80211_hwsim+0x0/0x31c
mac80211_hwsim: Initializing radio 0
PM: Adding info for No Bus:hwsim0
PM: Adding info for No Bus:phy0
PM: Adding info for No Bus:wmaster0
phy0: Failed to select rate control algorithm
phy0: Failed to initialize rate control algorithm
PM: Removing info for No Bus:wmaster0
PM: Removing info for No Bus:phy0
mac80211_hwsim: ieee80211_register_hw failed (-2)
PM: Removing info for No Bus:hwsim0
initcall init_mac80211_hwsim+0x0/0x31c returned -2 after 58 msecs
initcall init_mac80211_hwsim+0x0/0x31c returned with error code -2
calling dmfe_init_module+0x0/0xea
dmfe: Davicom DM9xxx net driver, version 1.36.4 (2002-01-17)
initcall dmfe_init_module+0x0/0xea returned 0 after 5 msecs

So at least as far as the init_mac80211_hwsim() deinit crash goes:

Tested-by: Ingo Molnar <[email protected]>

Ingo

2008-07-21 19:43:47

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

Ingo Molnar wrote:
> * Larry Finger <[email protected]> wrote:
>
>> Ingo Molnar wrote:
>>> * Ian Schram <[email protected]> wrote:
>>>
>>>> I was looking at this out of interest, but I'm in no way familiar
>>>> with the code.
>>> thanks Ian for the patch, i'll test it.
>>>
>>> Note that it was whitespace damaged, find below a tidied up version of
>>> the patch that i've applied to tip/out-of-tree.
>>>
>>> Ingo
>> This patch may be needed to fix error handling in the hw_sim code, but
>> I get the crash even with that code disabled. I'm currently bisecting
>> to find the culprit.
>
> ok. I just reactivated CONFIG_MAC80211_HWSIM, applied Ian's fix and the
> crash went away:
>
> calling iwl4965_init+0x0/0x6c
> iwl4965: Intel(R) Wireless WiFi Link 4965AGN driver for Linux, 1.3.27kd
> iwl4965: Copyright(c) 2003-2008 Intel Corporation
> initcall iwl4965_init+0x0/0x6c returned 0 after 10 msecs
> calling init_mac80211_hwsim+0x0/0x31c
> mac80211_hwsim: Initializing radio 0
> PM: Adding info for No Bus:hwsim0
> PM: Adding info for No Bus:phy0
> PM: Adding info for No Bus:wmaster0
> phy0: Failed to select rate control algorithm
> phy0: Failed to initialize rate control algorithm
> PM: Removing info for No Bus:wmaster0
> PM: Removing info for No Bus:phy0
> mac80211_hwsim: ieee80211_register_hw failed (-2)
> PM: Removing info for No Bus:hwsim0
> initcall init_mac80211_hwsim+0x0/0x31c returned -2 after 58 msecs
> initcall init_mac80211_hwsim+0x0/0x31c returned with error code -2
> calling dmfe_init_module+0x0/0xea
> dmfe: Davicom DM9xxx net driver, version 1.36.4 (2002-01-17)
> initcall dmfe_init_module+0x0/0xea returned 0 after 5 msecs
>
> So at least as far as the init_mac80211_hwsim() deinit crash goes:
>
> Tested-by: Ingo Molnar <[email protected]>

Yes, I'm chasing a distinct bug. The header for mine is

Jul 21 12:19:37 larrylap kernel: kernel BUG at net/core/dev.c:1328!
Jul 21 12:19:37 larrylap kernel: invalid opcode: 0000 [1] SMP
Jul 21 12:19:37 larrylap kernel: CPU 0
Jul 21 12:19:37 larrylap kernel: Modules linked in: af_packet rfkill_input nfs
lockd nfs_acl sunrpc cpufreq_conservative cpu
freq_userspace cpufreq_powersave powernow_k8 fuse loop dm_mod arc4 ecb
crypto_blkcipher b43 firmware_class rfkill mac80211 c
fg80211 snd_hda_intel snd_pcm snd_timer led_class snd k8temp input_polldev
sr_mod soundcore button battery hwmon cdrom force
deth ac serio_raw ssb snd_page_alloc sg ehci_hcd sd_mod ohci_hcd usbcore edd fan
thermal processor ext3 mbcache jbd pata_amd
ahci libata scsi_mod dock
Jul 21 12:19:37 larrylap kernel: Pid: 2057, comm: b43 Not tainted
2.6.26-Linus-git-05253-g14b395e #1
Jul 21 12:19:37 larrylap kernel: RIP: 0010:[<ffffffff8039ec4d>]
[<ffffffff8039ec4d>] __netif_schedule+0x12/0x75
Jul 21 12:19:37 larrylap kernel: RSP: 0000:ffff8800b9ae1de0 EFLAGS: 00010246

With an invalid opcode, mine is likely due to stack corruption.

Larry

2008-07-21 19:44:46

[permalink] [raw]

Subject: Re: [crash] kernel BUG at net/core/dev.c:1328!

* David Miller <[email protected]> wrote:

> From: Linus Torvalds <[email protected]>
> Date: Mon, 21 Jul 2008 11:35:21 -0700 (PDT)
>
> > Maybe the network drivers are few enough that it will get fixed, or
> > maybe the WARN_ON_ONCE() will just be removed and the rule not
> > reinforced.
> >
> > I personally suspect the latter, since it seems to happen with just
> > about _any_ random network driver, including the common and
> > well-maintained ones (ie the Gods only help us for the truly
> > odd/random cases)
>
> Yes, we'll see how this plays out.
>
> Ian Schram just posted a patch for the NULL pointer derfer in wireless
> Ingo reported, so we'll see if that bug will be fixed now as well.

Yes, the fix from Ian below solved the CONFIG_MAC80211_HWSIM=y crash i
was getting. I have no other pending issues other than a few low-prio
ne2000 build failures.

Thanks guys,

Ingo

--------->
commit 2f77dd3a3b5c3a27298fa0a09d8703c09c633fc6
Author: Ian Schram <[email protected]>
Date: Mon Jul 21 20:18:25 2008 +0200

mac80211_hwsim.c: fix: BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

I was looking at this out of interest, but I'm in no way familiar with
the code.

Looks to me that the error handling code in mac80211_hwsim is awkward.
Which leads to it calling ieee80211_unregister_hw even when
ieee80211_register_hw failed.

The function has a for loop where it generates all simulated radios.
when something fails, the error handling will call mac80211_hwsim_free
which frees all simulated radios who's pointer isn't zero. However the
information stored is insufficient to determine whether or not the call
to ieee80211_register_hw succeeded or not for a specific radio. The
included patch makes init_mac80211_hwsim clean up the current simulated
radio, and then calls into mac80211_hwsim_free to clean up all the
radios that did succeed.

This however doesn't explain why the rate control registration failed..
build tested this, but had some problems reproducing the original
problem.

Signed-off-by: Ian Schram <[email protected]>
Signed-off-by: Ingo Molnar <[email protected]>
---
drivers/net/wireless/mac80211_hwsim.c | 18 ++++++++++++------
1 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/drivers/net/wireless/mac80211_hwsim.c b/drivers/net/wireless/mac80211_hwsim.c
index 913dc9f..5816230 100644
--- a/drivers/net/wireless/mac80211_hwsim.c
+++ b/drivers/net/wireless/mac80211_hwsim.c
@@ -364,8 +364,7 @@ static void mac80211_hwsim_free(void)
struct mac80211_hwsim_data *data;
data = hwsim_radios[i]->priv;
ieee80211_unregister_hw(hwsim_radios[i]);
- if (!IS_ERR(data->dev))
- device_unregister(data->dev);
+ device_unregister(data->dev);
ieee80211_free_hw(hwsim_radios[i]);
}
}
@@ -437,7 +436,7 @@ static int __init init_mac80211_hwsim(void)
"mac80211_hwsim: device_create_drvdata "
"failed (%ld)\n", PTR_ERR(data->dev));
err = -ENOMEM;
- goto failed;
+ goto failed_drvdata;
}
data->dev->driver = &mac80211_hwsim_driver;

@@ -461,7 +460,7 @@ static int __init init_mac80211_hwsim(void)
if (err < 0) {
printk(KERN_DEBUG "mac80211_hwsim: "
"ieee80211_register_hw failed (%d)\n", err);
- goto failed;
+ goto failed_hw;
}

printk(KERN_DEBUG "%s: hwaddr %s registered\n",
@@ -479,9 +478,9 @@ static int __init init_mac80211_hwsim(void)
rtnl_lock();

err = dev_alloc_name(hwsim_mon, hwsim_mon->name);
- if (err < 0) {
+ if (err < 0)
goto failed_mon;
- }
+

err = register_netdevice(hwsim_mon);
if (err < 0)
@@ -494,7 +493,14 @@ static int __init init_mac80211_hwsim(void)
failed_mon:
rtnl_unlock();
free_netdev(hwsim_mon);
+ mac80211_hwsim_free();
+ return err;

+failed_hw:
+ device_unregister(data->dev);
+failed_drvdata:
+ ieee80211_free_hw(hw);
+ hwsim_radios[i] = 0;
failed:
mac80211_hwsim_free();
return err;

2008-07-21 19:48:32

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

On Mon, 21 Jul 2008, Larry Finger wrote:
>
> Yes, I'm chasing a distinct bug. The header for mine is
>
> Jul 21 12:19:37 larrylap kernel: kernel BUG at net/core/dev.c:1328!

Ok, that one is fixed now in my tree. Or at least it's turned into a
warning, so the machine should work.

> With an invalid opcode, mine is likely due to stack corruption.

No, invalid opcode is because we use the "ud2" instruction for BUG(),
which causes an invalid op exception. So any BUG[_ON]() will always cause
that on x86.

Linus

2008-07-21 20:11:41

[permalink] [raw]

Subject: Re: [crash] kernel BUG at net/core/dev.c:1328!

From: Stefan Richter <[email protected]>
Date: Mon, 21 Jul 2008 21:20:20 +0200

> In the meantime: Is there perhaps something obviously wrong with
> drivers/ieee1394/eth1394.c's netdevice initialization? We do it in
> ether1394_add_host(), and shortly thereafter the crashing
> ether1394_host_reset() is called. So we have essentially
>
> (add host)
> dev = alloc_netdev(...);
> initialize various members in dev...
> register_netdev(dev);
>
> (host reset)
> netif_stop_queue(dev);
> discard some stale 1394 stuff if there were some...
> netif_wake_queue(dev); <-- crashes in __netif_schedule(dev);

You should only do a netif_stop_queue() in your device
initialization, at the very end of ->open() processing
when you've fully committed to returning success.

You should not, in particular, be doing a netif_wake_queue()
before you've even done a netif_start_queue().

Many of these drivers are using netif_{stop,wake}_queue()
to stop packet flow, in particular when link state changes,
and netif_carrier_{on,off}() already does all of that for
you.

Really, anything outside of:

1) netif_start_queue() in ->open()
2) netif_stop_queue() in ->stop()
3) netif_{stop,wake}_queue() in the TX packet handling path

is superfluous.

2008-07-21 20:16:19

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

From: Linus Torvalds <[email protected]>
Date: Mon, 21 Jul 2008 12:47:58 -0700 (PDT)

> On Mon, 21 Jul 2008, Larry Finger wrote:
> > With an invalid opcode, mine is likely due to stack corruption.
>
> No, invalid opcode is because we use the "ud2" instruction for BUG(),
> which causes an invalid op exception. So any BUG[_ON]() will always cause
> that on x86.

Is there really no more backtrace from that crash message?
It would tell me what driver it's in.

There is some "comm: b43" in the log so I'll check that one.

2008-07-21 20:20:29

[permalink] [raw]

Subject: Re: [crash] kernel BUG at net/core/dev.c:1328!

From: Ingo Molnar <[email protected]>
Date: Mon, 21 Jul 2008 21:44:14 +0200

>
> * David Miller <[email protected]> wrote:
>
> > From: Linus Torvalds <[email protected]>
> > Date: Mon, 21 Jul 2008 11:35:21 -0700 (PDT)
> >
> > > Maybe the network drivers are few enough that it will get fixed, or
> > > maybe the WARN_ON_ONCE() will just be removed and the rule not
> > > reinforced.
> > >
> > > I personally suspect the latter, since it seems to happen with just
> > > about _any_ random network driver, including the common and
> > > well-maintained ones (ie the Gods only help us for the truly
> > > odd/random cases)
> >
> > Yes, we'll see how this plays out.
> >
> > Ian Schram just posted a patch for the NULL pointer derfer in wireless
> > Ingo reported, so we'll see if that bug will be fixed now as well.
>
> Yes, the fix from Ian below solved the CONFIG_MAC80211_HWSIM=y crash i
> was getting. I have no other pending issues other than a few low-prio
> ne2000 build failures.
>
> Thanks guys,

Thanks everyone, applied.

2008-07-21 20:21:20

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

From: Larry Finger <[email protected]>
Date: Mon, 21 Jul 2008 14:43:34 -0500

> Jul 21 12:19:37 larrylap kernel: kernel BUG at net/core/dev.c:1328!
> Jul 21 12:19:37 larrylap kernel: invalid opcode: 0000 [1] SMP
> Jul 21 12:19:37 larrylap kernel: CPU 0
> Jul 21 12:19:37 larrylap kernel: Modules linked in: af_packet rfkill_input nfs
> lockd nfs_acl sunrpc cpufreq_conservative cpu
> freq_userspace cpufreq_powersave powernow_k8 fuse loop dm_mod arc4 ecb
> crypto_blkcipher b43 firmware_class rfkill mac80211 c
> fg80211 snd_hda_intel snd_pcm snd_timer led_class snd k8temp input_polldev
> sr_mod soundcore button battery hwmon cdrom force
> deth ac serio_raw ssb snd_page_alloc sg ehci_hcd sd_mod ohci_hcd usbcore edd fan
> thermal processor ext3 mbcache jbd pata_amd
> ahci libata scsi_mod dock
> Jul 21 12:19:37 larrylap kernel: Pid: 2057, comm: b43 Not tainted
> 2.6.26-Linus-git-05253-g14b395e #1
> Jul 21 12:19:37 larrylap kernel: RIP: 0010:[<ffffffff8039ec4d>]
> [<ffffffff8039ec4d>] __netif_schedule+0x12/0x75
> Jul 21 12:19:37 larrylap kernel: RSP: 0000:ffff8800b9ae1de0 EFLAGS: 00010246
>
> With an invalid opcode, mine is likely due to stack corruption.

No further backtrace? That will tell us what driver is causing
this.

2008-07-21 20:28:43

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

Linus Torvalds wrote:
>
> On Mon, 21 Jul 2008, Larry Finger wrote:
>> Yes, I'm chasing a distinct bug. The header for mine is
>>
>> Jul 21 12:19:37 larrylap kernel: kernel BUG at net/core/dev.c:1328!
>
> Ok, that one is fixed now in my tree. Or at least it's turned into a
> warning, so the machine should work.
>
>> With an invalid opcode, mine is likely due to stack corruption.
>
> No, invalid opcode is because we use the "ud2" instruction for BUG(),
> which causes an invalid op exception. So any BUG[_ON]() will always cause
> that on x86.

Thanks for the explanation.

With your latest tree, I do get the warning. Unfortunately, it still breaks my
wireless and I still need to do the bisection. That is complicated by getting a
kernel that won't build after the first try. I think I now have a workaround.

Larry

2008-07-21 20:33:35

[permalink] [raw]

Subject: Re: [GIT]: Networking

netfilter: make security table depend on NETFILTER_ADVANCED

Signed-off-by: Patrick McHardy <[email protected]>
diff --git a/net/ipv4/netfilter/Kconfig b/net/ipv4/netfilter/Kconfig
index f23e60c..90eb7cb 100644
--- a/net/ipv4/netfilter/Kconfig
+++ b/net/ipv4/netfilter/Kconfig
@@ -369,7 +369,7 @@ config IP_NF_SECURITY
tristate "Security table"
depends on IP_NF_IPTABLES
depends on SECURITY
- default m if NETFILTER_ADVANCED=n
+ depends on NETFILTER_ADVANCED
help
This option adds a `security' table to iptables, for use
with Mandatory Access Control (MAC) policy.
diff --git a/net/ipv6/netfilter/Kconfig b/net/ipv6/netfilter/Kconfig
index 689dec8..0cfcce7 100644
--- a/net/ipv6/netfilter/Kconfig
+++ b/net/ipv6/netfilter/Kconfig
@@ -213,7 +213,7 @@ config IP6_NF_SECURITY
tristate "Security table"
depends on IP6_NF_IPTABLES
depends on SECURITY
- default m if NETFILTER_ADVANCED=n
+ depends on NETFILTER_ADVANCED
help
This option adds a `security' table to iptables, for use
with Mandatory Access Control (MAC) policy.

Attachments:

x (1.07 kB)

2008-07-21 20:39:05

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

David Miller wrote:
>
> No further backtrace? That will tell us what driver is causing
> this.

Yes, I have a full backtrace.

It starts with possible recursive locking in NetworkManager, and goes directly
into the Warning - this came from a later pull of Linus's tree.

Jul 21 15:11:07 larrylap kernel: [ INFO: possible recursive locking detected ]
Jul 21 15:11:07 larrylap kernel: 2.6.26-Linus-git-05614-ge89970a #8
Jul 21 15:11:07 larrylap kernel: ---------------------------------------------
Jul 21 15:11:07 larrylap kernel: NetworkManager/2661 is trying to acquire lock:
Jul 21 15:11:07 larrylap kernel: (&dev->addr_list_lock){-...}, at:
[<ffffffff803a2961>] dev_mc_sync+0x19/0x57
Jul 21 15:11:07 larrylap kernel:
Jul 21 15:11:07 larrylap kernel: but task is already holding lock:
Jul 21 15:11:07 larrylap kernel: (&dev->addr_list_lock){-...}, at:
[<ffffffff8039e7c5>] dev_set_rx_mode+0x19/0x2e
Jul 21 15:11:07 larrylap kernel:
Jul 21 15:11:07 larrylap kernel: other info that might help us debug this:
Jul 21 15:11:07 larrylap kernel: 2 locks held by NetworkManager/2661:
Jul 21 15:11:07 larrylap kernel: #0: (rtnl_mutex){--..}, at:
[<ffffffff803a8318>] rtnetlink_rcv+0x12/0x27
Jul 21 15:11:07 larrylap kernel: #1: (&dev->addr_list_lock){-...}, at:
[<ffffffff8039e7c5>] dev_set_rx_mode+0x19/0x2e
Jul 21 15:11:07 larrylap kernel:
Jul 21 15:11:07 larrylap kernel: stack backtrace:
Jul 21 15:11:07 larrylap kernel: Pid: 2661, comm: NetworkManager Not tainted
2.6.26-Linus-git-05614-ge89970a #8
Jul 21 15:11:07 larrylap kernel:
Jul 21 15:11:07 larrylap kernel: Call Trace:
Jul 21 15:11:07 larrylap kernel: [<ffffffff80251b02>] __lock_acquire+0xb7b/0xecc
Jul 21 15:11:07 larrylap kernel: [<ffffffff80251ea4>] lock_acquire+0x51/0x6a
Jul 21 15:11:07 larrylap kernel: [<ffffffff803a2961>] dev_mc_sync+0x19/0x57
Jul 21 15:11:07 larrylap kernel: [<ffffffff80408f9c>] _spin_lock_bh+0x23/0x2c
Jul 21 15:11:07 larrylap kernel: [<ffffffff803a2961>] dev_mc_sync+0x19/0x57
Jul 21 15:11:07 larrylap kernel: [<ffffffff8039e7cd>] dev_set_rx_mode+0x21/0x2e
Jul 21 15:11:07 larrylap kernel: [<ffffffff803a036c>] dev_open+0x8e/0xb0
Jul 21 15:11:07 larrylap kernel: [<ffffffff8039fd13>] dev_change_flags+0xa6/0x164
Jul 21 15:11:07 larrylap kernel: [<ffffffff803a7421>] do_setlink+0x286/0x349
Jul 21 15:11:07 larrylap kernel: [<ffffffff803a832d>] rtnetlink_rcv_msg+0x0/0x1ec
Jul 21 15:11:07 larrylap syslog-ng[2488]: last message repeated 2 times
Jul 21 15:11:07 larrylap kernel: [<ffffffff803a766e>] rtnl_setlink+0x10b/0x10d
Jul 21 15:11:07 larrylap kernel: [<ffffffff803a832d>] rtnetlink_rcv_msg+0x0/0x1ec
Jul 21 15:11:07 larrylap kernel: [<ffffffff803b1bcf>] netlink_rcv_skb+0x34/0x7d
Jul 21 15:11:07 larrylap kernel: [<ffffffff803a8327>] rtnetlink_rcv+0x21/0x27
Jul 21 15:11:07 larrylap kernel: [<ffffffff803b16cf>] netlink_unicast+0x1f0/0x261
Jul 21 15:11:07 larrylap kernel: [<ffffffff8039a448>] __alloc_skb+0x66/0x12a
Jul 21 15:11:07 larrylap kernel: [<ffffffff803b19a8>] netlink_sendmsg+0x268/0x27b
Jul 21 15:11:07 larrylap rpc.idmapd[2783]: main:
fcntl(/var/lib/nfs/rpc_pipefs/nfs): Invalid argument
Jul 21 15:11:07 larrylap kernel: [<ffffffff80393b8d>] sock_sendmsg+0xcb/0xe3
Jul 21 15:11:07 larrylap kernel: [<ffffffff80246aab>]
autoremove_wake_function+0x0/0x2e
Jul 21 15:11:07 larrylap kernel: [<ffffffff8039b194>] verify_iovec+0x46/0x82
Jul 21 15:11:07 larrylap kernel: [<ffffffff80393dbc>] sys_sendmsg+0x217/0x28a
Jul 21 15:11:07 larrylap kernel: [<ffffffff80393516>] sockfd_lookup_light+0x1a/0x52
Jul 21 15:11:07 larrylap kernel: [<ffffffff80250990>]
trace_hardirqs_on_caller+0xef/0x113
Jul 21 15:11:07 larrylap kernel: [<ffffffff80408b14>]
trace_hardirqs_on_thunk+0x3a/0x3f
Jul 21 15:11:07 larrylap kernel: [<ffffffff8020be9b>]
system_call_after_swapgs+0x7b/0x80
Jul 21 15:11:07 larrylap kernel:
Jul 21 15:11:07 larrylap kernel: NET: Registered protocol family 17
Jul 21 15:11:08 larrylap kernel: ------------[ cut here ]------------
Jul 21 15:11:08 larrylap kernel: WARNING: at net/core/dev.c:1330
__netif_schedule+0x2c/0x98()
Jul 21 15:11:08 larrylap kernel: Modules linked in: snd_seq_device af_packet nfs
lockd nfs_acl sunrpc rfkill_input cpufreq_conservative cpufreq_userspace
cpufreq_powersave powernow_k8 fuse loop dm_mod arc4 ecb crypto_blkcipher b43
firmware_class rfkill snd_hda_intel mac80211 cfg80211 snd_pcm snd_timer
led_class snd soundcore input_polldev ac k8temp button snd_page_alloc battery
sr_mod forcedeth cdrom serio_raw hwmon ssb sg ehci_hcd sd_mod ohci_hcd usbcore
edd fan thermal processor ext3 mbcache jbd pata_amd ahci libata scsi_mod dock
Jul 21 15:11:08 larrylap kernel: Pid: 2035, comm: b43 Not tainted
2.6.26-Linus-git-05614-ge89970a #8
Jul 21 15:11:08 larrylap kernel:
Jul 21 15:11:08 larrylap kernel: Call Trace:
Jul 21 15:11:08 larrylap kernel: [<ffffffff80233f6d>] warn_on_slowpath+0x51/0x8c
Jul 21 15:11:08 larrylap kernel: [<ffffffff8039d7f3>] __netif_schedule+0x2c/0x98
Jul 21 15:11:08 larrylap kernel: [<ffffffffa018b44d>]
ieee80211_scan_completed+0x25b/0x2e1 [mac80211]
Jul 21 15:11:08 larrylap kernel: [<ffffffffa018b6ce>]
ieee80211_sta_scan_work+0x0/0x1b8 [mac80211]
Jul 21 15:11:08 larrylap kernel: [<ffffffff8024325e>] run_workqueue+0xf0/0x1f2
Jul 21 15:11:08 larrylap kernel: [<ffffffff8024343b>] worker_thread+0xdb/0xea
Jul 21 15:11:08 larrylap kernel: [<ffffffff80246aab>]
autoremove_wake_function+0x0/0x2e
Jul 21 15:11:08 larrylap avahi-daemon[2877]: Found user 'avahi' (UID 102) and
group 'avahi' (GID 104).
Jul 21 15:11:09 larrylap kernel: [<ffffffff80243360>] worker_thread+0x0/0xea
Jul 21 15:11:09 larrylap kernel: [<ffffffff8024678b>] kthread+0x47/0x73
Jul 21 15:11:09 larrylap avahi-daemon[2877]: Successfully dropped root privileges.
Jul 21 15:11:09 larrylap kernel: [<ffffffff80408b14>]
trace_hardirqs_on_thunk+0x3a/0x3f
Jul 21 15:11:09 larrylap avahi-daemon[2877]: avahi-daemon 0.6.22 starting up.
Jul 21 15:11:09 larrylap kernel: [<ffffffff8020cea9>] child_rip+0xa/0x11
Jul 21 15:11:09 larrylap kernel: [<ffffffff8020c4df>] restore_args+0x0/0x30
Jul 21 15:11:09 larrylap kernel: [<ffffffff8024671f>] kthreadd+0x188/0x1ad
Jul 21 15:11:09 larrylap kernel: [<ffffffff80246744>] kthread+0x0/0x73
Jul 21 15:11:09 larrylap kernel: [<ffffffff8020ce9f>] child_rip+0x0/0x11
Jul 21 15:11:09 larrylap kernel:
Jul 21 15:11:09 larrylap kernel: ---[ end trace 030d0589d3c6c7f5 ]---

Larry

2008-07-21 20:46:24

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

From: Larry Finger <[email protected]>
Date: Mon, 21 Jul 2008 15:38:47 -0500

> David Miller wrote:
> >
> > No further backtrace? That will tell us what driver is causing
> > this.
>
> Yes, I have a full backtrace.
>
> It starts with possible recursive locking in NetworkManager, and goes directly
> into the Warning - this came from a later pull of Linus's tree.

That helps a lot, I'm looking at this now.

Thanks.

2008-07-21 20:52:15

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

David Miller wrote:
> From: Larry Finger <[email protected]>
> Date: Mon, 21 Jul 2008 15:38:47 -0500
>
>> David Miller wrote:
>>> No further backtrace? That will tell us what driver is causing
>>> this.
>> Yes, I have a full backtrace.
>>
>> It starts with possible recursive locking in NetworkManager, and goes directly
>> into the Warning - this came from a later pull of Linus's tree.
>
> That helps a lot, I'm looking at this now.

I'm guessing this needs similar lockdep class initializations
to _xmit_lock since it essentially has the same nesting rules.

2008-07-21 21:01:34

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

From: Patrick McHardy <[email protected]>
Date: Mon, 21 Jul 2008 22:51:53 +0200

> David Miller wrote:
> > From: Larry Finger <[email protected]>
> > Date: Mon, 21 Jul 2008 15:38:47 -0500
> >
> >> David Miller wrote:
> >>> No further backtrace? That will tell us what driver is causing
> >>> this.
> >> Yes, I have a full backtrace.
> >>
> >> It starts with possible recursive locking in NetworkManager, and goes directly
> >> into the Warning - this came from a later pull of Linus's tree.
> >
> > That helps a lot, I'm looking at this now.
>
> I'm guessing this needs similar lockdep class initializations
> to _xmit_lock since it essentially has the same nesting rules.

Yes, I figured that out just now :-)

Maybe something like the following should do it?

diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index 9737c06..a641eea 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -5041,6 +5041,7 @@ static int bond_check_params(struct bond_params *params)
}

static struct lock_class_key bonding_netdev_xmit_lock_key;
+static struct lock_class_key bonding_netdev_addr_lock_key;

static void bond_set_lockdep_class_one(struct net_device *dev,
struct netdev_queue *txq,
@@ -5052,6 +5053,8 @@ static void bond_set_lockdep_class_one(struct net_device *dev,

static void bond_set_lockdep_class(struct net_device *dev)
{
+ lockdep_set_class(&dev->addr_list_lock,
+ &bonding_netdev_addr_lock_key);
netdev_for_each_tx_queue(dev, bond_set_lockdep_class_one, NULL);
}

diff --git a/drivers/net/hamradio/bpqether.c b/drivers/net/hamradio/bpqether.c
index b6500b2..58f4b1d 100644
--- a/drivers/net/hamradio/bpqether.c
+++ b/drivers/net/hamradio/bpqether.c
@@ -123,6 +123,7 @@ static LIST_HEAD(bpq_devices);
* off into a separate class since they always nest.
*/
static struct lock_class_key bpq_netdev_xmit_lock_key;
+static struct lock_class_key bpq_netdev_addr_lock_key;

static void bpq_set_lockdep_class_one(struct net_device *dev,
struct netdev_queue *txq,
@@ -133,6 +134,7 @@ static void bpq_set_lockdep_class_one(struct net_device *dev,

static void bpq_set_lockdep_class(struct net_device *dev)
{
+ lockdep_set_class(&dev->addr_list_lock, &bpq_netdev_addr_lock_key);
netdev_for_each_tx_queue(dev, bpq_set_lockdep_class_one, NULL);
}

diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
index efbc155..4239450 100644
--- a/drivers/net/macvlan.c
+++ b/drivers/net/macvlan.c
@@ -276,6 +276,7 @@ static int macvlan_change_mtu(struct net_device *dev, int new_mtu)
* separate class since they always nest.
*/
static struct lock_class_key macvlan_netdev_xmit_lock_key;
+static struct lock_class_key macvlan_netdev_addr_lock_key;

#define MACVLAN_FEATURES \
(NETIF_F_SG | NETIF_F_ALL_CSUM | NETIF_F_HIGHDMA | NETIF_F_FRAGLIST | \
@@ -295,6 +296,8 @@ static void macvlan_set_lockdep_class_one(struct net_device *dev,

static void macvlan_set_lockdep_class(struct net_device *dev)
{
+ lockdep_set_class(&dev->addr_list_lock,
+ &macvlan_netdev_addr_lock_key);
netdev_for_each_tx_queue(dev, macvlan_set_lockdep_class_one, NULL);
}

diff --git a/drivers/net/wireless/hostap/hostap_hw.c b/drivers/net/wireless/hostap/hostap_hw.c
index 13d5882..3153fe9 100644
--- a/drivers/net/wireless/hostap/hostap_hw.c
+++ b/drivers/net/wireless/hostap/hostap_hw.c
@@ -3101,6 +3101,7 @@ static void prism2_clear_set_tim_queue(local_info_t *local)
* This is a natural nesting, which needs a split lock type.
*/
static struct lock_class_key hostap_netdev_xmit_lock_key;
+static struct lock_class_key hostap_netdev_addr_lock_key;

static void prism2_set_lockdep_class_one(struct net_device *dev,
struct netdev_queue *txq,
@@ -3112,6 +3113,8 @@ static void prism2_set_lockdep_class_one(struct net_device *dev,

static void prism2_set_lockdep_class(struct net_device *dev)
{
+ lockdep_set_class(&dev->addr_list_lock,
+ &hostap_netdev_addr_lock_key);
netdev_for_each_tx_queue(dev, prism2_set_lockdep_class_one, NULL);
}

diff --git a/net/8021q/vlan_dev.c b/net/8021q/vlan_dev.c
index f42bc2b..4bf014e 100644
--- a/net/8021q/vlan_dev.c
+++ b/net/8021q/vlan_dev.c
@@ -569,6 +569,7 @@ static void vlan_dev_set_rx_mode(struct net_device *vlan_dev)
* separate class since they always nest.
*/
static struct lock_class_key vlan_netdev_xmit_lock_key;
+static struct lock_class_key vlan_netdev_addr_lock_key;

static void vlan_dev_set_lockdep_one(struct net_device *dev,
struct netdev_queue *txq,
@@ -581,6 +582,9 @@ static void vlan_dev_set_lockdep_one(struct net_device *dev,

static void vlan_dev_set_lockdep_class(struct net_device *dev, int subclass)
{
+ lockdep_set_class_and_subclass(&dev->addr_list_lock,
+ &vlan_netdev_addr_lock_key,
+ subclass);
netdev_for_each_tx_queue(dev, vlan_dev_set_lockdep_one, &subclass);
}

diff --git a/net/netrom/af_netrom.c b/net/netrom/af_netrom.c
index fccc250..532e4fa 100644
--- a/net/netrom/af_netrom.c
+++ b/net/netrom/af_netrom.c
@@ -73,6 +73,7 @@ static const struct proto_ops nr_proto_ops;
* separate class since they always nest.
*/
static struct lock_class_key nr_netdev_xmit_lock_key;
+static struct lock_class_key nr_netdev_addr_lock_key;

static void nr_set_lockdep_one(struct net_device *dev,
struct netdev_queue *txq,
@@ -83,6 +84,7 @@ static void nr_set_lockdep_one(struct net_device *dev,

static void nr_set_lockdep_key(struct net_device *dev)
{
+ lockdep_set_class(&dev->addr_list_lock, &nr_netdev_addr_lock_key);
netdev_for_each_tx_queue(dev, nr_set_lockdep_one, NULL);
}

diff --git a/net/rose/af_rose.c b/net/rose/af_rose.c
index dbc963b..a7f1ce1 100644
--- a/net/rose/af_rose.c
+++ b/net/rose/af_rose.c
@@ -74,6 +74,7 @@ ax25_address rose_callsign;
* separate class since they always nest.
*/
static struct lock_class_key rose_netdev_xmit_lock_key;
+static struct lock_class_key rose_netdev_addr_lock_key;

static void rose_set_lockdep_one(struct net_device *dev,
struct netdev_queue *txq,
@@ -84,6 +85,7 @@ static void rose_set_lockdep_one(struct net_device *dev,

static void rose_set_lockdep_key(struct net_device *dev)
{
+ lockdep_set_class(&dev->addr_list_lock, &rose_netdev_addr_lock_key);
netdev_for_each_tx_queue(dev, rose_set_lockdep_one, NULL);
}

2008-07-21 21:06:23

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

David Miller wrote:
> From: Patrick McHardy <[email protected]>
> Date: Mon, 21 Jul 2008 22:51:53 +0200
>
>> David Miller wrote:
>>> From: Larry Finger <[email protected]>
>>> Date: Mon, 21 Jul 2008 15:38:47 -0500
>>>
>>>> David Miller wrote:
>>>>> No further backtrace? That will tell us what driver is causing
>>>>> this.
>>>> Yes, I have a full backtrace.
>>>>
>>>> It starts with possible recursive locking in NetworkManager, and goes directly
>>>> into the Warning - this came from a later pull of Linus's tree.
>>> That helps a lot, I'm looking at this now.
>> I'm guessing this needs similar lockdep class initializations
>> to _xmit_lock since it essentially has the same nesting rules.
>
> Yes, I figured that out just now :-)
>
> Maybe something like the following should do it?

It looks correct in any case. I'm not sure whether it fixes
this lockdep warning though, according to the backtrace and
module list its b43 and dev_mc_sync in net/mac80211/main.c
that are causing the error, which don't seem to be included
in your patch. I'm unable to find where it previously
initialized the xmit_lock lockdep class though, so I must
be missing something :)

2008-07-21 21:27:32

[permalink] [raw]

Subject: Re: [crash] kernel BUG at net/core/dev.c:1328!

David Miller wrote:
> You should only do a netif_stop_queue() in your device
> initialization, at the very end of ->open() processing
> when you've fully committed to returning success.
>
> You should not, in particular, be doing a netif_wake_queue()
> before you've even done a netif_start_queue().
>
> Many of these drivers are using netif_{stop,wake}_queue()
> to stop packet flow, in particular when link state changes,
> and netif_carrier_{on,off}() already does all of that for
> you.
>
> Really, anything outside of:
>
> 1) netif_start_queue() in ->open()
> 2) netif_stop_queue() in ->stop()
> 3) netif_{stop,wake}_queue() in the TX packet handling path
>
> is superfluous.

Thanks, I wasn't aware of this. I'll try to fix up eth1394 this way as
time permits.
--
Stefan Richter
-=====-==--- -=== =-=-=
http://arcgraph.de/sr/

2008-07-21 21:35:48

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

net: set lockdep class for dev->addr_list_lock

Initialize dev->addr_list_lock lockdep classes equally to dev->_xmit_lock.

Signed-off-by: Patrick McHardy <[email protected]>

diff --git a/net/core/dev.c b/net/core/dev.c
index 2eed17b..9cfed90 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -299,6 +299,7 @@ static const char *netdev_lock_name[] =
"_xmit_NONE"};

static struct lock_class_key netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)];
+static struct lock_class_key netdev_addr_lock_key[ARRAY_SIZE(netdev_lock_type)];

static inline unsigned short netdev_lock_pos(unsigned short dev_type)
{
@@ -311,8 +312,8 @@ static inline unsigned short netdev_lock_pos(unsigned short dev_type)
return ARRAY_SIZE(netdev_lock_type) - 1;
}

-static inline void netdev_set_lockdep_class(spinlock_t *lock,
- unsigned short dev_type)
+static inline void netdev_set_xmit_lockdep_class(spinlock_t *lock,
+ unsigned short dev_type)
{
int i;

@@ -320,9 +321,23 @@ static inline void netdev_set_lockdep_class(spinlock_t *lock,
lockdep_set_class_and_name(lock, &netdev_xmit_lock_key[i],
netdev_lock_name[i]);
}
+
+static inline void netdev_set_addr_lockdep_class(spinlock_t *lock,
+ unsigned short dev_type)
+{
+ int i;
+
+ i = netdev_lock_pos(dev_type);
+ lockdep_set_class_and_name(lock, &netdev_addr_lock_key[i],
+ netdev_lock_name[i]);
+}
#else
-static inline void netdev_set_lockdep_class(spinlock_t *lock,
- unsigned short dev_type)
+static inline void netdev_set_xmit_lockdep_class(spinlock_t *lock,
+ unsigned short dev_type)
+{
+}
+static inline void netdev_set_addr_lockdep_class(spinlock_t *lock,
+ unsigned short dev_type)
{
}
#endif
@@ -3843,14 +3858,15 @@ static void __netdev_init_queue_locks_one(struct net_device *dev,
void *_unused)
{
spin_lock_init(&dev_queue->_xmit_lock);
- netdev_set_lockdep_class(&dev_queue->_xmit_lock, dev->type);
+ netdev_set_xmit_lockdep_class(&dev_queue->_xmit_lock, dev->type);
dev_queue->xmit_lock_owner = -1;
}

-static void netdev_init_queue_locks(struct net_device *dev)
+static void netdev_init_locks(struct net_device *dev)
{
netdev_for_each_tx_queue(dev, __netdev_init_queue_locks_one, NULL);
__netdev_init_queue_locks_one(dev, &dev->rx_queue, NULL);
+ netdev_set_addr_lockdep_class(&dev->addr_list_lock, dev->type);
}

/**
@@ -3888,7 +3904,7 @@ int register_netdevice(struct net_device *dev)
net = dev_net(dev);

spin_lock_init(&dev->addr_list_lock);
- netdev_init_queue_locks(dev);
+ netdev_init_locks(dev);

dev->iflink = -1;

Attachments:

x (2.52 kB)

2008-07-21 21:43:19

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

net: set lockdep class for dev->addr_list_lock

Initialize dev->addr_list_lock lockdep classes equally to dev->_xmit_lock.

Signed-off-by: Patrick McHardy <[email protected]>

diff --git a/net/core/dev.c b/net/core/dev.c
index 2eed17b..6f8b6c5 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -299,6 +299,7 @@ static const char *netdev_lock_name[] =
"_xmit_NONE"};

static struct lock_class_key netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)];
+static struct lock_class_key netdev_addr_lock_key[ARRAY_SIZE(netdev_lock_type)];

static inline unsigned short netdev_lock_pos(unsigned short dev_type)
{
@@ -311,8 +312,8 @@ static inline unsigned short netdev_lock_pos(unsigned short dev_type)
return ARRAY_SIZE(netdev_lock_type) - 1;
}

-static inline void netdev_set_lockdep_class(spinlock_t *lock,
- unsigned short dev_type)
+static inline void netdev_set_xmit_lockdep_class(spinlock_t *lock,
+ unsigned short dev_type)
{
int i;

@@ -320,9 +321,22 @@ static inline void netdev_set_lockdep_class(spinlock_t *lock,
lockdep_set_class_and_name(lock, &netdev_xmit_lock_key[i],
netdev_lock_name[i]);
}
+
+static inline void netdev_set_addr_lockdep_class(struct net_device *dev)
+{
+ int i;
+
+ i = netdev_lock_pos(dev->type);
+ lockdep_set_class_and_name(&dev->addr_list_lock,
+ &netdev_addr_lock_key[i],
+ netdev_lock_name[i]);
+}
#else
-static inline void netdev_set_lockdep_class(spinlock_t *lock,
- unsigned short dev_type)
+static inline void netdev_set_xmit_lockdep_class(spinlock_t *lock,
+ unsigned short dev_type)
+{
+}
+static inline void netdev_set_addr_lockdep_class(struct net_device *dev)
{
}
#endif
@@ -3843,7 +3857,7 @@ static void __netdev_init_queue_locks_one(struct net_device *dev,
void *_unused)
{
spin_lock_init(&dev_queue->_xmit_lock);
- netdev_set_lockdep_class(&dev_queue->_xmit_lock, dev->type);
+ netdev_set_xmit_lockdep_class(&dev_queue->_xmit_lock, dev->type);
dev_queue->xmit_lock_owner = -1;
}

@@ -3888,6 +3902,7 @@ int register_netdevice(struct net_device *dev)
net = dev_net(dev);

spin_lock_init(&dev->addr_list_lock);
+ netdev_set_addr_lockdep_class(dev);
netdev_init_queue_locks(dev);

dev->iflink = -1;

Attachments:

x (2.18 kB)

2008-07-21 21:51:39

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

Patrick McHardy wrote:
> Patrick McHardy wrote:
>> David Miller wrote:
>>> Maybe something like the following should do it?
>>
>>
>> It looks correct in any case. I'm not sure whether it fixes
>> this lockdep warning though, according to the backtrace and
>> module list its b43 and dev_mc_sync in net/mac80211/main.c
>> that are causing the error, which don't seem to be included
>> in your patch. I'm unable to find where it previously
>> initialized the xmit_lock lockdep class though, so I must
>> be missing something :)
>
> This is what I was missing, we're setting a lockdep class
> by default depending on dev->type. This patch combined
> with yours should fix all addr_list_lock warnings.

No cigar yet. I tried davem's patch first, then yours on top of his. I still get
both the recursive locking and the kernel warning.

BTW, wireless doesn't work but if I plug in the wire, then networking is OK. It
seems to be in mac80211, which is strange because I routinely run the latest
wireless-testing kernel, and all the wireless bits should be there already.

I'm still plugging away at the bisection. I think I got away from the kernel
that won't build.

Larry

2008-07-21 22:05:37

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

diff --git a/net/core/dev.c b/net/core/dev.c
index 2eed17b..371b1a0 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -259,7 +259,7 @@ static RAW_NOTIFIER_HEAD(netdev_chain);

DEFINE_PER_CPU(struct softnet_data, softnet_data);

-#ifdef CONFIG_DEBUG_LOCK_ALLOC
+#ifdef CONFIG_LOCKDEP
/*
* register_netdevice() inits txq->_xmit_lock and sets lockdep class
* according to dev->type

Attachments:

x (389.00 B)

2008-07-21 22:40:32

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

Patrick McHardy wrote:
> Larry Finger wrote:
>> Patrick McHardy wrote:
>>>
>>> This is what I was missing, we're setting a lockdep class
>>> by default depending on dev->type. This patch combined
>>> with yours should fix all addr_list_lock warnings.
>>
>> No cigar yet. I tried davem's patch first, then yours on top of his. I
>> still get both the recursive locking and the kernel warning.
>
> Does this one earn me my cigar? :)

Sorry :(

I used the davem patch, the second version of your first one, and your second
one. Both problems persist.

Still plugging away on bisection.

Larry

2008-07-21 23:15:31

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

From: Larry Finger <[email protected]>
Date: Mon, 21 Jul 2008 17:40:10 -0500

> Sorry :(
>
> I used the davem patch, the second version of your first one, and your second
> one. Both problems persist.
>
> Still plugging away on bisection.

GIT bisecting the lockdep problem is surely going the land you on:

commit e308a5d806c852f56590ffdd3834d0df0cbed8d7
Author: David S. Miller <[email protected]>
Date: Tue Jul 15 00:13:44 2008 -0700

netdev: Add netdev->addr_list_lock protection.

Add netif_addr_{lock,unlock}{,_bh}() helpers.

Use them to protect operations that operate on or read
the network device unicast and multicast address lists.

Also use them in cases where the code simply wants to
block calls into the driver's ->set_rx_mode() and
->set_multicast_list() methods.

Signed-off-by: David S. Miller <[email protected]>

2008-07-22 06:34:43

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

David Miller wrote:
> From: Larry Finger <[email protected]>
> Date: Mon, 21 Jul 2008 17:40:10 -0500
>
>> Sorry :(
>>
>> I used the davem patch, the second version of your first one, and your second
>> one. Both problems persist.
>>
>> Still plugging away on bisection.
>
> GIT bisecting the lockdep problem is surely going the land you on:
>
> commit e308a5d806c852f56590ffdd3834d0df0cbed8d7

No. It landed on this one.

37437bb2e1ae8af470dfcd5b4ff454110894ccaf is first bad commit
commit 37437bb2e1ae8af470dfcd5b4ff454110894ccaf
Author: David S. Miller <[email protected]>
Date: Wed Jul 16 02:15:04 2008 -0700

pkt_sched: Schedule qdiscs instead of netdev_queue.

When we have shared qdiscs, packets come out of the qdiscs
for multiple transmit queues.

Therefore it doesn't make any sense to schedule the transmit
queue when logically we cannot know ahead of time the TX
queue of the SKB that the qdisc->dequeue() will give us.

Just for sanity I added a BUG check to make sure we never
get into a state where the noop_qdisc is scheduled.

Signed-off-by: David S. Miller <[email protected]>

:040000 040000 4d13d1fb1ae37d9720c3db6b1368866e78621f55
f1a0f5e5a191e7904b528d9e10069a4324a5d328 M include
:040000 040000 3515aad52a2cdaaba85feeffc0944d7f07a19c96
4854d4f4df9726a2e8837037f82bde807bed2ede M net

Larry

2008-07-22 10:47:12

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

On 22-07-2008 08:34, Larry Finger wrote:
...
>>> I used the davem patch, the second version of your first one, and
>>> your second one. Both problems persist.

Could you send lockdep info after Patrick's "set lockdep classes"
patch?

Jarek P.

2008-07-22 11:22:07

[permalink] [raw]

Subject: [TCP bug] stuck distcc connections in latest -git

* Ingo Molnar <[email protected]> wrote:

> ok, have updated the testboxes to your latest push.
>
> Btw., otherwise the big networking pull held up pretty well on a
> healthy range of testboxes i have, [...]

hm, the distcc TCP hangs are back:

Distcc client box (quad, 10.0.1.16) running v2.6.24:

dione:~> netstat -nt | grep -vw TIME_WAIT | grep 3632
tcp 0 250455 10.0.1.16:55559 10.0.1.19:3632 ESTABLISHED
tcp 0 254743 10.0.1.16:56096 10.0.1.19:3632 ESTABLISHED
tcp 0 219617 10.0.1.16:55674 10.0.1.19:3632 ESTABLISHED

[ ^--- note the stuck send-queue ]

Distcc server box (16-way, 10.0.1.19) running very-latest:

phoenix:~> netstat -nt | grep 10.0.1.16 | grep 3632

tcp 0 0 10.0.1.19:3632 10.0.1.16:55559 ESTABLISHED
tcp 0 0 10.0.1.19:3632 10.0.1.16:56096 ESTABLISHED
tcp 0 0 10.0.1.19:3632 10.0.1.16:55674 ESTABLISHED

tcp 0 0 10.0.1.19:3632 10.0.1.16:34411 ESTABLISHED
tcp 0 0 10.0.1.19:3632 10.0.1.16:51094 ESTABLISHED
tcp 0 0 10.0.1.19:3632 10.0.1.16:60787 ESTABLISHED
tcp 0 0 10.0.1.19:3632 10.0.1.16:50874 ESTABLISHED

I.e. the client side send-queue is stuck in established state, server
side thinks it's a proper established connection. Nobody makes any
progress.

Also note the final 4 connections on the server side - those are not
present on the client box.

The hung condition seemed permanent (i waited a couple of minutes).

Then i shut down the distccd on the server side, which propagated to the
client:

distcc[18496] (dcc_pump_sendfile) ERROR: sendfile failed: Broken pipe
distcc[18496] (dcc_readx) ERROR: unexpected eof on fd4
distcc[18496] (dcc_r_token_int) ERROR: read failed while waiting for token "DONE"
distcc[18496] Warning: failed to distribute kernel/futex.c to ph/20, running locally instead

Server side lingered in FIN_WAIT2 a bit:

Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 10.0.1.19:3632 10.0.1.16:56096 FIN_WAIT2
tcp 0 0 10.0.1.19:3632 10.0.1.16:55559 FIN_WAIT2

I retried the same build 10 times and it would not reproduce - so this
again is a hard to reproduce condition. (and there's no chance to get a
proper tcpdump either, at these traffic levels)

Ingo

2008-07-22 11:32:34

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

From: Larry Finger <[email protected]>
Date: Tue, 22 Jul 2008 01:34:28 -0500

> David Miller wrote:
> > From: Larry Finger <[email protected]>
> > Date: Mon, 21 Jul 2008 17:40:10 -0500
> >
> >> Sorry :(
> >>
> >> I used the davem patch, the second version of your first one, and your second
> >> one. Both problems persist.
> >>
> >> Still plugging away on bisection.
> >
> > GIT bisecting the lockdep problem is surely going the land you on:
> >
> > commit e308a5d806c852f56590ffdd3834d0df0cbed8d7
>
> No. It landed on this one.

For the lockdep warnings?

2008-07-22 12:52:39

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

David Miller wrote:
> From: Larry Finger <[email protected]>
> Date: Tue, 22 Jul 2008 01:34:28 -0500
>
>> David Miller wrote:
>>> From: Larry Finger <[email protected]>
>>> Date: Mon, 21 Jul 2008 17:40:10 -0500
>>>
>>>> Sorry :(
>>>>
>>>> I used the davem patch, the second version of your first one, and your second
>>>> one. Both problems persist.
>>>>
>>>> Still plugging away on bisection.
>>> GIT bisecting the lockdep problem is surely going the land you on:
>>>
>>> commit e308a5d806c852f56590ffdd3834d0df0cbed8d7
>> No. It landed on this one.
>
> For the lockdep warnings?

No - this one triggers the kernel BUG as follows:

------------[ cut here ]------------
kernel BUG at net/core/dev.c:1328!
invalid opcode: 0000 [1] SMP
CPU 0
Modules linked in: af_packet rfkill_input nfs lockd nfs_acl sunrpc
cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop
dm_mod arc4 ecb crypto_blkcipher b43 firmware_class rfkill mac80211 cfg80211
led_class input_polldev k8temp sr_mod battery ac ssb button hwmon forcedeth
cdrom serio_raw sg ohci_hcd ehci_hcd sd_mod usbcore edd fan thermal processor
ext3 mbcache jbd pata_amd ahci libata scsi_mod dock
Pid: 2003, comm: b43 Not tainted 2.6.26-rc8-Linus-git-01424-g37437bb #43
RIP: 0010:[<ffffffff803958c6>] [<ffffffff803958c6>] __netif_schedule+0x12/0x75
RSP: 0018:ffff8100b9e33de0 EFLAGS: 00010246
RAX: ffff8100b63819c0 RBX: ffffffff80545300 RCX: ffff8100b6381980
RDX: 00000000ffffffff RSI: 0000000000000001 RDI: ffffffff80545300
RBP: ffff8100b7b45158 R08: ffff8100b89d8000 R09: ffff8100b9d26000
R10: ffff8100b7b44480 R11: ffffffffa01239ef R12: ffff8100b7b44480
R13: ffff8100b9d26000 R14: ffff8100b89d8000 R15: 0000000000000000
FS: 00007f494406a6f0(0000) GS:ffffffff8055e000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007f49440933dc CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process b43 (pid: 2003, threadinfo ffff8100b9e32000, task ffff8100b4a3e480)
Stack: ffff8100b7b45158 ffff8100b89d8900 ffff8100b7b45158 ffffffffa0158455
ffff8100ba3287c0 0000000000000246 0000000000000000 0000000000000000
ffff8100b9e33e70 ffff8100b7b451b8 ffff8100ba3287c0 ffff8100b7b451b0
Call Trace:
[<ffffffffa0158455>] ? :mac80211:ieee80211_scan_completed+0x25b/0x2e1
[<ffffffffa01586d6>] ? :mac80211:ieee80211_sta_scan_work+0x0/0x1b8
[<ffffffff8023f7d7>] ? run_workqueue+0xf1/0x1f3
[<ffffffff8023f9b4>] ? worker_thread+0xdb/0xea
[<ffffffff80243017>] ? autoremove_wake_function+0x0/0x2e
[<ffffffff8023f8d9>] ? worker_thread+0x0/0xea
[<ffffffff80242cff>] ? kthread+0x47/0x73
[<ffffffff80402845>] ? trace_hardirqs_on_thunk+0x35/0x3a
[<ffffffff8020cd48>] ? child_rip+0xa/0x12
[<ffffffff8020c45f>] ? restore_args+0x0/0x30
[<ffffffff8021d3b6>] ? flat_send_IPI_mask+0x0/0x67
[<ffffffff80242c93>] ? kthreadd+0x188/0x1ad
[<ffffffff80242c93>] ? kthreadd+0x188/0x1ad
[<ffffffff80242cb8>] ? kthread+0x0/0x73
[<ffffffff8020cd3e>] ? child_rip+0x0/0x12

Code: 00 00 75 0a 55 9d 5e 5b 5d e9 32 64 eb ff e8 21 73 eb ff 55 9d 59 5b 5d c3
55 53 48 89 fb 48 83 ec 08 48 81 ff 00 53 54 80 75 04 <0f> 0b eb fe 48 8d 47 30
f0 0f ba 28 01 19 d2 85 d2 75 4c 9c 5d
RIP [<ffffffff803958c6>] __netif_schedule+0x12/0x75
RSP <ffff8100b9e33de0>
---[ end trace 396dc6bdf73da468 ]---

I'll have to trace back to see which of the bisections produced both the lockdep
and the kernel bug.

Larry

2008-07-22 13:02:25

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

2008-07-22 13:46:00

by David Newall

[permalink] [raw]

Subject: Re: [TCP bug] stuck distcc connections in latest -git

Ingo Molnar wrote:
> hm, the distcc TCP hangs are back:
>

The missing four client-side connections are more interesting than the
unsent data.

> I.e. the client side send-queue is stuck in established state, server
> side thinks it's a proper established connection. Nobody makes any
> progress.
>

I might be missing something obvious, but I don't think there's anything
unusual in the three sessions displayed on the client. They should be
"ESTABLISHED", and on the server, too, just as they are.

> Also note the final 4 connections on the server side - those are not
> present on the client box.
>

Now this is interesting. I would be much more interested in how the
client's sides for these disappeared.

> The hung condition seemed permanent (i waited a couple of minutes).
>

Not nearly long enough. Retransmits can be sent as infrequently as per
180 seconds. I think there's an argument to use one of the the various
patches that reduce your TCP_RTO_MAX, for example OBATA Noboru's
(http://marc.info/?l=linux-netdev&m=118422471428855): you don't have to
wait unreasonably long before seeing a retransmit. Remember, three minutes!

> I retried the same build 10 times and it would not reproduce - so this
> again is a hard to reproduce condition. (and there's no chance to get a
> proper tcpdump either, at these traffic levels)

You really should start that capture, and on both client and server.
You don't need to dump everything, only traffic to or from server:distcc.

2008-07-22 13:57:59

[permalink] [raw]

Subject: Re: [TCP bug] stuck distcc connections in latest -git

* David Newall <[email protected]> wrote:

> > The hung condition seemed permanent (i waited a couple of minutes).
>
> Not nearly long enough. Retransmits can be sent as infrequently as
> per 180 seconds. I think there's an argument to use one of the the
> various patches that reduce your TCP_RTO_MAX, for example OBATA
> Noboru's (http://marc.info/?l=linux-netdev&m=118422471428855): you
> don't have to wait unreasonably long before seeing a retransmit.
> Remember, three minutes!

i know, i waited much more than 180 minutes - about 15 minutes. That is
more than enough for this LAN connection.

It's all on the LAN directly via a single gigabit switch and no packet
dropping. I noticed the hung build immediately as it happened.

> > I retried the same build 10 times and it would not reproduce - so
> > this again is a hard to reproduce condition. (and there's no chance
> > to get a proper tcpdump either, at these traffic levels)
>
> You really should start that capture, and on both client and server.
> You don't need to dump everything, only traffic to or from
> server:distcc.

It's not feasible. That box did in excess of 200 GB of network traffic
in the past 7 hours alone. ~10 clients are doing make -j200 type of
kernel builds to this 16way buildbox so it is not realistic to tcpdump
it - especially given the rarity of this problem. (it has not reoccured
since then) The network is local LAN, gigabit ethernet over a single
gigabit switch.

Ingo

2008-07-22 14:54:06

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

Larry Finger wrote:
> David Miller wrote:
>> From: Larry Finger <[email protected]>
>> Date: Tue, 22 Jul 2008 01:34:28 -0500
>>
>>>> GIT bisecting the lockdep problem is surely going the land you on:
>>>>
>>>> commit e308a5d806c852f56590ffdd3834d0df0cbed8d7
>>> No. It landed on this one.
>>
>> For the lockdep warnings?
>
> When I was just one commit later, I got both the lockdep warning and the
> BUG. This is the commit in question.

I actually don't see how you could still get the warning with
Dave's patch and the two I sent applied.

The warning is triggered by the dev_mc_sync call in
ieee80211_set_multicast_list:

dev_mc_sync(local->mdev, dev);

local->mdev is the wmaster device, which has its type set to
ARPHRD_IEEE80211. dev is regular wireless device with type set
to ARPHRD_ETHER. So they have distinct lockdep classes set
by register_netdevice.

The warning is:

Jul 21 15:11:07 larrylap kernel: NetworkManager/2661 is trying to
acquire lock:
Jul 21 15:11:07 larrylap kernel: (&dev->addr_list_lock){-...}, at:
[<ffffffff803a2961>] dev_mc_sync+0x19/0x57
Jul 21 15:11:07 larrylap kernel:
Jul 21 15:11:07 larrylap kernel: but task is already holding lock:
Jul 21 15:11:07 larrylap kernel: (&dev->addr_list_lock){-...}, at:
[<ffffffff8039e7c5>] dev_set_rx_mode+0x19/0x2e
Jul 21 15:11:07 larrylap kernel:

The only already held is dev->addr_list_lock, the one taken
by dev_mc_sync is local->mdev->addr_list_lock. And this shouldn't
cause any warnings because of the distinct lockdep classes.

Could you please retry with the three patches attached to this
mail? If the lockdep warning still triggers, please post it again.

Attachments:

01.diff (5.45 kB)
02.diff (2.18 kB)
03.diff (389.00 B)
Download all attachments

2008-07-22 14:54:40

by David Newall

[permalink] [raw]

Subject: Re: [TCP bug] stuck distcc connections in latest -git

Ingo Molnar wrote:
> * David Newall <[email protected]> wrote:
>
>> You really should start that capture, and on both client and server.
>> You don't need to dump everything, only traffic to or from
>> server:distcc.
>>
>
> It's not feasible. That box did in excess of 200 GB of network traffic
> in the past 7 hours alone.

You only need distcc traffic, and perhaps only after it's hung. With
250k outstanding per socket, are you certain that no traffic was sent?
Is it certain that one packet wasn't being sent each three minutes? I
suppose you're right and the stack really is stuck, but this is such an
easy thing to check and eliminate that you should do so. I suppose,
too, that you should trace the server-side processes and confirm that
they are waiting for socket input. You should dump tcp (for the distcc
port) next time the problem recurs and also check that the server
processes are waiting for socket input.

2008-07-22 15:35:19

[permalink] [raw]

Subject: Re: [TCP bug] stuck distcc connections in latest -git

* David Newall <[email protected]> wrote:

> Ingo Molnar wrote:
> > * David Newall <[email protected]> wrote:
> >
> >> You really should start that capture, and on both client and server.
> >> You don't need to dump everything, only traffic to or from
> >> server:distcc.
> >>
> >
> > It's not feasible. That box did in excess of 200 GB of network traffic
> > in the past 7 hours alone.
>
> You only need distcc traffic, and perhaps only after it's hung. With
> 250k outstanding per socket, are you certain that no traffic was sent?
> Is it certain that one packet wasn't being sent each three minutes? I
> suppose you're right and the stack really is stuck, but this is such
> an easy thing to check and eliminate that you should do so. I
> suppose, too, that you should trace the server-side processes and
> confirm that they are waiting for socket input. You should dump tcp
> (for the distcc port) next time the problem recurs and also check that
> the server processes are waiting for socket input.

ok, will do that if it happens again.

Ingo

2008-07-22 16:39:54

[permalink] [raw]

Subject: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

David and Patrick,

Here is the latest on this problem.

I pulled from Linus's tree this morning and now have git-05752-g93ded9b. The
kernel WARNING from __netif_schedule and the lockdep warning are present with or
without the patches from yesterday.

As I stated earlier, the kernel WARNING (it was a BUG then) was introduced in
commit 37437bb2 when the BUG statement was entered.

The lockdep warning started with the next commit (16361127).

I am not using any network traffic shaping. Is it correct that the faulty
condition is not that q == &noop_qdisc, but that __netif_schedule was called
when that condition exists?

The lockdep warning is:

=============================================
[ INFO: possible recursive locking detected ]
2.6.26-Linus-git-05752-g93ded9b #49
---------------------------------------------
NetworkManager/2611 is trying to acquire lock:
(&dev->addr_list_lock){-...}, at: [<ffffffff803a2ad1>] dev_mc_sync+0x19/0x57

but task is already holding lock:
(&dev->addr_list_lock){-...}, at: [<ffffffff8039e909>] dev_set_rx_mode+0x19/0x2e

other info that might help us debug this:
2 locks held by NetworkManager/2611:
#0: (rtnl_mutex){--..}, at: [<ffffffff803a8488>] rtnetlink_rcv+0x12/0x27
#1: (&dev->addr_list_lock){-...}, at: [<ffffffff8039e909>]
dev_set_rx_mode+0x19/0x2e

stack backtrace:
Pid: 2611, comm: NetworkManager Not tainted 2.6.26-Linus-git-05752-g93ded9b #49

Call Trace:
[<ffffffff80251b02>] __lock_acquire+0xb7b/0xecc
[<ffffffff80251ea4>] lock_acquire+0x51/0x6a
[<ffffffff803a2ad1>] dev_mc_sync+0x19/0x57
[<ffffffff8040b3fc>] _spin_lock_bh+0x23/0x2c
[<ffffffff803a2ad1>] dev_mc_sync+0x19/0x57
[<ffffffff8039e911>] dev_set_rx_mode+0x21/0x2e
[<ffffffff803a04da>] dev_open+0x8e/0xb0
[<ffffffff8039fe84>] dev_change_flags+0xa6/0x163
[<ffffffff803a7591>] do_setlink+0x286/0x349
[<ffffffff803a849d>] rtnetlink_rcv_msg+0x0/0x1ec
[<ffffffff803a849d>] rtnetlink_rcv_msg+0x0/0x1ec
[<ffffffff803a849d>] rtnetlink_rcv_msg+0x0/0x1ec
[<ffffffff803a77de>] rtnl_setlink+0x10b/0x10d
[<ffffffff803a849d>] rtnetlink_rcv_msg+0x0/0x1ec
[<ffffffff803b416f>] netlink_rcv_skb+0x34/0x7d
[<ffffffff803a8497>] rtnetlink_rcv+0x21/0x27
[<ffffffff803b3c6f>] netlink_unicast+0x1f0/0x261
[<ffffffff8039a58d>] __alloc_skb+0x66/0x12a
[<ffffffff803b3f48>] netlink_sendmsg+0x268/0x27b
[<ffffffff80393cb9>] sock_sendmsg+0xcb/0xe3
[<ffffffff80246aab>] autoremove_wake_function+0x0/0x2e
[<ffffffff8039b2d8>] verify_iovec+0x46/0x82
[<ffffffff80393ee8>] sys_sendmsg+0x217/0x28a
[<ffffffff80393642>] sockfd_lookup_light+0x1a/0x52
[<ffffffff80250990>] trace_hardirqs_on_caller+0xef/0x113
[<ffffffff8040af74>] trace_hardirqs_on_thunk+0x3a/0x3f
[<ffffffff8020be9b>] system_call_after_swapgs+0x7b/0x80
========================================================

The logged data for the WARNING is as follows:

------------[ cut here ]------------
WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()
Modules linked in: af_packet nfs lockd nfs_acl rfkill_input sunrpc
cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop
dm_mod arc4 ecb crypto_blkcipher b43 firmware_class rfkill mac80211 cfg80211
led_class input_polldev battery ac button ssb serio_raw forcedeth sr_mod cdrom
k8temp hwmon sg sd_mod ehci_hcd ohci_hcd usbcore edd fan thermal processor ext3
mbcache jbd pata_amd ahci libata scsi_mod dock
Pid: 1990, comm: b43 Not tainted 2.6.26-Linus-git-05752-g93ded9b #49

Call Trace:
[<ffffffff80233f6d>] warn_on_slowpath+0x51/0x8c
[<ffffffff8039d937>] __netif_schedule+0x2c/0x98
[<ffffffffa015445d>] ieee80211_scan_completed+0x26b/0x2f1 [mac80211]
[<ffffffffa01546de>] ieee80211_sta_scan_work+0x0/0x1b8 [mac80211]
[<ffffffff8024325e>] run_workqueue+0xf0/0x1f2
[<ffffffff8024343b>] worker_thread+0xdb/0xea
[<ffffffff80246aab>] autoremove_wake_function+0x0/0x2e
[<ffffffff80243360>] worker_thread+0x0/0xea
[<ffffffff8024678b>] kthread+0x47/0x73
[<ffffffff8040af74>] trace_hardirqs_on_thunk+0x3a/0x3f
[<ffffffff8020cea9>] child_rip+0xa/0x11
[<ffffffff8020c4df>] restore_args+0x0/0x30
[<ffffffff8024671f>] kthreadd+0x188/0x1ad
[<ffffffff80246744>] kthread+0x0/0x73
[<ffffffff8020ce9f>] child_rip+0x0/0x11

---[ end trace 42d234b678daea7a ]---

Other info I have found. The call to __netif_schedule from
ieee80211_scan_completed is through the following code from
include/linux/netdevice.h:

/**
* netif_wake_queue - restart transmit
* @dev: network device
*
* Allow upper layers to call the device hard_start_xmit routine.
* Used for flow control when transmit resources are available.
*/
static inline void netif_tx_wake_queue(struct netdev_queue *dev_queue)
{
#ifdef CONFIG_NETPOLL_TRAP
if (netpoll_trap()) {
clear_bit(__QUEUE_STATE_XOFF, &dev_queue->state);
return;
}
#endif
if (test_and_clear_bit(__QUEUE_STATE_XOFF, &dev_queue->state))
__netif_schedule(dev_queue->qdisc);
}

It doesn't make any difference if CONFIG_NETPOLL_TRAP is defined or not.

Please let me know if I can provide any further information,

Larry

2008-07-22 17:20:22

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

Larry Finger wrote:
> I pulled from Linus's tree this morning and now have git-05752-g93ded9b.
> The kernel WARNING from __netif_schedule and the lockdep warning are
> present with or without the patches from yesterday.
>
> As I stated earlier, the kernel WARNING (it was a BUG then) was
> introduced in commit 37437bb2 when the BUG statement was entered.
>
> The lockdep warning started with the next commit (16361127).
>
> The lockdep warning is:
>
> =============================================
> [ INFO: possible recursive locking detected ]
> 2.6.26-Linus-git-05752-g93ded9b #49

^^^^^^^^^^^^^^ dirty?

This kernel is not using the patches we sent.

2008-07-22 18:40:27

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

Patrick McHardy wrote:
> Larry Finger wrote:
>>
>> =============================================
>> [ INFO: possible recursive locking detected ]
>> 2.6.26-Linus-git-05752-g93ded9b #49
>
> ^^^^^^^^^^^^^^ dirty?
>
> This kernel is not using the patches we sent.

No, but they didn't make any difference. I tried with all 3 applied, then backed
them out one by one. That was the state when I posted before.

Here are the dumps with all 3 patches applied:

=============================================
[ INFO: possible recursive locking detected ]
2.6.26-Linus-05752-g93ded9b-dirty #53
---------------------------------------------
b43/1997 is trying to acquire lock:
(_xmit_IEEE80211#2){-...}, at: [<ffffffffa028f322>]
ieee80211_scan_completed+0x130/0x2e1 [mac80211]

but task is already holding lock:
(_xmit_IEEE80211#2){-...}, at: [<ffffffffa028f322>]
ieee80211_scan_completed+0x130/0x2e1 [mac80211]

other info that might help us debug this:
3 locks held by b43/1997:
#0: ((name)){--..}, at: [<ffffffff80245185>] run_workqueue+0xa7/0x1f2
#1: (&(&local->scan_work)->work){--..}, at: [<ffffffff80245185>]
run_workqueue+0xa7/0x1f2
#2: (_xmit_IEEE80211#2){-...}, at: [<ffffffffa028f322>]
ieee80211_scan_completed+0x130/0x2e1 [mac80211]

stack backtrace:
Pid: 1997, comm: b43 Not tainted 2.6.26-Linus-05752-g93ded9b-dirty #53

Call Trace:
[<ffffffff80255616>] __lock_acquire+0xb7b/0xecc
[<ffffffff8040c9a0>] __mutex_unlock_slowpath+0x100/0x10b
[<ffffffff802559b8>] lock_acquire+0x51/0x6a
[<ffffffffa028f322>] ieee80211_scan_completed+0x130/0x2e1 [mac80211]
[<ffffffff8040dc08>] _spin_lock+0x1e/0x27
[<ffffffffa028f322>] ieee80211_scan_completed+0x130/0x2e1 [mac80211]
[<ffffffffa028f6ce>] ieee80211_sta_scan_work+0x0/0x1b8 [mac80211]
[<ffffffff802451ce>] run_workqueue+0xf0/0x1f2
[<ffffffff802453ab>] worker_thread+0xdb/0xea
[<ffffffff80248a5f>] autoremove_wake_function+0x0/0x2e
[<ffffffff802452d0>] worker_thread+0x0/0xea
[<ffffffff80248731>] kthread+0x47/0x73
[<ffffffff8040d7b1>] trace_hardirqs_on_thunk+0x3a/0x3f
[<ffffffff8020ceb9>] child_rip+0xa/0x11
[<ffffffff8020c4ef>] restore_args+0x0/0x30
[<ffffffff802486c5>] kthreadd+0x19a/0x1bf
[<ffffffff802486ea>] kthread+0x0/0x73
[<ffffffff8020ceaf>] child_rip+0x0/0x11

------------[ cut here ]------------
WARNING: at net/core/dev.c:1344 __netif_schedule+0x2c/0x98()
Modules linked in: af_packet rfkill_input nfs lockd nfs_acl sunrpc
cpufreq_conservative cpufreq_userspace cpufreq_powersave powernow_k8 fuse loop
dm_mod arc4 ecb crypto_blkcipher b43 firmware_class rfkill mac80211
snd_hda_intel cfg80211 led_class input_polldev k8temp snd_pcm snd_timer battery
hwmon sr_mod forcedeth ssb joydev ac button serio_raw cdrom snd soundcore
snd_page_alloc sg sd_mod ohci_hcd ehci_hcd usbcore edd fan thermal processor
ext3 mbcache jbd pata_amd ahci libata scsi_mod dock
Pid: 1997, comm: b43 Not tainted 2.6.26-Linus-05752-g93ded9b-dirty #53

Call Trace:
[<ffffffff80235d49>] warn_on_slowpath+0x51/0x8c
[<ffffffff8040d7b1>] trace_hardirqs_on_thunk+0x3a/0x3f
[<ffffffff803a2413>] __netif_schedule+0x2c/0x98
[<ffffffffa028f44d>] ieee80211_scan_completed+0x25b/0x2e1 [mac80211]
[<ffffffffa028f6ce>] ieee80211_sta_scan_work+0x0/0x1b8 [mac80211]
[<ffffffff802451ce>] run_workqueue+0xf0/0x1f2
[<ffffffff802453ab>] worker_thread+0xdb/0xea
[<ffffffff80248a5f>] autoremove_wake_function+0x0/0x2e
[<ffffffff802452d0>] worker_thread+0x0/0xea
[<ffffffff80248731>] kthread+0x47/0x73
[<ffffffff8040d7b1>] trace_hardirqs_on_thunk+0x3a/0x3f
[<ffffffff8020ceb9>] child_rip+0xa/0x11
[<ffffffff8020c4ef>] restore_args+0x0/0x30
[<ffffffff802486c5>] kthreadd+0x19a/0x1bf
[<ffffffff802486ea>] kthread+0x0/0x73
[<ffffffff8020ceaf>] child_rip+0x0/0x11

---[ end trace 88fab857dc2a4242 ]---

Larry

2008-07-22 18:45:13

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

Larry Finger wrote:
> Patrick McHardy wrote:
>> Larry Finger wrote:
>>>
>>> =============================================
>>> [ INFO: possible recursive locking detected ]
>>> 2.6.26-Linus-git-05752-g93ded9b #49
>>
>> ^^^^^^^^^^^^^^ dirty?
>>
>> This kernel is not using the patches we sent.
>
> No, but they didn't make any difference. I tried with all 3 applied,
> then backed them out one by one. That was the state when I posted before.

Well, this is a completely different warning.

>
> Here are the dumps with all 3 patches applied:
>
> =============================================
> [ INFO: possible recursive locking detected ]
> 2.6.26-Linus-05752-g93ded9b-dirty #53
> ---------------------------------------------
> b43/1997 is trying to acquire lock:
> (_xmit_IEEE80211#2){-...}, at: [<ffffffffa028f322>]
> ieee80211_scan_completed+0x130/0x2e1 [mac80211]
>
> but task is already holding lock:
> (_xmit_IEEE80211#2){-...}, at: [<ffffffffa028f322>]
> ieee80211_scan_completed+0x130/0x2e1 [mac80211]

2008-07-22 19:32:37

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

Patrick McHardy wrote:
> Larry Finger wrote:
>> Patrick McHardy wrote:
>>> Larry Finger wrote:
>>>>
>>>> =============================================
>>>> [ INFO: possible recursive locking detected ]
>>>> 2.6.26-Linus-git-05752-g93ded9b #49
>>>
>>> ^^^^^^^^^^^^^^ dirty?
>>>
>>> This kernel is not using the patches we sent.
>>
>> No, but they didn't make any difference. I tried with all 3 applied,
>> then backed them out one by one. That was the state when I posted before.
>
> Well, this is a completely different warning.
>
>>
>> Here are the dumps with all 3 patches applied:
>>
>> =============================================
>> [ INFO: possible recursive locking detected ]
>> 2.6.26-Linus-05752-g93ded9b-dirty #53
>> ---------------------------------------------
>> b43/1997 is trying to acquire lock:
>> (_xmit_IEEE80211#2){-...}, at: [<ffffffffa028f322>]
>> ieee80211_scan_completed+0x130/0x2e1 [mac80211]
>>
>> but task is already holding lock:
>> (_xmit_IEEE80211#2){-...}, at: [<ffffffffa028f322>]
>> ieee80211_scan_completed+0x130/0x2e1 [mac80211]

Sorry. After all those kernel builds I got bleary-eyed and just looked for
recursive locking without any regard for the details.

Larry

2008-07-22 20:43:52

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

From: Larry Finger <[email protected]>
Date: Tue, 22 Jul 2008 07:52:24 -0500

> David Miller wrote:
> > From: Larry Finger <[email protected]>
> > Date: Tue, 22 Jul 2008 01:34:28 -0500
> >
> >> David Miller wrote:
> >>> From: Larry Finger <[email protected]>
> >>> Date: Mon, 21 Jul 2008 17:40:10 -0500
> >>>
> >>>> Sorry :(
> >>>>
> >>>> I used the davem patch, the second version of your first one, and your second
> >>>> one. Both problems persist.
> >>>>
> >>>> Still plugging away on bisection.
> >>> GIT bisecting the lockdep problem is surely going the land you on:
> >>>
> >>> commit e308a5d806c852f56590ffdd3834d0df0cbed8d7
> >> No. It landed on this one.
> >
> > For the lockdep warnings?
>
> No - this one triggers the kernel BUG as follows:

Well, we were trying to fix the lockdep warnings. One
thing at a time :-)

And that BUG is now just a WARN in Linus's tree, so if
you update you should get further along.

2008-07-22 21:14:47

[permalink] [raw]

Subject: Re: [TCP bug] stuck distcc connections in latest -git

On Tue, Jul 22, 2008 at 05:34:43PM +0200, Ingo Molnar wrote:
> * David Newall <[email protected]> wrote:
>
> > Ingo Molnar wrote:
> > > * David Newall <[email protected]> wrote:
> > >
> > >> You really should start that capture, and on both client and server.
> > >> You don't need to dump everything, only traffic to or from
> > >> server:distcc.
> > >>
> > >
> > > It's not feasible. That box did in excess of 200 GB of network traffic
> > > in the past 7 hours alone.
> >
> > You only need distcc traffic, and perhaps only after it's hung. With
> > 250k outstanding per socket, are you certain that no traffic was sent?
> > Is it certain that one packet wasn't being sent each three minutes? I
> > suppose you're right and the stack really is stuck, but this is such
> > an easy thing to check and eliminate that you should do so. I
> > suppose, too, that you should trace the server-side processes and
> > confirm that they are waiting for socket input. You should dump tcp
> > (for the distcc port) next time the problem recurs and also check that
> > the server processes are waiting for socket input.
>
> ok, will do that if it happens again.

Ingo,

if it can help, I have a "capture" script which allows you to define
a size and will rotate captures within that size. That's what I'm
using to troubleshoot rarely occuring problems in datacenters, so
it's horrible but efficient :-)

You just have to stop it once the problem has happened again. Ping
me if you're interested (I'm lazy to start my laptop right just for
it now in fact).

Willy

2008-07-22 21:19:58

[permalink] [raw]

Subject: Re: [crash] BUG: unable to handle kernel NULL pointer dereference at 0000000000000370

From: Patrick McHardy <[email protected]>
Date: Tue, 22 Jul 2008 16:53:30 +0200

> Could you please retry with the three patches attached to this
> mail? If the lockdep warning still triggers, please post it again.

Since I'm convinced the original lockdep spurious warning
issue is cured, and we're looking at something different
now, I've integrated all of these fixes together as one
commit as below.

Thanks a lot Patrick.

netdev: Handle ->addr_list_lock just like ->_xmit_lock for lockdep.

The new address list lock needs to handle the same device layering
issues that the _xmit_lock one does.

This integrates work done by Patrick McHardy.

Signed-off-by: David S. Miller <[email protected]>

diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index 9737c06..a641eea 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -5041,6 +5041,7 @@ static int bond_check_params(struct bond_params *params)
}

static struct lock_class_key bonding_netdev_xmit_lock_key;
+static struct lock_class_key bonding_netdev_addr_lock_key;

static void bond_set_lockdep_class_one(struct net_device *dev,
struct netdev_queue *txq,
@@ -5052,6 +5053,8 @@ static void bond_set_lockdep_class_one(struct net_device *dev,

static void bond_set_lockdep_class(struct net_device *dev)
{
+ lockdep_set_class(&dev->addr_list_lock,
+ &bonding_netdev_addr_lock_key);
netdev_for_each_tx_queue(dev, bond_set_lockdep_class_one, NULL);
}

diff --git a/drivers/net/hamradio/bpqether.c b/drivers/net/hamradio/bpqether.c
index b6500b2..58f4b1d 100644
--- a/drivers/net/hamradio/bpqether.c
+++ b/drivers/net/hamradio/bpqether.c
@@ -123,6 +123,7 @@ static LIST_HEAD(bpq_devices);
* off into a separate class since they always nest.
*/
static struct lock_class_key bpq_netdev_xmit_lock_key;
+static struct lock_class_key bpq_netdev_addr_lock_key;

static void bpq_set_lockdep_class_one(struct net_device *dev,
struct netdev_queue *txq,
@@ -133,6 +134,7 @@ static void bpq_set_lockdep_class_one(struct net_device *dev,

static void bpq_set_lockdep_class(struct net_device *dev)
{
+ lockdep_set_class(&dev->addr_list_lock, &bpq_netdev_addr_lock_key);
netdev_for_each_tx_queue(dev, bpq_set_lockdep_class_one, NULL);
}

diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
index efbc155..4239450 100644
--- a/drivers/net/macvlan.c
+++ b/drivers/net/macvlan.c
@@ -276,6 +276,7 @@ static int macvlan_change_mtu(struct net_device *dev, int new_mtu)
* separate class since they always nest.
*/
static struct lock_class_key macvlan_netdev_xmit_lock_key;
+static struct lock_class_key macvlan_netdev_addr_lock_key;

#define MACVLAN_FEATURES \
(NETIF_F_SG | NETIF_F_ALL_CSUM | NETIF_F_HIGHDMA | NETIF_F_FRAGLIST | \
@@ -295,6 +296,8 @@ static void macvlan_set_lockdep_class_one(struct net_device *dev,

static void macvlan_set_lockdep_class(struct net_device *dev)
{
+ lockdep_set_class(&dev->addr_list_lock,
+ &macvlan_netdev_addr_lock_key);
netdev_for_each_tx_queue(dev, macvlan_set_lockdep_class_one, NULL);
}

diff --git a/drivers/net/wireless/hostap/hostap_hw.c b/drivers/net/wireless/hostap/hostap_hw.c
index 13d5882..3153fe9 100644
--- a/drivers/net/wireless/hostap/hostap_hw.c
+++ b/drivers/net/wireless/hostap/hostap_hw.c
@@ -3101,6 +3101,7 @@ static void prism2_clear_set_tim_queue(local_info_t *local)
* This is a natural nesting, which needs a split lock type.
*/
static struct lock_class_key hostap_netdev_xmit_lock_key;
+static struct lock_class_key hostap_netdev_addr_lock_key;

static void prism2_set_lockdep_class_one(struct net_device *dev,
struct netdev_queue *txq,
@@ -3112,6 +3113,8 @@ static void prism2_set_lockdep_class_one(struct net_device *dev,

static void prism2_set_lockdep_class(struct net_device *dev)
{
+ lockdep_set_class(&dev->addr_list_lock,
+ &hostap_netdev_addr_lock_key);
netdev_for_each_tx_queue(dev, prism2_set_lockdep_class_one, NULL);
}

diff --git a/net/8021q/vlan_dev.c b/net/8021q/vlan_dev.c
index f42bc2b..4bf014e 100644
--- a/net/8021q/vlan_dev.c
+++ b/net/8021q/vlan_dev.c
@@ -569,6 +569,7 @@ static void vlan_dev_set_rx_mode(struct net_device *vlan_dev)
* separate class since they always nest.
*/
static struct lock_class_key vlan_netdev_xmit_lock_key;
+static struct lock_class_key vlan_netdev_addr_lock_key;

static void vlan_dev_set_lockdep_one(struct net_device *dev,
struct netdev_queue *txq,
@@ -581,6 +582,9 @@ static void vlan_dev_set_lockdep_one(struct net_device *dev,

static void vlan_dev_set_lockdep_class(struct net_device *dev, int subclass)
{
+ lockdep_set_class_and_subclass(&dev->addr_list_lock,
+ &vlan_netdev_addr_lock_key,
+ subclass);
netdev_for_each_tx_queue(dev, vlan_dev_set_lockdep_one, &subclass);
}

diff --git a/net/core/dev.c b/net/core/dev.c
index 65eea83..6bf217d 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -261,7 +261,7 @@ static RAW_NOTIFIER_HEAD(netdev_chain);

DEFINE_PER_CPU(struct softnet_data, softnet_data);

-#ifdef CONFIG_DEBUG_LOCK_ALLOC
+#ifdef CONFIG_LOCKDEP
/*
* register_netdevice() inits txq->_xmit_lock and sets lockdep class
* according to dev->type
@@ -301,6 +301,7 @@ static const char *netdev_lock_name[] =
"_xmit_NONE"};

static struct lock_class_key netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)];
+static struct lock_class_key netdev_addr_lock_key[ARRAY_SIZE(netdev_lock_type)];

static inline unsigned short netdev_lock_pos(unsigned short dev_type)
{
@@ -313,8 +314,8 @@ static inline unsigned short netdev_lock_pos(unsigned short dev_type)
return ARRAY_SIZE(netdev_lock_type) - 1;
}

-static inline void netdev_set_lockdep_class(spinlock_t *lock,
- unsigned short dev_type)
+static inline void netdev_set_xmit_lockdep_class(spinlock_t *lock,
+ unsigned short dev_type)
{
int i;

@@ -322,9 +323,22 @@ static inline void netdev_set_lockdep_class(spinlock_t *lock,
lockdep_set_class_and_name(lock, &netdev_xmit_lock_key[i],
netdev_lock_name[i]);
}
+
+static inline void netdev_set_addr_lockdep_class(struct net_device *dev)
+{
+ int i;
+
+ i = netdev_lock_pos(dev->type);
+ lockdep_set_class_and_name(&dev->addr_list_lock,
+ &netdev_addr_lock_key[i],
+ netdev_lock_name[i]);
+}
#else
-static inline void netdev_set_lockdep_class(spinlock_t *lock,
- unsigned short dev_type)
+static inline void netdev_set_xmit_lockdep_class(spinlock_t *lock,
+ unsigned short dev_type)
+{
+}
+static inline void netdev_set_addr_lockdep_class(struct net_device *dev)
{
}
#endif
@@ -3851,7 +3865,7 @@ static void __netdev_init_queue_locks_one(struct net_device *dev,
void *_unused)
{
spin_lock_init(&dev_queue->_xmit_lock);
- netdev_set_lockdep_class(&dev_queue->_xmit_lock, dev->type);
+ netdev_set_xmit_lockdep_class(&dev_queue->_xmit_lock, dev->type);
dev_queue->xmit_lock_owner = -1;
}

@@ -3896,6 +3910,7 @@ int register_netdevice(struct net_device *dev)
net = dev_net(dev);

spin_lock_init(&dev->addr_list_lock);
+ netdev_set_addr_lockdep_class(dev);
netdev_init_queue_locks(dev);

dev->iflink = -1;
diff --git a/net/netrom/af_netrom.c b/net/netrom/af_netrom.c
index fccc250..532e4fa 100644
--- a/net/netrom/af_netrom.c
+++ b/net/netrom/af_netrom.c
@@ -73,6 +73,7 @@ static const struct proto_ops nr_proto_ops;
* separate class since they always nest.
*/
static struct lock_class_key nr_netdev_xmit_lock_key;
+static struct lock_class_key nr_netdev_addr_lock_key;

static void nr_set_lockdep_one(struct net_device *dev,
struct netdev_queue *txq,
@@ -83,6 +84,7 @@ static void nr_set_lockdep_one(struct net_device *dev,

static void nr_set_lockdep_key(struct net_device *dev)
{
+ lockdep_set_class(&dev->addr_list_lock, &nr_netdev_addr_lock_key);
netdev_for_each_tx_queue(dev, nr_set_lockdep_one, NULL);
}

diff --git a/net/rose/af_rose.c b/net/rose/af_rose.c
index dbc963b..a7f1ce1 100644
--- a/net/rose/af_rose.c
+++ b/net/rose/af_rose.c
@@ -74,6 +74,7 @@ ax25_address rose_callsign;
* separate class since they always nest.
*/
static struct lock_class_key rose_netdev_xmit_lock_key;
+static struct lock_class_key rose_netdev_addr_lock_key;

static void rose_set_lockdep_one(struct net_device *dev,
struct netdev_queue *txq,
@@ -84,6 +85,7 @@ static void rose_set_lockdep_one(struct net_device *dev,

static void rose_set_lockdep_key(struct net_device *dev)
{
+ lockdep_set_class(&dev->addr_list_lock, &rose_netdev_addr_lock_key);
netdev_for_each_tx_queue(dev, rose_set_lockdep_one, NULL);
}

2008-07-22 23:04:24

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

From: Larry Finger <[email protected]>
Date: Tue, 22 Jul 2008 13:39:08 -0500

> =============================================
> [ INFO: possible recursive locking detected ]
> 2.6.26-Linus-05752-g93ded9b-dirty #53
> ---------------------------------------------
> b43/1997 is trying to acquire lock:
> (_xmit_IEEE80211#2){-...}, at: [<ffffffffa028f322>]
> ieee80211_scan_completed+0x130/0x2e1 [mac80211]
>
> but task is already holding lock:
> (_xmit_IEEE80211#2){-...}, at: [<ffffffffa028f322>]
> ieee80211_scan_completed+0x130/0x2e1 [mac80211]
>
> other info that might help us debug this:
> 3 locks held by b43/1997:
> #0: ((name)){--..}, at: [<ffffffff80245185>] run_workqueue+0xa7/0x1f2
> #1: (&(&local->scan_work)->work){--..}, at: [<ffffffff80245185>]
> run_workqueue+0xa7/0x1f2
> #2: (_xmit_IEEE80211#2){-...}, at: [<ffffffffa028f322>]
> ieee80211_scan_completed+0x130/0x2e1 [mac80211]
>
> stack backtrace:
> Pid: 1997, comm: b43 Not tainted 2.6.26-Linus-05752-g93ded9b-dirty #53
>
> Call Trace:
> [<ffffffff80255616>] __lock_acquire+0xb7b/0xecc
> [<ffffffff8040c9a0>] __mutex_unlock_slowpath+0x100/0x10b
> [<ffffffff802559b8>] lock_acquire+0x51/0x6a
> [<ffffffffa028f322>] ieee80211_scan_completed+0x130/0x2e1 [mac80211]
> [<ffffffff8040dc08>] _spin_lock+0x1e/0x27
> [<ffffffffa028f322>] ieee80211_scan_completed+0x130/0x2e1 [mac80211]
> [<ffffffffa028f6ce>] ieee80211_sta_scan_work+0x0/0x1b8 [mac80211]
> [<ffffffff802451ce>] run_workqueue+0xf0/0x1f2
> [<ffffffff802453ab>] worker_thread+0xdb/0xea
> [<ffffffff80248a5f>] autoremove_wake_function+0x0/0x2e
> [<ffffffff802452d0>] worker_thread+0x0/0xea
> [<ffffffff80248731>] kthread+0x47/0x73
> [<ffffffff8040d7b1>] trace_hardirqs_on_thunk+0x3a/0x3f
> [<ffffffff8020ceb9>] child_rip+0xa/0x11
> [<ffffffff8020c4ef>] restore_args+0x0/0x30
> [<ffffffff802486c5>] kthreadd+0x19a/0x1bf
> [<ffffffff802486ea>] kthread+0x0/0x73
> [<ffffffff8020ceaf>] child_rip+0x0/0x11

Lockdep doesn't like that we have an array of objects (the TX queues)
and we're iterating over them grabbing all of their locks.

Does anyone know how to teach lockdep that this is OK?

2008-07-23 06:15:44

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On 23-07-2008 01:04, David Miller wrote:
> From: Larry Finger <[email protected]>
> Date: Tue, 22 Jul 2008 13:39:08 -0500
>
>> =============================================
>> [ INFO: possible recursive locking detected ]
>> 2.6.26-Linus-05752-g93ded9b-dirty #53
>> ---------------------------------------------
>> b43/1997 is trying to acquire lock:
>> (_xmit_IEEE80211#2){-...}, at: [<ffffffffa028f322>]
>> ieee80211_scan_completed+0x130/0x2e1 [mac80211]
>>
>> but task is already holding lock:
>> (_xmit_IEEE80211#2){-...}, at: [<ffffffffa028f322>]
>> ieee80211_scan_completed+0x130/0x2e1 [mac80211]
...
> Lockdep doesn't like that we have an array of objects (the TX queues)
> and we're iterating over them grabbing all of their locks.
>
> Does anyone know how to teach lockdep that this is OK?

I guess, David Miller knows...:

http://permalink.gmane.org/gmane.linux.network/99784

Jarek P.

PS: if there is nothing new in lockdep the classical method would
be to change this static array:

static struct lock_class_key
netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)];

to

static struct lock_class_key
netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)][MAX_NUM_TX_QUEUES];

and set lockdep classes per queue as well. (If we are sure we don't
need lockdep subclasses anywhere this could be optimized by using
one lock_class_key per 8 queues and spin_lock_nested()).

2008-07-23 07:59:33

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

From: Jarek Poplawski <[email protected]>
Date: Wed, 23 Jul 2008 06:20:36 +0000

> PS: if there is nothing new in lockdep the classical method would
> be to change this static array:
>
> static struct lock_class_key
> netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)];
>
> to
>
> static struct lock_class_key
> netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)][MAX_NUM_TX_QUEUES];
>
> and set lockdep classes per queue as well. (If we are sure we don't
> need lockdep subclasses anywhere this could be optimized by using
> one lock_class_key per 8 queues and spin_lock_nested()).

Unfortunately MAX_NUM_TX_QUEUES is USHORT_MAX, so this isn't really
a feasible approach.

spin_lock_nested() isn't all that viable either, as the subclass
limit is something like 8.

2008-07-23 08:28:00

[permalink] [raw]

Subject: Re: [TCP bug] stuck distcc connections in latest -git

* Willy Tarreau <[email protected]> wrote:

> On Tue, Jul 22, 2008 at 05:34:43PM +0200, Ingo Molnar wrote:
> > * David Newall <[email protected]> wrote:
> >
> > > Ingo Molnar wrote:
> > > > * David Newall <[email protected]> wrote:
> > > >
> > > >> You really should start that capture, and on both client and server.
> > > >> You don't need to dump everything, only traffic to or from
> > > >> server:distcc.
> > > >>
> > > >
> > > > It's not feasible. That box did in excess of 200 GB of network traffic
> > > > in the past 7 hours alone.
> > >
> > > You only need distcc traffic, and perhaps only after it's hung. With
> > > 250k outstanding per socket, are you certain that no traffic was sent?
> > > Is it certain that one packet wasn't being sent each three minutes? I
> > > suppose you're right and the stack really is stuck, but this is such
> > > an easy thing to check and eliminate that you should do so. I
> > > suppose, too, that you should trace the server-side processes and
> > > confirm that they are waiting for socket input. You should dump tcp
> > > (for the distcc port) next time the problem recurs and also check that
> > > the server processes are waiting for socket input.
> >
> > ok, will do that if it happens again.
>
> Ingo,
>
> if it can help, I have a "capture" script which allows you to define a
> size and will rotate captures within that size. That's what I'm using
> to troubleshoot rarely occuring problems in datacenters, so it's
> horrible but efficient :-)
>
> You just have to stop it once the problem has happened again. Ping me
> if you're interested (I'm lazy to start my laptop right just for it
> now in fact).

yeah, that would be handy, thanks.

Alas, the problem has not reoccured since then - more than a thousand
kernel builds down the line. Yesterday it triggered so quickly when i
updated the buildbox to the new kernel, and happened repeatedly when i
tried to build a new kernel, that i didnt assume it was something hard
to reproduce - but it went poof after i restarted distccd on the server.

So i'd suggest we do not count this as a regression, i've got no way at
the moment of reproducing it reliably.

Ingo

2008-07-23 08:49:59

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Wed, Jul 23, 2008 at 12:59:21AM -0700, David Miller wrote:
> From: Jarek Poplawski <[email protected]>
> Date: Wed, 23 Jul 2008 06:20:36 +0000
>
> > PS: if there is nothing new in lockdep the classical method would
> > be to change this static array:
> >
> > static struct lock_class_key
> > netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)];
> >
> > to
> >
> > static struct lock_class_key
> > netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)][MAX_NUM_TX_QUEUES];
> >
> > and set lockdep classes per queue as well. (If we are sure we don't
> > need lockdep subclasses anywhere this could be optimized by using
> > one lock_class_key per 8 queues and spin_lock_nested()).
>
> Unfortunately MAX_NUM_TX_QUEUES is USHORT_MAX, so this isn't really
> a feasible approach.

Is it used by real devices already? Maybe for the beginning we could
start with something less?

> spin_lock_nested() isn't all that viable either, as the subclass
> limit is something like 8.

This method would need to do some additional counting: depending of
a queue number each 8 subsequent queues share (are set to) the same
class and their number mod 8 gives the subqueue number for
spin_lock_nested().

I'll try to find if there is something new around this in lockdep.
(lockdep people added to CC.)

Jarek P.

2008-07-23 09:03:23

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Wed, 2008-07-23 at 08:54 +0000, Jarek Poplawski wrote:
> On Wed, Jul 23, 2008 at 12:59:21AM -0700, David Miller wrote:
> > From: Jarek Poplawski <[email protected]>
> > Date: Wed, 23 Jul 2008 06:20:36 +0000
> >
> > > PS: if there is nothing new in lockdep the classical method would
> > > be to change this static array:
> > >
> > > static struct lock_class_key
> > > netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)];
> > >
> > > to
> > >
> > > static struct lock_class_key
> > > netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)][MAX_NUM_TX_QUEUES];
> > >
> > > and set lockdep classes per queue as well. (If we are sure we don't
> > > need lockdep subclasses anywhere this could be optimized by using
> > > one lock_class_key per 8 queues and spin_lock_nested()).
> >
> > Unfortunately MAX_NUM_TX_QUEUES is USHORT_MAX, so this isn't really
> > a feasible approach.
>
> Is it used by real devices already? Maybe for the beginning we could
> start with something less?
>
> > spin_lock_nested() isn't all that viable either, as the subclass
> > limit is something like 8.
>
> This method would need to do some additional counting: depending of
> a queue number each 8 subsequent queues share (are set to) the same
> class and their number mod 8 gives the subqueue number for
> spin_lock_nested().
>
> I'll try to find if there is something new around this in lockdep.
> (lockdep people added to CC.)

There isn't.

Is there a static data structure that the driver needs to instantiate to
'create' a queue? Something like:

/* this imaginary e1000 hardware has 16 hardware queues */
static struct net_tx_queue e1000e_tx_queues[16];

In that case you can stick the key in there and do:

int e1000e_init_tx_queue(struct net_tx_queue *txq)
{
...

spin_lock_init(&txq->tx_lock);
lockdep_set_class(&txq->tx_lock, &txq->tx_lock_key);

...
}

( This is what the scheduler runqueues also do )

2008-07-23 09:30:07

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Wed, Jul 23, 2008 at 11:03:06AM +0200, Peter Zijlstra wrote:
> On Wed, 2008-07-23 at 08:54 +0000, Jarek Poplawski wrote:
> > On Wed, Jul 23, 2008 at 12:59:21AM -0700, David Miller wrote:
> > > From: Jarek Poplawski <[email protected]>
> > > Date: Wed, 23 Jul 2008 06:20:36 +0000
> > >
> > > > PS: if there is nothing new in lockdep the classical method would
> > > > be to change this static array:
> > > >
> > > > static struct lock_class_key
> > > > netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)];
> > > >
> > > > to
> > > >
> > > > static struct lock_class_key
> > > > netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)][MAX_NUM_TX_QUEUES];
> > > >
> > > > and set lockdep classes per queue as well. (If we are sure we don't
> > > > need lockdep subclasses anywhere this could be optimized by using
> > > > one lock_class_key per 8 queues and spin_lock_nested()).
> > >
> > > Unfortunately MAX_NUM_TX_QUEUES is USHORT_MAX, so this isn't really
> > > a feasible approach.
> >
> > Is it used by real devices already? Maybe for the beginning we could
> > start with something less?
> >
> > > spin_lock_nested() isn't all that viable either, as the subclass
> > > limit is something like 8.
> >
> > This method would need to do some additional counting: depending of
> > a queue number each 8 subsequent queues share (are set to) the same
> > class and their number mod 8 gives the subqueue number for
> > spin_lock_nested().
> >
> > I'll try to find if there is something new around this in lockdep.
> > (lockdep people added to CC.)
>
> There isn't.
>
> Is there a static data structure that the driver needs to instantiate to
> 'create' a queue? Something like:
>
> /* this imaginary e1000 hardware has 16 hardware queues */
> static struct net_tx_queue e1000e_tx_queues[16];

I guess, not.

Then, IMHO, we could be practical and simply skip lockdep validation
for "some" range of locks: e.g. to set the table for the first 256
queues only, and to do e.g. __raw_spin_lock() for bigger numbers. (If
there are any bad locking patterns this should be enough for checking.)

Jarek P.

2008-07-23 09:50:30

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Wed, 2008-07-23 at 09:35 +0000, Jarek Poplawski wrote:
> On Wed, Jul 23, 2008 at 11:03:06AM +0200, Peter Zijlstra wrote:
> > On Wed, 2008-07-23 at 08:54 +0000, Jarek Poplawski wrote:
> > > On Wed, Jul 23, 2008 at 12:59:21AM -0700, David Miller wrote:
> > > > From: Jarek Poplawski <[email protected]>
> > > > Date: Wed, 23 Jul 2008 06:20:36 +0000
> > > >
> > > > > PS: if there is nothing new in lockdep the classical method would
> > > > > be to change this static array:
> > > > >
> > > > > static struct lock_class_key
> > > > > netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)];
> > > > >
> > > > > to
> > > > >
> > > > > static struct lock_class_key
> > > > > netdev_xmit_lock_key[ARRAY_SIZE(netdev_lock_type)][MAX_NUM_TX_QUEUES];
> > > > >
> > > > > and set lockdep classes per queue as well. (If we are sure we don't
> > > > > need lockdep subclasses anywhere this could be optimized by using
> > > > > one lock_class_key per 8 queues and spin_lock_nested()).
> > > >
> > > > Unfortunately MAX_NUM_TX_QUEUES is USHORT_MAX, so this isn't really
> > > > a feasible approach.
> > >
> > > Is it used by real devices already? Maybe for the beginning we could
> > > start with something less?
> > >
> > > > spin_lock_nested() isn't all that viable either, as the subclass
> > > > limit is something like 8.
> > >
> > > This method would need to do some additional counting: depending of
> > > a queue number each 8 subsequent queues share (are set to) the same
> > > class and their number mod 8 gives the subqueue number for
> > > spin_lock_nested().
> > >
> > > I'll try to find if there is something new around this in lockdep.
> > > (lockdep people added to CC.)
> >
> > There isn't.
> >
> > Is there a static data structure that the driver needs to instantiate to
> > 'create' a queue? Something like:
> >
> > /* this imaginary e1000 hardware has 16 hardware queues */
> > static struct net_tx_queue e1000e_tx_queues[16];
>
> I guess, not.
>
> Then, IMHO, we could be practical and simply skip lockdep validation
> for "some" range of locks: e.g. to set the table for the first 256
> queues only, and to do e.g. __raw_spin_lock() for bigger numbers. (If
> there are any bad locking patterns this should be enough for checking.)

definite NAK on using raw spinlock ops...

I'll go look at this multiqueue stuff to see if there is anything to be
done.. David, what would be a good commit to start reading?

2008-07-23 10:09:18

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Wed, Jul 23, 2008 at 11:50:14AM +0200, Peter Zijlstra wrote:
...
> definite NAK on using raw spinlock ops...
>
> I'll go look at this multiqueue stuff to see if there is anything to be
> done.. David, what would be a good commit to start reading?

...In the case David ever sleeps: I think, the current Linus' git is
good enough (the problem is with netdevice.h: netif_tx_lock()).

Jarek P.

2008-07-23 10:58:30

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Wed, 2008-07-23 at 10:13 +0000, Jarek Poplawski wrote:
> On Wed, Jul 23, 2008 at 11:50:14AM +0200, Peter Zijlstra wrote:
> ....
> > definite NAK on using raw spinlock ops...
> >
> > I'll go look at this multiqueue stuff to see if there is anything to be
> > done.. David, what would be a good commit to start reading?
>
> ....In the case David ever sleeps: I think, the current Linus' git is
> good enough (the problem is with netdevice.h: netif_tx_lock()).

Ah, right,...

that takes a whole bunch of locks at once..

Is that really needed? - when I grep for its usage its surprisingly few
drivers using it and even less generic code.

When I look at the mac802.11 code in ieee80211_tx_pending() it looks
like it can do with just one lock at a time, instead of all - but I
might be missing some obvious details.

So I guess my question is, is netif_tx_lock() here to stay, or is the
right fix to convert all those drivers to use __netif_tx_lock() which
locks only a single queue?

2008-07-23 11:30:30

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Wed, Jul 23, 2008 at 12:58:16PM +0200, Peter Zijlstra wrote:
...
> Ah, right,...
>
> that takes a whole bunch of locks at once..
>
> Is that really needed? - when I grep for its usage its surprisingly few
> drivers using it and even less generic code.
>
> When I look at the mac802.11 code in ieee80211_tx_pending() it looks
> like it can do with just one lock at a time, instead of all - but I
> might be missing some obvious details.
>
> So I guess my question is, is netif_tx_lock() here to stay, or is the
> right fix to convert all those drivers to use __netif_tx_lock() which
> locks only a single queue?
>

It's a new thing mainly for new hardware/drivers, and just after
conversion (older drivers effectively use __netif_tx_lock()), so it'll
probably stay for some time until something better is found. David,
will tell the rest, I hope.

Jarek P.

2008-07-23 11:44:23

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Wed, Jul 23, 2008 at 11:35:19AM +0000, Jarek Poplawski wrote:
> On Wed, Jul 23, 2008 at 12:58:16PM +0200, Peter Zijlstra wrote:
...
> > When I look at the mac802.11 code in ieee80211_tx_pending() it looks
> > like it can do with just one lock at a time, instead of all - but I
> > might be missing some obvious details.
> >
> > So I guess my question is, is netif_tx_lock() here to stay, or is the
> > right fix to convert all those drivers to use __netif_tx_lock() which
> > locks only a single queue?
> >
>
> It's a new thing mainly for new hardware/drivers, and just after
> conversion (older drivers effectively use __netif_tx_lock()), so it'll
> probably stay for some time until something better is found. David,
> will tell the rest, I hope.

...And, of course, these new drivers should also lock a single queue
where possible.

Jarek P.

2008-07-23 20:14:52

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

From: Peter Zijlstra <[email protected]>
Date: Wed, 23 Jul 2008 12:58:16 +0200

> So I guess my question is, is netif_tx_lock() here to stay, or is the
> right fix to convert all those drivers to use __netif_tx_lock() which
> locks only a single queue?

It's staying.

It's trying to block all potential calls into the ->hard_start_xmit()
method of the driver, and the only reliable way to do that is to take
all the TX queue locks. And in one form or another, we're going to
have this "grab/release all the TX queue locks" construct.

I find it interesting that this cannot be simply described to lockdep
:-)

2008-07-23 20:16:24

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

From: Jarek Poplawski <[email protected]>
Date: Wed, 23 Jul 2008 11:49:14 +0000

> On Wed, Jul 23, 2008 at 11:35:19AM +0000, Jarek Poplawski wrote:
> > On Wed, Jul 23, 2008 at 12:58:16PM +0200, Peter Zijlstra wrote:
> ...
> > > When I look at the mac802.11 code in ieee80211_tx_pending() it looks
> > > like it can do with just one lock at a time, instead of all - but I
> > > might be missing some obvious details.
> > >
> > > So I guess my question is, is netif_tx_lock() here to stay, or is the
> > > right fix to convert all those drivers to use __netif_tx_lock() which
> > > locks only a single queue?
> > >
> >
> > It's a new thing mainly for new hardware/drivers, and just after
> > conversion (older drivers effectively use __netif_tx_lock()), so it'll
> > probably stay for some time until something better is found. David,
> > will tell the rest, I hope.
>
> ...And, of course, these new drivers should also lock a single queue
> where possible.

It isn't going away.

There will always be a need for a "stop all the TX queues" operation.

2008-07-23 20:43:04

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Wed, Jul 23, 2008 at 01:16:07PM -0700, David Miller wrote:
...
> There will always be a need for a "stop all the TX queues" operation.

The question is if the current way is "all correct". As a matter of
fact I think Peter's doubts could be justified: taking "USHORT_MAX"
locks looks really dubious (so maybe it's not so strange lockdep
didn't get used to this).

Jarek P.

2008-07-23 20:55:42

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

From: Jarek Poplawski <[email protected]>
Date: Wed, 23 Jul 2008 22:43:35 +0200

> On Wed, Jul 23, 2008 at 01:16:07PM -0700, David Miller wrote:
> ...
> > There will always be a need for a "stop all the TX queues" operation.
>
> The question is if the current way is "all correct". As a matter of
> fact I think Peter's doubts could be justified: taking "USHORT_MAX"
> locks looks really dubious (so maybe it's not so strange lockdep
> didn't get used to this).

There are, of course, potentially other ways to achieve the objective.

And for non-multiqueue aware devices (which is the vast majority of
the 400 or so networking drivers we have) there is only one queue and
thus one lock taken.

2008-07-23 23:43:14

[permalink] [raw]

Subject: Re: [GIT]: Networking

From: Patrick McHardy <[email protected]>
Date: Mon, 21 Jul 2008 22:33:18 +0200

> Its not needed today of course, so the attached patch changes it
> to depend on NETFILTER_ADVANCED and removes the default.

I've applied this, thanks Patrick.

2008-07-24 06:05:47

[permalink] [raw]

Subject: [TCP bug, regression] stuck distcc connections in latest -git

* Ingo Molnar <[email protected]> wrote:

> Alas, the problem has not reoccured since then - more than a thousand
> kernel builds down the line. [...]

the permanently hug distcc kernel build bug triggered again, twice.
First time it happened yesterday, i left it running overnight and it
never recovered after a 14+ hours of wait.

It shows a similar pattern, 'ESTABLISHED' state on both sides, but the
client-side is stuck and the server (running latest kernel) is seemingly
clueless about that fact:

client:

Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 375450 10.0.1.16:39201 10.0.1.19:3632 ESTABLISHED

server:

Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 10.0.1.19:3632 10.0.1.16:39201 ESTABLISHED

i waited ~30 minutes in this second case.

the client (running 2.6.24) does periodic 120 seconds retransmits:

07:40:48.255452 IP dione.39201 > phoenix.distcc: . 1608:2144(536) ack 1 win 584007:40:48.255547 IP phoenix.distcc > dione.39201: . ack 2144 win 65535
07:40:48.255564 IP dione.39201 > phoenix.distcc: . 67143:67679(536) ack 1 win 5840
07:40:48.255648 IP phoenix.distcc > dione.39201: . ack 2144 win 65535
07:42:48.255440 IP dione.39201 > phoenix.distcc: . 2144:2680(536) ack 1 win 5840
07:42:48.255559 IP phoenix.distcc > dione.39201: . ack 2680 win 65535
07:42:48.255570 IP dione.39201 > phoenix.distcc: . 67679:68215(536) ack 1 win 5840
07:42:48.255659 IP phoenix.distcc > dione.39201: . ack 2680 win 65535
07:44:48.255436 IP dione.39201 > phoenix.distcc: . 2680:3216(536) ack 1 win 584007:44:48.255570 IP phoenix.distcc > dione.39201: . ack 3216 win 65535
07:44:48.255585 IP dione.39201 > phoenix.distcc: . 68215:68751(536) ack 1 win 5840
07:44:48.255669 IP phoenix.distcc > dione.39201: . ack 3216 win 65535

the server (running the latest kernel) responds:

07:40:47.551098 IP dione.39201 > phoenix.distcc: . 1072:1608(536) ack 1 win 584007:40:47.551141 IP phoenix.distcc > dione.39201: . ack 1608 win 65535
07:40:47.551204 IP dione.39201 > phoenix.distcc: . 66607:67143(536) ack 1 win 5840
07:40:47.551213 IP phoenix.distcc > dione.39201: . ack 1608 win 65535
07:42:47.570994 IP dione.39201 > phoenix.distcc: . 1608:2144(536) ack 1 win 584007:42:47.571027 IP phoenix.distcc > dione.39201: . ack 2144 win 65535
07:42:47.571117 IP dione.39201 > phoenix.distcc: . 67143:67679(536) ack 1 win 5840
07:42:47.571127 IP phoenix.distcc > dione.39201: . ack 2144 win 65535
07:44:47.590901 IP dione.39201 > phoenix.distcc: . 2144:2680(536) ack 1 win 584007:44:47.590960 IP phoenix.distcc > dione.39201: . ack 2680 win 65535
07:44:47.591042 IP dione.39201 > phoenix.distcc: . 67679:68215(536) ack 1 win 5840
07:44:47.591054 IP phoenix.distcc > dione.39201: . ack 2680 win 65535

full client socket state:

dione:~> grep $(printf "%X\n" 39201) /proc/net/tcp
44: 1001000A:9921 1301000A:0E30 01 0005ABF2:00000000 01:00002B8A
00000000 500 0 63130083 2 ffff81000c762d00 120000 0 0 28 101

[ a few minutes later ]

44: 1001000A:9921 1301000A:0E30 01 0005A392:00000000 01:00002BF0
00000000 500 0 63130083 2 ffff81000c762d00 120000 0 0 32 101

[ i.e. the tx queue did increase by 2144 bytes - 4x 536 bytes ]

full server socket state:

phoenix:~> grep $(printf "%X\n" 39201) /proc/net/tcp
6: 1301000A:0E30 1001000A:9921 01 00000000:00000000 00:00000000
00000000 99 0 728382 1 ffff88042d8db280 300 4 30 2 -1

[ a few minutes later ]

6: 1301000A:0E30 1001000A:9921 01 00000000:00000000 00:00000000
00000000 99 0 728382 1 ffff88042d8db280 300 4 30 2 -1

[ i.e. no change - no pending packets ]

I've started a longer capture session as well - it seems the TCP stack
is slowly cycling through retransmissions of 536-byte packets, with
375450 bytes pending? At 120 seconds a pop that would be about 23 hours
to make any progress on - but i'm not sure i interpreted that right. It
all looks very weird.

The timestamps of the two boxes are synced up to within about 1 second:

earth4:~> for N in dione phoenix; do ssh $N date; done
Thu Jul 24 07:44:02 CEST 2008
Thu Jul 24 07:44:02 CEST 2008

( but the two boxes are responding to each other fine, so ordering of
events is not a question here. )

Any other state you'd like to see before i continue with -tip testing?

Ingo

2008-07-24 06:33:36

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

here's a longer log from the server, with sequences, flags, etc:

08:06:47.809947 IP (tos 0x0, ttl 64, id 13998, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234555110:234555646(536) ack 2272574194 win 5840
08:06:47.809974 IP (tos 0x0, ttl 64, id 27389, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x9900 (correct), 2272574194:2272574194(0) ack 234555646 win 65535
08:06:47.810051 IP (tos 0x0, ttl 64, id 13999, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234620645:234621181(536) ack 2272574194 win 5840
08:06:47.810065 IP (tos 0x0, ttl 64, id 27390, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x9900 (correct), 2272574194:2272574194(0) ack 234555646 win 65535
08:08:47.829813 IP (tos 0x0, ttl 64, id 14000, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234555646:234556182(536) ack 2272574194 win 5840
08:08:47.829909 IP (tos 0x0, ttl 64, id 27391, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x96e8 (correct), 2272574194:2272574194(0) ack 234556182 win 65535
08:08:47.830009 IP (tos 0x0, ttl 64, id 14001, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234621181:234621717(536) ack 2272574194 win 5840
08:08:47.830026 IP (tos 0x0, ttl 64, id 27392, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x96e8 (correct), 2272574194:2272574194(0) ack 234556182 win 65535
08:10:47.849756 IP (tos 0x0, ttl 64, id 14002, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234556182:234556718(536) ack 2272574194 win 5840
08:10:47.849845 IP (tos 0x0, ttl 64, id 27393, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x94d0 (correct), 2272574194:2272574194(0) ack 234556718 win 65535
08:10:47.849936 IP (tos 0x0, ttl 64, id 14003, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234621717:234622253(536) ack 2272574194 win 5840
08:10:47.849951 IP (tos 0x0, ttl 64, id 27394, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x94d0 (correct), 2272574194:2272574194(0) ack 234556718 win 65535
08:12:47.870018 IP (tos 0x0, ttl 64, id 14004, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234556718:234557254(536) ack 2272574194 win 5840
08:12:47.870091 IP (tos 0x0, ttl 64, id 27395, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x92b8 (correct), 2272574194:2272574194(0) ack 234557254 win 65535
08:12:47.870217 IP (tos 0x0, ttl 64, id 14005, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234622253:234622789(536) ack 2272574194 win 5840
08:12:47.870266 IP (tos 0x0, ttl 64, id 27396, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x92b8 (correct), 2272574194:2272574194(0) ack 234557254 win 65535
08:14:47.889925 IP (tos 0x0, ttl 64, id 14006, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234557254:234557790(536) ack 2272574194 win 5840
08:14:47.890006 IP (tos 0x0, ttl 64, id 27397, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x90a0 (correct), 2272574194:2272574194(0) ack 234557790 win 65535
08:14:47.890105 IP (tos 0x0, ttl 64, id 14007, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234622789:234623325(536) ack 2272574194 win 5840
08:14:47.890120 IP (tos 0x0, ttl 64, id 27398, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x90a0 (correct), 2272574194:2272574194(0) ack 234557790 win 65535
08:16:47.909483 IP (tos 0x0, ttl 64, id 14008, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234557790:234558326(536) ack 2272574194 win 5840
08:16:47.909583 IP (tos 0x0, ttl 64, id 27399, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8e88 (correct), 2272574194:2272574194(0) ack 234558326 win 65535
08:16:47.909700 IP (tos 0x0, ttl 64, id 14009, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234623325:234623861(536) ack 2272574194 win 5840
08:16:47.909717 IP (tos 0x0, ttl 64, id 27400, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8e88 (correct), 2272574194:2272574194(0) ack 234558326 win 65535
08:18:47.929372 IP (tos 0x0, ttl 64, id 14010, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234558326:234558862(536) ack 2272574194 win 5840
08:18:47.929428 IP (tos 0x0, ttl 64, id 27401, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8c70 (correct), 2272574194:2272574194(0) ack 234558862 win 65535
08:18:47.929506 IP (tos 0x0, ttl 64, id 14011, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234623861:234624397(536) ack 2272574194 win 5840
08:18:47.929519 IP (tos 0x0, ttl 64, id 27402, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8c70 (correct), 2272574194:2272574194(0) ack 234558862 win 65535
08:20:47.949661 IP (tos 0x0, ttl 64, id 14012, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234558862:234559398(536) ack 2272574194 win 5840
08:20:47.949761 IP (tos 0x0, ttl 64, id 27403, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8a58 (correct), 2272574194:2272574194(0) ack 234559398 win 65535
08:20:47.949867 IP (tos 0x0, ttl 64, id 14013, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234624397:234624933(536) ack 2272574194 win 5840
08:20:47.949884 IP (tos 0x0, ttl 64, id 27404, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8a58 (correct), 2272574194:2272574194(0) ack 234559398 win 65535
08:22:47.969196 IP (tos 0x0, ttl 64, id 14014, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234559398:234559934(536) ack 2272574194 win 5840
08:22:47.969291 IP (tos 0x0, ttl 64, id 27405, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8840 (correct), 2272574194:2272574194(0) ack 234559934 win 65535
08:22:47.969372 IP (tos 0x0, ttl 64, id 14015, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234624933:234625469(536) ack 2272574194 win 5840
08:22:47.969433 IP (tos 0x0, ttl 64, id 27406, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8840 (correct), 2272574194:2272574194(0) ack 234559934 win 65535
08:24:47.989458 IP (tos 0x0, ttl 64, id 14016, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234559934:234560470(536) ack 2272574194 win 5840
08:24:47.989518 IP (tos 0x0, ttl 64, id 27407, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8628 (correct), 2272574194:2272574194(0) ack 234560470 win 65535
08:24:47.989642 IP (tos 0x0, ttl 64, id 14017, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234625469:234626005(536) ack 2272574194 win 5840
08:24:47.989655 IP (tos 0x0, ttl 64, id 27408, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8628 (correct), 2272574194:2272574194(0) ack 234560470 win 65535
08:26:48.009355 IP (tos 0x0, ttl 64, id 14018, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234560470:234561006(536) ack 2272574194 win 5840
08:26:48.009432 IP (tos 0x0, ttl 64, id 27409, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8410 (correct), 2272574194:2272574194(0) ack 234561006 win 65535
08:26:48.009523 IP (tos 0x0, ttl 64, id 14019, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234626005:234626541(536) ack 2272574194 win 5840
08:26:48.009539 IP (tos 0x0, ttl 64, id 27410, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8410 (correct), 2272574194:2272574194(0) ack 234561006 win 65535
08:28:48.028909 IP (tos 0x0, ttl 64, id 14020, offset 0, flags [DF], proto TCP (6), length 290) dione.39201 > phoenix.distcc: . 234561006:234561256(250) ack 2272574194 win 5840
08:28:48.028961 IP (tos 0x0, ttl 64, id 27411, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8316 (correct), 2272574194:2272574194(0) ack 234561256 win 65535
08:28:48.029044 IP (tos 0x0, ttl 64, id 14021, offset 0, flags [DF], proto TCP (6), length 147) dione.39201 > phoenix.distcc: P 234626541:234626648(107) ack 2272574194 win 5840
08:28:48.029057 IP (tos 0x0, ttl 64, id 27412, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8316 (correct), 2272574194:2272574194(0) ack 234561256 win 65535
08:30:48.049167 IP (tos 0x0, ttl 64, id 14022, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234561256:234561792(536) ack 2272574194 win 5840
08:30:48.049223 IP (tos 0x0, ttl 64, id 27413, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535
08:30:48.049341 IP (tos 0x0, ttl 64, id 14023, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234626648:234627184(536) ack 2272574194 win 5840
08:30:48.049348 IP (tos 0x0, ttl 64, id 14024, offset 0, flags [DF], proto TCP (6), length 183) dione.39201 > phoenix.distcc: . 234627184:234627327(143) ack 2272574194 win 5840
08:30:48.049354 IP (tos 0x0, ttl 64, id 27414, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535
08:30:48.049359 IP (tos 0x0, ttl 64, id 27415, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535

2008-07-24 07:00:54

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Wed, 2008-07-23 at 13:14 -0700, David Miller wrote:
> From: Peter Zijlstra <[email protected]>
> Date: Wed, 23 Jul 2008 12:58:16 +0200
>
> > So I guess my question is, is netif_tx_lock() here to stay, or is the
> > right fix to convert all those drivers to use __netif_tx_lock() which
> > locks only a single queue?
>
> It's staying.
>
> It's trying to block all potential calls into the ->hard_start_xmit()
> method of the driver, and the only reliable way to do that is to take
> all the TX queue locks. And in one form or another, we're going to
> have this "grab/release all the TX queue locks" construct.
>
> I find it interesting that this cannot be simply described to lockdep
> :-)

If you think its OK to take USHORT_MAX locks at once, I'm afraid we'll
have to agree to disagree :-/

Thing is, lockdep wants to be able to describe the locking hierarchy
with classes, and each class needs to be in static storage for various
reasons.

So if you make a locking hierarchy that is USHORT_MAX deep, you need at
least that amount of static classes.

Also, you'll run into the fact that lockdep will only track like 48 held
locks, after that it self terminates.

I'm aware of only 2 sites in the kernel that break this limit. The
down-side of stretching this limit is that deep lock chains come with
costs (esp so on -rt), so I'm not particularly eager to grow this - it
might give the impresssion its a good idea to have very long lock
chains.

2008-07-24 07:34:55

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

On Thu, Jul 24, 2008 at 08:32:42AM +0200, Ingo Molnar wrote:
>
> here's a longer log from the server, with sequences, flags, etc:

good. The sequence numbers from dione to phoenix are bouncing back and
forth because a big data chunk was lost, so dione is alternatively
sending and old segment (retransmit) which, when ACKed, allows it to
slide the window forward and send one more chunk from the window tip.

> 08:06:47.809947 IP (tos 0x0, ttl 64, id 13998, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234555110:234555646(536) ack 2272574194 win 5840
> 08:06:47.809974 IP (tos 0x0, ttl 64, id 27389, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x9900 (correct), 2272574194:2272574194(0) ack 234555646 win 65535
> 08:06:47.810051 IP (tos 0x0, ttl 64, id 13999, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234620645:234621181(536) ack 2272574194 win 5840
> 08:06:47.810065 IP (tos 0x0, ttl 64, id 27390, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x9900 (correct), 2272574194:2272574194(0) ack 234555646 win 65535
> 08:08:47.829813 IP (tos 0x0, ttl 64, id 14000, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234555646:234556182(536) ack 2272574194 win 5840
> 08:08:47.829909 IP (tos 0x0, ttl 64, id 27391, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x96e8 (correct), 2272574194:2272574194(0) ack 234556182 win 65535
> 08:08:47.830009 IP (tos 0x0, ttl 64, id 14001, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234621181:234621717(536) ack 2272574194 win 5840
(...)
> 08:30:48.049167 IP (tos 0x0, ttl 64, id 14022, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234561256:234561792(536) ack 2272574194 win 5840
> 08:30:48.049223 IP (tos 0x0, ttl 64, id 27413, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535
> 08:30:48.049341 IP (tos 0x0, ttl 64, id 14023, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234626648:234627184(536) ack 2272574194 win 5840
> 08:30:48.049348 IP (tos 0x0, ttl 64, id 14024, offset 0, flags [DF], proto TCP (6), length 183) dione.39201 > phoenix.distcc: . 234627184:234627327(143) ack 2272574194 win 5840

here it looks like dione believes it does not have the chunk starting
at 234561792. Maybe it moved its window too far ? It only sends the
tip of the window.

> 08:30:48.049354 IP (tos 0x0, ttl 64, id 27414, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535
> 08:30:48.049359 IP (tos 0x0, ttl 64, id 27415, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535

And phoenix insists on getting what is missing from the window.

So dione is wrong here.

Regards,
Willy

2008-07-24 07:54:59

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

Ingo Molnar <[email protected]> wrote:
>
> here's a longer log from the server, with sequences, flags, etc:

Sorry but which is the server, phoenix?

> 08:28:48.029057 IP (tos 0x0, ttl 64, id 27412, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8316 (correct), 2272574194:2272574194(0) ack 234561256 win 65535
> 08:30:48.049167 IP (tos 0x0, ttl 64, id 14022, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234561256:234561792(536) ack 2272574194 win 5840
> 08:30:48.049223 IP (tos 0x0, ttl 64, id 27413, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535
> 08:30:48.049341 IP (tos 0x0, ttl 64, id 14023, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234626648:234627184(536) ack 2272574194 win 5840
> 08:30:48.049348 IP (tos 0x0, ttl 64, id 14024, offset 0, flags [DF], proto TCP (6), length 183) dione.39201 > phoenix.distcc: . 234627184:234627327(143) ack 2272574194 win 5840
> 08:30:48.049354 IP (tos 0x0, ttl 64, id 27414, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535
> 08:30:48.049359 IP (tos 0x0, ttl 64, id 27415, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535

No further traffic is generated for this connection?

For a problem like this, you really need to dump on both sides
to get the full picture. Otherwise it's difficult to tell why
we have a gap after 234561792 which is never filled in by dione.

BTW any reason why you appear to have SACK off?

Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

2008-07-24 08:26:22

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

On Thu, Jul 24, 2008 at 03:53:40PM +0800, Herbert Xu wrote:
> Ingo Molnar <[email protected]> wrote:
> >
> > here's a longer log from the server, with sequences, flags, etc:
>
> Sorry but which is the server, phoenix?
>
> > 08:28:48.029057 IP (tos 0x0, ttl 64, id 27412, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8316 (correct), 2272574194:2272574194(0) ack 234561256 win 65535
> > 08:30:48.049167 IP (tos 0x0, ttl 64, id 14022, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234561256:234561792(536) ack 2272574194 win 5840
> > 08:30:48.049223 IP (tos 0x0, ttl 64, id 27413, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535
> > 08:30:48.049341 IP (tos 0x0, ttl 64, id 14023, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234626648:234627184(536) ack 2272574194 win 5840
> > 08:30:48.049348 IP (tos 0x0, ttl 64, id 14024, offset 0, flags [DF], proto TCP (6), length 183) dione.39201 > phoenix.distcc: . 234627184:234627327(143) ack 2272574194 win 5840
> > 08:30:48.049354 IP (tos 0x0, ttl 64, id 27414, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535
> > 08:30:48.049359 IP (tos 0x0, ttl 64, id 27415, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535
>
> No further traffic is generated for this connection?
>
> For a problem like this, you really need to dump on both sides
> to get the full picture. Otherwise it's difficult to tell why
> we have a gap after 234561792 which is never filled in by dione.

Given the TTLs, it looks to me like both are on the same LAN. Also,
234561792 is the window trail it seems.

> BTW any reason why you appear to have SACK off?

and also MSS is at the lowest value (536), maybe because of numerous
losses on large segments ?

Willy

2008-07-24 08:29:07

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

* Herbert Xu <[email protected]> wrote:

> Ingo Molnar <[email protected]> wrote:
> >
> > here's a longer log from the server, with sequences, flags, etc:
>
> Sorry but which is the server, phoenix?

yeah.

> > 08:28:48.029057 IP (tos 0x0, ttl 64, id 27412, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x8316 (correct), 2272574194:2272574194(0) ack 234561256 win 65535
> > 08:30:48.049167 IP (tos 0x0, ttl 64, id 14022, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234561256:234561792(536) ack 2272574194 win 5840
> > 08:30:48.049223 IP (tos 0x0, ttl 64, id 27413, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535
> > 08:30:48.049341 IP (tos 0x0, ttl 64, id 14023, offset 0, flags [DF], proto TCP (6), length 576) dione.39201 > phoenix.distcc: . 234626648:234627184(536) ack 2272574194 win 5840
> > 08:30:48.049348 IP (tos 0x0, ttl 64, id 14024, offset 0, flags [DF], proto TCP (6), length 183) dione.39201 > phoenix.distcc: . 234627184:234627327(143) ack 2272574194 win 5840
> > 08:30:48.049354 IP (tos 0x0, ttl 64, id 27414, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535
> > 08:30:48.049359 IP (tos 0x0, ttl 64, id 27415, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535
>
> No further traffic is generated for this connection?

you mean besides the periodic 2 minute stuff? Correct.

> For a problem like this, you really need to dump on both sides to get
> the full picture. Otherwise it's difficult to tell why we have a gap
> after 234561792 which is never filled in by dione.

well, i dumped on both sides and posted the result (in my first mail),
didnt i?

> BTW any reason why you appear to have SACK off?

default distro setup, Fedora 9 and Fedora 8. No extra tweaking that i'm
aware of.

Fortunately this is my lucky day and i just triggered an half-incident,
a "half hang". The connection was stuck for about 40 minutes - and 2
minutes after i started tracing it on both sides it got unstuck. If the
stall is shorter than that i'd normally not notice it. (except for a
drop in testing throughput)

Server (phoenix) and client (dione) dumps attached.

Note, these hung TCP problems started after i upgraded the server's
kernel to the big networking merge. Somewhere in the 24 hours preceding
my first report:

Date: Tue, 22 Jul 2008 13:21:33 +0200

I upgrade the server's kernel daily, so the regression window is pretty
accurate. So the prime suspect is one of these 1232 commits:

| commit db6d8c7a4027b48d797b369a53f8470aaeed7063
| Merge: 3a53337... fb65a7c...
| Author: Linus Torvalds <[email protected]>
| Date: Sun Jul 20 17:43:29 2008 -0700
|
| Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6
|
| * git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6: (1232 commits)

i.e.:

last known good kernel is: v2.6.26-3465-g5b664cb
first known broken kernel is: v2.6.26-4811-gdb6d8c7
last tested (still broken) kernel is: v2.6.26-6077-gc010b2f

Still not reproducible enough to bisect it.

Thanks,

Ingo

------------------>
[traced on the client, dione]

10:03:23.590481 IP (tos 0x0, ttl 64, id 46098, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xb6ce), 1979565914:1979566450(536) ack 2260258133 win 5840
10:03:23.590593 IP (tos 0x0, ttl 64, id 19850, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5a6b (correct), 2260258133:2260258133(0) ack 1979566450 win 65535
10:03:23.590605 IP (tos 0x0, ttl 64, id 46099, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x2bd1), 1979631449:1979631985(536) ack 2260258133 win 5840
10:03:23.590693 IP (tos 0x0, ttl 64, id 19851, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5a6b (correct), 2260258133:2260258133(0) ack 1979566450 win 65535
10:05:23.590442 IP (tos 0x0, ttl 64, id 46100, offset 0, flags [DF], proto TCP (6), length 183) dione.40344 > phoenix.distcc: ., cksum 0x16cc (incorrect (-> 0xc006), 1979566450:1979566593(143) ack 2260258133 win 5840
10:05:23.590580 IP (tos 0x0, ttl 64, id 19852, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x59dc (correct), 2260258133:2260258133(0) ack 1979566593 win 65535
10:05:23.590605 IP (tos 0x0, ttl 64, id 46101, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xa38a), 1979566593:1979567129(536) ack 2260258133 win 5840
10:05:23.590608 IP (tos 0x0, ttl 64, id 46102, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x7d22), 1979567129:1979567665(536) ack 2260258133 win 5840
10:05:23.590879 IP (tos 0x0, ttl 64, id 19853, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5a6b (correct), 2260258133:2260258133(0) ack 1979631985 win 65535
10:05:23.590976 IP (tos 0x0, ttl 64, id 46103, offset 0, flags [DF], proto TCP (6), length 1648) dione.40344 > phoenix.distcc: ., cksum 0x1c85 (incorrect (-> 0x2436), 1979631985:1979633593(1608) ack 2260258133 win 5840
10:05:23.590982 IP (tos 0x0, ttl 64, id 19854, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5a6b (correct), 2260258133:2260258133(0) ack 1979631985 win 65535
10:05:23.590986 IP (tos 0x0, ttl 64, id 46106, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x0eb7), 1979633593:1979634129(536) ack 2260258133 win 5840
10:05:23.591079 IP (tos 0x0, ttl 64, id 19855, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5853 (correct), 2260258133:2260258133(0) ack 1979632521 win 65535
10:05:23.591087 IP (tos 0x0, ttl 64, id 46107, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x3e75), 1979634129:1979634665(536) ack 2260258133 win 5840
10:05:23.591090 IP (tos 0x0, ttl 64, id 19856, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x563b (correct), 2260258133:2260258133(0) ack 1979633057 win 65535
10:05:23.591094 IP (tos 0x0, ttl 64, id 46108, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x48d3), 1979634665:1979635737(1072) ack 2260258133 win 5840
10:05:23.591098 IP (tos 0x0, ttl 64, id 19857, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5423 (correct), 2260258133:2260258133(0) ack 1979633593 win 65535
10:05:23.591103 IP (tos 0x0, ttl 64, id 46110, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xe7b1), 1979635737:1979636809(1072) ack 2260258133 win 5840
10:05:23.591107 IP (tos 0x0, ttl 64, id 19858, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x520b (correct), 2260258133:2260258133(0) ack 1979634129 win 65535
10:05:23.591111 IP (tos 0x0, ttl 64, id 46112, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x5fa7), 1979636809:1979637881(1072) ack 2260258133 win 5840
10:05:23.591179 IP (tos 0x0, ttl 64, id 19859, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4ff3 (correct), 2260258133:2260258133(0) ack 1979634665 win 65535
10:05:23.591186 IP (tos 0x0, ttl 64, id 46114, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x8a24), 1979637881:1979638953(1072) ack 2260258133 win 5840
10:05:23.591190 IP (tos 0x0, ttl 64, id 19860, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4ddb (correct), 2260258133:2260258133(0) ack 1979635201 win 65535
10:05:23.591194 IP (tos 0x0, ttl 64, id 46116, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x0da7), 1979638953:1979640025(1072) ack 2260258133 win 5840
10:05:23.591198 IP (tos 0x0, ttl 64, id 19861, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4bc3 (correct), 2260258133:2260258133(0) ack 1979635737 win 65535
10:05:23.591202 IP (tos 0x0, ttl 64, id 46118, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x7008), 1979640025:1979641097(1072) ack 2260258133 win 5840
10:05:23.591206 IP (tos 0x0, ttl 64, id 19862, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x49ab (correct), 2260258133:2260258133(0) ack 1979636273 win 65535
10:05:23.591210 IP (tos 0x0, ttl 64, id 46120, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x9591), 1979641097:1979642169(1072) ack 2260258133 win 5840
10:05:23.591214 IP (tos 0x0, ttl 64, id 19863, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4793 (correct), 2260258133:2260258133(0) ack 1979636809 win 65535
10:05:23.591219 IP (tos 0x0, ttl 64, id 46122, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x3377), 1979642169:1979643241(1072) ack 2260258133 win 5840
10:05:23.591222 IP (tos 0x0, ttl 64, id 19864, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x457b (correct), 2260258133:2260258133(0) ack 1979637345 win 65535
10:05:23.591228 IP (tos 0x0, ttl 64, id 46124, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xf421), 1979643241:1979644313(1072) ack 2260258133 win 5840
10:05:23.591232 IP (tos 0x0, ttl 64, id 19865, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4363 (correct), 2260258133:2260258133(0) ack 1979637881 win 65535
10:05:23.591236 IP (tos 0x0, ttl 64, id 46126, offset 0, flags [DF], proto TCP (6), length 433) dione.40344 > phoenix.distcc: ., cksum 0x17c6 (incorrect (-> 0x4f0c), 1979644313:1979644706(393) ack 2260258133 win 5840
10:05:23.591239 IP (tos 0x0, ttl 64, id 46127, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xd389), 1979644706:1979645242(536) ack 2260258133 win 5840
10:05:23.591254 IP (tos 0x0, ttl 64, id 19866, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x414b (correct), 2260258133:2260258133(0) ack 1979638417 win 65535
10:05:23.591257 IP (tos 0x0, ttl 64, id 46128, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x10ea), 1979645242:1979646314(1072) ack 2260258133 win 5840
10:05:23.591259 IP (tos 0x0, ttl 64, id 19867, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3f33 (correct), 2260258133:2260258133(0) ack 1979638953 win 65535
10:05:23.591262 IP (tos 0x0, ttl 64, id 46130, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x7867), 1979646314:1979647386(1072) ack 2260258133 win 5840
10:05:23.591264 IP (tos 0x0, ttl 64, id 19868, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3d1b (correct), 2260258133:2260258133(0) ack 1979639489 win 65535
10:05:23.591267 IP (tos 0x0, ttl 64, id 46132, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x0843), 1979647386:1979648458(1072) ack 2260258133 win 5840
10:05:23.591269 IP (tos 0x0, ttl 64, id 19869, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3b03 (correct), 2260258133:2260258133(0) ack 1979640025 win 65535
10:05:23.591272 IP (tos 0x0, ttl 64, id 46134, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x3bd3), 1979648458:1979649530(1072) ack 2260258133 win 5840
10:05:23.591274 IP (tos 0x0, ttl 64, id 19870, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x38eb (correct), 2260258133:2260258133(0) ack 1979640561 win 65535
10:05:23.591276 IP (tos 0x0, ttl 64, id 46136, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xe070), 1979649530:1979650602(1072) ack 2260258133 win 5840
10:05:23.591330 IP (tos 0x0, ttl 64, id 19871, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x36d3 (correct), 2260258133:2260258133(0) ack 1979641097 win 65535
10:05:23.591333 IP (tos 0x0, ttl 64, id 46138, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xc112), 1979650602:1979651674(1072) ack 2260258133 win 5840
10:05:23.591336 IP (tos 0x0, ttl 64, id 19872, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x34bb (correct), 2260258133:2260258133(0) ack 1979641633 win 65535
10:05:23.591338 IP (tos 0x0, ttl 64, id 46140, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x49a1), 1979651674:1979652746(1072) ack 2260258133 win 5840
10:05:23.591341 IP (tos 0x0, ttl 64, id 19873, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x32a3 (correct), 2260258133:2260258133(0) ack 1979642169 win 65535
10:05:23.591343 IP (tos 0x0, ttl 64, id 46142, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xd0d1), 1979652746:1979653818(1072) ack 2260258133 win 5840
10:05:23.591345 IP (tos 0x0, ttl 64, id 19874, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x308b (correct), 2260258133:2260258133(0) ack 1979642705 win 65535
10:05:23.591348 IP (tos 0x0, ttl 64, id 46144, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x835e), 1979653818:1979654890(1072) ack 2260258133 win 5840
10:05:23.591350 IP (tos 0x0, ttl 64, id 19875, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2e73 (correct), 2260258133:2260258133(0) ack 1979643241 win 65535
10:05:23.591352 IP (tos 0x0, ttl 64, id 46146, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xffb0), 1979654890:1979655962(1072) ack 2260258133 win 5840
10:05:23.591354 IP (tos 0x0, ttl 64, id 19876, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2c5b (correct), 2260258133:2260258133(0) ack 1979643777 win 65535
10:05:23.591357 IP (tos 0x0, ttl 64, id 46148, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x4b8d), 1979655962:1979656498(536) ack 2260258133 win 5840
10:05:23.591359 IP (tos 0x0, ttl 64, id 19877, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2a43 (correct), 2260258133:2260258133(0) ack 1979644313 win 65535
10:05:23.591361 IP (tos 0x0, ttl 64, id 46149, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x7825), 1979656498:1979657034(536) ack 2260258133 win 5840
10:05:23.591362 IP (tos 0x0, ttl 64, id 19878, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x28ba (correct), 2260258133:2260258133(0) ack 1979644706 win 65535
10:05:23.591364 IP (tos 0x0, ttl 64, id 46150, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xdf49), 1979657034:1979658106(1072) ack 2260258133 win 5840
10:05:23.591367 IP (tos 0x0, ttl 64, id 19879, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x26a2 (correct), 2260258133:2260258133(0) ack 1979645242 win 65535
10:05:23.591369 IP (tos 0x0, ttl 64, id 46152, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xcd06), 1979658106:1979658642(536) ack 2260258133 win 5840
10:05:23.591371 IP (tos 0x0, ttl 64, id 19880, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x248a (correct), 2260258133:2260258133(0) ack 1979645778 win 65535
10:05:23.591373 IP (tos 0x0, ttl 64, id 46153, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xf71e), 1979658642:1979659178(536) ack 2260258133 win 5840
10:05:23.591374 IP (tos 0x0, ttl 64, id 19881, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2272 (correct), 2260258133:2260258133(0) ack 1979646314 win 65535
10:05:23.591377 IP (tos 0x0, ttl 64, id 46154, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x2b14), 1979659178:1979659714(536) ack 2260258133 win 5840
10:05:23.591378 IP (tos 0x0, ttl 64, id 19882, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x205a (correct), 2260258133:2260258133(0) ack 1979646850 win 65535
10:05:23.591381 IP (tos 0x0, ttl 64, id 46155, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xb4b3), 1979659714:1979660250(536) ack 2260258133 win 5840
10:05:23.591382 IP (tos 0x0, ttl 64, id 19883, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1e42 (correct), 2260258133:2260258133(0) ack 1979647386 win 65535
10:05:23.591384 IP (tos 0x0, ttl 64, id 46156, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xfe09), 1979660250:1979661322(1072) ack 2260258133 win 5840
10:05:23.591387 IP (tos 0x0, ttl 64, id 19884, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1c2a (correct), 2260258133:2260258133(0) ack 1979647922 win 65535
10:05:23.591389 IP (tos 0x0, ttl 64, id 46158, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x0961), 1979661322:1979661858(536) ack 2260258133 win 5840
10:05:23.591391 IP (tos 0x0, ttl 64, id 19885, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1a12 (correct), 2260258133:2260258133(0) ack 1979648458 win 65535
10:05:23.591392 IP (tos 0x0, ttl 64, id 46159, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xacb2), 1979661858:1979662394(536) ack 2260258133 win 5840
10:05:23.591427 IP (tos 0x0, ttl 64, id 19886, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x17fa (correct), 2260258133:2260258133(0) ack 1979648994 win 65535
10:05:23.591431 IP (tos 0x0, ttl 64, id 46160, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x0efd), 1979662394:1979662930(536) ack 2260258133 win 5840
10:05:23.591433 IP (tos 0x0, ttl 64, id 19887, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x15e2 (correct), 2260258133:2260258133(0) ack 1979649530 win 65535
10:05:23.591434 IP (tos 0x0, ttl 64, id 46161, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x79a1), 1979662930:1979663466(536) ack 2260258133 win 5840
10:05:23.591436 IP (tos 0x0, ttl 64, id 19888, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x13ca (correct), 2260258133:2260258133(0) ack 1979650066 win 65535
10:05:23.591438 IP (tos 0x0, ttl 64, id 46162, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xc709), 1979663466:1979664002(536) ack 2260258133 win 5840
10:05:23.591440 IP (tos 0x0, ttl 64, id 19889, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x11b2 (correct), 2260258133:2260258133(0) ack 1979650602 win 65535
10:05:23.591442 IP (tos 0x0, ttl 64, id 46163, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x6bcf), 1979664002:1979664538(536) ack 2260258133 win 5840
10:05:23.591443 IP (tos 0x0, ttl 64, id 19890, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0f9a (correct), 2260258133:2260258133(0) ack 1979651138 win 65535
10:05:23.591446 IP (tos 0x0, ttl 64, id 46164, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xc469), 1979664538:1979665610(1072) ack 2260258133 win 5840
10:05:23.591448 IP (tos 0x0, ttl 64, id 19891, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0d82 (correct), 2260258133:2260258133(0) ack 1979651674 win 65535
10:05:23.591450 IP (tos 0x0, ttl 64, id 46166, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xbd17), 1979665610:1979666146(536) ack 2260258133 win 5840
10:05:23.591452 IP (tos 0x0, ttl 64, id 19892, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0b6a (correct), 2260258133:2260258133(0) ack 1979652210 win 65535
10:05:23.591455 IP (tos 0x0, ttl 64, id 46167, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x5899), 1979666146:1979666682(536) ack 2260258133 win 5840
10:05:23.591456 IP (tos 0x0, ttl 64, id 19893, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0952 (correct), 2260258133:2260258133(0) ack 1979652746 win 65535
10:05:23.591458 IP (tos 0x0, ttl 64, id 46168, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xd662), 1979666682:1979667218(536) ack 2260258133 win 5840
10:05:23.591460 IP (tos 0x0, ttl 64, id 19894, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x073a (correct), 2260258133:2260258133(0) ack 1979653282 win 65535
10:05:23.591462 IP (tos 0x0, ttl 64, id 46169, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xc61f), 1979667218:1979667754(536) ack 2260258133 win 5840
10:05:23.591478 IP (tos 0x0, ttl 64, id 19895, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0522 (correct), 2260258133:2260258133(0) ack 1979653818 win 65535
10:05:23.591481 IP (tos 0x0, ttl 64, id 46170, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x85e0), 1979667754:1979668290(536) ack 2260258133 win 5840
10:05:23.591483 IP (tos 0x0, ttl 64, id 19896, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x030a (correct), 2260258133:2260258133(0) ack 1979654354 win 65535
10:05:23.591486 IP (tos 0x0, ttl 64, id 46171, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xaa37), 1979668290:1979668826(536) ack 2260258133 win 5840
10:05:23.591487 IP (tos 0x0, ttl 64, id 19897, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x00f2 (correct), 2260258133:2260258133(0) ack 1979654890 win 65535
10:05:23.591489 IP (tos 0x0, ttl 64, id 46172, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x01c9), 1979668826:1979669362(536) ack 2260258133 win 5840
10:05:23.591491 IP (tos 0x0, ttl 64, id 19898, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfed9 (correct), 2260258133:2260258133(0) ack 1979655426 win 65535
10:05:23.591494 IP (tos 0x0, ttl 64, id 46173, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xe9d9), 1979669362:1979669898(536) ack 2260258133 win 5840
10:05:23.591497 IP (tos 0x0, ttl 64, id 19899, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfcc1 (correct), 2260258133:2260258133(0) ack 1979655962 win 65535
10:05:23.591499 IP (tos 0x0, ttl 64, id 46174, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xea9c), 1979669898:1979670434(536) ack 2260258133 win 5840
10:05:23.591528 IP (tos 0x0, ttl 64, id 19900, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfaa9 (correct), 2260258133:2260258133(0) ack 1979656498 win 65535
10:05:23.591531 IP (tos 0x0, ttl 64, id 46175, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xeb5e), 1979670434:1979670970(536) ack 2260258133 win 5840
10:05:23.591533 IP (tos 0x0, ttl 64, id 19901, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf891 (correct), 2260258133:2260258133(0) ack 1979657034 win 65535
10:05:23.591535 IP (tos 0x0, ttl 64, id 46176, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x80a0), 1979670970:1979671506(536) ack 2260258133 win 5840
10:05:23.591536 IP (tos 0x0, ttl 64, id 19902, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf679 (correct), 2260258133:2260258133(0) ack 1979657570 win 65535
10:05:23.591539 IP (tos 0x0, ttl 64, id 46177, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x8520), 1979671506:1979672042(536) ack 2260258133 win 5840
10:05:23.591540 IP (tos 0x0, ttl 64, id 19903, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf461 (correct), 2260258133:2260258133(0) ack 1979658106 win 65535
10:05:23.591543 IP (tos 0x0, ttl 64, id 46178, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x52db), 1979672042:1979672578(536) ack 2260258133 win 5840
10:05:23.591544 IP (tos 0x0, ttl 64, id 19904, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf249 (correct), 2260258133:2260258133(0) ack 1979658642 win 65535
10:05:23.591546 IP (tos 0x0, ttl 64, id 46179, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x2fce), 1979672578:1979673650(1072) ack 2260258133 win 5840
10:05:23.591549 IP (tos 0x0, ttl 64, id 19905, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf031 (correct), 2260258133:2260258133(0) ack 1979659178 win 65535
10:05:23.591551 IP (tos 0x0, ttl 64, id 46181, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x7537), 1979673650:1979674186(536) ack 2260258133 win 5840
10:05:23.591553 IP (tos 0x0, ttl 64, id 19906, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xee19 (correct), 2260258133:2260258133(0) ack 1979659714 win 65535
10:05:23.591555 IP (tos 0x0, ttl 64, id 46182, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x5a7a), 1979674186:1979674722(536) ack 2260258133 win 5840
10:05:23.591556 IP (tos 0x0, ttl 64, id 19907, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xec01 (correct), 2260258133:2260258133(0) ack 1979660250 win 65535
10:05:23.591558 IP (tos 0x0, ttl 64, id 46183, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x8257), 1979674722:1979675258(536) ack 2260258133 win 5840
10:05:23.591578 IP (tos 0x0, ttl 64, id 19908, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe9e9 (correct), 2260258133:2260258133(0) ack 1979660786 win 65535
10:05:23.591582 IP (tos 0x0, ttl 64, id 46184, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x0c07), 1979675258:1979675794(536) ack 2260258133 win 5840
10:05:23.591583 IP (tos 0x0, ttl 64, id 19909, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe7d1 (correct), 2260258133:2260258133(0) ack 1979661322 win 65535
10:05:23.591586 IP (tos 0x0, ttl 64, id 46185, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x4cec), 1979675794:1979676330(536) ack 2260258133 win 5840
10:05:23.591587 IP (tos 0x0, ttl 64, id 19910, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe5b9 (correct), 2260258133:2260258133(0) ack 1979661858 win 65535
10:05:23.591589 IP (tos 0x0, ttl 64, id 46186, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x3599), 1979676330:1979676866(536) ack 2260258133 win 5840
10:05:23.591590 IP (tos 0x0, ttl 64, id 19911, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe3a1 (correct), 2260258133:2260258133(0) ack 1979662394 win 65535
10:05:23.591592 IP (tos 0x0, ttl 64, id 46187, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x8a1b), 1979676866:1979677402(536) ack 2260258133 win 5840
10:05:23.591593 IP (tos 0x0, ttl 64, id 19912, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe189 (correct), 2260258133:2260258133(0) ack 1979662930 win 65535
10:05:23.591596 IP (tos 0x0, ttl 64, id 46188, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x2335), 1979677402:1979677938(536) ack 2260258133 win 5840
10:05:23.591597 IP (tos 0x0, ttl 64, id 19913, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xdf71 (correct), 2260258133:2260258133(0) ack 1979663466 win 65535
10:05:23.591599 IP (tos 0x0, ttl 64, id 46189, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x2c5e), 1979677938:1979678474(536) ack 2260258133 win 5840
10:05:23.591601 IP (tos 0x0, ttl 64, id 19914, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xdd59 (correct), 2260258133:2260258133(0) ack 1979664002 win 65535
10:05:23.591604 IP (tos 0x0, ttl 64, id 46190, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xaa8b), 1979678474:1979679010(536) ack 2260258133 win 5840
10:05:23.591607 IP (tos 0x0, ttl 64, id 19915, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xdb41 (correct), 2260258133:2260258133(0) ack 1979664538 win 65535
10:05:23.591609 IP (tos 0x0, ttl 64, id 46191, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xd126), 1979679010:1979679546(536) ack 2260258133 win 5840
10:05:23.591611 IP (tos 0x0, ttl 64, id 19916, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd929 (correct), 2260258133:2260258133(0) ack 1979665074 win 65535
10:05:23.591614 IP (tos 0x0, ttl 64, id 46192, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xba76), 1979679546:1979680082(536) ack 2260258133 win 5840
10:05:23.591615 IP (tos 0x0, ttl 64, id 19917, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd711 (correct), 2260258133:2260258133(0) ack 1979665610 win 65535
10:05:23.591617 IP (tos 0x0, ttl 64, id 46193, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x774f), 1979680082:1979680618(536) ack 2260258133 win 5840
10:05:23.591678 IP (tos 0x0, ttl 64, id 19918, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd4f9 (correct), 2260258133:2260258133(0) ack 1979666146 win 65535
10:05:23.591681 IP (tos 0x0, ttl 64, id 46194, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x603a), 1979680618:1979681154(536) ack 2260258133 win 5840
10:05:23.591683 IP (tos 0x0, ttl 64, id 19919, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd2e1 (correct), 2260258133:2260258133(0) ack 1979666682 win 65535
10:05:23.591686 IP (tos 0x0, ttl 64, id 46195, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x544a), 1979681154:1979681690(536) ack 2260258133 win 5840
10:05:23.591687 IP (tos 0x0, ttl 64, id 19920, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd0c9 (correct), 2260258133:2260258133(0) ack 1979667218 win 65535
10:05:23.591689 IP (tos 0x0, ttl 64, id 46196, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xc45b), 1979681690:1979682226(536) ack 2260258133 win 5840
10:05:23.591690 IP (tos 0x0, ttl 64, id 19921, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xceb1 (correct), 2260258133:2260258133(0) ack 1979667754 win 65535
10:05:23.591693 IP (tos 0x0, ttl 64, id 46197, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xf8bc), 1979682226:1979682762(536) ack 2260258133 win 5840
10:05:23.591694 IP (tos 0x0, ttl 64, id 19922, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xcc99 (correct), 2260258133:2260258133(0) ack 1979668290 win 65535
10:05:23.591696 IP (tos 0x0, ttl 64, id 46198, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xf997), 1979682762:1979683298(536) ack 2260258133 win 5840
10:05:23.591697 IP (tos 0x0, ttl 64, id 19923, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xca81 (correct), 2260258133:2260258133(0) ack 1979668826 win 65535
10:05:23.591699 IP (tos 0x0, ttl 64, id 46199, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x4b6b), 1979683298:1979683834(536) ack 2260258133 win 5840
10:05:23.591701 IP (tos 0x0, ttl 64, id 19924, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc869 (correct), 2260258133:2260258133(0) ack 1979669362 win 65535
10:05:23.591703 IP (tos 0x0, ttl 64, id 46200, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x9b8b), 1979683834:1979684370(536) ack 2260258133 win 5840
10:05:23.591704 IP (tos 0x0, ttl 64, id 19925, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc651 (correct), 2260258133:2260258133(0) ack 1979669898 win 65535
10:05:23.591707 IP (tos 0x0, ttl 64, id 46201, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xd84e), 1979684370:1979684906(536) ack 2260258133 win 5840
10:05:23.591708 IP (tos 0x0, ttl 64, id 19926, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc439 (correct), 2260258133:2260258133(0) ack 1979670434 win 65535
10:05:23.591710 IP (tos 0x0, ttl 64, id 46202, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x2b48), 1979684906:1979685442(536) ack 2260258133 win 5840
10:05:23.591712 IP (tos 0x0, ttl 64, id 19927, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc221 (correct), 2260258133:2260258133(0) ack 1979670970 win 65535
10:05:23.591713 IP (tos 0x0, ttl 64, id 46203, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x5916), 1979685442:1979685978(536) ack 2260258133 win 5840
10:05:23.591715 IP (tos 0x0, ttl 64, id 19928, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc009 (correct), 2260258133:2260258133(0) ack 1979671506 win 65535
10:05:23.591717 IP (tos 0x0, ttl 64, id 46204, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xbe9a), 1979685978:1979686514(536) ack 2260258133 win 5840
10:05:23.591719 IP (tos 0x0, ttl 64, id 19929, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xbdf1 (correct), 2260258133:2260258133(0) ack 1979672042 win 65535
10:05:23.591721 IP (tos 0x0, ttl 64, id 46205, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xb3b0), 1979686514:1979687050(536) ack 2260258133 win 5840
10:05:23.591722 IP (tos 0x0, ttl 64, id 19930, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xbbd9 (correct), 2260258133:2260258133(0) ack 1979672578 win 65535
10:05:23.591724 IP (tos 0x0, ttl 64, id 46206, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xa518), 1979687050:1979688122(1072) ack 2260258133 win 5840
10:05:23.591726 IP (tos 0x0, ttl 64, id 19931, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb9c1 (correct), 2260258133:2260258133(0) ack 1979673114 win 65535
10:05:23.591729 IP (tos 0x0, ttl 64, id 46208, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x247f), 1979688122:1979688658(536) ack 2260258133 win 5840
10:05:23.591730 IP (tos 0x0, ttl 64, id 19932, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb7a9 (correct), 2260258133:2260258133(0) ack 1979673650 win 65535
10:05:23.591732 IP (tos 0x0, ttl 64, id 46209, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xdc70), 1979688658:1979689194(536) ack 2260258133 win 5840
10:05:23.591734 IP (tos 0x0, ttl 64, id 19933, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb591 (correct), 2260258133:2260258133(0) ack 1979674186 win 65535
10:05:23.591736 IP (tos 0x0, ttl 64, id 46210, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xc420), 1979689194:1979689730(536) ack 2260258133 win 5840
10:05:23.591739 IP (tos 0x0, ttl 64, id 19934, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb379 (correct), 2260258133:2260258133(0) ack 1979674722 win 65535
10:05:23.591741 IP (tos 0x0, ttl 64, id 46211, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x6b75), 1979689730:1979690266(536) ack 2260258133 win 5840
10:05:23.591743 IP (tos 0x0, ttl 64, id 19935, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb161 (correct), 2260258133:2260258133(0) ack 1979675258 win 65535
10:05:23.591745 IP (tos 0x0, ttl 64, id 46212, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x9f8c), 1979690266:1979690802(536) ack 2260258133 win 5840
10:05:23.591746 IP (tos 0x0, ttl 64, id 19936, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xaf49 (correct), 2260258133:2260258133(0) ack 1979675794 win 65535
10:05:23.591748 IP (tos 0x0, ttl 64, id 46213, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xf695), 1979690802:1979691338(536) ack 2260258133 win 5840
10:05:23.591752 IP (tos 0x0, ttl 64, id 19937, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xad31 (correct), 2260258133:2260258133(0) ack 1979676330 win 65535
10:05:23.591755 IP (tos 0x0, ttl 64, id 46214, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x0f0f), 1979691338:1979691874(536) ack 2260258133 win 5840
10:05:23.591777 IP (tos 0x0, ttl 64, id 19938, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xab19 (correct), 2260258133:2260258133(0) ack 1979676866 win 65535
10:05:23.591780 IP (tos 0x0, ttl 64, id 46215, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x3612), 1979691874:1979692410(536) ack 2260258133 win 5840
10:05:23.591782 IP (tos 0x0, ttl 64, id 19939, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa901 (correct), 2260258133:2260258133(0) ack 1979677402 win 65535
10:05:23.591784 IP (tos 0x0, ttl 64, id 46216, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xe8f0), 1979692410:1979692946(536) ack 2260258133 win 5840
10:05:23.591786 IP (tos 0x0, ttl 64, id 19940, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa6e9 (correct), 2260258133:2260258133(0) ack 1979677938 win 65535
10:05:23.591788 IP (tos 0x0, ttl 64, id 46217, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x7b58), 1979692946:1979693482(536) ack 2260258133 win 5840
10:05:23.591789 IP (tos 0x0, ttl 64, id 19941, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa4d1 (correct), 2260258133:2260258133(0) ack 1979678474 win 65535
10:05:23.591791 IP (tos 0x0, ttl 64, id 46218, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x4dcc), 1979693482:1979694018(536) ack 2260258133 win 5840
10:05:23.591792 IP (tos 0x0, ttl 64, id 19942, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa2b9 (correct), 2260258133:2260258133(0) ack 1979679010 win 65535
10:05:23.591794 IP (tos 0x0, ttl 64, id 46219, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xe1c4), 1979694018:1979694554(536) ack 2260258133 win 5840
10:05:23.591796 IP (tos 0x0, ttl 64, id 19943, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa0a1 (correct), 2260258133:2260258133(0) ack 1979679546 win 65535
10:05:23.591798 IP (tos 0x0, ttl 64, id 46220, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xba54), 1979694554:1979695090(536) ack 2260258133 win 5840
10:05:23.591800 IP (tos 0x0, ttl 64, id 19944, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9e89 (correct), 2260258133:2260258133(0) ack 1979680082 win 65535
10:05:23.591802 IP (tos 0x0, ttl 64, id 46221, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xf5a2), 1979695090:1979695626(536) ack 2260258133 win 5840
10:05:23.591803 IP (tos 0x0, ttl 64, id 19945, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9c71 (correct), 2260258133:2260258133(0) ack 1979680618 win 65535
10:05:23.591805 IP (tos 0x0, ttl 64, id 46222, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x490c), 1979695626:1979696162(536) ack 2260258133 win 5840
10:05:23.591807 IP (tos 0x0, ttl 64, id 19946, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9a59 (correct), 2260258133:2260258133(0) ack 1979681154 win 65535
10:05:23.591811 IP (tos 0x0, ttl 64, id 46223, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x5eac), 1979696162:1979696698(536) ack 2260258133 win 5840
10:05:23.591812 IP (tos 0x0, ttl 64, id 19947, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9841 (correct), 2260258133:2260258133(0) ack 1979681690 win 65535
10:05:23.591814 IP (tos 0x0, ttl 64, id 46224, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x08a7), 1979696698:1979697234(536) ack 2260258133 win 5840
10:05:23.591816 IP (tos 0x0, ttl 64, id 19948, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9629 (correct), 2260258133:2260258133(0) ack 1979682226 win 65535
10:05:23.591818 IP (tos 0x0, ttl 64, id 46225, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x6d23), 1979697234:1979697770(536) ack 2260258133 win 5840
10:05:23.591819 IP (tos 0x0, ttl 64, id 19949, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9411 (correct), 2260258133:2260258133(0) ack 1979682762 win 65535
10:05:23.591821 IP (tos 0x0, ttl 64, id 46226, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xc833), 1979697770:1979698306(536) ack 2260258133 win 5840
10:05:23.591823 IP (tos 0x0, ttl 64, id 19950, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x91f9 (correct), 2260258133:2260258133(0) ack 1979683298 win 65535
10:05:23.591825 IP (tos 0x0, ttl 64, id 46227, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x78e9), 1979698306:1979698842(536) ack 2260258133 win 5840
10:05:23.591827 IP (tos 0x0, ttl 64, id 19951, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8fe1 (correct), 2260258133:2260258133(0) ack 1979683834 win 65535
10:05:23.591829 IP (tos 0x0, ttl 64, id 46228, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xc1db), 1979698842:1979699378(536) ack 2260258133 win 5840
10:05:23.591831 IP (tos 0x0, ttl 64, id 19952, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8dc9 (correct), 2260258133:2260258133(0) ack 1979684370 win 65535
10:05:23.591833 IP (tos 0x0, ttl 64, id 46229, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xcd2a), 1979699378:1979699914(536) ack 2260258133 win 5840
10:05:23.591834 IP (tos 0x0, ttl 64, id 19953, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8bb1 (correct), 2260258133:2260258133(0) ack 1979684906 win 65535
10:05:23.591836 IP (tos 0x0, ttl 64, id 46230, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x347f), 1979699914:1979700450(536) ack 2260258133 win 5840
10:05:23.591839 IP (tos 0x0, ttl 64, id 19954, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8999 (correct), 2260258133:2260258133(0) ack 1979685442 win 65535
10:05:23.591842 IP (tos 0x0, ttl 64, id 19955, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8781 (correct), 2260258133:2260258133(0) ack 1979685978 win 65535
10:05:23.591846 IP (tos 0x0, ttl 64, id 19956, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8569 (correct), 2260258133:2260258133(0) ack 1979686514 win 65535
10:05:23.591851 IP (tos 0x0, ttl 64, id 19957, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8351 (correct), 2260258133:2260258133(0) ack 1979687050 win 65535
10:05:23.591929 IP (tos 0x0, ttl 64, id 19962, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x78d9 (correct), 2260258133:2260258133(0) ack 1979689730 win 65535
10:05:23.591932 IP (tos 0x0, ttl 64, id 46240, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x0a1e), 1979705274:1979705810(536) ack 2260258133 win 5840
10:05:23.591934 IP (tos 0x0, ttl 64, id 19963, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x76c1 (correct), 2260258133:2260258133(0) ack 1979690266 win 65535
10:05:23.591936 IP (tos 0x0, ttl 64, id 46241, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x6f73), 1979705810:1979706346(536) ack 2260258133 win 5840
10:05:23.591938 IP (tos 0x0, ttl 64, id 19964, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x74a9 (correct), 2260258133:2260258133(0) ack 1979690802 win 65535
10:05:23.591940 IP (tos 0x0, ttl 64, id 46242, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xd5d3), 1979706346:1979706882(536) ack 2260258133 win 5840
10:05:23.591941 IP (tos 0x0, ttl 64, id 19965, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7291 (correct), 2260258133:2260258133(0) ack 1979691338 win 65535
10:05:23.591943 IP (tos 0x0, ttl 64, id 46243, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x46e8), 1979706882:1979707418(536) ack 2260258133 win 5840
10:05:23.591944 IP (tos 0x0, ttl 64, id 19966, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7079 (correct), 2260258133:2260258133(0) ack 1979691874 win 65535
10:05:23.591946 IP (tos 0x0, ttl 64, id 46244, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xc9ee), 1979707418:1979707954(536) ack 2260258133 win 5840
10:05:23.591948 IP (tos 0x0, ttl 64, id 19967, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6e61 (correct), 2260258133:2260258133(0) ack 1979692410 win 65535
10:05:23.591950 IP (tos 0x0, ttl 64, id 46245, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x1cc0), 1979707954:1979708490(536) ack 2260258133 win 5840
10:05:23.591951 IP (tos 0x0, ttl 64, id 19968, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6c49 (correct), 2260258133:2260258133(0) ack 1979692946 win 65535
10:05:23.592525 IP (tos 0x0, ttl 64, id 19969, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6a31 (correct), 2260258133:2260258133(0) ack 1979693482 win 65535
10:05:23.592528 IP (tos 0x0, ttl 64, id 46247, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x5666), 1979709026:1979709562(536) ack 2260258133 win 5840
10:05:23.592530 IP (tos 0x0, ttl 64, id 19970, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6819 (correct), 2260258133:2260258133(0) ack 1979694018 win 65535
10:05:23.592532 IP (tos 0x0, ttl 64, id 46248, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: P, cksum 0x1855 (incorrect (-> 0x8fb5), 1979709562:1979710098(536) ack 2260258133 win 5840
10:05:23.592533 IP (tos 0x0, ttl 64, id 19971, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6601 (correct), 2260258133:2260258133(0) ack 1979694554 win 65535
10:05:23.592536 IP (tos 0x0, ttl 64, id 46249, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xbea5), 1979710098:1979710634(536) ack 2260258133 win 5840
10:05:23.592538 IP (tos 0x0, ttl 64, id 19972, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x63e9 (correct), 2260258133:2260258133(0) ack 1979695090 win 65535
10:05:23.592540 IP (tos 0x0, ttl 64, id 46250, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x4592), 1979710634:1979711170(536) ack 2260258133 win 5840
10:05:23.592541 IP (tos 0x0, ttl 64, id 19973, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x61d1 (correct), 2260258133:2260258133(0) ack 1979695626 win 65535
10:05:23.592543 IP (tos 0x0, ttl 64, id 46251, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x850b), 1979711170:1979711706(536) ack 2260258133 win 5840
10:05:23.592544 IP (tos 0x0, ttl 64, id 19974, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5fb9 (correct), 2260258133:2260258133(0) ack 1979696162 win 65535
10:05:23.592546 IP (tos 0x0, ttl 64, id 46252, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xb083), 1979711706:1979712242(536) ack 2260258133 win 5840
10:05:23.592547 IP (tos 0x0, ttl 64, id 19975, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5da1 (correct), 2260258133:2260258133(0) ack 1979696698 win 65535
10:05:23.592549 IP (tos 0x0, ttl 64, id 46253, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x2a6c), 1979712242:1979712778(536) ack 2260258133 win 5840
10:05:23.592551 IP (tos 0x0, ttl 64, id 19976, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5b89 (correct), 2260258133:2260258133(0) ack 1979697234 win 65535
10:05:23.592553 IP (tos 0x0, ttl 64, id 46254, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x48bc), 1979712778:1979713314(536) ack 2260258133 win 5840
10:05:23.592554 IP (tos 0x0, ttl 64, id 19977, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5971 (correct), 2260258133:2260258133(0) ack 1979697770 win 65535
10:05:23.592557 IP (tos 0x0, ttl 64, id 46255, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x12b7), 1979713314:1979713850(536) ack 2260258133 win 5840
10:05:23.592558 IP (tos 0x0, ttl 64, id 19978, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5759 (correct), 2260258133:2260258133(0) ack 1979698306 win 65535
10:05:23.592561 IP (tos 0x0, ttl 64, id 46256, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x94da), 1979713850:1979714386(536) ack 2260258133 win 5840
10:05:23.592562 IP (tos 0x0, ttl 64, id 19979, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5541 (correct), 2260258133:2260258133(0) ack 1979698842 win 65535
10:05:23.592564 IP (tos 0x0, ttl 64, id 46257, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x7562), 1979714386:1979714922(536) ack 2260258133 win 5840
10:05:23.592565 IP (tos 0x0, ttl 64, id 19980, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5329 (correct), 2260258133:2260258133(0) ack 1979699378 win 65535
10:05:23.592568 IP (tos 0x0, ttl 64, id 46258, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x7053), 1979714922:1979715458(536) ack 2260258133 win 5840
10:05:23.592569 IP (tos 0x0, ttl 64, id 19981, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5111 (correct), 2260258133:2260258133(0) ack 1979699914 win 65535
10:05:23.592571 IP (tos 0x0, ttl 64, id 46259, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x227a), 1979715458:1979715994(536) ack 2260258133 win 5840
10:05:23.592572 IP (tos 0x0, ttl 64, id 19982, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4ef9 (correct), 2260258133:2260258133(0) ack 1979700450 win 65535
10:05:23.592574 IP (tos 0x0, ttl 64, id 46260, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x6540), 1979715994:1979717066(1072) ack 2260258133 win 5840
10:05:23.592576 IP (tos 0x0, ttl 64, id 19983, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4ce1 (correct), 2260258133:2260258133(0) ack 1979700986 win 65535
10:05:23.592579 IP (tos 0x0, ttl 64, id 46262, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xb7e1), 1979717066:1979717602(536) ack 2260258133 win 5840
10:05:23.592580 IP (tos 0x0, ttl 64, id 19984, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4ac9 (correct), 2260258133:2260258133(0) ack 1979701522 win 65535
10:05:23.592583 IP (tos 0x0, ttl 64, id 46263, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xa084), 1979717602:1979718138(536) ack 2260258133 win 5840
10:05:23.592585 IP (tos 0x0, ttl 64, id 19985, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x48b1 (correct), 2260258133:2260258133(0) ack 1979702058 win 65535
10:05:23.592587 IP (tos 0x0, ttl 64, id 46264, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x364b), 1979718138:1979718674(536) ack 2260258133 win 5840
10:05:23.592589 IP (tos 0x0, ttl 64, id 19986, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4699 (correct), 2260258133:2260258133(0) ack 1979702594 win 65535
10:05:23.592591 IP (tos 0x0, ttl 64, id 46265, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x3872), 1979718674:1979719210(536) ack 2260258133 win 5840
10:05:23.592592 IP (tos 0x0, ttl 64, id 19987, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4481 (correct), 2260258133:2260258133(0) ack 1979703130 win 65535
10:05:23.592595 IP (tos 0x0, ttl 64, id 46266, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x2200), 1979719210:1979719746(536) ack 2260258133 win 5840
10:05:23.592596 IP (tos 0x0, ttl 64, id 19988, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4269 (correct), 2260258133:2260258133(0) ack 1979703666 win 65535
10:05:23.592598 IP (tos 0x0, ttl 64, id 46267, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x0837), 1979719746:1979720282(536) ack 2260258133 win 5840
10:05:23.592599 IP (tos 0x0, ttl 64, id 19989, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3e39 (correct), 2260258133:2260258133(0) ack 1979704738 win 65535
10:05:23.592601 IP (tos 0x0, ttl 64, id 46268, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xd07f), 1979720282:1979721354(1072) ack 2260258133 win 5840
10:05:23.592604 IP (tos 0x0, ttl 64, id 19990, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3a09 (correct), 2260258133:2260258133(0) ack 1979705810 win 65535
10:05:23.592606 IP (tos 0x0, ttl 64, id 46270, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x3c59), 1979721354:1979722426(1072) ack 2260258133 win 5840
10:05:23.592608 IP (tos 0x0, ttl 64, id 19991, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x35d9 (correct), 2260258133:2260258133(0) ack 1979706882 win 65535
10:05:23.592611 IP (tos 0x0, ttl 64, id 46272, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x6f49), 1979722426:1979723498(1072) ack 2260258133 win 5840
10:05:23.592613 IP (tos 0x0, ttl 64, id 19992, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x31a9 (correct), 2260258133:2260258133(0) ack 1979707954 win 65535
10:05:23.592615 IP (tos 0x0, ttl 64, id 46274, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xf4f1), 1979723498:1979724570(1072) ack 2260258133 win 5840
10:05:23.592628 IP (tos 0x0, ttl 64, id 19993, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2d79 (correct), 2260258133:2260258133(0) ack 1979709026 win 65535
10:05:23.592631 IP (tos 0x0, ttl 64, id 46276, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xcdd6), 1979724570:1979725642(1072) ack 2260258133 win 5840
10:05:23.592633 IP (tos 0x0, ttl 64, id 19994, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2949 (correct), 2260258133:2260258133(0) ack 1979710098 win 65535
10:05:23.592635 IP (tos 0x0, ttl 64, id 46278, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x2b34), 1979725642:1979726714(1072) ack 2260258133 win 5840
10:05:23.592637 IP (tos 0x0, ttl 64, id 19995, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2519 (correct), 2260258133:2260258133(0) ack 1979711170 win 65535
10:05:23.592639 IP (tos 0x0, ttl 64, id 46280, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x5e7b), 1979726714:1979727786(1072) ack 2260258133 win 5840
10:05:23.592641 IP (tos 0x0, ttl 64, id 19996, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x20e9 (correct), 2260258133:2260258133(0) ack 1979712242 win 65535
10:05:23.592643 IP (tos 0x0, ttl 64, id 46282, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x20e4), 1979727786:1979728858(1072) ack 2260258133 win 5840
10:05:23.592645 IP (tos 0x0, ttl 64, id 19997, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1cb9 (correct), 2260258133:2260258133(0) ack 1979713314 win 65535
10:05:23.592648 IP (tos 0x0, ttl 64, id 46284, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x4d20), 1979728858:1979729930(1072) ack 2260258133 win 5840
10:05:23.592650 IP (tos 0x0, ttl 64, id 19998, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1889 (correct), 2260258133:2260258133(0) ack 1979714386 win 65535
10:05:23.592652 IP (tos 0x0, ttl 64, id 46286, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x4346), 1979729930:1979731002(1072) ack 2260258133 win 5840
10:05:23.592654 IP (tos 0x0, ttl 64, id 19999, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1459 (correct), 2260258133:2260258133(0) ack 1979715458 win 65535
10:05:23.592656 IP (tos 0x0, ttl 64, id 46288, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xfadd), 1979731002:1979732074(1072) ack 2260258133 win 5840
10:05:23.592660 IP (tos 0x0, ttl 64, id 20000, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1029 (correct), 2260258133:2260258133(0) ack 1979716530 win 65535
10:05:23.592663 IP (tos 0x0, ttl 64, id 46290, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x622e), 1979732074:1979733146(1072) ack 2260258133 win 5840
10:05:23.592675 IP (tos 0x0, ttl 64, id 20001, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0bf9 (correct), 2260258133:2260258133(0) ack 1979717602 win 65535
10:05:23.592678 IP (tos 0x0, ttl 64, id 46292, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x87ff), 1979733146:1979734218(1072) ack 2260258133 win 5840
10:05:23.592681 IP (tos 0x0, ttl 64, id 20002, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x07c9 (correct), 2260258133:2260258133(0) ack 1979718674 win 65535
10:05:23.592683 IP (tos 0x0, ttl 64, id 46294, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x8c01), 1979734218:1979735290(1072) ack 2260258133 win 5840
10:05:23.592685 IP (tos 0x0, ttl 64, id 20003, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0399 (correct), 2260258133:2260258133(0) ack 1979719746 win 65535
10:05:23.592687 IP (tos 0x0, ttl 64, id 46296, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xc229), 1979735290:1979736362(1072) ack 2260258133 win 5840
10:05:23.592726 IP (tos 0x0, ttl 64, id 20004, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xff68 (correct), 2260258133:2260258133(0) ack 1979720818 win 65535
10:05:23.592729 IP (tos 0x0, ttl 64, id 46298, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x3c79), 1979736362:1979737434(1072) ack 2260258133 win 5840
10:05:23.592731 IP (tos 0x0, ttl 64, id 20005, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfb38 (correct), 2260258133:2260258133(0) ack 1979721890 win 65535
10:05:23.592734 IP (tos 0x0, ttl 64, id 46300, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x4d00), 1979737434:1979738506(1072) ack 2260258133 win 5840
10:05:23.592736 IP (tos 0x0, ttl 64, id 20006, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf708 (correct), 2260258133:2260258133(0) ack 1979722962 win 65535
10:05:23.592738 IP (tos 0x0, ttl 64, id 46302, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xc88b), 1979738506:1979739578(1072) ack 2260258133 win 5840
10:05:23.592740 IP (tos 0x0, ttl 64, id 20007, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf2d8 (correct), 2260258133:2260258133(0) ack 1979724034 win 65535
10:05:23.592743 IP (tos 0x0, ttl 64, id 46304, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x252a), 1979739578:1979740650(1072) ack 2260258133 win 5840
10:05:23.592776 IP (tos 0x0, ttl 64, id 20008, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xeea8 (correct), 2260258133:2260258133(0) ack 1979725106 win 65535
10:05:23.592780 IP (tos 0x0, ttl 64, id 46306, offset 0, flags [DF], proto TCP (6), length 1648) dione.40344 > phoenix.distcc: ., cksum 0x1c85 (incorrect (-> 0xe96f), 1979740650:1979742258(1608) ack 2260258133 win 5840
10:05:23.592782 IP (tos 0x0, ttl 64, id 20009, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xea78 (correct), 2260258133:2260258133(0) ack 1979726178 win 65535
10:05:23.592784 IP (tos 0x0, ttl 64, id 46309, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x41ec), 1979742258:1979742794(536) ack 2260258133 win 5840
10:05:23.592786 IP (tos 0x0, ttl 64, id 46310, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x7642), 1979742794:1979743330(536) ack 2260258133 win 5840
10:05:23.592787 IP (tos 0x0, ttl 64, id 20010, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe648 (correct), 2260258133:2260258133(0) ack 1979727250 win 65535
10:05:23.592790 IP (tos 0x0, ttl 64, id 46311, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x7309), 1979743330:1979744402(1072) ack 2260258133 win 5840
10:05:23.592792 IP (tos 0x0, ttl 64, id 20011, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe218 (correct), 2260258133:2260258133(0) ack 1979728322 win 65535
10:05:23.592795 IP (tos 0x0, ttl 64, id 46313, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xde30), 1979744402:1979745474(1072) ack 2260258133 win 5840
10:05:23.592825 IP (tos 0x0, ttl 64, id 20012, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xdde8 (correct), 2260258133:2260258133(0) ack 1979729394 win 65535
10:05:23.592829 IP (tos 0x0, ttl 64, id 46315, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xd0c5), 1979745474:1979746546(1072) ack 2260258133 win 5840
10:05:23.592831 IP (tos 0x0, ttl 64, id 20013, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd9b8 (correct), 2260258133:2260258133(0) ack 1979730466 win 65535
10:05:23.592834 IP (tos 0x0, ttl 64, id 46317, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x0bde), 1979746546:1979747618(1072) ack 2260258133 win 5840
10:05:23.592836 IP (tos 0x0, ttl 64, id 20014, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd588 (correct), 2260258133:2260258133(0) ack 1979731538 win 65535
10:05:23.592839 IP (tos 0x0, ttl 64, id 46319, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xbf84), 1979747618:1979748690(1072) ack 2260258133 win 5840
10:05:23.592875 IP (tos 0x0, ttl 64, id 20015, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd158 (correct), 2260258133:2260258133(0) ack 1979732610 win 65535
10:05:23.592879 IP (tos 0x0, ttl 64, id 46321, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xf352), 1979748690:1979749762(1072) ack 2260258133 win 5840
10:05:23.592881 IP (tos 0x0, ttl 64, id 20016, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xcd28 (correct), 2260258133:2260258133(0) ack 1979733682 win 65535
10:05:23.592884 IP (tos 0x0, ttl 64, id 46323, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x4bd9), 1979749762:1979750834(1072) ack 2260258133 win 5840
10:05:23.592886 IP (tos 0x0, ttl 64, id 20017, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc8f8 (correct), 2260258133:2260258133(0) ack 1979734754 win 65535
10:05:23.592889 IP (tos 0x0, ttl 64, id 46325, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x5806), 1979750834:1979751906(1072) ack 2260258133 win 5840
10:05:23.592926 IP (tos 0x0, ttl 64, id 20018, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc4c8 (correct), 2260258133:2260258133(0) ack 1979735826 win 65535
10:05:23.592930 IP (tos 0x0, ttl 64, id 46327, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x2dda), 1979751906:1979752978(1072) ack 2260258133 win 5840
10:05:23.592932 IP (tos 0x0, ttl 64, id 20019, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc098 (correct), 2260258133:2260258133(0) ack 1979736898 win 65535
10:05:23.592935 IP (tos 0x0, ttl 64, id 46329, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x0534), 1979752978:1979754050(1072) ack 2260258133 win 5840
10:05:23.592938 IP (tos 0x0, ttl 64, id 20020, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xbc68 (correct), 2260258133:2260258133(0) ack 1979737970 win 65535
10:05:23.592940 IP (tos 0x0, ttl 64, id 46331, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xe89d), 1979754050:1979755122(1072) ack 2260258133 win 5840
10:05:23.592975 IP (tos 0x0, ttl 64, id 20021, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb838 (correct), 2260258133:2260258133(0) ack 1979739042 win 65535
10:05:23.592978 IP (tos 0x0, ttl 64, id 46333, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x58cb), 1979755122:1979756194(1072) ack 2260258133 win 5840
10:05:23.592980 IP (tos 0x0, ttl 64, id 20022, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb408 (correct), 2260258133:2260258133(0) ack 1979740114 win 65535
10:05:23.592983 IP (tos 0x0, ttl 64, id 46335, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x5bf2), 1979756194:1979757266(1072) ack 2260258133 win 5840
10:05:23.592985 IP (tos 0x0, ttl 64, id 20023, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xafd8 (correct), 2260258133:2260258133(0) ack 1979741186 win 65535
10:05:23.592988 IP (tos 0x0, ttl 64, id 46337, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x4592), 1979757266:1979758338(1072) ack 2260258133 win 5840
10:05:23.592990 IP (tos 0x0, ttl 64, id 20024, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xaba8 (correct), 2260258133:2260258133(0) ack 1979742258 win 65535
10:05:23.592993 IP (tos 0x0, ttl 64, id 46339, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x6c1b), 1979758338:1979759410(1072) ack 2260258133 win 5840
10:05:23.592996 IP (tos 0x0, ttl 64, id 20025, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa778 (correct), 2260258133:2260258133(0) ack 1979743330 win 65535
10:05:23.592999 IP (tos 0x0, ttl 64, id 46341, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x1f54), 1979759410:1979760482(1072) ack 2260258133 win 5840
10:05:23.593025 IP (tos 0x0, ttl 64, id 20026, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa348 (correct), 2260258133:2260258133(0) ack 1979744402 win 65535
10:05:23.593028 IP (tos 0x0, ttl 64, id 46343, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xdab7), 1979760482:1979761554(1072) ack 2260258133 win 5840
10:05:23.593031 IP (tos 0x0, ttl 64, id 20027, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9f18 (correct), 2260258133:2260258133(0) ack 1979745474 win 65535
10:05:23.593033 IP (tos 0x0, ttl 64, id 46345, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x45d7), 1979761554:1979762626(1072) ack 2260258133 win 5840
10:05:23.593075 IP (tos 0x0, ttl 64, id 20028, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9ae8 (correct), 2260258133:2260258133(0) ack 1979746546 win 65535
10:05:23.593078 IP (tos 0x0, ttl 64, id 46347, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xd943), 1979762626:1979763698(1072) ack 2260258133 win 5840
10:05:23.593081 IP (tos 0x0, ttl 64, id 20029, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x96b8 (correct), 2260258133:2260258133(0) ack 1979747618 win 65535
10:05:23.593083 IP (tos 0x0, ttl 64, id 46349, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x93b7), 1979763698:1979764770(1072) ack 2260258133 win 5840
10:05:23.593126 IP (tos 0x0, ttl 64, id 20030, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9288 (correct), 2260258133:2260258133(0) ack 1979748690 win 65535
10:05:23.593129 IP (tos 0x0, ttl 64, id 46351, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xcddd), 1979764770:1979765842(1072) ack 2260258133 win 5840
10:05:23.593131 IP (tos 0x0, ttl 64, id 20031, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8e58 (correct), 2260258133:2260258133(0) ack 1979749762 win 65535
10:05:23.593133 IP (tos 0x0, ttl 64, id 46353, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x4597), 1979765842:1979766914(1072) ack 2260258133 win 5840
10:05:23.593136 IP (tos 0x0, ttl 64, id 20032, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8a28 (correct), 2260258133:2260258133(0) ack 1979750834 win 65535
10:05:23.593138 IP (tos 0x0, ttl 64, id 46355, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x7032), 1979766914:1979767986(1072) ack 2260258133 win 5840
10:05:23.593140 IP (tos 0x0, ttl 64, id 20033, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x85f8 (correct), 2260258133:2260258133(0) ack 1979751906 win 65535
10:05:23.593142 IP (tos 0x0, ttl 64, id 46357, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xe4c4), 1979767986:1979769058(1072) ack 2260258133 win 5840
10:05:23.593144 IP (tos 0x0, ttl 64, id 20034, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x81c8 (correct), 2260258133:2260258133(0) ack 1979752978 win 65535
10:05:23.593146 IP (tos 0x0, ttl 64, id 46359, offset 0, flags [DF], proto TCP (6), length 1648) dione.40344 > phoenix.distcc: ., cksum 0x1c85 (incorrect (-> 0x3453), 1979769058:1979770666(1608) ack 2260258133 win 5840
10:05:23.593175 IP (tos 0x0, ttl 64, id 20035, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7d98 (correct), 2260258133:2260258133(0) ack 1979754050 win 65535
10:05:23.593178 IP (tos 0x0, ttl 64, id 46362, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x63e6), 1979770666:1979771738(1072) ack 2260258133 win 5840
10:05:23.593180 IP (tos 0x0, ttl 64, id 20036, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7968 (correct), 2260258133:2260258133(0) ack 1979755122 win 65535
10:05:23.593182 IP (tos 0x0, ttl 64, id 46364, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xdfe0), 1979771738:1979772810(1072) ack 2260258133 win 5840
10:05:23.593184 IP (tos 0x0, ttl 64, id 20037, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7538 (correct), 2260258133:2260258133(0) ack 1979756194 win 65535
10:05:23.593186 IP (tos 0x0, ttl 64, id 46366, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x02c0), 1979772810:1979773882(1072) ack 2260258133 win 5840
10:05:23.593224 IP (tos 0x0, ttl 64, id 20038, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7108 (correct), 2260258133:2260258133(0) ack 1979757266 win 65535
10:05:23.593228 IP (tos 0x0, ttl 64, id 46368, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x993d), 1979773882:1979774954(1072) ack 2260258133 win 5840
10:05:23.593230 IP (tos 0x0, ttl 64, id 20039, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6cd8 (correct), 2260258133:2260258133(0) ack 1979758338 win 65535
10:05:23.593232 IP (tos 0x0, ttl 64, id 46370, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x3a5f), 1979774954:1979775490(536) ack 2260258133 win 5840
10:05:23.593234 IP (tos 0x0, ttl 64, id 46371, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x8d5f), 1979775490:1979776026(536) ack 2260258133 win 5840
10:05:23.593236 IP (tos 0x0, ttl 64, id 20040, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x68a8 (correct), 2260258133:2260258133(0) ack 1979759410 win 65535
10:05:23.593238 IP (tos 0x0, ttl 64, id 46372, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x3d19), 1979776026:1979777098(1072) ack 2260258133 win 5840
10:05:23.593241 IP (tos 0x0, ttl 64, id 20041, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6478 (correct), 2260258133:2260258133(0) ack 1979760482 win 65535
10:05:23.593243 IP (tos 0x0, ttl 64, id 46374, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x30aa), 1979777098:1979778170(1072) ack 2260258133 win 5840
10:05:23.593276 IP (tos 0x0, ttl 64, id 20042, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6048 (correct), 2260258133:2260258133(0) ack 1979761554 win 65535
10:05:23.593279 IP (tos 0x0, ttl 64, id 46376, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x45a2), 1979778170:1979779242(1072) ack 2260258133 win 5840
10:05:23.593282 IP (tos 0x0, ttl 64, id 20043, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5c18 (correct), 2260258133:2260258133(0) ack 1979762626 win 65535
10:05:23.593284 IP (tos 0x0, ttl 64, id 46378, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x24fe), 1979779242:1979780314(1072) ack 2260258133 win 5840
10:05:23.593286 IP (tos 0x0, ttl 64, id 20044, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x57e8 (correct), 2260258133:2260258133(0) ack 1979763698 win 65535
10:05:23.593289 IP (tos 0x0, ttl 64, id 46380, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x747e), 1979780314:1979781386(1072) ack 2260258133 win 5840
10:05:23.593324 IP (tos 0x0, ttl 64, id 20045, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x53b8 (correct), 2260258133:2260258133(0) ack 1979764770 win 65535
10:05:23.593327 IP (tos 0x0, ttl 64, id 46382, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x7b2c), 1979781386:1979782458(1072) ack 2260258133 win 5840
10:05:23.593329 IP (tos 0x0, ttl 64, id 20046, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4f88 (correct), 2260258133:2260258133(0) ack 1979765842 win 65535
10:05:23.593332 IP (tos 0x0, ttl 64, id 46384, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xe8d3), 1979782458:1979783530(1072) ack 2260258133 win 5840
10:05:23.593334 IP (tos 0x0, ttl 64, id 20047, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4b58 (correct), 2260258133:2260258133(0) ack 1979766914 win 65535
10:05:23.593336 IP (tos 0x0, ttl 64, id 46386, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x007d), 1979783530:1979784602(1072) ack 2260258133 win 5840
10:05:23.593375 IP (tos 0x0, ttl 64, id 20048, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4728 (correct), 2260258133:2260258133(0) ack 1979767986 win 65535
10:05:23.593378 IP (tos 0x0, ttl 64, id 46388, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xb086), 1979784602:1979785674(1072) ack 2260258133 win 5840
10:05:23.593380 IP (tos 0x0, ttl 64, id 20049, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x42f8 (correct), 2260258133:2260258133(0) ack 1979769058 win 65535
10:05:23.593382 IP (tos 0x0, ttl 64, id 46390, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xaf52), 1979785674:1979786746(1072) ack 2260258133 win 5840
10:05:23.593384 IP (tos 0x0, ttl 64, id 20050, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3ec8 (correct), 2260258133:2260258133(0) ack 1979770130 win 65535
10:05:23.593387 IP (tos 0x0, ttl 64, id 46392, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x0e0b), 1979786746:1979787818(1072) ack 2260258133 win 5840
10:05:23.593388 IP (tos 0x0, ttl 64, id 20051, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3a98 (correct), 2260258133:2260258133(0) ack 1979771202 win 65535
10:05:23.593391 IP (tos 0x0, ttl 64, id 46394, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x7680), 1979787818:1979788890(1072) ack 2260258133 win 5840
10:05:23.593425 IP (tos 0x0, ttl 64, id 20052, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3668 (correct), 2260258133:2260258133(0) ack 1979772274 win 65535
10:05:23.593429 IP (tos 0x0, ttl 64, id 46396, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x329f), 1979788890:1979789962(1072) ack 2260258133 win 5840
10:05:23.593431 IP (tos 0x0, ttl 64, id 20053, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3238 (correct), 2260258133:2260258133(0) ack 1979773346 win 65535
10:05:23.593434 IP (tos 0x0, ttl 64, id 46398, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x1cf9), 1979789962:1979791034(1072) ack 2260258133 win 5840
10:05:23.593436 IP (tos 0x0, ttl 64, id 20054, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2e08 (correct), 2260258133:2260258133(0) ack 1979774418 win 65535
10:05:23.593438 IP (tos 0x0, ttl 64, id 46400, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x19f5), 1979791034:1979792106(1072) ack 2260258133 win 5840
10:05:23.593475 IP (tos 0x0, ttl 64, id 20055, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x29d8 (correct), 2260258133:2260258133(0) ack 1979775490 win 65535
10:05:23.593478 IP (tos 0x0, ttl 64, id 46402, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xb0fd), 1979792106:1979793178(1072) ack 2260258133 win 5840
10:05:23.593480 IP (tos 0x0, ttl 64, id 20056, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x25a8 (correct), 2260258133:2260258133(0) ack 1979776562 win 65535
10:05:23.593483 IP (tos 0x0, ttl 64, id 46404, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xb5e2), 1979793178:1979794250(1072) ack 2260258133 win 5840
10:05:23.593485 IP (tos 0x0, ttl 64, id 20057, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2178 (correct), 2260258133:2260258133(0) ack 1979777634 win 65535
10:05:23.593487 IP (tos 0x0, ttl 64, id 46406, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xe950), 1979794250:1979795322(1072) ack 2260258133 win 5840
10:05:23.593524 IP (tos 0x0, ttl 64, id 20058, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1d48 (correct), 2260258133:2260258133(0) ack 1979778706 win 65535
10:05:23.593527 IP (tos 0x0, ttl 64, id 46408, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x9434), 1979795322:1979796394(1072) ack 2260258133 win 5840
10:05:23.593529 IP (tos 0x0, ttl 64, id 20059, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1918 (correct), 2260258133:2260258133(0) ack 1979779778 win 65535
10:05:23.593533 IP (tos 0x0, ttl 64, id 46410, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xb348), 1979796394:1979797466(1072) ack 2260258133 win 5840
10:05:23.593535 IP (tos 0x0, ttl 64, id 20060, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x14e8 (correct), 2260258133:2260258133(0) ack 1979780850 win 65535
10:05:23.593537 IP (tos 0x0, ttl 64, id 46412, offset 0, flags [DF], proto TCP (6), length 1648) dione.40344 > phoenix.distcc: ., cksum 0x1c85 (incorrect (-> 0xeb0a), 1979797466:1979799074(1608) ack 2260258133 win 5840
10:05:23.593539 IP (tos 0x0, ttl 64, id 20061, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x10b8 (correct), 2260258133:2260258133(0) ack 1979781922 win 65535
10:05:23.593542 IP (tos 0x0, ttl 64, id 46415, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x6ed1), 1979799074:1979800146(1072) ack 2260258133 win 5840
10:05:23.593574 IP (tos 0x0, ttl 64, id 20062, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0c88 (correct), 2260258133:2260258133(0) ack 1979782994 win 65535
10:05:23.593577 IP (tos 0x0, ttl 64, id 46417, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xed63), 1979800146:1979801218(1072) ack 2260258133 win 5840
10:05:23.593580 IP (tos 0x0, ttl 64, id 20063, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0858 (correct), 2260258133:2260258133(0) ack 1979784066 win 65535
10:05:23.593583 IP (tos 0x0, ttl 64, id 46419, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x2fc8), 1979801218:1979802290(1072) ack 2260258133 win 5840
10:05:23.593585 IP (tos 0x0, ttl 64, id 20064, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0428 (correct), 2260258133:2260258133(0) ack 1979785138 win 65535
10:05:23.593588 IP (tos 0x0, ttl 64, id 46421, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xbdec), 1979802290:1979803362(1072) ack 2260258133 win 5840
10:05:23.593625 IP (tos 0x0, ttl 64, id 20065, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfff7 (correct), 2260258133:2260258133(0) ack 1979786210 win 65535
10:05:23.593629 IP (tos 0x0, ttl 64, id 46423, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x8d89), 1979803362:1979804434(1072) ack 2260258133 win 5840
10:05:23.593631 IP (tos 0x0, ttl 64, id 20066, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfbc7 (correct), 2260258133:2260258133(0) ack 1979787282 win 65535
10:05:23.593634 IP (tos 0x0, ttl 64, id 46425, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xf8a1), 1979804434:1979805506(1072) ack 2260258133 win 5840
10:05:23.593636 IP (tos 0x0, ttl 64, id 20067, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf797 (correct), 2260258133:2260258133(0) ack 1979788354 win 65535
10:05:23.593639 IP (tos 0x0, ttl 64, id 46427, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xe481), 1979805506:1979806578(1072) ack 2260258133 win 5840
10:05:23.593674 IP (tos 0x0, ttl 64, id 20068, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf367 (correct), 2260258133:2260258133(0) ack 1979789426 win 65535
10:05:23.593677 IP (tos 0x0, ttl 64, id 46429, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xcd5e), 1979806578:1979807650(1072) ack 2260258133 win 5840
10:05:23.593680 IP (tos 0x0, ttl 64, id 20069, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xef37 (correct), 2260258133:2260258133(0) ack 1979790498 win 65535
10:05:23.593682 IP (tos 0x0, ttl 64, id 46431, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: P, cksum 0x1855 (incorrect (-> 0x2e6b), 1979807650:1979808186(536) ack 2260258133 win 5840
10:05:23.593684 IP (tos 0x0, ttl 64, id 46432, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x614f), 1979808186:1979808722(536) ack 2260258133 win 5840
10:05:23.593687 IP (tos 0x0, ttl 64, id 20070, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xeb07 (correct), 2260258133:2260258133(0) ack 1979791570 win 65535
10:05:23.593690 IP (tos 0x0, ttl 64, id 46433, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xcf1f), 1979808722:1979809794(1072) ack 2260258133 win 5840
10:05:23.593692 IP (tos 0x0, ttl 64, id 20071, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe6d7 (correct), 2260258133:2260258133(0) ack 1979792642 win 65535
10:05:23.593695 IP (tos 0x0, ttl 64, id 46435, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x77bb), 1979809794:1979810866(1072) ack 2260258133 win 5840
10:05:23.593724 IP (tos 0x0, ttl 64, id 20072, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe2a7 (correct), 2260258133:2260258133(0) ack 1979793714 win 65535
10:05:23.593727 IP (tos 0x0, ttl 64, id 46437, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x6107), 1979810866:1979811938(1072) ack 2260258133 win 5840
10:05:23.593730 IP (tos 0x0, ttl 64, id 20073, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xde77 (correct), 2260258133:2260258133(0) ack 1979794786 win 65535
10:05:23.593733 IP (tos 0x0, ttl 64, id 46439, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x241a), 1979811938:1979813010(1072) ack 2260258133 win 5840
10:05:23.593735 IP (tos 0x0, ttl 64, id 20074, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xda47 (correct), 2260258133:2260258133(0) ack 1979795858 win 65535
10:05:23.593738 IP (tos 0x0, ttl 64, id 46441, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x388b), 1979813010:1979814082(1072) ack 2260258133 win 5840
10:05:23.593774 IP (tos 0x0, ttl 64, id 20075, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd617 (correct), 2260258133:2260258133(0) ack 1979796930 win 65535
10:05:23.593778 IP (tos 0x0, ttl 64, id 46443, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x077a), 1979814082:1979815154(1072) ack 2260258133 win 5840
10:05:23.593780 IP (tos 0x0, ttl 64, id 20076, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd1e7 (correct), 2260258133:2260258133(0) ack 1979798002 win 65535
10:05:23.593783 IP (tos 0x0, ttl 64, id 46445, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x272d), 1979815154:1979816226(1072) ack 2260258133 win 5840
10:05:23.593785 IP (tos 0x0, ttl 64, id 20077, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xcdb7 (correct), 2260258133:2260258133(0) ack 1979799074 win 65535
10:05:23.593788 IP (tos 0x0, ttl 64, id 46447, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x5108), 1979816226:1979817298(1072) ack 2260258133 win 5840
10:05:23.593790 IP (tos 0x0, ttl 64, id 20078, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc987 (correct), 2260258133:2260258133(0) ack 1979800146 win 65535
10:05:23.593792 IP (tos 0x0, ttl 64, id 46449, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x496c), 1979817298:1979818370(1072) ack 2260258133 win 5840
10:05:23.593824 IP (tos 0x0, ttl 64, id 20079, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc557 (correct), 2260258133:2260258133(0) ack 1979801218 win 65535
10:05:23.593827 IP (tos 0x0, ttl 64, id 46451, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x8a17), 1979818370:1979819442(1072) ack 2260258133 win 5840
10:05:23.593829 IP (tos 0x0, ttl 64, id 20080, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc127 (correct), 2260258133:2260258133(0) ack 1979802290 win 65535
10:05:23.593832 IP (tos 0x0, ttl 64, id 46453, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xa11a), 1979819442:1979820514(1072) ack 2260258133 win 5840
10:05:23.593835 IP (tos 0x0, ttl 64, id 20081, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xbcf7 (correct), 2260258133:2260258133(0) ack 1979803362 win 65535
10:05:23.593837 IP (tos 0x0, ttl 64, id 46455, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x2ef0), 1979820514:1979821586(1072) ack 2260258133 win 5840
10:05:23.593874 IP (tos 0x0, ttl 64, id 20082, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb8c7 (correct), 2260258133:2260258133(0) ack 1979804434 win 65535
10:05:23.593877 IP (tos 0x0, ttl 64, id 46457, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x8153), 1979821586:1979822658(1072) ack 2260258133 win 5840
10:05:23.593879 IP (tos 0x0, ttl 64, id 20083, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb497 (correct), 2260258133:2260258133(0) ack 1979805506 win 65535
10:05:23.593882 IP (tos 0x0, ttl 64, id 46459, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xb26b), 1979822658:1979823730(1072) ack 2260258133 win 5840
10:05:23.593884 IP (tos 0x0, ttl 64, id 20084, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb067 (correct), 2260258133:2260258133(0) ack 1979806578 win 65535
10:05:23.593886 IP (tos 0x0, ttl 64, id 46461, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x3cdc), 1979823730:1979824802(1072) ack 2260258133 win 5840
10:05:23.593924 IP (tos 0x0, ttl 64, id 20085, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xac37 (correct), 2260258133:2260258133(0) ack 1979807650 win 65535
10:05:23.593927 IP (tos 0x0, ttl 64, id 46463, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xeb21), 1979824802:1979825874(1072) ack 2260258133 win 5840
10:05:23.593929 IP (tos 0x0, ttl 64, id 20086, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa807 (correct), 2260258133:2260258133(0) ack 1979808722 win 65535
10:05:23.593931 IP (tos 0x0, ttl 64, id 46465, offset 0, flags [DF], proto TCP (6), length 1648) dione.40344 > phoenix.distcc: ., cksum 0x1c85 (incorrect (-> 0x58c9), 1979825874:1979827482(1608) ack 2260258133 win 5840
10:05:23.593973 IP (tos 0x0, ttl 64, id 20087, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa3d7 (correct), 2260258133:2260258133(0) ack 1979809794 win 65535
10:05:23.593976 IP (tos 0x0, ttl 64, id 46468, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x1cd2), 1979827482:1979828554(1072) ack 2260258133 win 5840
10:05:23.593978 IP (tos 0x0, ttl 64, id 20088, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9fa7 (correct), 2260258133:2260258133(0) ack 1979810866 win 65535
10:05:23.593981 IP (tos 0x0, ttl 64, id 46470, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x437a), 1979828554:1979829626(1072) ack 2260258133 win 5840
10:05:23.593982 IP (tos 0x0, ttl 64, id 20089, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9b77 (correct), 2260258133:2260258133(0) ack 1979811938 win 65535
10:05:23.593985 IP (tos 0x0, ttl 64, id 46472, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xe4cc), 1979829626:1979830698(1072) ack 2260258133 win 5840
10:05:23.593987 IP (tos 0x0, ttl 64, id 20090, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9747 (correct), 2260258133:2260258133(0) ack 1979813010 win 65535
10:05:23.593989 IP (tos 0x0, ttl 64, id 46474, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xf8ec), 1979830698:1979831770(1072) ack 2260258133 win 5840
10:05:23.593991 IP (tos 0x0, ttl 64, id 20091, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9317 (correct), 2260258133:2260258133(0) ack 1979814082 win 65535
10:05:23.593993 IP (tos 0x0, ttl 64, id 46476, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x590e), 1979831770:1979832842(1072) ack 2260258133 win 5840
10:05:23.593995 IP (tos 0x0, ttl 64, id 20092, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8ee7 (correct), 2260258133:2260258133(0) ack 1979815154 win 65535
10:05:23.593997 IP (tos 0x0, ttl 64, id 46478, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x4a1d), 1979832842:1979833914(1072) ack 2260258133 win 5840
10:05:23.594023 IP (tos 0x0, ttl 64, id 20093, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8ab7 (correct), 2260258133:2260258133(0) ack 1979816226 win 65535
10:05:23.594027 IP (tos 0x0, ttl 64, id 46480, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xad03), 1979833914:1979834986(1072) ack 2260258133 win 5840
10:05:23.594074 IP (tos 0x0, ttl 64, id 20094, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8687 (correct), 2260258133:2260258133(0) ack 1979817298 win 65535
10:05:23.594077 IP (tos 0x0, ttl 64, id 46482, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x1787), 1979834986:1979836058(1072) ack 2260258133 win 5840
10:05:23.594080 IP (tos 0x0, ttl 64, id 20095, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8257 (correct), 2260258133:2260258133(0) ack 1979818370 win 65535
10:05:23.594082 IP (tos 0x0, ttl 64, id 46484, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x3108), 1979836058:1979837130(1072) ack 2260258133 win 5840
10:05:23.594084 IP (tos 0x0, ttl 64, id 20096, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7e27 (correct), 2260258133:2260258133(0) ack 1979819442 win 65535
10:05:23.594087 IP (tos 0x0, ttl 64, id 46486, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xe5bf), 1979837130:1979838202(1072) ack 2260258133 win 5840
10:05:23.594089 IP (tos 0x0, ttl 64, id 20097, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x79f7 (correct), 2260258133:2260258133(0) ack 1979820514 win 65535
10:05:23.594091 IP (tos 0x0, ttl 64, id 46488, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x4a5e), 1979838202:1979839274(1072) ack 2260258133 win 5840
10:05:23.594093 IP (tos 0x0, ttl 64, id 20098, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x75c7 (correct), 2260258133:2260258133(0) ack 1979821586 win 65535
10:05:23.594095 IP (tos 0x0, ttl 64, id 46490, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x0bc1), 1979839274:1979840346(1072) ack 2260258133 win 5840
10:05:23.594096 IP (tos 0x0, ttl 64, id 20099, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7197 (correct), 2260258133:2260258133(0) ack 1979822658 win 65535
10:05:23.594098 IP (tos 0x0, ttl 64, id 46492, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xab2d), 1979840346:1979840882(536) ack 2260258133 win 5840
10:05:23.594100 IP (tos 0x0, ttl 64, id 46493, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x56a2), 1979840882:1979841418(536) ack 2260258133 win 5840
10:05:23.594124 IP (tos 0x0, ttl 64, id 20100, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6d67 (correct), 2260258133:2260258133(0) ack 1979823730 win 65535
10:05:23.594126 IP (tos 0x0, ttl 64, id 46494, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xe6a5), 1979841418:1979842490(1072) ack 2260258133 win 5840
10:05:23.594128 IP (tos 0x0, ttl 64, id 20101, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6937 (correct), 2260258133:2260258133(0) ack 1979824802 win 65535
10:05:23.594130 IP (tos 0x0, ttl 64, id 46496, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xa8be), 1979842490:1979843562(1072) ack 2260258133 win 5840
10:05:23.594174 IP (tos 0x0, ttl 64, id 20102, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6507 (correct), 2260258133:2260258133(0) ack 1979825874 win 65535
10:05:23.594177 IP (tos 0x0, ttl 64, id 46498, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xdd94), 1979843562:1979844634(1072) ack 2260258133 win 5840
10:05:23.594180 IP (tos 0x0, ttl 64, id 20103, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x60d7 (correct), 2260258133:2260258133(0) ack 1979826946 win 65535
10:05:23.594182 IP (tos 0x0, ttl 64, id 46500, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xc674), 1979844634:1979845706(1072) ack 2260258133 win 5840
10:05:23.594184 IP (tos 0x0, ttl 64, id 20104, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5ca7 (correct), 2260258133:2260258133(0) ack 1979828018 win 65535
10:05:23.594187 IP (tos 0x0, ttl 64, id 46502, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x3f94), 1979845706:1979846778(1072) ack 2260258133 win 5840
10:05:23.594188 IP (tos 0x0, ttl 64, id 20105, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5877 (correct), 2260258133:2260258133(0) ack 1979829090 win 65535
10:05:23.594191 IP (tos 0x0, ttl 64, id 46504, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x89a4), 1979846778:1979847850(1072) ack 2260258133 win 5840
10:05:23.594223 IP (tos 0x0, ttl 64, id 20106, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5447 (correct), 2260258133:2260258133(0) ack 1979830162 win 65535
10:05:23.594226 IP (tos 0x0, ttl 64, id 46506, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x6d13), 1979847850:1979848922(1072) ack 2260258133 win 5840
10:05:23.594228 IP (tos 0x0, ttl 64, id 20107, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5017 (correct), 2260258133:2260258133(0) ack 1979831234 win 65535
10:05:23.594231 IP (tos 0x0, ttl 64, id 46508, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x439e), 1979848922:1979849994(1072) ack 2260258133 win 5840
10:05:23.594274 IP (tos 0x0, ttl 64, id 20108, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4be7 (correct), 2260258133:2260258133(0) ack 1979832306 win 65535
10:05:23.594277 IP (tos 0x0, ttl 64, id 46510, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x155e), 1979849994:1979851066(1072) ack 2260258133 win 5840
10:05:23.594279 IP (tos 0x0, ttl 64, id 20109, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x47b7 (correct), 2260258133:2260258133(0) ack 1979833378 win 65535
10:05:23.594282 IP (tos 0x0, ttl 64, id 46512, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x5933), 1979851066:1979852138(1072) ack 2260258133 win 5840
10:05:23.594284 IP (tos 0x0, ttl 64, id 20110, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4387 (correct), 2260258133:2260258133(0) ack 1979834450 win 65535
10:05:23.594287 IP (tos 0x0, ttl 64, id 46514, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x43e1), 1979852138:1979853210(1072) ack 2260258133 win 5840
10:05:23.594289 IP (tos 0x0, ttl 64, id 20111, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3f57 (correct), 2260258133:2260258133(0) ack 1979835522 win 65535
10:05:23.594292 IP (tos 0x0, ttl 64, id 46516, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x15cd), 1979853210:1979854282(1072) ack 2260258133 win 5840
10:05:23.594323 IP (tos 0x0, ttl 64, id 20112, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3b27 (correct), 2260258133:2260258133(0) ack 1979836594 win 65535
10:05:23.594326 IP (tos 0x0, ttl 64, id 46518, offset 0, flags [DF], proto TCP (6), length 1648) dione.40344 > phoenix.distcc: ., cksum 0x1c85 (incorrect (-> 0x8bdd), 1979854282:1979855890(1608) ack 2260258133 win 5840
10:05:23.594329 IP (tos 0x0, ttl 64, id 20113, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x36f7 (correct), 2260258133:2260258133(0) ack 1979837666 win 65535
10:05:23.594331 IP (tos 0x0, ttl 64, id 46521, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x1397), 1979855890:1979856962(1072) ack 2260258133 win 5840
10:05:23.594333 IP (tos 0x0, ttl 64, id 20114, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x32c7 (correct), 2260258133:2260258133(0) ack 1979838738 win 65535
10:05:23.594336 IP (tos 0x0, ttl 64, id 46523, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xb2f5), 1979856962:1979858034(1072) ack 2260258133 win 5840
10:05:23.594373 IP (tos 0x0, ttl 64, id 20115, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2e97 (correct), 2260258133:2260258133(0) ack 1979839810 win 65535
10:05:23.594376 IP (tos 0x0, ttl 64, id 46525, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x432e), 1979858034:1979859106(1072) ack 2260258133 win 5840
10:05:23.594379 IP (tos 0x0, ttl 64, id 20116, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2a67 (correct), 2260258133:2260258133(0) ack 1979840882 win 65535
10:05:23.594381 IP (tos 0x0, ttl 64, id 46527, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x03cf), 1979859106:1979860178(1072) ack 2260258133 win 5840
10:05:23.594384 IP (tos 0x0, ttl 64, id 20117, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2637 (correct), 2260258133:2260258133(0) ack 1979841954 win 65535
10:05:23.594387 IP (tos 0x0, ttl 64, id 46529, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x2c9b), 1979860178:1979861250(1072) ack 2260258133 win 5840
10:05:23.594389 IP (tos 0x0, ttl 64, id 20118, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2207 (correct), 2260258133:2260258133(0) ack 1979843026 win 65535
10:05:23.594391 IP (tos 0x0, ttl 64, id 46531, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x0c0d), 1979861250:1979862322(1072) ack 2260258133 win 5840
10:05:23.594424 IP (tos 0x0, ttl 64, id 20119, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1dd7 (correct), 2260258133:2260258133(0) ack 1979844098 win 65535
10:05:23.594427 IP (tos 0x0, ttl 64, id 46533, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x390e), 1979862322:1979863394(1072) ack 2260258133 win 5840
10:05:23.594429 IP (tos 0x0, ttl 64, id 20120, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x19a7 (correct), 2260258133:2260258133(0) ack 1979845170 win 65535
10:05:23.594432 IP (tos 0x0, ttl 64, id 46535, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x995c), 1979863394:1979864466(1072) ack 2260258133 win 5840
10:05:23.594434 IP (tos 0x0, ttl 64, id 20121, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1577 (correct), 2260258133:2260258133(0) ack 1979846242 win 65535
10:05:23.594436 IP (tos 0x0, ttl 64, id 46537, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x9612), 1979864466:1979865538(1072) ack 2260258133 win 5840
10:05:23.594440 IP (tos 0x0, ttl 64, id 20122, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1147 (correct), 2260258133:2260258133(0) ack 1979847314 win 65535
10:05:23.594472 IP (tos 0x0, ttl 64, id 20123, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0d17 (correct), 2260258133:2260258133(0) ack 1979848386 win 65535
10:05:23.594476 IP (tos 0x0, ttl 64, id 46541, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xd7c5), 1979866610:1979867682(1072) ack 2260258133 win 5840
10:05:23.594478 IP (tos 0x0, ttl 64, id 20124, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x08e7 (correct), 2260258133:2260258133(0) ack 1979849458 win 65535
10:05:23.594480 IP (tos 0x0, ttl 64, id 46543, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xdd22), 1979867682:1979868754(1072) ack 2260258133 win 5840
10:05:23.594522 IP (tos 0x0, ttl 64, id 20125, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x04b7 (correct), 2260258133:2260258133(0) ack 1979850530 win 65535
10:05:23.594526 IP (tos 0x0, ttl 64, id 46545, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x7538), 1979868754:1979869826(1072) ack 2260258133 win 5840
10:05:23.594528 IP (tos 0x0, ttl 64, id 20126, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0087 (correct), 2260258133:2260258133(0) ack 1979851602 win 65535
10:05:23.594530 IP (tos 0x0, ttl 64, id 46547, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x765a), 1979869826:1979870898(1072) ack 2260258133 win 5840
10:05:23.594532 IP (tos 0x0, ttl 64, id 20127, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfc56 (correct), 2260258133:2260258133(0) ack 1979852674 win 65535
10:05:23.594535 IP (tos 0x0, ttl 64, id 46549, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x690b), 1979870898:1979871970(1072) ack 2260258133 win 5840
10:05:23.594537 IP (tos 0x0, ttl 64, id 20128, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf826 (correct), 2260258133:2260258133(0) ack 1979853746 win 65535
10:05:23.594540 IP (tos 0x0, ttl 64, id 46551, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xfa9b), 1979871970:1979873042(1072) ack 2260258133 win 5840
10:05:23.594572 IP (tos 0x0, ttl 64, id 20129, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf3f6 (correct), 2260258133:2260258133(0) ack 1979854818 win 65535
10:05:23.594576 IP (tos 0x0, ttl 64, id 46553, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x7701), 1979873042:1979873578(536) ack 2260258133 win 5840
10:05:23.594578 IP (tos 0x0, ttl 64, id 46554, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x69d8), 1979873578:1979874114(536) ack 2260258133 win 5840
10:05:23.594579 IP (tos 0x0, ttl 64, id 20130, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xefc6 (correct), 2260258133:2260258133(0) ack 1979855890 win 65535
10:05:23.594581 IP (tos 0x0, ttl 64, id 46555, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x206c), 1979874114:1979875186(1072) ack 2260258133 win 5840
10:05:23.594623 IP (tos 0x0, ttl 64, id 20133, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe336 (correct), 2260258133:2260258133(0) ack 1979859106 win 65535
10:05:23.594626 IP (tos 0x0, ttl 64, id 46561, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xb6bf), 1979877330:1979878402(1072) ack 2260258133 win 5840
10:05:23.594628 IP (tos 0x0, ttl 64, id 20134, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe11e (correct), 2260258133:2260258133(0) ack 1979859642 win 65535
10:05:23.594631 IP (tos 0x0, ttl 64, id 46563, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x3264), 1979878402:1979878938(536) ack 2260258133 win 5840
10:05:23.594632 IP (tos 0x0, ttl 64, id 20135, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xdf06 (correct), 2260258133:2260258133(0) ack 1979860178 win 65535
10:05:23.594635 IP (tos 0x0, ttl 64, id 20136, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xdcee (correct), 2260258133:2260258133(0) ack 1979860714 win 65535
10:05:23.594639 IP (tos 0x0, ttl 64, id 20137, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xdad6 (correct), 2260258133:2260258133(0) ack 1979861250 win 65535
10:05:23.594672 IP (tos 0x0, ttl 64, id 20138, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd8be (correct), 2260258133:2260258133(0) ack 1979861786 win 65535
10:05:23.594675 IP (tos 0x0, ttl 64, id 46567, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x2b6f), 1979880546:1979881618(1072) ack 2260258133 win 5840
10:05:23.594678 IP (tos 0x0, ttl 64, id 20139, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd6a6 (correct), 2260258133:2260258133(0) ack 1979862322 win 65535
10:05:23.594680 IP (tos 0x0, ttl 64, id 46569, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x7579), 1979881618:1979882154(536) ack 2260258133 win 5840
10:05:23.594681 IP (tos 0x0, ttl 64, id 20140, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd48e (correct), 2260258133:2260258133(0) ack 1979862858 win 65535
10:05:23.594683 IP (tos 0x0, ttl 64, id 46570, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xa70b), 1979882154:1979882690(536) ack 2260258133 win 5840
10:05:23.594685 IP (tos 0x0, ttl 64, id 20141, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd276 (correct), 2260258133:2260258133(0) ack 1979863394 win 65535
10:05:23.594688 IP (tos 0x0, ttl 64, id 20142, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd05e (correct), 2260258133:2260258133(0) ack 1979863930 win 65535
10:05:23.594692 IP (tos 0x0, ttl 64, id 20143, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xce46 (correct), 2260258133:2260258133(0) ack 1979864466 win 65535
10:05:23.594773 IP (tos 0x0, ttl 64, id 20148, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc3ce (correct), 2260258133:2260258133(0) ack 1979867146 win 65535
10:05:23.594776 IP (tos 0x0, ttl 64, id 46578, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xa8f1), 1979886442:1979886978(536) ack 2260258133 win 5840
10:05:23.594778 IP (tos 0x0, ttl 64, id 20149, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc1b6 (correct), 2260258133:2260258133(0) ack 1979867682 win 65535
10:05:23.594780 IP (tos 0x0, ttl 64, id 46579, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xa1ee), 1979886978:1979887514(536) ack 2260258133 win 5840
10:05:23.594781 IP (tos 0x0, ttl 64, id 20150, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xbf9e (correct), 2260258133:2260258133(0) ack 1979868218 win 65535
10:05:23.594783 IP (tos 0x0, ttl 64, id 46580, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x3a3c), 1979887514:1979888050(536) ack 2260258133 win 5840
10:05:23.594784 IP (tos 0x0, ttl 64, id 20151, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xbd86 (correct), 2260258133:2260258133(0) ack 1979868754 win 65535
10:05:23.594786 IP (tos 0x0, ttl 64, id 46581, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xa327), 1979888050:1979888586(536) ack 2260258133 win 5840
10:05:23.594787 IP (tos 0x0, ttl 64, id 20152, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xbb6e (correct), 2260258133:2260258133(0) ack 1979869290 win 65535
10:05:23.594790 IP (tos 0x0, ttl 64, id 46582, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x5d36), 1979888586:1979889122(536) ack 2260258133 win 5840
10:05:23.594791 IP (tos 0x0, ttl 64, id 20153, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb956 (correct), 2260258133:2260258133(0) ack 1979869826 win 65535
10:05:23.594794 IP (tos 0x0, ttl 64, id 46583, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x251b), 1979889122:1979889658(536) ack 2260258133 win 5840
10:05:23.594795 IP (tos 0x0, ttl 64, id 20154, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb73e (correct), 2260258133:2260258133(0) ack 1979870362 win 65535
10:05:23.594800 IP (tos 0x0, ttl 64, id 20155, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb526 (correct), 2260258133:2260258133(0) ack 1979870898 win 65535
10:05:23.594872 IP (tos 0x0, ttl 64, id 20163, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa466 (correct), 2260258133:2260258133(0) ack 1979875186 win 65535
10:05:23.594876 IP (tos 0x0, ttl 64, id 46593, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x5709), 1979894482:1979895018(536) ack 2260258133 win 5840
10:05:23.594877 IP (tos 0x0, ttl 64, id 20164, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa24e (correct), 2260258133:2260258133(0) ack 1979875722 win 65535
10:05:23.594879 IP (tos 0x0, ttl 64, id 46594, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xf8b5), 1979895018:1979896090(1072) ack 2260258133 win 5840
10:05:23.594881 IP (tos 0x0, ttl 64, id 20165, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa036 (correct), 2260258133:2260258133(0) ack 1979876258 win 65535
10:05:23.594883 IP (tos 0x0, ttl 64, id 46596, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x556c), 1979896090:1979896626(536) ack 2260258133 win 5840
10:05:23.594885 IP (tos 0x0, ttl 64, id 20166, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9e1e (correct), 2260258133:2260258133(0) ack 1979876794 win 65535
10:05:23.594887 IP (tos 0x0, ttl 64, id 46597, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x2a3c), 1979896626:1979897162(536) ack 2260258133 win 5840
10:05:23.594888 IP (tos 0x0, ttl 64, id 20167, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9c06 (correct), 2260258133:2260258133(0) ack 1979877330 win 65535
10:05:23.594890 IP (tos 0x0, ttl 64, id 46598, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xac70), 1979897162:1979897698(536) ack 2260258133 win 5840
10:05:23.594892 IP (tos 0x0, ttl 64, id 20168, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x99ee (correct), 2260258133:2260258133(0) ack 1979877866 win 65535
10:05:23.594972 IP (tos 0x0, ttl 64, id 20177, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8716 (correct), 2260258133:2260258133(0) ack 1979882690 win 65535
10:05:23.594975 IP (tos 0x0, ttl 64, id 46608, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xa874), 1979902522:1979903058(536) ack 2260258133 win 5840
10:05:23.594977 IP (tos 0x0, ttl 64, id 20178, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x84fe (correct), 2260258133:2260258133(0) ack 1979883226 win 65535
10:05:23.594979 IP (tos 0x0, ttl 64, id 46609, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x89bb), 1979903058:1979903594(536) ack 2260258133 win 5840
10:05:23.594980 IP (tos 0x0, ttl 64, id 20179, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x82e6 (correct), 2260258133:2260258133(0) ack 1979883762 win 65535
10:05:23.594982 IP (tos 0x0, ttl 64, id 46610, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xc089), 1979903594:1979904130(536) ack 2260258133 win 5840
10:05:23.594983 IP (tos 0x0, ttl 64, id 20180, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x80ce (correct), 2260258133:2260258133(0) ack 1979884298 win 65535
10:05:23.594985 IP (tos 0x0, ttl 64, id 46611, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xac95), 1979904130:1979904666(536) ack 2260258133 win 5840
10:05:23.594986 IP (tos 0x0, ttl 64, id 20181, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7eb6 (correct), 2260258133:2260258133(0) ack 1979884834 win 65535
10:05:23.594988 IP (tos 0x0, ttl 64, id 46612, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xe0f2), 1979904666:1979905202(536) ack 2260258133 win 5840
10:05:23.594989 IP (tos 0x0, ttl 64, id 20182, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7c9e (correct), 2260258133:2260258133(0) ack 1979885370 win 65535
10:05:23.594992 IP (tos 0x0, ttl 64, id 46613, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x29de), 1979905202:1979905738(536) ack 2260258133 win 5840
10:05:23.594993 IP (tos 0x0, ttl 64, id 20183, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7a86 (correct), 2260258133:2260258133(0) ack 1979885906 win 65535
10:05:23.594994 IP (tos 0x0, ttl 64, id 46614, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: P, cksum 0x1855 (incorrect (-> 0xc40d), 1979905738:1979906274(536) ack 2260258133 win 5840
10:05:23.595072 IP (tos 0x0, ttl 64, id 20197, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5d36 (correct), 2260258133:2260258133(0) ack 1979893410 win 65535
10:05:23.595075 IP (tos 0x0, ttl 64, id 46629, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xdfe4), 1979913778:1979914314(536) ack 2260258133 win 5840
10:05:23.595076 IP (tos 0x0, ttl 64, id 20198, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5b1e (correct), 2260258133:2260258133(0) ack 1979893946 win 65535
10:05:23.595079 IP (tos 0x0, ttl 64, id 46630, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x6597), 1979914314:1979914850(536) ack 2260258133 win 5840
10:05:23.595080 IP (tos 0x0, ttl 64, id 20199, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5906 (correct), 2260258133:2260258133(0) ack 1979894482 win 65535
10:05:23.595082 IP (tos 0x0, ttl 64, id 46631, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x6740), 1979914850:1979915386(536) ack 2260258133 win 5840
10:05:23.595083 IP (tos 0x0, ttl 64, id 20200, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x56ee (correct), 2260258133:2260258133(0) ack 1979895018 win 65535
10:05:23.595085 IP (tos 0x0, ttl 64, id 46632, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x3dbb), 1979915386:1979915922(536) ack 2260258133 win 5840
10:05:23.595087 IP (tos 0x0, ttl 64, id 20201, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x54d6 (correct), 2260258133:2260258133(0) ack 1979895554 win 65535
10:05:23.595172 IP (tos 0x0, ttl 64, id 20211, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3fe6 (correct), 2260258133:2260258133(0) ack 1979900914 win 65535
10:05:23.595175 IP (tos 0x0, ttl 64, id 46643, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x1324), 1979921282:1979921818(536) ack 2260258133 win 5840
10:05:23.595176 IP (tos 0x0, ttl 64, id 20212, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3dce (correct), 2260258133:2260258133(0) ack 1979901450 win 65535
10:05:23.595178 IP (tos 0x0, ttl 64, id 46644, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xfa30), 1979921818:1979922354(536) ack 2260258133 win 5840
10:05:23.595179 IP (tos 0x0, ttl 64, id 20213, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3bb6 (correct), 2260258133:2260258133(0) ack 1979901986 win 65535
10:05:23.595182 IP (tos 0x0, ttl 64, id 46645, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xc91b), 1979922354:1979922890(536) ack 2260258133 win 5840
10:05:23.595183 IP (tos 0x0, ttl 64, id 20214, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x399e (correct), 2260258133:2260258133(0) ack 1979902522 win 65535
10:05:23.595185 IP (tos 0x0, ttl 64, id 46646, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x2721), 1979922890:1979923426(536) ack 2260258133 win 5840
10:05:23.595186 IP (tos 0x0, ttl 64, id 20215, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3786 (correct), 2260258133:2260258133(0) ack 1979903058 win 65535
10:05:23.595188 IP (tos 0x0, ttl 64, id 46647, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x0b03), 1979923426:1979923962(536) ack 2260258133 win 5840
10:05:23.595190 IP (tos 0x0, ttl 64, id 20216, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x356e (correct), 2260258133:2260258133(0) ack 1979903594 win 65535
10:05:23.595192 IP (tos 0x0, ttl 64, id 46648, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xdf6d), 1979923962:1979925034(1072) ack 2260258133 win 5840
10:05:23.595194 IP (tos 0x0, ttl 64, id 20217, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3356 (correct), 2260258133:2260258133(0) ack 1979904130 win 65535
10:05:23.595197 IP (tos 0x0, ttl 64, id 20218, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x313e (correct), 2260258133:2260258133(0) ack 1979904666 win 65535
10:05:23.595322 IP (tos 0x0, ttl 64, id 20236, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0b8e (correct), 2260258133:2260258133(0) ack 1979914314 win 65535
10:05:23.595325 IP (tos 0x0, ttl 64, id 46669, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xd48b), 1979935218:1979935754(536) ack 2260258133 win 5840
10:05:23.595326 IP (tos 0x0, ttl 64, id 20237, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0976 (correct), 2260258133:2260258133(0) ack 1979914850 win 65535
10:05:23.595328 IP (tos 0x0, ttl 64, id 46670, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xf8c9), 1979935754:1979936290(536) ack 2260258133 win 5840
10:05:23.595329 IP (tos 0x0, ttl 64, id 20238, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x075e (correct), 2260258133:2260258133(0) ack 1979915386 win 65535
10:05:23.595331 IP (tos 0x0, ttl 64, id 46671, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xa441), 1979936290:1979936826(536) ack 2260258133 win 5840
10:05:23.595332 IP (tos 0x0, ttl 64, id 20239, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0546 (correct), 2260258133:2260258133(0) ack 1979915922 win 65535
10:05:23.595334 IP (tos 0x0, ttl 64, id 46672, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x6bf7), 1979936826:1979937362(536) ack 2260258133 win 5840
10:05:23.595336 IP (tos 0x0, ttl 64, id 20240, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x032e (correct), 2260258133:2260258133(0) ack 1979916458 win 65535
10:05:23.595338 IP (tos 0x0, ttl 64, id 46673, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xd6c7), 1979937362:1979937898(536) ack 2260258133 win 5840
10:05:23.595339 IP (tos 0x0, ttl 64, id 20241, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0116 (correct), 2260258133:2260258133(0) ack 1979916994 win 65535
10:05:23.595341 IP (tos 0x0, ttl 64, id 46674, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x8dbd), 1979937898:1979938434(536) ack 2260258133 win 5840
10:05:23.595342 IP (tos 0x0, ttl 64, id 20242, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfefd (correct), 2260258133:2260258133(0) ack 1979917530 win 65535
10:05:23.595344 IP (tos 0x0, ttl 64, id 46675, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x6059), 1979938434:1979938970(536) ack 2260258133 win 5840
10:05:23.595348 IP (tos 0x0, ttl 64, id 20243, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfce5 (correct), 2260258133:2260258133(0) ack 1979918066 win 65535
10:05:23.595421 IP (tos 0x0, ttl 64, id 20259, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xdb65 (correct), 2260258133:2260258133(0) ack 1979926642 win 65535
10:05:23.595424 IP (tos 0x0, ttl 64, id 46693, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x9c32), 1979948082:1979948618(536) ack 2260258133 win 5840
10:05:23.595426 IP (tos 0x0, ttl 64, id 20260, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd94d (correct), 2260258133:2260258133(0) ack 1979927178 win 65535
10:05:23.595428 IP (tos 0x0, ttl 64, id 46694, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xe158), 1979948618:1979949154(536) ack 2260258133 win 5840
10:05:23.595429 IP (tos 0x0, ttl 64, id 20261, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd735 (correct), 2260258133:2260258133(0) ack 1979927714 win 65535
10:05:23.595432 IP (tos 0x0, ttl 64, id 20262, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd51d (correct), 2260258133:2260258133(0) ack 1979928250 win 65535
10:05:23.595522 IP (tos 0x0, ttl 64, id 20273, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xbe15 (correct), 2260258133:2260258133(0) ack 1979934146 win 65535
10:05:23.595525 IP (tos 0x0, ttl 64, id 46708, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x97a3), 1979956122:1979956658(536) ack 2260258133 win 5840
10:05:23.595527 IP (tos 0x0, ttl 64, id 20274, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xbbfd (correct), 2260258133:2260258133(0) ack 1979934682 win 65535
10:05:23.595529 IP (tos 0x0, ttl 64, id 46709, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xb57c), 1979956658:1979957194(536) ack 2260258133 win 5840
10:05:23.595530 IP (tos 0x0, ttl 64, id 20275, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb9e5 (correct), 2260258133:2260258133(0) ack 1979935218 win 65535
10:05:23.595532 IP (tos 0x0, ttl 64, id 46710, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x117e), 1979957194:1979957730(536) ack 2260258133 win 5840
10:05:23.595533 IP (tos 0x0, ttl 64, id 20276, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb7cd (correct), 2260258133:2260258133(0) ack 1979935754 win 65535
10:05:23.595535 IP (tos 0x0, ttl 64, id 46711, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x49fb), 1979957730:1979958266(536) ack 2260258133 win 5840
10:05:23.595536 IP (tos 0x0, ttl 64, id 20277, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb5b5 (correct), 2260258133:2260258133(0) ack 1979936290 win 65535
10:05:23.595538 IP (tos 0x0, ttl 64, id 46712, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x0f52), 1979958266:1979958802(536) ack 2260258133 win 5840
10:05:23.595539 IP (tos 0x0, ttl 64, id 20278, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb39d (correct), 2260258133:2260258133(0) ack 1979936826 win 65535
10:05:23.595541 IP (tos 0x0, ttl 64, id 46713, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x3b33), 1979958802:1979959338(536) ack 2260258133 win 5840
10:05:23.595542 IP (tos 0x0, ttl 64, id 20279, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb185 (correct), 2260258133:2260258133(0) ack 1979937362 win 65535
10:05:23.595544 IP (tos 0x0, ttl 64, id 46714, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xe889), 1979959338:1979959874(536) ack 2260258133 win 5840
10:05:23.595672 IP (tos 0x0, ttl 64, id 20304, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7d2d (correct), 2260258133:2260258133(0) ack 1979950762 win 65535
10:05:23.595675 IP (tos 0x0, ttl 64, id 46740, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xd20c), 1979973274:1979973810(536) ack 2260258133 win 5840
10:05:23.595676 IP (tos 0x0, ttl 64, id 20305, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7b15 (correct), 2260258133:2260258133(0) ack 1979951298 win 65535
10:05:23.595678 IP (tos 0x0, ttl 64, id 46741, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x66f2), 1979973810:1979974346(536) ack 2260258133 win 5840
10:05:23.595680 IP (tos 0x0, ttl 64, id 20306, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x78fd (correct), 2260258133:2260258133(0) ack 1979951834 win 65535
10:05:23.595681 IP (tos 0x0, ttl 64, id 46742, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x754d), 1979974346:1979974882(536) ack 2260258133 win 5840
10:05:23.595683 IP (tos 0x0, ttl 64, id 20307, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x76e5 (correct), 2260258133:2260258133(0) ack 1979952370 win 65535
10:05:23.595685 IP (tos 0x0, ttl 64, id 46743, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x51b5), 1979974882:1979975418(536) ack 2260258133 win 5840
10:05:23.595686 IP (tos 0x0, ttl 64, id 20308, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x74cd (correct), 2260258133:2260258133(0) ack 1979952906 win 65535
10:05:23.595688 IP (tos 0x0, ttl 64, id 46744, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x0fb0), 1979975418:1979975954(536) ack 2260258133 win 5840
10:05:23.595689 IP (tos 0x0, ttl 64, id 20309, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x72b5 (correct), 2260258133:2260258133(0) ack 1979953442 win 65535
10:05:23.595691 IP (tos 0x0, ttl 64, id 46745, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x672e), 1979975954:1979976490(536) ack 2260258133 win 5840
10:05:23.595692 IP (tos 0x0, ttl 64, id 20310, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x709d (correct), 2260258133:2260258133(0) ack 1979953978 win 65535
10:05:23.595694 IP (tos 0x0, ttl 64, id 46746, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x3fed), 1979976490:1979977026(536) ack 2260258133 win 5840
10:05:23.595771 IP (tos 0x0, ttl 64, id 20319, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5dc5 (correct), 2260258133:2260258133(0) ack 1979958802 win 65535
10:05:23.595774 IP (tos 0x0, ttl 64, id 46755, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xbf38), 1979981314:1979981850(536) ack 2260258133 win 5840
10:05:23.595776 IP (tos 0x0, ttl 64, id 20320, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5bad (correct), 2260258133:2260258133(0) ack 1979959338 win 65535
10:05:23.595778 IP (tos 0x0, ttl 64, id 46756, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x2a02), 1979981850:1979982922(1072) ack 2260258133 win 5840
10:05:23.595780 IP (tos 0x0, ttl 64, id 20321, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5995 (correct), 2260258133:2260258133(0) ack 1979959874 win 65535
10:05:23.595782 IP (tos 0x0, ttl 64, id 46758, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x491b), 1979982922:1979983458(536) ack 2260258133 win 5840
10:05:23.595783 IP (tos 0x0, ttl 64, id 20322, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x577d (correct), 2260258133:2260258133(0) ack 1979960410 win 65535
10:05:23.595785 IP (tos 0x0, ttl 64, id 46759, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xc55f), 1979983458:1979983994(536) ack 2260258133 win 5840
10:05:23.595786 IP (tos 0x0, ttl 64, id 20323, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5565 (correct), 2260258133:2260258133(0) ack 1979960946 win 65535
10:05:23.595788 IP (tos 0x0, ttl 64, id 46760, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x3150), 1979983994:1979984530(536) ack 2260258133 win 5840
10:05:23.595789 IP (tos 0x0, ttl 64, id 20324, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x534d (correct), 2260258133:2260258133(0) ack 1979961482 win 65535
10:05:23.595792 IP (tos 0x0, ttl 64, id 46761, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xe5d0), 1979984530:1979985066(536) ack 2260258133 win 5840
10:05:23.595792 IP (tos 0x0, ttl 64, id 20325, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5135 (correct), 2260258133:2260258133(0) ack 1979962018 win 65535
10:05:23.595795 IP (tos 0x0, ttl 64, id 20326, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4f1d (correct), 2260258133:2260258133(0) ack 1979962554 win 65535
10:05:23.595921 IP (tos 0x0, ttl 64, id 20352, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x18ad (correct), 2260258133:2260258133(0) ack 1979976490 win 65535
10:05:23.595925 IP (tos 0x0, ttl 64, id 46790, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xf4e9), 1980000074:1980000610(536) ack 2260258133 win 5840
10:05:23.595926 IP (tos 0x0, ttl 64, id 20353, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1695 (correct), 2260258133:2260258133(0) ack 1979977026 win 65535
10:05:23.595928 IP (tos 0x0, ttl 64, id 46791, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x67f3), 1980000610:1980001146(536) ack 2260258133 win 5840
10:05:23.595929 IP (tos 0x0, ttl 64, id 20354, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x147d (correct), 2260258133:2260258133(0) ack 1979977562 win 65535
10:05:23.595931 IP (tos 0x0, ttl 64, id 46792, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xa26b), 1980001146:1980001682(536) ack 2260258133 win 5840
10:05:23.595932 IP (tos 0x0, ttl 64, id 20355, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1265 (correct), 2260258133:2260258133(0) ack 1979978098 win 65535
10:05:23.595934 IP (tos 0x0, ttl 64, id 46793, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x5a2a), 1980001682:1980002218(536) ack 2260258133 win 5840
10:05:23.596020 IP (tos 0x0, ttl 64, id 20356, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x104d (correct), 2260258133:2260258133(0) ack 1979978634 win 65535
10:05:23.596023 IP (tos 0x0, ttl 64, id 46794, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x1d1d), 1980002218:1980002754(536) ack 2260258133 win 5840
10:05:23.596024 IP (tos 0x0, ttl 64, id 20357, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0e35 (correct), 2260258133:2260258133(0) ack 1979979170 win 65535
10:05:23.596027 IP (tos 0x0, ttl 64, id 46795, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x1d95), 1980002754:1980003290(536) ack 2260258133 win 5840
10:05:23.596028 IP (tos 0x0, ttl 64, id 20358, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0c1d (correct), 2260258133:2260258133(0) ack 1979979706 win 65535
10:05:23.596030 IP (tos 0x0, ttl 64, id 46796, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xf42a), 1980003290:1980003826(536) ack 2260258133 win 5840
10:05:23.596031 IP (tos 0x0, ttl 64, id 20359, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0a05 (correct), 2260258133:2260258133(0) ack 1979980242 win 65535
10:05:23.596033 IP (tos 0x0, ttl 64, id 46797, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xb9ca), 1980003826:1980004362(536) ack 2260258133 win 5840
10:05:23.596034 IP (tos 0x0, ttl 64, id 20360, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x07ed (correct), 2260258133:2260258133(0) ack 1979980778 win 65535
10:05:23.596036 IP (tos 0x0, ttl 64, id 46798, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xd6a9), 1980004362:1980004898(536) ack 2260258133 win 5840
10:05:23.596037 IP (tos 0x0, ttl 64, id 20361, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x05d5 (correct), 2260258133:2260258133(0) ack 1979981314 win 65535
10:05:23.596039 IP (tos 0x0, ttl 64, id 46799, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x2078), 1980004898:1980005434(536) ack 2260258133 win 5840
10:05:23.596040 IP (tos 0x0, ttl 64, id 20362, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x03bd (correct), 2260258133:2260258133(0) ack 1979981850 win 65535
10:05:23.596042 IP (tos 0x0, ttl 64, id 46800, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x8bd3), 1980005434:1980005970(536) ack 2260258133 win 5840
10:05:23.596043 IP (tos 0x0, ttl 64, id 20363, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x01a5 (correct), 2260258133:2260258133(0) ack 1979982386 win 65535
10:05:23.596045 IP (tos 0x0, ttl 64, id 46801, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x4d3a), 1980005970:1980006506(536) ack 2260258133 win 5840
10:05:23.596047 IP (tos 0x0, ttl 64, id 20364, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xff8c (correct), 2260258133:2260258133(0) ack 1979982922 win 65535
10:05:23.596048 IP (tos 0x0, ttl 64, id 46802, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x4531), 1980006506:1980007042(536) ack 2260258133 win 5840
10:05:23.596050 IP (tos 0x0, ttl 64, id 20365, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfd74 (correct), 2260258133:2260258133(0) ack 1979983458 win 65535
10:05:23.596053 IP (tos 0x0, ttl 64, id 46803, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x4bb3), 1980007042:1980007578(536) ack 2260258133 win 5840
10:05:23.596054 IP (tos 0x0, ttl 64, id 20366, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfb5c (correct), 2260258133:2260258133(0) ack 1979983994 win 65535
10:05:23.596055 IP (tos 0x0, ttl 64, id 46804, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x500e), 1980007578:1980008114(536) ack 2260258133 win 5840
10:05:23.596056 IP (tos 0x0, ttl 64, id 20367, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf944 (correct), 2260258133:2260258133(0) ack 1979984530 win 65535
10:05:23.596059 IP (tos 0x0, ttl 64, id 20368, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf72c (correct), 2260258133:2260258133(0) ack 1979985066 win 65535
10:05:23.596119 IP (tos 0x0, ttl 64, id 20383, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd7c4 (correct), 2260258133:2260258133(0) ack 1979993106 win 65535
10:05:23.596171 IP (tos 0x0, ttl 64, id 20390, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc91c (correct), 2260258133:2260258133(0) ack 1979996858 win 65535
10:05:23.596174 IP (tos 0x0, ttl 64, id 46829, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x6077), 1980020978:1980021514(536) ack 2260258133 win 5840
10:05:23.596175 IP (tos 0x0, ttl 64, id 20391, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc704 (correct), 2260258133:2260258133(0) ack 1979997394 win 65535
10:05:23.596177 IP (tos 0x0, ttl 64, id 46830, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x51e6), 1980021514:1980022050(536) ack 2260258133 win 5840
10:05:23.596178 IP (tos 0x0, ttl 64, id 20392, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc4ec (correct), 2260258133:2260258133(0) ack 1979997930 win 65535
10:05:23.596180 IP (tos 0x0, ttl 64, id 46831, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0xc561), 1980022050:1980022586(536) ack 2260258133 win 5840
10:05:23.596270 IP (tos 0x0, ttl 64, id 20406, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8c64 (correct), 2260258133:2260258133(0) ack 1980012402 win 65535
10:05:23.596273 IP (tos 0x0, ttl 64, id 46858, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: P, cksum 0x1855 (incorrect (-> 0x85df), 1980036522:1980037058(536) ack 2260258133 win 5840
10:05:23.596275 IP (tos 0x0, ttl 64, id 46859, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x6dc3), 1980037058:1980037594(536) ack 2260258133 win 5840
10:05:23.596276 IP (tos 0x0, ttl 64, id 20407, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8834 (correct), 2260258133:2260258133(0) ack 1980013474 win 65535
10:05:23.596280 IP (tos 0x0, ttl 64, id 20408, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8404 (correct), 2260258133:2260258133(0) ack 1980014546 win 65535
10:05:23.596285 IP (tos 0x0, ttl 64, id 20409, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7fd4 (correct), 2260258133:2260258133(0) ack 1980015618 win 65535
10:05:23.596289 IP (tos 0x0, ttl 64, id 20410, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7ba4 (correct), 2260258133:2260258133(0) ack 1980016690 win 65535
10:05:23.596370 IP (tos 0x0, ttl 64, id 20417, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5e54 (correct), 2260258133:2260258133(0) ack 1980024194 win 65535
10:05:23.596374 IP (tos 0x0, ttl 64, id 46880, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x57d6), 1980048314:1980049386(1072) ack 2260258133 win 5840
10:05:23.596375 IP (tos 0x0, ttl 64, id 20418, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5a24 (correct), 2260258133:2260258133(0) ack 1980025266 win 65535
10:05:23.596378 IP (tos 0x0, ttl 64, id 46882, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x623a), 1980049386:1980050458(1072) ack 2260258133 win 5840
10:05:23.596379 IP (tos 0x0, ttl 64, id 20419, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x55f4 (correct), 2260258133:2260258133(0) ack 1980026338 win 65535
10:05:23.596381 IP (tos 0x0, ttl 64, id 46884, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x1968), 1980050458:1980051530(1072) ack 2260258133 win 5840
10:05:23.596383 IP (tos 0x0, ttl 64, id 20420, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x51c4 (correct), 2260258133:2260258133(0) ack 1980027410 win 65535
10:05:23.596385 IP (tos 0x0, ttl 64, id 46886, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xc14a), 1980051530:1980052602(1072) ack 2260258133 win 5840
10:05:23.596387 IP (tos 0x0, ttl 64, id 20421, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4d94 (correct), 2260258133:2260258133(0) ack 1980028482 win 65535
10:05:23.596967 IP (tos 0x0, ttl 64, id 20422, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4964 (correct), 2260258133:2260258133(0) ack 1980029554 win 65535
10:05:23.596971 IP (tos 0x0, ttl 64, id 46890, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x840c), 1980053674:1980054746(1072) ack 2260258133 win 5840
10:05:23.596973 IP (tos 0x0, ttl 64, id 20423, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4534 (correct), 2260258133:2260258133(0) ack 1980030626 win 65535
10:05:23.596975 IP (tos 0x0, ttl 64, id 46892, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x08ba), 1980054746:1980055818(1072) ack 2260258133 win 5840
10:05:23.596976 IP (tos 0x0, ttl 64, id 20424, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4104 (correct), 2260258133:2260258133(0) ack 1980031698 win 65535
10:05:23.596979 IP (tos 0x0, ttl 64, id 46894, offset 0, flags [DF], proto TCP (6), length 1648) dione.40344 > phoenix.distcc: ., cksum 0x1c85 (incorrect (-> 0x8154), 1980055818:1980057426(1608) ack 2260258133 win 5840
10:05:23.597019 IP (tos 0x0, ttl 64, id 20425, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3cd4 (correct), 2260258133:2260258133(0) ack 1980032770 win 65535
10:05:23.597022 IP (tos 0x0, ttl 64, id 46897, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xcb5d), 1980057426:1980058498(1072) ack 2260258133 win 5840
10:05:23.597023 IP (tos 0x0, ttl 64, id 20426, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x38a4 (correct), 2260258133:2260258133(0) ack 1980033842 win 65535
10:05:23.597026 IP (tos 0x0, ttl 64, id 46899, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x33dc), 1980058498:1980059570(1072) ack 2260258133 win 5840
10:05:23.597028 IP (tos 0x0, ttl 64, id 20427, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3474 (correct), 2260258133:2260258133(0) ack 1980034914 win 65535
10:05:23.597030 IP (tos 0x0, ttl 64, id 46901, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xf8b2), 1980059570:1980060642(1072) ack 2260258133 win 5840
10:05:23.597032 IP (tos 0x0, ttl 64, id 20428, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3044 (correct), 2260258133:2260258133(0) ack 1980035986 win 65535
10:05:23.597034 IP (tos 0x0, ttl 64, id 46903, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x1d20), 1980060642:1980061714(1072) ack 2260258133 win 5840
10:05:23.597036 IP (tos 0x0, ttl 64, id 20429, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2c14 (correct), 2260258133:2260258133(0) ack 1980037058 win 65535
10:05:23.597039 IP (tos 0x0, ttl 64, id 46905, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xdc53), 1980061714:1980062786(1072) ack 2260258133 win 5840
10:05:23.597040 IP (tos 0x0, ttl 64, id 20430, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x27e4 (correct), 2260258133:2260258133(0) ack 1980038130 win 65535
10:05:23.597043 IP (tos 0x0, ttl 64, id 46907, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x3785), 1980062786:1980063858(1072) ack 2260258133 win 5840
10:05:23.597044 IP (tos 0x0, ttl 64, id 20431, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x23b4 (correct), 2260258133:2260258133(0) ack 1980039202 win 65535
10:05:23.597047 IP (tos 0x0, ttl 64, id 46909, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xa5b8), 1980063858:1980064930(1072) ack 2260258133 win 5840
10:05:23.597049 IP (tos 0x0, ttl 64, id 20432, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1f84 (correct), 2260258133:2260258133(0) ack 1980040274 win 65535
10:05:23.597051 IP (tos 0x0, ttl 64, id 46911, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x4b61), 1980064930:1980066002(1072) ack 2260258133 win 5840
10:05:23.597053 IP (tos 0x0, ttl 64, id 20433, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1b54 (correct), 2260258133:2260258133(0) ack 1980041346 win 65535
10:05:23.597055 IP (tos 0x0, ttl 64, id 46913, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xb61b), 1980066002:1980067074(1072) ack 2260258133 win 5840
10:05:23.597057 IP (tos 0x0, ttl 64, id 20434, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1724 (correct), 2260258133:2260258133(0) ack 1980042418 win 65535
10:05:23.597059 IP (tos 0x0, ttl 64, id 46915, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xabf9), 1980067074:1980068146(1072) ack 2260258133 win 5840
10:05:23.597061 IP (tos 0x0, ttl 64, id 20435, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x12f4 (correct), 2260258133:2260258133(0) ack 1980043490 win 65535
10:05:23.597063 IP (tos 0x0, ttl 64, id 46917, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x2116), 1980068146:1980069218(1072) ack 2260258133 win 5840
10:05:23.597065 IP (tos 0x0, ttl 64, id 20436, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0ec4 (correct), 2260258133:2260258133(0) ack 1980044562 win 65535
10:05:23.597067 IP (tos 0x0, ttl 64, id 46919, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x5c77), 1980069218:1980069754(536) ack 2260258133 win 5840
10:05:23.597068 IP (tos 0x0, ttl 64, id 46920, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: ., cksum 0x1855 (incorrect (-> 0x8107), 1980069754:1980070290(536) ack 2260258133 win 5840
10:05:23.597070 IP (tos 0x0, ttl 64, id 20437, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0a94 (correct), 2260258133:2260258133(0) ack 1980045634 win 65535
10:05:23.597072 IP (tos 0x0, ttl 64, id 46921, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x1c56), 1980070290:1980071362(1072) ack 2260258133 win 5840
10:05:23.597074 IP (tos 0x0, ttl 64, id 20438, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0664 (correct), 2260258133:2260258133(0) ack 1980046706 win 65535
10:05:23.597076 IP (tos 0x0, ttl 64, id 46923, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x571c), 1980071362:1980072434(1072) ack 2260258133 win 5840
10:05:23.597078 IP (tos 0x0, ttl 64, id 20439, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0234 (correct), 2260258133:2260258133(0) ack 1980047778 win 65535
10:05:23.597080 IP (tos 0x0, ttl 64, id 46925, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x312a), 1980072434:1980073506(1072) ack 2260258133 win 5840
10:05:23.597081 IP (tos 0x0, ttl 64, id 20440, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfe03 (correct), 2260258133:2260258133(0) ack 1980048850 win 65535
10:05:23.597084 IP (tos 0x0, ttl 64, id 46927, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x7272), 1980073506:1980074578(1072) ack 2260258133 win 5840
10:05:23.597086 IP (tos 0x0, ttl 64, id 20441, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf9d3 (correct), 2260258133:2260258133(0) ack 1980049922 win 65535
10:05:23.597089 IP (tos 0x0, ttl 64, id 46929, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x9aa9), 1980074578:1980075650(1072) ack 2260258133 win 5840
10:05:23.597092 IP (tos 0x0, ttl 64, id 20442, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf5a3 (correct), 2260258133:2260258133(0) ack 1980050994 win 65535
10:05:23.597095 IP (tos 0x0, ttl 64, id 46931, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x3ee6), 1980075650:1980076722(1072) ack 2260258133 win 5840
10:05:23.597118 IP (tos 0x0, ttl 64, id 20443, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf173 (correct), 2260258133:2260258133(0) ack 1980052066 win 65535
10:05:23.597121 IP (tos 0x0, ttl 64, id 46933, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xe496), 1980076722:1980077794(1072) ack 2260258133 win 5840
10:05:23.597124 IP (tos 0x0, ttl 64, id 20444, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xed43 (correct), 2260258133:2260258133(0) ack 1980053138 win 65535
10:05:23.597126 IP (tos 0x0, ttl 64, id 46935, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xaea4), 1980077794:1980078866(1072) ack 2260258133 win 5840
10:05:23.597128 IP (tos 0x0, ttl 64, id 20445, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe913 (correct), 2260258133:2260258133(0) ack 1980054210 win 65535
10:05:23.597130 IP (tos 0x0, ttl 64, id 46937, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xf86a), 1980078866:1980079938(1072) ack 2260258133 win 5840
10:05:23.597132 IP (tos 0x0, ttl 64, id 20446, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe4e3 (correct), 2260258133:2260258133(0) ack 1980055282 win 65535
10:05:23.597134 IP (tos 0x0, ttl 64, id 46939, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0x68f9), 1980079938:1980081010(1072) ack 2260258133 win 5840
10:05:23.597136 IP (tos 0x0, ttl 64, id 20447, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe0b3 (correct), 2260258133:2260258133(0) ack 1980056354 win 65535
10:05:23.597138 IP (tos 0x0, ttl 64, id 46941, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xb1a1), 1980081010:1980082082(1072) ack 2260258133 win 5840
10:05:23.597140 IP (tos 0x0, ttl 64, id 20448, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xdc83 (correct), 2260258133:2260258133(0) ack 1980057426 win 65535
10:05:23.597142 IP (tos 0x0, ttl 64, id 46943, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xbc8a), 1980082082:1980083154(1072) ack 2260258133 win 5840
10:05:23.597144 IP (tos 0x0, ttl 64, id 20449, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd853 (correct), 2260258133:2260258133(0) ack 1980058498 win 65535
10:05:23.597146 IP (tos 0x0, ttl 64, id 46945, offset 0, flags [DF], proto TCP (6), length 1112) dione.40344 > phoenix.distcc: ., cksum 0x1a6d (incorrect (-> 0xc6cf), 1980083154:1980084226(1072) ack 2260258133 win 5840
10:05:23.597148 IP (tos 0x0, ttl 64, id 20450, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd423 (correct), 2260258133:2260258133(0) ack 1980059570 win 65535
10:05:23.597150 IP (tos 0x0, ttl 64, id 46947, offset 0, flags [DF], proto TCP (6), length 1648) dione.40344 > phoenix.distcc: ., cksum 0x1c85 (incorrect (-> 0x5cfb), 1980084226:1980085834(1608) ack 2260258133 win 5840
10:05:23.597151 IP (tos 0x0, ttl 64, id 20451, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xcff3 (correct), 2260258133:2260258133(0) ack 1980060642 win 65535
10:05:23.597153 IP (tos 0x0, ttl 64, id 46950, offset 0, flags [DF], proto TCP (6), length 351) dione.40344 > phoenix.distcc: P, cksum 0x1774 (incorrect (-> 0xe29c), 1980085834:1980086145(311) ack 2260258133 win 5840
10:05:23.597167 IP (tos 0x0, ttl 64, id 20452, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xcbc3 (correct), 2260258133:2260258133(0) ack 1980061714 win 65535
10:05:23.597169 IP (tos 0x0, ttl 64, id 20453, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc793 (correct), 2260258133:2260258133(0) ack 1980062786 win 65535
10:05:23.597219 IP (tos 0x0, ttl 64, id 20454, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xc363 (correct), 2260258133:2260258133(0) ack 1980063858 win 65535
10:05:23.597222 IP (tos 0x0, ttl 64, id 20455, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xbf33 (correct), 2260258133:2260258133(0) ack 1980064930 win 65535
10:05:23.597223 IP (tos 0x0, ttl 64, id 20456, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xbb03 (correct), 2260258133:2260258133(0) ack 1980066002 win 65535
10:05:23.597225 IP (tos 0x0, ttl 64, id 20457, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb6d3 (correct), 2260258133:2260258133(0) ack 1980067074 win 65535
10:05:23.597226 IP (tos 0x0, ttl 64, id 20458, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb2a3 (correct), 2260258133:2260258133(0) ack 1980068146 win 65535
10:05:23.597268 IP (tos 0x0, ttl 64, id 20459, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xae73 (correct), 2260258133:2260258133(0) ack 1980069218 win 65535
10:05:23.597270 IP (tos 0x0, ttl 64, id 20460, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xaa43 (correct), 2260258133:2260258133(0) ack 1980070290 win 65535
10:05:23.597272 IP (tos 0x0, ttl 64, id 20461, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa613 (correct), 2260258133:2260258133(0) ack 1980071362 win 65535
10:05:23.597318 IP (tos 0x0, ttl 64, id 20462, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa1e3 (correct), 2260258133:2260258133(0) ack 1980072434 win 65535
10:05:23.597320 IP (tos 0x0, ttl 64, id 20463, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9db3 (correct), 2260258133:2260258133(0) ack 1980073506 win 65535
10:05:23.597321 IP (tos 0x0, ttl 64, id 20464, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9983 (correct), 2260258133:2260258133(0) ack 1980074578 win 65535
10:05:23.597367 IP (tos 0x0, ttl 64, id 20465, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9553 (correct), 2260258133:2260258133(0) ack 1980075650 win 65535
10:05:23.597369 IP (tos 0x0, ttl 64, id 20466, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9123 (correct), 2260258133:2260258133(0) ack 1980076722 win 65535
10:05:23.597371 IP (tos 0x0, ttl 64, id 20467, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8cf3 (correct), 2260258133:2260258133(0) ack 1980077794 win 65535
10:05:23.597372 IP (tos 0x0, ttl 64, id 20468, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x88c3 (correct), 2260258133:2260258133(0) ack 1980078866 win 65535
10:05:23.597418 IP (tos 0x0, ttl 64, id 20469, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8493 (correct), 2260258133:2260258133(0) ack 1980079938 win 65535
10:05:23.597420 IP (tos 0x0, ttl 64, id 20470, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8063 (correct), 2260258133:2260258133(0) ack 1980081010 win 65535
10:05:23.597421 IP (tos 0x0, ttl 64, id 20471, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7c33 (correct), 2260258133:2260258133(0) ack 1980082082 win 65535
10:05:23.597468 IP (tos 0x0, ttl 64, id 20472, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7803 (correct), 2260258133:2260258133(0) ack 1980083154 win 65535
10:05:23.597470 IP (tos 0x0, ttl 64, id 20473, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x73d3 (correct), 2260258133:2260258133(0) ack 1980084226 win 65535
10:05:23.597471 IP (tos 0x0, ttl 64, id 20474, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6fa3 (correct), 2260258133:2260258133(0) ack 1980085298 win 65535
10:05:23.597473 IP (tos 0x0, ttl 64, id 20475, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6c54 (correct), 2260258133:2260258133(0) ack 1980086145 win 65535
10:05:23.998379 IP (tos 0x0, ttl 64, id 20476, offset 0, flags [DF], proto TCP (6), length 1500) phoenix.distcc > dione.40344: ., cksum 0xdb6a (correct), 2260258133:2260259593(1460) ack 1980086145 win 65535
10:05:23.998388 IP (tos 0x0, ttl 64, id 46951, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0x4468 (correct), 1980086145:1980086145(0) ack 2260259593 win 8760
10:05:23.998391 IP (tos 0x0, ttl 64, id 20477, offset 0, flags [DF], proto TCP (6), length 1500) phoenix.distcc > dione.40344: ., cksum 0xc6e2 (correct), 2260259593:2260261053(1460) ack 1980086145 win 65535
10:05:23.998394 IP (tos 0x0, ttl 64, id 46952, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0x334c (correct), 1980086145:1980086145(0) ack 2260261053 win 11680
10:05:23.998477 IP (tos 0x0, ttl 64, id 20478, offset 0, flags [DF], proto TCP (6), length 1500) phoenix.distcc > dione.40344: ., cksum 0x52a0 (correct), 2260261053:2260262513(1460) ack 1980086145 win 65535
10:05:23.998484 IP (tos 0x0, ttl 64, id 46953, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0x2230 (correct), 1980086145:1980086145(0) ack 2260262513 win 14600
10:05:23.998486 IP (tos 0x0, ttl 64, id 20479, offset 0, flags [DF], proto TCP (6), length 1500) phoenix.distcc > dione.40344: ., cksum 0x577f (correct), 2260262513:2260263973(1460) ack 1980086145 win 65535
10:05:23.998489 IP (tos 0x0, ttl 64, id 46954, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0x1114 (correct), 1980086145:1980086145(0) ack 2260263973 win 17520
10:05:23.998528 IP (tos 0x0, ttl 64, id 20480, offset 0, flags [DF], proto TCP (6), length 1500) phoenix.distcc > dione.40344: ., cksum 0x3212 (correct), 2260263973:2260265433(1460) ack 1980086145 win 65535
10:05:23.998532 IP (tos 0x0, ttl 64, id 46955, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0xfff7 (correct), 1980086145:1980086145(0) ack 2260265433 win 20440
10:05:23.998534 IP (tos 0x0, ttl 64, id 20481, offset 0, flags [DF], proto TCP (6), length 1500) phoenix.distcc > dione.40344: ., cksum 0x0f43 (correct), 2260265433:2260266893(1460) ack 1980086145 win 65535
10:05:23.998537 IP (tos 0x0, ttl 64, id 46956, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0xeedb (correct), 1980086145:1980086145(0) ack 2260266893 win 23360
10:05:23.998578 IP (tos 0x0, ttl 64, id 20482, offset 0, flags [DF], proto TCP (6), length 1500) phoenix.distcc > dione.40344: ., cksum 0xb97b (correct), 2260266893:2260268353(1460) ack 1980086145 win 65535
10:05:23.998582 IP (tos 0x0, ttl 64, id 46957, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0xddbf (correct), 1980086145:1980086145(0) ack 2260268353 win 26280
10:05:23.998586 IP (tos 0x0, ttl 64, id 20483, offset 0, flags [DF], proto TCP (6), length 1500) phoenix.distcc > dione.40344: P, cksum 0x0297 (correct), 2260268353:2260269813(1460) ack 1980086145 win 65535
10:05:23.998588 IP (tos 0x0, ttl 64, id 46958, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0xcca3 (correct), 1980086145:1980086145(0) ack 2260269813 win 29200
10:05:23.998590 IP (tos 0x0, ttl 64, id 20484, offset 0, flags [DF], proto TCP (6), length 924) phoenix.distcc > dione.40344: P, cksum 0x67c5 (correct), 2260269813:2260270697(884) ack 1980086145 win 65535
10:05:23.998599 IP (tos 0x0, ttl 64, id 46959, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0xbdc7 (correct), 1980086145:1980086145(0) ack 2260270697 win 32120
10:05:23.998645 IP (tos 0x0, ttl 64, id 46960, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: F, cksum 0xbdc6 (correct), 1980086145:1980086145(0) ack 2260270697 win 32120
10:05:23.999025 IP (tos 0x0, ttl 64, id 20485, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: F, cksum 0x3b3e (correct), 2260270697:2260270697(0) ack 1980086146 win 65535
10:05:23.999030 IP (tos 0x0, ttl 64, id 46961, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0xbdc5 (correct), 1980086146:1980086146(0) ack 2260270698 win 32120

[traced on the server, phoenix]

10:03:24.304962 IP (tos 0x0, ttl 64, id 46098, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979565914:1979566450(536) ack 2260258133 win 5840
10:03:24.304989 IP (tos 0x0, ttl 64, id 19850, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5a6b (correct), 2260258133:2260258133(0) ack 1979566450 win 65535
10:03:24.305063 IP (tos 0x0, ttl 64, id 46099, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979631449:1979631985(536) ack 2260258133 win 5840
10:03:24.305076 IP (tos 0x0, ttl 64, id 19851, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5a6b (correct), 2260258133:2260258133(0) ack 1979566450 win 65535
10:05:24.324738 IP (tos 0x0, ttl 64, id 46100, offset 0, flags [DF], proto TCP (6), length 183) dione.40344 > phoenix.distcc: . 1979566450:1979566593(143) ack 2260258133 win 5840
10:05:24.324789 IP (tos 0x0, ttl 64, id 19852, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x59dc (correct), 2260258133:2260258133(0) ack 1979566593 win 65535
10:05:24.324910 IP (tos 0x0, ttl 64, id 46101, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979566593:1979567129(536) ack 2260258133 win 5840
10:05:24.324918 IP (tos 0x0, ttl 64, id 46102, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979567129:1979567665(536) ack 2260258133 win 5840
10:05:24.325121 IP (tos 0x0, ttl 64, id 19853, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5a6b (correct), 2260258133:2260258133(0) ack 1979631985 win 65535
10:05:24.325131 IP (tos 0x0, ttl 64, id 19854, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5a6b (correct), 2260258133:2260258133(0) ack 1979631985 win 65535
10:05:24.325282 IP (tos 0x0, ttl 64, id 46103, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979631985:1979632521(536) ack 2260258133 win 5840
10:05:24.325291 IP (tos 0x0, ttl 64, id 19855, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5853 (correct), 2260258133:2260258133(0) ack 1979632521 win 65535
10:05:24.325293 IP (tos 0x0, ttl 64, id 46104, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979632521:1979633057(536) ack 2260258133 win 5840
10:05:24.325300 IP (tos 0x0, ttl 64, id 19856, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x563b (correct), 2260258133:2260258133(0) ack 1979633057 win 65535
10:05:24.325301 IP (tos 0x0, ttl 64, id 46105, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979633057:1979633593(536) ack 2260258133 win 5840
10:05:24.325307 IP (tos 0x0, ttl 64, id 19857, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5423 (correct), 2260258133:2260258133(0) ack 1979633593 win 65535
10:05:24.325308 IP (tos 0x0, ttl 64, id 46106, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979633593:1979634129(536) ack 2260258133 win 5840
10:05:24.325315 IP (tos 0x0, ttl 64, id 19858, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x520b (correct), 2260258133:2260258133(0) ack 1979634129 win 65535
10:05:24.325386 IP (tos 0x0, ttl 64, id 46107, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979634129:1979634665(536) ack 2260258133 win 5840
10:05:24.325398 IP (tos 0x0, ttl 64, id 19859, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4ff3 (correct), 2260258133:2260258133(0) ack 1979634665 win 65535
10:05:24.325399 IP (tos 0x0, ttl 64, id 46108, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979634665:1979635201(536) ack 2260258133 win 5840
10:05:24.325407 IP (tos 0x0, ttl 64, id 19860, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4ddb (correct), 2260258133:2260258133(0) ack 1979635201 win 65535
10:05:24.325409 IP (tos 0x0, ttl 64, id 46109, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979635201:1979635737(536) ack 2260258133 win 5840
10:05:24.325415 IP (tos 0x0, ttl 64, id 19861, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4bc3 (correct), 2260258133:2260258133(0) ack 1979635737 win 65535
10:05:24.325422 IP (tos 0x0, ttl 64, id 46110, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979635737:1979636273(536) ack 2260258133 win 5840
10:05:24.325428 IP (tos 0x0, ttl 64, id 19862, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x49ab (correct), 2260258133:2260258133(0) ack 1979636273 win 65535
10:05:24.325429 IP (tos 0x0, ttl 64, id 46111, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979636273:1979636809(536) ack 2260258133 win 5840
10:05:24.325434 IP (tos 0x0, ttl 64, id 19863, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4793 (correct), 2260258133:2260258133(0) ack 1979636809 win 65535
10:05:24.325435 IP (tos 0x0, ttl 64, id 46112, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979636809:1979637345(536) ack 2260258133 win 5840
10:05:24.325437 IP (tos 0x0, ttl 64, id 46113, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979637345:1979637881(536) ack 2260258133 win 5840
10:05:24.325444 IP (tos 0x0, ttl 64, id 19864, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x457b (correct), 2260258133:2260258133(0) ack 1979637345 win 65535
10:05:24.325449 IP (tos 0x0, ttl 64, id 19865, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4363 (correct), 2260258133:2260258133(0) ack 1979637881 win 65535
10:05:24.325491 IP (tos 0x0, ttl 64, id 46114, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979637881:1979638417(536) ack 2260258133 win 5840
10:05:24.325498 IP (tos 0x0, ttl 64, id 46115, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979638417:1979638953(536) ack 2260258133 win 5840
10:05:24.325502 IP (tos 0x0, ttl 64, id 19866, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x414b (correct), 2260258133:2260258133(0) ack 1979638417 win 65535
10:05:24.325506 IP (tos 0x0, ttl 64, id 46116, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979638953:1979639489(536) ack 2260258133 win 5840
10:05:24.325508 IP (tos 0x0, ttl 64, id 19867, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3f33 (correct), 2260258133:2260258133(0) ack 1979638953 win 65535
10:05:24.325517 IP (tos 0x0, ttl 64, id 19868, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3d1b (correct), 2260258133:2260258133(0) ack 1979639489 win 65535
10:05:24.325519 IP (tos 0x0, ttl 64, id 46117, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979639489:1979640025(536) ack 2260258133 win 5840
10:05:24.325525 IP (tos 0x0, ttl 64, id 19869, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3b03 (correct), 2260258133:2260258133(0) ack 1979640025 win 65535
10:05:24.325526 IP (tos 0x0, ttl 64, id 46118, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979640025:1979640561(536) ack 2260258133 win 5840
10:05:24.325531 IP (tos 0x0, ttl 64, id 19870, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x38eb (correct), 2260258133:2260258133(0) ack 1979640561 win 65535
10:05:24.325532 IP (tos 0x0, ttl 64, id 46119, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979640561:1979641097(536) ack 2260258133 win 5840
10:05:24.325540 IP (tos 0x0, ttl 64, id 19871, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x36d3 (correct), 2260258133:2260258133(0) ack 1979641097 win 65535
10:05:24.325548 IP (tos 0x0, ttl 64, id 46120, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979641097:1979641633(536) ack 2260258133 win 5840
10:05:24.325554 IP (tos 0x0, ttl 64, id 46121, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979641633:1979642169(536) ack 2260258133 win 5840
10:05:24.325559 IP (tos 0x0, ttl 64, id 19872, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x34bb (correct), 2260258133:2260258133(0) ack 1979641633 win 65535
10:05:24.325560 IP (tos 0x0, ttl 64, id 46122, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979642169:1979642705(536) ack 2260258133 win 5840
10:05:24.325563 IP (tos 0x0, ttl 64, id 46123, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979642705:1979643241(536) ack 2260258133 win 5840
10:05:24.325564 IP (tos 0x0, ttl 64, id 19873, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x32a3 (correct), 2260258133:2260258133(0) ack 1979642169 win 65535
10:05:24.325570 IP (tos 0x0, ttl 64, id 19874, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x308b (correct), 2260258133:2260258133(0) ack 1979642705 win 65535
10:05:24.325574 IP (tos 0x0, ttl 64, id 19875, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2e73 (correct), 2260258133:2260258133(0) ack 1979643241 win 65535
10:05:24.325577 IP (tos 0x0, ttl 64, id 46124, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979643241:1979643777(536) ack 2260258133 win 5840
10:05:24.325585 IP (tos 0x0, ttl 64, id 19876, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2c5b (correct), 2260258133:2260258133(0) ack 1979643777 win 65535
10:05:24.325587 IP (tos 0x0, ttl 64, id 46125, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979643777:1979644313(536) ack 2260258133 win 5840
10:05:24.325591 IP (tos 0x0, ttl 64, id 46126, offset 0, flags [DF], proto TCP (6), length 433) dione.40344 > phoenix.distcc: . 1979644313:1979644706(393) ack 2260258133 win 5840
10:05:24.325592 IP (tos 0x0, ttl 64, id 19877, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2a43 (correct), 2260258133:2260258133(0) ack 1979644313 win 65535
10:05:24.325593 IP (tos 0x0, ttl 64, id 46127, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979644706:1979645242(536) ack 2260258133 win 5840
10:05:24.325599 IP (tos 0x0, ttl 64, id 19878, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x28ba (correct), 2260258133:2260258133(0) ack 1979644706 win 65535
10:05:24.325603 IP (tos 0x0, ttl 64, id 19879, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x26a2 (correct), 2260258133:2260258133(0) ack 1979645242 win 65535
10:05:24.325607 IP (tos 0x0, ttl 64, id 46128, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979645242:1979645778(536) ack 2260258133 win 5840
10:05:24.325615 IP (tos 0x0, ttl 64, id 19880, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x248a (correct), 2260258133:2260258133(0) ack 1979645778 win 65535
10:05:24.325617 IP (tos 0x0, ttl 64, id 46129, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979645778:1979646314(536) ack 2260258133 win 5840
10:05:24.325619 IP (tos 0x0, ttl 64, id 46130, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979646314:1979646850(536) ack 2260258133 win 5840
10:05:24.325622 IP (tos 0x0, ttl 64, id 46131, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979646850:1979647386(536) ack 2260258133 win 5840
10:05:24.325627 IP (tos 0x0, ttl 64, id 19881, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2272 (correct), 2260258133:2260258133(0) ack 1979646314 win 65535
10:05:24.325630 IP (tos 0x0, ttl 64, id 46132, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979647386:1979647922(536) ack 2260258133 win 5840
10:05:24.325632 IP (tos 0x0, ttl 64, id 19882, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x205a (correct), 2260258133:2260258133(0) ack 1979646850 win 65535
10:05:24.325637 IP (tos 0x0, ttl 64, id 19883, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1e42 (correct), 2260258133:2260258133(0) ack 1979647386 win 65535
10:05:24.325635 IP (tos 0x0, ttl 64, id 46133, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979647922:1979648458(536) ack 2260258133 win 5840
10:05:24.325641 IP (tos 0x0, ttl 64, id 19884, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1c2a (correct), 2260258133:2260258133(0) ack 1979647922 win 65535
10:05:24.325647 IP (tos 0x0, ttl 64, id 19885, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1a12 (correct), 2260258133:2260258133(0) ack 1979648458 win 65535
10:05:24.325654 IP (tos 0x0, ttl 64, id 46134, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979648458:1979648994(536) ack 2260258133 win 5840
10:05:24.325660 IP (tos 0x0, ttl 64, id 19886, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x17fa (correct), 2260258133:2260258133(0) ack 1979648994 win 65535
10:05:24.325661 IP (tos 0x0, ttl 64, id 46135, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979648994:1979649530(536) ack 2260258133 win 5840
10:05:24.325666 IP (tos 0x0, ttl 64, id 19887, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x15e2 (correct), 2260258133:2260258133(0) ack 1979649530 win 65535
10:05:24.325668 IP (tos 0x0, ttl 64, id 46136, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979649530:1979650066(536) ack 2260258133 win 5840
10:05:24.325671 IP (tos 0x0, ttl 64, id 46137, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979650066:1979650602(536) ack 2260258133 win 5840
10:05:24.325676 IP (tos 0x0, ttl 64, id 19888, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x13ca (correct), 2260258133:2260258133(0) ack 1979650066 win 65535
10:05:24.325680 IP (tos 0x0, ttl 64, id 46138, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979650602:1979651138(536) ack 2260258133 win 5840
10:05:24.325681 IP (tos 0x0, ttl 64, id 19889, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x11b2 (correct), 2260258133:2260258133(0) ack 1979650602 win 65535
10:05:24.325690 IP (tos 0x0, ttl 64, id 19890, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0f9a (correct), 2260258133:2260258133(0) ack 1979651138 win 65535
10:05:24.325692 IP (tos 0x0, ttl 64, id 46139, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979651138:1979651674(536) ack 2260258133 win 5840
10:05:24.325698 IP (tos 0x0, ttl 64, id 19891, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0d82 (correct), 2260258133:2260258133(0) ack 1979651674 win 65535
10:05:24.325700 IP (tos 0x0, ttl 64, id 46140, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979651674:1979652210(536) ack 2260258133 win 5840
10:05:24.325702 IP (tos 0x0, ttl 64, id 46141, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979652210:1979652746(536) ack 2260258133 win 5840
10:05:24.325706 IP (tos 0x0, ttl 64, id 46142, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979652746:1979653282(536) ack 2260258133 win 5840
10:05:24.325706 IP (tos 0x0, ttl 64, id 19892, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0b6a (correct), 2260258133:2260258133(0) ack 1979652210 win 65535
10:05:24.325711 IP (tos 0x0, ttl 64, id 19893, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0952 (correct), 2260258133:2260258133(0) ack 1979652746 win 65535
10:05:24.325714 IP (tos 0x0, ttl 64, id 19894, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x073a (correct), 2260258133:2260258133(0) ack 1979653282 win 65535
10:05:24.325722 IP (tos 0x0, ttl 64, id 46143, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979653282:1979653818(536) ack 2260258133 win 5840
10:05:24.325728 IP (tos 0x0, ttl 64, id 19895, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0522 (correct), 2260258133:2260258133(0) ack 1979653818 win 65535
10:05:24.325730 IP (tos 0x0, ttl 64, id 46144, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979653818:1979654354(536) ack 2260258133 win 5840
10:05:24.325734 IP (tos 0x0, ttl 64, id 46145, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979654354:1979654890(536) ack 2260258133 win 5840
10:05:24.325736 IP (tos 0x0, ttl 64, id 46146, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979654890:1979655426(536) ack 2260258133 win 5840
10:05:24.325737 IP (tos 0x0, ttl 64, id 19896, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x030a (correct), 2260258133:2260258133(0) ack 1979654354 win 65535
10:05:24.325742 IP (tos 0x0, ttl 64, id 19897, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x00f2 (correct), 2260258133:2260258133(0) ack 1979654890 win 65535
10:05:24.325745 IP (tos 0x0, ttl 64, id 46147, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979655426:1979655962(536) ack 2260258133 win 5840
10:05:24.325746 IP (tos 0x0, ttl 64, id 19898, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfed9 (correct), 2260258133:2260258133(0) ack 1979655426 win 65535
10:05:24.325756 IP (tos 0x0, ttl 64, id 19899, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfcc1 (correct), 2260258133:2260258133(0) ack 1979655962 win 65535
10:05:24.325758 IP (tos 0x0, ttl 64, id 46148, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979655962:1979656498(536) ack 2260258133 win 5840
10:05:24.325762 IP (tos 0x0, ttl 64, id 46149, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979656498:1979657034(536) ack 2260258133 win 5840
10:05:24.325764 IP (tos 0x0, ttl 64, id 46150, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979657034:1979657570(536) ack 2260258133 win 5840
10:05:24.325765 IP (tos 0x0, ttl 64, id 19900, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfaa9 (correct), 2260258133:2260258133(0) ack 1979656498 win 65535
10:05:24.325770 IP (tos 0x0, ttl 64, id 19901, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf891 (correct), 2260258133:2260258133(0) ack 1979657034 win 65535
10:05:24.325775 IP (tos 0x0, ttl 64, id 19902, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf679 (correct), 2260258133:2260258133(0) ack 1979657570 win 65535
10:05:24.325781 IP (tos 0x0, ttl 64, id 46151, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979657570:1979658106(536) ack 2260258133 win 5840
10:05:24.325787 IP (tos 0x0, ttl 64, id 19903, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf461 (correct), 2260258133:2260258133(0) ack 1979658106 win 65535
10:05:24.325789 IP (tos 0x0, ttl 64, id 46152, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979658106:1979658642(536) ack 2260258133 win 5840
10:05:24.325791 IP (tos 0x0, ttl 64, id 46153, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979658642:1979659178(536) ack 2260258133 win 5840
10:05:24.325794 IP (tos 0x0, ttl 64, id 46154, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979659178:1979659714(536) ack 2260258133 win 5840
10:05:24.325795 IP (tos 0x0, ttl 64, id 19904, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf249 (correct), 2260258133:2260258133(0) ack 1979658642 win 65535
10:05:24.325796 IP (tos 0x0, ttl 64, id 46155, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979659714:1979660250(536) ack 2260258133 win 5840
10:05:24.325800 IP (tos 0x0, ttl 64, id 19905, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf031 (correct), 2260258133:2260258133(0) ack 1979659178 win 65535
10:05:24.325806 IP (tos 0x0, ttl 64, id 19906, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xee19 (correct), 2260258133:2260258133(0) ack 1979659714 win 65535
10:05:24.325811 IP (tos 0x0, ttl 64, id 19907, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xec01 (correct), 2260258133:2260258133(0) ack 1979660250 win 65535
10:05:24.325814 IP (tos 0x0, ttl 64, id 46156, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979660250:1979660786(536) ack 2260258133 win 5840
10:05:24.325823 IP (tos 0x0, ttl 64, id 19908, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe9e9 (correct), 2260258133:2260258133(0) ack 1979660786 win 65535
10:05:24.325824 IP (tos 0x0, ttl 64, id 46157, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979660786:1979661322(536) ack 2260258133 win 5840
10:05:24.325830 IP (tos 0x0, ttl 64, id 19909, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe7d1 (correct), 2260258133:2260258133(0) ack 1979661322 win 65535
10:05:24.325831 IP (tos 0x0, ttl 64, id 46158, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979661322:1979661858(536) ack 2260258133 win 5840
10:05:24.325833 IP (tos 0x0, ttl 64, id 46159, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979661858:1979662394(536) ack 2260258133 win 5840
10:05:24.325837 IP (tos 0x0, ttl 64, id 46160, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979662394:1979662930(536) ack 2260258133 win 5840
10:05:24.325838 IP (tos 0x0, ttl 64, id 19910, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe5b9 (correct), 2260258133:2260258133(0) ack 1979661858 win 65535
10:05:24.325839 IP (tos 0x0, ttl 64, id 46161, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979662930:1979663466(536) ack 2260258133 win 5840
10:05:24.325841 IP (tos 0x0, ttl 64, id 46162, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979663466:1979664002(536) ack 2260258133 win 5840
10:05:24.325844 IP (tos 0x0, ttl 64, id 19911, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe3a1 (correct), 2260258133:2260258133(0) ack 1979662394 win 65535
10:05:24.325845 IP (tos 0x0, ttl 64, id 46163, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979664002:1979664538(536) ack 2260258133 win 5840
10:05:24.325847 IP (tos 0x0, ttl 64, id 46164, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979664538:1979665074(536) ack 2260258133 win 5840
10:05:24.325850 IP (tos 0x0, ttl 64, id 19912, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe189 (correct), 2260258133:2260258133(0) ack 1979662930 win 65535
10:05:24.326731 IP (tos 0x0, ttl 64, id 46217, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979692946:1979693482(536) ack 2260258133 win 5840
10:05:24.326739 IP (tos 0x0, ttl 64, id 46218, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979693482:1979694018(536) ack 2260258133 win 5840
10:05:24.326741 IP (tos 0x0, ttl 64, id 46219, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979694018:1979694554(536) ack 2260258133 win 5840
10:05:24.326743 IP (tos 0x0, ttl 64, id 46220, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979694554:1979695090(536) ack 2260258133 win 5840
10:05:24.326745 IP (tos 0x0, ttl 64, id 46221, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979695090:1979695626(536) ack 2260258133 win 5840
10:05:24.326746 IP (tos 0x0, ttl 64, id 19969, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6a31 (correct), 2260258133:2260258133(0) ack 1979693482 win 65535
10:05:24.326746 IP (tos 0x0, ttl 64, id 46222, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979695626:1979696162(536) ack 2260258133 win 5840
10:05:24.326749 IP (tos 0x0, ttl 64, id 46223, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979696162:1979696698(536) ack 2260258133 win 5840
10:05:24.326751 IP (tos 0x0, ttl 64, id 19970, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6819 (correct), 2260258133:2260258133(0) ack 1979694018 win 65535
10:05:24.326753 IP (tos 0x0, ttl 64, id 46224, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979696698:1979697234(536) ack 2260258133 win 5840
10:05:24.326756 IP (tos 0x0, ttl 64, id 46225, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979697234:1979697770(536) ack 2260258133 win 5840
10:05:24.326757 IP (tos 0x0, ttl 64, id 19971, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6601 (correct), 2260258133:2260258133(0) ack 1979694554 win 65535
10:05:24.326760 IP (tos 0x0, ttl 64, id 46226, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979697770:1979698306(536) ack 2260258133 win 5840
10:05:24.326763 IP (tos 0x0, ttl 64, id 19972, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x63e9 (correct), 2260258133:2260258133(0) ack 1979695090 win 65535
10:05:24.326764 IP (tos 0x0, ttl 64, id 46227, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979698306:1979698842(536) ack 2260258133 win 5840
10:05:24.326767 IP (tos 0x0, ttl 64, id 46228, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979698842:1979699378(536) ack 2260258133 win 5840
10:05:24.326768 IP (tos 0x0, ttl 64, id 19973, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x61d1 (correct), 2260258133:2260258133(0) ack 1979695626 win 65535
10:05:24.326771 IP (tos 0x0, ttl 64, id 46229, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979699378:1979699914(536) ack 2260258133 win 5840
10:05:24.326772 IP (tos 0x0, ttl 64, id 19974, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5fb9 (correct), 2260258133:2260258133(0) ack 1979696162 win 65535
10:05:24.326775 IP (tos 0x0, ttl 64, id 46230, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979699914:1979700450(536) ack 2260258133 win 5840
10:05:24.326778 IP (tos 0x0, ttl 64, id 19975, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5da1 (correct), 2260258133:2260258133(0) ack 1979696698 win 65535
10:05:24.326780 IP (tos 0x0, ttl 64, id 46231, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979700450:1979700986(536) ack 2260258133 win 5840
10:05:24.326783 IP (tos 0x0, ttl 64, id 19976, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5b89 (correct), 2260258133:2260258133(0) ack 1979697234 win 65535
10:05:24.326783 IP (tos 0x0, ttl 64, id 46232, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979700986:1979701522(536) ack 2260258133 win 5840
10:05:24.326787 IP (tos 0x0, ttl 64, id 46233, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979701522:1979702058(536) ack 2260258133 win 5840
10:05:24.326789 IP (tos 0x0, ttl 64, id 19977, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5971 (correct), 2260258133:2260258133(0) ack 1979697770 win 65535
10:05:24.326791 IP (tos 0x0, ttl 64, id 46234, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979702058:1979702594(536) ack 2260258133 win 5840
10:05:24.326793 IP (tos 0x0, ttl 64, id 19978, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5759 (correct), 2260258133:2260258133(0) ack 1979698306 win 65535
10:05:24.326795 IP (tos 0x0, ttl 64, id 46235, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979702594:1979703130(536) ack 2260258133 win 5840
10:05:24.326798 IP (tos 0x0, ttl 64, id 19979, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5541 (correct), 2260258133:2260258133(0) ack 1979698842 win 65535
10:05:24.326798 IP (tos 0x0, ttl 64, id 46236, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979703130:1979703666(536) ack 2260258133 win 5840
10:05:24.326802 IP (tos 0x0, ttl 64, id 46237, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979703666:1979704202(536) ack 2260258133 win 5840
10:05:24.326803 IP (tos 0x0, ttl 64, id 19980, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5329 (correct), 2260258133:2260258133(0) ack 1979699378 win 65535
10:05:24.326805 IP (tos 0x0, ttl 64, id 46238, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979704202:1979704738(536) ack 2260258133 win 5840
10:05:24.326808 IP (tos 0x0, ttl 64, id 19981, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5111 (correct), 2260258133:2260258133(0) ack 1979699914 win 65535
10:05:24.326810 IP (tos 0x0, ttl 64, id 46239, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979704738:1979705274(536) ack 2260258133 win 5840
10:05:24.326813 IP (tos 0x0, ttl 64, id 19982, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4ef9 (correct), 2260258133:2260258133(0) ack 1979700450 win 65535
10:05:24.326815 IP (tos 0x0, ttl 64, id 46240, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979705274:1979705810(536) ack 2260258133 win 5840
10:05:24.326818 IP (tos 0x0, ttl 64, id 19983, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4ce1 (correct), 2260258133:2260258133(0) ack 1979700986 win 65535
10:05:24.326817 IP (tos 0x0, ttl 64, id 46241, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979705810:1979706346(536) ack 2260258133 win 5840
10:05:24.326822 IP (tos 0x0, ttl 64, id 46242, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979706346:1979706882(536) ack 2260258133 win 5840
10:05:24.326823 IP (tos 0x0, ttl 64, id 19984, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4ac9 (correct), 2260258133:2260258133(0) ack 1979701522 win 65535
10:05:24.326826 IP (tos 0x0, ttl 64, id 46243, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979706882:1979707418(536) ack 2260258133 win 5840
10:05:24.326828 IP (tos 0x0, ttl 64, id 19985, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x48b1 (correct), 2260258133:2260258133(0) ack 1979702058 win 65535
10:05:24.326831 IP (tos 0x0, ttl 64, id 46244, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979707418:1979707954(536) ack 2260258133 win 5840
10:05:24.326833 IP (tos 0x0, ttl 64, id 19986, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4699 (correct), 2260258133:2260258133(0) ack 1979702594 win 65535
10:05:24.326836 IP (tos 0x0, ttl 64, id 46245, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979707954:1979708490(536) ack 2260258133 win 5840
10:05:24.326838 IP (tos 0x0, ttl 64, id 19987, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4481 (correct), 2260258133:2260258133(0) ack 1979703130 win 65535
10:05:24.326842 IP (tos 0x0, ttl 64, id 46246, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979708490:1979709026(536) ack 2260258133 win 5840
10:05:24.326844 IP (tos 0x0, ttl 64, id 19988, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4269 (correct), 2260258133:2260258133(0) ack 1979703666 win 65535
10:05:24.326847 IP (tos 0x0, ttl 64, id 46247, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979709026:1979709562(536) ack 2260258133 win 5840
10:05:24.326851 IP (tos 0x0, ttl 64, id 46248, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: P 1979709562:1979710098(536) ack 2260258133 win 5840
10:05:24.326852 IP (tos 0x0, ttl 64, id 19989, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3e39 (correct), 2260258133:2260258133(0) ack 1979704738 win 65535
10:05:24.326853 IP (tos 0x0, ttl 64, id 46249, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979710098:1979710634(536) ack 2260258133 win 5840
10:05:24.326856 IP (tos 0x0, ttl 64, id 46250, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979710634:1979711170(536) ack 2260258133 win 5840
10:05:24.326858 IP (tos 0x0, ttl 64, id 19990, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3a09 (correct), 2260258133:2260258133(0) ack 1979705810 win 65535
10:05:24.326860 IP (tos 0x0, ttl 64, id 46251, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979711170:1979711706(536) ack 2260258133 win 5840
10:05:24.326864 IP (tos 0x0, ttl 64, id 19991, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x35d9 (correct), 2260258133:2260258133(0) ack 1979706882 win 65535
10:05:24.326865 IP (tos 0x0, ttl 64, id 46252, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979711706:1979712242(536) ack 2260258133 win 5840
10:05:24.326868 IP (tos 0x0, ttl 64, id 46253, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979712242:1979712778(536) ack 2260258133 win 5840
10:05:24.326871 IP (tos 0x0, ttl 64, id 19992, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x31a9 (correct), 2260258133:2260258133(0) ack 1979707954 win 65535
10:05:24.326872 IP (tos 0x0, ttl 64, id 46254, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979712778:1979713314(536) ack 2260258133 win 5840
10:05:24.326875 IP (tos 0x0, ttl 64, id 46255, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979713314:1979713850(536) ack 2260258133 win 5840
10:05:24.326877 IP (tos 0x0, ttl 64, id 19993, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2d79 (correct), 2260258133:2260258133(0) ack 1979709026 win 65535
10:05:24.326880 IP (tos 0x0, ttl 64, id 46256, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979713850:1979714386(536) ack 2260258133 win 5840
10:05:24.326884 IP (tos 0x0, ttl 64, id 19994, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2949 (correct), 2260258133:2260258133(0) ack 1979710098 win 65535
10:05:24.326885 IP (tos 0x0, ttl 64, id 46257, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979714386:1979714922(536) ack 2260258133 win 5840
10:05:24.326887 IP (tos 0x0, ttl 64, id 46258, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979714922:1979715458(536) ack 2260258133 win 5840
10:05:24.326890 IP (tos 0x0, ttl 64, id 19995, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2519 (correct), 2260258133:2260258133(0) ack 1979711170 win 65535
10:05:24.326891 IP (tos 0x0, ttl 64, id 46259, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979715458:1979715994(536) ack 2260258133 win 5840
10:05:24.326894 IP (tos 0x0, ttl 64, id 46260, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979715994:1979716530(536) ack 2260258133 win 5840
10:05:24.326896 IP (tos 0x0, ttl 64, id 19996, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x20e9 (correct), 2260258133:2260258133(0) ack 1979712242 win 65535
10:05:24.326898 IP (tos 0x0, ttl 64, id 46261, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979716530:1979717066(536) ack 2260258133 win 5840
10:05:24.326903 IP (tos 0x0, ttl 64, id 19997, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1cb9 (correct), 2260258133:2260258133(0) ack 1979713314 win 65535
10:05:24.326909 IP (tos 0x0, ttl 64, id 19998, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1889 (correct), 2260258133:2260258133(0) ack 1979714386 win 65535
10:05:24.326914 IP (tos 0x0, ttl 64, id 19999, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1459 (correct), 2260258133:2260258133(0) ack 1979715458 win 65535
10:05:24.326919 IP (tos 0x0, ttl 64, id 20000, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1029 (correct), 2260258133:2260258133(0) ack 1979716530 win 65535
10:05:24.326921 IP (tos 0x0, ttl 64, id 46262, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979717066:1979717602(536) ack 2260258133 win 5840
10:05:24.326929 IP (tos 0x0, ttl 64, id 20001, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0bf9 (correct), 2260258133:2260258133(0) ack 1979717602 win 65535
10:05:24.326931 IP (tos 0x0, ttl 64, id 46263, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979717602:1979718138(536) ack 2260258133 win 5840
10:05:24.326933 IP (tos 0x0, ttl 64, id 46264, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979718138:1979718674(536) ack 2260258133 win 5840
10:05:24.326938 IP (tos 0x0, ttl 64, id 20002, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x07c9 (correct), 2260258133:2260258133(0) ack 1979718674 win 65535
10:05:24.326939 IP (tos 0x0, ttl 64, id 46265, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979718674:1979719210(536) ack 2260258133 win 5840
10:05:24.326941 IP (tos 0x0, ttl 64, id 46266, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979719210:1979719746(536) ack 2260258133 win 5840
10:05:24.326945 IP (tos 0x0, ttl 64, id 20003, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0399 (correct), 2260258133:2260258133(0) ack 1979719746 win 65535
10:05:24.326946 IP (tos 0x0, ttl 64, id 46267, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979719746:1979720282(536) ack 2260258133 win 5840
10:05:24.326948 IP (tos 0x0, ttl 64, id 46268, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979720282:1979720818(536) ack 2260258133 win 5840
10:05:24.326953 IP (tos 0x0, ttl 64, id 20004, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xff68 (correct), 2260258133:2260258133(0) ack 1979720818 win 65535
10:05:24.326954 IP (tos 0x0, ttl 64, id 46269, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979720818:1979721354(536) ack 2260258133 win 5840
10:05:24.326956 IP (tos 0x0, ttl 64, id 46270, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979721354:1979721890(536) ack 2260258133 win 5840
10:05:24.326960 IP (tos 0x0, ttl 64, id 20005, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfb38 (correct), 2260258133:2260258133(0) ack 1979721890 win 65535
10:05:24.326961 IP (tos 0x0, ttl 64, id 46271, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979721890:1979722426(536) ack 2260258133 win 5840
10:05:24.326990 IP (tos 0x0, ttl 64, id 46272, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979722426:1979722962(536) ack 2260258133 win 5840
10:05:24.326993 IP (tos 0x0, ttl 64, id 20006, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf708 (correct), 2260258133:2260258133(0) ack 1979722962 win 65535
10:05:24.327035 IP (tos 0x0, ttl 64, id 46281, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979727250:1979727786(536) ack 2260258133 win 5840
10:05:24.327076 IP (tos 0x0, ttl 64, id 46286, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979729930:1979730466(536) ack 2260258133 win 5840
10:05:24.327091 IP (tos 0x0, ttl 64, id 46288, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979731002:1979731538(536) ack 2260258133 win 5840
10:05:24.327151 IP (tos 0x0, ttl 64, id 46296, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979735290:1979735826(536) ack 2260258133 win 5840
10:05:24.327229 IP (tos 0x0, ttl 64, id 46306, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979740650:1979741186(536) ack 2260258133 win 5840
10:05:24.327306 IP (tos 0x0, ttl 64, id 20028, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9ae8 (correct), 2260258133:2260258133(0) ack 1979746546 win 65535
10:05:24.327337 IP (tos 0x0, ttl 64, id 46320, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979748154:1979748690(536) ack 2260258133 win 5840
10:05:24.327362 IP (tos 0x0, ttl 64, id 20032, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8a28 (correct), 2260258133:2260258133(0) ack 1979750834 win 65535
10:05:24.327375 IP (tos 0x0, ttl 64, id 20033, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x85f8 (correct), 2260258133:2260258133(0) ack 1979751906 win 65535
10:05:24.327386 IP (tos 0x0, ttl 64, id 20034, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x81c8 (correct), 2260258133:2260258133(0) ack 1979752978 win 65535
10:05:24.327410 IP (tos 0x0, ttl 64, id 20035, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7d98 (correct), 2260258133:2260258133(0) ack 1979754050 win 65535
10:05:24.327436 IP (tos 0x0, ttl 64, id 20037, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7538 (correct), 2260258133:2260258133(0) ack 1979756194 win 65535
10:05:24.327473 IP (tos 0x0, ttl 64, id 46339, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979758338:1979758874(536) ack 2260258133 win 5840
10:05:24.327497 IP (tos 0x0, ttl 64, id 20041, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6478 (correct), 2260258133:2260258133(0) ack 1979760482 win 65535
10:05:24.327517 IP (tos 0x0, ttl 64, id 20042, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x6048 (correct), 2260258133:2260258133(0) ack 1979761554 win 65535
10:05:24.327528 IP (tos 0x0, ttl 64, id 20043, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x5c18 (correct), 2260258133:2260258133(0) ack 1979762626 win 65535
10:05:24.327539 IP (tos 0x0, ttl 64, id 20044, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x57e8 (correct), 2260258133:2260258133(0) ack 1979763698 win 65535
10:05:24.327556 IP (tos 0x0, ttl 64, id 20045, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x53b8 (correct), 2260258133:2260258133(0) ack 1979764770 win 65535
10:05:24.327606 IP (tos 0x0, ttl 64, id 46357, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979767986:1979768522(536) ack 2260258133 win 5840
10:05:24.327649 IP (tos 0x0, ttl 64, id 20051, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3a98 (correct), 2260258133:2260258133(0) ack 1979771202 win 65535
10:05:24.327694 IP (tos 0x0, ttl 64, id 46369, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979774418:1979774954(536) ack 2260258133 win 5840
10:05:24.327754 IP (tos 0x0, ttl 64, id 20058, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1d48 (correct), 2260258133:2260258133(0) ack 1979778706 win 65535
10:05:24.327765 IP (tos 0x0, ttl 64, id 20059, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1918 (correct), 2260258133:2260258133(0) ack 1979779778 win 65535
10:05:24.327780 IP (tos 0x0, ttl 64, id 20060, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x14e8 (correct), 2260258133:2260258133(0) ack 1979780850 win 65535
10:05:24.327794 IP (tos 0x0, ttl 64, id 20061, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x10b8 (correct), 2260258133:2260258133(0) ack 1979781922 win 65535
10:05:24.327816 IP (tos 0x0, ttl 64, id 20062, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0c88 (correct), 2260258133:2260258133(0) ack 1979782994 win 65535
10:05:24.327842 IP (tos 0x0, ttl 64, id 20064, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0428 (correct), 2260258133:2260258133(0) ack 1979785138 win 65535
10:05:24.327863 IP (tos 0x0, ttl 64, id 20065, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfff7 (correct), 2260258133:2260258133(0) ack 1979786210 win 65535
10:05:24.327874 IP (tos 0x0, ttl 64, id 20066, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfbc7 (correct), 2260258133:2260258133(0) ack 1979787282 win 65535
10:05:24.327890 IP (tos 0x0, ttl 64, id 20067, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf797 (correct), 2260258133:2260258133(0) ack 1979788354 win 65535
10:05:24.327914 IP (tos 0x0, ttl 64, id 20068, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf367 (correct), 2260258133:2260258133(0) ack 1979789426 win 65535
10:05:24.327921 IP (tos 0x0, ttl 64, id 20069, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xef37 (correct), 2260258133:2260258133(0) ack 1979790498 win 65535
10:05:24.327934 IP (tos 0x0, ttl 64, id 20070, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xeb07 (correct), 2260258133:2260258133(0) ack 1979791570 win 65535
10:05:24.327947 IP (tos 0x0, ttl 64, id 20071, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe6d7 (correct), 2260258133:2260258133(0) ack 1979792642 win 65535
10:05:24.327972 IP (tos 0x0, ttl 64, id 20072, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe2a7 (correct), 2260258133:2260258133(0) ack 1979793714 win 65535
10:05:24.328005 IP (tos 0x0, ttl 64, id 46410, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979796394:1979796930(536) ack 2260258133 win 5840
10:05:24.328039 IP (tos 0x0, ttl 64, id 20077, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xcdb7 (correct), 2260258133:2260258133(0) ack 1979799074 win 65535
10:05:24.328086 IP (tos 0x0, ttl 64, id 46422, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979802826:1979803362(536) ack 2260258133 win 5840
10:05:24.328122 IP (tos 0x0, ttl 64, id 20083, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb497 (correct), 2260258133:2260258133(0) ack 1979805506 win 65535
10:05:24.328135 IP (tos 0x0, ttl 64, id 20084, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xb067 (correct), 2260258133:2260258133(0) ack 1979806578 win 65535
10:05:24.328151 IP (tos 0x0, ttl 64, id 20085, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xac37 (correct), 2260258133:2260258133(0) ack 1979807650 win 65535
10:05:24.328164 IP (tos 0x0, ttl 64, id 20086, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa807 (correct), 2260258133:2260258133(0) ack 1979808722 win 65535
10:05:24.328185 IP (tos 0x0, ttl 64, id 46434, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979809258:1979809794(536) ack 2260258133 win 5840
10:05:24.328197 IP (tos 0x0, ttl 64, id 20087, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa3d7 (correct), 2260258133:2260258133(0) ack 1979809794 win 65535
10:05:24.328204 IP (tos 0x0, ttl 64, id 20088, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9fa7 (correct), 2260258133:2260258133(0) ack 1979810866 win 65535
10:05:24.328248 IP (tos 0x0, ttl 64, id 46444, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979814618:1979815154(536) ack 2260258133 win 5840
10:05:24.328293 IP (tos 0x0, ttl 64, id 20094, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8687 (correct), 2260258133:2260258133(0) ack 1979817298 win 65535
10:05:24.328352 IP (tos 0x0, ttl 64, id 46458, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979822122:1979822658(536) ack 2260258133 win 5840
10:05:24.328396 IP (tos 0x0, ttl 64, id 46464, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979825338:1979825874(536) ack 2260258133 win 5840
10:05:24.328426 IP (tos 0x0, ttl 64, id 20103, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x60d7 (correct), 2260258133:2260258133(0) ack 1979826946 win 65535
10:05:24.328459 IP (tos 0x0, ttl 64, id 46472, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979829626:1979830162(536) ack 2260258133 win 5840
10:05:24.328524 IP (tos 0x0, ttl 64, id 20110, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4387 (correct), 2260258133:2260258133(0) ack 1979834450 win 65535
10:05:24.328535 IP (tos 0x0, ttl 64, id 46482, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979834986:1979835522(536) ack 2260258133 win 5840
10:05:24.328569 IP (tos 0x0, ttl 64, id 20113, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x36f7 (correct), 2260258133:2260258133(0) ack 1979837666 win 65535
10:05:24.328626 IP (tos 0x0, ttl 64, id 46494, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979841418:1979841954(536) ack 2260258133 win 5840
10:05:24.328684 IP (tos 0x0, ttl 64, id 20121, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1577 (correct), 2260258133:2260258133(0) ack 1979846242 win 65535
10:05:24.328710 IP (tos 0x0, ttl 64, id 46506, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979847850:1979848386(536) ack 2260258133 win 5840
10:05:24.328743 IP (tos 0x0, ttl 64, id 46510, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979849994:1979850530(536) ack 2260258133 win 5840
10:05:24.328755 IP (tos 0x0, ttl 64, id 20125, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x04b7 (correct), 2260258133:2260258133(0) ack 1979850530 win 65535
10:05:24.328836 IP (tos 0x0, ttl 64, id 46523, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979856962:1979857498(536) ack 2260258133 win 5840
10:05:24.328862 IP (tos 0x0, ttl 64, id 20133, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe336 (correct), 2260258133:2260258133(0) ack 1979859106 win 65535
10:05:24.328978 IP (tos 0x0, ttl 64, id 46541, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1979866610:1979867146(536) ack 2260258133 win 5840
10:05:24.329135 IP (tos 0x0, ttl 64, id 20168, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x99ee (correct), 2260258133:2260258133(0) ack 1979877866 win 65535
10:05:24.331208 IP (tos 0x0, ttl 64, id 46844, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980029018:1980029554(536) ack 2260258133 win 5840
10:05:24.331216 IP (tos 0x0, ttl 64, id 46845, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980029554:1980030090(536) ack 2260258133 win 5840
10:05:24.331218 IP (tos 0x0, ttl 64, id 46846, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980030090:1980030626(536) ack 2260258133 win 5840
10:05:24.331220 IP (tos 0x0, ttl 64, id 46847, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980030626:1980031162(536) ack 2260258133 win 5840
10:05:24.331222 IP (tos 0x0, ttl 64, id 46848, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980031162:1980031698(536) ack 2260258133 win 5840
10:05:24.331221 IP (tos 0x0, ttl 64, id 20422, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4964 (correct), 2260258133:2260258133(0) ack 1980029554 win 65535
10:05:24.331223 IP (tos 0x0, ttl 64, id 46849, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980031698:1980032234(536) ack 2260258133 win 5840
10:05:24.331225 IP (tos 0x0, ttl 64, id 46850, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980032234:1980032770(536) ack 2260258133 win 5840
10:05:24.331229 IP (tos 0x0, ttl 64, id 20423, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4534 (correct), 2260258133:2260258133(0) ack 1980030626 win 65535
10:05:24.331229 IP (tos 0x0, ttl 64, id 46851, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980032770:1980033306(536) ack 2260258133 win 5840
10:05:24.331232 IP (tos 0x0, ttl 64, id 46852, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980033306:1980033842(536) ack 2260258133 win 5840
10:05:24.331236 IP (tos 0x0, ttl 64, id 46853, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980033842:1980034378(536) ack 2260258133 win 5840
10:05:24.331236 IP (tos 0x0, ttl 64, id 20424, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x4104 (correct), 2260258133:2260258133(0) ack 1980031698 win 65535
10:05:24.331238 IP (tos 0x0, ttl 64, id 46854, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980034378:1980034914(536) ack 2260258133 win 5840
10:05:24.331240 IP (tos 0x0, ttl 64, id 46855, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980034914:1980035450(536) ack 2260258133 win 5840
10:05:24.331242 IP (tos 0x0, ttl 64, id 20425, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3cd4 (correct), 2260258133:2260258133(0) ack 1980032770 win 65535
10:05:24.331244 IP (tos 0x0, ttl 64, id 46856, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980035450:1980035986(536) ack 2260258133 win 5840
10:05:24.331248 IP (tos 0x0, ttl 64, id 20426, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x38a4 (correct), 2260258133:2260258133(0) ack 1980033842 win 65535
10:05:24.331249 IP (tos 0x0, ttl 64, id 46857, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980035986:1980036522(536) ack 2260258133 win 5840
10:05:24.331251 IP (tos 0x0, ttl 64, id 46858, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: P 1980036522:1980037058(536) ack 2260258133 win 5840
10:05:24.331254 IP (tos 0x0, ttl 64, id 20427, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3474 (correct), 2260258133:2260258133(0) ack 1980034914 win 65535
10:05:24.331255 IP (tos 0x0, ttl 64, id 46859, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980037058:1980037594(536) ack 2260258133 win 5840
10:05:24.331258 IP (tos 0x0, ttl 64, id 46860, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980037594:1980038130(536) ack 2260258133 win 5840
10:05:24.331262 IP (tos 0x0, ttl 64, id 46861, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980038130:1980038666(536) ack 2260258133 win 5840
10:05:24.331262 IP (tos 0x0, ttl 64, id 20428, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x3044 (correct), 2260258133:2260258133(0) ack 1980035986 win 65535
10:05:24.331264 IP (tos 0x0, ttl 64, id 46862, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980038666:1980039202(536) ack 2260258133 win 5840
10:05:24.331266 IP (tos 0x0, ttl 64, id 46863, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980039202:1980039738(536) ack 2260258133 win 5840
10:05:24.331268 IP (tos 0x0, ttl 64, id 20429, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x2c14 (correct), 2260258133:2260258133(0) ack 1980037058 win 65535
10:05:24.331270 IP (tos 0x0, ttl 64, id 46864, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980039738:1980040274(536) ack 2260258133 win 5840
10:05:24.331275 IP (tos 0x0, ttl 64, id 46865, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980040274:1980040810(536) ack 2260258133 win 5840
10:05:24.331274 IP (tos 0x0, ttl 64, id 20430, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x27e4 (correct), 2260258133:2260258133(0) ack 1980038130 win 65535
10:05:24.331277 IP (tos 0x0, ttl 64, id 46866, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980040810:1980041346(536) ack 2260258133 win 5840
10:05:24.331281 IP (tos 0x0, ttl 64, id 20431, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x23b4 (correct), 2260258133:2260258133(0) ack 1980039202 win 65535
10:05:24.331282 IP (tos 0x0, ttl 64, id 46867, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980041346:1980041882(536) ack 2260258133 win 5840
10:05:24.331284 IP (tos 0x0, ttl 64, id 46868, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980041882:1980042418(536) ack 2260258133 win 5840
10:05:24.331288 IP (tos 0x0, ttl 64, id 46869, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980042418:1980042954(536) ack 2260258133 win 5840
10:05:24.331288 IP (tos 0x0, ttl 64, id 20432, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1f84 (correct), 2260258133:2260258133(0) ack 1980040274 win 65535
10:05:24.331290 IP (tos 0x0, ttl 64, id 46870, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980042954:1980043490(536) ack 2260258133 win 5840
10:05:24.331295 IP (tos 0x0, ttl 64, id 46871, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980043490:1980044026(536) ack 2260258133 win 5840
10:05:24.331294 IP (tos 0x0, ttl 64, id 20433, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1b54 (correct), 2260258133:2260258133(0) ack 1980041346 win 65535
10:05:24.331297 IP (tos 0x0, ttl 64, id 46872, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980044026:1980044562(536) ack 2260258133 win 5840
10:05:24.331301 IP (tos 0x0, ttl 64, id 46873, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980044562:1980045098(536) ack 2260258133 win 5840
10:05:24.331301 IP (tos 0x0, ttl 64, id 20434, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x1724 (correct), 2260258133:2260258133(0) ack 1980042418 win 65535
10:05:24.331303 IP (tos 0x0, ttl 64, id 46874, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980045098:1980045634(536) ack 2260258133 win 5840
10:05:24.331308 IP (tos 0x0, ttl 64, id 20435, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x12f4 (correct), 2260258133:2260258133(0) ack 1980043490 win 65535
10:05:24.331305 IP (tos 0x0, ttl 64, id 46875, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980045634:1980046170(536) ack 2260258133 win 5840
10:05:24.331311 IP (tos 0x0, ttl 64, id 46876, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980046170:1980046706(536) ack 2260258133 win 5840
10:05:24.331314 IP (tos 0x0, ttl 64, id 20436, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0ec4 (correct), 2260258133:2260258133(0) ack 1980044562 win 65535
10:05:24.331315 IP (tos 0x0, ttl 64, id 46877, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980046706:1980047242(536) ack 2260258133 win 5840
10:05:24.331317 IP (tos 0x0, ttl 64, id 46878, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980047242:1980047778(536) ack 2260258133 win 5840
10:05:24.331319 IP (tos 0x0, ttl 64, id 46879, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980047778:1980048314(536) ack 2260258133 win 5840
10:05:24.331320 IP (tos 0x0, ttl 64, id 20437, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0a94 (correct), 2260258133:2260258133(0) ack 1980045634 win 65535
10:05:24.331323 IP (tos 0x0, ttl 64, id 46880, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980048314:1980048850(536) ack 2260258133 win 5840
10:05:24.331326 IP (tos 0x0, ttl 64, id 20438, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0664 (correct), 2260258133:2260258133(0) ack 1980046706 win 65535
10:05:24.331327 IP (tos 0x0, ttl 64, id 46881, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980048850:1980049386(536) ack 2260258133 win 5840
10:05:24.331330 IP (tos 0x0, ttl 64, id 46882, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980049386:1980049922(536) ack 2260258133 win 5840
10:05:24.331333 IP (tos 0x0, ttl 64, id 20439, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x0234 (correct), 2260258133:2260258133(0) ack 1980047778 win 65535
10:05:24.331334 IP (tos 0x0, ttl 64, id 46883, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980049922:1980050458(536) ack 2260258133 win 5840
10:05:24.331339 IP (tos 0x0, ttl 64, id 46884, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980050458:1980050994(536) ack 2260258133 win 5840
10:05:24.331340 IP (tos 0x0, ttl 64, id 20440, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xfe03 (correct), 2260258133:2260258133(0) ack 1980048850 win 65535
10:05:24.331341 IP (tos 0x0, ttl 64, id 46885, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980050994:1980051530(536) ack 2260258133 win 5840
10:05:24.331344 IP (tos 0x0, ttl 64, id 46886, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980051530:1980052066(536) ack 2260258133 win 5840
10:05:24.331346 IP (tos 0x0, ttl 64, id 20441, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf9d3 (correct), 2260258133:2260258133(0) ack 1980049922 win 65535
10:05:24.331348 IP (tos 0x0, ttl 64, id 46887, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980052066:1980052602(536) ack 2260258133 win 5840
10:05:24.331352 IP (tos 0x0, ttl 64, id 20442, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf5a3 (correct), 2260258133:2260258133(0) ack 1980050994 win 65535
10:05:24.331353 IP (tos 0x0, ttl 64, id 46888, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980052602:1980053138(536) ack 2260258133 win 5840
10:05:24.331355 IP (tos 0x0, ttl 64, id 46889, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980053138:1980053674(536) ack 2260258133 win 5840
10:05:24.331359 IP (tos 0x0, ttl 64, id 20443, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xf173 (correct), 2260258133:2260258133(0) ack 1980052066 win 65535
10:05:24.331359 IP (tos 0x0, ttl 64, id 46890, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980053674:1980054210(536) ack 2260258133 win 5840
10:05:24.331361 IP (tos 0x0, ttl 64, id 46891, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980054210:1980054746(536) ack 2260258133 win 5840
10:05:24.331365 IP (tos 0x0, ttl 64, id 20444, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xed43 (correct), 2260258133:2260258133(0) ack 1980053138 win 65535
10:05:24.331366 IP (tos 0x0, ttl 64, id 46892, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980054746:1980055282(536) ack 2260258133 win 5840
10:05:24.331368 IP (tos 0x0, ttl 64, id 46893, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980055282:1980055818(536) ack 2260258133 win 5840
10:05:24.331372 IP (tos 0x0, ttl 64, id 20445, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe913 (correct), 2260258133:2260258133(0) ack 1980054210 win 65535
10:05:24.331372 IP (tos 0x0, ttl 64, id 46894, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980055818:1980056354(536) ack 2260258133 win 5840
10:05:24.331375 IP (tos 0x0, ttl 64, id 46895, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980056354:1980056890(536) ack 2260258133 win 5840
10:05:24.331379 IP (tos 0x0, ttl 64, id 46896, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980056890:1980057426(536) ack 2260258133 win 5840
10:05:24.331379 IP (tos 0x0, ttl 64, id 20446, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe4e3 (correct), 2260258133:2260258133(0) ack 1980055282 win 65535
10:05:24.331381 IP (tos 0x0, ttl 64, id 46897, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980057426:1980057962(536) ack 2260258133 win 5840
10:05:24.331385 IP (tos 0x0, ttl 64, id 20447, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xe0b3 (correct), 2260258133:2260258133(0) ack 1980056354 win 65535
10:05:24.331386 IP (tos 0x0, ttl 64, id 46898, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980057962:1980058498(536) ack 2260258133 win 5840
10:05:24.331388 IP (tos 0x0, ttl 64, id 46899, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980058498:1980059034(536) ack 2260258133 win 5840
10:05:24.331390 IP (tos 0x0, ttl 64, id 46900, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980059034:1980059570(536) ack 2260258133 win 5840
10:05:24.331392 IP (tos 0x0, ttl 64, id 20448, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xdc83 (correct), 2260258133:2260258133(0) ack 1980057426 win 65535
10:05:24.331394 IP (tos 0x0, ttl 64, id 46901, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980059570:1980060106(536) ack 2260258133 win 5840
10:05:24.331399 IP (tos 0x0, ttl 64, id 20449, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd853 (correct), 2260258133:2260258133(0) ack 1980058498 win 65535
10:05:24.331405 IP (tos 0x0, ttl 64, id 20450, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xd423 (correct), 2260258133:2260258133(0) ack 1980059570 win 65535
10:05:24.331411 IP (tos 0x0, ttl 64, id 20451, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xcff3 (correct), 2260258133:2260258133(0) ack 1980060642 win 65535
10:05:24.331524 IP (tos 0x0, ttl 64, id 20460, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xaa43 (correct), 2260258133:2260258133(0) ack 1980070290 win 65535
10:05:24.331531 IP (tos 0x0, ttl 64, id 20461, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa613 (correct), 2260258133:2260258133(0) ack 1980071362 win 65535
10:05:24.331537 IP (tos 0x0, ttl 64, id 20462, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0xa1e3 (correct), 2260258133:2260258133(0) ack 1980072434 win 65535
10:05:24.331548 IP (tos 0x0, ttl 64, id 20463, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x9db3 (correct), 2260258133:2260258133(0) ack 1980073506 win 65535
10:05:24.331574 IP (tos 0x0, ttl 64, id 46929, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980074578:1980075114(536) ack 2260258133 win 5840
10:05:24.331586 IP (tos 0x0, ttl 64, id 46931, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980075650:1980076186(536) ack 2260258133 win 5840
10:05:24.331639 IP (tos 0x0, ttl 64, id 20469, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8493 (correct), 2260258133:2260258133(0) ack 1980079938 win 65535
10:05:24.331647 IP (tos 0x0, ttl 64, id 46939, offset 0, flags [DF], proto TCP (6), length 576) dione.40344 > phoenix.distcc: . 1980079938:1980080474(536) ack 2260258133 win 5840
10:05:24.331657 IP (tos 0x0, ttl 64, id 20470, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x8063 (correct), 2260258133:2260258133(0) ack 1980081010 win 65535
10:05:24.331678 IP (tos 0x0, ttl 64, id 20471, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7c33 (correct), 2260258133:2260258133(0) ack 1980082082 win 65535
10:05:24.331688 IP (tos 0x0, ttl 64, id 20472, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: ., cksum 0x7803 (correct), 2260258133:2260258133(0) ack 1980083154 win 65535
10:05:24.732609 IP (tos 0x0, ttl 64, id 20476, offset 0, flags [DF], proto TCP (6), length 2960) phoenix.distcc > dione.40344: . 2260258133:2260261053(2920) ack 1980086145 win 65535
10:05:24.732740 IP (tos 0x0, ttl 64, id 46951, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0x4468 (correct), 1980086145:1980086145(0) ack 2260259593 win 8760
10:05:24.732753 IP (tos 0x0, ttl 64, id 20478, offset 0, flags [DF], proto TCP (6), length 2960) phoenix.distcc > dione.40344: . 2260261053:2260263973(2920) ack 1980086145 win 65535
10:05:24.732756 IP (tos 0x0, ttl 64, id 46952, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0x334c (correct), 1980086145:1980086145(0) ack 2260261053 win 11680
10:05:24.732761 IP (tos 0x0, ttl 64, id 20480, offset 0, flags [DF], proto TCP (6), length 2960) phoenix.distcc > dione.40344: . 2260263973:2260266893(2920) ack 1980086145 win 65535
10:05:24.732835 IP (tos 0x0, ttl 64, id 46953, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0x2230 (correct), 1980086145:1980086145(0) ack 2260262513 win 14600
10:05:24.732841 IP (tos 0x0, ttl 64, id 20482, offset 0, flags [DF], proto TCP (6), length 2960) phoenix.distcc > dione.40344: P 2260266893:2260269813(2920) ack 1980086145 win 65535
10:05:24.732843 IP (tos 0x0, ttl 64, id 46954, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0x1114 (correct), 1980086145:1980086145(0) ack 2260263973 win 17520
10:05:24.732848 IP (tos 0x0, ttl 64, id 20484, offset 0, flags [DF], proto TCP (6), length 924) phoenix.distcc > dione.40344: P 2260269813:2260270697(884) ack 1980086145 win 65535
10:05:24.732885 IP (tos 0x0, ttl 64, id 46955, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0xfff7 (correct), 1980086145:1980086145(0) ack 2260265433 win 20440
10:05:24.732889 IP (tos 0x0, ttl 64, id 46956, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0xeedb (correct), 1980086145:1980086145(0) ack 2260266893 win 23360
10:05:24.732933 IP (tos 0x0, ttl 64, id 46957, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0xddbf (correct), 1980086145:1980086145(0) ack 2260268353 win 26280
10:05:24.732937 IP (tos 0x0, ttl 64, id 46958, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0xcca3 (correct), 1980086145:1980086145(0) ack 2260269813 win 29200
10:05:24.732952 IP (tos 0x0, ttl 64, id 46959, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0xbdc7 (correct), 1980086145:1980086145(0) ack 2260270697 win 32120
10:05:24.732996 IP (tos 0x0, ttl 64, id 46960, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: F, cksum 0xbdc6 (correct), 1980086145:1980086145(0) ack 2260270697 win 32120
10:05:24.733318 IP (tos 0x0, ttl 64, id 20485, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.40344: F, cksum 0x3b3e (correct), 2260270697:2260270697(0) ack 1980086146 win 65535
10:05:24.733389 IP (tos 0x0, ttl 64, id 46961, offset 0, flags [DF], proto TCP (6), length 40) dione.40344 > phoenix.distcc: ., cksum 0xbdc5 (correct), 1980086146:1980086146(0) ack 2260270698 win 32120

2008-07-24 08:36:30

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

* Willy Tarreau <[email protected]> wrote:

> > 08:30:48.049354 IP (tos 0x0, ttl 64, id 27414, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535
> > 08:30:48.049359 IP (tos 0x0, ttl 64, id 27415, offset 0, flags [DF], proto TCP (6), length 40) phoenix.distcc > dione.39201: ., cksum 0x80fe (correct), 2272574194:2272574194(0) ack 234561792 win 65535
>
> And phoenix insists on getting what is missing from the window.
>
> So dione is wrong here.

hm, but dione has been running the same 2.6.24.7-92.fc8 kernel for a
long time:

/var/log/yum.log-20080528:May 27 23:44:36 Installed: kernel - 2.6.24.7-92.fc8.x86_64

and has 6 days uptime at the moment and no trouble communicating with
any other buildbox - except the [only] one running latest -git.

Ingo

2008-07-24 08:37:04

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

From: Ingo Molnar <[email protected]>
Date: Thu, 24 Jul 2008 10:27:30 +0200

> * Herbert Xu <[email protected]> wrote:
>
> > BTW any reason why you appear to have SACK off?
>
> default distro setup, Fedora 9 and Fedora 8. No extra tweaking that i'm
> aware of.

It's probably a side effect of their turning on syncookies in /etc/sysctl.conf :-/

I thought we added code that would allow enabling those options
even with syncookies enabled? There is code in net/ipv4/syncookies.c
function cookie_check_timestamp() to handle this.

Of course this is a recent addition, so if either side doesn't
have that change.

2008-07-24 09:06:21

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

Ingo Molnar <[email protected]> wrote:
>
> the client (running 2.6.24) does periodic 120 seconds retransmits:
>
> 07:40:48.255452 IP dione.39201 > phoenix.distcc: . 1608:2144(536) ack 1 win 584007:40:48.255547 IP phoenix.distcc > dione.39201: . ack 2144 win 65535
> 07:40:48.255564 IP dione.39201 > phoenix.distcc: . 67143:67679(536) ack 1 win 5840
> 07:40:48.255648 IP phoenix.distcc > dione.39201: . ack 2144 win 65535
> 07:42:48.255440 IP dione.39201 > phoenix.distcc: . 2144:2680(536) ack 1 win 5840
> 07:42:48.255559 IP phoenix.distcc > dione.39201: . ack 2680 win 65535
> 07:42:48.255570 IP dione.39201 > phoenix.distcc: . 67679:68215(536) ack 1 win 5840
> 07:42:48.255659 IP phoenix.distcc > dione.39201: . ack 2680 win 65535
> 07:44:48.255436 IP dione.39201 > phoenix.distcc: . 2680:3216(536) ack 1 win 584007:44:48.255570 IP phoenix.distcc > dione.39201: . ack 3216 win 65535
> 07:44:48.255585 IP dione.39201 > phoenix.distcc: . 68215:68751(536) ack 1 win 5840
> 07:44:48.255669 IP phoenix.distcc > dione.39201: . ack 3216 win 65535

OK, something's seriously screwed up on dione's kernel. Could
you please disable syncookies (which should enable SACK for you)
and see if the problem goes away?

I think our Reno code may have been broken.

Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

2008-07-24 09:11:04

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Wed, 2008-07-23 at 13:16 -0700, David Miller wrote:
> From: Jarek Poplawski <[email protected]>
> Date: Wed, 23 Jul 2008 11:49:14 +0000
>
> > On Wed, Jul 23, 2008 at 11:35:19AM +0000, Jarek Poplawski wrote:
> > > On Wed, Jul 23, 2008 at 12:58:16PM +0200, Peter Zijlstra wrote:
> > ...
> > > > When I look at the mac802.11 code in ieee80211_tx_pending() it looks
> > > > like it can do with just one lock at a time, instead of all - but I
> > > > might be missing some obvious details.
> > > >
> > > > So I guess my question is, is netif_tx_lock() here to stay, or is the
> > > > right fix to convert all those drivers to use __netif_tx_lock() which
> > > > locks only a single queue?
> > > >
> > >
> > > It's a new thing mainly for new hardware/drivers, and just after
> > > conversion (older drivers effectively use __netif_tx_lock()), so it'll
> > > probably stay for some time until something better is found. David,
> > > will tell the rest, I hope.
> >
> > ...And, of course, these new drivers should also lock a single queue
> > where possible.
>
> It isn't going away.
>
> There will always be a need for a "stop all the TX queues" operation.

Ok, then how about something like this, the idea is to wrap the per tx
lock with a read lock of the device and let the netif_tx_lock() be the
write side, therefore excluding all device locks, but not incure the
cacheline bouncing on the read side by using per-cpu counters like rcu
does.

This of course requires that netif_tx_lock() is rare, otherwise stuff
will go bounce anyway...

Probably missed a few details,.. but I think the below ought to show the
idea...

struct tx_lock {
int busy;
spinlock_t lock;
unsigned long *counters;
};

int tx_lock_init(struct tx_lock *txl)
{
txl->busy = 0;
spin_lock_init(&txl->lock);
txl->counters = alloc_percpu(unsigned long);

if (!txl->counters)
return -ENOMEM;

return 0;
}

void __netif_tx_lock(struct netdev_queue *txq, cpu)
{
struct net_device *dev = txq->dev;

if (rcu_dereference(dev->tx_lock.busy)) {
spin_lock(&dev->tx_lock.lock);
(*percpu_ptr(dev->tx_lock.counters, cpu))++;
spin_unlock(&dev->tx_lock.lock);
} else
(*percpu_ptr(dev->tx_lock.counters, cpu))++;

spin_lock(&txq->_xmit_lock);
txq->xmit_lock_owner = cpu;
}

void __netif_tx_unlock(struct netdev_queue *txq)
{
struct net_device *dev = txq->dev;

(*percpu_ptr(dev->tx_lock.counters, txq->xmit_lock_owner))--;
txq->xmit_lock_owner = -1;
spin_unlock(&txq->xmit_lock);
}

unsigned long tx_lock_read_counters(struct tx_lock *txl)
{
int i;
unsigned long counter = 0;

/* can use online - the inc/dec are matched per cpu */
for_each_online_cpu(i)
counter += *percpu_ptr(txl->counters, i);

return counter;
}

void netif_tx_lock(struct net_device *dev)
{
spin_lock(&dev->tx_lock.lock);
rcu_assign_pointer(dev->tx_lock.busy, 1);

while (tx_lock_read_counters(&dev->tx_lock)
cpu_relax();
}

void netif_tx_unlock(struct net_device *dev)
{
rcu_assign_pointer(dev->tx_lock.busy, 0);
smp_wmb(); /* because rcu_assign_pointer is broken */
spin_unlock(&dev->tx_lock.lock);
}

2008-07-24 09:20:52

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

From: Peter Zijlstra <[email protected]>
Date: Thu, 24 Jul 2008 11:10:48 +0200

> Ok, then how about something like this, the idea is to wrap the per tx
> lock with a read lock of the device and let the netif_tx_lock() be the
> write side, therefore excluding all device locks, but not incure the
> cacheline bouncing on the read side by using per-cpu counters like rcu
> does.
>
> This of course requires that netif_tx_lock() is rare, otherwise stuff
> will go bounce anyway...
>
> Probably missed a few details,.. but I think the below ought to show the
> idea...

Thanks for the effort, but I don't think we can seriously consider
this.

This lock is taken for every packet transmitted by the system, adding
another memory reference (the RCU deref) and a counter bump is just
not something we can just add to placate lockdep. We going through
all of this effort to seperate the TX locking into individual
queues, it would be silly to go back and make it more expensive.

I have other ideas which I've expanded upon in other emails. They
involve creating a netif_tx_freeze() interface and getting the drivers
to start using it.

2008-07-24 09:23:18

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

BTW, if someone can contact the Fedora folks, specifically whoever
added that syncookies enable to /etc/sysctl.conf, and let them know
that they've set our TCP stack back by 10 years by disabling SACK
and timestamps, I'd really appreciate it.

2008-07-24 09:26:21

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

* Herbert Xu <[email protected]> wrote:

> Ingo Molnar <[email protected]> wrote:
> >
> > the client (running 2.6.24) does periodic 120 seconds retransmits:
> >
> > 07:40:48.255452 IP dione.39201 > phoenix.distcc: . 1608:2144(536) ack 1 win 584007:40:48.255547 IP phoenix.distcc > dione.39201: . ack 2144 win 65535
> > 07:40:48.255564 IP dione.39201 > phoenix.distcc: . 67143:67679(536) ack 1 win 5840
> > 07:40:48.255648 IP phoenix.distcc > dione.39201: . ack 2144 win 65535
> > 07:42:48.255440 IP dione.39201 > phoenix.distcc: . 2144:2680(536) ack 1 win 5840
> > 07:42:48.255559 IP phoenix.distcc > dione.39201: . ack 2680 win 65535
> > 07:42:48.255570 IP dione.39201 > phoenix.distcc: . 67679:68215(536) ack 1 win 5840
> > 07:42:48.255659 IP phoenix.distcc > dione.39201: . ack 2680 win 65535
> > 07:44:48.255436 IP dione.39201 > phoenix.distcc: . 2680:3216(536) ack 1 win 584007:44:48.255570 IP phoenix.distcc > dione.39201: . ack 3216 win 65535
> > 07:44:48.255585 IP dione.39201 > phoenix.distcc: . 68215:68751(536) ack 1 win 5840
> > 07:44:48.255669 IP phoenix.distcc > dione.39201: . ack 3216 win 65535
>
> OK, something's seriously screwed up on dione's kernel. Could you
> please disable syncookies (which should enable SACK for you) and see
> if the problem goes away?
>
> I think our Reno code may have been broken.

ok, i've booted the latest kernel on dione and the builds are running on
it now. It's using these settings currently:

# cat /proc/sys/net/ipv4/tcp_syncookies
0
# cat /proc/sys/net/ipv4/tcp_sack
1

Just to make sure i have the right test-plan: assuming this
suspected-good kernel is trouble-free for say 48 hours, i should be able
to switch back to the suspected-broken variant, by doing:

echo 1 > /proc/sys/net/ipv4/tcp_syncookies

without rebooting the kernel, right? And then if the bug comes back
again we've nailed it down to the area you suspect, correct?

Ingo

2008-07-24 09:27:20

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Thu, 2008-07-24 at 02:20 -0700, David Miller wrote:
> From: Peter Zijlstra <[email protected]>
> Date: Thu, 24 Jul 2008 11:10:48 +0200
>
> > Ok, then how about something like this, the idea is to wrap the per tx
> > lock with a read lock of the device and let the netif_tx_lock() be the
> > write side, therefore excluding all device locks, but not incure the
> > cacheline bouncing on the read side by using per-cpu counters like rcu
> > does.
> >
> > This of course requires that netif_tx_lock() is rare, otherwise stuff
> > will go bounce anyway...
> >
> > Probably missed a few details,.. but I think the below ought to show the
> > idea...
>
> Thanks for the effort, but I don't think we can seriously consider
> this.
>
> This lock is taken for every packet transmitted by the system, adding
> another memory reference (the RCU deref) and a counter bump is just
> not something we can just add to placate lockdep. We going through
> all of this effort to seperate the TX locking into individual
> queues, it would be silly to go back and make it more expensive.

Well, not only lockdep, taking a very large number of locks is expensive
as well.

> I have other ideas which I've expanded upon in other emails. They
> involve creating a netif_tx_freeze() interface and getting the drivers
> to start using it.

OK, as long as we get there :-)

2008-07-24 09:30:03

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

From: Ingo Molnar <[email protected]>
Date: Thu, 24 Jul 2008 11:25:18 +0200

> ok, i've booted the latest kernel on dione and the builds are running on
> it now. It's using these settings currently:
>
> # cat /proc/sys/net/ipv4/tcp_syncookies
> 0
> # cat /proc/sys/net/ipv4/tcp_sack
> 1
>
> Just to make sure i have the right test-plan: assuming this
> suspected-good kernel is trouble-free for say 48 hours, i should be able
> to switch back to the suspected-broken variant, by doing:
>
> echo 1 > /proc/sys/net/ipv4/tcp_syncookies
>
> without rebooting the kernel, right? And then if the bug comes back
> again we've nailed it down to the area you suspect, correct?

You don't need the explicit tcp_sack=1, it's going to be '1' by
default.

Simply switching syncookies on and off will turn both SACK and
timestamps off when running the dist kernel.

2008-07-24 09:32:24

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

From: Peter Zijlstra <[email protected]>
Date: Thu, 24 Jul 2008 11:27:05 +0200

> Well, not only lockdep, taking a very large number of locks is expensive
> as well.

Right now it would be on the order of 16 or 32 for
real hardware.

Much less than the scheduler currently takes on some
of my systems, so currently you are the pot calling the
kettle black. :-)

USHORT_MAX is just the upper hard limit imposed by the
software interface merely as a side effect of storing
the queue number of the SKB as a u16.

2008-07-24 09:35:46

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

* David Miller <[email protected]> wrote:

> BTW, if someone can contact the Fedora folks, specifically whoever
> added that syncookies enable to /etc/sysctl.conf, and let them know
> that they've set our TCP stack back by 10 years by disabling SACK and
> timestamps, I'd really appreciate it.

Server box (F9) has:

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

timestamp of /etc/sysctl.conf: 2008-05-15, installation default i think.

Other (F8) box has:

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 0

timestamp of /etc/sysctl.conf: 2007-01-28, touched 1.5 years ago.

Have not checked the install defaults - if i touched them it was a long
time ago.

Ingo

2008-07-24 09:36:57

by Ilpo Järvinen

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

On Thu, 24 Jul 2008, Herbert Xu wrote:

> Ingo Molnar <[email protected]> wrote:
> >
> > the client (running 2.6.24) does periodic 120 seconds retransmits:
> >
> > 07:40:48.255452 IP dione.39201 > phoenix.distcc: . 1608:2144(536) ack 1 win 584007:40:48.255547 IP phoenix.distcc > dione.39201: . ack 2144 win 65535
> > 07:40:48.255564 IP dione.39201 > phoenix.distcc: . 67143:67679(536) ack 1 win 5840
> > 07:40:48.255648 IP phoenix.distcc > dione.39201: . ack 2144 win 65535
> > 07:42:48.255440 IP dione.39201 > phoenix.distcc: . 2144:2680(536) ack 1 win 5840
> > 07:42:48.255559 IP phoenix.distcc > dione.39201: . ack 2680 win 65535
> > 07:42:48.255570 IP dione.39201 > phoenix.distcc: . 67679:68215(536) ack 1 win 5840
> > 07:42:48.255659 IP phoenix.distcc > dione.39201: . ack 2680 win 65535
> > 07:44:48.255436 IP dione.39201 > phoenix.distcc: . 2680:3216(536) ack 1 win 584007:44:48.255570 IP phoenix.distcc > dione.39201: . ack 3216 win 65535
> > 07:44:48.255585 IP dione.39201 > phoenix.distcc: . 68215:68751(536) ack 1 win 5840
> > 07:44:48.255669 IP phoenix.distcc > dione.39201: . ack 3216 win 65535
>
> OK, something's seriously screwed up on dione's kernel. Could
> you please disable syncookies (which should enable SACK for you)
> and see if the problem goes away?

This looks like the FRTO bugs we fixed in 2.6.25.7, afaik, 2.6.24.y wasn't
anymore updated at that time so it's a bit obsolete...

--
i.

2008-07-24 10:03:47

by Ilpo Järvinen

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

On Thu, 24 Jul 2008, Ilpo J?rvinen wrote:

> On Thu, 24 Jul 2008, Herbert Xu wrote:
>
> > Ingo Molnar <[email protected]> wrote:
> > >
> > > the client (running 2.6.24) does periodic 120 seconds retransmits:
> > >
> > > 07:40:48.255452 IP dione.39201 > phoenix.distcc: . 1608:2144(536) ack 1 win 584007:40:48.255547 IP phoenix.distcc > dione.39201: . ack 2144 win 65535
> > > 07:40:48.255564 IP dione.39201 > phoenix.distcc: . 67143:67679(536) ack 1 win 5840
> > > 07:40:48.255648 IP phoenix.distcc > dione.39201: . ack 2144 win 65535
> > > 07:42:48.255440 IP dione.39201 > phoenix.distcc: . 2144:2680(536) ack 1 win 5840
> > > 07:42:48.255559 IP phoenix.distcc > dione.39201: . ack 2680 win 65535
> > > 07:42:48.255570 IP dione.39201 > phoenix.distcc: . 67679:68215(536) ack 1 win 5840
> > > 07:42:48.255659 IP phoenix.distcc > dione.39201: . ack 2680 win 65535
> > > 07:44:48.255436 IP dione.39201 > phoenix.distcc: . 2680:3216(536) ack 1 win 584007:44:48.255570 IP phoenix.distcc > dione.39201: . ack 3216 win 65535
> > > 07:44:48.255585 IP dione.39201 > phoenix.distcc: . 68215:68751(536) ack 1 win 5840
> > > 07:44:48.255669 IP phoenix.distcc > dione.39201: . ack 3216 win 65535
> >
> > OK, something's seriously screwed up on dione's kernel. Could
> > you please disable syncookies (which should enable SACK for you)
> > and see if the problem goes away?
>
> This looks like the FRTO bugs we fixed in 2.6.25.7, afaik, 2.6.24.y wasn't
> anymore updated at that time so it's a bit obsolete...

But there might be something very interesting on the opposite end change
that is pointed out by this behavior, since one needs considerable amount
of losses in the outstanding window to triggers long delays (the bug was
that FRTO never fallback to go-back-n retransmissions, so one RTO was
necessary per loss), like you found out there's slow progress made and the
situation can resolve (by a big cumulative ACK once all losses are
cleared). Would the receiver for some reason not accept the new data
segment that FRTO sends after getting the ACK of the retransmissions,
the RTO loop would continue forever with FRTO bug (unless userland tears
the connection down).

--
i.

2008-07-24 10:09:31

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Thu, 2008-07-24 at 02:32 -0700, David Miller wrote:
> From: Peter Zijlstra <[email protected]>
> Date: Thu, 24 Jul 2008 11:27:05 +0200
>
> > Well, not only lockdep, taking a very large number of locks is expensive
> > as well.
>
> Right now it would be on the order of 16 or 32 for
> real hardware.
>
> Much less than the scheduler currently takes on some
> of my systems, so currently you are the pot calling the
> kettle black. :-)

One nit, and then I'll let this issue rest :-)

The scheduler has a long lock dependancy chain (nr_cpu_ids rq locks),
but it never takes all of them at the same time. Any one code path will
at most hold two rq locks.

2008-07-24 10:38:55

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Thursday 24 July 2008 20:08, Peter Zijlstra wrote:
> On Thu, 2008-07-24 at 02:32 -0700, David Miller wrote:
> > From: Peter Zijlstra <[email protected]>
> > Date: Thu, 24 Jul 2008 11:27:05 +0200
> >
> > > Well, not only lockdep, taking a very large number of locks is
> > > expensive as well.
> >
> > Right now it would be on the order of 16 or 32 for
> > real hardware.
> >
> > Much less than the scheduler currently takes on some
> > of my systems, so currently you are the pot calling the
> > kettle black. :-)
>
> One nit, and then I'll let this issue rest :-)
>
> The scheduler has a long lock dependancy chain (nr_cpu_ids rq locks),
> but it never takes all of them at the same time. Any one code path will
> at most hold two rq locks.

Aside from lockdep, is there a particular problem with taking 64k locks
at once? (in a very slow path, of course) I don't think it causes a
problem with preempt_count, does it cause issues with -rt kernel?

Hey, something kind of cool (and OT) I've just thought of that we can
do with ticket locks is to take tickets for 2 (or 64K) nested locks,
and then wait for them both (all), so the cost is N*lock + longest spin,
rather than N*lock + N*avg spin.

That would mean even at the worst case of a huge amount of contention
on all 64K locks, it should only take a couple of ms to take all of
them (assuming max spin time isn't ridiculous).

Probably not the kind of feature we want to expose widely, but for
really special things like the scheduler, it might be a neat hack to
save a few cycles ;) Traditional implementations would just have
#define spin_lock_async spin_lock
#define spin_lock_async_wait do {} while (0)

Sorry it's offtopic, but if I didn't post it, I'd forget to. Might be
a fun quick hack for someone.

2008-07-24 10:56:25

by Miklos Szeredi

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Thu, 24 Jul 2008, Nick Piggin wrote:
> Hey, something kind of cool (and OT) I've just thought of that we can
> do with ticket locks is to take tickets for 2 (or 64K) nested locks,
> and then wait for them both (all), so the cost is N*lock + longest spin,
> rather than N*lock + N*avg spin.

Isn't this deadlocky?

E.g. one task takes ticket x=1, then other task comes in and takes x=2
and y=1, then first task takes y=2. Then neither can actually
complete both locks.

Miklos

2008-07-24 11:00:11

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Thu, 2008-07-24 at 20:38 +1000, Nick Piggin wrote:
> On Thursday 24 July 2008 20:08, Peter Zijlstra wrote:
> > On Thu, 2008-07-24 at 02:32 -0700, David Miller wrote:
> > > From: Peter Zijlstra <[email protected]>
> > > Date: Thu, 24 Jul 2008 11:27:05 +0200
> > >
> > > > Well, not only lockdep, taking a very large number of locks is
> > > > expensive as well.
> > >
> > > Right now it would be on the order of 16 or 32 for
> > > real hardware.
> > >
> > > Much less than the scheduler currently takes on some
> > > of my systems, so currently you are the pot calling the
> > > kettle black. :-)
> >
> > One nit, and then I'll let this issue rest :-)
> >
> > The scheduler has a long lock dependancy chain (nr_cpu_ids rq locks),
> > but it never takes all of them at the same time. Any one code path will
> > at most hold two rq locks.
>
> Aside from lockdep, is there a particular problem with taking 64k locks
> at once? (in a very slow path, of course) I don't think it causes a
> problem with preempt_count, does it cause issues with -rt kernel?

PI-chains might explode I guess, Thomas?

Besides that, I just have this voice in my head telling me that
minimizing the number of locks held is a good thing.

> Hey, something kind of cool (and OT) I've just thought of that we can
> do with ticket locks is to take tickets for 2 (or 64K) nested locks,
> and then wait for them both (all), so the cost is N*lock + longest spin,
> rather than N*lock + N*avg spin.
>
> That would mean even at the worst case of a huge amount of contention
> on all 64K locks, it should only take a couple of ms to take all of
> them (assuming max spin time isn't ridiculous).
>
> Probably not the kind of feature we want to expose widely, but for
> really special things like the scheduler, it might be a neat hack to
> save a few cycles ;) Traditional implementations would just have
> #define spin_lock_async spin_lock
> #define spin_lock_async_wait do {} while (0)
>
> Sorry it's offtopic, but if I didn't post it, I'd forget to. Might be
> a fun quick hack for someone.

It might just be worth it for double_rq_lock() - if you can sort out the
deadlock potential Miklos just raised ;-)

2008-07-24 11:07:27

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Thursday 24 July 2008 20:55, Miklos Szeredi wrote:
> On Thu, 24 Jul 2008, Nick Piggin wrote:
> > Hey, something kind of cool (and OT) I've just thought of that we can
> > do with ticket locks is to take tickets for 2 (or 64K) nested locks,
> > and then wait for them both (all), so the cost is N*lock + longest spin,
> > rather than N*lock + N*avg spin.
>
> Isn't this deadlocky?
>
> E.g. one task takes ticket x=1, then other task comes in and takes x=2
> and y=1, then first task takes y=2. Then neither can actually
> complete both locks.

Oh duh of course you still need mutual exclusion from the first lock
to order the subsequent :P

So yeah it only works for N > 2 locks, and you have to spin_lock the
first one... so unsuitable for scheduler.

2008-07-24 11:13:34

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

On Thu, Jul 24, 2008 at 11:25:18AM +0200, Ingo Molnar wrote:
>
> without rebooting the kernel, right? And then if the bug comes back
> again we've nailed it down to the area you suspect, correct?

Yes, except that you'll have to reestablish those connections
to make the new setting take effect. In either case please
verify with tcpdump to make sure that both sides are proposing
sack at the start of the connection. Alternatively see whether
sacks are generated in case of packet loss.

Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

2008-07-24 11:57:46

[permalink] [raw]

Subject: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

here's a new type of crash a -tip testbox triggered today:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
IP: [<0000000000000000>] 0x0
PGD 0
Oops: 0010 [1] PREEMPT
CPU 0
Pid: 5331, comm: ssh Not tainted 2.6.26 #21613
RIP: 0010:[<0000000000000000>] [<0000000000000000>] 0x0
RSP: 0018:ffff88003b959a70 EFLAGS: 00010217
RAX: ffff88003b9ba91c RBX: ffff88003b9baac0 RCX: 0000000000000002
RDX: ffff88003b9baac0 RSI: 0000000000000014 RDI: ffff88003b9ba3d8
RBP: ffff88003b959a98 R08: ffff88003e434130 R09: ffff88003b9598b8
R10: 0000000000002000 R11: 00000000d85507ac R12: ffff88003b9ba3d8
R13: 0000000000000002 R14: ffff88003e06a020 R15: 0000000000000000
FS: 00007fb2b7806780(0000) GS:ffffffff80c1c180(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 000000003b9e8000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ssh (pid: 5331, threadinfo ffff88003b958000, task ffff88003b99a000)
Stack: ffffffff80777f25 ffff88003b959a98 ffffffff80c24170 ffff88003b959b28
0000000000000004 ffff88003b959ae8 ffffffff80733107 ffff88003b959ab8
ffff88003b9ba3d8 ffffffff80c1ed60 0000000000000004 ffff88003b9ba3d8
Call Trace:
[<ffffffff80777f25>] ? ipv4_confirm+0x8d/0x122
[<ffffffff80733107>] nf_iterate+0x43/0xa5
[<ffffffff8074bbb2>] ? ip_finish_output+0x0/0x62
[<ffffffff807331cd>] nf_hook_slow+0x64/0xcb
[<ffffffff8074bbb2>] ? ip_finish_output+0x0/0x62
[<ffffffff8074b63b>] nf_hook_thresh+0x1e/0x20
[<ffffffff8074bd44>] ip_output+0xb3/0xc9
[<ffffffff8074a27e>] dst_output+0xb/0xd
[<ffffffff8074b6ef>] ip_local_out+0x1e/0x22
[<ffffffff8074c75c>] ip_queue_xmit+0x2a6/0x321
[<ffffffff80241961>] ? sub_preempt_count+0x2d/0x40
[<ffffffff8028f239>] ? slob_alloc+0x11b/0x248
[<ffffffff8075b85a>] tcp_transmit_skb+0x5ec/0x61b
[<ffffffff807122b6>] ? kmalloc+0xc/0xe
[<ffffffff8075d7ae>] tcp_connect+0x2a4/0x344
[<ffffffff80760cf1>] tcp_v4_connect+0x3cd/0x425
[<ffffffff8076bc7d>] inet_stream_connect+0x9a/0x24d
[<ffffffff80292988>] ? fget_light+0x38/0x85
[<ffffffff8070c60c>] sys_connect+0x68/0x8b
[<ffffffff8025dfab>] ? up_read+0x9/0xb
[<ffffffff8023727c>] ? do_page_fault+0x55f/0x92f
[<ffffffff8022479a>] system_call_fastpath+0x16/0x1b

with this config:

http://redhat.com/~mingo/misc/config-Thu_Jul_24_13_23_34_CEST_2008.bad

i'll try a bisection.

Ingo

2008-07-24 12:01:21

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

* Ingo Molnar <[email protected]> wrote:

> here's a new type of crash a -tip testbox triggered today:
>
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
> IP: [<0000000000000000>] 0x0
> PGD 0
> Oops: 0010 [1] PREEMPT
> CPU 0
> Pid: 5331, comm: ssh Not tainted 2.6.26 #21613
> RIP: 0010:[<0000000000000000>] [<0000000000000000>] 0x0
> RSP: 0018:ffff88003b959a70 EFLAGS: 00010217
> RAX: ffff88003b9ba91c RBX: ffff88003b9baac0 RCX: 0000000000000002
> RDX: ffff88003b9baac0 RSI: 0000000000000014 RDI: ffff88003b9ba3d8
> RBP: ffff88003b959a98 R08: ffff88003e434130 R09: ffff88003b9598b8
> R10: 0000000000002000 R11: 00000000d85507ac R12: ffff88003b9ba3d8
> R13: 0000000000000002 R14: ffff88003e06a020 R15: 0000000000000000
> FS: 00007fb2b7806780(0000) GS:ffffffff80c1c180(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000000 CR3: 000000003b9e8000 CR4: 00000000000006a0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process ssh (pid: 5331, threadinfo ffff88003b958000, task ffff88003b99a000)
> Stack: ffffffff80777f25 ffff88003b959a98 ffffffff80c24170 ffff88003b959b28
> 0000000000000004 ffff88003b959ae8 ffffffff80733107 ffff88003b959ab8
> ffff88003b9ba3d8 ffffffff80c1ed60 0000000000000004 ffff88003b9ba3d8
> Call Trace:
> [<ffffffff80777f25>] ? ipv4_confirm+0x8d/0x122
> [<ffffffff80733107>] nf_iterate+0x43/0xa5
> [<ffffffff8074bbb2>] ? ip_finish_output+0x0/0x62
> [<ffffffff807331cd>] nf_hook_slow+0x64/0xcb
> [<ffffffff8074bbb2>] ? ip_finish_output+0x0/0x62
> [<ffffffff8074b63b>] nf_hook_thresh+0x1e/0x20
> [<ffffffff8074bd44>] ip_output+0xb3/0xc9
> [<ffffffff8074a27e>] dst_output+0xb/0xd
> [<ffffffff8074b6ef>] ip_local_out+0x1e/0x22
> [<ffffffff8074c75c>] ip_queue_xmit+0x2a6/0x321
> [<ffffffff80241961>] ? sub_preempt_count+0x2d/0x40
> [<ffffffff8028f239>] ? slob_alloc+0x11b/0x248
> [<ffffffff8075b85a>] tcp_transmit_skb+0x5ec/0x61b
> [<ffffffff807122b6>] ? kmalloc+0xc/0xe
> [<ffffffff8075d7ae>] tcp_connect+0x2a4/0x344
> [<ffffffff80760cf1>] tcp_v4_connect+0x3cd/0x425
> [<ffffffff8076bc7d>] inet_stream_connect+0x9a/0x24d
> [<ffffffff80292988>] ? fget_light+0x38/0x85
> [<ffffffff8070c60c>] sys_connect+0x68/0x8b
> [<ffffffff8025dfab>] ? up_read+0x9/0xb
> [<ffffffff8023727c>] ? do_page_fault+0x55f/0x92f
> [<ffffffff8022479a>] system_call_fastpath+0x16/0x1b

here's the full bootlog:

http://redhat.com/~mingo/misc/crash-Thu_Jul_24_13_23_34_CEST_2008.log

kernel is latest -git, v2.6.26-6371-g338b9bb.

Ingo

2008-07-24 12:04:22

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

Ingo Molnar wrote:
> * Ingo Molnar <[email protected]> wrote:
>
>> here's a new type of crash a -tip testbox triggered today:
>>
>> BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
>> IP: [<0000000000000000>] 0x0
>> PGD 0
>> Oops: 0010 [1] PREEMPT
>> CPU 0
>> Pid: 5331, comm: ssh Not tainted 2.6.26 #21613
>> RIP: 0010:[<0000000000000000>] [<0000000000000000>] 0x0
>> RSP: 0018:ffff88003b959a70 EFLAGS: 00010217
>> RAX: ffff88003b9ba91c RBX: ffff88003b9baac0 RCX: 0000000000000002
>> RDX: ffff88003b9baac0 RSI: 0000000000000014 RDI: ffff88003b9ba3d8
>> RBP: ffff88003b959a98 R08: ffff88003e434130 R09: ffff88003b9598b8
>> R10: 0000000000002000 R11: 00000000d85507ac R12: ffff88003b9ba3d8
>> R13: 0000000000000002 R14: ffff88003e06a020 R15: 0000000000000000
>> FS: 00007fb2b7806780(0000) GS:ffffffff80c1c180(0000) knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> CR2: 0000000000000000 CR3: 000000003b9e8000 CR4: 00000000000006a0
>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Process ssh (pid: 5331, threadinfo ffff88003b958000, task ffff88003b99a000)
>> Stack: ffffffff80777f25 ffff88003b959a98 ffffffff80c24170 ffff88003b959b28
>> 0000000000000004 ffff88003b959ae8 ffffffff80733107 ffff88003b959ab8
>> ffff88003b9ba3d8 ffffffff80c1ed60 0000000000000004 ffff88003b9ba3d8
>> Call Trace:
>> [<ffffffff80777f25>] ? ipv4_confirm+0x8d/0x122
>> [<ffffffff80733107>] nf_iterate+0x43/0xa5

Does reverting 31d8519c fix this?

2008-07-24 12:23:01

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, Jul 24, 2008 at 02:03:50PM +0200, Patrick McHardy wrote:
>
> Does reverting 31d8519c fix this?

Regardless of whether this is the problem, banning ksize because
it can be abused is like banning cars because they can kill people.

For example, Ethernet skbs are 1500 bytes long, so using ksize
we could potentially use the left-over memory for temporary storage.
Without it all this memory goes to waste. On a machine that's
doing a lot of networking this could save a signifcant amount
of memory.

Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

2008-07-24 12:41:11

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

Hi Herbert,

On Thu, Jul 24, 2008 at 02:03:50PM +0200, Patrick McHardy wrote:
>> Does reverting 31d8519c fix this?

On Thu, Jul 24, 2008 at 3:22 PM, Herbert Xu <[email protected]> wrote:
> Regardless of whether this is the problem, banning ksize because
> it can be abused is like banning cars because they can kill people.
>
> For example, Ethernet skbs are 1500 bytes long, so using ksize
> we could potentially use the left-over memory for temporary storage.

Using ksize() for skbs will crash your kernel for some configurations
because calling that function for memory allocated with
kmem_cache_alloc() is not supported by all the allocators (well,
SLOB).

Pekka

2008-07-24 12:44:28

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, Jul 24, 2008 at 3:03 PM, Patrick McHardy <[email protected]> wrote:
> Ingo Molnar wrote:
>>
>> * Ingo Molnar <[email protected]> wrote:
>>
>>> here's a new type of crash a -tip testbox triggered today:
>>>
>>> BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
>>> IP: [<0000000000000000>] 0x0
>>> PGD 0 Oops: 0010 [1] PREEMPT CPU 0 Pid: 5331, comm: ssh Not tainted
>>> 2.6.26 #21613
>>> RIP: 0010:[<0000000000000000>] [<0000000000000000>] 0x0
>>> RSP: 0018:ffff88003b959a70 EFLAGS: 00010217
>>> RAX: ffff88003b9ba91c RBX: ffff88003b9baac0 RCX: 0000000000000002
>>> RDX: ffff88003b9baac0 RSI: 0000000000000014 RDI: ffff88003b9ba3d8
>>> RBP: ffff88003b959a98 R08: ffff88003e434130 R09: ffff88003b9598b8
>>> R10: 0000000000002000 R11: 00000000d85507ac R12: ffff88003b9ba3d8
>>> R13: 0000000000000002 R14: ffff88003e06a020 R15: 0000000000000000
>>> FS: 00007fb2b7806780(0000) GS:ffffffff80c1c180(0000)
>>> knlGS:0000000000000000
>>> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>> CR2: 0000000000000000 CR3: 000000003b9e8000 CR4: 00000000000006a0
>>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>> Process ssh (pid: 5331, threadinfo ffff88003b958000, task
>>> ffff88003b99a000)
>>> Stack: ffffffff80777f25 ffff88003b959a98 ffffffff80c24170
>>> ffff88003b959b28
>>> 0000000000000004 ffff88003b959ae8 ffffffff80733107 ffff88003b959ab8
>>> ffff88003b9ba3d8 ffffffff80c1ed60 0000000000000004 ffff88003b9ba3d8
>>> Call Trace:
>>> [<ffffffff80777f25>] ? ipv4_confirm+0x8d/0x122
>>> [<ffffffff80733107>] nf_iterate+0x43/0xa5
>
> Does reverting 31d8519c fix this?

Hmm, why do you think it's related? It looks like elem->hook() is a
NULL pointer but my patch shouldn't make any difference here...

2008-07-24 12:50:14

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

Pekka Enberg wrote:
> On Thu, Jul 24, 2008 at 3:03 PM, Patrick McHardy <[email protected]> wrote:
>> Ingo Molnar wrote:
>>> * Ingo Molnar <[email protected]> wrote:
>>>
>>>> here's a new type of crash a -tip testbox triggered today:
>>>>
>>>> BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
>>>> IP: [<0000000000000000>] 0x0
>>>> ...
>>>> Call Trace:
>>>> [<ffffffff80777f25>] ? ipv4_confirm+0x8d/0x122
>>>> [<ffffffff80733107>] nf_iterate+0x43/0xa5
>> Does reverting 31d8519c fix this?
>
> Hmm, why do you think it's related? It looks like elem->hook() is a
> NULL pointer but my patch shouldn't make any difference here...

No, its already in ipv4_confirm, so its most likely helper->help()
thats NULL, which is contained in an extension.

The reason why I think its this patch is (quoting from an old
email that I never got a response to):

---
Your patch introduced a use-after-free and double-free.
krealloc() frees the old pointer, but it is still used
for the ->move operations, then freed again.

To fix this I think we need a __krealloc() that doesn't
free the old memory, especially since it must not be
freed immediately because it may still be used in a RCU
read side (see the last part in the patch attached to
this mail (based on a kernel without your patch)).
---

I've attached that patch again.

Attachments:

01.diff (3.23 kB)

2008-07-24 12:51:20

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, Jul 24, 2008 at 03:40:40PM +0300, Pekka Enberg wrote:
>
> Using ksize() for skbs will crash your kernel for some configurations
> because calling that function for memory allocated with
> kmem_cache_alloc() is not supported by all the allocators (well,
> SLOB).

So how about fixing the interface so that it can return an error
to indicate that the allocator doesn't support it? You're taking
away an entire interface just because an underlying implementation
that's used by a very small proportion of users doesn't do the
right thing.

Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

2008-07-24 12:56:31

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thursday 24 July 2008 22:50, Herbert Xu wrote:
> On Thu, Jul 24, 2008 at 03:40:40PM +0300, Pekka Enberg wrote:
> > Using ksize() for skbs will crash your kernel for some configurations
> > because calling that function for memory allocated with
> > kmem_cache_alloc() is not supported by all the allocators (well,
> > SLOB).
>
> So how about fixing the interface so that it can return an error
> to indicate that the allocator doesn't support it? You're taking
> away an entire interface just because an underlying implementation
> that's used by a very small proportion of users doesn't do the
> right thing.

You could easily add some function which returns a constant error for
SLOB and ksize for others, because slob typically contains very little
slack space (unless explicit alignment is asked for).

OTOH, skb allocation uses kmalloc don't they? So you could still use
SLOB ksize for that I guess.

2008-07-24 13:03:36

[permalink] [raw]

Subject: Re: [TCP bug, regression] stuck distcc connections in latest -git

On Thu, Jul 24, 2008 at 02:22:59AM -0700, David Miller wrote:
>
> BTW, if someone can contact the Fedora folks, specifically whoever
> added that syncookies enable to /etc/sysctl.conf, and let them know
> that they've set our TCP stack back by 10 years by disabling SACK
> and timestamps, I'd really appreciate it.

But syn cookies should not trigger unless the syn backlog is full, right ?
Anyway, it would also explain why Ingo's running no default MSS (though
it should be correctly negociated by syn cookies).

willy

2008-07-24 13:05:27

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, Jul 24, 2008 at 10:56:08PM +1000, Nick Piggin wrote:
>
> OTOH, skb allocation uses kmalloc don't they? So you could still use
> SLOB ksize for that I guess.

Yes I was referring to the data portion which is kmalloc'ed.
That is also why I'm interested in ksize because a priori we
don't know exactly how big it's going to be. However, we do
know that statistically 1500 will dominate.

I'm not interested in ksize for kmem_cache at all. So in fact
we could have something simpler that's based on kmalloc's rounding
algorithm instead.

Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

2008-07-24 13:14:13

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thursday 24 July 2008 23:04, Herbert Xu wrote:
> On Thu, Jul 24, 2008 at 10:56:08PM +1000, Nick Piggin wrote:
> > OTOH, skb allocation uses kmalloc don't they? So you could still use
> > SLOB ksize for that I guess.
>
> Yes I was referring to the data portion which is kmalloc'ed.
> That is also why I'm interested in ksize because a priori we
> don't know exactly how big it's going to be. However, we do
> know that statistically 1500 will dominate.
>
> I'm not interested in ksize for kmem_cache at all. So in fact
> we could have something simpler that's based on kmalloc's rounding
> algorithm instead.

Yes you could definitely have a function that returns allocated
bytes for a given kmalloc size. Should be about as fast or faster
than extracting the size from the kaddr...

2008-07-24 13:16:12

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, 2008-07-24 at 20:50 +0800, Herbert Xu wrote:
> On Thu, Jul 24, 2008 at 03:40:40PM +0300, Pekka Enberg wrote:
> >
> > Using ksize() for skbs will crash your kernel for some configurations
> > because calling that function for memory allocated with
> > kmem_cache_alloc() is not supported by all the allocators (well,
> > SLOB).
>
> So how about fixing the interface so that it can return an error
> to indicate that the allocator doesn't support it?

You don't honestly expect people to correctly code to such a standard,
do you? People will assume that ksize never fails, they will be wrong,
and computers will die.

> You're taking
> away an entire interface just because an underlying implementation
> that's used by a very small proportion of users doesn't do the
> right thing.

Umm, no. There were very few users to being with, so it was actually a
fairly large proportion. And that suggested the interface was a bad
idea.

--
Mathematics is the supreme nostalgia of our time.

2008-07-24 13:23:49

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, 2008-07-24 at 14:49 +0200, Patrick McHardy wrote:
> Pekka Enberg wrote:
> > On Thu, Jul 24, 2008 at 3:03 PM, Patrick McHardy <[email protected]> wrote:
> >> Ingo Molnar wrote:
> >>> * Ingo Molnar <[email protected]> wrote:
> >>>
> >>>> here's a new type of crash a -tip testbox triggered today:
> >>>>
> >>>> BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
> >>>> IP: [<0000000000000000>] 0x0
> >>>> ...
> >>>> Call Trace:
> >>>> [<ffffffff80777f25>] ? ipv4_confirm+0x8d/0x122
> >>>> [<ffffffff80733107>] nf_iterate+0x43/0xa5
> >> Does reverting 31d8519c fix this?
> >
> > Hmm, why do you think it's related? It looks like elem->hook() is a
> > NULL pointer but my patch shouldn't make any difference here...
>
> No, its already in ipv4_confirm, so its most likely helper->help()
> thats NULL, which is contained in an extension.
>
> The reason why I think its this patch is (quoting from an old
> email that I never got a response to):

Oh, I'm really sorry about that.

> ---
> Your patch introduced a use-after-free and double-free.
> krealloc() frees the old pointer, but it is still used
> for the ->move operations, then freed again.
>
> To fix this I think we need a __krealloc() that doesn't
> free the old memory, especially since it must not be
> freed immediately because it may still be used in a RCU
> read side (see the last part in the patch attached to
> this mail (based on a kernel without your patch)).

Agreed. Something like this, perhaps?

[PATCH] netfilter: fix double-free and use-after free

As suggested by Patrick McHardy, introduce a __krealloc() that doesn't
free the original buffer to fix a double-free and use-after-free bug
introduced by me in netfilter that uses RCU.

Reported-by: Patrick McHardy <[email protected]>
Signed-off-by: Pekka Enberg <[email protected]>
---

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 9aa90a6..be6f1d4 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -96,6 +96,7 @@ int kmem_ptr_validate(struct kmem_cache *cachep, const void *ptr);
/*
* Common kmalloc functions provided by all allocators
*/
+void * __must_check __krealloc(const void *, size_t, gfp_t);
void * __must_check krealloc(const void *, size_t, gfp_t);
void kfree(const void *);
size_t ksize(const void *);
diff --git a/mm/util.c b/mm/util.c
index 8f18683..6ef9e99 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -68,25 +68,22 @@ void *kmemdup(const void *src, size_t len, gfp_t gfp)
EXPORT_SYMBOL(kmemdup);

/**
- * krealloc - reallocate memory. The contents will remain unchanged.
+ * __krealloc - like krealloc() but don't free @p.
* @p: object to reallocate memory for.
* @new_size: how many bytes of memory are required.
* @flags: the type of memory to allocate.
*
- * The contents of the object pointed to are preserved up to the
- * lesser of the new and old sizes. If @p is %NULL, krealloc()
- * behaves exactly like kmalloc(). If @size is 0 and @p is not a
- * %NULL pointer, the object pointed to is freed.
+ * This function is like krealloc() except it never frees the originally
+ * allocated buffer. Use this if you don't want to free the buffer immediately
+ * like, for example, with RCU.
*/
-void *krealloc(const void *p, size_t new_size, gfp_t flags)
+void *__krealloc(const void *p, size_t new_size, gfp_t flags)
{
void *ret;
size_t ks = 0;

- if (unlikely(!new_size)) {
- kfree(p);
+ if (unlikely(!new_size))
return ZERO_SIZE_PTR;
- }

if (p)
ks = ksize(p);
@@ -95,10 +92,37 @@ void *krealloc(const void *p, size_t new_size, gfp_t flags)
return (void *)p;

ret = kmalloc_track_caller(new_size, flags);
- if (ret && p) {
+ if (ret && p)
memcpy(ret, p, ks);
+
+ return ret;
+}
+EXPORT_SYMBOL(__krealloc);
+
+/**
+ * krealloc - reallocate memory. The contents will remain unchanged.
+ * @p: object to reallocate memory for.
+ * @new_size: how many bytes of memory are required.
+ * @flags: the type of memory to allocate.
+ *
+ * The contents of the object pointed to are preserved up to the
+ * lesser of the new and old sizes. If @p is %NULL, krealloc()
+ * behaves exactly like kmalloc(). If @size is 0 and @p is not a
+ * %NULL pointer, the object pointed to is freed.
+ */
+void *krealloc(const void *p, size_t new_size, gfp_t flags)
+{
+ void *ret;
+
+ if (unlikely(!new_size)) {
kfree(p);
+ return ZERO_SIZE_PTR;
}
+
+ ret = __krealloc(p, new_size, flags);
+ if (ret && p != ret)
+ kfree(p);
+
return ret;
}
EXPORT_SYMBOL(krealloc);
diff --git a/net/netfilter/nf_conntrack_extend.c b/net/netfilter/nf_conntrack_extend.c
index 3469bc7..c956ef7 100644
--- a/net/netfilter/nf_conntrack_extend.c
+++ b/net/netfilter/nf_conntrack_extend.c
@@ -95,7 +95,7 @@ void *__nf_ct_ext_add(struct nf_conn *ct, enum nf_ct_ext_id id, gfp_t gfp)
newlen = newoff + t->len;
rcu_read_unlock();

- new = krealloc(ct->ext, newlen, gfp);
+ new = __krealloc(ct->ext, newlen, gfp);
if (!new)
return NULL;

2008-07-24 13:31:47

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

Pekka Enberg wrote:
> On Thu, 2008-07-24 at 14:49 +0200, Patrick McHardy wrote:
>> Pekka Enberg wrote:
>>> Hmm, why do you think it's related? It looks like elem->hook() is a
>>> NULL pointer but my patch shouldn't make any difference here...
>> No, its already in ipv4_confirm, so its most likely helper->help()
>> thats NULL, which is contained in an extension.
>>
>> The reason why I think its this patch is (quoting from an old
>> email that I never got a response to):
>
> Oh, I'm really sorry about that.

No problem :)

>> ---
>> Your patch introduced a use-after-free and double-free.
>> krealloc() frees the old pointer, but it is still used
>> for the ->move operations, then freed again.
>>
>> To fix this I think we need a __krealloc() that doesn't
>> free the old memory, especially since it must not be
>> freed immediately because it may still be used in a RCU
>> read side (see the last part in the patch attached to
>> this mail (based on a kernel without your patch)).
>
> Agreed. Something like this, perhaps?
>
> [PATCH] netfilter: fix double-free and use-after free
>
> As suggested by Patrick McHardy, introduce a __krealloc() that doesn't
> free the original buffer to fix a double-free and use-after-free bug
> introduced by me in netfilter that uses RCU.
>
> Reported-by: Patrick McHardy <[email protected]>
> Signed-off-by: Pekka Enberg <[email protected]>

Looks good to me, thanks.

2008-07-24 13:32:42

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, Jul 24, 2008 at 10:56:08PM +1000, Nick Piggin wrote:
> > > OTOH, skb allocation uses kmalloc don't they? So you could still use
> > > SLOB ksize for that I guess.

On Thursday 24 July 2008 23:04, Herbert Xu wrote:
> > Yes I was referring to the data portion which is kmalloc'ed.
> > That is also why I'm interested in ksize because a priori we
> > don't know exactly how big it's going to be. However, we do
> > know that statistically 1500 will dominate.
> >
> > I'm not interested in ksize for kmem_cache at all. So in fact
> > we could have something simpler that's based on kmalloc's rounding
> > algorithm instead.

On Thu, 2008-07-24 at 23:13 +1000, Nick Piggin wrote:
> Yes you could definitely have a function that returns allocated
> bytes for a given kmalloc size. Should be about as fast or faster
> than extracting the size from the kaddr...

Yup, makes sense.

2008-07-24 13:34:51

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, 2008-07-24 at 15:31 +0200, Patrick McHardy wrote:
> Pekka Enberg wrote:
> > On Thu, 2008-07-24 at 14:49 +0200, Patrick McHardy wrote:
> >> Pekka Enberg wrote:
> >>> Hmm, why do you think it's related? It looks like elem->hook() is a
> >>> NULL pointer but my patch shouldn't make any difference here...
> >> No, its already in ipv4_confirm, so its most likely helper->help()
> >> thats NULL, which is contained in an extension.
> >>
> >> The reason why I think its this patch is (quoting from an old
> >> email that I never got a response to):
> >
> > Oh, I'm really sorry about that.
>
> No problem :)
>
> >> ---
> >> Your patch introduced a use-after-free and double-free.
> >> krealloc() frees the old pointer, but it is still used
> >> for the ->move operations, then freed again.
> >>
> >> To fix this I think we need a __krealloc() that doesn't
> >> free the old memory, especially since it must not be
> >> freed immediately because it may still be used in a RCU
> >> read side (see the last part in the patch attached to
> >> this mail (based on a kernel without your patch)).
> >
> > Agreed. Something like this, perhaps?
> >
> > [PATCH] netfilter: fix double-free and use-after free
> >
> > As suggested by Patrick McHardy, introduce a __krealloc() that doesn't
> > free the original buffer to fix a double-free and use-after-free bug
> > introduced by me in netfilter that uses RCU.
> >
> > Reported-by: Patrick McHardy <[email protected]>
> > Signed-off-by: Pekka Enberg <[email protected]>
>
> Looks good to me, thanks.

Ingo, can you please test this? Andrew, I'm at OLS so can you pick up
the patch in your tree?

2008-07-24 14:25:34

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

* Ingo Molnar <[email protected]> wrote:

> here's the full bootlog:
>
> http://redhat.com/~mingo/misc/crash-Thu_Jul_24_13_23_34_CEST_2008.log
>
> kernel is latest -git, v2.6.26-6371-g338b9bb.

FYI, this was the most complex bisection i have ever done under Linux.

Firstly, automated bisection honed in on an already known bad commit, so
i had to do another, manual bisection.

There i hit four other regression and i had to work them around at
multiple bisection points to be able to bisect this regression. I also
had to do a "nested" sub-bisection to fix one of the bisection points.

In case someone else has to bisect in the future in this region as well,
here is the list:

1) I hit the mac802 hwsim NULL dereference bootup crash and
cherry-picked its fix, 3a33cc108d1. Alas, that didnt work - so i
tweaked the .config. (hoping that it would not change the crash
pattern - fortunately it didnt)

2) build failure:

net/built-in.o: In function `dev_queue_xmit':
: undefined reference to `qdisc_calculate_pkt_len'
net/built-in.o: In function `__qdisc_destroy':
sch_generic.c:(.text+0x22874): undefined reference to qdisc_put_stab'
net/built-in.o: In function `ieee80211_requeue':
: undefined reference to `qdisc_calculate_pkt_len'

had to do a secondary, nested bisection to figure out that the
build at this commit point was broken by 175f9c1bba9b ("net_sched:
Add size table for qdiscs") and reverted it.

3) on bisection point 11ea859d64b i got a hard lockup:

BUG: NMI Watchdog detected LOCKUP on CPU0, ip ffffffff8028ef43, registers:

Call Trace:
[<ffffffff8028f3c6>] slob_alloc+0x9c/0x248
[<ffffffff8028f59d>] kmem_cache_alloc_node+0x2b/0x5e
[<ffffffff80292e42>] get_empty_filp+0x4f/0xee
[<ffffffff8029c635>] __path_lookup_intent_open+0x2f/0x9f
[<ffffffff8029c719>] path_lookup_open+0xc/0xe
[<ffffffff8029c7c8>] do_filp_open+0xad/0x7e7
[<ffffffff80272bec>] ? trace_preempt_on+0x9/0xb
[<ffffffff802419d9>] ? sub_preempt_count+0x2d/0x40
[<ffffffff802903ca>] ? get_unused_fd_flags+0x110/0x128
[<ffffffff80290433>] do_sys_open+0x51/0xd8
[<ffffffff802904e3>] sys_open+0x1b/0x1d
[<ffffffff802247aa>] system_call_after_swapgs+0x9a/0x9f

This might or might not be the same regression so i picked another
bisection point from the middle of the range, 11ea859d64b.

4) On one of the last bisection points (c3ee841) i hit this KVM
regression:

arch/x86/kvm/built-in.o:(.text.fixup+0x1): relocation truncated to fit: R_X86_64_32 against `.text'

i cherry-picked the fix 33a37eb411d.

Thus in this second bisection i arrived to:

| ae6134bdf3197206fba95563d755d2fa50d90ddd is first bad commit
| commit ae6134bdf3197206fba95563d755d2fa50d90ddd
| Author: Micah Dowty <[email protected]>
| Date: Mon Jul 21 09:59:09 2008 -0700
|
| hdlcdrv: Fix CRC calculation.

Ok: recent change, networking related, i had this driver enabled, looks
plausible. Here's the bisection log:

# bad: [338b9bb2] Merge branch 'x86/auditsc' of git://git.kernel.org
# good: [bce7f795] Linux 2.6.26
# good: [bce7f795] Linux 2.6.26
# good: [bce7f795] Linux 2.6.26
# good: [43146521] Merge branch 'release-2.6.27' of git://git.kernel.
# good: [906f25b3] Revert "net_sched: Add size table for qdiscs"
# good: [72a73693] Merge branch 'x86/for-linus' of git://git.kernel.o
# bad: [0988c371] Merge branch 'x86-fixes-for-linus' of git://git.ke
# bad: [a6e2ba82] block: make /proc/diskstats only build if CONFIG_P
# bad: [e89970a5] Merge git://git.kernel.org/pub/scm/linux/kernel/gi
# bad: [ae6134bd] hdlcdrv: Fix CRC calculation.
# good: [c3ee841b] pkt_sched: Remove unused variable skb in dev_deact
# good: [867d79f4] net: In __netif_schedule() use WARN_ON instead of
# good: [d3678b4b] Revert "pkt_sched: Make default qdisc nonshared-mu

[ and this bisection found a small Git buglet as well: it shows a
small git-bisect buglet. (Those three 2.6.26 log entries are there
because git-bisect failed to check out the target bisection point two
times due to untracked files.) ]

So i reverted "hdlcdrv: Fix CRC calculation." (ae6134bdf) ...

... but got the same crash again. So either the crash is not
deterministic or one of the bisection points had to be wrong. (which,
given the multitude of other regressions, is really not a surprise)

Then i tried both suggested fix patches Patrick sent me (a suggested
revert and an netfilter/RCU use-after-free fix), but none of them solved
the crash.

So i looked at the failed bisection log again, and noticed that the
'bad' commit ae6134bdf is just next to a string of netfilter commits:

5547cd0: netfilter: nf_conntrack_sctp: fix sparse warnings
c71529e: netfilter: nf_nat_sip: c= is optional for session
db1a75b: netfilter: xt_TCPMSS: collapse tcpmss_reverse_mtu{4,6} into one function
72961ec: netfilter: nfnetlink_log: send complete hardware header
280763c: netfilter: xt_time: fix time's time_mt()'s use of do_div()
5840157: netfilter: accounting rework: ct_extend + 64bit counters (v4)
07a7c10: netlink: add NLA_PUT_BE64 macro
0dbff68: netfilter: nf_nat_core: eliminate useless find_appropriate_src for IP_N
ae6134b: hdlcdrv: Fix CRC calculation.

... and the crash was in netfilter after all.

So i cornered the bug by checking:

5547cd0 => crash
ae6134bdf => good

... and starting off that point again, with the third bisection.

Thus i finally arrived to:

# good: [ae6134bd] hdlcdrv: Fix CRC calculation.
# bad: [5547cd0d] netfilter: nf_conntrack_sctp: fix sparse warnings
# bad: [280763c6] netfilter: xt_time: fix time's time_mt()'s use of
# good: [07a7c10b] netlink: add NLA_PUT_BE64 macro
# bad: [58401573] netfilter: accounting rework: ct_extend + 64bit co

| 584015727a3b88b46602b20077b46cd04f8b4ab3 is first bad commit
| commit 584015727a3b88b46602b20077b46cd04f8b4ab3
| Author: Krzysztof Piotr Oledzki <[email protected]>
| AuthorDate: Mon Jul 21 10:01:34 2008 -0700
| Commit: David S. Miller <[email protected]>
| CommitDate: Mon Jul 21 10:10:58 2008 -0700
|
| netfilter: accounting rework: ct_extend + 64bit counters (v4)
[...]
| Signed-off-by: Krzysztof Piotr Oledzki <[email protected]>
| Signed-off-by: Patrick McHardy <[email protected]>
| Signed-off-by: David S. Miller <[email protected]>

Which i double-checked by reverting that commit from -git as well and
that solved the crash. Find the tested reverter patch below.

Ingo

----------------------->
commit 548d9ef5fc65d921d20528de7b4d50e6cf0a1a15
Author: Ingo Molnar <[email protected]>
Date: Thu Jul 24 15:16:22 2008 +0200

Revert "netfilter: accounting rework: ct_extend + 64bit counters (v4)"

This reverts commit 584015727a3b88b46602b20077b46cd04f8b4ab3.

Signed-off-by: Ingo Molnar <[email protected]>

---
Documentation/feature-removal-schedule.txt | 10 --
Documentation/kernel-parameters.txt | 7 --
include/linux/netfilter/nf_conntrack_common.h | 8 ++-
include/linux/netfilter/nfnetlink_conntrack.h | 8 +-
include/net/netfilter/nf_conntrack.h | 6 +
include/net/netfilter/nf_conntrack_acct.h | 51 ----------
include/net/netfilter/nf_conntrack_extend.h | 2 -
.../netfilter/nf_conntrack_l3proto_ipv4_compat.c | 18 +++-
net/netfilter/Kconfig | 9 --
net/netfilter/Makefile | 2 +-
net/netfilter/nf_conntrack_acct.c | 104 --------------------
net/netfilter/nf_conntrack_core.c | 39 +++-----
net/netfilter/nf_conntrack_netlink.c | 44 +++++----
net/netfilter/nf_conntrack_standalone.c | 18 +++-
net/netfilter/xt_connbytes.c | 8 +-
15 files changed, 86 insertions(+), 248 deletions(-)

diff --git a/Documentation/feature-removal-schedule.txt b/Documentation/feature-removal-schedule.txt
index 9f73587..86334b6 100644
--- a/Documentation/feature-removal-schedule.txt
+++ b/Documentation/feature-removal-schedule.txt
@@ -336,13 +336,3 @@ When: After the only user (hal) has seen a release with the patches
Why: Over 1K .text/.data size reduction, data is available in other
ways (ioctls)
Who: Johannes Berg <[email protected]>
-
----------------------------
-
-What: CONFIG_NF_CT_ACCT
-When: 2.6.29
-Why: Accounting can now be enabled/disabled without kernel recompilation.
- Currently used only to set a default value for a feature that is also
- controlled by a kernel/module/sysfs/sysctl parameter.
-Who: Krzysztof Piotr Oledzki <[email protected]>
-
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 931f960..1d7e6b0 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -1279,13 +1279,6 @@ and is between 256 and 4096 characters. It is defined in the file
This usage is only documented in each driver source
file if at all.

- nf_conntrack.acct=
- [NETFILTER] Enable connection tracking flow accounting
- 0 to disable accounting
- 1 to enable accounting
- Default value depends on CONFIG_NF_CT_ACCT that is
- going to be removed in 2.6.29.
-
nfsaddrs= [NFS]
See Documentation/filesystems/nfsroot.txt.

diff --git a/include/linux/netfilter/nf_conntrack_common.h b/include/linux/netfilter/nf_conntrack_common.h
index 885cbe2..bad1eb7 100644
--- a/include/linux/netfilter/nf_conntrack_common.h
+++ b/include/linux/netfilter/nf_conntrack_common.h
@@ -122,7 +122,7 @@ enum ip_conntrack_events
IPCT_NATINFO_BIT = 10,
IPCT_NATINFO = (1 << IPCT_NATINFO_BIT),

- /* Counter highest bit has been set, unused */
+ /* Counter highest bit has been set */
IPCT_COUNTER_FILLING_BIT = 11,
IPCT_COUNTER_FILLING = (1 << IPCT_COUNTER_FILLING_BIT),

@@ -145,6 +145,12 @@ enum ip_conntrack_expect_events {
};

#ifdef __KERNEL__
+struct ip_conntrack_counter
+{
+ u_int32_t packets;
+ u_int32_t bytes;
+};
+
struct ip_conntrack_stat
{
unsigned int searched;
diff --git a/include/linux/netfilter/nfnetlink_conntrack.h b/include/linux/netfilter/nfnetlink_conntrack.h
index c19595c..759bc04 100644
--- a/include/linux/netfilter/nfnetlink_conntrack.h
+++ b/include/linux/netfilter/nfnetlink_conntrack.h
@@ -115,10 +115,10 @@ enum ctattr_protoinfo_sctp {

enum ctattr_counters {
CTA_COUNTERS_UNSPEC,
- CTA_COUNTERS_PACKETS, /* 64bit counters */
- CTA_COUNTERS_BYTES, /* 64bit counters */
- CTA_COUNTERS32_PACKETS, /* old 32bit counters, unused */
- CTA_COUNTERS32_BYTES, /* old 32bit counters, unused */
+ CTA_COUNTERS_PACKETS, /* old 64bit counters */
+ CTA_COUNTERS_BYTES, /* old 64bit counters */
+ CTA_COUNTERS32_PACKETS,
+ CTA_COUNTERS32_BYTES,
__CTA_COUNTERS_MAX
};
#define CTA_COUNTERS_MAX (__CTA_COUNTERS_MAX - 1)
diff --git a/include/net/netfilter/nf_conntrack.h b/include/net/netfilter/nf_conntrack.h
index 0741ad5..8f5b757 100644
--- a/include/net/netfilter/nf_conntrack.h
+++ b/include/net/netfilter/nf_conntrack.h
@@ -88,6 +88,7 @@ struct nf_conn_help {
u8 expecting[NF_CT_MAX_EXPECT_CLASSES];
};

+
#include <net/netfilter/ipv4/nf_conntrack_ipv4.h>
#include <net/netfilter/ipv6/nf_conntrack_ipv6.h>

@@ -110,6 +111,11 @@ struct nf_conn
/* Timer function; drops refcnt when it goes off. */
struct timer_list timeout;

+#ifdef CONFIG_NF_CT_ACCT
+ /* Accounting Information (same cache line as other written members) */
+ struct ip_conntrack_counter counters[IP_CT_DIR_MAX];
+#endif
+
#if defined(CONFIG_NF_CONNTRACK_MARK)
u_int32_t mark;
#endif
diff --git a/include/net/netfilter/nf_conntrack_acct.h b/include/net/netfilter/nf_conntrack_acct.h
deleted file mode 100644
index 5d5ae55..0000000
--- a/include/net/netfilter/nf_conntrack_acct.h
+++ /dev/null
@@ -1,51 +0,0 @@
-/*
- * (C) 2008 Krzysztof Piotr Oledzki <[email protected]>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#ifndef _NF_CONNTRACK_ACCT_H
-#define _NF_CONNTRACK_ACCT_H
-#include <linux/netfilter/nf_conntrack_common.h>
-#include <linux/netfilter/nf_conntrack_tuple_common.h>
-#include <net/netfilter/nf_conntrack.h>
-#include <net/netfilter/nf_conntrack_extend.h>
-
-struct nf_conn_counter {
- u_int64_t packets;
- u_int64_t bytes;
-};
-
-extern int nf_ct_acct;
-
-static inline
-struct nf_conn_counter *nf_conn_acct_find(const struct nf_conn *ct)
-{
- return nf_ct_ext_find(ct, NF_CT_EXT_ACCT);
-}
-
-static inline
-struct nf_conn_counter *nf_ct_acct_ext_add(struct nf_conn *ct, gfp_t gfp)
-{
- struct nf_conn_counter *acct;
-
- if (!nf_ct_acct)
- return NULL;
-
- acct = nf_ct_ext_add(ct, NF_CT_EXT_ACCT, gfp);
- if (!acct)
- pr_debug("failed to add accounting extension area");
-
-
- return acct;
-};
-
-extern unsigned int
-seq_print_acct(struct seq_file *s, const struct nf_conn *ct, int dir);
-
-extern int nf_conntrack_acct_init(void);
-extern void nf_conntrack_acct_fini(void);
-
-#endif /* _NF_CONNTRACK_ACCT_H */
diff --git a/include/net/netfilter/nf_conntrack_extend.h b/include/net/netfilter/nf_conntrack_extend.h
index da8ee52..f80c0ed 100644
--- a/include/net/netfilter/nf_conntrack_extend.h
+++ b/include/net/netfilter/nf_conntrack_extend.h
@@ -7,13 +7,11 @@ enum nf_ct_ext_id
{
NF_CT_EXT_HELPER,
NF_CT_EXT_NAT,
- NF_CT_EXT_ACCT,
NF_CT_EXT_NUM,
};

#define NF_CT_EXT_HELPER_TYPE struct nf_conn_help
#define NF_CT_EXT_NAT_TYPE struct nf_conn_nat
-#define NF_CT_EXT_ACCT_TYPE struct nf_conn_counter

/* Extensions: optional stuff which isn't permanently in struct. */
struct nf_ct_ext {
diff --git a/net/ipv4/netfilter/nf_conntrack_l3proto_ipv4_compat.c b/net/ipv4/netfilter/nf_conntrack_l3proto_ipv4_compat.c
index 3a02072..40a46d4 100644
--- a/net/ipv4/netfilter/nf_conntrack_l3proto_ipv4_compat.c
+++ b/net/ipv4/netfilter/nf_conntrack_l3proto_ipv4_compat.c
@@ -18,7 +18,19 @@
#include <net/netfilter/nf_conntrack_l3proto.h>
#include <net/netfilter/nf_conntrack_l4proto.h>
#include <net/netfilter/nf_conntrack_expect.h>
-#include <net/netfilter/nf_conntrack_acct.h>
+
+#ifdef CONFIG_NF_CT_ACCT
+static unsigned int
+seq_print_counters(struct seq_file *s,
+ const struct ip_conntrack_counter *counter)
+{
+ return seq_printf(s, "packets=%llu bytes=%llu ",
+ (unsigned long long)counter->packets,
+ (unsigned long long)counter->bytes);
+}
+#else
+#define seq_print_counters(x, y) 0
+#endif

struct ct_iter_state {
unsigned int bucket;
@@ -115,7 +127,7 @@ static int ct_seq_show(struct seq_file *s, void *v)
l3proto, l4proto))
return -ENOSPC;

- if (seq_print_acct(s, ct, IP_CT_DIR_ORIGINAL))
+ if (seq_print_counters(s, &ct->counters[IP_CT_DIR_ORIGINAL]))
return -ENOSPC;

if (!(test_bit(IPS_SEEN_REPLY_BIT, &ct->status)))
@@ -126,7 +138,7 @@ static int ct_seq_show(struct seq_file *s, void *v)
l3proto, l4proto))
return -ENOSPC;

- if (seq_print_acct(s, ct, IP_CT_DIR_REPLY))
+ if (seq_print_counters(s, &ct->counters[IP_CT_DIR_REPLY]))
return -ENOSPC;

if (test_bit(IPS_ASSURED_BIT, &ct->status))
diff --git a/net/netfilter/Kconfig b/net/netfilter/Kconfig
index ee898e7..316c7af 100644
--- a/net/netfilter/Kconfig
+++ b/net/netfilter/Kconfig
@@ -49,15 +49,6 @@ config NF_CT_ACCT
Those counters can be used for flow-based accounting or the
`connbytes' match.

- Please note that currently this option only sets a default state.
- You may change it at boot time with nf_conntrack.acct=0/1 kernel
- paramater or by loading the nf_conntrack module with acct=0/1.
-
- You may also disable/enable it on a running system with:
- sysctl net.netfilter.nf_conntrack_acct=0/1
-
- This option will be removed in 2.6.29.
-
If unsure, say `N'.

config NF_CONNTRACK_MARK
diff --git a/net/netfilter/Makefile b/net/netfilter/Makefile
index 3bd2cc5..5c4b183 100644
--- a/net/netfilter/Makefile
+++ b/net/netfilter/Makefile
@@ -1,6 +1,6 @@
netfilter-objs := core.o nf_log.o nf_queue.o nf_sockopt.o

-nf_conntrack-y := nf_conntrack_core.o nf_conntrack_standalone.o nf_conntrack_expect.o nf_conntrack_helper.o nf_conntrack_proto.o nf_conntrack_l3proto_generic.o nf_conntrack_proto_generic.o nf_conntrack_proto_tcp.o nf_conntrack_proto_udp.o nf_conntrack_extend.o nf_conntrack_acct.o
+nf_conntrack-y := nf_conntrack_core.o nf_conntrack_standalone.o nf_conntrack_expect.o nf_conntrack_helper.o nf_conntrack_proto.o nf_conntrack_l3proto_generic.o nf_conntrack_proto_generic.o nf_conntrack_proto_tcp.o nf_conntrack_proto_udp.o nf_conntrack_extend.o
nf_conntrack-$(CONFIG_NF_CONNTRACK_EVENTS) += nf_conntrack_ecache.o

obj-$(CONFIG_NETFILTER) = netfilter.o
diff --git a/net/netfilter/nf_conntrack_acct.c b/net/netfilter/nf_conntrack_acct.c
deleted file mode 100644
index 59bd8b9..0000000
--- a/net/netfilter/nf_conntrack_acct.c
+++ /dev/null
@@ -1,104 +0,0 @@
-/* Accouting handling for netfilter. */
-
-/*
- * (C) 2008 Krzysztof Piotr Oledzki <[email protected]>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/netfilter.h>
-#include <linux/kernel.h>
-#include <linux/moduleparam.h>
-
-#include <net/netfilter/nf_conntrack.h>
-#include <net/netfilter/nf_conntrack_extend.h>
-#include <net/netfilter/nf_conntrack_acct.h>
-
-#ifdef CONFIG_NF_CT_ACCT
-#define NF_CT_ACCT_DEFAULT 1
-#else
-#define NF_CT_ACCT_DEFAULT 0
-#endif
-
-int nf_ct_acct __read_mostly = NF_CT_ACCT_DEFAULT;
-EXPORT_SYMBOL_GPL(nf_ct_acct);
-
-module_param_named(acct, nf_ct_acct, bool, 0644);
-MODULE_PARM_DESC(acct, "Enable connection tracking flow accounting.");
-
-#ifdef CONFIG_SYSCTL
-static struct ctl_table_header *acct_sysctl_header;
-static struct ctl_table acct_sysctl_table[] = {
- {
- .ctl_name = CTL_UNNUMBERED,
- .procname = "nf_conntrack_acct",
- .data = &nf_ct_acct,
- .maxlen = sizeof(unsigned int),
- .mode = 0644,
- .proc_handler = &proc_dointvec,
- },
- {}
-};
-#endif /* CONFIG_SYSCTL */
-
-unsigned int
-seq_print_acct(struct seq_file *s, const struct nf_conn *ct, int dir)
-{
- struct nf_conn_counter *acct;
-
- acct = nf_conn_acct_find(ct);
- if (!acct)
- return 0;
-
- return seq_printf(s, "packets=%llu bytes=%llu ",
- (unsigned long long)acct[dir].packets,
- (unsigned long long)acct[dir].bytes);
-};
-EXPORT_SYMBOL_GPL(seq_print_acct);
-
-static struct nf_ct_ext_type acct_extend __read_mostly = {
- .len = sizeof(struct nf_conn_counter[IP_CT_DIR_MAX]),
- .align = __alignof__(struct nf_conn_counter[IP_CT_DIR_MAX]),
- .id = NF_CT_EXT_ACCT,
-};
-
-int nf_conntrack_acct_init(void)
-{
- int ret;
-
-#ifdef CONFIG_NF_CT_ACCT
- printk(KERN_WARNING "CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Plase use\n");
- printk(KERN_WARNING "nf_conntrack.acct=1 kernel paramater, acct=1 nf_conntrack module option or\n");
- printk(KERN_WARNING "sysctl net.netfilter.nf_conntrack_acct=1 to enable it.\n");
-#endif
-
- ret = nf_ct_extend_register(&acct_extend);
- if (ret < 0) {
- printk(KERN_ERR "nf_conntrack_acct: Unable to register extension\n");
- return ret;
- }
-
-#ifdef CONFIG_SYSCTL
- acct_sysctl_header = register_sysctl_paths(nf_net_netfilter_sysctl_path,
- acct_sysctl_table);
-
- if (!acct_sysctl_header) {
- nf_ct_extend_unregister(&acct_extend);
-
- printk(KERN_ERR "nf_conntrack_acct: can't register to sysctl.\n");
- return -ENOMEM;
- }
-#endif
-
- return 0;
-}
-
-void nf_conntrack_acct_fini(void)
-{
-#ifdef CONFIG_SYSCTL
- unregister_sysctl_table(acct_sysctl_header);
-#endif
- nf_ct_extend_unregister(&acct_extend);
-}
diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c
index c519d09..28d03e6 100644
--- a/net/netfilter/nf_conntrack_core.c
+++ b/net/netfilter/nf_conntrack_core.c
@@ -37,7 +37,6 @@
#include <net/netfilter/nf_conntrack_helper.h>
#include <net/netfilter/nf_conntrack_core.h>
#include <net/netfilter/nf_conntrack_extend.h>
-#include <net/netfilter/nf_conntrack_acct.h>

#define NF_CONNTRACK_VERSION "0.5.0"

@@ -556,8 +555,6 @@ init_conntrack(const struct nf_conntrack_tuple *tuple,
return NULL;
}

- nf_ct_acct_ext_add(ct, GFP_ATOMIC);
-
spin_lock_bh(&nf_conntrack_lock);
exp = nf_ct_find_expectation(tuple);
if (exp) {
@@ -831,16 +828,17 @@ void __nf_ct_refresh_acct(struct nf_conn *ct,
}

acct:
+#ifdef CONFIG_NF_CT_ACCT
if (do_acct) {
- struct nf_conn_counter *acct;
+ ct->counters[CTINFO2DIR(ctinfo)].packets++;
+ ct->counters[CTINFO2DIR(ctinfo)].bytes +=
+ skb->len - skb_network_offset(skb);

- acct = nf_conn_acct_find(ct);
- if (acct) {
- acct[CTINFO2DIR(ctinfo)].packets++;
- acct[CTINFO2DIR(ctinfo)].bytes +=
- skb->len - skb_network_offset(skb);
- }
+ if ((ct->counters[CTINFO2DIR(ctinfo)].packets & 0x80000000)
+ || (ct->counters[CTINFO2DIR(ctinfo)].bytes & 0x80000000))
+ event |= IPCT_COUNTER_FILLING;
}
+#endif

spin_unlock_bh(&nf_conntrack_lock);

@@ -855,19 +853,15 @@ bool __nf_ct_kill_acct(struct nf_conn *ct,
const struct sk_buff *skb,
int do_acct)
{
+#ifdef CONFIG_NF_CT_ACCT
if (do_acct) {
- struct nf_conn_counter *acct;
-
spin_lock_bh(&nf_conntrack_lock);
- acct = nf_conn_acct_find(ct);
- if (acct) {
- acct[CTINFO2DIR(ctinfo)].packets++;
- acct[CTINFO2DIR(ctinfo)].bytes +=
- skb->len - skb_network_offset(skb);
- }
+ ct->counters[CTINFO2DIR(ctinfo)].packets++;
+ ct->counters[CTINFO2DIR(ctinfo)].bytes +=
+ skb->len - skb_network_offset(skb);
spin_unlock_bh(&nf_conntrack_lock);
}
-
+#endif
if (del_timer(&ct->timeout)) {
ct->timeout.function((unsigned long)ct);
return true;
@@ -1035,7 +1029,6 @@ void nf_conntrack_cleanup(void)
nf_conntrack_proto_fini();
nf_conntrack_helper_fini();
nf_conntrack_expect_fini();
- nf_conntrack_acct_fini();
}

struct hlist_head *nf_ct_alloc_hashtable(unsigned int *sizep, int *vmalloced)
@@ -1175,10 +1168,6 @@ int __init nf_conntrack_init(void)
if (ret < 0)
goto out_fini_expect;

- ret = nf_conntrack_acct_init();
- if (ret < 0)
- goto out_fini_helper;
-
/* For use by REJECT target */
rcu_assign_pointer(ip_ct_attach, nf_conntrack_attach);
rcu_assign_pointer(nf_ct_destroy, destroy_conntrack);
@@ -1191,8 +1180,6 @@ int __init nf_conntrack_init(void)

return ret;

-out_fini_helper:
- nf_conntrack_helper_fini();
out_fini_expect:
nf_conntrack_expect_fini();
out_fini_proto:
diff --git a/net/netfilter/nf_conntrack_netlink.c b/net/netfilter/nf_conntrack_netlink.c
index 105a616..95a7967 100644
--- a/net/netfilter/nf_conntrack_netlink.c
+++ b/net/netfilter/nf_conntrack_netlink.c
@@ -37,7 +37,6 @@
#include <net/netfilter/nf_conntrack_l3proto.h>
#include <net/netfilter/nf_conntrack_l4proto.h>
#include <net/netfilter/nf_conntrack_tuple.h>
-#include <net/netfilter/nf_conntrack_acct.h>
#ifdef CONFIG_NF_NAT_NEEDED
#include <net/netfilter/nf_nat_core.h>
#include <net/netfilter/nf_nat_protocol.h>
@@ -207,26 +206,22 @@ nla_put_failure:
return -1;
}

+#ifdef CONFIG_NF_CT_ACCT
static int
ctnetlink_dump_counters(struct sk_buff *skb, const struct nf_conn *ct,
enum ip_conntrack_dir dir)
{
enum ctattr_type type = dir ? CTA_COUNTERS_REPLY: CTA_COUNTERS_ORIG;
struct nlattr *nest_count;
- const struct nf_conn_counter *acct;
-
- acct = nf_conn_acct_find(ct);
- if (!acct)
- return 0;

nest_count = nla_nest_start(skb, type | NLA_F_NESTED);
if (!nest_count)
goto nla_put_failure;

- NLA_PUT_BE64(skb, CTA_COUNTERS_PACKETS,
- cpu_to_be64(acct[dir].packets));
- NLA_PUT_BE64(skb, CTA_COUNTERS_BYTES,
- cpu_to_be64(acct[dir].bytes));
+ NLA_PUT_BE32(skb, CTA_COUNTERS32_PACKETS,
+ htonl(ct->counters[dir].packets));
+ NLA_PUT_BE32(skb, CTA_COUNTERS32_BYTES,
+ htonl(ct->counters[dir].bytes));

nla_nest_end(skb, nest_count);

@@ -235,6 +230,9 @@ ctnetlink_dump_counters(struct sk_buff *skb, const struct nf_conn *ct,
nla_put_failure:
return -1;
}
+#else
+#define ctnetlink_dump_counters(a, b, c) (0)
+#endif

#ifdef CONFIG_NF_CONNTRACK_MARK
static inline int
@@ -503,6 +501,11 @@ static int ctnetlink_conntrack_event(struct notifier_block *this,
goto nla_put_failure;
#endif

+ if (events & IPCT_COUNTER_FILLING &&
+ (ctnetlink_dump_counters(skb, ct, IP_CT_DIR_ORIGINAL) < 0 ||
+ ctnetlink_dump_counters(skb, ct, IP_CT_DIR_REPLY) < 0))
+ goto nla_put_failure;
+
if (events & IPCT_RELATED &&
ctnetlink_dump_master(skb, ct) < 0)
goto nla_put_failure;
@@ -573,15 +576,11 @@ restart:
cb->args[1] = (unsigned long)ct;
goto out;
}
-
+#ifdef CONFIG_NF_CT_ACCT
if (NFNL_MSG_TYPE(cb->nlh->nlmsg_type) ==
- IPCTNL_MSG_CT_GET_CTRZERO) {
- struct nf_conn_counter *acct;
-
- acct = nf_conn_acct_find(ct);
- if (acct)
- memset(acct, 0, sizeof(struct nf_conn_counter[IP_CT_DIR_MAX]));
- }
+ IPCTNL_MSG_CT_GET_CTRZERO)
+ memset(&ct->counters, 0, sizeof(ct->counters));
+#endif
}
if (cb->args[1]) {
cb->args[1] = 0;
@@ -833,9 +832,14 @@ ctnetlink_get_conntrack(struct sock *ctnl, struct sk_buff *skb,
u_int8_t u3 = nfmsg->nfgen_family;
int err = 0;

- if (nlh->nlmsg_flags & NLM_F_DUMP)
+ if (nlh->nlmsg_flags & NLM_F_DUMP) {
+#ifndef CONFIG_NF_CT_ACCT
+ if (NFNL_MSG_TYPE(nlh->nlmsg_type) == IPCTNL_MSG_CT_GET_CTRZERO)
+ return -ENOTSUPP;
+#endif
return netlink_dump_start(ctnl, skb, nlh, ctnetlink_dump_table,
ctnetlink_done);
+ }

if (cda[CTA_TUPLE_ORIG])
err = ctnetlink_parse_tuple(cda, &tuple, CTA_TUPLE_ORIG, u3);
@@ -1148,8 +1152,6 @@ ctnetlink_create_conntrack(struct nlattr *cda[],
goto err;
}

- nf_ct_acct_ext_add(ct, GFP_KERNEL);
-
#if defined(CONFIG_NF_CONNTRACK_MARK)
if (cda[CTA_MARK])
ct->mark = ntohl(nla_get_be32(cda[CTA_MARK]));
diff --git a/net/netfilter/nf_conntrack_standalone.c b/net/netfilter/nf_conntrack_standalone.c
index 869ef93..46ea542 100644
--- a/net/netfilter/nf_conntrack_standalone.c
+++ b/net/netfilter/nf_conntrack_standalone.c
@@ -25,7 +25,6 @@
#include <net/netfilter/nf_conntrack_l4proto.h>
#include <net/netfilter/nf_conntrack_expect.h>
#include <net/netfilter/nf_conntrack_helper.h>
-#include <net/netfilter/nf_conntrack_acct.h>

MODULE_LICENSE("GPL");

@@ -39,6 +38,19 @@ print_tuple(struct seq_file *s, const struct nf_conntrack_tuple *tuple,
}
EXPORT_SYMBOL_GPL(print_tuple);

+#ifdef CONFIG_NF_CT_ACCT
+static unsigned int
+seq_print_counters(struct seq_file *s,
+ const struct ip_conntrack_counter *counter)
+{
+ return seq_printf(s, "packets=%llu bytes=%llu ",
+ (unsigned long long)counter->packets,
+ (unsigned long long)counter->bytes);
+}
+#else
+#define seq_print_counters(x, y) 0
+#endif
+
struct ct_iter_state {
unsigned int bucket;
};
@@ -134,7 +146,7 @@ static int ct_seq_show(struct seq_file *s, void *v)
l3proto, l4proto))
return -ENOSPC;

- if (seq_print_acct(s, ct, IP_CT_DIR_ORIGINAL))
+ if (seq_print_counters(s, &ct->counters[IP_CT_DIR_ORIGINAL]))
return -ENOSPC;

if (!(test_bit(IPS_SEEN_REPLY_BIT, &ct->status)))
@@ -145,7 +157,7 @@ static int ct_seq_show(struct seq_file *s, void *v)
l3proto, l4proto))
return -ENOSPC;

- if (seq_print_acct(s, ct, IP_CT_DIR_REPLY))
+ if (seq_print_counters(s, &ct->counters[IP_CT_DIR_REPLY]))
return -ENOSPC;

if (test_bit(IPS_ASSURED_BIT, &ct->status))
diff --git a/net/netfilter/xt_connbytes.c b/net/netfilter/xt_connbytes.c
index 3e39c4f..d7e8983 100644
--- a/net/netfilter/xt_connbytes.c
+++ b/net/netfilter/xt_connbytes.c
@@ -8,7 +8,6 @@
#include <linux/netfilter/x_tables.h>
#include <linux/netfilter/xt_connbytes.h>
#include <net/netfilter/nf_conntrack.h>
-#include <net/netfilter/nf_conntrack_acct.h>

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Harald Welte <[email protected]>");
@@ -28,15 +27,12 @@ connbytes_mt(const struct sk_buff *skb, const struct net_device *in,
u_int64_t what = 0; /* initialize to make gcc happy */
u_int64_t bytes = 0;
u_int64_t pkts = 0;
- const struct nf_conn_counter *counters;
+ const struct ip_conntrack_counter *counters;

ct = nf_ct_get(skb, &ctinfo);
if (!ct)
return false;
-
- counters = nf_conn_acct_find(ct);
- if (!counters)
- return false;
+ counters = ct->counters;

switch (sinfo->what) {
case XT_CONNBYTES_PKTS:

2008-07-24 14:38:18

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, Jul 24, 2008 at 08:11:49AM -0500, Matt Mackall wrote:
>
> You don't honestly expect people to correctly code to such a standard,
> do you? People will assume that ksize never fails, they will be wrong,
> and computers will die.

If we follow this argument then most of our interfaces would be
broken.

> > You're taking
> > away an entire interface just because an underlying implementation
> > that's used by a very small proportion of users doesn't do the
> > right thing.
>
> Umm, no. There were very few users to being with, so it was actually a
> fairly large proportion. And that suggested the interface was a bad
> idea.

Huh? I'm talking about users of the kernel. A very small proportion
of those use SLOB.

Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

2008-07-24 15:23:22

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

Ingo Molnar wrote:
> Then i tried both suggested fix patches Patrick sent me (a suggested
> revert and an netfilter/RCU use-after-free fix), but none of them solved
> the crash.

Just to make sure - the "netfilter/RCU use-after-free fix" was the
patch from Pekka?

> Thus i finally arrived to:
>
> # good: [ae6134bd] hdlcdrv: Fix CRC calculation.
> # bad: [5547cd0d] netfilter: nf_conntrack_sctp: fix sparse warnings
> # bad: [280763c6] netfilter: xt_time: fix time's time_mt()'s use of
> # good: [07a7c10b] netlink: add NLA_PUT_BE64 macro
> # bad: [58401573] netfilter: accounting rework: ct_extend + 64bit co
>
> | 584015727a3b88b46602b20077b46cd04f8b4ab3 is first bad commit
> | commit 584015727a3b88b46602b20077b46cd04f8b4ab3
> | Author: Krzysztof Piotr Oledzki <[email protected]>
> | AuthorDate: Mon Jul 21 10:01:34 2008 -0700
> | Commit: David S. Miller <[email protected]>
> | CommitDate: Mon Jul 21 10:10:58 2008 -0700
> |
> | netfilter: accounting rework: ct_extend + 64bit counters (v4)
> [...]
> | Signed-off-by: Krzysztof Piotr Oledzki <[email protected]>
> | Signed-off-by: Patrick McHardy <[email protected]>
> | Signed-off-by: David S. Miller <[email protected]>
>
> Which i double-checked by reverting that commit from -git as well and
> that solved the crash. Find the tested reverter patch below.

Thats odd. I don't think anything is wrong with that patch
itself, its more likely that its triggering a bug in
ct_extend. You config has a few helper enabled (FTP, H.323,
TFTP) and the crash is when trying to call the helper functions.
Did you actually have traffic of one of these protocols?

2008-07-24 15:34:22

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

* Patrick McHardy <[email protected]> wrote:

> Ingo Molnar wrote:
>> Then i tried both suggested fix patches Patrick sent me (a suggested
>> revert and an netfilter/RCU use-after-free fix), but none of them
>> solved the crash.
>
> Just to make sure - the "netfilter/RCU use-after-free fix" was the
> patch from Pekka?

yes. You can see it in tip/out-of-tree:

http://people.redhat.com/mingo/tip.git/README

>> Thus i finally arrived to:
>>
>> # good: [ae6134bd] hdlcdrv: Fix CRC calculation.
>> # bad: [5547cd0d] netfilter: nf_conntrack_sctp: fix sparse warnings
>> # bad: [280763c6] netfilter: xt_time: fix time's time_mt()'s use of
>> # good: [07a7c10b] netlink: add NLA_PUT_BE64 macro
>> # bad: [58401573] netfilter: accounting rework: ct_extend + 64bit co
>>
>> | 584015727a3b88b46602b20077b46cd04f8b4ab3 is first bad commit
>> | commit 584015727a3b88b46602b20077b46cd04f8b4ab3
>> | Author: Krzysztof Piotr Oledzki <[email protected]>
>> | AuthorDate: Mon Jul 21 10:01:34 2008 -0700
>> | Commit: David S. Miller <[email protected]>
>> | CommitDate: Mon Jul 21 10:10:58 2008 -0700
>> |
>> | netfilter: accounting rework: ct_extend + 64bit counters (v4)
>> [...]
>> | Signed-off-by: Krzysztof Piotr Oledzki <[email protected]>
>> | Signed-off-by: Patrick McHardy <[email protected]>
>> | Signed-off-by: David S. Miller <[email protected]>
>>
>> Which i double-checked by reverting that commit from -git as well and
>> that solved the crash. Find the tested reverter patch below.
>
> Thats odd. I don't think anything is wrong with that patch itself, its
> more likely that its triggering a bug in ct_extend. You config has a
> few helper enabled (FTP, H.323, TFTP) and the crash is when trying to
> call the helper functions. Did you actually have traffic of one of
> these protocols?

no, that's not likely - it's a default distro bootup.

Ingo

2008-07-24 15:35:24

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

Ingo Molnar wrote:
> * Patrick McHardy <[email protected]> wrote:
>
>>> | netfilter: accounting rework: ct_extend + 64bit counters (v4)
>>> [...]
>>> | Signed-off-by: Krzysztof Piotr Oledzki <[email protected]>
>>> | Signed-off-by: Patrick McHardy <[email protected]>
>>> | Signed-off-by: David S. Miller <[email protected]>
>>>
>>> Which i double-checked by reverting that commit from -git as well and
>>> that solved the crash. Find the tested reverter patch below.
>> Thats odd. I don't think anything is wrong with that patch itself, its
>> more likely that its triggering a bug in ct_extend. You config has a
>> few helper enabled (FTP, H.323, TFTP) and the crash is when trying to
>> call the helper functions. Did you actually have traffic of one of
>> these protocols?
>
> no, that's not likely - it's a default distro bootup.

OK thanks. I'll look into it.

2008-07-24 17:52:00

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, 2008-07-24 at 22:37 +0800, Herbert Xu wrote:
> On Thu, Jul 24, 2008 at 08:11:49AM -0500, Matt Mackall wrote:
> >
> > You don't honestly expect people to correctly code to such a standard,
> > do you? People will assume that ksize never fails, they will be wrong,
> > and computers will die.
>
> If we follow this argument then most of our interfaces would be
> broken.

Let's try this again: did you know that ksize could fail depending on
kernel configuration? Most of us would answer no. That suggests the API
is bad. This ranks 12 on Rusty's spectrum of user-friendly APIs:

http://ozlabs.org/~rusty/ols-2003-keynote/img51.html

> > > You're taking
> > > away an entire interface just because an underlying implementation
> > > that's used by a very small proportion of users doesn't do the
> > > right thing.
> >
> > Umm, no. There were very few users to being with, so it was actually a
> > fairly large proportion. And that suggested the interface was a bad
> > idea.
>
> Huh? I'm talking about users of the kernel. A very small proportion
> of those use SLOB.

Ahh. I see what you're saying. You may be right about that, or you may
be wrong: I don't know which of the millions of mass market embedded
Linux devices out there are using it.

And of course I argue that SLOB did do the right thing, which was only
allowing ksize on kmalloced objects. It's an accident of implementation
that kmalloc and kmem_cache_alloc use the same underlying allocator. It
has not been true at all points in the past, it's not true for some
users in the present, and it may not be true for most users in the
future. Thus, it's a bad idea to try to use ksize on something that
wasn't kmalloced.

If you have an argument for reintroducing ksize based on an actual use
case, let's please move on to (or back to) it so that we can have some
substance to this discussion.

--
Mathematics is the supreme nostalgia of our time.

2008-07-24 18:09:53

by Krzysztof Olędzki

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, 24 Jul 2008, Ingo Molnar wrote:

>
> * Patrick McHardy <[email protected]> wrote:
>
>> Ingo Molnar wrote:
>>> Then i tried both suggested fix patches Patrick sent me (a suggested
>>> revert and an netfilter/RCU use-after-free fix), but none of them
>>> solved the crash.
>>
>> Just to make sure - the "netfilter/RCU use-after-free fix" was the
>> patch from Pekka?
>
> yes. You can see it in tip/out-of-tree:
>
> http://people.redhat.com/mingo/tip.git/README
>
>>> Thus i finally arrived to:
>>>
>>> # good: [ae6134bd] hdlcdrv: Fix CRC calculation.
>>> # bad: [5547cd0d] netfilter: nf_conntrack_sctp: fix sparse warnings
>>> # bad: [280763c6] netfilter: xt_time: fix time's time_mt()'s use of
>>> # good: [07a7c10b] netlink: add NLA_PUT_BE64 macro
>>> # bad: [58401573] netfilter: accounting rework: ct_extend + 64bit co
>>>
>>> | 584015727a3b88b46602b20077b46cd04f8b4ab3 is first bad commit
>>> | commit 584015727a3b88b46602b20077b46cd04f8b4ab3
>>> | Author: Krzysztof Piotr Oledzki <[email protected]>
>>> | AuthorDate: Mon Jul 21 10:01:34 2008 -0700
>>> | Commit: David S. Miller <[email protected]>
>>> | CommitDate: Mon Jul 21 10:10:58 2008 -0700
>>> |
>>> | netfilter: accounting rework: ct_extend + 64bit counters (v4)
>>> [...]
>>> | Signed-off-by: Krzysztof Piotr Oledzki <[email protected]>
>>> | Signed-off-by: Patrick McHardy <[email protected]>
>>> | Signed-off-by: David S. Miller <[email protected]>
>>>
>>> Which i double-checked by reverting that commit from -git as well and
>>> that solved the crash. Find the tested reverter patch below.
>>
>> Thats odd. I don't think anything is wrong with that patch itself, its
>> more likely that its triggering a bug in ct_extend. You config has a
>> few helper enabled (FTP, H.323, TFTP) and the crash is when trying to
>> call the helper functions. Did you actually have traffic of one of
>> these protocols?
>
> no, that's not likely - it's a default distro bootup.

The commit makes ct_extend area to be used *very* frequently. Could you
try to boot your kernel with nf_conntrack.acct=0 to disable accounting?
Does it help?

Best regards,

Krzysztof Ol?dzki

2008-07-24 18:54:19

by Andrew Morton

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, 24 Jul 2008 16:34:36 +0300 Pekka Enberg <[email protected]> wrote:

> > >> Your patch introduced a use-after-free and double-free.
> > >> krealloc() frees the old pointer, but it is still used
> > >> for the ->move operations, then freed again.
> > >>
> > >> To fix this I think we need a __krealloc() that doesn't
> > >> free the old memory, especially since it must not be
> > >> freed immediately because it may still be used in a RCU
> > >> read side (see the last part in the patch attached to
> > >> this mail (based on a kernel without your patch)).
> > >
> > > Agreed. Something like this, perhaps?
> > >
> > > [PATCH] netfilter: fix double-free and use-after free
> > >
> > > As suggested by Patrick McHardy, introduce a __krealloc() that doesn't
> > > free the original buffer to fix a double-free and use-after-free bug
> > > introduced by me in netfilter that uses RCU.
> > >
> > > Reported-by: Patrick McHardy <[email protected]>
> > > Signed-off-by: Pekka Enberg <[email protected]>
> >
> > Looks good to me, thanks.
>
> Ingo, can you please test this? Andrew, I'm at OLS so can you pick up
> the patch in your tree?

Sure. Or Patrick can do so and it can be merged via the net tree.

Ingo, did this patch actually fix something over there?

2008-07-24 18:56:06

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, 2008-07-24 at 11:51 -0700, Andrew Morton wrote:
> On Thu, 24 Jul 2008 16:34:36 +0300 Pekka Enberg <[email protected]> wrote:
>
> > > >> Your patch introduced a use-after-free and double-free.
> > > >> krealloc() frees the old pointer, but it is still used
> > > >> for the ->move operations, then freed again.
> > > >>
> > > >> To fix this I think we need a __krealloc() that doesn't
> > > >> free the old memory, especially since it must not be
> > > >> freed immediately because it may still be used in a RCU
> > > >> read side (see the last part in the patch attached to
> > > >> this mail (based on a kernel without your patch)).
> > > >
> > > > Agreed. Something like this, perhaps?
> > > >
> > > > [PATCH] netfilter: fix double-free and use-after free
> > > >
> > > > As suggested by Patrick McHardy, introduce a __krealloc() that doesn't
> > > > free the original buffer to fix a double-free and use-after-free bug
> > > > introduced by me in netfilter that uses RCU.
> > > >
> > > > Reported-by: Patrick McHardy <[email protected]>
> > > > Signed-off-by: Pekka Enberg <[email protected]>
> > >
> > > Looks good to me, thanks.
> >
> > Ingo, can you please test this? Andrew, I'm at OLS so can you pick up
> > the patch in your tree?
>
> Sure. Or Patrick can do so and it can be merged via the net tree.
>
> Ingo, did this patch actually fix something over there?

Apparently it didn't but it did fix Dieter's problem:

http://lkml.org/lkml/2008/7/24/337

Dieter, can we add a Tested-by tag from you to this patch?

2008-07-24 19:26:28

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, 2008-07-24 at 16:32 +0300, Pekka Enberg wrote:
> On Thu, Jul 24, 2008 at 10:56:08PM +1000, Nick Piggin wrote:
> > > > OTOH, skb allocation uses kmalloc don't they? So you could still use
> > > > SLOB ksize for that I guess.
>
> On Thursday 24 July 2008 23:04, Herbert Xu wrote:
> > > Yes I was referring to the data portion which is kmalloc'ed.
> > > That is also why I'm interested in ksize because a priori we
> > > don't know exactly how big it's going to be. However, we do
> > > know that statistically 1500 will dominate.
> > >
> > > I'm not interested in ksize for kmem_cache at all. So in fact
> > > we could have something simpler that's based on kmalloc's rounding
> > > algorithm instead.
>
> On Thu, 2008-07-24 at 23:13 +1000, Nick Piggin wrote:
> > Yes you could definitely have a function that returns allocated
> > bytes for a given kmalloc size. Should be about as fast or faster
> > than extracting the size from the kaddr...
>
> Yup, makes sense.

On the other hand, I can imagine useful allocator changes where this
would not be a constant of requested size. For instance, imagine we had
a classless bucket allocator, but with a heuristic to try a larger
bucket when it wasn't cheap/possible to allocate a right-sized object
(because of memory pressure, etc.) and larger ones were available.
This sort of thing is a pretty small change for SLAB/SLUB.

--
Mathematics is the supreme nostalgia of our time.

2008-07-24 19:37:18

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

* Andrew Morton <[email protected]> wrote:

> On Thu, 24 Jul 2008 16:34:36 +0300 Pekka Enberg <[email protected]> wrote:
>
> > > >> Your patch introduced a use-after-free and double-free.
> > > >> krealloc() frees the old pointer, but it is still used
> > > >> for the ->move operations, then freed again.
> > > >>
> > > >> To fix this I think we need a __krealloc() that doesn't
> > > >> free the old memory, especially since it must not be
> > > >> freed immediately because it may still be used in a RCU
> > > >> read side (see the last part in the patch attached to
> > > >> this mail (based on a kernel without your patch)).
> > > >
> > > > Agreed. Something like this, perhaps?
> > > >
> > > > [PATCH] netfilter: fix double-free and use-after free
> > > >
> > > > As suggested by Patrick McHardy, introduce a __krealloc() that doesn't
> > > > free the original buffer to fix a double-free and use-after-free bug
> > > > introduced by me in netfilter that uses RCU.
> > > >
> > > > Reported-by: Patrick McHardy <[email protected]>
> > > > Signed-off-by: Pekka Enberg <[email protected]>
> > >
> > > Looks good to me, thanks.
> >
> > Ingo, can you please test this? Andrew, I'm at OLS so can you pick up
> > the patch in your tree?
>
> Sure. Or Patrick can do so and it can be merged via the net tree.
>
> Ingo, did this patch actually fix something over there?

not that i know of. I'll do some re-tests tomorrow to make sure.

Ingo

2008-07-24 20:58:23

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

From: Pekka Enberg <[email protected]>
Date: Thu, 24 Jul 2008 21:55:44 +0300

> On Thu, 2008-07-24 at 11:51 -0700, Andrew Morton wrote:
> > Sure. Or Patrick can do so and it can be merged via the net tree.
> >
> > Ingo, did this patch actually fix something over there?
>
> Apparently it didn't but it did fix Dieter's problem:
>
> http://lkml.org/lkml/2008/7/24/337
>
> Dieter, can we add a Tested-by tag from you to this patch?

Patrick will toss it my way, and it'll get integrated
via net-2.6 as a result.

2008-07-24 21:20:00

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, 24 Jul 2008, Patrick McHardy wrote:
>
> To fix this I think we need a __krealloc() that doesn't
> free the old memory, especially since it must not be
> freed immediately because it may still be used in a RCU
> read side (see the last part in the patch attached to
> this mail (based on a kernel without your patch)).

Hmm. Don't you need to fix some of the ordering of the initialization too?

If there are possible readers that happen in parallel with changing this
thing, don't you need to protect the update of "ext->len" against the
actual changes? And the readers should probably have a read barrier
between checking "len" and actually looking at the values? Finally, why do
the "ct->ext" dereference thing, when we know it has to be equal to "new"?

ie something like this on the writing side (in _addition_ to both the
patches already seen), but I didn't do the reading side (ie there are no
"smp_rmb()"'s on the reading side)

And no, I don't know the code, so I don't know who/what can read those
things with RCU, so maybe there is some reason why the actual data doesn't
need protecting. But I somehow doubt it.

Linus

---
net/netfilter/nf_conntrack_extend.c | 9 +++++----
1 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/net/netfilter/nf_conntrack_extend.c b/net/netfilter/nf_conntrack_extend.c
index 3469bc7..135e095 100644
--- a/net/netfilter/nf_conntrack_extend.c
+++ b/net/netfilter/nf_conntrack_extend.c
@@ -115,10 +115,11 @@ void *__nf_ct_ext_add(struct nf_conn *ct, enum nf_ct_ext_id id, gfp_t gfp)
ct->ext = new;
}

- ct->ext->offset[id] = newoff;
- ct->ext->len = newlen;
- memset((void *)ct->ext + newoff, 0, newlen - newoff);
- return (void *)ct->ext + newoff;
+ new->offset[id] = newoff;
+ memset((void *)new + newoff, 0, newlen - newoff);
+ smp_wmb();
+ new->len = newlen;
+ return (void *)new + newoff;
}
EXPORT_SYMBOL(__nf_ct_ext_add);

2008-07-24 22:09:32

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

From: Linus Torvalds <[email protected]>
Date: Thu, 24 Jul 2008 14:13:42 -0700 (PDT)

> Finally, why do the "ct->ext" dereference thing, when we know it has
> to be equal to "new"?

Actually in the old code this precondition didn't hold, which explains
how it is.

The old code looked like:

if (newlen >= ksize(ct->ext)) {
new = kmalloc(newlen, gfp);
if (!new)
return NULL;
...
ct->ext = new;
}

ct->ext->offset[id] = newoff;
ct->ext->len = newlen;
memset((void *)ct->ext + newoff, 0, newlen - newoff);
return (void *)ct->ext + newoff;

and in that context 'new' is only assigned in the "newlen >=" guarded
code block.

Anyways, it does seem that we should indeed only update the
new larger length only after we've initialized the contents.

Note that we could make krealloc() and friends clear out the trailing
bits of the new buffer, and therefore the caller wouldn't even need to
be mindful of such things.

I don't know if that's desirable in general, probably it isn't.

2008-07-25 01:41:11

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Thu, Jul 24, 2008 at 12:47:19PM -0500, Matt Mackall wrote:
>
> Let's try this again: did you know that ksize could fail depending on
> kernel configuration? Most of us would answer no. That suggests the API
> is bad. This ranks 12 on Rusty's spectrum of user-friendly APIs:

I think you misunderstood my argument. I never suggested changing
the existing ksize interface to return an error onto unsuspecting
users. I suggested creating a new interface that is explicitly
designed to return an error if the underlying implementation
is unable to support this.

Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

2008-07-25 03:03:42

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Fri, 2008-07-25 at 09:39 +0800, Herbert Xu wrote:
> On Thu, Jul 24, 2008 at 12:47:19PM -0500, Matt Mackall wrote:
> >
> > Let's try this again: did you know that ksize could fail depending on
> > kernel configuration? Most of us would answer no. That suggests the API
> > is bad. This ranks 12 on Rusty's spectrum of user-friendly APIs:
>
> I think you misunderstood my argument. I never suggested changing
> the existing ksize interface to return an error onto unsuspecting
> users. I suggested creating a new interface that is explicitly
> designed to return an error if the underlying implementation
> is unable to support this.

I think that could probably be made to work. Perhaps something like:

size_t kmalloc_extra(void *); /* how many extra bytes in this kmalloc?
*/

Which, if it didn't work, could return a nice safe 0. We could argue
about signedness a bit, but I think this would always be safe.

This will also work with all our current kmalloc implementations. The
trouble was calling ksize() on kmem_cache_alloc objects, which happens
to work with SLAB and SLOB.

--
Mathematics is the supreme nostalgia of our time.

2008-07-25 06:20:59

[permalink] [raw]

Subject: [lockdep warning] AOE / networking: aoenet_xmit: noop_qdisc.q.lock, INFO: inconsistent lock state at 0000000000000370

not sure whether it got reported already, but -tip testing triggered
this new ATA over Ethernet lockdep warning on latest -git:

=================================
[ INFO: inconsistent lock state ]
2.6.26-tip-00346-gb4d3941-dirty #13925
---------------------------------
inconsistent {softirq-on-W} -> {in-softirq-W} usage.
swapper/0 [HC0[0]:SC1[2]:HE1:SE0] takes:
(noop_qdisc.q.lock){-+..}, at: [<ffffffff80880ba8>] dev_queue_xmit+0xd8/0x520
{softirq-on-W} state was registered at:
[<ffffffff80266517>] __lock_acquire+0x3a7/0x10e0
[<ffffffff802672a7>] lock_acquire+0x57/0x80
[<ffffffff80a080da>] _spin_lock+0x2a/0x40
[<ffffffff808a141f>] shutdown_scheduler_queue+0x3f/0x60
[<ffffffff808a1481>] dev_shutdown+0x41/0x90
[<ffffffff8087e9da>] rollback_registered+0x6a/0x100
[<ffffffff8087ea92>] unregister_netdevice+0x22/0x80
[<ffffffff80999bcc>] ieee80211_register_hw+0x2ec/0x3b0
[<ffffffff80eb0dbc>] init_mac80211_hwsim+0x1bc/0x370
[<ffffffff80209045>] do_one_initcall+0x45/0x180
[<ffffffff80e7fb75>] kernel_init+0x1d5/0x2f0
[<ffffffff8020d599>] child_rip+0xa/0x11
[<ffffffffffffffff>] 0xffffffffffffffff
irq event stamp: 326488
hardirqs last enabled at (326488): [<ffffffff80265cbd>] trace_hardirqs_on+0xd/0x10
hardirqs last disabled at (326487): [<ffffffff802642ed>] trace_hardirqs_off+0xd/0x10
softirqs last enabled at (326450): [<ffffffff80248462>] __do_softirq+0x102/0x120
softirqs last disabled at (326461): [<ffffffff8020d86c>] call_softirq+0x1c/0x30

other info that might help us debug this:
1 lock held by swapper/0:
#0: (rcu_read_lock){..--}, at: [<ffffffff80880b2b>] dev_queue_xmit+0x5b/0x520

stack backtrace:
Pid: 0, comm: swapper Not tainted 2.6.26-tip-00346-gb4d3941-dirty #13925

Call Trace:
<IRQ> [<ffffffff80264c6b>] print_usage_bug+0x18b/0x190
[<ffffffff80265a67>] mark_lock+0x517/0x570
[<ffffffff802664ce>] __lock_acquire+0x35e/0x10e0
[<ffffffff80265c5b>] ? trace_hardirqs_on_caller+0x7b/0xd0
[<ffffffff802672a7>] lock_acquire+0x57/0x80
[<ffffffff80880ba8>] ? dev_queue_xmit+0xd8/0x520
[<ffffffff80a080da>] _spin_lock+0x2a/0x40
[<ffffffff80880b2b>] ? dev_queue_xmit+0x5b/0x520
[<ffffffff80880ba8>] dev_queue_xmit+0xd8/0x520
[<ffffffff807aba2f>] aoenet_xmit+0x2f/0x40
[<ffffffff807a9cd0>] aoecmd_cfg+0x140/0x170
[<ffffffff807ab890>] ? discover_timer+0x0/0x110
[<ffffffff807ab92a>] discover_timer+0x9a/0x110
[<ffffffff8024bf3b>] run_timer_softirq+0x20b/0x250
[<ffffffff802483f4>] __do_softirq+0x94/0x120
[<ffffffff80260e6d>] ? clockevents_program_event+0x5d/0x90
[<ffffffff8020d86c>] call_softirq+0x1c/0x30
[<ffffffff8021018a>] do_softirq+0x6a/0xa0
[<ffffffff80248349>] irq_exit+0x99/0xb0
[<ffffffff80220842>] smp_apic_timer_interrupt+0x92/0xd0
[<ffffffff8020d34b>] apic_timer_interrupt+0x6b/0x70
<EOI> [<ffffffff80214afc>] ? mwait_idle+0x4c/0x60
[<ffffffff80214af3>] ? mwait_idle+0x43/0x60
[<ffffffff8020af51>] ? cpu_idle+0xa1/0x110
[<ffffffff80a00990>] ? start_secondary+0x150/0x1a0

with this config:

http://redhat.com/~mingo/misc/log-Fri_Jul_25_06_07_52_CEST_2008.bad

full bootlog:

http://redhat.com/~mingo/misc/config-Fri_Jul_25_06_07_52_CEST_2008.bad

i suspect this too is about the changed locking dependencies.

Ingo

2008-07-25 06:25:33

[permalink] [raw]

Subject: Re: [lockdep warning] AOE / networking: aoenet_xmit: noop_qdisc.q.lock, INFO: inconsistent lock state at 0000000000000370

From: Ingo Molnar <[email protected]>
Date: Fri, 25 Jul 2008 08:20:09 +0200

> not sure whether it got reported already, but -tip testing triggered
> this new ATA over Ethernet lockdep warning on latest -git:

Thanks for the report, does this fix it?

diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index 4ac7e3a..43abd4d 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -736,9 +736,9 @@ static void shutdown_scheduler_queue(struct net_device *dev,
dev_queue->qdisc = qdisc_default;
dev_queue->qdisc_sleeping = qdisc_default;

- spin_lock(root_lock);
+ spin_lock_bh(root_lock);
qdisc_destroy(qdisc);
- spin_unlock(root_lock);
+ spin_unlock_bh(root_lock);
}
}

2008-07-25 07:26:55

[permalink] [raw]

Subject: Re: [lockdep warning] AOE / networking: aoenet_xmit: noop_qdisc.q.lock, INFO: inconsistent lock state at 0000000000000370

* David Miller <[email protected]> wrote:

> From: Ingo Molnar <[email protected]>
> Date: Fri, 25 Jul 2008 08:20:09 +0200
>
> > not sure whether it got reported already, but -tip testing triggered
> > this new ATA over Ethernet lockdep warning on latest -git:
>
> Thanks for the report, does this fix it?

i've tested it and the warning went away - thanks David!

Tested-by: Ingo Molnar <[email protected]>

Ingo

2008-07-25 08:02:29

by Dieter Ries

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

Pekka Enberg schrieb:
> On Thu, 2008-07-24 at 11:51 -0700, Andrew Morton wrote:
>> On Thu, 24 Jul 2008 16:34:36 +0300 Pekka Enberg <[email protected]> wrote:
>>
>>>>>> Your patch introduced a use-after-free and double-free.
>>>>>> krealloc() frees the old pointer, but it is still used
>>>>>> for the ->move operations, then freed again.
>>>>>>
>>>>>> To fix this I think we need a __krealloc() that doesn't
>>>>>> free the old memory, especially since it must not be
>>>>>> freed immediately because it may still be used in a RCU
>>>>>> read side (see the last part in the patch attached to
>>>>>> this mail (based on a kernel without your patch)).
>>>>> Agreed. Something like this, perhaps?
>>>>>
>>>>> [PATCH] netfilter: fix double-free and use-after free
>>>>>
>>>>> As suggested by Patrick McHardy, introduce a __krealloc() that doesn't
>>>>> free the original buffer to fix a double-free and use-after-free bug
>>>>> introduced by me in netfilter that uses RCU.
>>>>>
>>>>> Reported-by: Patrick McHardy <[email protected]>
>>>>> Signed-off-by: Pekka Enberg <[email protected]>
>>>> Looks good to me, thanks.
>>> Ingo, can you please test this? Andrew, I'm at OLS so can you pick up
>>> the patch in your tree?
>> Sure. Or Patrick can do so and it can be merged via the net tree.
>>
>> Ingo, did this patch actually fix something over there?
>
> Apparently it didn't but it did fix Dieter's problem:
>
> http://lkml.org/lkml/2008/7/24/337
>
> Dieter, can we add a Tested-by tag from you to this patch?
>

Yes, it definitely fixed my issue and I have not encountered
further problems with the patch. The machine
is running fine with it.

Do I have to explicitly add my
Tested-by: Dieter Ries <[email protected]>
tag somewhere (if yes, where?) or is this enough for you?

cu
Dieter

2008-07-25 08:23:41

[permalink] [raw]

Subject: Re: [lockdep warning] AOE / networking: aoenet_xmit: noop_qdisc.q.lock, INFO: inconsistent lock state at 0000000000000370

From: Ingo Molnar <[email protected]>
Date: Fri, 25 Jul 2008 09:26:24 +0200

>
> * David Miller <[email protected]> wrote:
>
> > From: Ingo Molnar <[email protected]>
> > Date: Fri, 25 Jul 2008 08:20:09 +0200
> >
> > > not sure whether it got reported already, but -tip testing triggered
> > > this new ATA over Ethernet lockdep warning on latest -git:
> >
> > Thanks for the report, does this fix it?
>
> i've tested it and the warning went away - thanks David!
>
> Tested-by: Ingo Molnar <[email protected]>

Thanks for testing.

2008-07-25 09:09:46

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Friday 25 July 2008 05:21, Matt Mackall wrote:
> On Thu, 2008-07-24 at 16:32 +0300, Pekka Enberg wrote:
> > On Thu, Jul 24, 2008 at 10:56:08PM +1000, Nick Piggin wrote:
> > > > > OTOH, skb allocation uses kmalloc don't they? So you could still
> > > > > use SLOB ksize for that I guess.
> >
> > On Thursday 24 July 2008 23:04, Herbert Xu wrote:
> > > > Yes I was referring to the data portion which is kmalloc'ed.
> > > > That is also why I'm interested in ksize because a priori we
> > > > don't know exactly how big it's going to be. However, we do
> > > > know that statistically 1500 will dominate.
> > > >
> > > > I'm not interested in ksize for kmem_cache at all. So in fact
> > > > we could have something simpler that's based on kmalloc's rounding
> > > > algorithm instead.
> >
> > On Thu, 2008-07-24 at 23:13 +1000, Nick Piggin wrote:
> > > Yes you could definitely have a function that returns allocated
> > > bytes for a given kmalloc size. Should be about as fast or faster
> > > than extracting the size from the kaddr...
> >
> > Yup, makes sense.
>
> On the other hand, I can imagine useful allocator changes where this
> would not be a constant of requested size. For instance, imagine we had
> a classless bucket allocator, but with a heuristic to try a larger
> bucket when it wasn't cheap/possible to allocate a right-sized object
> (because of memory pressure, etc.) and larger ones were available.
> This sort of thing is a pretty small change for SLAB/SLUB.

If that comes up, you could make the API take the pointer as well,
then just filter out the new argument in allocators that don't use
it.

2008-07-25 10:42:20

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

On Fri, 2008-07-25 at 10:02 +0200, Dieter Ries wrote:
> Pekka Enberg schrieb:
> > On Thu, 2008-07-24 at 11:51 -0700, Andrew Morton wrote:
> >> On Thu, 24 Jul 2008 16:34:36 +0300 Pekka Enberg <[email protected]> wrote:
> >>
> >>>>>> Your patch introduced a use-after-free and double-free.
> >>>>>> krealloc() frees the old pointer, but it is still used
> >>>>>> for the ->move operations, then freed again.
> >>>>>>
> >>>>>> To fix this I think we need a __krealloc() that doesn't
> >>>>>> free the old memory, especially since it must not be
> >>>>>> freed immediately because it may still be used in a RCU
> >>>>>> read side (see the last part in the patch attached to
> >>>>>> this mail (based on a kernel without your patch)).
> >>>>> Agreed. Something like this, perhaps?
> >>>>>
> >>>>> [PATCH] netfilter: fix double-free and use-after free
> >>>>>
> >>>>> As suggested by Patrick McHardy, introduce a __krealloc() that doesn't
> >>>>> free the original buffer to fix a double-free and use-after-free bug
> >>>>> introduced by me in netfilter that uses RCU.
> >>>>>
> >>>>> Reported-by: Patrick McHardy <[email protected]>
> >>>>> Signed-off-by: Pekka Enberg <[email protected]>
> >>>> Looks good to me, thanks.
> >>> Ingo, can you please test this? Andrew, I'm at OLS so can you pick up
> >>> the patch in your tree?
> >> Sure. Or Patrick can do so and it can be merged via the net tree.
> >>
> >> Ingo, did this patch actually fix something over there?
> >
> > Apparently it didn't but it did fix Dieter's problem:
> >
> > http://lkml.org/lkml/2008/7/24/337
> >
> > Dieter, can we add a Tested-by tag from you to this patch?
> >
>
> Yes, it definitely fixed my issue and I have not encountered
> further problems with the patch. The machine
> is running fine with it.
>
> Do I have to explicitly add my
> Tested-by: Dieter Ries <[email protected]>
> tag somewhere (if yes, where?) or is this enough for you?

No, that's fine. The people who are merging the patch can pick it up.
Thanks a lot for your help!

Pekka

2008-07-25 17:52:32

by Ingo Oeser

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

Hi David,

David Miller schrieb:
> From: Peter Zijlstra <[email protected]>
> Date: Wed, 23 Jul 2008 12:58:16 +0200
>
> > So I guess my question is, is netif_tx_lock() here to stay, or is the
> > right fix to convert all those drivers to use __netif_tx_lock() which
> > locks only a single queue?
>
> It's staying.
>
> It's trying to block all potential calls into the ->hard_start_xmit()
> method of the driver, and the only reliable way to do that is to take
> all the TX queue locks. And in one form or another, we're going to
> have this "grab/release all the TX queue locks" construct.
>
> I find it interesting that this cannot be simply described to lockdep
> :-)

I'm sure as hell, I miss sth. but can't it be done by this pseudo-code:

netif_tx_lock(device)
{
mutex_lock(device->queue_entry_mutex);
foreach_queue_entries(queue, device->queues)
{
spin_lock(queue->tx_lock);
set_noop_tx_handler(queue);
spin_unlock(queue->tx_lock);
}
mutex_unlock(device->queue_entry_mutex);
}

netif_tx_unlock(device)
{
mutex_lock(device->queue_entry_mutex);
foreach_queue_entries(queue, device->queues)
{
spin_lock(queue->tx_lock);
set_useful_tx_handler(queue);
spin_unlock(queue->tx_lock);
}
mutex_unlock(device->queue_entry_mutex);
}

Then protect use of the queues by queue->tx_lock in transmit path.
The first setup of the queue doesn't need to be protected, since no-one
knows the device. The final cleanup of the device doesn't need to be
protected either, because netif_tx_lock() and netif_tx_unlock() should
not be called after entering the final cleanup.

Some VM locking works this way...

Best Regards

Ingo Oeser

2008-07-25 18:36:06

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Fri, Jul 25, 2008 at 07:04:36PM +0200, Ingo Oeser wrote:
...
> I'm sure as hell, I miss sth. but can't it be done by this pseudo-code:

...And I really doubt it can't be done like this.

Jarek P.

>
> netif_tx_lock(device)
> {
> mutex_lock(device->queue_entry_mutex);
> foreach_queue_entries(queue, device->queues)
> {
> spin_lock(queue->tx_lock);
> set_noop_tx_handler(queue);
> spin_unlock(queue->tx_lock);
> }
> mutex_unlock(device->queue_entry_mutex);
> }
>
> netif_tx_unlock(device)
> {
> mutex_lock(device->queue_entry_mutex);
> foreach_queue_entries(queue, device->queues)
> {
> spin_lock(queue->tx_lock);
> set_useful_tx_handler(queue);
> spin_unlock(queue->tx_lock);
> }
> mutex_unlock(device->queue_entry_mutex);
> }
>
> Then protect use of the queues by queue->tx_lock in transmit path.
> The first setup of the queue doesn't need to be protected, since no-one
> knows the device. The final cleanup of the device doesn't need to be
> protected either, because netif_tx_lock() and netif_tx_unlock() should
> not be called after entering the final cleanup.
>
> Some VM locking works this way...
>
>
> Best Regards
>
> Ingo Oeser

2008-07-25 19:17:25

by Johannes Berg

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Fri, 2008-07-25 at 20:36 +0200, Jarek Poplawski wrote:
> On Fri, Jul 25, 2008 at 07:04:36PM +0200, Ingo Oeser wrote:
> ...
> > I'm sure as hell, I miss sth. but can't it be done by this pseudo-code:
>
> ...And I really doubt it can't be done like this.

Umm, of course it cannot, because then we'd have to take the mutex in
the TX path, which we cannot. We cannot have another lock in the TX
path, what's so hard to understand about? We need to be able to lock all
queues to lock out multiple tx paths at once in some (really) slow paths
but not have any extra lock overhead for the tx path, especially not a
single lock.

johannes

Attachments:

signature.asc (836.00 B)
This is a digitally signed message part

2008-07-25 19:33:34

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Fri, Jul 25, 2008 at 09:16:24PM +0200, Johannes Berg wrote:
> On Fri, 2008-07-25 at 20:36 +0200, Jarek Poplawski wrote:
> > On Fri, Jul 25, 2008 at 07:04:36PM +0200, Ingo Oeser wrote:
> > ...
> > > I'm sure as hell, I miss sth. but can't it be done by this pseudo-code:
> >
> > ...And I really doubt it can't be done like this.
>
> Umm, of course it cannot, because then we'd have to take the mutex in
> the TX path, which we cannot. We cannot have another lock in the TX
> path, what's so hard to understand about? We need to be able to lock all
> queues to lock out multiple tx paths at once in some (really) slow paths
> but not have any extra lock overhead for the tx path, especially not a
> single lock.

But this mutex doesn't have to be mutex. And it's not for the tx path,
only for "service" just like netif_tx_lock(). The fast path needs only
queue->tx_lock.

Jarek P.

2008-07-25 19:37:05

by Johannes Berg

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Fri, 2008-07-25 at 21:34 +0200, Jarek Poplawski wrote:

> > Umm, of course it cannot, because then we'd have to take the mutex in
> > the TX path, which we cannot. We cannot have another lock in the TX
> > path, what's so hard to understand about? We need to be able to lock all
> > queues to lock out multiple tx paths at once in some (really) slow paths
> > but not have any extra lock overhead for the tx path, especially not a
> > single lock.
>
> But this mutex doesn't have to be mutex. And it's not for the tx path,
> only for "service" just like netif_tx_lock(). The fast path needs only
> queue->tx_lock.

No, we need to be able to lock out multiple TX paths at once.

johannes

Attachments:

signature.asc (836.00 B)
This is a digitally signed message part

2008-07-25 20:00:43

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Fri, Jul 25, 2008 at 09:36:15PM +0200, Johannes Berg wrote:
> On Fri, 2008-07-25 at 21:34 +0200, Jarek Poplawski wrote:
>
> > > Umm, of course it cannot, because then we'd have to take the mutex in
> > > the TX path, which we cannot. We cannot have another lock in the TX
> > > path, what's so hard to understand about? We need to be able to lock all
> > > queues to lock out multiple tx paths at once in some (really) slow paths
> > > but not have any extra lock overhead for the tx path, especially not a
> > > single lock.
> >
> > But this mutex doesn't have to be mutex. And it's not for the tx path,
> > only for "service" just like netif_tx_lock(). The fast path needs only
> > queue->tx_lock.
>
> No, we need to be able to lock out multiple TX paths at once.

IMHO, it can do the same. We could e.g. insert a locked spinlock into
this noop_tx_handler(), to give everyone some waiting.

Jarek P.

2008-07-26 09:19:00

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

From: Jarek Poplawski <[email protected]>
Date: Fri, 25 Jul 2008 22:01:37 +0200

> On Fri, Jul 25, 2008 at 09:36:15PM +0200, Johannes Berg wrote:
> > On Fri, 2008-07-25 at 21:34 +0200, Jarek Poplawski wrote:
> >
> > No, we need to be able to lock out multiple TX paths at once.
>
> IMHO, it can do the same. We could e.g. insert a locked spinlock into
> this noop_tx_handler(), to give everyone some waiting.

I think there might be an easier way, but we may have
to modify the state bits a little.

Every call into ->hard_start_xmit() is made like this:

1. lock TX queue
2. check TX queue stopped
3. call ->hard_start_xmit() if not stopped

This means that we can in fact do something like:

unsigned int i;

for (i = 0; i < dev->num_tx_queues; i++) {
struct netdev_queue *txq;

txq = netdev_get_tx_queue(dev, i);
spin_lock_bh(&txq->_xmit_lock);
netif_tx_freeze_queue(txq);
spin_unlock_bh(&txq->_xmit_lock);
}

netif_tx_freeze_queue() just sets a new bit we add.

Then we go to the ->hard_start_xmit() call sites and check this new
"frozen" bit as well as the existing "stopped" bit.

When we unfreeze each queue later, we see if it is stopped, and if not
we schedule it's qdisc for packet processing.

A patch below shows how the guarding would work. It doesn't implement
the actual freeze/unfreeze.

We need to use a side-state bit to do this because we don't
want this operation to get all mixed up with the queue waking
operations that the driver TX reclaim code will be doing
asynchronously.

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index b4d056c..cba98fb 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -440,6 +440,7 @@ static inline void napi_synchronize(const struct napi_struct *n)
enum netdev_queue_state_t
{
__QUEUE_STATE_XOFF,
+ __QUEUE_STATE_FROZEN,
};

struct netdev_queue {
@@ -1099,6 +1100,11 @@ static inline int netif_queue_stopped(const struct net_device *dev)
return netif_tx_queue_stopped(netdev_get_tx_queue(dev, 0));
}

+static inline int netif_tx_queue_frozen(const struct netdev_queue *dev_queue)
+{
+ return test_bit(__QUEUE_STATE_FROZEN, &dev_queue->state);
+}
+
/**
* netif_running - test if up
* @dev: network device
diff --git a/net/core/netpoll.c b/net/core/netpoll.c
index c127208..6c7af39 100644
--- a/net/core/netpoll.c
+++ b/net/core/netpoll.c
@@ -70,6 +70,7 @@ static void queue_process(struct work_struct *work)
local_irq_save(flags);
__netif_tx_lock(txq, smp_processor_id());
if (netif_tx_queue_stopped(txq) ||
+ netif_tx_queue_frozen(txq) ||
dev->hard_start_xmit(skb, dev) != NETDEV_TX_OK) {
skb_queue_head(&npinfo->txq, skb);
__netif_tx_unlock(txq);
diff --git a/net/core/pktgen.c b/net/core/pktgen.c
index c7d484f..3284605 100644
--- a/net/core/pktgen.c
+++ b/net/core/pktgen.c
@@ -3305,6 +3305,7 @@ static __inline__ void pktgen_xmit(struct pktgen_dev *pkt_dev)

txq = netdev_get_tx_queue(odev, queue_map);
if (netif_tx_queue_stopped(txq) ||
+ netif_tx_queue_frozen(txq) ||
need_resched()) {
idle_start = getCurUs();

@@ -3320,7 +3321,8 @@ static __inline__ void pktgen_xmit(struct pktgen_dev *pkt_dev)

pkt_dev->idle_acc += getCurUs() - idle_start;

- if (netif_tx_queue_stopped(txq)) {
+ if (netif_tx_queue_stopped(txq) ||
+ netif_tx_queue_frozen(txq)) {
pkt_dev->next_tx_us = getCurUs(); /* TODO */
pkt_dev->next_tx_ns = 0;
goto out; /* Try the next interface */
@@ -3352,7 +3354,8 @@ static __inline__ void pktgen_xmit(struct pktgen_dev *pkt_dev)
txq = netdev_get_tx_queue(odev, queue_map);

__netif_tx_lock_bh(txq);
- if (!netif_tx_queue_stopped(txq)) {
+ if (!netif_tx_queue_stopped(txq) &&
+ !netif_tx_queue_frozen(txq)) {

atomic_inc(&(pkt_dev->skb->users));
retry_now:
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index fd2a6ca..f17551a 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -135,7 +135,8 @@ static inline int qdisc_restart(struct Qdisc *q)
txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));

HARD_TX_LOCK(dev, txq, smp_processor_id());
- if (!netif_subqueue_stopped(dev, skb))
+ if (!netif_tx_queue_stopped(txq) &&
+ !netif_tx_queue_frozen(txq))
ret = dev_hard_start_xmit(skb, dev, txq);
HARD_TX_UNLOCK(dev, txq);

@@ -162,7 +163,8 @@ static inline int qdisc_restart(struct Qdisc *q)
break;
}

- if (ret && netif_tx_queue_stopped(txq))
+ if (ret && (netif_tx_queue_stopped(txq) ||
+ netif_tx_queue_frozen(txq)))
ret = 0;

return ret;

2008-07-26 10:53:32

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Sat, Jul 26, 2008 at 02:18:46AM -0700, David Miller wrote:
> From: Jarek Poplawski <[email protected]>
> Date: Fri, 25 Jul 2008 22:01:37 +0200
>
> > On Fri, Jul 25, 2008 at 09:36:15PM +0200, Johannes Berg wrote:
> > > On Fri, 2008-07-25 at 21:34 +0200, Jarek Poplawski wrote:
> > >
> > > No, we need to be able to lock out multiple TX paths at once.
> >
> > IMHO, it can do the same. We could e.g. insert a locked spinlock into
> > this noop_tx_handler(), to give everyone some waiting.
>
> I think there might be an easier way, but we may have
> to modify the state bits a little.

Yes, this looks definitely easier, but here is this one little bit
more, plus additional code to handle this in various places. Ingo's
proposal needs a (one?!) bit more thinking in one place, but it
shouldn't add even a bit to tx path (and it looks really cool!). Of
course, it could be re-considered in some other time too.

BTW, it seems with "Ingo's method" this netif_queue_stopped() check
could be removed too - the change of handlers could be done with
single qdiscs as well.

Jarek P.

2008-07-26 13:17:55

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Sat, Jul 26, 2008 at 02:18:46AM -0700, David Miller wrote:
...
> I think there might be an easier way, but we may have
> to modify the state bits a little.
>
> Every call into ->hard_start_xmit() is made like this:
>
> 1. lock TX queue
> 2. check TX queue stopped
> 3. call ->hard_start_xmit() if not stopped
>
> This means that we can in fact do something like:
>
> unsigned int i;
>
> for (i = 0; i < dev->num_tx_queues; i++) {
> struct netdev_queue *txq;
>
> txq = netdev_get_tx_queue(dev, i);
> spin_lock_bh(&txq->_xmit_lock);
> netif_tx_freeze_queue(txq);
> spin_unlock_bh(&txq->_xmit_lock);
> }
>
> netif_tx_freeze_queue() just sets a new bit we add.
>
> Then we go to the ->hard_start_xmit() call sites and check this new
> "frozen" bit as well as the existing "stopped" bit.
>
> When we unfreeze each queue later, we see if it is stopped, and if not
> we schedule it's qdisc for packet processing.

I guess some additional synchronization will be added yet to prevent
parallel freeze and especially unfreeze.

Jarek P.

2008-07-26 13:45:56

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

Andrew Morton wrote:
> On Thu, 24 Jul 2008 16:34:36 +0300 Pekka Enberg <[email protected]> wrote:
>
>>>> [PATCH] netfilter: fix double-free and use-after free
>>>>
>> Ingo, can you please test this? Andrew, I'm at OLS so can you pick up
>> the patch in your tree?
>
> Sure. Or Patrick can do so and it can be merged via the net tree.

I've picked it up and will merge it via Dave.

2008-07-26 13:47:34

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

Linus Torvalds wrote:
>
> On Thu, 24 Jul 2008, Patrick McHardy wrote:
>> To fix this I think we need a __krealloc() that doesn't
>> free the old memory, especially since it must not be
>> freed immediately because it may still be used in a RCU
>> read side (see the last part in the patch attached to
>> this mail (based on a kernel without your patch)).
>
> Hmm. Don't you need to fix some of the ordering of the initialization too?
>
> If there are possible readers that happen in parallel with changing this
> thing, don't you need to protect the update of "ext->len" against the
> actual changes? And the readers should probably have a read barrier
> between checking "len" and actually looking at the values?

Extensions can only be added while the conntrack is "unconfirmed",
meaning its not in the hash tables yet and the reference is exclusive.
The reason why we need RCU at all is that the extension areas
might already visible and contained in RCU protected lists.

> Finally, why do
> the "ct->ext" dereference thing, when we know it has to be equal to "new"?

Thats a relict of the old code, which allocated "new" conditionally.
I've taken that part from your patch without the smb_wmb(), which
shouldn't be necessary.

2008-07-26 16:09:21

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

Ingo Molnar wrote:
> * Andrew Morton <[email protected]> wrote:
>
>> Ingo, did this patch actually fix something over there?
>
> not that i know of. I'll do some re-tests tomorrow to make sure.

I managed to reproduce the bug you reported, but it reliably
goes away if I add Pekka's patch. Did you get a chance to
retest already?

2008-07-26 17:36:28

[permalink] [raw]

Subject: Re: [regression] nf_iterate(), BUG: unable to handle kernel NULL pointer dereference

* Patrick McHardy <[email protected]> wrote:

> Ingo Molnar wrote:
>> * Andrew Morton <[email protected]> wrote:
>>
>>> Ingo, did this patch actually fix something over there?
>>
>> not that i know of. I'll do some re-tests tomorrow to make sure.
>
> I managed to reproduce the bug you reported, but it reliably goes away
> if I add Pekka's patch. Did you get a chance to retest already?

not yet - please consider it fixed, i'll let you know if something
similar to that occurs.

Ingo

2008-07-27 00:34:47

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

From: Jarek Poplawski <[email protected]>
Date: Sat, 26 Jul 2008 15:18:38 +0200

> I guess some additional synchronization will be added yet to prevent
> parallel freeze and especially unfreeze.

Yes, that could be a problem. Using test_and_set_bit() can
guard the freezing sequence itself, but it won't handle
letting two threads of control freeze and unfreeze safely
without a reference count.

We want this thing to be able to be used flexbly, which means
we can't just assume that this is a short code sequence and
the unfreeze will come quickly. That pretty much rules
out using a new lock around the operation or anything
like that.

So I guess we could replace the state bit with a reference
count. It doesn't even need to be atomic since it is set
and tested under dev_queue->_xmit_lock

2008-07-27 20:38:20

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Sat, Jul 26, 2008 at 05:34:34PM -0700, David Miller wrote:
> From: Jarek Poplawski <[email protected]>
> Date: Sat, 26 Jul 2008 15:18:38 +0200
>
> > I guess some additional synchronization will be added yet to prevent
> > parallel freeze and especially unfreeze.
>
> Yes, that could be a problem. Using test_and_set_bit() can
> guard the freezing sequence itself, but it won't handle
> letting two threads of control freeze and unfreeze safely
> without a reference count.
>
> We want this thing to be able to be used flexbly, which means
> we can't just assume that this is a short code sequence and
> the unfreeze will come quickly. That pretty much rules
> out using a new lock around the operation or anything
> like that.
>
> So I guess we could replace the state bit with a reference
> count. It doesn't even need to be atomic since it is set
> and tested under dev_queue->_xmit_lock

Looks like enough to me. (Probably it could even share space with
the state.)

Jarek P.

2008-07-31 12:29:47

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

From: Jarek Poplawski <[email protected]>
Date: Sun, 27 Jul 2008 22:37:57 +0200

> Looks like enough to me. (Probably it could even share space with
> the state.)

So I made some progress on this, three things:

1) I remember why I choose a to use a bit in my design, it's so that
it does not increase the costs of the checks in the fast paths.
test_bit(X) && test_bit(Y) can be combined into a single test by
the compiler.

2) We can't use the reference counting scheme, because we don't want
to let a second cpu into these protected code paths just because
another is in the middle of using a freeze too.

3) So we can simply put a top-level TX spinlock around these things.

Therefore all the hot paths:

a) grab _xmit_lock
b) check XOFF and FROZEN
c) only call ->hard_start_xmit() if both bits are clear

netif_tx_lock() does:

1) grab netdev->tx_global_lock
2) for_each_tx_queue() {
lock(txq);
set_bit(FROZEN);
unlock(txq);
}

and unlock does:

1) for_each_tx_queue() {
clear_bit(FROZEN);
if (!test_bit(XOFF))
__netif_schedule();
}
2) release netdev->tx_global_lock

And this seems to satisfy all the constraints which are:

1) Must act like a lock and protect execution of the code path
which occurs inside of "netif_tx_{lock,unlock}()"

2) Must ensure no cpus are executing inside of ->hard_start_xmit()
after netif_tx_lock() returns.

3) Must not try to grab all the TX queue locks at once.

This top-level tx_global_lock also simplifies the freezing, as
it makes sure only one cpu is initiating or finishing a freeze
at any given time.

I've also adjusted code that really and truly only wanted to
lock one queue at a time, which in particular was IFB and the
teql scheduler.

It's late here, but I'll start testing the following patch on my
multiqueue capable cards after some sleep.

diff --git a/drivers/net/ifb.c b/drivers/net/ifb.c
index 0960e69..e4fbefc 100644
--- a/drivers/net/ifb.c
+++ b/drivers/net/ifb.c
@@ -69,18 +69,20 @@ static void ri_tasklet(unsigned long dev)
struct net_device *_dev = (struct net_device *)dev;
struct ifb_private *dp = netdev_priv(_dev);
struct net_device_stats *stats = &_dev->stats;
+ struct netdev_queue *txq;
struct sk_buff *skb;

+ txq = netdev_get_tx_queue(_dev, 0);
dp->st_task_enter++;
if ((skb = skb_peek(&dp->tq)) == NULL) {
dp->st_txq_refl_try++;
- if (netif_tx_trylock(_dev)) {
+ if (__netif_tx_trylock(txq)) {
dp->st_rxq_enter++;
while ((skb = skb_dequeue(&dp->rq)) != NULL) {
skb_queue_tail(&dp->tq, skb);
dp->st_rx2tx_tran++;
}
- netif_tx_unlock(_dev);
+ __netif_tx_unlock(txq);
} else {
/* reschedule */
dp->st_rxq_notenter++;
@@ -115,7 +117,7 @@ static void ri_tasklet(unsigned long dev)
BUG();
}

- if (netif_tx_trylock(_dev)) {
+ if (__netif_tx_trylock(txq)) {
dp->st_rxq_check++;
if ((skb = skb_peek(&dp->rq)) == NULL) {
dp->tasklet_pending = 0;
@@ -123,10 +125,10 @@ static void ri_tasklet(unsigned long dev)
netif_wake_queue(_dev);
} else {
dp->st_rxq_rsch++;
- netif_tx_unlock(_dev);
+ __netif_tx_unlock(txq);
goto resched;
}
- netif_tx_unlock(_dev);
+ __netif_tx_unlock(txq);
} else {
resched:
dp->tasklet_pending = 1;
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index b4d056c..ee583f6 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -440,6 +440,7 @@ static inline void napi_synchronize(const struct napi_struct *n)
enum netdev_queue_state_t
{
__QUEUE_STATE_XOFF,
+ __QUEUE_STATE_FROZEN,
};

struct netdev_queue {
@@ -636,7 +637,7 @@ struct net_device
unsigned int real_num_tx_queues;

unsigned long tx_queue_len; /* Max frames per queue allowed */
-
+ spinlock_t tx_global_lock;
/*
* One part is mostly used on xmit path (device)
*/
@@ -1099,6 +1100,11 @@ static inline int netif_queue_stopped(const struct net_device *dev)
return netif_tx_queue_stopped(netdev_get_tx_queue(dev, 0));
}

+static inline int netif_tx_queue_frozen(const struct netdev_queue *dev_queue)
+{
+ return test_bit(__QUEUE_STATE_FROZEN, &dev_queue->state);
+}
+
/**
* netif_running - test if up
* @dev: network device
@@ -1475,6 +1481,26 @@ static inline void __netif_tx_lock_bh(struct netdev_queue *txq)
txq->xmit_lock_owner = smp_processor_id();
}

+static inline int __netif_tx_trylock(struct netdev_queue *txq)
+{
+ int ok = spin_trylock(&txq->_xmit_lock);
+ if (likely(ok))
+ txq->xmit_lock_owner = smp_processor_id();
+ return ok;
+}
+
+static inline void __netif_tx_unlock(struct netdev_queue *txq)
+{
+ txq->xmit_lock_owner = -1;
+ spin_unlock(&txq->_xmit_lock);
+}
+
+static inline void __netif_tx_unlock_bh(struct netdev_queue *txq)
+{
+ txq->xmit_lock_owner = -1;
+ spin_unlock_bh(&txq->_xmit_lock);
+}
+
/**
* netif_tx_lock - grab network device transmit lock
* @dev: network device
@@ -1484,12 +1510,23 @@ static inline void __netif_tx_lock_bh(struct netdev_queue *txq)
*/
static inline void netif_tx_lock(struct net_device *dev)
{
- int cpu = smp_processor_id();
unsigned int i;
+ int cpu;

+ spin_lock(&dev->tx_global_lock);
+ cpu = smp_processor_id();
for (i = 0; i < dev->num_tx_queues; i++) {
struct netdev_queue *txq = netdev_get_tx_queue(dev, i);
+
+ /* We are the only thread of execution doing a
+ * freeze, but we have to grab the _xmit_lock in
+ * order to synchronize with threads which are in
+ * the ->hard_start_xmit() handler and already
+ * checked the frozen bit.
+ */
__netif_tx_lock(txq, cpu);
+ set_bit(__QUEUE_STATE_FROZEN, &txq->state);
+ __netif_tx_unlock(txq);
}
}

@@ -1499,40 +1536,22 @@ static inline void netif_tx_lock_bh(struct net_device *dev)
netif_tx_lock(dev);
}

-static inline int __netif_tx_trylock(struct netdev_queue *txq)
-{
- int ok = spin_trylock(&txq->_xmit_lock);
- if (likely(ok))
- txq->xmit_lock_owner = smp_processor_id();
- return ok;
-}
-
-static inline int netif_tx_trylock(struct net_device *dev)
-{
- return __netif_tx_trylock(netdev_get_tx_queue(dev, 0));
-}
-
-static inline void __netif_tx_unlock(struct netdev_queue *txq)
-{
- txq->xmit_lock_owner = -1;
- spin_unlock(&txq->_xmit_lock);
-}
-
-static inline void __netif_tx_unlock_bh(struct netdev_queue *txq)
-{
- txq->xmit_lock_owner = -1;
- spin_unlock_bh(&txq->_xmit_lock);
-}
-
static inline void netif_tx_unlock(struct net_device *dev)
{
unsigned int i;

for (i = 0; i < dev->num_tx_queues; i++) {
struct netdev_queue *txq = netdev_get_tx_queue(dev, i);
- __netif_tx_unlock(txq);
- }

+ /* No need to grab the _xmit_lock here. If the
+ * queue is not stopped for another reason, we
+ * force a schedule.
+ */
+ clear_bit(__QUEUE_STATE_FROZEN, &txq->state);
+ if (!test_bit(__QUEUE_STATE_XOFF, &txq->state))
+ __netif_schedule(txq->qdisc);
+ }
+ spin_unlock(&dev->tx_global_lock);
}

static inline void netif_tx_unlock_bh(struct net_device *dev)
@@ -1556,13 +1575,18 @@ static inline void netif_tx_unlock_bh(struct net_device *dev)
static inline void netif_tx_disable(struct net_device *dev)
{
unsigned int i;
+ int cpu;

- netif_tx_lock_bh(dev);
+ local_bh_disable();
+ cpu = smp_processor_id();
for (i = 0; i < dev->num_tx_queues; i++) {
struct netdev_queue *txq = netdev_get_tx_queue(dev, i);
+
+ __netif_tx_lock(txq, cpu);
netif_tx_stop_queue(txq);
+ __netif_tx_unlock(txq);
}
- netif_tx_unlock_bh(dev);
+ local_bh_enable();
}

static inline void netif_addr_lock(struct net_device *dev)
diff --git a/net/core/dev.c b/net/core/dev.c
index 63d6bcd..69320a5 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -4200,6 +4200,7 @@ static void netdev_init_queues(struct net_device *dev)
{
netdev_init_one_queue(dev, &dev->rx_queue, NULL);
netdev_for_each_tx_queue(dev, netdev_init_one_queue, NULL);
+ spin_lock_init(&dev->tx_global_lock);
}

/**
diff --git a/net/core/netpoll.c b/net/core/netpoll.c
index c127208..6c7af39 100644
--- a/net/core/netpoll.c
+++ b/net/core/netpoll.c
@@ -70,6 +70,7 @@ static void queue_process(struct work_struct *work)
local_irq_save(flags);
__netif_tx_lock(txq, smp_processor_id());
if (netif_tx_queue_stopped(txq) ||
+ netif_tx_queue_frozen(txq) ||
dev->hard_start_xmit(skb, dev) != NETDEV_TX_OK) {
skb_queue_head(&npinfo->txq, skb);
__netif_tx_unlock(txq);
diff --git a/net/core/pktgen.c b/net/core/pktgen.c
index c7d484f..3284605 100644
--- a/net/core/pktgen.c
+++ b/net/core/pktgen.c
@@ -3305,6 +3305,7 @@ static __inline__ void pktgen_xmit(struct pktgen_dev *pkt_dev)

txq = netdev_get_tx_queue(odev, queue_map);
if (netif_tx_queue_stopped(txq) ||
+ netif_tx_queue_frozen(txq) ||
need_resched()) {
idle_start = getCurUs();

@@ -3320,7 +3321,8 @@ static __inline__ void pktgen_xmit(struct pktgen_dev *pkt_dev)

pkt_dev->idle_acc += getCurUs() - idle_start;

- if (netif_tx_queue_stopped(txq)) {
+ if (netif_tx_queue_stopped(txq) ||
+ netif_tx_queue_frozen(txq)) {
pkt_dev->next_tx_us = getCurUs(); /* TODO */
pkt_dev->next_tx_ns = 0;
goto out; /* Try the next interface */
@@ -3352,7 +3354,8 @@ static __inline__ void pktgen_xmit(struct pktgen_dev *pkt_dev)
txq = netdev_get_tx_queue(odev, queue_map);

__netif_tx_lock_bh(txq);
- if (!netif_tx_queue_stopped(txq)) {
+ if (!netif_tx_queue_stopped(txq) &&
+ !netif_tx_queue_frozen(txq)) {

atomic_inc(&(pkt_dev->skb->users));
retry_now:
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index 345838a..9c9cd4d 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -135,7 +135,8 @@ static inline int qdisc_restart(struct Qdisc *q)
txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));

HARD_TX_LOCK(dev, txq, smp_processor_id());
- if (!netif_subqueue_stopped(dev, skb))
+ if (!netif_tx_queue_stopped(txq) &&
+ !netif_tx_queue_frozen(txq))
ret = dev_hard_start_xmit(skb, dev, txq);
HARD_TX_UNLOCK(dev, txq);

@@ -162,7 +163,8 @@ static inline int qdisc_restart(struct Qdisc *q)
break;
}

- if (ret && netif_tx_queue_stopped(txq))
+ if (ret && (netif_tx_queue_stopped(txq) ||
+ netif_tx_queue_frozen(txq)))
ret = 0;

return ret;
diff --git a/net/sched/sch_teql.c b/net/sched/sch_teql.c
index 5372236..2c35c67 100644
--- a/net/sched/sch_teql.c
+++ b/net/sched/sch_teql.c
@@ -305,10 +305,11 @@ restart:

switch (teql_resolve(skb, skb_res, slave)) {
case 0:
- if (netif_tx_trylock(slave)) {
- if (!__netif_subqueue_stopped(slave, subq) &&
+ if (__netif_tx_trylock(slave_txq)) {
+ if (!netif_tx_queue_stopped(slave_txq) &&
+ !netif_tx_queue_frozen(slave_txq) &&
slave->hard_start_xmit(skb, slave) == 0) {
- netif_tx_unlock(slave);
+ __netif_tx_unlock(slave_txq);
master->slaves = NEXT_SLAVE(q);
netif_wake_queue(dev);
master->stats.tx_packets++;
@@ -316,7 +317,7 @@ restart:
qdisc_pkt_len(skb);
return 0;
}
- netif_tx_unlock(slave);
+ __netif_tx_unlock(slave_txq);
}
if (netif_queue_stopped(dev))
busy = 1;

2008-07-31 12:38:57

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Thursday 31 July 2008 22:29, David Miller wrote:
> From: Jarek Poplawski <[email protected]>
> Date: Sun, 27 Jul 2008 22:37:57 +0200
>
> > Looks like enough to me. (Probably it could even share space with
> > the state.)
>
> So I made some progress on this, three things:
>
> 1) I remember why I choose a to use a bit in my design, it's so that
> it does not increase the costs of the checks in the fast paths.
> test_bit(X) && test_bit(Y) can be combined into a single test by
> the compiler.

Except for the braindead volatile that gets stuck on the bitops pointer.

Last time I complained about this, a lot of noise was made and I think
Linus wanted it to stay around so we could pass volatile pointers to
bitops & co without warnings. I say we should just remove the volatile
and kill any callers that might warn...

2008-07-31 12:44:21

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

From: Nick Piggin <[email protected]>
Date: Thu, 31 Jul 2008 22:38:19 +1000

> Except for the braindead volatile that gets stuck on the bitops pointer.
>
> Last time I complained about this, a lot of noise was made and I think
> Linus wanted it to stay around so we could pass volatile pointers to
> bitops & co without warnings. I say we should just remove the volatile
> and kill any callers that might warn...

Ho hum... :)

Another way to approach that, and keep the volatile, is to have
a "test_flags()" interface that takes the bit mask of values
you want to test for cases where you know it is a single word
flags value.

The downside is that this kind of interface is easy to use
incorrectly especially when accesses to the same flags use
bot test_bit() and test_flags().

2008-08-01 04:27:41

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

From: David Miller <[email protected]>
Date: Thu, 31 Jul 2008 05:29:32 -0700 (PDT)

> It's late here, but I'll start testing the following patch on my
> multiqueue capable cards after some sleep.

As a quick followup, I tested this on a machine where I had
a multiqueue interface and could reproduce the lockdep warnings,
and the patch makes them go away.

So I've pushed the patch into net-2.6 and will send it to Linus.

2008-08-01 06:43:14

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Thu, Jul 31, 2008 at 05:29:32AM -0700, David Miller wrote:
> From: Jarek Poplawski <[email protected]>
> Date: Sun, 27 Jul 2008 22:37:57 +0200
>
> > Looks like enough to me. (Probably it could even share space with
> > the state.)

Alas I've some doubts here...

...
> static inline void netif_tx_unlock(struct net_device *dev)
> {
> unsigned int i;
>
> for (i = 0; i < dev->num_tx_queues; i++) {
> struct netdev_queue *txq = netdev_get_tx_queue(dev, i);
> - __netif_tx_unlock(txq);
> - }
>
> + /* No need to grab the _xmit_lock here. If the
> + * queue is not stopped for another reason, we
> + * force a schedule.
> + */
> + clear_bit(__QUEUE_STATE_FROZEN, &txq->state);

The comments in asm-x86/bitops.h to set_bit/clear_bit are rather queer
about reordering on non x86: isn't eg. smp_mb_before_clear_bit()
useful here?

> + if (!test_bit(__QUEUE_STATE_XOFF, &txq->state))
> + __netif_schedule(txq->qdisc);
> + }
> + spin_unlock(&dev->tx_global_lock);
> }
...

> diff --git a/net/core/dev.c b/net/core/dev.c
> index 63d6bcd..69320a5 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -4200,6 +4200,7 @@ static void netdev_init_queues(struct net_device *dev)
> {
> netdev_init_one_queue(dev, &dev->rx_queue, NULL);
> netdev_for_each_tx_queue(dev, netdev_init_one_queue, NULL);
> + spin_lock_init(&dev->tx_global_lock);

This will probably need some lockdep annotations similar to
_xmit_lock.

> diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
> index 345838a..9c9cd4d 100644
> --- a/net/sched/sch_generic.c
> +++ b/net/sched/sch_generic.c
> @@ -135,7 +135,8 @@ static inline int qdisc_restart(struct Qdisc *q)
> txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));
>
> HARD_TX_LOCK(dev, txq, smp_processor_id());
> - if (!netif_subqueue_stopped(dev, skb))
> + if (!netif_tx_queue_stopped(txq) &&
> + !netif_tx_queue_frozen(txq))
> ret = dev_hard_start_xmit(skb, dev, txq);
> HARD_TX_UNLOCK(dev, txq);

This thing is the most doubtful to me: before this patch callers would
wait on this lock. Now they take the lock without problems, check the
flags, and let to take this lock again, doing some re-queing in the
meantime.

So, it seems HARD_TX_LOCK should rather do some busy looping now with
a trylock, and re-checking the _FROZEN flag. Maybe even this should
be done in __netif_tx_lock(). On the other hand, this shouldn't block
too much the owner of tx_global_lock() with taking such a lock.

Jarek P.

2008-08-01 06:56:47

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Fri, Aug 01, 2008 at 06:48:10AM +0000, Jarek Poplawski wrote:
> On Thu, Jul 31, 2008 at 05:29:32AM -0700, David Miller wrote:
...
> > diff --git a/net/core/dev.c b/net/core/dev.c
> > index 63d6bcd..69320a5 100644
> > --- a/net/core/dev.c
> > +++ b/net/core/dev.c
> > @@ -4200,6 +4200,7 @@ static void netdev_init_queues(struct net_device *dev)
> > {
> > netdev_init_one_queue(dev, &dev->rx_queue, NULL);
> > netdev_for_each_tx_queue(dev, netdev_init_one_queue, NULL);
> > + spin_lock_init(&dev->tx_global_lock);
>
> This will probably need some lockdep annotations similar to
> _xmit_lock.

...BTW, we probably could also consider some optimization here: the
xmit_lock of the first queue could be treated as special, and only
the owner could do such a freezing. This would save changes of
functionality to non mq devices. On the other hand, it would need
remembering about this special treatment (so, eg. a separate lockdep
initialization than all the others).

Jarek P.

2008-08-01 07:00:24

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

From: Jarek Poplawski <[email protected]>
Date: Fri, 1 Aug 2008 06:48:10 +0000

> On Thu, Jul 31, 2008 at 05:29:32AM -0700, David Miller wrote:
> > + /* No need to grab the _xmit_lock here. If the
> > + * queue is not stopped for another reason, we
> > + * force a schedule.
> > + */
> > + clear_bit(__QUEUE_STATE_FROZEN, &txq->state);
>
> The comments in asm-x86/bitops.h to set_bit/clear_bit are rather queer
> about reordering on non x86: isn't eg. smp_mb_before_clear_bit()
> useful here?

It doesn't matter, we need no synchronization here at all.

We unconditionally perform a __netif_schedule(), and that
will run the TX queue on the local cpu. We will take the
_xmit_lock at least once time if in fact the queue was not
stopped before the first froze it.

> > diff --git a/net/core/dev.c b/net/core/dev.c
> > index 63d6bcd..69320a5 100644
> > --- a/net/core/dev.c
> > +++ b/net/core/dev.c
> > @@ -4200,6 +4200,7 @@ static void netdev_init_queues(struct net_device *dev)
> > {
> > netdev_init_one_queue(dev, &dev->rx_queue, NULL);
> > netdev_for_each_tx_queue(dev, netdev_init_one_queue, NULL);
> > + spin_lock_init(&dev->tx_global_lock);
>
> This will probably need some lockdep annotations similar to
> _xmit_lock.

I highly doubt it. It will never be taken nested with another
device's instance.

It is only ->hard_start_xmit() leading to another ->hard_start_xmit()
where this can currently happen, but tx_global_lock will not be used
in such paths.

> > @@ -135,7 +135,8 @@ static inline int qdisc_restart(struct Qdisc *q)
> > txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));
> >
> > HARD_TX_LOCK(dev, txq, smp_processor_id());
> > - if (!netif_subqueue_stopped(dev, skb))
> > + if (!netif_tx_queue_stopped(txq) &&
> > + !netif_tx_queue_frozen(txq))
> > ret = dev_hard_start_xmit(skb, dev, txq);
> > HARD_TX_UNLOCK(dev, txq);
>
> This thing is the most doubtful to me: before this patch callers would
> wait on this lock. Now they take the lock without problems, check the
> flags, and let to take this lock again, doing some re-queing in the
> meantime.
>
> So, it seems HARD_TX_LOCK should rather do some busy looping now with
> a trylock, and re-checking the _FROZEN flag. Maybe even this should
> be done in __netif_tx_lock(). On the other hand, this shouldn't block
> too much the owner of tx_global_lock() with taking such a lock.

'ret' will be NETDEV_TX_BUSY in such a case (finding the queue
frozen), which will cause the while() loop in __qdisc_run() to
terminate.

The freezer will unconditionally schedule a new __qdisc_run()
when it unfreezes the queue.

Sure it's possible for some cpus to bang in and out of there
a few times, but that's completely harmless. And it can only
happen a few times since this freeze state is only held across
a critical section.

2008-08-01 07:01:58

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

From: Jarek Poplawski <[email protected]>
Date: Fri, 1 Aug 2008 07:01:50 +0000

> On Fri, Aug 01, 2008 at 06:48:10AM +0000, Jarek Poplawski wrote:
> > On Thu, Jul 31, 2008 at 05:29:32AM -0700, David Miller wrote:
> ...
> > > diff --git a/net/core/dev.c b/net/core/dev.c
> > > index 63d6bcd..69320a5 100644
> > > --- a/net/core/dev.c
> > > +++ b/net/core/dev.c
> > > @@ -4200,6 +4200,7 @@ static void netdev_init_queues(struct net_device *dev)
> > > {
> > > netdev_init_one_queue(dev, &dev->rx_queue, NULL);
> > > netdev_for_each_tx_queue(dev, netdev_init_one_queue, NULL);
> > > + spin_lock_init(&dev->tx_global_lock);
> >
> > This will probably need some lockdep annotations similar to
> > _xmit_lock.
>
> ...BTW, we probably could also consider some optimization here: the
> xmit_lock of the first queue could be treated as special, and only
> the owner could do such a freezing. This would save changes of
> functionality to non mq devices. On the other hand, it would need
> remembering about this special treatment (so, eg. a separate lockdep
> initialization than all the others).

I think special casing the zero's queue's lock is a bad idea.
Having a real top-level synchronizer is a powerful tool and
we could use it for other things.

2008-08-01 07:09:44

[permalink] [raw]

Subject: Re: Kernel WARNING: at net/core/dev.c:1330 __netif_schedule+0x2c/0x98()

On Thu, 2008-07-31 at 21:27 -0700, David Miller wrote:
> From: David Miller <[email protected]>
> Date: Thu, 31 Jul 2008 05:29:32 -0700 (PDT)
>
> > It's late here, but I'll start testing the following patch on my
> > multiqueue capable cards after some sleep.
>
> As a quick followup, I tested this on a machine where I had
> a multiqueue interface and could reproduce the lockdep warnings,
> and the patch makes them go away.
>
> So I've pushed the patch into net-2.6 and will send it to Linus.

Thanks david!

2008-08-01 07:36:28