I've found what I believe is a potential DoS condition in IPSec using Debian
but I need help isolating the culprit. First my system:
Debian 3.1 stable with all updates as of 9/2.
Custom Linux kernel 2.6.13 (no patches)
IPTables 1.3.3 (no patches)
Shorewall 2.4.3
I use my system as a multipurpose firewall/DHCP/DNS gateway on my broadband
home connection. I have another system with the same specs on a remote
network.
Just recently I installed the racoon debian package on both machines and
established an IPSec VPN tunnel between the two routers. All has worked
generally as expected and I have connectivity between my local and remote
LAN while having regular internet service running alongside. Here is my
racoon-tool
config:
#
# Configuration file for racoon-tool
#
# See racoon-tool.conf(5) for details
#
global:
log: notify
connection(test):
src_range: 192.168.4.0/24
dst_range: 192.168.1.0/24
src_ip: A.B.C.D
dst_ip: W.X.Y.Z
authentication_algorithm: hmac_sha1
admin_status: yes
peer(A.B.C.D):
passive: off
verify_identifier: on
lifetime: time 30 min
hash_algorithm[0]: sha1
encryption_algorithm[0]: aes
my_identifier: address A.B.C.D
peers_identifier: address W.X.Y.Z ####
Now with this config, I can ping the remote LAN from my Win XP machine
successfully using "ping -t -l 1024 192.168.1.2".
I was just randomly messing around with some settings when I ran the
following:
"ping -t -l 2048 192.168.1.2"
Now I'm well aware that this command won't likely succeed due to the
excessive size of the packet (2048 bytes). What I wasn't expecting was that
I would be totally and completely unable to access my router via SSH or have
any internet connectivity. Yes, the oversize ping packet took down all the
network connectivity on my router. Upon further inspection I noticed the
packet had actually caused a kernel panic (visible only on the monitor, now
also unresponsive).
I shut down with a hard power down, rebooted, and tried again...with the
dead same result. This oversize ping packet seems to repeatedly crash the
system.
I tried while booting the current Debian stock kernel (2.6.8-2-686), and
this problem was NOT present. I think due to this, it is unlikely related
to any supporting software on my system.
I'm not sure where to proceed from here. I'd be glad to send in the kernel
traces, but I don't know how to capture them...they aren't being logged
anywhere that I can find, so I need to know how to capture them and submit
them if they're necessary.
I want to emphasize that this is somehow related to IPSec, as I tested the
exact same command but substituted http://www.google.com for my remote LAN IP
address and there were NO crashes/problems whatsoever.
This bug seems to be a high threat because:
-It occurs reproducibly.
-It occurs in the latest kernel with *no* patches.
-It did not occur with a previous kernel.
-It causes a complete system failure.
-It is extremely simple to exploit.
I will gladly provide more details upon request. I'm not experienced enough
at the low level system aspects to provide detailed debugging info from
scratch, but I do have extensive experience with administering these systems
so I can give any information necessary with a little guidance.
Note: I'm trying posting this under a 2nd email address. I tried posting
with my subscribed email address, but the post has not appeared on the list
in 24 hours, even though I'm receiving posts normally. If it results in
double posts I apologize.
-
Matt LaPlante
Matt LaPlante <[email protected]> wrote:
>
> network connectivity on my router. Upon further inspection I noticed the
> packet had actually caused a kernel panic (visible only on the monitor, now
> also unresponsive).
Thanks for the report. I'll try to track it down.
If you could jot down the important bits of the panic message
(IP, Call-Trace) it would help me find the problem much quicker.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
> -----Original Message-----
> From: [email protected] [mailto:linux-kernel-
> [email protected]] On Behalf Of Herbert Xu
> Sent: Sunday, September 04, 2005 4:24 AM
> To: Matt LaPlante
> Cc: [email protected]
> Subject: Re: Potential IPSec DoS/Kernel Panic with 2.6.13
>
> Matt LaPlante <[email protected]> wrote:
> >
> > network connectivity on my router. Upon further inspection I noticed
> the
> > packet had actually caused a kernel panic (visible only on the monitor,
> now
> > also unresponsive).
>
> Thanks for the report. I'll try to track it down.
>
> If you could jot down the important bits of the panic message
> (IP, Call-Trace) it would help me find the problem much quicker.
I'd be more than happy to help you track this one down. The problem here is
that the panic scrolls up and off the screen after which the system is
unusable. Is there a way for me to capture it or redirect it somewhere that
I can read it? I can also include my kernel config or any other system
details of interest. Thanks.
-
Matt
On 9/4/05, Matt LaPlante <[email protected]> wrote:
> > -----Original Message-----
> > From: [email protected] [mailto:linux-kernel-
> > [email protected]] On Behalf Of Herbert Xu
> > Sent: Sunday, September 04, 2005 4:24 AM
> > To: Matt LaPlante
> > Cc: [email protected]
> > Subject: Re: Potential IPSec DoS/Kernel Panic with 2.6.13
> >
> > Matt LaPlante <[email protected]> wrote:
> > >
> > > network connectivity on my router. Upon further inspection I noticed
> > the
> > > packet had actually caused a kernel panic (visible only on the monitor,
> > now
> > > also unresponsive).
> >
> > Thanks for the report. I'll try to track it down.
> >
> > If you could jot down the important bits of the panic message
> > (IP, Call-Trace) it would help me find the problem much quicker.
>
> I'd be more than happy to help you track this one down. The problem here is
> that the panic scrolls up and off the screen after which the system is
> unusable. Is there a way for me to capture it or redirect it somewhere that
> I can read it? I can also include my kernel config or any other system
> details of interest. Thanks.
>
serial console over a cross-over cable to a second box.
netconsole will let you put the console on a different box over the network.
console on line printer will let you have a permanent record of the
console output on paper.
See
Documentation/serial-console.txt
Documentation/networking/netconsole.txt
the help entry for "config LP_CONSOLE" (in drivers/char/Kconfig)
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
> -----Original Message-----
> From: [email protected] [mailto:linux-kernel-
> [email protected]] On Behalf Of Jesper Juhl
> Sent: Sunday, September 04, 2005 2:49 PM
> To: Matt LaPlante
> Cc: Herbert Xu; [email protected]
> Subject: Re: Potential IPSec DoS/Kernel Panic with 2.6.13
>
> >
> serial console over a cross-over cable to a second box.
> netconsole will let you put the console on a different box over the
> network.
> console on line printer will let you have a permanent record of the
> console output on paper.
>
> See
> Documentation/serial-console.txt
> Documentation/networking/netconsole.txt
> the help entry for "config LP_CONSOLE" (in drivers/char/Kconfig)
>
Well I've tried for several hours now to get netconsole working, but it just
won't give me output. I've tried using both built in as well as module, and
I've tried two different clients using both netcat and syslogd on different
ports. The most output I ever get is when loading the module I manage to
receive one message saying "netconsole: network logging started". I've
verified all the netconsole settings are correct in the kernel logs when it
loads. I'm not one to give up easily, but this one's got me stumped.
I know the option to use a printer is out...I might be able to manage a
serial connection, but I'll have to rebuild my kernel to support it. I'll
look into that later...
-
Matt
On 9/5/05, Matt LaPlante <[email protected]> wrote:
>
> > -----Original Message-----
> > From: [email protected] [mailto:linux-kernel-
> > [email protected]] On Behalf Of Jesper Juhl
> > Sent: Sunday, September 04, 2005 2:49 PM
> > To: Matt LaPlante
> > Cc: Herbert Xu; [email protected]
> > Subject: Re: Potential IPSec DoS/Kernel Panic with 2.6.13
> >
> > >
> > serial console over a cross-over cable to a second box.
> > netconsole will let you put the console on a different box over the
> > network.
> > console on line printer will let you have a permanent record of the
> > console output on paper.
> >
> > See
> > Documentation/serial-console.txt
> > Documentation/networking/netconsole.txt
> > the help entry for "config LP_CONSOLE" (in drivers/char/Kconfig)
> >
>
> Well I've tried for several hours now to get netconsole working, but it just
> won't give me output. I've tried using both built in as well as module, and
> I've tried two different clients using both netcat and syslogd on different
> ports. The most output I ever get is when loading the module I manage to
> receive one message saying "netconsole: network logging started". I've
> verified all the netconsole settings are correct in the kernel logs when it
> loads. I'm not one to give up easily, but this one's got me stumped.
>
> I know the option to use a printer is out...I might be able to manage a
> serial connection, but I'll have to rebuild my kernel to support it. I'll
> look into that later...
>
Hmm, just a guess; since your bug kills networking I'd suspect that
perhaps netconsole is not the best option. Serial console or printer
(if you have one obviously) would probably be better options.
If you can't get netconsole to work even for stuff prior to triggering
the oops, then that's a bit odd. Perhaps if you posted a description
of exactely what it is you are doing someone could point out the
error... Personally I haven't used netconsole for quite a while, so
my memory of how to set it up is a bit rusty, but I don't recall it
being /that/ tricky.
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
I did a direct serial connection and boy was that a lot easier. Anyway,
without further adeu, the crash:
#############################
Unable to handle kernel paging request at virtual address 50c86502
printing eip:
c01bd216
*pde = 00000000
Oops: 0000 [#1]
PREEMPT
Modules linked in: aes_i586 esp6 ah6 ipcomp esp4 ah4 af_key xfrm_user
michael_mic deflate twofish khazad wp512 arc4 serpent tea sha512 blowfish
sha256 md5 md4 cast6 cast5 des crypto_null zlib_deflate ipv6 ipt_ULOG
ipt_REJECT ipt_LOG ipt_state ipt_pkttype ipt_CONNMARK ipt_connmark ipt_owner
ipt_recent ipt_iprange ipt_multiport ip_nat_irc ip_nat_tftp ip_conntrack_irc
ip_conntrack_tftp evdev floppy pcspkr intel_agp via_rhine mii tulip agpgart
psmouse ide_cd cdrom
CPU: 0
EIP: 0060:[<c01bd216>] Not tainted VLI
EFLAGS: 00010216 (2.6.13-Firewall.CyberDog.v12)
EIP is at sha1_update+0x96/0xe0
eax: cd485f0c ebx: 0000000c ecx: 00000003 edx: 00000244
esi: 50c86502 edi: cd485f5c ebp: 50c86502 esp: c0333c40
ds: 007b es: 007b ss: 0068
Process swapper (pid: 0, threadinfo=c0332000 task=c02edb80)
Stack: 00000244 cd485f0c 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
Call Trace:
[<c01bc881>] update+0x91/0xd0
[<c01bcf11>] crypto_hmac_update+0x11/0x20
[<c02a28c3>] skb_icv_walk+0xe3/0x200
[<d0aa5d23>] esp_hmac_digest+0x63/0xa0 [esp4]
[<c01bcf00>] crypto_hmac_update+0x0/0x20
[<c01bc524>] cbc_encrypt+0x44/0x50
[<d0aa5350>] esp_output+0x350/0x420 [esp4]
[<c029c920>] xfrm4_output+0x70/0x170
[<c0260510>] ip_forward_finish+0x0/0x40
[<c02603bf>] ip_forward+0x16f/0x2c0
[<c0260510>] ip_forward_finish+0x0/0x40
[<c025ed85>] ip_rcv+0x365/0x4d0
[<c025f0a0>] ip_rcv_finish+0x0/0x280
[<c02a67a6>] packet_rcv_spkt+0x186/0x260
[<d09e77f0>] tulip_poll+0x0/0x680 [tulip]
[<c0244ea9>] netif_receive_skb+0x1a9/0x260
[<d09e7c55>] tulip_poll+0x465/0x680 [tulip]
[<d09df692>] rhine_interrupt+0x42/0x240 [via_rhine]
[<c02450fc>] net_rx_action+0xac/0x1b0
[<c0119c19>] __do_softirq+0x79/0x90
[<c0119c57>] do_softirq+0x27/0x30
[<c0119d25>] irq_exit+0x35/0x40
[<c01047be>] do_IRQ+0x1e/0x30
[<c0102df2>] common_interrupt+0x1a/0x20
[<c0100bf0>] default_idle+0x0/0x30
[<c0100c13>] default_idle+0x23/0x30
[<c0100cb0>] cpu_idle+0x50/0x60
[<c0334787>] start_kernel+0x177/0x1c0
[<c0334340>] unknown_bootoption+0x0/0x1c0
Code: 83 e1 03 74 02 f3 a4 81 c4 48 01 00 00 5b 5e 5f 5d c3 8d 76 00 8b 44
24 04 bb 40 00 00 00 29 f3 89 d9 8d 7c 06 1c 89 ee c1 e9 02 <f3> a5 89 d9 83
e1 03 74 02 f3 a4 89 c6 8d 7c 24 08 89 c2 83 c6
<0>Kernel panic - not syncing: Fatal exception in interrupt
#############################
Hope that helps! If you need more info just ask!
-
Matt