I forgot to mention. I am running RedHat 6.2 and the kernel release is
2.2.14-5smp.
Joaquin
----- Forwarded message from Joaquin Rapela <[email protected]> -----
Date: Wed, 27 Dec 2000 16:25:18 -0800
From: Joaquin Rapela <[email protected]>
To: [email protected]
Subject: SYN_SENT block
User-Agent: Mutt/1.2.5i
Hello,
I am not a linux kernel guy. I am running a spider that sometimes gets blocked
for long periods of time. I run a "netstat -nto" and I observe a socket in
state SYS_SENT that seems to be blocked. Its timer keeps on incrementing.
Is there any way to avoid this blocking? Is this a bug in the kernel or
something wrong in my TCP/IP configuration/settings.
Thanks in advance, Joaquin
----- End forwarded message -----
--
Joaquin Rapela
PhD Student, Signal and Image Processing Institute
University of Southern California
3740 McClintock Ave, EEB 424
Los Angeles, CA 90089-2564
email: [email protected]
tel: (213) 740-6430
fax: (213) 740-4651
On Wed, Dec 27, 2000 at 05:38:10PM -0800, Joaquin Rapela wrote:
> I forgot to mention. I am running RedHat 6.2 and the kernel release is
> 2.2.14-5smp.
It depends upon what that spider is doing. You didn't exactly
elaborate on what it was. Is it a "web spider" or a "spam harvestor"
(web spider looking for E-Mail addresses) or something else? You cross
into the wrong areas, and some sites will cut you off (warning ALWAYS
obey robots.txt). If you get cut off by a firewall, you will most often
get dropped and every SYN packet you send will get ignored and you
will be stuck in SYN SENT state till you time out. This is deliberate
on our part. Someone trespassing out of bounds in my network gets
cut off at the firewall and gets nothing back... That does several
things that I WANT. It blocks them from determining anything residing
behind my firewall, it blocks them from determining the state of
ports and servers, it slows them down when scanning my address
space, and it makes my entire network look like it is totally
unpopulated (and uninteresting). I like that... :-)
Access the wrong URL and some IDS (Instrusion Detection System)
fires and you can no longer contact anything in an entire netblock. You
won't even get an error back. Hell! We don't want to tell you we are
refusing to talk to you. We want you to ignore us and go away like we
didn't even exist!
That being said, if you reach a dead site (not currently reachable
for one reason or another) this can also happen. Normally you might
expect an ICMP UNREACHABLE. If the target site is behind a firewall
that blocks ICMP UNREACHABLE {HOST|NET_UNREACHABLE} (lots do this) you
will hang in that state till you time out because you never got an
error back from a down system.
That's life in the big city... You run a spider, you can expect
that. Sites will be hostile to you and cut you off. If you trespass
out of bounds into some honey-pot or sugarplum, you get what you
deserve, that's why we set these traps up on our servers. My robots.txt
will warn off anyone legitimate. Some spider straying beyond what is
allowed by robots.txt is going to hit a trap and be blackholed. It's
exactly what it deserves.
> Joaquin
> ----- Forwarded message from Joaquin Rapela <[email protected]> -----
> Date: Wed, 27 Dec 2000 16:25:18 -0800
> From: Joaquin Rapela <[email protected]>
> To: [email protected]
> Subject: SYN_SENT block
> User-Agent: Mutt/1.2.5i
>
> Hello,
>
> I am not a linux kernel guy. I am running a spider that sometimes gets blocked
> for long periods of time. I run a "netstat -nto" and I observe a socket in
> state SYS_SENT that seems to be blocked. Its timer keeps on incrementing.
Yup... You sent a SYN (and maybe a couple of retries) and got
absolutely nothing back. Zip. Zero. Ziltch. Of course it's timer
keeps incrementing. How else would it time out?
> Is there any way to avoid this blocking? Is this a bug in the kernel or
> something wrong in my TCP/IP configuration/settings.
Neither. You are working fine. You are being deliberately
torpedoed by systems or firewalls configured to return no error and
to ignore you. Deal.
> Thanks in advance, Joaquin
> ----- End forwarded message -----
>
> --
> Joaquin Rapela
> PhD Student, Signal and Image Processing Institute
> University of Southern California
> 3740 McClintock Ave, EEB 424
> Los Angeles, CA 90089-2564
> email: [email protected]
> tel: (213) 740-6430
> fax: (213) 740-4651
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> Please read the FAQ at http://www.tux.org/lkml/
Mike
--
Michael H. Warfield | (770) 985-6132 | [email protected]
(The Mad Wizard) | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0xDF1DD471 | possible worlds. A pessimist is sure of it!