Hello,
I have a (well, at least one) show-stopping problems with the 2.6
kernel while doing heavy I/O. I have a (software) RAID1 of network block
devices (nbd0 and nbd1) set up on two identical machines in an
active-passive HA cluster configuration. When the "primary" node goes
down and comes back up it recovers the RAID as follows:
Start RAID in degraded mode with remote device (nbd1)
Hot-add local device (nbd0)
This all works. Hot-adding the local device causes a resync and that is
where the problems begin. Once the resync begins the system becomes
unusable. Anything that wants to write something to the syslog socket
("/dev/log") syncronously hangs until the resync completes. The system
load goes up to 18 or so. Writing stuff to the local disk ("/etc" for
example, which is not part of the RAID) sometimes hangs. When the resync
is complete everything is happy again. Resyncing takes about 25 minutes
(20GB over a dedicated network interface to the client at 1000Mbps) and
makes the recovery time unacceptable. Also, during this recovery the OOM
killer will occasionally be invoked and kill something randomly even
though there is typically plenty of unused swap lying around before
(though perhaps "java" just decides to eat all of that VERY quickly and I
don't notice this, since that is what the OOM killer choses to kill.)
Does anyone have any ideas ?
Information about the systems:
Info: Linux cog1 2.6.9-5.0.5.ELsmp #1 SMP Fri Apr 8 14:29:47 EDT 2005 i686 i686 i386 GNU/Linux
Dist: RedHat Enterprise Linux 4
Spec:
2 x 3.2GHz Xeon (each system, with hyperthreading so 4 logical processors)
4GB of physical RAM
2GB of configured swap (partition, contigious)
2 x 1000Mbps (Intel 82546GB) network cards (HA cluster link is
provided by a cross over cable between the two nodes)
> Info: Linux cog1 2.6.9-5.0.5.ELsmp #1 SMP Fri Apr 8 14:29:47 EDT 2005 i686 i686
> i386 GNU/Linux
> Dist: RedHat Enterprise Linux 4
> Spec:
> 2 x 3.2GHz Xeon (each system, with hyperthreading so 4 logical processors)
> 4GB of physical RAM
> 2GB of configured swap (partition, contigious)
> 2 x 1000Mbps (Intel 82546GB) network cards (HA cluster link is
> provided by a cross over cable between the two nodes)
> -
Since you are using a vendor kernel which is older than the current 2.6 kernel.org one - you are better off posting to appropriate vendor mailing list or ask them for support if you have a contract. Or else, try to reproduce the problem with latest kernel.org kernel and then re-post the information here.
Parag
Bernd,
The ENBD project requires a kernel patch for 2.6 support and I
would like to remain with the vendor supplied (and "blessed" kernel) for
support reasons.
Since I cannot provide feedback on the latest version of the
kernel and I don't want to have an "unblessed" kernel, I've opened a
ticket with RedHat attached to my support contract for this problem. I
mainly posted my question here to see if this was a known issue with 2.6
in general or NBD-specific.
I had not heard of "DRBD" before now, but it looks interesting. I
am looking into it further.
Thanks,
Roy Keene
On Tue, 31 May 2005, Bernd Schubert wrote:
> Hello Roy,
>
> Roy Keene wrote:
>
>> Hello,
>>
>> ??I?have?a?(well,?at?least?one)?show-stopping?problems?with?the?2.6
>> kernel while doing heavy I/O.??I?have?a?(software)?RAID1?of?network?block
>> devices (nbd0 and nbd1) set up on two identical machines in an
>
> what about using drbd or enbd? AFAIK both are much better tested/suited for
> network raid.
>
> Cheers,
> ?Bernd
>
>
> --
> Bernd Schubert
> PCI / Theoretische Chemie
> Universit?t Heidelberg
> INF 229
> 69120 Heidelberg
> e-mail: [email protected]
>
On May 31, 2005, at 15:00:46, Roy Keene wrote:
> I had not heard of "DRBD" before now, but it looks interesting.
> I am looking into it further.
For DRBD, I recommend first installing DRBD, then setting up and
installing Heartbeat. On Debian the process is something like
the following:
# apt-get install kernel-package kernel-source-${VERSION} drbd0.7-
module-source drbd0.7-utils heartbeat
# cd /usr/src
# tar -xjf kernel-source-${VERSION}.tar.bz2
# cd kernel-source-${VERSION}
# cp /boot/config-${VERSION}-whatever ./.config
# make-kpkg --revision ${REVISION} --append-to-version -$
{MOREVERSION} --added-modules drbd --config oldconfig --us --uc
configure modules_image
# dpkg -i /usr/src/drbd0.7-module-${VERSION}-${MOREVERSION}_$
{DRBD_VERSION}+${REVISION}_${ARCH}.deb
Then read the heartbeat docs to make yourself a /etc/ha.d/haresources
file. Mine looks like this:
# Address monarch.csl.tjhsst.edu, Kerberos master, LDAP master
king IPaddr::198.38.19.1 Kerberos::TJHSST.EDU::CSL.TJHSST.EDU
# webkdc.tjhsst.edu
king IPaddr::198.38.19.2
# weblogin.tjhsst.edu
king IPaddr::198.38.19.3
# cups.csl.tjhsst.edu
king IPaddr::198.38.19.8
# mirror.tjhsst.edu
king IPaddr::198.38.19.9
# AFS volumes
king drbddisk::afs0 AFSMount::/vicepa
emperor drbddisk::afs1 AFSMount::/vicepb
NOTE: AFSMount is a custom heartbeat script, and I use a slightly
modified drbddisk script as well.
Cheers,
Kyle Moffett
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$
L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+
PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$
r !y?(-)
------END GEEK CODE BLOCK------