Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1764113AbYBZRpo (ORCPT ); Tue, 26 Feb 2008 12:45:44 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1764017AbYBZRpB (ORCPT ); Tue, 26 Feb 2008 12:45:01 -0500 Received: from systemlinux.org ([83.151.29.59]:40874 "EHLO m18s25.vlinux.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1763697AbYBZRou (ORCPT ); Tue, 26 Feb 2008 12:44:50 -0500 Date: Tue, 26 Feb 2008 18:43:14 +0100 From: Andre Noll To: adam radford Cc: Tony Battersby , Johannes =?iso-8859-1?Q?W=F6rner?= , James Bottomley , linux-scsi , linux-kernel Subject: Performance problems with 3ware 9500S-4LP and 2.6.25-rc3 Message-ID: <20080226174314.GA30767@skl-net.de> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="J2SCkAp4GZ/dPZZf" Content-Disposition: inline User-Agent: Mutt/1.5.9i Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 94337 Lines: 3546 --J2SCkAp4GZ/dPZZf Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi we are experiencing massive performance problems with two of our Linux servers that contain 3ware controllers on a Tyan mainboard and a couple of 1T disks. During the daily cron job that uses rsync to sync a 500G file system =66rom another machine to the raid on the 3ware controller the load jumps up, and the machine becomes sluggish as hell. For example, an ssh login to that machine takes minutes to complete and ldap becomes unreliable while the rsync job is running. Even Nagios complains about the machine being down while rsync is running. We tried the Cent-OS 2.6.18-based kernel, 2.6.23.y and linus-git from today, but all three kernels show the same very poor performance as soon as data is written to the disks on the 3ware controller. In particular commit 1e6c38c, i.e. [SCSI] 3w-9xxx: fix abysmal write performance on some motherboards which is contained in linus-current but not in the other two kernels mentioned above does not seem to make any difference. We also tried different Raid Configurations, to no avail. ATM we're using a raid10 over 4 disks with write cache enabled. Below there's some more info about the card, dmesg and lspci output and our kernel config. A similar machine works fine with FreeBSD, so I really think it's a problem with the linux driver. ATM this machine is only used as a fallback for the main server, so we'll be able to reboot and test patches. Thanks Andre -------------------------------------------------------------------- =46rom the 3DM2 web interface: Model: 9500S-4LP Firmware: FE9X 2.08.00.009 Driver: 2.26.02.010 BIOS: BE9X 2.03.01.052 Memory Installed 112 MB # of Ports 4 # of Units 1 # of Drives 4 -------------------------------------------------------------------- =46rom dmesg (linus-git): Driver 'sd' needs updating - please use bus_type methods 3ware 9000 Storage Controller device driver for Linux v2.26.02.010. ACPI: PCI Interrupt 0000:03:03.0[A] -> GSI 24 (level, low) -> IRQ 24 input: ImPS/2 Logitech Wheel Mouse as /class/input/input0 input: AT Translated Set 2 keyboard as /class/input/input1 scsi0 : 3ware 9000 Storage Controller 3w-9xxx: scsi0: Found a 3ware 9000 Storage Controller at 0xfe8ffc00, IRQ: = 24. 3w-9xxx: scsi0: Firmware FE9X 2.08.00.009, BIOS BE9X 2.03.01.052, Ports: 4. scsi 0:0:0:0: Direct-Access AMCC 9500S-4LP DISK 2.08 PQ: 0 ANSI:= 3 sd 0:0:0:0: [sda] 3906207744 512-byte hardware sectors (1999978 MB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Mode Sense: 23 00 00 00 sd 0:0:0:0: [sda] Write cache: enabled, read cache: disabled, doesn't supp= ort DPO or FUA sd 0:0:0:0: [sda] 3906207744 512-byte hardware sectors (1999978 MB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Mode Sense: 23 00 00 00 sd 0:0:0:0: [sda] Write cache: enabled, read cache: disabled, doesn't supp= ort DPO or FUA sda: sda1 sda2 sda3 sd 0:0:0:0: [sda] Attached SCSI disk -------------------------------------------------------------------- lspci -v -v -v usage: scp [-1246BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file] [-l limit] [-o ssh_option] [-P port] [-S program] [[user@]host1:]file1 [...] [[user@]host2:]file2 00:00.0 Host bridge: Intel Corporation E7501 Memory Controller Hub (rev 01) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Steppi= ng- SERR- FastB2B- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=3Dfast >TAbort- SERR- TAbort- SERR- TAbort- Reset- FastB2B- 00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI = Controller #1 (rev 02) (prog-if 00 [UHCI]) Subsystem: Intel Corporation Unknown device 24d0 Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Steppi= ng- SERR- FastB2B- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=3Dmedium >TAbort- SERR- TAbort- SERR- TAbort- SERR- TAbort- SERR- TAbort- Reset- FastB2B- 00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface= Bridge (rev 02) Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Steppi= ng- SERR- FastB2B- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=3Dmedium >TAbort- SERR- TAbort- SERR- TAbort- SERR- TAbort- SERR- TAbort- SERR- TAbort- SERR- TAbort- SERR- TAbort- Reset- FastB2B- Capabilities: [50] PCI-X bridge device Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=3D100MHz Status: Dev=3D02:1d.0 64bit+ 133MHz+ SCD- USC- SCO- SRD- Upstream: Capacity=3D65535 CommitmentLimit=3D65535 Downstream: Capacity=3D65535 CommitmentLimit=3D65535 02:1e.0 PIC: Intel Corporation 82870P2 P64H2 I/OxAPIC (rev 04) (prog-if 20 = [IO(X)-APIC]) Subsystem: Intel Corporation 82870P2 P64H2 I/OxAPIC Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Steppi= ng- SERR- FastB2B- Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=3Dfast >TAbort- SERR- TAbort- SERR- TAbort- Reset- FastB2B- Capabilities: [50] PCI-X bridge device Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=3Dconv Status: Dev=3D02:1f.0 64bit+ 133MHz+ SCD- USC- SCO- SRD- Upstream: Capacity=3D65535 CommitmentLimit=3D65535 Downstream: Capacity=3D65535 CommitmentLimit=3D65535 03:03.0 RAID bus controller: 3ware Inc 9xxx-series SATA-RAID Subsystem: 3ware Inc 9xxx-series SATA-RAID Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV+ VGASnoop- ParErr- Steppi= ng- SERR- FastB2B- Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=3Dmedium >TAbort- SERR- TAbort- SERR- TAbort- SERR-