Hi there,
We're having issues with our Adaptec RAID controller and I was
wondering if anyone would be able to advise on how to go about
resolving them. :)
The system:
RHEL 5.1 x86_64
Kernel 2.6.18-53.1.4.el5
aacraid 1.1-5[2437]-mh4 (The default module shipped with the RHEL kernel)
The controller:
Adaptec 3405 + BBU
BIOS : 5.2-0 (12415)
Firmware : 5.2-0 (12415)
Driver : 1.1-5 (2437)
Boot Flash : 5.2-0 (12415)
The disks:
4x Seagate ST373455SS (Firmware 0002) in a RAID10
The issue:
We continually get what seem to be adapter lockups with the error:
aacraid: Host adapter abort request (3,0,0,0)
aacraid: Host adapter reset request. SCSI hang ?
The controller hangs for a while, recovers, and everything continues normally.
We've tried a replacement controller and a replacement set of disks
(we swapped out all 4 disks) to no avail.
The problem seems (?) to happen during spikes of IO -- like when the
PostgreSQL autovacuum daemon kicks in. But not always.
>From searching around LKML and Google, this seems to be a fairly
common issue, but I'm not quite clear on the cause (hardware?
software? firmware?) or the resolution.
Any help would be greatly appreciated.
Thanks!
Regards,
Omar
On Sat, 16 Feb 2008 17:38:13 +1100
"Omar Kilani" <[email protected]> wrote:
> Hi there,
>
> We're having issues with our Adaptec RAID controller and I was
> wondering if anyone would be able to advise on how to go about
> resolving them. :)
>
> The system:
>
> RHEL 5.1 x86_64
> Kernel 2.6.18-53.1.4.el5
> aacraid 1.1-5[2437]-mh4 (The default module shipped with the RHEL kernel)
Hi,
Maybe starting there is better :
https://www.redhat.com/apps/support/
Best,
Francois
On Sat, 16 Feb 2008 17:38:13 +1100
"Omar Kilani" <[email protected]> wrote:
> Hi there,
>
> We're having issues with our Adaptec RAID controller and I was
> wondering if anyone would be able to advise on how to go about
> resolving them. :)
>
> The system:
>
> RHEL 5.1 x86_64
> Kernel 2.6.18-53.1.4.el5
> aacraid 1.1-5[2437]-mh4 (The default module shipped with the RHEL kernel)
For vendor enterprise kernels you really need to go and ask the vendor or
vendor list, they diverge from the base kernels quite a bit (as they are
full of backports not updates) and the base code is [by Linux standards]
very old.
Alan
This 'Host adapter reset request. SCSI hang ?' message hides a rather complicated-to-explain underlying hardware behavior.
Aacraid based controllers have an underlying timeout/recovery cycle that is 35 seconds long, there is a driver patch for this POST RHEL5 (but is going into RHEL5.2) that increases the default Linux per-device timeout to 45 seconds. The default in some SCSI subsystems was 60 seconds in the past, but is now standardized at 30 seconds which results in an interference pattern between the controller and Linux' SCSI subsystem. The alternate workaround is for the user to adjust the timeout in sysfs if it is shorter than this value. This is the only likely Linux driver issue that you may be having, everything is out of the scope for Linux or the aacraid driver; and needs to be addressed separately. This is not the panacea, so do NOT get your hopes up, and do NOT feel that the warning behavior is in fact a problem that needs to be solved (!)
Keep in mind that I/O completions on these controllers when everything is working is typically in the millisecond region or less.
The 3405 (or generically any aacraid based controller) is likely going through an error correction cycle on the SAS/SATA bus that is delaying the completion of I/O beyond the Linux default timeout set for the device, this may be a hardware issue (i.e.: driver or OS will not change anything) and typically will be of little concern to the Linux Kernel or Distribution folks. Or this may be a problem with an overly aggressive default timeout value as outlined immediately above. The fact that all you get are these messages and I/O continues indicates that everything is working as-designed dealing with or working around whatever the hardware issue is.
That does not mean the driver is free and clear, perhaps we are reaching a fundamental resource limit in the controller, on the bus, on the enclosure or in the drives. You may be able to mitigate this by adjusting the maximum queue depth down somewhat. You can adjust this in sysfs or indirectly through driver's controller-wide limit using insmod parameter numacb. If you find this to be the case and it is prevalent, we may need to adjust the rules within the driver regarding load balancing and queue depths to improve the general reliability and responsiveness of the system. A balance between reliability, performance, responsiveness and periodic warning messages in your logs? Target devices typically have the ability to handle at least 32 outstanding commands, often more.
My guess is that if the drives, enclosure and the controllers are all up to date (latest Firmware on all) and swapping components has not resulted in any changes in behavior, and this problem persists at per-device queue depths below 32 and timeouts of 60 seconds, that you likely have a drive compatibility problem.
One of the first incompatibilities that needs to be reconciled that comes to mind is the use of desktop class (typically self correcting, which can take more than ten seconds to respond to I/O requests) or off-spec (i.e., higher error rate drives often shipped into low-budget markets) drives. The controller needs to work with these drives, and it does, and is dealing with them in an as-designed manner. If these are the drives you have, then you are getting what you paid for. If this is the case, you can reduce the annoyance by increasing the component's timeout by programming an updated value in sysfs.
Keep in mind that if the purchaser has in their mind that they wanted an enterprise class drive, and the supplier (or the manufacturer) supplied you desktop class drives; that they are often very willing to swap out the desktop class drives in exchange for the enterprise class drives. The difference is often merely as simple as a Drive Firmware behavior.
I would not increase this timeout much beyond 120 seconds, for anything larger starts affecting your rough guarantee of I/O delivery required for server network connections. The timeout results in a quescing of I/O to the device and results in the servicing of any starved requests that may be backed up behind resource limits in the components and thus also has an affect on dealing with the fundamental resource limit issues indicated above. The messages you are seeing, although annoying, are actually a desirable, albeit clunky, handshake somewhat ensuring the guarantee of I/O delivery. It is clunky because it also leads to a pause in more recently submitted I/O as it offers up ten seconds of silence in it's honor :-(
Another problem can arise surrounding the enclosure compatibility, for it may be negotiating in bad faith with the components, or has an incompatibility with the enclosure services in the controller. Especially if the enclosure has not been certified for use with the controller.
Another is that the drive is server/enterprise class and because it is a late-model unit, specifically has never been certified with the controller and has an odd behavior not yet worked around within the contoller's firmware.
If either of these incompatibilities is the case, it is in Adaptec's interest to work with you via the technical support department to resolve this problem. They will no doubt immediately ask you for the 'diagnostic dump'. If that does not bear any fruit, then they will work with you to either internally duplicate the problem, or as a last resort require your system to duplicate the issue to acquire the all-important SAS/SATA trace.
I hope you can see why these messages are discussed ad-nauseam on the network, fingers pointing in all different directions. Good luck working on this issue now armed with this understanding!
My cookbook for the 'aacraid: Host adapter reset request. SCSI hang ?' message:
- Check for any updated firmware for the controller, targets and enclosure on the respective manufacturer's websites.
- If you get a BlinkLED (rare) following this message, contact the controller supplier's technical support department
- Check per-device queue depth in sysfs to make sure it is reasonable.
- Check per-device timeout in sysfs to make sure it is reasonable.
- Engage Drive supplier's technical support department to check through compatibility or drive class issues.
- Engage Enclosure supplier's technical support department to check through compatibility issues.
- Engage Controller supplier's (some aacraid controllers are OEM'd through other channels) technical support department to check through compatibility issues.
- Tech support will ask for the 'support.zip' and diagnostic dumps to triage issue.
- Tech support may further engage you to work with you to acquire additional details from your specific system.
- Or ... be happy with what class of underlying target components you have selected; for a warning does not mean there is a problem that needs to be resolved.
Sincerely -- Mark Salyzyn
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Omar Kilani
> Sent: Saturday, February 16, 2008 1:38 AM
> To: [email protected]
> Subject: aacraid: Host adapter reset request. SCSI hang ?
>
> Hi there,
>
> We're having issues with our Adaptec RAID controller and I was
> wondering if anyone would be able to advise on how to go about
> resolving them. :)
>
> The system:
>
> RHEL 5.1 x86_64
> Kernel 2.6.18-53.1.4.el5
> aacraid 1.1-5[2437]-mh4 (The default module shipped with the
> RHEL kernel)
>
> The controller:
>
> Adaptec 3405 + BBU
>
> BIOS : 5.2-0 (12415)
> Firmware : 5.2-0 (12415)
> Driver : 1.1-5 (2437)
> Boot Flash : 5.2-0 (12415)
>
> The disks:
>
> 4x Seagate ST373455SS (Firmware 0002) in a RAID10
>
> The issue:
>
> We continually get what seem to be adapter lockups with the error:
>
> aacraid: Host adapter abort request (3,0,0,0)
> aacraid: Host adapter reset request. SCSI hang ?
>
> The controller hangs for a while, recovers, and everything
> continues normally.
>
> We've tried a replacement controller and a replacement set of disks
> (we swapped out all 4 disks) to no avail.
>
> The problem seems (?) to happen during spikes of IO -- like when the
> PostgreSQL autovacuum daemon kicks in. But not always.
>
> From searching around LKML and Google, this seems to be a fairly
> common issue, but I'm not quite clear on the cause (hardware?
> software? firmware?) or the resolution.
>
> Any help would be greatly appreciated.
>
> Thanks!
>
> Regards,
> Omar
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
> pattern between the controller and Linux' SCSI subsystem. The alternate workaround is for the user to adjust the timeout in sysfs if it is shorter than this value. This is the only
Does that actually work. I consistently see libata reporting 60 seconds
but the commands are on 30 second timeouts and I've yet to get to the
bottom of this
Alan Cox [mailto:[email protected]] sez:
> Sent: Monday, February 18, 2008 9:54 AM
> To: Salyzyn, Mark
> Cc: Omar Kilani; [email protected]
> Subject: Re: aacraid: Host adapter reset request. SCSI hang ?
>
> > pattern between the controller and Linux' SCSI subsystem.
> > The alternate workaround is for the user to adjust the
> > timeout in sysfs if it is shorter than this value.
> Does that actually work. I consistently see libata reporting
> 60 seconds but the commands are on 30 second timeouts and I've
> yet to get to the bottom of this
I 'assumed' it worked, the customer that reported a related problem went silent after I suggested the change.
libata's relationship with scsi may be different than aacraid's relationship? <shrug>
Sincerely -- Mark Salyzyn
Eeek (a scientific term) - v
- realization that a unit test was not performed.
- See 'Screeeeeeeee' for more critical definition.