From: Theodore Tso Subject: Re: [patch] document flash/RAID dangers Date: Wed, 26 Aug 2009 08:40:58 -0400 Message-ID: <20090826124058.GK32712@mit.edu> References: <20090825233701.GH4300@elf.ucw.cz> <4A947839.4010601@redhat.com> <20090826000657.GK4300@elf.ucw.cz> <4A947E05.8070406@redhat.com> <20090826002045.GO4300@elf.ucw.cz> <4A9481BE.1030308@redhat.com> <20090826003803.GP4300@elf.ucw.cz> <4A9485A6.1010803@redhat.com> <20090826112121.GD26595@elf.ucw.cz> <4A952370.50603@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Pavel Machek , david@lang.hm, Florian Weimer , Goswin von Brederlow , Rob Landley , kernel list , Andrew Morton , mtk.manpages@gmail.com, rdunlap@xenotime.net, linux-doc@vger.kernel.org, linux-ext4@vger.kernel.org, corbet@lwn.net To: Ric Wheeler Return-path: Received: from THUNK.ORG ([69.25.196.29]:35753 "EHLO thunker.thunk.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932850AbZHZMlL (ORCPT ); Wed, 26 Aug 2009 08:41:11 -0400 Content-Disposition: inline In-Reply-To: <4A952370.50603@redhat.com> Sender: linux-ext4-owner@vger.kernel.org List-ID: On Wed, Aug 26, 2009 at 07:58:40AM -0400, Ric Wheeler wrote: >> Drive in raid 5 failed; hot spare was available (no idea about >> UPS). System apparently locked up trying to talk to the failed drive, >> or maybe admin just was not patient enough, so he just powercycled the >> array. He lost the array. >> >> So while most people will not agressively powercycle the RAID array, >> drive failure still provokes little tested error paths, and getting >> unclean shutdown is quite easy in such case. > > Then what we need to document is do not power cycle an array during a > rebuild, right? Well, the softwar raid layer could be improved so that it implements scrubbing by default (i.e., have the md package install a cron job to implement a periodict scrub pass automatically). The MD code could also regularly check to make sure the hot spare is OK; the other possibility is that hot spare, which hadn't been used in a long time, had silently failed. > In the end, there are cascading failures that will defeat any data > protection scheme, but that does not mean that the value of that scheme > is zero. We need to be get more people to use RAID (including MD5) and > try to enhance it as we go. Just using a single disk is not a good > thing... Yep; the solution is to improve the storage devices. It is *not* to encourage people to think RAID is not worth it, or that somehow ext2 is better than ext3 because it runs fsck's all the time at boot up. That's just crazy talk. - Ted