On Monday August 24, [email protected] wrote:
> > +Don't damage the old data on a failed write (ATOMIC-WRITES)
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +Either whole sector is correctly written or nothing is written during
> > +powerfail.
> > +
> > + ? ? ? Because RAM tends to fail faster than rest of system during
> > + ? ? ? powerfail, special hw killing DMA transfers may be necessary;
> > + ? ? ? otherwise, disks may write garbage during powerfail.
> > + ? ? ? This may be quite common on generic PC machines.
> > +
> > + ? ? ? Note that atomic write is very hard to guarantee for RAID-4/5/6,
> > + ? ? ? because it needs to write both changed data, and parity, to
> > + ? ? ? different disks. (But it will only really show up in degraded mode).
> > + ? ? ? UPS for RAID array should help.
>
> Can someone clarify if this is true in raid-6 with just a single disk
> failure? I don't see why it would be.
It does affect raid6 with a single drive missing.
After an unclean shutdown you cannot trust any Parity block as it
is possible that some of the blocks in the stripe have been updated,
but others have not. So you must assume that all parity blocks are
wrong and update them. If you have a missing disk you cannot do that.
To take a more concrete example, imagine a 5 device RAID6 with
3 data blocks D0 D1 D2 as well a P and Q on some stripe.
Suppose that we crashed while updating D0, which would have involved
writing out D0, P and Q.
On restart, suppose D2 is missing. It is possible that 0, 1, 2, or 3
of D0, P and Q have been updated and the others not.
We can try to recompute D2 from D0 D1 and P, from
D0 P and Q or from D1, P and Q.
We could conceivably try each of those and if they all produce the
same result we might be confident of it.
If two produced the same result and the other was different we could
use a voting process to choose the 'best'. And in this particular
case I think that would work. If 0 or 3 had been updates, all would
be the same. If only 1 was updated, then the combinations that
exclude it will match. If 2 were updated, then the combinations that
exclude the non-updated block will match.
But if both D0 and D1 were being updated I think there would be too
many combinations and it would be very possibly that all three
computed values for D2 would be different.
So yes: a singly degraded RAID6 cannot promise no data corruption
after an unclean shutdown. That is why "mdadm" will not assemble such
an array unless you use "--force" to acknowledge that there has been a
problem.
NeilBrown
On Wed, 26 Aug 2009, Neil Brown wrote:
> On Monday August 24, [email protected] wrote:
>>> +Don't damage the old data on a failed write (ATOMIC-WRITES)
>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +Either whole sector is correctly written or nothing is written during
>>> +powerfail.
>>> +
>>> + ? ? ? Because RAM tends to fail faster than rest of system during
>>> + ? ? ? powerfail, special hw killing DMA transfers may be necessary;
>>> + ? ? ? otherwise, disks may write garbage during powerfail.
>>> + ? ? ? This may be quite common on generic PC machines.
>>> +
>>> + ? ? ? Note that atomic write is very hard to guarantee for RAID-4/5/6,
>>> + ? ? ? because it needs to write both changed data, and parity, to
>>> + ? ? ? different disks. (But it will only really show up in degraded mode).
>>> + ? ? ? UPS for RAID array should help.
>>
>> Can someone clarify if this is true in raid-6 with just a single disk
>> failure? I don't see why it would be.
>
> It does affect raid6 with a single drive missing.
>
> After an unclean shutdown you cannot trust any Parity block as it
> is possible that some of the blocks in the stripe have been updated,
> but others have not. So you must assume that all parity blocks are
> wrong and update them. If you have a missing disk you cannot do that.
>
> To take a more concrete example, imagine a 5 device RAID6 with
> 3 data blocks D0 D1 D2 as well a P and Q on some stripe.
> Suppose that we crashed while updating D0, which would have involved
> writing out D0, P and Q.
> On restart, suppose D2 is missing. It is possible that 0, 1, 2, or 3
> of D0, P and Q have been updated and the others not.
> We can try to recompute D2 from D0 D1 and P, from
> D0 P and Q or from D1, P and Q.
>
> We could conceivably try each of those and if they all produce the
> same result we might be confident of it.
> If two produced the same result and the other was different we could
> use a voting process to choose the 'best'. And in this particular
> case I think that would work. If 0 or 3 had been updates, all would
> be the same. If only 1 was updated, then the combinations that
> exclude it will match. If 2 were updated, then the combinations that
> exclude the non-updated block will match.
>
> But if both D0 and D1 were being updated I think there would be too
> many combinations and it would be very possibly that all three
> computed values for D2 would be different.
>
> So yes: a singly degraded RAID6 cannot promise no data corruption
> after an unclean shutdown. That is why "mdadm" will not assemble such
> an array unless you use "--force" to acknowledge that there has been a
> problem.
thanks for this detail, I would not have expected a partially degraded
raid 6 array to be this sensitive to problems.
assuming that the degradation happens prior to the power failure, what
could be done to make this safer and more predictable.
off the top of my head (and possibly an extreme performance hit, not
nessasarily suitable for everyone) is there something that could be done
with ordering the writes to the various drives?
David Lang