On Fri, Oct 30, 2009 at 4:22 AM, <[email protected]> wrote:
> http://bugzilla.kernel.org/show_bug.cgi?id=14354
>
> --- Comment #152 from Alexey Fisher <[email protected]> ?2009-10-30 08:22:10 ---
> Ted,
> Thank you for explanation :)
> Notice: i learning computer forensic, and was trained to mount all evidence
> systems with "-o ro" to not contaminate it. It seems like ext4 break this
> tradition, so many forensics will surprised ?why md5sum do not match.
Ted, (Alexey there is a response to further down).
I have not followed this thread ultra-closely but Alexey's comment got
my attention.
Ignoring computer forensics, with LVM snapshots, hardware raid array
snapshots, etc. even in the presence of a dirty log, we need to be
able to mount a drive in true read-only fashion fro many backup
operations to function correctly.
XFS added an extra mount flag for that 5 or so years ago.
I hope ext4 either has or will add a true read-only mount option.
Maybe Eric Sandeen remembers the actual drivers for adding that
feature to XFS.
Alexey,
I do computer forensics as part of my job (see my signature). Never
trust the -o ro flag with any filesystem type to keep evidence from
being modified. It is not designed for forensic use. And it is hard
to test because it may work in most scenarios, but then under certain
situations, the journal gets applied, or cleared, etc.
fyi: Yes I have read where doing so is advised, but I think that
technique was developed back before Journaled filesystems were common.
With a modern FS, it is just not a reliable technique in all
situations.
If you must mount a filesystem readonly to perform an exam, then use a
hardware write-blocker to prevent modification. If the filesystem
cannot be mounted readonly because a writeblocker is in use, then you
know you have issues.
The reality is that in more complex exams, we clone the original
evidence, then perform part of the exam in a live environment. This
clearly modifies the clone, but not the original. But the process
should be repeatable by simply making more clones, etc.
Greg
--
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
Preservation and Forensic processing of Exchange Repositories White Paper -
<http://www.norcrossgroup.com/forms/whitepapers/tng_whitepaper_fpe.html>
The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
Greg Freemyer wrote:
> On Fri, Oct 30, 2009 at 4:22 AM, <[email protected]> wrote:
>> http://bugzilla.kernel.org/show_bug.cgi?id=14354
>>
>> --- Comment #152 from Alexey Fisher <[email protected]> 2009-10-30 08:22:10 ---
>> Ted,
>> Thank you for explanation :)
>> Notice: i learning computer forensic, and was trained to mount all evidence
>> systems with "-o ro" to not contaminate it. It seems like ext4 break this
>> tradition, so many forensics will surprised why md5sum do not match.
>
> Ted, (Alexey there is a response to further down).
>
> I have not followed this thread ultra-closely but Alexey's comment got
> my attention.
>
> Ignoring computer forensics, with LVM snapshots, hardware raid array
> snapshots, etc. even in the presence of a dirty log, we need to be
> able to mount a drive in true read-only fashion fro many backup
> operations to function correctly.
>
> XFS added an extra mount flag for that 5 or so years ago.
> I hope ext4 either has or will add a true read-only mount option.
> Maybe Eric Sandeen remembers the actual drivers for adding that
> feature to XFS.
>
After a little brief digging I'm not sure when the xfs mount option went
in or why...
But for both
xfs: mount -o ro,norecovery
and
ext[34]: mount -o ro,noload
I don't think either one should touch the disk.
Also, both should skip journal replay if you set the block device
readonly prior to mount (hdparm -r can do this).
-Eric
Am Freitag, den 30.10.2009, 10:14 -0500 schrieb Eric Sandeen:
> Greg Freemyer wrote:
> > On Fri, Oct 30, 2009 at 4:22 AM, <[email protected]> wrote:
> >> http://bugzilla.kernel.org/show_bug.cgi?id=14354
> >>
> >> --- Comment #152 from Alexey Fisher <[email protected]> 2009-10-30 08:22:10 ---
> >> Ted,
> >> Thank you for explanation :)
> >> Notice: i learning computer forensic, and was trained to mount all evidence
> >> systems with "-o ro" to not contaminate it. It seems like ext4 break this
> >> tradition, so many forensics will surprised why md5sum do not match.
> >
> > Ted, (Alexey there is a response to further down).
> >
> > I have not followed this thread ultra-closely but Alexey's comment got
> > my attention.
> >
> > Ignoring computer forensics, with LVM snapshots, hardware raid array
> > snapshots, etc. even in the presence of a dirty log, we need to be
> > able to mount a drive in true read-only fashion fro many backup
> > operations to function correctly.
> >
> > XFS added an extra mount flag for that 5 or so years ago.
> > I hope ext4 either has or will add a true read-only mount option.
> > Maybe Eric Sandeen remembers the actual drivers for adding that
> > feature to XFS.
> >
>
> After a little brief digging I'm not sure when the xfs mount option went
> in or why...
>
> But for both
>
> xfs: mount -o ro,norecovery
>
> and
>
> ext[34]: mount -o ro,noload
>
> I don't think either one should touch the disk.
>
> Also, both should skip journal replay if you set the block device
> readonly prior to mount (hdparm -r can do this).
Interesting tip, thank you.
But there is some problems:
1. "hdparm -r" will set complete drive to ro mode. This is bad if i
use /dev/sda1 for root and /dev/sda5 need to be forced readonly.
2. the fact xfs and ext[3,4] use different options for true_ro make
things complicated.
3. the definition of ro is broken.
4. many frustrated admins who mounted part of raid1 only with "-o ro"
Regards,
Alexey
Am Freitag, den 30.10.2009, 10:20 -0400 schrieb Greg Freemyer:
> On Fri, Oct 30, 2009 at 4:22 AM, <[email protected]> wrote:
> > http://bugzilla.kernel.org/show_bug.cgi?id=14354
> >
> > --- Comment #152 from Alexey Fisher <[email protected]> 2009-10-30 08:22:10 ---
> > Ted,
> > Thank you for explanation :)
> > Notice: i learning computer forensic, and was trained to mount all evidence
> > systems with "-o ro" to not contaminate it. It seems like ext4 break this
> > tradition, so many forensics will surprised why md5sum do not match.
>
> Ted, (Alexey there is a response to further down).
>
> I have not followed this thread ultra-closely but Alexey's comment got
> my attention.
>
> Ignoring computer forensics, with LVM snapshots, hardware raid array
> snapshots, etc. even in the presence of a dirty log, we need to be
> able to mount a drive in true read-only fashion fro many backup
> operations to function correctly.
>
> XFS added an extra mount flag for that 5 or so years ago.
>
> I hope ext4 either has or will add a true read-only mount option.
> Maybe Eric Sandeen remembers the actual drivers for adding that
> feature to XFS.
>
> Alexey,
>
> I do computer forensics as part of my job (see my signature). Never
> trust the -o ro flag with any filesystem type to keep evidence from
> being modified. It is not designed for forensic use. And it is hard
> to test because it may work in most scenarios, but then under certain
> situations, the journal gets applied, or cleared, etc.
>
> fyi: Yes I have read where doing so is advised, but I think that
> technique was developed back before Journaled filesystems were common.
> With a modern FS, it is just not a reliable technique in all
> situations.
>
> If you must mount a filesystem readonly to perform an exam, then use a
> hardware write-blocker to prevent modification. If the filesystem
> cannot be mounted readonly because a writeblocker is in use, then you
> know you have issues.
>
> The reality is that in more complex exams, we clone the original
> evidence, then perform part of the exam in a live environment. This
> clearly modifies the clone, but not the original. But the process
> should be repeatable by simply making more clones, etc.
>
> Greg
Greg,
thank you for the comment,
as a student i do not own a hardware write-blocker. But even for normal
admin work, this can cause some frustration. With your comment i
realized that i probably screwed up some raid1 array. I used "-o ro" to
open one of disks. :( need to check it now.
Alexey Fisher wrote:
> Am Freitag, den 30.10.2009, 10:14 -0500 schrieb Eric Sandeen:
...
>> After a little brief digging I'm not sure when the xfs mount option went
>> in or why...
>>
>> But for both
>>
>> xfs: mount -o ro,norecovery
>>
>> and
>>
>> ext[34]: mount -o ro,noload
>>
>> I don't think either one should touch the disk.
>>
>> Also, both should skip journal replay if you set the block device
>> readonly prior to mount (hdparm -r can do this).
>
> Interesting tip, thank you.
> But there is some problems:
> 1. "hdparm -r" will set complete drive to ro mode. This is bad if i
> use /dev/sda1 for root and /dev/sda5 need to be forced readonly.
So point it at the partition not the drive:
[root@neon ~]# hdparm -r 1 /dev/sda1
/dev/sda1:
setting readonly to 1 (on)
readonly = 1 (on)
[root@neon ~]# hdparm -r /dev/sda2
/dev/sda2:
readonly = 0 (off)
It doesn't change the hardware, it sets a flag on the kernel's block
device structure.
> 2. the fact xfs and ext[3,4] use different options for true_ro make
> things complicated.
the hazards of being an open source sysadmin I guess.
> 3. the definition of ro is broken.
depends on what you mean by ro. A user can only read from the
filesystem so it is accurate in that respect. Is "ro" for the fs or the
bdev? Semantic differences but not necessarily broken.
> 4. many frustrated admins who mounted part of raid1 only with "-o ro"
Dunno what you mean by that ...
-Eric
> Regards,
> Alexey
Am Freitag, den 30.10.2009, 11:14 -0500 schrieb Eric Sandeen:
> Alexey Fisher wrote:
> > Am Freitag, den 30.10.2009, 10:14 -0500 schrieb Eric Sandeen:
>
> ...
>
> >> After a little brief digging I'm not sure when the xfs mount option went
> >> in or why...
> >>
> >> But for both
> >>
> >> xfs: mount -o ro,norecovery
> >>
> >> and
> >>
> >> ext[34]: mount -o ro,noload
> >>
> >> I don't think either one should touch the disk.
> >>
> >> Also, both should skip journal replay if you set the block device
> >> readonly prior to mount (hdparm -r can do this).
> >
> > Interesting tip, thank you.
> > But there is some problems:
> > 1. "hdparm -r" will set complete drive to ro mode. This is bad if i
> > use /dev/sda1 for root and /dev/sda5 need to be forced readonly.
>
> So point it at the partition not the drive:
>
> [root@neon ~]# hdparm -r 1 /dev/sda1
>
> /dev/sda1:
> setting readonly to 1 (on)
> readonly = 1 (on)
> [root@neon ~]# hdparm -r /dev/sda2
>
> /dev/sda2:
> readonly = 0 (off)
>
> It doesn't change the hardware, it sets a flag on the kernel's block
> device structure.
ok, got it. Every day learning something new.
It was not clear for me, after i read man hdparm: "Get/set read-only
flag for the device. When set, Linux disallows write operations on
the device."
> > 2. the fact xfs and ext[3,4] use different options for true_ro make
> > things complicated.
>
> the hazards of being an open source sysadmin I guess.
:( are there any plans to unify mount options?
> > 3. the definition of ro is broken.
>
> depends on what you mean by ro. A user can only read from the
> filesystem so it is accurate in that respect. Is "ro" for the fs or the
> bdev? Semantic differences but not necessarily broken.
Hmm... bdev. any chance to do temporary recovery and load it as external
journal if ro used? Anyway, you already pointed me to hdparm, so i can
use it too.
> > 4. many frustrated admins who mounted part of raid1 only with "-o ro"
>
> Dunno what you mean by that ...
raid1 is down, so you need for some reasons to mount ro only one disk of
the array. Needed to do it for short time (i used -o ro), now i know
this probably was a bad idea (bad me, should read documentation). Need
to check my raid now. Suddenly i'm not alone who doing this :(
> -Eric
>
> > Regards,
> > Alexey
Eric, Greg,
Thank you
Regards,
Alexey.
Alexey Fisher wrote:
> Am Freitag, den 30.10.2009, 11:14 -0500 schrieb Eric Sandeen:
>> Alexey Fisher wrote:
>>> Am Freitag, den 30.10.2009, 10:14 -0500 schrieb Eric Sandeen:
>> ...
>>
>>>> After a little brief digging I'm not sure when the xfs mount option went
>>>> in or why...
>>>>
>>>> But for both
>>>>
>>>> xfs: mount -o ro,norecovery
>>>>
>>>> and
>>>>
>>>> ext[34]: mount -o ro,noload
>>>>
>>>> I don't think either one should touch the disk.
>>>>
>>>> Also, both should skip journal replay if you set the block device
>>>> readonly prior to mount (hdparm -r can do this).
>>> Interesting tip, thank you.
>>> But there is some problems:
>>> 1. "hdparm -r" will set complete drive to ro mode. This is bad if i
>>> use /dev/sda1 for root and /dev/sda5 need to be forced readonly.
>> So point it at the partition not the drive:
>>
>> [root@neon ~]# hdparm -r 1 /dev/sda1
>>
>> /dev/sda1:
>> setting readonly to 1 (on)
>> readonly = 1 (on)
>> [root@neon ~]# hdparm -r /dev/sda2
>>
>> /dev/sda2:
>> readonly = 0 (off)
>>
>> It doesn't change the hardware, it sets a flag on the kernel's block
>> device structure.
>
> ok, got it. Every day learning something new.
> It was not clear for me, after i read man hdparm: "Get/set read-only
> flag for the device. When set, Linux disallows write operations on
> the device."
>
>>> 2. the fact xfs and ext[3,4] use different options for true_ro make
>>> things complicated.
>> the hazards of being an open source sysadmin I guess.
>
> :( are there any plans to unify mount options?
Some of this gets done; barrier options now match across xfs & ext4, I'm
actually just writing a patch for ext3.
Doing the same for noload/norecovery would be pretty trivial.
>>> 3. the definition of ro is broken.
>> depends on what you mean by ro. A user can only read from the
>> filesystem so it is accurate in that respect. Is "ro" for the fs or the
>> bdev? Semantic differences but not necessarily broken.
>
> Hmm... bdev. any chance to do temporary recovery and load it as external
> journal if ro used? Anyway, you already pointed me to hdparm, so i can
> use it too.
There were patches floated to in-ram recovery for those blocks so that
you could have a consistent fs w/o touching the disk but it didn't seem
to get far.
>>> 4. many frustrated admins who mounted part of raid1 only with "-o ro"
>> Dunno what you mean by that ...
>
> raid1 is down, so you need for some reasons to mount ro only one disk of
> the array. Needed to do it for short time (i used -o ro), now i know
> this probably was a bad idea (bad me, should read documentation). Need
> to check my raid now. Suddenly i'm not alone who doing this :(
oh I see. Yup....
-Eric
2009/10/30 Eric Sandeen <[email protected]>:
> Alexey Fisher wrote:
>> Hmm... bdev. any chance to do temporary recovery and load it as external
>> journal if ro used? Anyway, you already pointed me to hdparm, so i can
>> use it too.
>
> There were patches floated to in-ram recovery for those blocks so that you
> could have a consistent fs w/o touching the disk but it didn't seem to get
> far.
Those were mine, I think. I got stuck at the point of needing to
handle escaped blocks -- I couldn't find any way of reliably and
efficiently getting hold of a buffer after it had been read from disk
but before anybody else had a chance to access it. I think if I could
have cracked that then I could have got it working. The only other
missing piece of functionality was handling the transition from
read-only (unrecovered) to read-write.
It would have needed a very thorough audit of the places where ext3
was doing ll_rw_block and such like, though.
Cheers,
Duane.
--
"I never could learn to drink that blood and call it wine" - Bob Dylan
On Fri, Oct 30, 2009 at 10:20:35AM -0400, Greg Freemyer wrote:
> Ignoring computer forensics, with LVM snapshots, hardware raid array
> snapshots, etc. even in the presence of a dirty log, we need to be
> able to mount a drive in true read-only fashion fro many backup
> operations to function correctly.
Can you go into more detail about "many backup operations"?
> XFS added an extra mount flag for that 5 or so years ago.
As Eric has already pointed out, "norecovery" and "noload" mean the
same thing. But not recovering the journal is dangerous; the file
system is not necessarily going to be consistent, and while the kernel
_shouldn't_ crash given an inconsistent filesystem image --- and a lot
of fsfuzzer testing and bug fixing means that it _probably_ won't
crash --- taking a backup of an inconsistent file system image due to
the journal recovery being suppressed isn't such a great idea.
As I mentioned, trying to _simulate_ a journal recovery by using the
journal instead of data blocks for those blocks in the journal is
possible, but it's a non-trival task to code up. A Google Summer of
Student project could probably do it, but it's not a day or half-day
project.
If someone is interested in simulating a journal recovery in a true ro
fashion, I'm happy to lay out the design for such a thing. Contact me
if you're interested....
- Ted
P.S. We can certainly add an alias so that ext4 understands
norecovery much like XFS does. If we are going to standardize on a
mount option, I'd agree that XFS's norecovery is probably a better
choice than ext3/4's noload.
On Sun, Nov 1, 2009 at 12:45 AM, Theodore Tso <[email protected]> wrote:
> On Fri, Oct 30, 2009 at 10:20:35AM -0400, Greg Freemyer wrote:
>> Ignoring computer forensics, with LVM snapshots, hardware raid array
>> snapshots, etc. even in the presence of a dirty log, we need to be
>> able to mount a drive in true read-only fashion fro many backup
>> operations to function correctly.
>
> Can you go into more detail about "many backup operations"?
One example is a hardware raid array that creates readonly snapshots
or clones. (Lots of those exist in the real world).
So the typical backup procedure is:
====
Queisce application (databases, etc. have utils to do this.)
Queisce filesystem (xfs_freeze -f can be done from userspace. is there
a ext4 util?)
issue raid array command to create snapshot.
release filesystem (xfs_freeze -u)
release the app (util provided by app).
Mount the snapshot readonly (true readonly with zero writes to the
block device).
Backup the readonly snapshot (to tape, etc.).
===
I believe XFS had 2 issues related to this process when first
implemented in linux.
1) It required the UUID to be unique. Obviously in the above scenario
it is not, so "mount -o nouuid" was added for xfs.
2) Journal replay was originally aways attempted in the above process,
so the "mount -o norecovery" option was added to force a true readonly
mount.
ext4 may already support mounting of readonly clones, but if not it
needs to before it will qualify as a data center ready filesystem.
Greg
--
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
Preservation and Forensic processing of Exchange Repositories White Paper -
<http://www.norcrossgroup.com/forms/whitepapers/tng_whitepaper_fpe.html>
The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
On 2009-11-02, at 14:59, Greg Freemyer wrote:
> One example is a hardware raid array that creates readonly snapshots
> or clones. (Lots of those exist in the real world).
>
> So the typical backup procedure is:
>
> ====
> Queisce application (databases, etc. have utils to do this.)
>
> Queisce filesystem (xfs_freeze -f can be done from userspace. is there
> a ext4 util?)
>
> issue raid array command to create snapshot.
>
> release filesystem (xfs_freeze -u)
>
> release the app (util provided by app).
>
> Mount the snapshot readonly (true readonly with zero writes to the
> block device).
>
> Backup the readonly snapshot (to tape, etc.).
I thought Takashi Sato was working on allowing a filesystem freeze
ioctl from userspace? This would hook into the filesystem-specific
freeze code so that when the ioctl() returns the on-disk filesystem
is fully consistent and does not even require journal replay.
> I believe XFS had 2 issues related to this process when first
> implemented in linux.
>
> 1) It required the UUID to be unique. Obviously in the above scenario
> it is not, so "mount -o nouuid" was added for xfs.
>
> 2) Journal replay was originally aways attempted in the above process,
> so the "mount -o norecovery" option was added to force a true readonly
> mount.
>
> ext4 may already support mounting of readonly clones, but if not it
> needs to before it will qualify as a data center ready filesystem.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Andreas Dilger wrote:
> On 2009-11-02, at 14:59, Greg Freemyer wrote:
>> One example is a hardware raid array that creates readonly snapshots
>> or clones. (Lots of those exist in the real world).
>>
>> So the typical backup procedure is:
>>
>> ====
>> Queisce application (databases, etc. have utils to do this.)
>>
>> Queisce filesystem (xfs_freeze -f can be done from userspace. is there
>> a ext4 util?)
>>
>> issue raid array command to create snapshot.
>>
>> release filesystem (xfs_freeze -u)
>>
>> release the app (util provided by app).
>>
>> Mount the snapshot readonly (true readonly with zero writes to the
>> block device).
>>
>> Backup the readonly snapshot (to tape, etc.).
>
> I thought Takashi Sato was working on allowing a filesystem freeze
> ioctl from userspace? This would hook into the filesystem-specific
> freeze code so that when the ioctl() returns the on-disk filesystem
> is fully consistent and does not even require journal replay.
That's in and done; most recent xfsprogs' xfs_freeze utility will even
freeze non-xfs filesystems now :) Otherwise a wrapper utility around
the ioctl would be trivial to write.
>> I believe XFS had 2 issues related to this process when first
>> implemented in linux.
>>
>> 1) It required the UUID to be unique. Obviously in the above scenario
>> it is not, so "mount -o nouuid" was added for xfs.
>>
>> 2) Journal replay was originally aways attempted in the above process,
>> so the "mount -o norecovery" option was added to force a true readonly
>> mount.
these days a frozen xfs fs should be consistent & not need replay.
-Eric
>> ext4 may already support mounting of readonly clones, but if not it
>> needs to before it will qualify as a data center ready filesystem.
On Mon, Nov 02, 2009 at 04:59:18PM -0500, Greg Freemyer wrote:
> Queisce application (databases, etc. have utils to do this.)
>
> Queisce filesystem (xfs_freeze -f can be done from userspace. is there
> a ext4 util?)
>
> issue raid array command to create snapshot.
>
> release filesystem (xfs_freeze -u)
>
> release the app (util provided by app).
>
> Mount the snapshot readonly (true readonly with zero writes to the
> block device).
>
> Backup the readonly snapshot (to tape, etc.).
This works today using ext4. The freeze ioctl quiesces the filesystem
and leaves the journal in clean state, so there is no need for a
journal replay. I take snapshots using LVM all the time. It's
slightly more work if you are using a hardware raid array since you
can't use a single integrated "lvcreate --snapshot" command, but
should Just Work today.
- Ted
On 2009-11-02, at 16:02, Eric Sandeen wrote:
> Andreas Dilger wrote:
>> I thought Takashi Sato was working on allowing a filesystem freeze
>> ioctl from userspace? This would hook into the filesystem-specific
>> freeze code so that when the ioctl() returns the on-disk filesystem
>> is fully consistent and does not even require journal replay.
>
> That's in and done; most recent xfsprogs' xfs_freeze utility will
> even freeze non-xfs filesystems now :) Otherwise a wrapper utility
> around the ioctl would be trivial to write.
It probably makes sense to add a tune2fs option to freeze and unfreeze
the
filesystem? That would allow ext* users to have an available/documented
command even if they don't have xfsprogs installed.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Andreas Dilger wrote:
> On 2009-11-02, at 16:02, Eric Sandeen wrote:
>> Andreas Dilger wrote:
>>> I thought Takashi Sato was working on allowing a filesystem freeze
>>> ioctl from userspace? This would hook into the filesystem-specific
>>> freeze code so that when the ioctl() returns the on-disk filesystem
>>> is fully consistent and does not even require journal replay.
>> That's in and done; most recent xfsprogs' xfs_freeze utility will
>> even freeze non-xfs filesystems now :) Otherwise a wrapper utility
>> around the ioctl would be trivial to write.
>
>
> It probably makes sense to add a tune2fs option to freeze and unfreeze
> the
> filesystem? That would allow ext* users to have an available/documented
> command even if they don't have xfsprogs installed.
I'd rather see a simple tool in util-linux-ng since it's no longer
fs-specific, rather than putting something in each filesystem's toolbox.
However, the command is a little dangerous; you can lock up the machine
by freezing the wrong filesystem at the wrong time... IO can stack up
(nothing stops mmap writes yet, today) and the unfreeze may require
memory, which may require writeback to ... the frozen filesystem....
In this case there is a fair bit of rope to hang oneself ;)
-Eric