This is my first proper kernel code proposal so please bear with me!
=Summary for busy kernel hackers=
Problem: laptop_mode wants to keep applications from waking the hard
disks but fsync calls can "sneak through". (IMHO this is a bug.)
Proposed solution: Pretend the fsync was executed and successful.
Insert two lines into the fsync and fdatasync calls in fs/sync.c:
if (unlikely(laptop_mode))
return 0;
=More background information and flamewar suggestions below=
What is Laptop_Mode?
Laptop_Mode is a feature that saves power by forced buffering of hard
disk writes in memory. It flushes the data out to disk only e.g. every
15 minutes, means it can be off most of the time. Check this for
Andrew's intro: http://lwn.net/Articles/1652/. It is disabled by
default in most distributions afaik, e.g.
http://samwel.tk/laptop_mode/packages/ubuntu.
The Problem: Any time an application uses fsync, the hard disk wakes
up despite laptop_mode being active. This not only causes higher
energy consumption than necessary (either because laptop_mode can't be
used or because of constat hd spin ups), it also damages hard disks,
as they are created to last only a certain amount of head load cycles.
See https://ata.wiki.kernel.org/index.php/Known_issues#Drives_which_perform_frequent_head_unloads_under_Linux
for details. Applications that use fsync include LibreOffice, Firefox
3 (used to a lot), Syslog
(https://www-304.ibm.com/support/docview.wss?uid=swg21255028) and e.g.
anything which uses sqlite in synchronious mode
(http://www.sqlite.org/speed.html).
The Fix was already suggested by tytso here, but not followed up:
http://tytso.livejournal.com/2009/03/15/
Existing Workarounds:
- Libraries: Severeal libraries exist which you can use with
LD_PRELOAD to disable fsync for certain applications.
- Applications: You could disable fsync in the applications.
Why are they not enough?
- Lib raries: Libraries don't allow a switch to comfortably enable
and disable fsync. You always miss a program. It's a hassle.
- Applications: Disabling the fsync call may not make sense in the
libraries. One good example is Office Software. Normally, I do want it
to fsync the file after saving. But on battery I prefer the data to be
left in memory. So fixing LibreOffice would not make a lot of sense.
Not saving there would also not be ideal: If it crashes, I would end
up with nothing. If I keep autosave active, I will at least be save
from the software crashing.
The big advantige of this proposal is not only that laptop_mode will
work as advertized and prevent all automatic spin ups to write data.
("sync" on the command line still works.), but also that this behavior
is already implemented in many distributions via laptop-mode to switch
depending on battery/AC status.
Side effects:
This would make laptop_mode a great way to cheat benchmarks that rely
on fsync for their timing, (to ensure data is actually written to disk
before the timer ends). People who don't care so much about data
safety can gain some disk speed with one switch. Crashing applications
are _not_ affected. Their data is written to memory, just not flushed
to disk.
"But this means work will be lost"
This is a complaint against this solution made e.g. here in the second
part. http://lwn.net/Articles/323780/
Yes, this is true. But Laptop_Mode makes that very, very clear. The
configuration file actually calls the setting
"LM_BATT_MAX_LOST_WORK_SECONDS". Also laptop_mode decreases and then
disables itself when the battery runs down. And finally, I don't know
distributions that enable it out of the box. If a user does enable
laptop_mode without reading anything at all about what it means, I
don't think he should be protected. If a distribution uses it by
default, it should warn users about possible consequences.
In short: That it makes laptop_mode work as advertised IHO is no valid
point against this solution. And if, we could consider a sending a
printk the first time an fsync is skipped.
"This is not necessary with SSDs"
I'm not sure, but I think that SSDs will also keep an active SATA
connection if you write to them all the time. They also have standby
modes which also save power. But yes, this make not or hardly make
sense with them. But as long as you don't send me one, I still like a
fix. Also, this might extend SSD lifetime as this reduces the total
times a sector is written.
Testing:
I've been using this workaround on my netbook for over six months now.
It works as expected for me with all software in a Ubuntu 9.10
environment and saves me at least 0.5 Watt or roughly 10 % battery
time - without destroying my hard disk. I have seen no negative side
effects.
What I didn't test:
I didn't test this on different platforms and environments. But as
it's generic code I expect no different behavior. I didn't benchmark
how this might affect performance due to the additional if() check for
every fsync call. But as the fsync is rather expensive and not used
*that* much, I think it should have not noticeable impact.
Implementation Ideas:
If this patch does turn out to affect some scenarios negatively, a
switch would be necessary. Laptop_Mode is in proc, but we don't want
more files in there. In any case, a configuration on a by device level
would make sense (e.g. no fsync on sda when laptop_mode active). That
would fit nicely into sys and allow to disable fsync only for the hard
disk, but not for the SSD or usb stick if both are present. But let's
try not to blow this up too much.
Where's the Trojan Horse?
Laptop_Mode was the trojan horse, this is a soldier.
There's no need. This already is "_obviously_ good at first sight". ;-)
Great Flame Topics (please add [flame] to the subject for better overview!):
- User Space vs. Kernel Space Fixes
- Is this obsoleted by SSDs
- Is this a bug or a feature
Thanks for your feedback! Thanks for the feedback at LinuxTag Kernel
Kwestioning.
> Problem: laptop_mode wants to keep applications from waking the hard
> disks but fsync calls can "sneak through". (IMHO this is a bug.)
Its a standards requirement and thing many apps rely on. Also you can do
it perfectly well in user space for all apps. Thats why you can
preconfigure preloads and it's why you have glic library source code!
> Proposed solution: Pretend the fsync was executed and successful.
> Insert two lines into the fsync and fdatasync calls in fs/sync.c:
> if (unlikely(laptop_mode))
> return 0;
Whoops you've just risked corrupting any app which relies on fsync for
ordering if there is a crash.
> "This is not necessary with SSDs"
> I'm not sure, but I think that SSDs will also keep an active SATA
> connection if you write to them all the time. They also have standby
On SSD the impact is miniscule as far as I can tell. A graph would tell
you more.
The "right" way to do this is more complicated by far. An fsync is an
ordering guarantee so you need to implement that ordering guarantee even
if you don't force writes to physical media. That at least cuts down most
failures (but not all - eg commits with a network component such as email
receives)
Anyway you can do it in userspace trivially if you want.
On Thu, May 19, 2011 at 4:39 PM, Alan Cox <[email protected]> wrote:
>> > That at least cuts down most failures (but not all - eg commits with a network
>> > component such as email receives)
>>
>> I don't understand your email example.
>
> Think about sendmail and SMTP
>
> Mail arrives
> Commit to disk
> fsync
> [X]
> Confirm receipt to other system
> Other system removes it from the queue it holds
>
> So if it crashes at X even though you've kept ordering and you've
> got an internally consistent view (as if it crashed earlier), that
> has been observed and acted upon by another system - so your email
> just went into the cosmic trashcan.
>
>
Ok, I think I understand your example. But isn't that the user's risk
and more a problem of the general idea of laptop mode? If I enable
laptop_mode and set MAX_LOST_WORK_SECONDS, I risk to lose data. I
would say that includes email and whatever else may happen. (And if I
don't switch it on, I risk running out of battery and being unable to
create data...). btw. I don't think it would be a good idea for people
to use sendmail or other crucial data storage applications with
laptop_mode active. I would see this as a switch for e.g. notebooks at
conferences, in class, lectures, etc.
(sorry, lost the CCs)
On Thu, May 19, 2011 at 3:43 PM, Alan Cox <[email protected]> wrote:
>> Problem: laptop_mode wants to keep applications from waking the hard
>> disks but fsync calls can "sneak through". (IMHO this is a bug.)
>
> Its a standards requirement and thing many apps rely on. Also you can do
> it perfectly well in user space for all apps. Thats why you can
> preconfigure preloads and it's why you have glic library source code!
Yes, it's another possibility I've considered. So then the patch
should go into glibc and glibc check for active laptop_mode? But if
implement it inside a library won't I get the same ordering guarantee
failure?
>
>> Proposed solution: Pretend the fsync was executed and successful.
>> Insert two lines into the fsync and fdatasync calls in fs/sync.c:
>> if (unlikely(laptop_mode))
>> return 0;
>
> Whoops you've just risked corrupting any app which relies on fsync for
> ordering if there is a crash.
(...)
> The "right" way to do this is more complicated by far. An fsync is an
> ordering guarantee so you need to implement that ordering guarantee even
> if you don't force writes to physical media.
Ok. No problem, so the exit point would be a different one.
> That at least cuts down most failures (but not all - eg commits with a network
> component such as email receives)
I don't understand your email example.
On Thu, May 19, 2011 at 03:34:46PM +0200, Dennis Jansen wrote:
> This is my first proper kernel code proposal so please bear with me!
>
> =Summary for busy kernel hackers=
> Problem: laptop_mode wants to keep applications from waking the hard
> disks but fsync calls can "sneak through". (IMHO this is a bug.)
>
> Proposed solution: Pretend the fsync was executed and successful.
> Insert two lines into the fsync and fdatasync calls in fs/sync.c:
> if (unlikely(laptop_mode))
> return 0;
No, no, no, no, no, no, no, no, no, no.
There is _absolutely no justification_ for putting people's data at
risk like this. If you want to do make fsync/fdatasync calls
no-ops, then go install libeatmydata on your systems. It's your
data, and you make the decision to risk it, not us.
And BTW, I just added your name to my "ignore fileystem/data
corruption bug reports from these people" list, because that's
exactly where you'll end up with if you follow this path.
Cheers,
Dave.
--
Dave Chinner
[email protected]
On Fri, May 20, 2011 at 5:39 AM, Dave Chinner <[email protected]> wrote:
> On Thu, May 19, 2011 at 03:34:46PM +0200, Dennis Jansen wrote:
>> This is my first proper kernel code proposal so please bear with me!
>>
>> =Summary for busy kernel hackers=
>> Problem: laptop_mode wants to keep applications from waking the hard
>> disks but fsync calls can "sneak through". (IMHO this is a bug.)
>>
>> Proposed solution: Pretend the fsync was executed and successful.
>> Insert two lines into the fsync and fdatasync calls in fs/sync.c:
>> if (unlikely(laptop_mode))
>> return 0;
>
> No, no, no, no, no, no, no, no, no, no.
>
> There is _absolutely no justification_ for putting people's data at
> risk like this. If you want to do make fsync/fdatasync calls
> no-ops, then go install libeatmydata on your systems. It's your
> data, and you make the decision to risk it, not us.
1. I thought I (may) make that decision by using laptop mode.
2. libeatmydata would _always_ be active.
3. A lib doesn't fix the ordering guarantee problem.
4. It's clear that it's not the right code. (And it is a rfc and my
first one, too...)
Any suggestions?
Thanks!
Dennis
On Thu, 19 May 2011 15:34:46 +0200, Dennis Jansen said:
> Testing:
> I've been using this workaround on my netbook for over six months now.
> It works as expected for me with all software in a Ubuntu 9.10
> environment and saves me at least 0.5 Watt or roughly 10 % battery
> time - without destroying my hard disk. I have seen no negative side
> effects.
>
> What I didn't test:
> I didn't test this on different platforms and environments. But as
> it's generic code I expect no different behavior. I didn't benchmark
> how this might affect performance due to the additional if() check for
> every fsync call. But as the fsync is rather expensive and not used
> *that* much, I think it should have not noticeable impact.
How much destructive testing did you do? In the 6 months, how many times did
the system crash (or had the battery pulled out, or whatever) while large
amounts of data were still pending after apps thought they were fsync'ed? How
much crash testing was done against apps that use fsync for ordering or
correctness reasons?
> In short: That it makes laptop_mode work as advertised IHO is no valid
> point against this solution. And if, we could consider a sending a
> printk the first time an fsync is skipped.
That would be the same printk that the user never actually *sees* because your
patch suppressed syslogd's fsync to guarantee it made it to disk, so it was
lost when the system crashed soon thereafter, along with the user's work? ;)
On Thu, 19 May 2011 17:02:21 +0200, "D. Jansen" said:
> On Thu, May 19, 2011 at 3:43 PM, Alan Cox <[email protected]> wrote:
> > That at least cuts down most failures (but not all - eg commits with a network
> > component such as email receives)
>
> I don't understand your email example.
If you don't understand Alan's example, maybe you shouldn't be messing with
code that's used for correctness guarantees. But I'll spell it out for you:
You turn on laptop mode. You check your mail. You download 5 new messages,
which your laptop then *assures* the mail server "I've got them, we're cool".
Now, this is a pretty heavy responsibility (for example, RFC2821 says this:
6.1 Reliable Delivery and Replies by Email
When the receiver-SMTP accepts a piece of mail (by sending a "250 OK"
message in response to DATA), it is accepting responsibility for
delivering or relaying the message. It must take this responsibility
seriously. It MUST NOT lose the message for frivolous reasons, such
as because the host later crashes or because of a predictable
resource shortage.
So - the mail server then deletes the mail because the laptop has told it "Yes,
I've gotten it, it's stable, and even if I crash I won't lose it". But since
the mail program's fsync() calls got suppressed, it really *isn't* stable, so
when you crash, the mails are gone. Poof.
Second order effects crop up after that - after you recover from the crash,
your end has no memory of downloading the 5 messages, so it tries again - only
to have the mail server say "No such message". This is the sort of inconsistency
that gives guys at the support desk indigestion...
Understand better now?
On Fri, May 20, 2011 at 5:28 PM, <[email protected]> wrote:
> On Thu, 19 May 2011 15:34:46 +0200, Dennis Jansen said:
>
>> Testing:
>> I've been using this workaround on my netbook for over six months now.
>> It works as expected for me with all software in a Ubuntu 9.10
>> environment and saves me at least 0.5 Watt or roughly 10 % battery
>> time - without destroying my hard disk. I have seen no negative side
>> effects.
>
> How much destructive testing did you do? In the 6 months, how many times did
> the system crash (or had the battery pulled out, or whatever) while large
> amounts of data were still pending after apps thought they were fsync'ed? How
> much crash testing was done against apps that use fsync for ordering or
> correctness reasons?
I don't see the point in verifying the obvious. Of course applications
that rely on fsync will lose data.
The real problem comes with ordering correctness, which could actually
_destroy previous data_ as well.
In my scenario (office applications, browsing) I have not hit such a problem.
Does anyone know a Linux app that actually does rely on ordering
correctness? Is that one which is used on a laptop? In laptop mode?
(-> when on battery?) Because so far the discussion seems to be
running in circles around Alan's mailer daemon. I would just shut that
down on enabling laptop mode. Problem solved. But I don't run that on
my laptop, anyway, esp. in laptop mode. I wonder who would, too.
>
>> In short: That it makes laptop_mode work as advertised IHO is no valid
>> point against this solution. And if, we could consider a sending a
>> printk the first time an fsync is skipped.
>
> That would be the same printk that the user never actually *sees* because your
> patch suppressed syslogd's fsync to guarantee it made it to disk, so it was
> lost when the system crashed soon thereafter, along with the user's work? ;)
That is a good point! So one last fsync after the message? :) Or a beep?
But I understand something else here as well: Don't give users the
choice to easily destroy their systems. It would be like having a
packaged libeatmydata that a user could just download and install
(like this one http://packages.ubuntu.com/oneiric/eatmydata) but with
fsync disabled for all programs by default upon installation. It would
be a hassle to support, because users would just enable it without
understanding what they do and later complain that data is lost. Could
it be that this is a big part of the reason people don't like to even
think about the idea in detail? Maybe we could then discuss more about
how to prevent users from using this without knowing about what they
do.
Apart from ordering, I don't see the big difference between data lost
through normal writes (postponed in laptop mode) and fsync writes
(postponed in laptop mode). Are there really apps that corrupt their
data without fsync? (That you end up without the new data like new
emails is a given and normal with laptop mode.) Which?
Thanks, I already got the email example when Alan explained it. It's
just that I doubt that the email server scenario is very likely. Email
client, yes. But an email client can leave the data on the server.
Let's not say "this is not possible, because a badly configured system
could cause a problem." Isn't that always the case?
On the other hand, one could also say that applications simply should
all not use fsync in the first place, but e.g. featherstick
(http://lwn.net/Articles/354861/) instead.
It seems the resistance to this idea is very high, though I think more
for fear of abuse and/or badly configured systems than due to normal
_laptop mode_ use cases.
Cheers
ps. Please don't put me into any filters or ignore lists just yet,
because I _am_ aware of the data loss risk. And I won't complain about
data loss on that system, but just quietly enjoy my ~ 0.5 W power
savings, even with sqlite-based and other fsyncing software running.
Maybe someone will sometime write a patch that "featherstickyfies"
fsync in laptop mode.
On Fri, May 20, 2011 at 06:40:49PM +0200, D. Jansen wrote:
> On Fri, May 20, 2011 at 5:28 PM, <[email protected]> wrote:
> > On Thu, 19 May 2011 15:34:46 +0200, Dennis Jansen said:
> >
> >> Testing:
> >> I've been using this workaround on my netbook for over six months now.
> >> It works as expected for me with all software in a Ubuntu 9.10
> >> environment and saves me at least 0.5 Watt or roughly 10 % battery
> >> time - without destroying my hard disk. I have seen no negative side
> >> effects.
> >
> > How much destructive testing did you do? ?In the 6 months, how many times did
> > the system crash (or had the battery pulled out, or whatever) while large
> > amounts of data were still pending after apps thought they were fsync'ed? How
> > much crash testing was done against apps that use fsync for ordering or
> > correctness reasons?
>
> I don't see the point in verifying the obvious. Of course applications
> that rely on fsync will lose data.
> The real problem comes with ordering correctness, which could actually
> _destroy previous data_ as well.
> In my scenario (office applications, browsing) I have not hit such a problem.
how about making fsync block until the harddisk spins up ?
this would also enable you to detect these apps you wouldnt be using in
laptop mode anyways ?
--
torben Hohn
On Sat, May 21, 2011 at 12:03 AM, torbenh <[email protected]> wrote:
> On Fri, May 20, 2011 at 06:40:49PM +0200, D. Jansen wrote:
>> On Fri, May 20, 2011 at 5:28 PM, <[email protected]> wrote:
>> > On Thu, 19 May 2011 15:34:46 +0200, Dennis Jansen said:
>> >
>> >> Testing:
>> >> I've been using this workaround on my netbook for over six months now.
>> >> It works as expected for me with all software in a Ubuntu 9.10
>> >> environment and saves me at least 0.5 Watt or roughly 10 % battery
>> >> time - without destroying my hard disk. I have seen no negative side
>> >> effects.
>> >
>> > How much destructive testing did you do? In the 6 months, how many times did
>> > the system crash (or had the battery pulled out, or whatever) while large
>> > amounts of data were still pending after apps thought they were fsync'ed? How
>> > much crash testing was done against apps that use fsync for ordering or
>> > correctness reasons?
>>
>> I don't see the point in verifying the obvious. Of course applications
>> that rely on fsync will lose data.
>> The real problem comes with ordering correctness, which could actually
>> _destroy previous data_ as well.
>> In my scenario (office applications, browsing) I have not hit such a problem.
>
> how about making fsync block until the harddisk spins up ?
> this would also enable you to detect these apps you wouldnt be using in
> laptop mode anyways ?
Very nice idea. I had tried that some time ago. Depending on the error codes I
have, the apps either failed to save and complained to the user - or -
kept trying to fsync again and again. Of course the latter wouldn't
really be a problem unless a race occurs. And we have the benefit of
being honest to user
space and letting them know we are not fsyncing. And the fsync occurs
as soon as we disable laptop mode and plug into AC. I have to try that
out again.
On Fri, May 20, 2011 at 08:01:17AM +0200, D. Jansen wrote:
> On Fri, May 20, 2011 at 5:39 AM, Dave Chinner <[email protected]> wrote:
> > On Thu, May 19, 2011 at 03:34:46PM +0200, Dennis Jansen wrote:
> >> This is my first proper kernel code proposal so please bear with me!
> >>
> >> =Summary for busy kernel hackers=
> >> Problem: laptop_mode wants to keep applications from waking the hard
> >> disks but fsync calls can "sneak through". (IMHO this is a bug.)
> >>
> >> Proposed solution: Pretend the fsync was executed and successful.
> >> Insert two lines into the fsync and fdatasync calls in fs/sync.c:
> >> if (unlikely(laptop_mode))
> >> ? ?return 0;
> >
> > No, no, no, no, no, no, no, no, no, no.
> >
> > There is _absolutely no justification_ for putting people's data at
> > risk like this. ?If you want to do make fsync/fdatasync calls
> > no-ops, then go install libeatmydata on your systems. It's your
> > data, and you make the decision to risk it, not us.
>
> 1. I thought I (may) make that decision by using laptop mode.
Laptop mode does not change fsync guarantees, so no, you ar enot
making a decision to throw data away when you select laptop mode.
> 2. libeatmydata would _always_ be active.
Your choice, really. Just use it as a preload for your apps that you
want to avoid fsync for.
> 3. A lib doesn't fix the ordering guarantee problem.
A properly implemented filesystem will not have ordering problems
just because fsyncs are not issued.
> 4. It's clear that it's not the right code. (And it is a rfc and my
> first one, too...)
>
> Any suggestions?
Your problem to solve in userspace. Changing a generic,
widely used kernel option like this is not allowed, and in general
when there are ways in userspace of playing fast and loose with
data, it's up to the user to do that in userspace, not the kernel...
Cheers,
Dave.
--
Dave Chinner
[email protected]
Am Sonntag, 22. Mai 2011, 02:48:33 schrieb Dave Chinner:
> On Fri, May 20, 2011 at 08:01:17AM +0200, D. Jansen wrote:
> > On Fri, May 20, 2011 at 5:39 AM, Dave Chinner <[email protected]> wrote:
> > 1. I thought I (may) make that decision by using laptop mode.
>
> Laptop mode does not change fsync guarantees, so no, you ar enot
> making a decision to throw data away when you select laptop mode.
You do however decide to sync less often, resulting in a potentially
larger loss of data.
> > 3. A lib doesn't fix the ordering guarantee problem.
>
> A properly implemented filesystem will not have ordering problems
> just because fsyncs are not issued.
But user space will have this problem. A single task's sequence of
write(); fsync(); write(); does give an implicit guarantee of ordering
to user space. To keep that guarantee, you need kernel support,
because only the kernel can issue the necessary barrier/flush commands.
A filesystem is not required to make sure your writes hit the disk
in the order you issued them, unless you use fsync, which gives
an even larger guarantee, but also implies ordering.
Now you may say that you want full fsync or nothing, but this is a personal
preference.
Regards
Oliver
On Fri, 20 May 2011, D. Jansen wrote:
> On Fri, May 20, 2011 at 5:28 PM, <[email protected]> wrote:
> > On Thu, 19 May 2011 15:34:46 +0200, Dennis Jansen said:
> >
> >> Testing:
> >> I've been using this workaround on my netbook for over six months now.
> >> It works as expected for me with all software in a Ubuntu 9.10
> >> environment and saves me at least 0.5 Watt or roughly 10 % battery
> >> time - without destroying my hard disk. I have seen no negative side
> >> effects.
> >
> > How much destructive testing did you do? In the 6 months, how many times did
> > the system crash (or had the battery pulled out, or whatever) while large
> > amounts of data were still pending after apps thought they were fsync'ed? How
> > much crash testing was done against apps that use fsync for ordering or
> > correctness reasons?
>
> I don't see the point in verifying the obvious. Of course applications
> that rely on fsync will lose data.
> The real problem comes with ordering correctness, which could actually
> _destroy previous data_ as well.
> In my scenario (office applications, browsing) I have not hit such a problem.
>
> Does anyone know a Linux app that actually does rely on ordering
> correctness? Is that one which is used on a laptop? In laptop mode?
> (-> when on battery?) Because so far the discussion seems to be
> running in circles around Alan's mailer daemon. I would just shut that
> down on enabling laptop mode. Problem solved. But I don't run that on
> my laptop, anyway, esp. in laptop mode. I wonder who would, too.
>
PostgreSQL's Write Ahead Log (WAL) would be one such example as far as I
can see ( http://www.westnet.com/~gsmith/content/postgresql/TuningPGWAL.htm ).
And yes, people run that on laptops while on battery (I know that for a
fact since I do that myself).
--
Jesper Juhl <[email protected]> http://www.chaosbits.net/
Don't top-post http://www.catb.org/jargon/html/T/top-post.html
Plain text mails only, please.
On Mon, May 23, 2011 at 10:12 AM, Oliver Neukum <[email protected]> wrote:
>> > 3. A lib doesn't fix the ordering guarantee problem.
>>
>> A properly implemented filesystem will not have ordering problems
>> just because fsyncs are not issued.
>
> But user space will have this problem. A single task's sequence of
> write(); fsync(); write(); does give an implicit guarantee of ordering
> to user space. To keep that guarantee, you need kernel support,
> because only the kernel can issue the necessary barrier/flush commands.
> A filesystem is not required to make sure your writes hit the disk
> in the order you issued them, unless you use fsync, which gives
> an even larger guarantee, but also implies ordering.
If this is correct, we do need some form of kernel support. To ensure
that one fsync means the write comes before another. Because how would
it be implemented in user space to ensure write ordering without
fsync? The library could only request the kernel to enforce write
ordering. Otherwise it would have to catch and buffer all writes
itself and then commit them with fsyncs in between. _That_ doesn't
sound like a good idea at all...
And the alternative would be that a write of
1. XXXXXXXXXXXXXXXX (file)
2. XXHHHHHXXXXXXX (write H)
3. XXHHHNNNNXXXX (write N)
would become
1. XXXXXXXXXXXXXXXX (file)
2. XXXXXXNNNNXXXX (write N)
3. XXHHHHHNNXXXX (write H)
And we have data that was never supposed to be on disk this way...
On Mon, May 23, 2011 at 10:12:06AM +0200, Oliver Neukum wrote:
> Am Sonntag, 22. Mai 2011, 02:48:33 schrieb Dave Chinner:
> > On Fri, May 20, 2011 at 08:01:17AM +0200, D. Jansen wrote:
> > > On Fri, May 20, 2011 at 5:39 AM, Dave Chinner <[email protected]> wrote:
>
> > > 1. I thought I (may) make that decision by using laptop mode.
> >
> > Laptop mode does not change fsync guarantees, so no, you ar enot
> > making a decision to throw data away when you select laptop mode.
>
> You do however decide to sync less often, resulting in a potentially
> larger loss of data.
>
> > > 3. A lib doesn't fix the ordering guarantee problem.
> >
> > A properly implemented filesystem will not have ordering problems
> > just because fsyncs are not issued.
>
> But user space will have this problem. A single task's sequence of
> write(); fsync(); write(); does give an implicit guarantee of ordering
> to user space.
Oh, you're talking about application level write ordering. IO
"ordering" in filesystem speak is about guaranteeing the order of
data vs metadata writes for ensuring consistency after a crash
(e.g. ext3/4 default "data=ordered" mode). that's what I was
refering to, not anythign to do with applications.
Besides, having to work out how to handle subtle application write
ordering bugs because you changed fsync semantics is simply another
reason for not changing behaviour in the first place.
Cheers,
Dave.
--
Dave Chinner
[email protected]
Am Mittwoch, 25. Mai 2011, 02:00:03 schrieb Dave Chinner:
> Oh, you're talking about application level write ordering. IO
> "ordering" in filesystem speak is about guaranteeing the order of
> data vs metadata writes for ensuring consistency after a crash
> (e.g. ext3/4 default "data=ordered" mode). that's what I was
> refering to, not anythign to do with applications.
>
> Besides, having to work out how to handle subtle application write
> ordering bugs because you changed fsync semantics is simply another
> reason for not changing behaviour in the first place.
Sure. I'd say changing the behavior is right out. The question is whether
we want an additional "superlaptop"-mode.
And even in this case it seems to me that fsync() cannot be reduced to
a nop because of ordering constraints. But perhaps we should then consider
exporting an ordering primitive to user space.
Regards
Oliver
--
- - -
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend?rffer, HRB 16746 (AG N?rnberg)
Maxfeldstra?e 5
90409 N?rnberg
Germany
- - -
On Wed, May 25, 2011 at 8:50 AM, Oliver Neukum <[email protected]> wrote:
>
> Am Mittwoch, 25. Mai 2011, 02:00:03 schrieb Dave Chinner:
> > Oh, you're talking about application level write ordering.
(...)
> > Besides, having to work out how to handle subtle application write
> > ordering bugs because you changed fsync semantics is simply another
> > reason for not changing behaviour in the first place.
>
> Sure. I'd say changing the behavior is right out. The question is whether
> we want an additional "superlaptop"-mode.
Exactly. 0.5 Watts is a lot of energy on a modern system. And on some
systems with proper link power management in sata and pcie it will be
even higher (~1 Watt I think). Think not only of the work that will be
lost if the system crashes (unlikely), but also of the work that could
never be done because the battery was gone (a certainty).
>
> And even in this case it seems to me that fsync() cannot be reduced to
> a nop because of ordering constraints. But perhaps we should then consider
> exporting an ordering primitive to user space.
That seems to be the big ordering issue. I had always assumed that
user space writes (by the same app to the same file) would be
committed in order. Is that really not the case?
Wouldn't most app programmers assume ordering? Wouldn't that always
possibly be an issue? Or do all the apps that require ordered writes
use fsync. There will surely be some who require ordering but don't
fsync. And without ordering, some apps won't be able to avoid fsync
without data safety issues.
It would seem that even ordering writes by default could not be a data
safety issue. But I guess performance would be affected negatively in
some cases. And behavior shouldn't be changed. So yes, it seems an
ordering primitive would be a good idea.
On May 26, 2011, at 3:01 AM, D. Jansen wrote:
> That seems to be the big ordering issue. I had always assumed that
> user space writes (by the same app to the same file) would be
> committed in order. Is that really not the case?
>
> Wouldn't most app programmers assume ordering? Wouldn't that always
> possibly be an issue? Or do all the apps that require ordered writes
> use fsync. There will surely be some who require ordering but don't
> fsync. And without ordering, some apps won't be able to avoid fsync
> without data safety issues.
I really don't like using the word "ordering" the way Dave used it,
because it's a file system lingo that *always* confuses civilians.
And "Insider" language like that isn't help for communication,
unless you're certain there are only experts in the room...
As Dave said earlier, "ordering" in the sense he was using it
refers strictly to ensuring consistency after a crash.
Now, there are two levels of consistency; one is file system
level consistency, and the other is application level
consistency. It used to be that desktop drives would
lie about forcing data to disk in response to a FLUSH
CACHE command, "yes sir, I promise the data is on
the disk, sir!", because it resulted in higher WINBENCH
scores. File systems engineers hated this, because
a primary tool we have for assuring that file systems
don't look like swiss cheese after a crash was completely
unreliable. Fortunately, those disks have largely
disappeared from the market place.
The suggestion of making fsync a no-op is essentially
asking for a knob that breaks application-level consistency
the same way those broken hard drives broke file system
consistency by making the FLUSH CACHE command
unreliable. Maybe improving battery lifetime is a more
honorable excuse than the purely mercenary goal of
selling more disk drives, but it can still break applications
after a crash.
Now, you may think that you're prepared by that. After all,
you're already prepared to say that you're willing to lose
the last 15 minutes of work or whatever, right?
Well, wrong. It's not so simple as that. If you're only
talking about simple, flat, human-readable text files,
maybe it would work that way. But what about complex,
binary databases? Like sqllite databases used by
Firefox and Chrome? Or MySQL databases? More
and more, sophisticated applications, even desktop
applications, are using these complex data stores,
and the libraries which update these complex data
stores rely on fsync() to prevent their database files
from looking like swiss cheese. If you crash while
fsync() has been disabled, the entire database file
could be completely trashed, which could be hours,
days, weeks, or months of work lost.
So the resistance that people like Dave have to your
proposal can be summed up by Confucious if you are
Chinese: ""Never impose on others what you would
not choose for yourself." Or if you are Jewish, the Rabbi
Hillel said: "That which is hateful to you, do not do to
your fellow. That is the whole Torah; the rest is the
explanation; go and learn." Or if you are a Muslim,
the Prophet Mohammed: "Hurt no one so that no one
may hurt you." Breaking fsync() is like hard drives that
break faith with file system authors by lying when they
say everything is safely written to stable storage. And
what are databases but complex file systems living inside
a single file?
-- Ted
On Thu, May 26, 2011 at 01:16:45PM +0200, D. Jansen wrote:
>
> exactly. so what now?
This needs to be solved at the application level. So what we need is
a coordinating daemon which sends a message to all applications
saying, "ok, here's an opportunity to flush data to the disk; if you
have dirty files, or databases that has data that needs to be flushed,
send it now", and then a little while later, it sends a message to all
applications, "ok, we're going to let the disk spin down now, hold
your horses and don't send any data to the disk".
If an application has too much data buffered in memory, and either
because (a) it's getting nervous that it doesn't want to lose a huge
amount of critical user data, or (b) it's about to exit, so it really
needs to save its state to disk, or (c) the buffered data is taking up
too much space, so it needs to flush its in-memory buffers, then the
application sends a message to the coordinating daemon, saying,
"please give me permission to write", and then waits for the "ok, you
can write to the disk" message.
That way, if you have multiple database-using applications active on
the system at the same time, they can coordinate their access to the
disk, and in a way such that when they stop writing, the data left in
the application-level database is in a coherent state.
The kernel can't do this by arbitrarily stopping writes from going to
the disk because the kernel doesn't know --- and can't possibly know
--- when it is a safe time to cut off write access to the disk. It
can try and guess, by stopping write access after an fsync(), but it
has to honor the fsync(). Consider what happens when an SMTP server
sends back a 200 OK message --- that gives permission for its opposite
number at the other end of the connection to delete the e-mail
message, since the local SMTP server has promised that the e-mail has
been saved to stable storage. It can't make that promise if fsync()
has been rigged to lie.
That's why at the end of the day, in order for application level data
coherency, the application has to be in control. If you want to save
power by coordinating access to the disk, that coordination has to
happen at the application level.
- Ted
Problem: any fsync call by any application spins up the hard disk any
time even in laptop_mode
On Thu, May 26, 2011 at 3:31 PM, Ted Ts'o <[email protected]> wrote:
> On Thu, May 26, 2011 at 01:16:45PM +0200, D. Jansen wrote:
>> exactly. so what now?
> This needs to be solved at the application level. So what we need is
> a coordinating daemon which sends a message to all applications
> saying, "ok, here's an opportunity to flush data to the disk; if you
> have dirty files, or databases that has data that needs to be flushed,
> send it now", and then a little while later, it sends a message to all
> applications, "ok, we're going to let the disk spin down now, hold
> your horses and don't send any data to the disk".
I'm not sure this approach is straight forward.
Because though there is no possibility to destroy data that is on disk
due to non FIFO flushing of application writes queued in the kernel,
which seems to be the main kernel level problem, yet new problems come
up.
Now there is
1) special support needed on the application side.
2) need for new out-of-kernel buffers.
3) need for inter-application write alignment nightmares. This sort of
structure could cause very uncomfortable bugs that prevent writes from
happening at all in cases that were not foreseen at all.
4) need for resources wasted through yet another daemon.
5) If the _application_, but not the kernel crashes, the data is safe.
In my experience this is the much more likely case than that the mail
server on my netbook optimized for battery time receives an email in
laptop mode, sends the other server "200" and then before the next
commit window my battery slips out and it's all gone.
6) _If_ there would be a special daemon I think it would make sense
for that daemon to provide all the infrastructure, buffering and
manage the fsync calls itself. This shouldn't and wouldn't be
replicated by every single application.
I think the alternative of ensuring the application writes are
committed in order would make more sense:
e..g a _user space library_ disables fsync etc. in laptop_mode if the
user chooses to do so and kernel support for forced FIFO ordering or
writes.
This would fix 1) 2) 3) 4) 5) 6).
Cheers,
Dennis
ps.
So you've re-thought this "All that is necessary is a kernel patch to
allow laptop_mode to disable fsync() calls(...)"
(http://tytso.livejournal.com/2009/03/15/). That post had inspired my
patch.
pps.
I know that an email getting lost is everyone's nightmare. But these
things do happen, anyway. It's not the end of the world if one email
is lost. (There's still a copy in the sent box. People will probably
notice, eventually. I recently found out an email properly sent and
filed in my gmail sent box never arrived for several of the people
without any error reports. If people need to be 100% sure an email
arrived they should ask for a confirmation and/or use a different
medium.) This just makes it much more likely.
On Thu, May 26, 2011 at 06:05:43PM +0200, D. Jansen wrote:
> Problem: any fsync call by any application spins up the hard disk any
> time even in laptop_mode
What you call a problem, I call a feature. If an application doesn't
participate in the write aggregation protocol, the worst that happens
is that you waste battery power. This I consider as a lesser evil
than data loss.
Similarly, if an application really _needs_ to write disk, and it
can't contact the coordinating daemon, or the coordinating daemon
doesn't respond in a reasonable amount of time, the application should
feel free to write the data to disk and fsync(). This might waste a
bit of power, but power is cheaper than lost data.
> Because though there is no possibility to destroy data that is on disk
> due to non FIFO flushing of application writes queued in the kernel,
> which seems to be the main kernel level problem, yet new problems come
> up.
I'm not sure what you're talking about here. Buffered data can always
be reordered in terms of when it is written to disk. This is
considered good, and normal. If you want to guarantee that
application writes are pushed out to disk, then either (a) use
O_DIRECT, or (b) use fsync(). Those are your two options.
If we didn't (for example) reorder writes to avoid the hard disk head
from seeking all over the disk, that would actually cause more power
to be consumed!
> Now there is
> 1) special support needed on the application side.
Yep, because this is fundamentally an application-level problem, and
the kernel doesn't have enough semantic information to solve the
database coherency problem.
> 2) need for new out-of-kernel buffers.
Yes. So?
> 3) need for inter-application write alignment nightmares. This sort of
> structure could cause very uncomfortable bugs that prevent writes from
> happening at all in cases that were not foreseen at all.
Huh? I think you are talking about order that buffered writes happen,
and there's no problem here. It's a feature that they can be
reordered. See above.
> 4) need for resources wasted through yet another daemon.
A daemon doesn't have to take up much space. If it is linked with all
of the GNOME libraries in the world, yeah, there'll be a problem, but
there's no reason that this daemon should take more than, say a few
tens of kilobytes at most.
> 5) If the _application_, but not the kernel crashes, the data is safe.
> In my experience this is the much more likely case than that the mail
> server on my netbook optimized for battery time receives an email in
> laptop mode, sends the other server "200" and then before the next
> commit window my battery slips out and it's all gone.
Huh? What's the problem that you're worried about here.
> I think the alternative of ensuring the application writes are
> committed in order would make more sense:
> e..g a _user space library_ disables fsync etc. in laptop_mode if the
> user chooses to do so and kernel support for forced FIFO ordering or
> writes.
> This would fix 1) 2) 3) 4) 5) 6).
And if you do this to a mysql daemon, or to a firefox or chrome
process which uses sqllite, and you crash at a wrong time, the entire
database could be scrambled. You can't fix this with your solution,
because you want to make fsync() lie to the database code. And so all
of the extra work (and power) consumed by the database code to try to
make its database writes be safe, will be compromised by making
fsync() unreliable.
> So you've re-thought this "All that is necessary is a kernel patch to
> allow laptop_mode to disable fsync() calls(...)"
> (http://tytso.livejournal.com/2009/03/15/). That post had inspired my
> patch.
I was thinking about things only from a file system perspective. The
problem is that more and more people are running databases or other
binary files which are updated in place on their laptops, and from a
more holistic perspective, we have to worry about making sure that
application-level databases are coherent in the face of a system
crash. (For example, you drop your mobile phone, or your tablet, or
your laptop, and the battery slips out.)
- Ted
On Thu, May 26, 2011 at 09:01:49AM +0200, D. Jansen wrote:
> Exactly. 0.5 Watts is a lot of energy on a modern system. And on some
> systems with proper link power management in sata and pcie it will be
> even higher (~1 Watt I think). Think not only of the work that will be
> lost if the system crashes (unlikely), but also of the work that could
> never be done because the battery was gone (a certainty).
I don't think link power management is relevant. The hysteresis there is
sufficiently low that it's only going to make a difference during the
point where you're actually throwing data across the link.
--
Matthew Garrett | [email protected]
On Thu, May 26, 2011 at 6:21 PM, Ted Ts'o <[email protected]> wrote:
> On Thu, May 26, 2011 at 06:05:43PM +0200, D. Jansen wrote:
>> Problem: any fsync call by any application spins up the hard disk any
>> time even in laptop_mode
>
> What you call a problem, I call a feature.
Problem: any fsync call by any application spins up the hard disk any
time even in laptop_mode and there's nothing the user can do about it
in user space - without risking that the application corrupts existing data if
the kernel decides to commit the queued writes in non-FIFO order OR
modifying every single application itself.
>> Because though there is no possibility to destroy data that is on disk
>> due to non FIFO flushing of application writes queued in the kernel,
>> which seems to be the main kernel level problem, yet new problems come
>> up.
>
> I'm not sure what you're talking about here. Buffered data can always
> be reordered in terms of when it is written to disk. This is
> considered good, and normal. If you want to guarantee that
> application writes are pushed out to disk, then either (a) use
> O_DIRECT, or (b) use fsync(). Those are your two options.
That reordering is exactly what I'm talking about. It wasn't my idea.
But if I understood it correctly, it's possible that the kernel
commits writes of an application, _to one and the same file_, in a
non-FIFO order, if the application does not fsync. And this _afaiu_
could result in the loss not only of new data, but complete corruption
of previously existing data in laptop mode without fsync.
But you're the expert. Is that really the case? If so, could it be
avoided without the daemon and application patching?
> If we didn't (for example) reorder writes to avoid the hard disk head
> from seeking all over the disk, that would actually cause more power
> to be consumed!
Yes, probably. But I doubt if that happens only once in a commit
window in laptop mode that the effect would destroy the gains. Also it
is not always necessary. Only writes to one file should be committed
in order. They could even be merged to one write - if they aren't
already: It seems the ordering is only necessary when an fsync occurs.
1) DDD_ (write D at 0)
2) _HHH (write H at 1) (fsync)
3) DHHH (result/merged write, in order)
As long as we don't end up with:
3) DDDH (out of order write, corrupt)
>> Now there is
(in a special write queue and coordination daemon)
>> 1) special support needed on the application side.
>
> Yep, because this is fundamentally an application-level problem, and
> the kernel doesn't have enough semantic information to solve the
> database coherency problem.
Well if we know that fsyncs mean the application needs the data to be
committed in order, couldn't we watch out for fsync calls and then (in
laptop mode when this feature specially requested by the user) switch
that application to fifo per file writes? (Disregarding the write
performance in that case.) Or we let the userspace eatmydata library
detect the same fsync and use a kernel api to switch that write to
fifo instead of fsyncing. A fifo write call might actually be useful
to other applications and scenarios as well. (trojan horse!)
Or the last write before the fsync is committed last. If reordering is
otherwise possible, this should avoid corruption and decrease
performance less. (Though we're not talking about writing hundreds of
MBs in laptop mode in my average use case scenario of office
applications and maybe a browser running.)
>
>> 2) need for new out-of-kernel buffers.
>
> Yes. So?
Shouldn't we try to avoid replicating existing infrastructure when possible?
>
>> 3) need for inter-application write alignment nightmares. This sort of
>> structure could cause very uncomfortable bugs that prevent writes from
>> happening at all in cases that were not foreseen at all.
>
> Huh? I think you are talking about order that buffered writes happen,
> and there's no problem here. It's a feature that they can be
> reordered. See above.
No, what I meant is that if there is a bug at any step of the
coordination between the applications and the daemon: in the daemon,
the software, their communication connection, etc., writes may not
occur and we may lose data without need.
>> 5) If the _application_, but not the kernel crashes, the data is safe.
>> In my experience this is the much more likely case than that the mail
>> server on my netbook optimized for battery time receives an email in
>> laptop mode, sends the other server "200" and then before the next
>> commit window my battery slips out and it's all gone.
>
> Huh? What's the problem that you're worried about here.
Your scenario sounds like this:
daemon announced when to flush data
until then application buffers data in it's user space.
This means if you save a file and the application crashes, e.g. segfaults
and is killed, the data is still in its queue and thus lost.
Without the daemon, the data would be in kernel space already and thus
safe from application crashes.
In my experience the kernel is very stable, applications are much less so.
And I really don't see this entering many applications. They would
probably say this is the task of the kernel itself or some other piece
of layer in between, but not the task of every single app developer to
reinvent write caching, coordination with the laptop writes daemon
etc. In the end we might have one or two special "write in laptop
mode" apps and as soon as I start a browser or any sqlite based app,
the problem is back.
>> I think the alternative of ensuring the application writes are
>> committed in order would make more sense:
>> e..g a _user space library_ disables fsync etc. in laptop_mode if the
>> user chooses to do so and kernel support for forced FIFO ordering or
>> writes.
>> This would fix 1) 2) 3) 4) 5) 6).
>
> And if you do this to a mysql daemon, or to a firefox or chrome
> process which uses sqllite, and you crash at a wrong time, the entire
> database could be scrambled.
Define crash at the wrong time. Because there is always a wrong time,
whether with laptop mode or without, with fsync or without.
> You can't fix this with your solution, because you want to make fsync()
> lie to the database code. And so all
> of the extra work (and power) consumed by the database code to try to
> make its database writes be safe, will be compromised by making
> fsync() unreliable.
Yes, I would like to have the liberty of extending the decrease of
safety of new data in favor of the choice of creating more new data
(due to longer run time) when in laptop mode.
I still want and use that safety, just not when I'm in laptop mode.
>
>> So you've re-thought this "All that is necessary is a kernel patch to
>> allow laptop_mode to disable fsync() calls(...)"
>> (http://tytso.livejournal.com/2009/03/15/). That post had inspired my
>> patch.
>
> I was thinking about things only from a file system perspective. The
> problem is that more and more people are running databases or other
> binary files which are updated in place on their laptops, and from a
> more holistic perspective, we have to worry about making sure that
> application-level databases are coherent in the face of a system
> crash. (For example, you drop your mobile phone, or your tablet, or
> your laptop, and the battery slips out.)
Exactly. Great example! Again, I very much agree.("Even") I don't want
to end up with
corrupt data. But I accept old data. Is there really no way to get there without
rewriting each and every application's fsync code?
Thanks for your insights!
On Thu, May 26, 2011 at 9:31 PM, Matthew Garrett <[email protected]> wrote:
> On Thu, May 26, 2011 at 09:01:49AM +0200, D. Jansen wrote:
>
>> Exactly. 0.5 Watts is a lot of energy on a modern system. And on some
>> systems with proper link power management in sata and pcie it will be
>> even higher (~1 Watt I think). Think not only of the work that will be
>> lost if the system crashes (unlikely), but also of the work that could
>> never be done because the battery was gone (a certainty).
>
> I don't think link power management is relevant. The hysteresis there is
> sufficiently low that it's only going to make a difference during the
> point where you're actually throwing data across the link.
>
Ok, thanks! I'm not sure how much power could be saved in that case on
different system.
All I know is that it's about 0.5 Watts on my system. All I want is to
help people to be able to take advantage of that extra bit to have
extra battery (in my case 10-20%) for last bit of <insert activity>,
in my case taking notes.
On May 27, 2011, at 3:12 AM, D. Jansen wrote:
>
> That reordering is exactly what I'm talking about. It wasn't my idea.
> But if I understood it correctly, it's possible that the kernel
> commits writes of an application, _to one and the same file_, in a
> non-FIFO order, if the application does not fsync. And this _afaiu_
> could result in the loss not only of new data, but complete corruption
> of previously existing data in laptop mode without fsync.
No, you're not understanding the problem. All layers of the storage
stack -- including the hard drive -- is allowed to reorder writes. So
even if the kernel sends data to the disk in the exact same order that
the application wrote it, it could still get written in a different order,
because the hard drive itself can reorder writes. This is necessary
for performance; if you didn't have this, the storage stack would be
dog slow, and would consume even more power.
So at least level, the only thing you can count upon is that if you want
to make sure everything is flushed to stable store, you need to send
an fsync() command at the application to file system level, or a barrier
or flush command at the OS to hard drive level.
So what databases do is the first write the changes they intend to
make to an intent log. Then they send an fsync() command; then
they write a commit block to the intent log; then they send another
fsync() command; and only then now that the transaction has been
committed to the commit log, do they start updating the table files.
(This is a highly simplified model, but it's good enough for this
discussion.)
Ordering doesn't matter, because nothing, including the hard drive,
guarantees ordering. What does matter is that the fsync() commands
act like barriers; writes before the fsync() command are guaranteed
to be written to the disk, and survive a reboot, before any writes after
the fsync() are processed. See?
This is why getting fsync() right is so critical; things are defined to work
this way, and programs like mysql and sqllite depend on things working
this way. You are proposing to break this.
> (Though we're not talking about writing hundreds of
> MBs in laptop mode in my average use case scenario of office
> applications and maybe a browser running.)
Firefox, in order to make their "awesome bar" work, is responsible for
300+ MB's worth of writes per click; so for every three clicks, you've
written a gigabyte. Any other questions?
>
> No, what I meant is that if there is a bug at any step of the
> coordination between the applications and the daemon: in the daemon,
> the software, their communication connection, etc., writes may not
> occur and we may lose data without need.
But the application will know that, and at the end of the day, if
the coordination is wrong, the application can always ignore the
daemon, write the data and call fsync(). So if there is any failure,
it fails safe; worst case you just waste more battery.
> Your scenario sounds like this:
> daemon announced when to flush data
> until then application buffers data in it's user space.
>
> This means if you save a file and the application crashes, e.g. segfaults
> and is killed, the data is still in its queue and thus lost.
If the application crashes, it will always lose data. If the application thinks
its flaky, it can always ignore protocol and force a disk write; as I said,
that will just burn battery, which is preferable to losing data.
>
> Exactly. Great example! Again, I very much agree.("Even") I don't want
> to end up with
> corrupt data. But I accept old data. Is there really no way to get there without
> rewriting each and every application's fsync code?
If the application is using a binary database file format, then no, if you subvert
fsync(), you can risk losing the entire database. But even if you use 100's
of flat files, if you care about the relationship between the flat files as having
critical meaning, then you can end up corrupting data even if you use lots
of flat files.
If you are willing to rewrite the *entire* database to a completely new file each
time you want to write out some data, and only delete the old database
once the new database has been written out, then you're fine. If the file is too
big you can delay the time period between a complete writeout of the
database. But then if you drop your laptop and the battery slips out, you'll
lose more data. Life is full of tradeoffs.
If the only editor you use is vi, and the only web browser you use is lynx, then
life is much simpler. If you want more complexity, AND you want more safety,
then you'll have to pay for that in terms of more battery usage.
-- Ted
>
> Thanks for your insights!
On Fri, 27 May 2011, Theodore Tso wrote:
> If you are willing to rewrite the *entire* database to a completely new file each
> time you want to write out some data, and only delete the old database
> once the new database has been written out, then you're fine. If the file is too
> big you can delay the time period between a complete writeout of the
> database. But then if you drop your laptop and the battery slips out, you'll
> lose more data. Life is full of tradeoffs.
actually, even in this case don't you need a barrier or fsync to make sure
the new file is completely written before the old file goes away?
David Lang
On Fri, May 27, 2011 at 4:17 PM, Theodore Tso <[email protected]> wrote:
> On May 27, 2011, at 3:12 AM, D. Jansen wrote:
>> That reordering is exactly what I'm talking about. It wasn't my idea.
>> But if I understood it correctly, it's possible that the kernel
>> commits writes of an application, _to one and the same file_, in a
>> non-FIFO order, if the application does not fsync. And this _afaiu_
>> could result in the loss not only of new data, but complete corruption
>> of previously existing data in laptop mode without fsync.
>
> No, you're not understanding the problem. All layers of the storage
> stack -- including the hard drive -- is allowed to reorder writes. So
> even if the kernel sends data to the disk in the exact same order that
> the application wrote it, it could still get written in a different order,
> because the hard drive itself can reorder writes. This is necessary
> for performance; if you didn't have this, the storage stack would be
> dog slow, and would consume even more power.
>
> So at least level, the only thing you can count upon is that if you want
> to make sure everything is flushed to stable store, you need to send
> an fsync() command at the application to file system level, or a barrier
> or flush command at the OS to hard drive level.
(...)
> Ordering doesn't matter, because nothing, including the hard drive,
> guarantees ordering. What does matter is that the fsync() commands
> act like barriers; writes before the fsync() command are guaranteed
> to be written to the disk, and survive a reboot, before any writes after
> the fsync() are processed. See?
Ok, thanks a lot! I understand a lot better now!
So we can't live without the fsyncs.
So what if we would queue the fsyncs along with the writes - we would
just fsync later instead of immediately, in between the writes as they
came in. Then by design previous data could not be corrupted, right?
We would do exactly the same thing, just later.
It'd be kind of a disk write time distortion field.
Thanks again for your feedback!
On Sun, 29 May 2011, D. Jansen wrote:
> On Fri, May 27, 2011 at 4:17 PM, Theodore Tso <[email protected]> wrote:
>> On May 27, 2011, at 3:12 AM, D. Jansen wrote:
>>> That reordering is exactly what I'm talking about. It wasn't my idea.
>>> But if I understood it correctly, it's possible that the kernel
>>> commits writes of an application, _to one and the same file_, in a
>>> non-FIFO order, if the application does not fsync. And this _afaiu_
>>> could result in the loss not only of new data, but complete corruption
>>> of previously existing data in laptop mode without fsync.
>>
>> No, you're not understanding the problem. All layers of the storage
>> stack -- including the hard drive -- is allowed to reorder writes. So
>> even if the kernel sends data to the disk in the exact same order that
>> the application wrote it, it could still get written in a different order,
>> because the hard drive itself can reorder writes. This is necessary
>> for performance; if you didn't have this, the storage stack would be
>> dog slow, and would consume even more power.
>>
>> So at least level, the only thing you can count upon is that if you want
>> to make sure everything is flushed to stable store, you need to send
>> an fsync() command at the application to file system level, or a barrier
>> or flush command at the OS to hard drive level.
> (...)
>> Ordering doesn't matter, because nothing, including the hard drive,
>> guarantees ordering. What does matter is that the fsync() commands
>> act like barriers; writes before the fsync() command are guaranteed
>> to be written to the disk, and survive a reboot, before any writes after
>> the fsync() are processed. See?
>
> Ok, thanks a lot! I understand a lot better now!
> So we can't live without the fsyncs.
>
> So what if we would queue the fsyncs along with the writes - we would
> just fsync later instead of immediately, in between the writes as they
> came in. Then by design previous data could not be corrupted, right?
> We would do exactly the same thing, just later.
> It'd be kind of a disk write time distortion field.
the problem is that the spec for fsync says that your program stops until
fsync finishes. If you don't do that then you will corrupt and loose data.
so if you delay fsync you will have your application (or desktop manager)
freeze until the fsync completes.
if what you are wanting is the ability to say 'these things must be
written before these other things to keep them from being corrupted, but I
don't care when they get written (or if they get lost in a crash)' then
what you want isn't fsync, it's a barrier.
David Lang
Am Montag, 30. Mai 2011, 03:53:18 schrieb [email protected]:
> > So what if we would queue the fsyncs along with the writes - we would
> > just fsync later instead of immediately, in between the writes as they
> > came in. Then by design previous data could not be corrupted, right?
> > We would do exactly the same thing, just later.
> > It'd be kind of a disk write time distortion field.
>
> the problem is that the spec for fsync says that your program stops until
> fsync finishes. If you don't do that then you will corrupt and loose data.
It is important to be precise.
You will loose data, but you will not get corruption.
> so if you delay fsync you will have your application (or desktop manager)
> freeze until the fsync completes.
>
> if what you are wanting is the ability to say 'these things must be
> written before these other things to keep them from being corrupted, but I
> don't care when they get written (or if they get lost in a crash)' then
> what you want isn't fsync, it's a barrier.
Yes, but where is the problem?
Regards
Oliver
--
- - -
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend?rffer, HRB 16746 (AG N?rnberg)
Maxfeldstra?e 5
90409 N?rnberg
Germany
- - -
On Mon, 30 May 2011 09:13:27 +0200, Oliver Neukum said:
> Am Montag, 30. Mai 2011, 03:53:18 schrieb [email protected]:
> > if what you are wanting is the ability to say 'these things must be
> > written before these other things to keep them from being corrupted, but I
> > don't care when they get written (or if they get lost in a crash)' then
> > what you want isn't fsync, it's a barrier.
>
> Yes, but where is the problem?
The problem is that *from userspace*, the closest approximation to a barrier
is fsync().
On Mon, May 30, 2011 at 3:53 AM, <[email protected]> wrote:
> On Sun, 29 May 2011, D. Jansen wrote:
>> On Fri, May 27, 2011 at 4:17 PM, Theodore Tso <[email protected]> wrote:
>>> On May 27, 2011, at 3:12 AM, D. Jansen wrote:
>>>> That reordering is exactly what I'm talking about. It wasn't my idea.
>>>> But if I understood it correctly, it's possible that the kernel
>>>> commits writes of an application, _to one and the same file_, in a
>>>> non-FIFO order, if the application does not fsync. And this _afaiu_
>>>> could result in the loss not only of new data, but complete corruption
>>>> of previously existing data in laptop mode without fsync.
>>>
>>> No, you're not understanding the problem. All layers of the storage
>>> stack -- including the hard drive -- is allowed to reorder writes. So
>>> even if the kernel sends data to the disk in the exact same order that
>>> the application wrote it, it could still get written in a different
>>> order,
>>> because the hard drive itself can reorder writes. This is necessary
>>> for performance; if you didn't have this, the storage stack would be
>>> dog slow, and would consume even more power.
>>>
>>> So at least level, the only thing you can count upon is that if you want
>>> to make sure everything is flushed to stable store, you need to send
>>> an fsync() command at the application to file system level, or a barrier
>>> or flush command at the OS to hard drive level.
>> (...)
>>> Ordering doesn't matter, because nothing, including the hard drive,
>>> guarantees ordering. What does matter is that the fsync() commands
>>> act like barriers; writes before the fsync() command are guaranteed
>>> to be written to the disk, and survive a reboot, before any writes after
>>> the fsync() are processed. See?
>>
>> Ok, thanks a lot! I understand a lot better now!
>> So we can't live without the fsyncs.
>>
>> So what if we would queue the fsyncs along with the writes - we would
>> just fsync later instead of immediately, in between the writes as they
>> came in. Then by design previous data could not be corrupted, right?
>> We would do exactly the same thing, just later.
>> It'd be kind of a disk write time distortion field.
>
> the problem is that the spec for fsync says that your program stops until
> fsync finishes. If you don't do that then you will corrupt and loose data.
>
> so if you delay fsync you will have your application (or desktop manager)
> freeze until the fsync completes.
So that would not be an option. Freezing until the end of the write
window is not what we want.
Neither is ignoring the fsync because that could corrupt data, esp. in
databases like sqlite.
>
> if what you are wanting is the ability to say 'these things must be written
> before these other things to keep them from being corrupted, but I don't
> care when they get written (or if they get lost in a crash)' then what you
> want isn't fsync, it's a barrier.
That sounds great!
So an fsync call in laptop mode could be interpreted as a barrier
and we would be reasonably save from corrupting old existing data?
On Mon, 30 May 2011, D. Jansen wrote:
> On Mon, May 30, 2011 at 3:53 AM, <[email protected]> wrote:
>> On Sun, 29 May 2011, D. Jansen wrote:
>>> On Fri, May 27, 2011 at 4:17 PM, Theodore Tso <[email protected]> wrote:
>>>> On May 27, 2011, at 3:12 AM, D. Jansen wrote:
>>>>> That reordering is exactly what I'm talking about. It wasn't my idea.
>>>>> But if I understood it correctly, it's possible that the kernel
>>>>> commits writes of an application, _to one and the same file_, in a
>>>>> non-FIFO order, if the application does not fsync. And this _afaiu_
>>>>> could result in the loss not only of new data, but complete corruption
>>>>> of previously existing data in laptop mode without fsync.
>>>>
>>>> No, you're not understanding the problem. All layers of the storage
>>>> stack -- including the hard drive -- is allowed to reorder writes. So
>>>> even if the kernel sends data to the disk in the exact same order that
>>>> the application wrote it, it could still get written in a different
>>>> order,
>>>> because the hard drive itself can reorder writes. This is necessary
>>>> for performance; if you didn't have this, the storage stack would be
>>>> dog slow, and would consume even more power.
>>>>
>>>> So at least level, the only thing you can count upon is that if you want
>>>> to make sure everything is flushed to stable store, you need to send
>>>> an fsync() command at the application to file system level, or a barrier
>>>> or flush command at the OS to hard drive level.
>>> (...)
>>>> Ordering doesn't matter, because nothing, including the hard drive,
>>>> guarantees ordering. What does matter is that the fsync() commands
>>>> act like barriers; writes before the fsync() command are guaranteed
>>>> to be written to the disk, and survive a reboot, before any writes after
>>>> the fsync() are processed. See?
>>>
>>> Ok, thanks a lot! I understand a lot better now!
>>> So we can't live without the fsyncs.
>>>
>>> So what if we would queue the fsyncs along with the writes - we would
>>> just fsync later instead of immediately, in between the writes as they
>>> came in. Then by design previous data could not be corrupted, right?
>>> We would do exactly the same thing, just later.
>>> It'd be kind of a disk write time distortion field.
>>
>> the problem is that the spec for fsync says that your program stops until
>> fsync finishes. If you don't do that then you will corrupt and loose data.
>>
>> so if you delay fsync you will have your application (or desktop manager)
>> freeze until the fsync completes.
>
> So that would not be an option. Freezing until the end of the write
> window is not what we want.
> Neither is ignoring the fsync because that could corrupt data, esp. in
> databases like sqlite.
>>
>> if what you are wanting is the ability to say 'these things must be written
>> before these other things to keep them from being corrupted, but I don't
>> care when they get written (or if they get lost in a crash)' then what you
>> want isn't fsync, it's a barrier.
>
> That sounds great!
> So an fsync call in laptop mode could be interpreted as a barrier
> and we would be reasonably save from corrupting old existing data?
no, you cannot just change a fsync to a barrier, in some cases the data
absolutly needs to be saved, not just ordered (remember the example of a
mail server telling the other system that the data can be deleted after a
fsync returns)
David Lang
On Mon, 30 May 2011, [email protected] wrote:
> On Mon, 30 May 2011 09:13:27 +0200, Oliver Neukum said:
>> Am Montag, 30. Mai 2011, 03:53:18 schrieb [email protected]:
>
>>> if what you are wanting is the ability to say 'these things must be
>>> written before these other things to keep them from being corrupted, but I
>>> don't care when they get written (or if they get lost in a crash)' then
>>> what you want isn't fsync, it's a barrier.
>>
>> Yes, but where is the problem?
>
> The problem is that *from userspace*, the closest approximation to a barrier
> is fsync().
good point, there really should be some way of doing a barrier from
userspace.
David Lang
On Mon, May 30, 2011 at 8:02 PM, <[email protected]> wrote:
> On Mon, 30 May 2011, D. Jansen wrote:
>> On Mon, May 30, 2011 at 3:53 AM, <[email protected]> wrote:
>>> On Sun, 29 May 2011, D. Jansen wrote:
>>>> On Fri, May 27, 2011 at 4:17 PM, Theodore Tso <[email protected]> wrote:
>>>>> On May 27, 2011, at 3:12 AM, D. Jansen wrote:
>>>>>> That reordering is exactly what I'm talking about. It wasn't my idea.
>>>>>> But if I understood it correctly, it's possible that the kernel
>>>>>> commits writes of an application, _to one and the same file_, in a
>>>>>> non-FIFO order, if the application does not fsync. And this _afaiu_
>>>>>> could result in the loss not only of new data, but complete corruption
>>>>>> of previously existing data in laptop mode without fsync.
>>>>>
>>>>> No, you're not understanding the problem. All layers of the storage
>>>>> stack -- including the hard drive -- is allowed to reorder writes. So
>>>>> even if the kernel sends data to the disk in the exact same order that
>>>>> the application wrote it, it could still get written in a different
>>>>> order,
>>>>> because the hard drive itself can reorder writes. This is necessary
>>>>> for performance; if you didn't have this, the storage stack would be
>>>>> dog slow, and would consume even more power.
>>>>>
>>>>> So at least level, the only thing you can count upon is that if you
>>>>> want
>>>>> to make sure everything is flushed to stable store, you need to send
>>>>> an fsync() command at the application to file system level, or a
>>>>> barrier
>>>>> or flush command at the OS to hard drive level.
>>>>
>>>> (...)
>>>>>
>>>>> Ordering doesn't matter, because nothing, including the hard drive,
>>>>> guarantees ordering. What does matter is that the fsync() commands
>>>>> act like barriers; writes before the fsync() command are guaranteed
>>>>> to be written to the disk, and survive a reboot, before any writes
>>>>> after
>>>>> the fsync() are processed. See?
>>>>
>>>> Ok, thanks a lot! I understand a lot better now!
>>>> So we can't live without the fsyncs.
>>>>
>>>> So what if we would queue the fsyncs along with the writes - we would
>>>> just fsync later instead of immediately, in between the writes as they
>>>> came in. Then by design previous data could not be corrupted, right?
>>>> We would do exactly the same thing, just later.
>>>> It'd be kind of a disk write time distortion field.
>>>
>>> the problem is that the spec for fsync says that your program stops until
>>> fsync finishes. If you don't do that then you will corrupt and loose
>>> data.
>>>
>>> so if you delay fsync you will have your application (or desktop manager)
>>> freeze until the fsync completes.
>>
>> So that would not be an option. Freezing until the end of the write
>> window is not what we want.
>> Neither is ignoring the fsync because that could corrupt data, esp. in
>> databases like sqlite.
>>>
>>> if what you are wanting is the ability to say 'these things must be
>>> written
>>> before these other things to keep them from being corrupted, but I don't
>>> care when they get written (or if they get lost in a crash)' then what
>>> you
>>> want isn't fsync, it's a barrier.
>>
>> That sounds great!
>> So an fsync call in laptop mode could be interpreted as a barrier
>> and we would be reasonably save from corrupting old existing data?
>
> no, you cannot just change a fsync to a barrier, in some cases the data
> absolutly needs to be saved, not just ordered (remember the example of a
> mail server telling the other system that the data can be deleted after a
> fsync returns)
I'm not really sure I why shouldn't have that choice as a user. Just
because someone else could be running a mailserver on his system and
configure it in a way that it doesn't behave as it should?
If he really wants to do that there's really nothing we can do to stop
him. I'm sure there are other ways existing kernel options can be used
to make software behave different than it should. Are we going to
remove them all now?
The big problem is that so far only fsync existed and lots of software
seemingly abuses it as an expensive write barrier. And it would really
be lovely to have the choice to stop that on an opt-in basis in laptop
mode.
On Mon, 30 May 2011 20:28:24 +0200, "D. Jansen" said:
> I'm not really sure I why shouldn't have that choice as a user. Just
> because someone else could be running a mailserver on his system and
> configure it in a way that it doesn't behave as it should?
It cuts both ways. If a piece of software really wants to be sure that the
fsync() semantics it's expecting are actually adhered to and refuses to run
otherwise, how does it tell that you're lying to it?
> The big problem is that so far only fsync existed and lots of software
> seemingly abuses it as an expensive write barrier. And it would really
> be lovely to have the choice to stop that on an opt-in basis in laptop
> mode.
It's not "seemingly abuses it". That's the existing userspace API for
inserting a barrier. The problem is that as defined, it will wait for the
writeback to actually finish - which is actually as good as you can get without
getting into the async I/O support. If there was a "insert barrier and return"
API, how would it report back that the barrier had failed with EIO after it had
already returned to userspace?
On Mon, 30 May 2011, D. Jansen wrote:
> On Mon, May 30, 2011 at 8:02 PM, <[email protected]> wrote:
>>
>> no, you cannot just change a fsync to a barrier, in some cases the data
>> absolutly needs to be saved, not just ordered (remember the example of a
>> mail server telling the other system that the data can be deleted after a
>> fsync returns)
>
> I'm not really sure I why shouldn't have that choice as a user. Just
> because someone else could be running a mailserver on his system and
> configure it in a way that it doesn't behave as it should?
> If he really wants to do that there's really nothing we can do to stop
> him. I'm sure there are other ways existing kernel options can be used
> to make software behave different than it should. Are we going to
> remove them all now?
>
> The big problem is that so far only fsync existed and lots of software
> seemingly abuses it as an expensive write barrier. And it would really
> be lovely to have the choice to stop that on an opt-in basis in laptop
> mode.
is the benifit of not spinning up the disk really worth the risk of
loosing data?
and should this really be a global across-the-board option?
the problem is that most users don't know what their system is running, or
what effect disaling fsync would have. those that do can probably use
LD_PRELOAD to override fsync calls.
it doesn't take running a mail server, even a mail client will have the
same risk. If you use POP for mail (a very common option) then you
download messages and tell the server to delete them. if you do not really
save them (one fsync after they are all saved), then you can loose
everything that you downloaded.
David Lang
On Mon, May 30, 2011 at 8:45 PM, <[email protected]> wrote:
> On Mon, 30 May 2011, D. Jansen wrote:
>
>> On Mon, May 30, 2011 at 8:02 PM, <[email protected]> wrote:
>>>
>>> no, you cannot just change a fsync to a barrier, in some cases the data
>>> absolutly needs to be saved, not just ordered (remember the example of a
>>> mail server telling the other system that the data can be deleted after a
>>> fsync returns)
>>
>> I'm not really sure I why shouldn't have that choice as a user. Just
>> because someone else could be running a mailserver on his system and
>> configure it in a way that it doesn't behave as it should?
>> If he really wants to do that there's really nothing we can do to stop
>> him. I'm sure there are other ways existing kernel options can be used
>> to make software behave different than it should. Are we going to
>> remove them all now?
>>
>> The big problem is that so far only fsync existed and lots of software
>> seemingly abuses it as an expensive write barrier. And it would really
>> be lovely to have the choice to stop that on an opt-in basis in laptop
>> mode.
>
> is the benifit of not spinning up the disk really worth the risk of loosing
> data?
Is that a choice I can make myself? Is that a choice I make with
laptop mode? Or is that a choice I may only make if I'm willing to
modify and compile my own kernel?
>
> and should this really be a global across-the-board option?
The point is option imo.
>
> the problem is that most users don't know what their system is running, or
> what effect disaling fsync would have. those that do can probably use
> LD_PRELOAD to override fsync calls.
As we found out, they can't. But if we export barrier, I hope a
library could wrap fsyncs into barriers. Is that the case?
>
> it doesn't take running a mail server, even a mail client will have the same
> risk. If you use POP for mail (a very common option) then you download
> messages and tell the server to delete them. if you do not really save them
> (one fsync after they are all saved), then you can loose everything that you
> downloaded.
Yes, I know. It's the same argument again and again. I understand not
everybody wants this. But some do. Some prefer working 10-20% longer
on battery (certainty) instead of possibly losing 5 % data
(possibility) or losing all your data (possibility if you use laptop
mode and the hard disk wakes up again and again and eventually wears
out). That's why there's laptop mode. And this would play into laptop
mode and prevent the hard disk from breaking down prematurely and
saving battery.
Please try it out before you say that it's not necessary in your use
case. It's useful and necessary in mine and other mobile use cases.
Please don't assume that all distributions activate this by default
and mailservers world wide use all their emails because they all crash
with laptop mode active.
I really don't care if it's in the kernel or elsewhere. But I care
that old data is not corrupted. And it seems this is not (yet)
possible with user space, e.g. a library in ld_preload. So I'm happy
for any hints how to accomplish this.
On Mon, May 30, 2011 at 8:43 PM, <[email protected]> wrote:
> On Mon, 30 May 2011 20:28:24 +0200, "D. Jansen" said:
>
>> I'm not really sure I why shouldn't have that choice as a user. Just
>> because someone else could be running a mailserver on his system and
>> configure it in a way that it doesn't behave as it should?
>
> It cuts both ways. If a piece of software really wants to be sure that the
> fsync() semantics it's expecting are actually adhered to and refuses to run
> otherwise, how does it tell that you're lying to it?
Well if we allow to use a library to wrap fsync into write barriers
that doesn't seem to matter because software doesn't need to know what
happens and isn't even supposed to be able to work around it.
If this was to go into the kernel as an option, the situation was
different. But really, if you look at this discussion and the feedback
so far I don't think that's an option we need to discuss at this
point...
>
>> The big problem is that so far only fsync existed and lots of software
>> seemingly abuses it as an expensive write barrier. And it would really
>> be lovely to have the choice to stop that on an opt-in basis in laptop
>> mode.
>
> It's not "seemingly abuses it". That's the existing userspace API for
> inserting a barrier. The problem is that as defined, it will wait for the
> writeback to actually finish - which is actually as good as you can get without
> getting into the async I/O support. If there was a "insert barrier and return"
> API, how would it report back that the barrier had failed with EIO after it had
> already returned to userspace?
I guess with another call some time later, to check for the success of
the previous call?
On Mon, 30 May 2011, D. Jansen wrote:
> On Mon, May 30, 2011 at 8:45 PM, <[email protected]> wrote:
>>
>>
>> the problem is that most users don't know what their system is running, or
>> what effect disaling fsync would have. those that do can probably use
>> LD_PRELOAD to override fsync calls.
>
> As we found out, they can't. But if we export barrier, I hope a
> library could wrap fsyncs into barriers. Is that the case?
a library can wrap fsync into anything.
>>
>> it doesn't take running a mail server, even a mail client will have the same
>> risk. If you use POP for mail (a very common option) then you download
>> messages and tell the server to delete them. if you do not really save them
>> (one fsync after they are all saved), then you can loose everything that you
>> downloaded.
>
> Yes, I know. It's the same argument again and again. I understand not
> everybody wants this. But some do. Some prefer working 10-20% longer
> on battery (certainty) instead of possibly losing 5 % data
> (possibility) or losing all your data (possibility if you use laptop
> mode and the hard disk wakes up again and again and eventually wears
> out).
those are some powerful numbers you are throwing around, can you back them
up?
do you really have so many fsync's going on that the disk spins up so much
that you would gain 10-20% battery life?
and what makes you think the extra spin-ups from fsyncs will cause your
hard drive to fail significantly earlier? (if you have a hard drive with a
limited number of spin-up cycles, you probably don't want to use laptop
mode at all)
why do you think it's a possibility of loosing only 5% of data?
David Lang
> That's why there's laptop mode. And this would play into laptop
> mode and prevent the hard disk from breaking down prematurely and
> saving battery.
>
> Please try it out before you say that it's not necessary in your use
> case. It's useful and necessary in mine and other mobile use cases.
> Please don't assume that all distributions activate this by default
> and mailservers world wide use all their emails because they all crash
> with laptop mode active.
>
> I really don't care if it's in the kernel or elsewhere. But I care
> that old data is not corrupted. And it seems this is not (yet)
> possible with user space, e.g. a library in ld_preload. So I'm happy
> for any hints how to accomplish this.
>
On Mon, May 30, 2011 at 10:53 PM, <[email protected]> wrote:
> On Mon, 30 May 2011, D. Jansen wrote:
>
>> On Mon, May 30, 2011 at 8:45 PM, <[email protected]> wrote:
>>>
>>>
>>> the problem is that most users don't know what their system is running,
>>> or
>>> what effect disaling fsync would have. those that do can probably use
>>> LD_PRELOAD to override fsync calls.
>>
>> As we found out, they can't. But if we export barrier, I hope a
>> library could wrap fsyncs into barriers. Is that the case?
>
> a library can wrap fsync into anything.
>
>>>
>>> it doesn't take running a mail server, even a mail client will have the
>>> same
>>> risk. If you use POP for mail (a very common option) then you download
>>> messages and tell the server to delete them. if you do not really save
>>> them
>>> (one fsync after they are all saved), then you can loose everything that
>>> you
>>> downloaded.
>>
>> Yes, I know. It's the same argument again and again. I understand not
>> everybody wants this. But some do. Some prefer working 10-20% longer
>> on battery (certainty) instead of possibly losing 5 % data
>> (possibility) or losing all your data (possibility if you use laptop
>> mode and the hard disk wakes up again and again and eventually wears
>> out).
>
> those are some powerful numbers you are throwing around, can you back them
> up?
Yes. My netbook uses 4.8 W with hard disk off. 0.5 W more with hard
disk on. A lot of the time, the display is off and the power
consumption drops to about 3.5 W. I don't know how I got to 20 % atm.
I can only do this if fsync is disabled. I tried before, but it would
go on for every write. It died, very prematurely.
>
> do you really have so many fsync's going on that the disk spins up so much
> that you would gain 10-20% battery life?
Yes. Every autosave in LibreOffice triggers one. And I want autosave,
but I want them in memory, not on disk.
>
> and what makes you think the extra spin-ups from fsyncs will cause your hard
> drive to fail significantly earlier? (if you have a hard drive with a
> limited number of spin-up cycles, you probably don't want to use laptop mode
> at all)
Experience, see above. Also, this is well described behavior. All hard
disks are only designed to last a certain number of head loads and
unloads. Spinning up and down even less.
>
> why do you think it's a possibility of loosing only 5% of data?
Well, it really depends on the way your configure laptop mode. But in
my case the laptop mode window is 20 minutes, the run time then is
about 10-12 hrs. I can actually lose less than 5 % of the data created
during the battery run time. It's a certain 1-1.5 hours (10-15 %) more
work or a possible 20 mins (3.3 % actually) loss of work. I think it's
a good deal.
On Mon, 30 May 2011, D. Jansen wrote:
> On Mon, May 30, 2011 at 8:45 PM, <[email protected]> wrote:
> > On Mon, 30 May 2011, D. Jansen wrote:
> >
> >> On Mon, May 30, 2011 at 8:02 PM, <[email protected]> wrote:
> >>>
> >>> no, you cannot just change a fsync to a barrier, in some cases the data
> >>> absolutly needs to be saved, not just ordered (remember the example of a
> >>> mail server telling the other system that the data can be deleted after a
> >>> fsync returns)
> >>
> >> I'm not really sure I why shouldn't have that choice as a user. Just
> >> because someone else could be running a mailserver on his system and
> >> configure it in a way that it doesn't behave as it should?
> >> If he really wants to do that there's really nothing we can do to stop
> >> him. I'm sure there are other ways existing kernel options can be used
> >> to make software behave different than it should. Are we going to
> >> remove them all now?
> >>
> >> The big problem is that so far only fsync existed and lots of software
> >> seemingly abuses it as an expensive write barrier. And it would really
> >> be lovely to have the choice to stop that on an opt-in basis in laptop
> >> mode.
> >
> > is the benifit of not spinning up the disk really worth the risk of loosing
> > data?
>
> Is that a choice I can make myself? Is that a choice I make with
> laptop mode? Or is that a choice I may only make if I'm willing to
> modify and compile my own kernel?
> >
> > and should this really be a global across-the-board option?
>
> The point is option imo.
> >
> > the problem is that most users don't know what their system is running, or
> > what effect disaling fsync would have. those that do can probably use
> > LD_PRELOAD to override fsync calls.
>
> As we found out, they can't. But if we export barrier, I hope a
> library could wrap fsyncs into barriers. Is that the case?
> >
> > it doesn't take running a mail server, even a mail client will have the same
> > risk. If you use POP for mail (a very common option) then you download
> > messages and tell the server to delete them. if you do not really save them
> > (one fsync after they are all saved), then you can loose everything that you
> > downloaded.
>
> Yes, I know. It's the same argument again and again. I understand not
> everybody wants this. But some do. Some prefer working 10-20% longer
> on battery (certainty) instead of possibly losing 5 % data
> (possibility) or losing all your data (possibility if you use laptop
> mode and the hard disk wakes up again and again and eventually wears
> out). That's why there's laptop mode. And this would play into laptop
> mode and prevent the hard disk from breaking down prematurely and
> saving battery.
>
But how can you say that you are only risking 5% data loss or only risking
your most recent data? You can't.
fsync provides guarantees and applications take advantage of that
guarantee and you have no way of knowing what breaking those guarantees
will mean for all applications in the system (not without a *lot* of
research work at least).
You may be willing to lose a few emails or a few recently modified plain
text files or office documents. But how would you feel if that multi
megabyte or gigabyte database that has been build up over months with your
work suddenly became completely unrecoverable because the application
maintaining it made some assumptions based on fsync's guarantees and those
guarantees were suddenly broken? My guess is you wouldn't be very happy.
Some applications may only need the ordering guarantee that fsync
provides, not the "data is safely on media" guarantee. So perhaps if a new
barrier syscall was provided that gave applications just the lighter
ordering guarantee, then over time those apps could transition to use it
instead of fsync. Apps that really do need the "data is safely on media"
guarantee would of course continue to call fsync, but the number of
overall fsync calls in the system would decrease over time - this would
help your battery life and we'd still have a working fsync for those apps
that really do need it.
How does that sound?
--
Jesper Juhl <[email protected]> http://www.chaosbits.net/
Don't top-post http://www.catb.org/jargon/html/T/top-post.html
Plain text mails only, please.
On Mon, 30 May 2011, D. Jansen wrote:
> On Mon, May 30, 2011 at 10:53 PM, <[email protected]> wrote:
>> On Mon, 30 May 2011, D. Jansen wrote:
>>
>>> On Mon, May 30, 2011 at 8:45 PM, <[email protected]> wrote:
>>>> it doesn't take running a mail server, even a mail client will have the
>>>> same
>>>> risk. If you use POP for mail (a very common option) then you download
>>>> messages and tell the server to delete them. if you do not really save
>>>> them
>>>> (one fsync after they are all saved), then you can loose everything that
>>>> you
>>>> downloaded.
>>>
>>> Yes, I know. It's the same argument again and again. I understand not
>>> everybody wants this. But some do. Some prefer working 10-20% longer
>>> on battery (certainty) instead of possibly losing 5 % data
>>> (possibility) or losing all your data (possibility if you use laptop
>>> mode and the hard disk wakes up again and again and eventually wears
>>> out).
>>
>> those are some powerful numbers you are throwing around, can you back them
>> up?
>
> Yes. My netbook uses 4.8 W with hard disk off. 0.5 W more with hard
> disk on. A lot of the time, the display is off and the power
> consumption drops to about 3.5 W. I don't know how I got to 20 % atm.
> I can only do this if fsync is disabled. I tried before, but it would
> go on for every write. It died, very prematurely.
what are you doing that's causing so many fsyncs?
>>
>> do you really have so many fsync's going on that the disk spins up so much
>> that you would gain 10-20% battery life?
>
> Yes. Every autosave in LibreOffice triggers one. And I want autosave,
> but I want them in memory, not on disk.
are you sure that it's doing a fsync, not just a normal write? in laptop
mode the drive will spin up occasionally for normal writes (to keep you
from loosing too much data). I would expect that autosave is infrequent
enough that (absent all other activity), each autosave will result in a
spin-up
>> and what makes you think the extra spin-ups from fsyncs will cause your hard
>> drive to fail significantly earlier? (if you have a hard drive with a
>> limited number of spin-up cycles, you probably don't want to use laptop mode
>> at all)
>
> Experience, see above. Also, this is well described behavior. All hard
> disks are only designed to last a certain number of head loads and
> unloads. Spinning up and down even less.
right, but the trade-off of laptop mode is that it will do _more_ spin ups
on the theory that it's better to spin up the drive once ina while than to
keep it running. If you are looking at total power consumption this is the
case, but if you are worried about load/unload cycles this is not the
case.
the question I was asking is how do you know that fsync is the cause of
these extra spin-ups? if you disable fsync, but still do the same number
of writes, willyou really have far fewer spin-up events? I don't think
that it would make nearly the difference ou think it is.
>> why do you think it's a possibility of loosing only 5% of data?
>
> Well, it really depends on the way your configure laptop mode. But in
> my case the laptop mode window is 20 minutes, the run time then is
> about 10-12 hrs. I can actually lose less than 5 % of the data created
> during the battery run time. It's a certain 1-1.5 hours (10-15 %) more
> work or a possible 20 mins (3.3 % actually) loss of work. I think it's
> a good deal.
as others point out, one missing write can cause you to loose a large
chunk of data, not just the data being worked on during that time.
so if you end up with a corrupt file, you loose 100% of the file, not just
the changes.
David Lang
On Mon, May 30, 2011 at 11:24:03PM +0200, D. Jansen wrote:
> > do you really have so many fsync's going on that the disk spins up so much
> > that you would gain 10-20% battery life?
>
> Yes. Every autosave in LibreOffice triggers one. And I want autosave,
> but I want them in memory, not on disk.
What on *heck* good does an autosave to memory do? Are you afraid
your X server is going to go poof, or OpenOffice is going to crash on
you? I can't remember the last time this has happened to me. It's
typically a system crash or a power loss that causes me to lose an
OpenOffice session.
> > and what makes you think the extra spin-ups from fsyncs will cause your hard
> > drive to fail significantly earlier? (if you have a hard drive with a
> > limited number of spin-up cycles, you probably don't want to use laptop mode
> > at all)
>
> Experience, see above. Also, this is well described behavior. All hard
> disks are only designed to last a certain number of head loads and
> unloads. Spinning up and down even less.
Modern laptop drives are designed for 600,000 to a million head loads
and unloads. Open Office by default autosaves once every 15 minutes.
So if you leave your system running (on battery?!?) 24x7, with open
office open all that time, even with HDD which is only rated for 600k
load cycles, that's 4.5 years. And of course, normally such a system
is powered all the time, and the hard drive shouldn't be spinning down
if you're on the AC mains.
The real fix is that applications need to be fixed to be less
write-happy. Firefox example, used to request a transactional update
to a sqllite database on every web click. Laptop mode isn't the right
place to fix this, since if you try to stop the writes from hitting
the disk, you'll still end up burning memory that can't be released
until the data is written to disk. My guess is that it's not
OpenOffice which is causing you the problem, unless you've adjusted
the autosave interval, in which case it's your own fault. It's
probably some application like firefox, or syslog. (Syslog can be
fixed by prefixing syslog.conf entries with the '-' character so that
the files aren't sync'ed after every log update, for example. Or not
to log as much in the first place.)
- Ted
On Mon, 30 May 2011, Ted Ts'o wrote:
> On Mon, May 30, 2011 at 11:24:03PM +0200, D. Jansen wrote:
> It's probably some application like firefox, or syslog. (Syslog can be
> fixed by prefixing syslog.conf entries with the '-' character so that
> the files aren't sync'ed after every log update, for example. Or not to
> log as much in the first place.)
note that rsyslog (which is the default on new versions of just about
every distro) changes this default so it no longer does a fsync for each
log entry. It can be configured to be ultra paranoid if you need it, but
by default it's not.
David Lang
Am Montag, 30. Mai 2011, 20:45:42 schrieb [email protected]:
> > The big problem is that so far only fsync existed and lots of software
> > seemingly abuses it as an expensive write barrier. And it would really
> > be lovely to have the choice to stop that on an opt-in basis in laptop
> > mode.
>
> is the benifit of not spinning up the disk really worth the risk of
> loosing data?
That is a policy question.
> and should this really be a global across-the-board option?
Possibly not. However this is a secondary question.
> the problem is that most users don't know what their system is running, or
> what effect disaling fsync would have. those that do can probably use
> LD_PRELOAD to override fsync calls.
Not without a loss of ordering, as we have discussed.
Regards
Oliver
--
- - -
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend?rffer, HRB 16746 (AG N?rnberg)
Maxfeldstra?e 5
90409 N?rnberg
Germany
- - -
On Mon, 30 May 2011 23:24:03 +0200, "D. Jansen" said:
> > do you really have so many fsync's going on that the disk spins up so much
> > that you would gain 10-20% battery life?
>
> Yes. Every autosave in LibreOffice triggers one. And I want autosave,
> but I want them in memory, not on disk.
I think what you want from autosave isn't what the LibreOffice team wants
from autosave. Remember, if they're in memory, they aren't *really* saved.
On Tue, May 31, 2011 at 4:03 AM, Ted Ts'o <[email protected]> wrote:
> On Mon, May 30, 2011 at 11:24:03PM +0200, D. Jansen wrote:
>> > do you really have so many fsync's going on that the disk spins up so much
>> > that you would gain 10-20% battery life?
>>
>> Yes. Every autosave in LibreOffice triggers one. And I want autosave,
>> but I want them in memory, not on disk.
>
> What on *heck* good does an autosave to memory do? Are you afraid
> your X server is going to go poof, or OpenOffice is going to crash on
> you?
Unfortunately, yes. This happens to me regularly, e.g. roughly every
10th resume. It's a poulsbo system. ;)
Another reason is a Java extension that makes it crash happy.
(Please don't tell me now that the real fix is to fix poulsbo...)
>I can't remember the last time this has happened to me. It's
> typically a system crash or a power loss that causes me to lose an
> OpenOffice session.
Well, good for you! Power loss didn't ever occur to me on the other
hand, at least not on my netbook.
>> > and what makes you think the extra spin-ups from fsyncs will cause your hard
>> > drive to fail significantly earlier? (if you have a hard drive with a
>> > limited number of spin-up cycles, you probably don't want to use laptop mode
>> > at all)
>>
>> Experience, see above. Also, this is well described behavior. All hard
>> disks are only designed to last a certain number of head loads and
>> unloads. Spinning up and down even less.
>
> Modern laptop drives are designed for 600,000 to a million head loads
> and unloads. Open Office by default autosaves once every 15 minutes.
> So if you leave your system running (on battery?!?) 24x7, with open
> office open all that time, even with HDD which is only rated for 600k
> load cycles, that's 4.5 years.
Yeah, exactly what I had thought -- enabling laptop mode for the first time.
After the hard disk started to show more read errors
and almost crashed so I had to exchange it, I reconsidered.
I wish specifications would be more in conformity with reality...
Good thing I watch my smart log and caught it in time to avoid data loss.
Though I do have a solid backup routine to avoid serious issues.
> And of course, normally such a system
> is powered all the time, and the hard drive shouldn't be spinning down
> if you're on the AC mains.
Well, I use my netbook on the go mostly. I guess we just have different habits.
>
> The real fix is that applications need to be fixed to be less
> write-happy. Firefox example, used to request a transactional update
> to a sqllite database on every web click. Laptop mode isn't the right
> place to fix this, since if you try to stop the writes from hitting
> the disk, you'll still end up burning memory that can't be released
> until the data is written to disk. (...)
That is another fix that everybody can benefit from. And another
reason to provide barrier support for user space.
(We have a new trojan horse!)
On Tue, 31 May 2011, D. Jansen wrote:
> On Tue, May 31, 2011 at 4:03 AM, Ted Ts'o <[email protected]> wrote:
>> On Mon, May 30, 2011 at 11:24:03PM +0200, D. Jansen wrote:
>>>> do you really have so many fsync's going on that the disk spins up so much
>>>> that you would gain 10-20% battery life?
>>>
>>> Yes. Every autosave in LibreOffice triggers one. And I want autosave,
>>> but I want them in memory, not on disk.
>>
>> What on *heck* good does an autosave to memory do? Are you afraid
>> your X server is going to go poof, or OpenOffice is going to crash on
>> you?
> Unfortunately, yes. This happens to me regularly, e.g. roughly every
> 10th resume. It's a poulsbo system. ;)
> Another reason is a Java extension that makes it crash happy.
> (Please don't tell me now that the real fix is to fix poulsbo...)
>
>> I can't remember the last time this has happened to me. It's
>> typically a system crash or a power loss that causes me to lose an
>> OpenOffice session.
>
> Well, good for you! Power loss didn't ever occur to me on the other
> hand, at least not on my netbook.
failure to resume is effectivly power loss. a autosave to ram would be
lost, just like with a power loss or system crash.
David Lang
On Tue, May 31, 2011 at 8:23 PM, <[email protected]> wrote:
> On Tue, 31 May 2011, D. Jansen wrote:
>
>> On Tue, May 31, 2011 at 4:03 AM, Ted Ts'o <[email protected]> wrote:
>>> I can't remember the last time this has happened to me. It's
>>> typically a system crash or a power loss that causes me to lose an
>>> OpenOffice session.
>>
>> Well, good for you! Power loss didn't ever occur to me on the other
>> hand, at least not on my netbook.
>
> failure to resume is effectivly power loss. a autosave to ram would be lost,
> just like with a power loss or system crash.
>
Well in my case only X crashes and the autosave is saved.
On Tue, 31 May 2011, D. Jansen wrote:
> On Tue, May 31, 2011 at 8:23 PM, <[email protected]> wrote:
>> On Tue, 31 May 2011, D. Jansen wrote:
>>
>>> On Tue, May 31, 2011 at 4:03 AM, Ted Ts'o <[email protected]> wrote:
>
>>>> I can't remember the last time this has happened to me. It's
>>>> typically a system crash or a power loss that causes me to lose an
>>>> OpenOffice session.
>>>
>>> Well, good for you! Power loss didn't ever occur to me on the other
>>> hand, at least not on my netbook.
>>
>> failure to resume is effectivly power loss. a autosave to ram would be lost,
>> just like with a power loss or system crash.
>>
> Well in my case only X crashes and the autosave is saved.
but you have no guarantee that it would be saved without fsync.
it looks like libreoffice does 2 fsyncs per snapshot saved. This is why
your data is saved.
If this is only every 15 minutes (the default), and you have the disk
spin-up time set to every 20 minutes, you will be spinning up the disk
just about as frequently anyway.
now if the system isn't flushing things to disk when it spins up for the
fsync, then there is something wrong.
but if it is working as designed, you get one spin-up every 15 min instead
of one spin-up every 20 minutes, that should not be enough to kill
anything.
David Lang
On Tue, May 31, 2011 at 8:54 PM, <[email protected]> wrote:
> On Tue, 31 May 2011, D. Jansen wrote:
>
>> On Tue, May 31, 2011 at 8:23 PM, <[email protected]> wrote:
>>>
>>> On Tue, 31 May 2011, D. Jansen wrote:
>>>
>>>> On Tue, May 31, 2011 at 4:03 AM, Ted Ts'o <[email protected]> wrote:
>>
>>>>> I can't remember the last time this has happened to me. It's
>>>>> typically a system crash or a power loss that causes me to lose an
>>>>> OpenOffice session.
>>>>
>>>> Well, good for you! Power loss didn't ever occur to me on the other
>>>> hand, at least not on my netbook.
>>>
>>> failure to resume is effectivly power loss. a autosave to ram would be
>>> lost,
>>> just like with a power loss or system crash.
>>>
>> Well in my case only X crashes and the autosave is saved.
>
> but you have no guarantee that it would be saved without fsync.
But I would with barriers.
>
> it looks like libreoffice does 2 fsyncs per snapshot saved. This is why your
> data is saved.
>
> If this is only every 15 minutes (the default), and you have the disk
> spin-up time set to every 20 minutes, you will be spinning up the disk just
> about as frequently anyway.
>
> now if the system isn't flushing things to disk when it spins up for the
> fsync, then there is something wrong.
>
> but if it is working as designed, you get one spin-up every 15 min instead
> of one spin-up every 20 minutes, that should not be enough to kill anything.
Yes, if there was nothing else, that's probably true. But _many_
applications use fsync, even KDE's plasma shell I think.
On Tue, May 31, 2011 at 4:03 AM, Ted Ts'o <[email protected]> wrote:
> The real fix is that applications need to be fixed to be less
> write-happy.
So do you agree to export barriers to user space? Because as you said
yourself, that would be the only way for apps to avoid fsync without
losing data safety. And since you want the problem fixed with the
apps, we need to provide them with the means to do so, right?