Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1946042Ab2JYODk (ORCPT ); Thu, 25 Oct 2012 10:03:40 -0400 Received: from li9-11.members.linode.com ([67.18.176.11]:57489 "EHLO imap.thunk.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933124Ab2JYODi (ORCPT ); Thu, 25 Oct 2012 10:03:38 -0400 Date: Thu, 25 Oct 2012 10:03:27 -0400 From: "Theodore Ts'o" To: david@lang.hm Cc: Nico Williams , General Discussion of SQLite Database , =?utf-8?B?5p2o6IuP56uL?= Yang Su Li , linux-fsdevel@vger.kernel.org, linux-kernel , drh@hwaci.com Subject: Re: [sqlite] light weight write barriers Message-ID: <20121025140327.GB13562@thunk.org> Mail-Followup-To: Theodore Ts'o , david@lang.hm, Nico Williams , General Discussion of SQLite Database , =?utf-8?B?5p2o6IuP56uL?= Yang Su Li , linux-fsdevel@vger.kernel.org, linux-kernel , drh@hwaci.com References:

<5086F5A7.9090406@vlnb.net>

<20121025060231.GC9860@thunk.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: tytso@thunk.org X-SA-Exim-Scanned: No (on imap.thunk.org); SAEximRunCond expanded to false Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2655 Lines: 51 On Wed, Oct 24, 2012 at 11:58:49PM -0700, david@lang.hm wrote: > The frustrating thing is that when people point out how things like > sqlite are so horribly slow, the reply seems to be "well, that's > what you get for doing so many fsyncs, don't do that", when there is > a 'problem' like the KDE "config loss" problem a few years ago, the > response is "well, that's what you get for not doing fsync" Sure... but the answer is to only do the fsync's when you need to. For example, if GNOME and KDE is rewriting the entire registry file each time the application is changing a single registry key, sure, if you rewrite the entire registry file, and then fsync after each rewrite before you replace the file, you will be safe. And if the application needs to update dozens or hundreds of registry keys (or every time the window gets moved or resized), then yes, it will be slow. But the application didn't have to do that! It could have updated all the registry keys in memory, and then update the registry file periodically instead. Similarly, Firefox didn't need to do a sqllite commit after every single time its history file was written, causing a third of a megabyte of write traffic each time you clicked on a web page. It could have batched its updates to the history file, since most of the time, you don't care about making sure the web history is written to stable store before you're allowed to click on a web page and visit the next web page. Or does rsyslog *really* need to issue an fsync after each log message? Or could it batch updates so that every N seconds, it flushes writes to the disk? (And this is a problem with most Android applications as well. Apparently the framework API's are such that it's easier for an application to treat each sqlite statement as an atomic update, so many/most application writers don't use explicit transaction boundaries, so updates don't get batched even though it would be more efficient if they did so.) Sometimes, the answer is not to try to create exotic database like functionality in the file system --- the answer is to be more intelligent at the application leyer. Not only will the application be more portable, it will also in the end be more efficient, since even with the most exotic database technologies, the most efficient transactional commit is the unneeded commit that you optimize away at the application layer. - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/