Tested with various 2.6.x i386/x86-64 SMP kernels and CPUs, for example
2.6.21.3/x86-64.
Process 1:
- lock file
- write(4096 + 16 bytes)
- unlock file
Process 2:
- lock file
- read(8192 bytes)
- unlock file
Sometimes read() returns only 4096 bytes. I'm locking the file, so I
don't think this should ever happen, right?
Attached a test program. Takes from a few seconds to half a minute with
my computer to print "page size cut".
On Wed, 2007-06-20 at 17:52 +0300, Timo Sirainen wrote:
> Sometimes read() returns only 4096 bytes. I'm locking the file, so I
> don't think this should ever happen, right?
Sorry, the problem was with file truncating so there's no bug with
locking. I didn't notice it first because it happened to work with
FreeBSD and Solaris.
Without locking there's a problem that I'd want to avoid, but I guess
it's not necessarily a bug (although it doesn't seem to happen with
FreeBSD or Solaris):
Process 1:
- create "foo2"
- rename() to "foo"
- lock
- write(4096 + 16 bytes)
- [fsync() here doesn't change anything]
- pwrite("1111" to offset=4096-4)
- unlock
Process 2:
- open("foo")
- read(8192 bytes)
read() sometimes returns 4096 bytes but with the "1111" already included
in the data. Is there a way to avoid this without locking the file while
reading? The "1111" tries to act as a kind of a lock.
Again attached a test program, which should really do what I
intended. :)
On 6/20/07, Timo Sirainen <[email protected]> wrote:
> On Wed, 2007-06-20 at 17:52 +0300, Timo Sirainen wrote:
> > Sometimes read() returns only 4096 bytes. I'm locking the file, so I
> > don't think this should ever happen, right?
man 2 read
read() is always allowed to return less than you asked for. You need
to go back and ask for the rest. That's why people wrap read() and
write() in loops, and use those wrapped versions instead.
> read() sometimes returns 4096 bytes but with the "1111" already included
> in the data.
Uhm, do you mean 'without the 1111'? 4096 with the 1111 sounds
perfectly fine. (Though 4096 without the 1111 is also perfectly fine.)
> Is there a way to avoid this without locking the file while
> reading? The "1111" tries to act as a kind of a lock.
I think you've misunderstood how read and write work. pread and pwrite
are no different in this respect -- they just allow an atomic
seek+read/write.
Ray
On Wed, 2007-06-20 at 09:22 -0700, Ray Lee wrote:
> On 6/20/07, Timo Sirainen <[email protected]> wrote:
> > On Wed, 2007-06-20 at 17:52 +0300, Timo Sirainen wrote:
> > > Sometimes read() returns only 4096 bytes. I'm locking the file, so I
> > > don't think this should ever happen, right?
>
> man 2 read
>
> read() is always allowed to return less than you asked for. You need
> to go back and ask for the rest. That's why people wrap read() and
> write() in loops, and use those wrapped versions instead.
I thought this wouldn't happen when reading from local filesystems.
Anyway, my real program was doing that, and it was still seeing partial
data. But looks like I can't reproduce it in my test program with two
pread()s, so I'll have to do some more debugging.
> I thought this wouldn't happen when reading from local filesystems.
> Anyway, my real program was doing that, and it was still seeing partial
> data. But looks like I can't reproduce it in my test program with two
> pread()s, so I'll have to do some more debugging.
It will occur if you are reading as someone else changes the file size.
Use file locking, it exists for a reason ;)
Alan
On Wed, 2007-06-20 at 18:05 +0100, Alan Cox wrote:
> > I thought this wouldn't happen when reading from local filesystems.
> > Anyway, my real program was doing that, and it was still seeing partial
> > data. But looks like I can't reproduce it in my test program with two
> > pread()s, so I'll have to do some more debugging.
Found the problem in my code. A buggy program, buggy test program and a
buggy brain while debugging doesn't seem to give the best results. :)
> It will occur if you are reading as someone else changes the file size.
> Use file locking, it exists for a reason ;)
Annoying extra overhead. Especially with NFS, when nowadays you can't
even use flock() to create local locks..
On Wed, 20 Jun 2007, Timo Sirainen wrote:
> > It will occur if you are reading as someone else changes the file size.
> > Use file locking, it exists for a reason ;)
> Annoying extra overhead. Especially with NFS, when nowadays you can't
> even use flock() to create local locks..
It's not only "nowadays", it has been like that for quite a some time.
On the other hand you can use fcntl(F_SETLK) though, which works even
through NFS (v3).
--
Jiri Kosina