On 14/04/11 16:50, Eric Sandeen wrote:
> On 4/14/11 9:59 AM, P?draig Brady wrote:
>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
>>>>> Hi P?draig,
>>>>>
>>>>> here you go:
>>>>> + filefrag -v unwritten.withdata
>>>>> Filesystem type is: ef53
>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)
>>>>> ext logical physical expected length flags
>>>>> 0 0 274432 2560 unwritten,eof
>>>>> unwritten.withdata: 1 extent found
>>>>>
>>>>> Please notice that this also happens with ext4 on the same kernel.
>>>>> Btrfs is fine.
>>>>
>>> `filefrag -vs` fixes the issue on both xfs and ext4.
>>
>> So in summary, currently on (2.6.39-rc3), the following
>> will (usually?) report a single unwritten extent,
>> on both ext4 and xfs
>>
>> fallocate -l 10MiB -n k
>> dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
>> filefrag -v k # grep for an extent without unwritten || fail
>
> right, that's what I see too in testing.
>
> But would the coreutils install have done a preallocation of the destination file?
>
> Otherwise this looks like a different bug...
>
>> This particular issue has been discussed so far at:
>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8411
>> Note there it was stated there that ext4 had this
>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
>
> ext4 got a fix, but not xfs, I guess. My poor brain can't remember, I think I started looking into it, but it's clearly still broken.
>
> Still, I don't know for sure what happened to Markus - did something preallocate, in his case?
Well that preallocate test is failing for him
when the source file is either on ext4 or xfs.
He noticed the issue initially on XFS when copying
none preallocated files, so XFS probably just has
the general issue of needing a sync before fiemap,
where as EXT4 just has this preallocate one
(though I've not seen it myself).
cheers,
P?draig.
On 4/14/11 10:52 AM, P?draig Brady wrote:
> On 14/04/11 16:50, Eric Sandeen wrote:
>> On 4/14/11 9:59 AM, P?draig Brady wrote:
>>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
>>>>>> Hi P?draig,
>>>>>>
>>>>>> here you go:
>>>>>> + filefrag -v unwritten.withdata
>>>>>> Filesystem type is: ef53
>>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)
>>>>>> ext logical physical expected length flags
>>>>>> 0 0 274432 2560 unwritten,eof
>>>>>> unwritten.withdata: 1 extent found
>>>>>>
>>>>>> Please notice that this also happens with ext4 on the same kernel.
>>>>>> Btrfs is fine.
>>>>>
>>>> `filefrag -vs` fixes the issue on both xfs and ext4.
>>>
>>> So in summary, currently on (2.6.39-rc3), the following
>>> will (usually?) report a single unwritten extent,
>>> on both ext4 and xfs
>>>
>>> fallocate -l 10MiB -n k
>>> dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
>>> filefrag -v k # grep for an extent without unwritten || fail
>>
>> right, that's what I see too in testing.
>>
>> But would the coreutils install have done a preallocation of the destination file?
>>
>> Otherwise this looks like a different bug...
>>
>>> This particular issue has been discussed so far at:
>>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8411
>>> Note there it was stated there that ext4 had this
>>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
>>
>> ext4 got a fix, but not xfs, I guess. My poor brain can't remember, I think I started looking into it, but it's clearly still broken.
>>
>> Still, I don't know for sure what happened to Markus - did something preallocate, in his case?
>
> Well that preallocate test is failing for him
> when the source file is either on ext4 or xfs.
> He noticed the issue initially on XFS when copying
> none preallocated files, so XFS probably just has
> the general issue of needing a sync before fiemap,
> where as EXT4 just has this preallocate one
> (though I've not seen it myself).
>
> cheers,
> P?draig.
>
well, if I simply take the preallocation step out of the testcase, it works fine on xfs without a sync.
So I still don't know what Markus hit...
-Eric
On 2011.04.14 at 10:56 -0500, Eric Sandeen wrote:
> On 4/14/11 10:52 AM, P?draig Brady wrote:
> > On 14/04/11 16:50, Eric Sandeen wrote:
> >> On 4/14/11 9:59 AM, P?draig Brady wrote:
> >>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
> >>>>>> Hi P?draig,
> >>>>>>
> >>>>>> here you go:
> >>>>>> + filefrag -v unwritten.withdata
> >>>>>> Filesystem type is: ef53
> >>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)
> >>>>>> ext logical physical expected length flags
> >>>>>> 0 0 274432 2560 unwritten,eof
> >>>>>> unwritten.withdata: 1 extent found
> >>>>>>
> >>>>>> Please notice that this also happens with ext4 on the same kernel.
> >>>>>> Btrfs is fine.
> >>>>>
> >>>> `filefrag -vs` fixes the issue on both xfs and ext4.
> >>>
> >>> So in summary, currently on (2.6.39-rc3), the following
> >>> will (usually?) report a single unwritten extent,
> >>> on both ext4 and xfs
> >>>
> >>> fallocate -l 10MiB -n k
> >>> dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
> >>> filefrag -v k # grep for an extent without unwritten || fail
> >>
> >> right, that's what I see too in testing.
> >>
> >> But would the coreutils install have done a preallocation of the destination file?
> >>
> >> Otherwise this looks like a different bug...
> >>
> >>> This particular issue has been discussed so far at:
> >>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8411
> >>> Note there it was stated there that ext4 had this
> >>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
> >>
> >> ext4 got a fix, but not xfs, I guess. My poor brain can't remember, I think I started looking into it, but it's clearly still broken.
> >>
> >> Still, I don't know for sure what happened to Markus - did something preallocate, in his case?
> >
> > Well that preallocate test is failing for him
> > when the source file is either on ext4 or xfs.
> > He noticed the issue initially on XFS when copying
> > none preallocated files, so XFS probably just has
> > the general issue of needing a sync before fiemap,
> > where as EXT4 just has this preallocate one
> > (though I've not seen it myself).
> >
> > cheers,
> > P?draig.
> >
>
> well, if I simply take the preallocation step out of the testcase, it works fine on xfs without a sync.
>
> So I still don't know what Markus hit...
Maybe it's delalloc:
x4 /tmp # dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
10+0 records in
10+0 records out
5120 bytes (5.1 kB) copied, 0.0021822 s, 2.3 MB/s
x4 /tmp # filefrag -v k
Filesystem type is: 58465342
File size of k is 5120 (2 blocks, blocksize 4096)
ext logical physical expected length flags
0 0 0 16 unknown,delalloc,eof
k: 1 extent found
x4 /tmp # sync
x4 /tmp # filefrag -v k
Filesystem type is: 58465342
File size of k is 5120 (2 blocks, blocksize 4096)
ext logical physical expected length flags
0 0 26960045 16 eof
k: 1 extent found
--
Markus
2011/4/14 Eric Sandeen <[email protected]>:
> On 4/14/11 10:52 AM, P?draig Brady wrote:
>> On 14/04/11 16:50, Eric Sandeen wrote:
>>> On 4/14/11 9:59 AM, P?draig Brady wrote:
>>>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
>>>>>>> Hi P?draig,
>>>>>>>
>>>>>>> here you go:
>>>>>>> + filefrag -v unwritten.withdata
>>>>>>> Filesystem type is: ef53
>>>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)
>>>>>>> ?ext logical physical expected length flags
>>>>>>> ? ?0 ? ? ? 0 ? 274432 ? ? ? ? ? ?2560 unwritten,eof
>>>>>>> unwritten.withdata: 1 extent found
>>>>>>>
>>>>>>> Please notice that this also happens with ext4 on the same kernel.
>>>>>>> Btrfs is fine.
>>>>>>
>>>>> `filefrag -vs` fixes the issue on both xfs and ext4.
>>>>
>>>> So in summary, currently on (2.6.39-rc3), the following
>>>> will (usually?) report a single unwritten extent,
>>>> on both ext4 and xfs
>>>>
>>>> ? fallocate -l 10MiB -n k
>>>> ? dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
>>>> ? filefrag -v k # grep for an extent without unwritten || fail
>>>
>>> right, that's what I see too in testing.
>>>
>>> But would the coreutils install have done a preallocation of the destination file?
>>>
>>> Otherwise this looks like a different bug...
>>>
>>>> This particular issue has been discussed so far at:
>>>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8411
>>>> Note there it was stated there that ext4 had this
>>>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
>>>
>>> ext4 got a fix, but not xfs, I guess. ?My poor brain can't remember, I think I started looking into it, but it's clearly still broken.
>>>
>>> Still, I don't know for sure what happened to Markus - did something preallocate, in his case?
>>
>> Well that preallocate test is failing for him
>> when the source file is either on ext4 or xfs.
>> He noticed the issue initially on XFS when copying
>> none preallocated files, so XFS probably just has
>> the general issue of needing a sync before fiemap,
>> where as EXT4 just has this preallocate one
>> (though I've not seen it myself).
>>
>> cheers,
>> P?draig.
>>
>
> well, if I simply take the preallocation step out of the testcase, it works fine on xfs without a sync.
>
> So I still don't know what Markus hit...
Sorry for that my patch ignored fallocate. The situation is like this:
An user allocated space for a file with fallocate, and write a small
part of it. and the written is not flushed.
The extent stays one unwritten extent in disk and memory with delayed
allocation.
In ext4 ext4_ext_walk_space() thinks an extent does not exist only if
there is no any extents on disk. So ext4_ext_walk_space()
thinks there is a extent and ext4_ext_fiemap_cb() thus ignores pagecache.
I think ext4_ext_walk_space() should take unwritten extent not exist.
Yongqiang.
>
> -Eric
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at ?http://vger.kernel.org/majordomo-info.html
>
--
Best Wishes
Yongqiang Yang
Hi,
I am off my working computer. Maybe below fix could fix the problem.
fs/ext4/extent.c
static int ext4_ext_walk_space(struct inode *inode, ext4_lblk_t block,
1877 } else if (block >= le32_to_cpu(ex->ee_block)) {
1878 /*
1879 * some part of requested space is covered
1880 * by found extent
1881 */
1882 start = block;
1883 end = le32_to_cpu(ex->ee_block)
1884 + ext4_ext_get_actual_len(ex);
1885 if (block + num < end)
1886 end = block + num;
+ if (!ext4_ext_is_uninitialized(ex))
1887 exists = 1;
1888 } else {
1889 BUG();
1890 }
On Fri, Apr 15, 2011 at 12:04 AM, Yongqiang Yang <[email protected]> wrote:
> 2011/4/14 Eric Sandeen <[email protected]>:
>> On 4/14/11 10:52 AM, P?draig Brady wrote:
>>> On 14/04/11 16:50, Eric Sandeen wrote:
>>>> On 4/14/11 9:59 AM, P?draig Brady wrote:
>>>>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
>>>>>>>> Hi P?draig,
>>>>>>>>
>>>>>>>> here you go:
>>>>>>>> + filefrag -v unwritten.withdata
>>>>>>>> Filesystem type is: ef53
>>>>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)
>>>>>>>> ?ext logical physical expected length flags
>>>>>>>> ? ?0 ? ? ? 0 ? 274432 ? ? ? ? ? ?2560 unwritten,eof
>>>>>>>> unwritten.withdata: 1 extent found
>>>>>>>>
>>>>>>>> Please notice that this also happens with ext4 on the same kernel.
>>>>>>>> Btrfs is fine.
>>>>>>>
>>>>>> `filefrag -vs` fixes the issue on both xfs and ext4.
>>>>>
>>>>> So in summary, currently on (2.6.39-rc3), the following
>>>>> will (usually?) report a single unwritten extent,
>>>>> on both ext4 and xfs
>>>>>
>>>>> ? fallocate -l 10MiB -n k
>>>>> ? dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
>>>>> ? filefrag -v k # grep for an extent without unwritten || fail
>>>>
>>>> right, that's what I see too in testing.
>>>>
>>>> But would the coreutils install have done a preallocation of the destination file?
>>>>
>>>> Otherwise this looks like a different bug...
>>>>
>>>>> This particular issue has been discussed so far at:
>>>>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8411
>>>>> Note there it was stated there that ext4 had this
>>>>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
>>>>
>>>> ext4 got a fix, but not xfs, I guess. ?My poor brain can't remember, I think I started looking into it, but it's clearly still broken.
>>>>
>>>> Still, I don't know for sure what happened to Markus - did something preallocate, in his case?
>>>
>>> Well that preallocate test is failing for him
>>> when the source file is either on ext4 or xfs.
>>> He noticed the issue initially on XFS when copying
>>> none preallocated files, so XFS probably just has
>>> the general issue of needing a sync before fiemap,
>>> where as EXT4 just has this preallocate one
>>> (though I've not seen it myself).
>>>
>>> cheers,
>>> P?draig.
>>>
>>
>> well, if I simply take the preallocation step out of the testcase, it works fine on xfs without a sync.
>>
>> So I still don't know what Markus hit...
> Sorry for that my patch ignored fallocate. ?The situation is like this:
> An user allocated space for a file with fallocate, and write a small
> part of it. and the written is not flushed.
> The extent stays one unwritten extent in disk and memory with delayed
> allocation.
>
> In ext4 ext4_ext_walk_space() thinks an extent does not exist only if
> there is no any extents on disk. ?So ext4_ext_walk_space()
> thinks there is a extent and ? ext4_ext_fiemap_cb() thus ignores pagecache.
>
> I think ext4_ext_walk_space() should take unwritten extent not exist.
>
> Yongqiang.
>>
>> -Eric
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
>> the body of a message to [email protected]
>> More majordomo info at ?http://vger.kernel.org/majordomo-info.html
>>
>
>
>
> --
> Best Wishes
> Yongqiang Yang
>
--
Best Wishes
Yongqiang Yang
On 4/14/11 11:03 AM, Markus Trippelsdorf wrote:
> On 2011.04.14 at 10:56 -0500, Eric Sandeen wrote:
...
>> well, if I simply take the preallocation step out of the testcase, it works fine on xfs without a sync.
>>
>> So I still don't know what Markus hit...
>
> Maybe it's delalloc:
>
> x4 /tmp # dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
> 10+0 records in
> 10+0 records out
> 5120 bytes (5.1 kB) copied, 0.0021822 s, 2.3 MB/s
> x4 /tmp # filefrag -v k
> Filesystem type is: 58465342
> File size of k is 5120 (2 blocks, blocksize 4096)
> ext logical physical expected length flags
> 0 0 0 16 unknown,delalloc,eof
> k: 1 extent found
> x4 /tmp # sync
> x4 /tmp # filefrag -v k
> Filesystem type is: 58465342
> File size of k is 5120 (2 blocks, blocksize 4096)
> ext logical physical expected length flags
> 0 0 26960045 16 eof
> k: 1 extent found
>
well filefrag still returns that it knows of a valid range from logical blocks 0 to 16, so cp should have something to copy...
-Eric