In some cases, using the 'truncate' command to extend a UDF file results
in a mismatch between the length of the file's extents (specifically, due
to incorrect length of the final NOT_ALLOCATED extent) and the information
(file) length. The discrepancy can prevent other operating systems
(i.e., Windows 10) from opening the file.
Two particular errors have been observed when extending a file:
1. The final extent is larger than it should be, having been rounded up
to a multiple of the block size.
B. The final extent is not shorter than it should be, due to not having
been updated when the file's information length was increased.
The first case could represent a design error, if coded intentionally
due to a misinterpretation of scantily-documented ECMA-167 "file tail"
rules. The standard specifies that the tail, if present, consists of
a sequence of "unrecorded and allocated" extents (only).
Signed-off-by: Steven J. Magnani <[email protected]>
--- a/fs/udf/inode.c 2019-05-24 21:17:33.659704533 -0500
+++ b/fs/udf/inode.c 2019-05-29 20:32:23.730129419 -0500
@@ -474,7 +474,8 @@ static struct buffer_head *udf_getblk(st
static int udf_do_extend_file(struct inode *inode,
struct extent_position *last_pos,
struct kernel_long_ad *last_ext,
- sector_t blocks)
+ sector_t blocks,
+ unsigned long partial_final_block)
{
sector_t add;
int count = 0, fake = !(last_ext->extLength & UDF_EXTENT_LENGTH_MASK);
@@ -486,7 +487,7 @@ static int udf_do_extend_file(struct ino
/* The previous extent is fake and we should not extend by anything
* - there's nothing to do... */
- if (!blocks && fake)
+ if (!blocks && !partial_final_block && fake)
return 0;
iinfo = UDF_I(inode);
@@ -524,6 +525,10 @@ static int udf_do_extend_file(struct ino
add = blocks;
blocks -= add;
last_ext->extLength += add << sb->s_blocksize_bits;
+ if (blocks == 0 && partial_final_block) {
+ last_ext->extLength -= sb->s_blocksize
+ - partial_final_block;
+ }
}
if (fake) {
@@ -566,6 +571,10 @@ static int udf_do_extend_file(struct ino
if (blocks) {
last_ext->extLength = EXT_NOT_RECORDED_NOT_ALLOCATED |
(blocks << sb->s_blocksize_bits);
+ if (partial_final_block) {
+ last_ext->extLength -= sb->s_blocksize
+ - partial_final_block;
+ }
err = udf_add_aext(inode, last_pos, &last_ext->extLocation,
last_ext->extLength, 1);
if (err)
@@ -605,6 +614,7 @@ static int udf_extend_file(struct inode
int8_t etype;
struct super_block *sb = inode->i_sb;
sector_t first_block = newsize >> sb->s_blocksize_bits, offset;
+ unsigned long partial_final_block;
int adsize;
struct udf_inode_info *iinfo = UDF_I(inode);
struct kernel_long_ad extent;
@@ -619,15 +629,17 @@ static int udf_extend_file(struct inode
etype = inode_bmap(inode, first_block, &epos, &eloc, &elen, &offset);
+ partial_final_block = newsize & (sb->s_blocksize - 1);
+
/* File has extent covering the new size (could happen when extending
* inside a block)? */
- if (etype != -1)
- return 0;
- if (newsize & (sb->s_blocksize - 1))
- offset++;
- /* Extended file just to the boundary of the last file block? */
- if (offset == 0)
- return 0;
+ if (etype == -1) {
+ if (partial_final_block)
+ offset++;
+ } else {
+ /* Extending file within the last file block */
+ offset = 0; /* Don't add any new blocks */
+ }
/* Truncate is extending the file by 'offset' blocks */
if ((!epos.bh && epos.offset == udf_file_entry_alloc_offset(inode)) ||
@@ -643,7 +655,8 @@ static int udf_extend_file(struct inode
&extent.extLength, 0);
extent.extLength |= etype << 30;
}
- err = udf_do_extend_file(inode, &epos, &extent, offset);
+ err = udf_do_extend_file(inode, &epos, &extent, offset,
+ partial_final_block);
if (err < 0)
goto out;
err = 0;
@@ -760,7 +773,7 @@ static sector_t inode_getblk(struct inod
startnum = (offset > 0);
}
/* Create extents for the hole between EOF and offset */
- ret = udf_do_extend_file(inode, &prev_epos, laarr, offset);
+ ret = udf_do_extend_file(inode, &prev_epos, laarr, offset, 0);
if (ret < 0) {
*err = ret;
newblock = 0;
On 6/4/19 7:31 AM, Steve Magnani wrote:
> B. The final extent is not shorter than it should be, due to not having
Oops: should have been
B. The final extent is shorter than it should be, due to not having
Hi Jan,
On 6/4/19 7:31 AM, Steve Magnani wrote:
> In some cases, using the 'truncate' command to extend a UDF file results
> in a mismatch between the length of the file's extents (specifically, due
> to incorrect length of the final NOT_ALLOCATED extent) and the information
> (file) length. The discrepancy can prevent other operating systems
> (i.e., Windows 10) from opening the file.
>
> Two particular errors have been observed when extending a file:
>
> 1. The final extent is larger than it should be, having been rounded up
> to a multiple of the block size.
>
> B. The final extent is shorter than it should be, due to not having
> been updated when the file's information length was increased.
Wondering if you've seen this, or if something got lost in a spam folder.
Regards,
------------------------------------------------------------------------
Steven J. Magnani "I claim this network for MARS!
http://www.digidescorp.com Earthling, return my space modulator!"
#include <standard.disclaimer>
Hi Steve!
On Sun 16-06-19 11:28:46, Steve Magnani wrote:
> On 6/4/19 7:31 AM, Steve Magnani wrote:
>
> > In some cases, using the 'truncate' command to extend a UDF file results
> > in a mismatch between the length of the file's extents (specifically, due
> > to incorrect length of the final NOT_ALLOCATED extent) and the information
> > (file) length. The discrepancy can prevent other operating systems
> > (i.e., Windows 10) from opening the file.
> >
> > Two particular errors have been observed when extending a file:
> >
> > 1. The final extent is larger than it should be, having been rounded up
> > to a multiple of the block size.
> >
> > B. The final extent is shorter than it should be, due to not having
> > been updated when the file's information length was increased.
>
> Wondering if you've seen this, or if something got lost in a spam folder.
Sorry for not getting to you earlier. I've seen the patches and they look
reasonable to me. I just wanted to have a one more closer look but last
weeks were rather busy so I didn't get to it. I'll look into it this week.
Thanks a lot for debugging the problem and sending the fixes!
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR
On 6/19/19 1:47 AM, Jan Kara wrote:
> Hi Steve!
>
> On Sun 16-06-19 11:28:46, Steve Magnani wrote:
>> On 6/4/19 7:31 AM, Steve Magnani wrote:
>>
>>> In some cases, using the 'truncate' command to extend a UDF file results
>>> in a mismatch between the length of the file's extents (specifically, due
>>> to incorrect length of the final NOT_ALLOCATED extent) and the information
>>> (file) length. The discrepancy can prevent other operating systems
>>> (i.e., Windows 10) from opening the file.
>>>
>>> Two particular errors have been observed when extending a file:
>>>
>>> 1. The final extent is larger than it should be, having been rounded up
>>> to a multiple of the block size.
>>>
>>> B. The final extent is shorter than it should be, due to not having
>>> been updated when the file's information length was increased.
>> Wondering if you've seen this, or if something got lost in a spam folder.
> Sorry for not getting to you earlier. I've seen the patches and they look
> reasonable to me. I just wanted to have a one more closer look but last
> weeks were rather busy so I didn't get to it. I'll look into it this week.
> Thanks a lot for debugging the problem and sending the fixes!
>
> Honza
No worries. If you're short on time I'd suggest looking first at the ways
udf_do_extend_file() can be called via inode_getblk(). Those were harder for
me to follow so if there is a bug it's most likely in one of those paths.
Regards,
------------------------------------------------------------------------
Steven J. Magnani "I claim this network for MARS!
http://www.digidescorp.com Earthling, return my space modulator!"
#include <standard.disclaimer>
On Tue 04-06-19 07:31:58, Steve Magnani wrote:
> In some cases, using the 'truncate' command to extend a UDF file results
> in a mismatch between the length of the file's extents (specifically, due
> to incorrect length of the final NOT_ALLOCATED extent) and the information
> (file) length. The discrepancy can prevent other operating systems
> (i.e., Windows 10) from opening the file.
>
> Two particular errors have been observed when extending a file:
>
> 1. The final extent is larger than it should be, having been rounded up
> to a multiple of the block size.
>
> B. The final extent is not shorter than it should be, due to not having
> been updated when the file's information length was increased.
>
> The first case could represent a design error, if coded intentionally
> due to a misinterpretation of scantily-documented ECMA-167 "file tail"
> rules. The standard specifies that the tail, if present, consists of
> a sequence of "unrecorded and allocated" extents (only).
>
> Signed-off-by: Steven J. Magnani <[email protected]>
Thanks for the testcase and the patch! I finally got to reading through
this in detail. In udf driver in Linux we are generally fine with the last
extent being rounded up to the block size. udf_truncate_tail_extent() is
generally responsible for truncating the last extent to appropriate size
once we are done with the inode. However there are two problems with this:
1) We used to do this inside udf_clear_inode() back in the old days but
then switched to a different scheme in commit 2c948b3f86e5f "udf: Avoid IO
in udf_clear_inode". So this actually breaks workloads where user calls
truncate(2) directly and there's no place where udf_truncate_tail_extent()
gets called.
2) udf_extend_file() sets i_lenExtents == i_size although the last extent
isn't properly rounded so even if udf_truncate_tail_extent() gets called
(which is actually the case for truncate(1) which does open, ftruncate,
close), it will think it has nothing to do and exit.
Now 2) is easily fixed by setting i_lenExtents to real length of extents we
have created. However that still leaves problem 1) which isn't easy to deal
with. After some though I think that your solution of making
udf_do_extend_file() always create appropriately sized extents makes
sense. However I dislike the calling convention you've chosen. When
udf_do_extend_file() needs to now byte length, then why not pass it to it
directly, instead of somewhat cumbersome "sector length + byte offset"
pair?
Will you update the patch please? Thanks!
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR
Hi Jan,
On 6/25/19 5:30 AM, Jan Kara wrote:
> On Tue 04-06-19 07:31:58, Steve Magnani wrote:
>> In some cases, using the 'truncate' command to extend a UDF file results
>> in a mismatch between the length of the file's extents (specifically, due
>> to incorrect length of the final NOT_ALLOCATED extent) and the information
>> (file) length. The discrepancy can prevent other operating systems
>> (i.e., Windows 10) from opening the file.
>>
>> Two particular errors have been observed when extending a file:
>>
>> 1. The final extent is larger than it should be, having been rounded up
>> to a multiple of the block size.
>>
>> B. The final extent is not shorter than it should be, due to not having
>> been updated when the file's information length was increased.
>>
>> The first case could represent a design error, if coded intentionally
>> due to a misinterpretation of scantily-documented ECMA-167 "file tail"
>> rules. The standard specifies that the tail, if present, consists of
>> a sequence of "unrecorded and allocated" extents (only).
>>
>> Signed-off-by: Steven J. Magnani <[email protected]>
> Thanks for the testcase and the patch! I finally got to reading through
> this in detail. In udf driver in Linux we are generally fine with the last
> extent being rounded up to the block size. udf_truncate_tail_extent() is
> generally responsible for truncating the last extent to appropriate size
> once we are done with the inode. However there are two problems with this:
>
> 1) We used to do this inside udf_clear_inode() back in the old days but
> then switched to a different scheme in commit 2c948b3f86e5f "udf: Avoid IO
> in udf_clear_inode". So this actually breaks workloads where user calls
> truncate(2) directly and there's no place where udf_truncate_tail_extent()
> gets called.
>
> 2) udf_extend_file() sets i_lenExtents == i_size although the last extent
> isn't properly rounded so even if udf_truncate_tail_extent() gets called
> (which is actually the case for truncate(1) which does open, ftruncate,
> close), it will think it has nothing to do and exit.
>
> Now 2) is easily fixed by setting i_lenExtents to real length of extents we
> have created. However that still leaves problem 1) which isn't easy to deal
> with. After some though I think that your solution of making
> udf_do_extend_file() always create appropriately sized extents makes
> sense. However I dislike the calling convention you've chosen. When
> udf_do_extend_file() needs to now byte length, then why not pass it to it
> directly, instead of somewhat cumbersome "sector length + byte offset"
> pair?
>
> Will you update the patch please? Thanks!
That sounds reasonable, but at first glance I think it might be more
confusing. The API as I reworked it now communicates two different
(although related) things - the number of blocks that need to be added,
and the number of bytes within the last block that are part of the file.
This is able to cover both the corner case of extending within the last
file block and extending beyond that:
partial_final_block = newsize & (sb->s_blocksize - 1);
/* File has extent covering the new size (could happen when extending
* inside a block)? */
if (etype == -1) {
if (partial_final_block)
offset++;
} else {
/* Extending file within the last file block */
offset = 0; /* Don't add any new blocks */
}
If it were as simple as passing to udf_do_extend_file() a loff_t
specifying the number of bytes to add, including both full blocks and a
final partial block, I would agree with you. But this isn't enough
information for udf_do_extend_file() to know whether the final partial
block requires a new block or not.
I will think about it some more. Maybe moving the 'extending within the
last file block' case out to udf_extend_file() would help.
Steve