Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Steve Magnani <steve.magnani@digidescorp.com>
Subject: Re: [RFC] udf: 2.01 interoperability issues with Windows 10
To:     =?UTF-8?Q?Pali_Roh=c3=a1r?= <pali.rohar@gmail.com>
Cc:     Jan Kara <jack@suse.cz>,
        =?UTF-8?Q?Vojt=c4=9bch_Vladyka?= <vojtech.vladyka@foxyco.cz>,
        linux-fsdevel@vger.kernel.org,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
References: <96e1ea00-ac12-015d-5c54-80a83f08b898@digidescorp.com>
 <20190709185638.pcgdbdqoqgur6id3@pali>
 <958ea915-3568-8f5a-581c-e5f0a673d30f@digidescorp.com>
 <20190709210457.kzjnigu6fwgxxq27@pali>
Message-ID: <2994ee3a-9e38-0ed0-652a-e85de704f8d1@digidescorp.com>
Date:   Wed, 10 Jul 2019 08:24:09 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.7.0
MIME-Version: 1.0
In-Reply-To: <20190709210457.kzjnigu6fwgxxq27@pali>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


On 7/9/19 4:04 PM, Pali Rohár wrote:
> On Tuesday 09 July 2019 15:14:38 Steve Magnani wrote:
>> On 7/9/19 1:56 PM, Pali Rohár wrote:
>>> Can grub2 recognize such disks?
>> I'm not sure what you're asking here. The physical interface to this drive is USB,
> It is USB mass storage device? If yes, then grub2 should be able to
> normally use. Read its content, etc... You can use "ls" grub command to
> list content of disk with supported filesystem.

Yes, Mass Storage Bulk-Only Transport.

>
>> and it's not designed for general-purpose storage (or booting). That said, if you
>> have some grub2 commands you want me to run against this drive/partition let me know.
> There is also some way for using grub's fs implementation to read disk
> images. It is primary used by grub's automated tests. I do not know
> right now how to use, I need to look grub documentation. But I have
> already used it during implementation of UDF UUID in grub.

Grub is not recognizing my USB drive, i.e. 'ls' does not show usb0 as an option.
I tried 'insmod usb' but that made no difference. Maybe grub does not support my
USB 3.0 host controller, I will retry on a USB2 port when I have a chance.

>>> Also can you check if libparted from git master branch can recognize
>>> such disk? In following commit I added support for recognizing UDF
>>> filesystem in libparted, it is only in git master branch, not released:
>>>
>>> http://git.savannah.gnu.org/cgit/parted.git/commit/?id=8740cfcff3ea839dd6dc8650dec0a466e9870625
>> Build failed:
>>    In file included from pt-tools.c:114:0:
>>    pt-tools.c: In function 'pt_limit_lookup':
>>    pt-limit.gperf:78:1: error: function might be candidate for attribute 'pure' [-Werror=suggest-attribute=pure]
>>
>> If you send me some other SHA to try I can attempt a rebuild.
> Try to use top of master branch. That mentioned commit is already in git
> master.
>
> And if it still produce that error, compile without -Werror flag (or add
> -Wno-error).

I had to configure with CFLAGS=-Wno-error.

It does not recognize Windows-formatted 4K-sector media:
   Disk /dev/sdb: 17.6TB
   Sector size (logical/physical): 4096B/4096B
   Partition Table: msdos
   Disk Flags:

   Number  Start   End     Size    Type     File system  Flags
    1      1049kB  17.6TB  17.6TB  primary


It does recognize mkudffs-formatted media.

>
>>> ISSUE 2: Inability of Windows chkdsk to analyze 4K-sector media
>>>            formatted by mkudffs.
>>> This is really bad :-(
>>>
>>>> It would be possible to work around this by tweaking mkudffs to
>>>> insert dummy BOOT2 components in between the BEA/NSR/TEA:
>>>>
>>>>     0000: 00 42 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .BEA01..........
>>>>     0800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00  .BOOT2..........
>>>>     1000: 00 4e 53 52 30 33 01 00 00 00 00 00 00 00 00 00  .NSR03..........
>>>>     1800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00  .BOOT2..........
>>>>     2000: 00 54 45 41 30 31 01 00 00 00 00 00 00 00 00 00  .TEA01..........
>>>>
>>>> That would introduce a slight ECMA-167 nonconformity, but Linux
>>>> does not object and Windows actually performs better. I would
>>>> have to tweak udffsck though since I believe this could confuse
>>>> its automatic detection of medium block size.
>>> I would like to avoid this hack. If chkdsk is unable to detect such
>>> filesystem, it is really a good idea to let it do try doing filesystem
>>> checks and recovery? You are saying that udfs.sys can recognize such
>>> disk and mount it. I think this should be enough.
>> Fair enough, but it's also reasonable to assume the bugginess is
>> limited to the VRS corner case. AFAIK that's the only place in ECMA-167
>> where there is a difference in layout specific to 4K sectors.
>> With the BOOT2 band-aid chkdsk is able to analyze filesystems on 4Kn media.
> Main problem with this hack is that it breaks detection of valid UDF
> filesystems which use VRS for block size detection. I do not know which
> implementation may use VRS for block size detection, but I do not see
> anything wrong in this approach.

I went through this with udffsck. The VRS is not very helpful in
determining block size because the only time the block size can be
determined conclusively is when the interval between VRS components
is > 2048 bytes. With an interval of 2048 bytes, the only conclusion
that can be drawn is that blocks are no larger than 2048 bytes.

>> I use chkdsk frequently to double-check UDF generation firmware
> Vojtěch wrote in his thesis that MS's chkdsk sometimes put UDF
> filesystem into more broken state as before.

Yes, I have personally experienced this. I don't have chkdsk do
repairs any more. In my case the problem may be that chkdsk
poorly handles the cascading corruption that resulted from this:

     https://lkml.org/lkml/2019/2/8/740

>> I am writing, and also udffsck work-in-progress.
> Have you used some Vojtěch's parts? Or are you writing it from scratch?
>
A udffsck discussion should probably continue in another thread.
Here let me just say that I have been enhancing Vojtěch's code,
in this fork:

   https://github.com/smagnani/udftools

...as time permits. Since winter ended the time I have available
for this has plummeted, so progress is very slow. But this recent
kernel driver patch grew out of work to make sure that udffsck
handles the UDF "file tail" properly:

   https://lkml.org/lkml/2019/6/4/551
   https://lkml.org/lkml/2019/6/30/181

------------------------------------------------------------------------
  Steven J. Magnani               "I claim this network for MARS!
  www.digidescorp.com               Earthling, return my space modulator!"

  #include <standard.disclaimer>