LinuxLists.cc - Moving unmaintained filesystems to staging

[permalink] [raw]

Subject: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)

Hi Matthew.

You probably put your stick into a cave with ancient sleeping dragons :)

Added in linux-m68k mailing list, as they likely have an opinion on how
to treat affs + RDB partition support. Also added in Jens Axboe about
patching that RDB support broken with 2 TB or larger harddisks issue
which had been in Linux kernel for 6 years while a patch exists that to
my testing back then solves the issue.

Matthew Wilcox - 26.04.18, 04:57:
> On Wed, Apr 25, 2018 at 10:30:29PM +0200, David Sterba wrote:
> > I had similar toughts some time ago while browsing the fs/
> > directory.
> > Access to the filesystem images can be reimplemented in FUSE, but
> > other than that, I don't think the in-kernel code would be missed.
> >
> > It's hard to know how many users are there. I was curious eg. about
> > bfs, befs, coda or feevxfs, looked at the last commits and searched
> > around web if there are any mentions or user community. But as long
> > as there's somebody listed in MAINTAINERS, the above are not
> > candidates for moving to staging or deletion.
>
> Yeah, it's pretty sad how few commits some of these filesystems have
> had in recent years. One can argue that they're stable and don't need
> to be fixed because they aren't broken, but I find it hard to believe
> that any of them were better-implemented than ext2 which still sees
> regular bugfixes.

Regarding affs there is a severe issue which is not in affs itself but
in the handling of Rigid Disk Block (RDB) partitions, the Amiga
partitioning standard, which is far more advanced than MBR: It overruns
for 2 TB or larger drives and then wraps over to the beginning of the
drive – I bet you can imagine what happens if you write to an area
larger than 2 TB. I learned this with an external 2TB RDB partitioned
harddisk back then, which I used for Sam440ep (a kind of successor for
old, classic Amiga hardware) backup + some Linux related stuff in
another partition.

Joanne Dow, a developer who developed hdwrench.library which HDToolBox
uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then,
but never officially put it officially through upstreaming as I offered
to make a good description and upstream it through Jens Axboe.

I may take this as a reason to… actually follow through this time,
hopefully remembering all the details in order to provide a meaningful
patch description – but I think mostly I can do just careful copy and
paste. Even tough I believe Joanne Dow´s fix only fixed my bug report
43511, but not 43511 which is more about a safeguarding issue in case of
future overflows, I still think it would be good to go in in case affs +
RDB stays in their current places.

However, in case you move affs to staging, I may be less motivated to do
so, but then I suggest you also move RDB partitioning support to
staging, cause this is the one that is known to be dangerously badly for
2 TB or larger disks. And yeah, I admit I did not follow through with
having that patch upstreamed. Probably I did not want to be responsible
in case my description would not have been absolutely accurate or the
patch breaks something else. I do not have that 2 TB drive anymore and
don´t feel like setting one up in a suitable way in order to go about
this patch, but my testing back then was quite elaborate and I still
feel pretty confident about it.

I totally get your motivation, but I would find it somewhat sad to see
the filesystems you mentioned go into staging. However, as I just shown
clearly, for the user it may be better, cause there may be unfixed
dangerous bugs. FUSE might be an interesting approach, but I bet it will
not solve the maintenance issue. If there is no one maintaining it in
the kernel, I think its unlikely to find someone adapting it to be a
FUSE filesystem and maintaining it. And then I am not aware of FUSE
based partitioning support. (And I think think ideally we´d had a
microkernel and run all filesystems in userspace processes with a
defined set of privileges, but that is simply not Linux as it is.)

Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in
AmigaOS 4.1
https://lkml.org/lkml/2012/6/17/6

Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size
instead of refusing to use it
https://bugzilla.kernel.org/show_bug.cgi?id=43521

Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big,
while OK in AmigaOS 4.1
https://bugzilla.kernel.org/show_bug.cgi?id=43511

I forward the relevant mail of Joanne, in

https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7

I even have the patch in diff format. And I just checked, the issue is
still unpatched as of 4.16.3.

---------- Weitergeleitete Nachricht ----------

Betreff: Re: Partitions: Amiga RDB partition on 2 TB disk way too big,
while OK in AmigaOS 4.1
Datum: Montag, 18. Juni 2012, 03:28:48 CEST
Von: jdow <[email protected]>
An: Martin Steigerwald <[email protected]>
Kopie: Geert Uytterhoeven <[email protected]>, linux-
[email protected], Jens Axboe <[email protected]>, linux-
[email protected]

On 2012/06/17 14:20, Martin Steigerwald wrote:
> Am Sonntag, 17. Juni 2012 schrieb jdow:
>> On 2012/06/17 09:36, Geert Uytterhoeven wrote:
>>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald
> <[email protected]> wrote:
>>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>> | JXFS 64 bit file system
>>>> |
>>>> | With AmigaOS 4.x a new file system has been introduced called
>>>> | JXFS. It is a totally new 64 bit file system that supports
>>>> | partitions up to 16 TB in size. It is a modern journalling file
>>>> | system, which means that it reduces data loss if data writes to
>>>> | the disk are interrupted. It is the fastest and most reliable
>>>> | file system ever created for AmigaOS.
>>>>
>>>> http://www.amigaos.net/content/1/features
>>>>
>>>> Well I asked AmigaOS 4 developers about this issue as well. Lets
see
>>>> what they say about 2 TB limits.
>>>
>>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
>>> 4096?
>>>
>>> block/partitions/amiga.c reads the block size from
>>> RigidDiskBlock.rdb_BlockBytes,
>>> but after conversion to 512-byte blocks, all further calculations
are
>>> done on "int", so it will overflow for disks larger than 2 TiB.
>>>
>>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
>>> bytes per block,
>>> so I'll have to get a deeper look into your RDB first...
> […]
>> Note that the work I did on the Linux RDB code eons ago took full
>> cognizance of the physical and virtual block sizes.
> […]
>> I've asked Martin for a digital copy of his RDBs and what he thinks
the
>> partition(s) should look like. I should also be told whether the disk
>> is supposed to be solely Amiga OSs or not. I gather it's not.
>
> Its all in the bug report. profile-binary.img should be it.
>
> Its small so I attach it.
>
> Meanwhile I try to get the currently supported maximum disk size out
from
> the OS 4 developers. Maybe JXFS is playing tricks that other
filesystems do
> not play or simply has a different implementation.
>
> Thanks,

I've sent Jens a proposed fix changing these lines in amiga.c.
===8<--- was
struct PartitionBlock *pb;
int start_sect, nr_sects, blk, part, res = 0;
int blksize = 1; /* Multiplier for disk block size */
===8<--- becomes changing line 338 into lines 338-339
struct PartitionBlock *pb;
sector_t start_sect, nr_sects;
int blk, part, res = 0;
int blksize = 1; /* Multiplier for disk block size */
===8<---

(I'm working without proper tools at the moment or I'd make a real diff.
Sorry.)

This fix may not be complete. The RDBs contain provisions for describing
the physical disk block size and how many physical blocks are used in a
file system's logical blocks. This MAY not work quite right large
physical
block sizes.

{^_^} Joanne Dow
--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

-------------------------------------------------------------
[…]

> As long as there's someone who can be pestered with bugs, I don't see
> the need to push their baby out of the tree. I'm a bit more nervous
> about hfsplus; if it has users and no maintainer, those users are at
> risk.
>
> Perhaps we could advertise on Craigslist or something ... maintainer
> wanted for LTR.

Thanks,
--
Martin

2018-04-26 10:40:07

[permalink] [raw]

Subject: Re: Moving unmaintained filesystems to staging

Pavel Machek - 26.04.18, 08:11:
> On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
> > Recently ncpfs got moved to staging. Also recently, we had some
> > fuzzer developers report bugs in hfs, which they deem a security
> > hole because Ubuntu attempts to automount an inserted USB device as
> > hfs.
>
> We promise "no-regressions" for code in main repository, no such
> promise for staging. We have quite a lot of code without maintainer.
>
> Moving code to staging means it will get broken -- staging was not
> designed for this. I believe moving anything there is bad idea.
>
> Staging is for ugly code, not for code that needs new maintainter.

Good point.

Moving things in and out to some "unmaintained" directory… I am not sure
about that either. I tend to think that moving code around does not
solve the underlying issue.

Which, according to what I got from Matthew, was that distributors
enable just about every filesystem they can enable which lead to hfs
being used for automounting an USB stick formatted with it.

In the end what may be beneficial would be hinting distributors and
people who compile their own kernels at what features are considered
stable, secure and well tested and what features are not, but how to
determine that? The hint could be some kernel option flag that would be
displayed by make *config. But then it probably needs also a
justification or at least a link to more information.

Actually did ever someone audit the whole kernel source?

Thanks,
--
Martin

2018-04-26 10:47:07

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

(adding debian-68k)

Hi Matthew!

On 04/26/2018 12:28 PM, Martin Steigerwald wrote:
> You probably put your stick into a cave with ancient sleeping dragons :)

Indeed.

> Added in linux-m68k mailing list, as they likely have an opinion on how
> to treat affs + RDB partition support. Also added in Jens Axboe about
> patching that RDB support broken with 2 TB or larger harddisks issue
> which had been in Linux kernel for 6 years while a patch exists that to
> my testing back then solves the issue.

The answer is that we are still very much actively using RDB and AFFS
supoort in the Linux kernel and if you were to remove it, you would
directly hit users.

I know it may sound crazy, but the Linux/m68k port (Atari, Mac, Amiga etc)
is a very actively used and maintained port which just recently received
three new drivers:

> https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?h=4.18/scsi-queue&id=3109e5ae0311e937d49a5325134e50b742ac5f4a
> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=861928f4e60e826cd8871c0c37f4b3d825b8d81d
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/ata/pata_gayle.c?id=9ab27d1d35fda0c5fce624083e92546a8545e7e5

The community around the m68k CPU is constantly developing new hardware
(new accelerator boards, networking cards, IDE controllers etc for the
Amiga and so on). So, the community and the port are anything but dead.

>> Yeah, it's pretty sad how few commits some of these filesystems have
>> had in recent years. One can argue that they're stable and don't need
>> to be fixed because they aren't broken, but I find it hard to believe
>> that any of them were better-implemented than ext2 which still sees
>> regular bugfixes.

Exactly. It works fine as is:

root@elgar:~> uname -a
Linux elgar 4.16.0-rc2-amiga-16784-ga8917fc #650 Mon Mar 5 15:32:52 NZDT 2018 m68k GNU/Linux
root@elgar:~> mount /dev/sda1 /mnt -taffs
root@elgar:~> ls -l /mnt | head
total 0
drwx------ 1 root root 0 Mar 30 2001 Alt
-rw------- 1 root root 1352 Mar 27 1997 Alt.info
drwx------ 1 root root 0 Nov 16 14:39 C
drwx------ 1 root root 0 Mar 27 1997 CS_Fonts
drwx------ 1 root root 0 Mar 27 1997 Classes
-rwx------ 1 root root 1132 Aug 14 1996 Classes.info
drwx------ 1 root root 0 Feb 10 2004 Commodities
-rw------- 1 root root 628 Jan 14 2002 Commodities.info
drwx------ 1 root root 0 Apr 10 1999 CyberTools
root@elgar:~> mount |grep affs
/dev/sda1 on /mnt type affs (rw,relatime,bs=512,volume=:)
root@elgar:~>

There is nothing at the moment that needs fixing.

> Regarding affs there is a severe issue which is not in affs itself but
> in the handling of Rigid Disk Block (RDB) partitions, the Amiga
> partitioning standard, which is far more advanced than MBR: It overruns
> for 2 TB or larger drives and then wraps over to the beginning of the
> drive – I bet you can imagine what happens if you write to an area
> larger than 2 TB. I learned this with an external 2TB RDB partitioned
> harddisk back then, which I used for Sam440ep (a kind of successor for
> old, classic Amiga hardware) backup + some Linux related stuff in
> another partition.

The usecase for RDB-partitioned disks larger than 2 TiB is rather
obscure, so I don't really consider this a problem. Amigas running
Linux can use GPT for the other disks.

> Joanne Dow, a developer who developed hdwrench.library which HDToolBox
> uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then,
> but never officially put it officially through upstreaming as I offered
> to make a good description and upstream it through Jens Axboe.

Could be an idea to do that.

> I may take this as a reason to… actually follow through this time,
> hopefully remembering all the details in order to provide a meaningful
> patch description – but I think mostly I can do just careful copy and
> paste. Even tough I believe Joanne Dow´s fix only fixed my bug report
> 43511, but not 43511 which is more about a safeguarding issue in case of
> future overflows, I still think it would be good to go in in case affs +
> RDB stays in their current places.

That would be cool. Let me know whether you need real Amiga hardware
for testing. We have plenty available.

> However, in case you move affs to staging, I may be less motivated to do
> so, but then I suggest you also move RDB partitioning support to
> staging, cause this is the one that is known to be dangerously badly for
> 2 TB or larger disks. And yeah, I admit I did not follow through with
> having that patch upstreamed. Probably I did not want to be responsible
> in case my description would not have been absolutely accurate or the
> patch breaks something else. I do not have that 2 TB drive anymore and
> don´t feel like setting one up in a suitable way in order to go about
> this patch, but my testing back then was quite elaborate and I still
> feel pretty confident about it.

I wholeheartedly object to move RDB and AFFS to staging and I guess
the Linux/m68k and Debian/m68k community agrees.

> I totally get your motivation, but I would find it somewhat sad to see
> the filesystems you mentioned go into staging. However, as I just shown
> clearly, for the user it may be better, cause there may be unfixed
> dangerous bugs.

No, it's not better for the user if you take something away which
works for 99% of us just fine.

> FUSE might be an interesting approach, but I bet it will
> not solve the maintenance issue. If there is no one maintaining it in
> the kernel, I think its unlikely to find someone adapting it to be a
> FUSE filesystem and maintaining it. And then I am not aware of FUSE
> based partitioning support. (And I think think ideally we´d had a
> microkernel and run all filesystems in userspace processes with a
> defined set of privileges, but that is simply not Linux as it is.)
>
> Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in
> AmigaOS 4.1
> https://lkml.org/lkml/2012/6/17/6
>
> Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size
> instead of refusing to use it
> https://bugzilla.kernel.org/show_bug.cgi?id=43521
>
> Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big,
> while OK in AmigaOS 4.1
> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>
> I forward the relevant mail of Joanne, in
>
> https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7
>
> I even have the patch in diff format. And I just checked, the issue is
> still unpatched as of 4.16.3.

Adrian

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - [email protected]
`. `' Freie Universitaet Berlin - [email protected]
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

2018-04-26 11:03:09

by Christoph Hellwig

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)

On Thu, Apr 26, 2018 at 12:28:40PM +0200, Martin Steigerwald wrote:
> Added in linux-m68k mailing list, as they likely have an opinion on how
> to treat affs + RDB partition support. Also added in Jens Axboe about
> patching that RDB support broken with 2 TB or larger harddisks issue
> which had been in Linux kernel for 6 years while a patch exists that to
> my testing back then solves the issue.

Jens can't do much unless you actually sent said patch to the
linux-block list.

2018-04-26 11:04:28

by David Sterba

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

On Thu, Apr 26, 2018 at 12:45:41PM +0200, John Paul Adrian Glaubitz wrote:
> (adding debian-68k)
>
> Hi Matthew!
>
> On 04/26/2018 12:28 PM, Martin Steigerwald wrote:
> > You probably put your stick into a cave with ancient sleeping dragons :)
>
> Indeed.
>
> > Added in linux-m68k mailing list, as they likely have an opinion on how
> > to treat affs + RDB partition support. Also added in Jens Axboe about
> > patching that RDB support broken with 2 TB or larger harddisks issue
> > which had been in Linux kernel for 6 years while a patch exists that to
> > my testing back then solves the issue.
>
> The answer is that we are still very much actively using RDB and AFFS
> supoort in the Linux kernel and if you were to remove it, you would
> directly hit users.

Based on that I think removing affs will not happen, but the upstream
maintenance status should be updated accordingly.

> I know it may sound crazy, but the Linux/m68k port (Atari, Mac, Amiga etc)
> is a very actively used and maintained port which just recently received
> three new drivers:
>
> > https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?h=4.18/scsi-queue&id=3109e5ae0311e937d49a5325134e50b742ac5f4a
> > https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=861928f4e60e826cd8871c0c37f4b3d825b8d81d
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/ata/pata_gayle.c?id=9ab27d1d35fda0c5fce624083e92546a8545e7e5
>
> The community around the m68k CPU is constantly developing new hardware
> (new accelerator boards, networking cards, IDE controllers etc for the
> Amiga and so on). So, the community and the port are anything but dead.
>
> > > Yeah, it's pretty sad how few commits some of these filesystems have
> > > had in recent years. One can argue that they're stable and don't need
> > > to be fixed because they aren't broken, but I find it hard to believe
> > > that any of them were better-implemented than ext2 which still sees
> > > regular bugfixes.
>
> Exactly. It works fine as is:
...
> There is nothing at the moment that needs fixing.

So, I'm willing to act as upstream maintainer for affs, send pull
requests with fixes if you ever need that (unless you find someone
else).

2018-04-26 11:07:40

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

On 04/26/2018 12:59 PM, David Sterba wrote:
>> The answer is that we are still very much actively using RDB and AFFS
>> supoort in the Linux kernel and if you were to remove it, you would
>> directly hit users.
>
> Based on that I think removing affs will not happen, but the upstream
> maintenance status should be updated accordingly.

As a fellow SUSE employee, I'm very happy to hear that someone
from SUSE is picking up the work <3. FWIW, Andreas Schwab from
SUSE maintains an internal port of openSUSE for m68k :).

>> Exactly. It works fine as is:
> ...
>> There is nothing at the moment that needs fixing.
>
> So, I'm willing to act as upstream maintainer for affs, send pull
> requests with fixes if you ever need that (unless you find someone
> else).

Thanks a thousand times. If we happen to meet at a SUSE event,
I'll invite you to beverage of your choice ;-).

Thanks!
Adrian

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - [email protected]
`. `' Freie Universitaet Berlin - [email protected]
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

2018-04-26 11:10:06

by Geert Uytterhoeven

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)

Hi Martin,

CC jdow

On Thu, Apr 26, 2018 at 12:28 PM, Martin Steigerwald
<[email protected]> wrote:
> You probably put your stick into a cave with ancient sleeping dragons :)
>
> Added in linux-m68k mailing list, as they likely have an opinion on how
> to treat affs + RDB partition support. Also added in Jens Axboe about
> patching that RDB support broken with 2 TB or larger harddisks issue
> which had been in Linux kernel for 6 years while a patch exists that to
> my testing back then solves the issue.

While non-native Linux filesystem support (e.g. affs/isofs/...) could be
handled by FUSE, moving RDB partition support to staging is not an option,
as it is the only partitioning scheme that Amigas can boot from.

If there are bugs in the RDB parser that people run into, they should
be fixed.
If there are limitations in the RDB format on large disks, that's still not
a reason to move it to staging (hi msdos partitioning!).

> Matthew Wilcox - 26.04.18, 04:57:
>> On Wed, Apr 25, 2018 at 10:30:29PM +0200, David Sterba wrote:
>> > I had similar toughts some time ago while browsing the fs/
>> > directory.
>> > Access to the filesystem images can be reimplemented in FUSE, but
>> > other than that, I don't think the in-kernel code would be missed.
>> >
>> > It's hard to know how many users are there. I was curious eg. about
>> > bfs, befs, coda or feevxfs, looked at the last commits and searched
>> > around web if there are any mentions or user community. But as long
>> > as there's somebody listed in MAINTAINERS, the above are not
>> > candidates for moving to staging or deletion.
>>
>> Yeah, it's pretty sad how few commits some of these filesystems have
>> had in recent years. One can argue that they're stable and don't need
>> to be fixed because they aren't broken, but I find it hard to believe
>> that any of them were better-implemented than ext2 which still sees
>> regular bugfixes.
>
> Regarding affs there is a severe issue which is not in affs itself but
> in the handling of Rigid Disk Block (RDB) partitions, the Amiga
> partitioning standard, which is far more advanced than MBR: It overruns
> for 2 TB or larger drives and then wraps over to the beginning of the
> drive – I bet you can imagine what happens if you write to an area
> larger than 2 TB. I learned this with an external 2TB RDB partitioned
> harddisk back then, which I used for Sam440ep (a kind of successor for
> old, classic Amiga hardware) backup + some Linux related stuff in
> another partition.
>
> Joanne Dow, a developer who developed hdwrench.library which HDToolBox
> uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then,
> but never officially put it officially through upstreaming as I offered
> to make a good description and upstream it through Jens Axboe.
>
> I may take this as a reason to… actually follow through this time,
> hopefully remembering all the details in order to provide a meaningful
> patch description – but I think mostly I can do just careful copy and
> paste. Even tough I believe Joanne Dow´s fix only fixed my bug report
> 43511, but not 43511 which is more about a safeguarding issue in case of
> future overflows, I still think it would be good to go in in case affs +
> RDB stays in their current places.
>
> However, in case you move affs to staging, I may be less motivated to do
> so, but then I suggest you also move RDB partitioning support to
> staging, cause this is the one that is known to be dangerously badly for
> 2 TB or larger disks. And yeah, I admit I did not follow through with
> having that patch upstreamed. Probably I did not want to be responsible
> in case my description would not have been absolutely accurate or the
> patch breaks something else. I do not have that 2 TB drive anymore and
> don´t feel like setting one up in a suitable way in order to go about
> this patch, but my testing back then was quite elaborate and I still
> feel pretty confident about it.
>
> I totally get your motivation, but I would find it somewhat sad to see
> the filesystems you mentioned go into staging. However, as I just shown
> clearly, for the user it may be better, cause there may be unfixed
> dangerous bugs. FUSE might be an interesting approach, but I bet it will
> not solve the maintenance issue. If there is no one maintaining it in
> the kernel, I think its unlikely to find someone adapting it to be a
> FUSE filesystem and maintaining it. And then I am not aware of FUSE
> based partitioning support. (And I think think ideally we´d had a
> microkernel and run all filesystems in userspace processes with a
> defined set of privileges, but that is simply not Linux as it is.)
>
> Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in
> AmigaOS 4.1
> https://lkml.org/lkml/2012/6/17/6
>
> Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size
> instead of refusing to use it
> https://bugzilla.kernel.org/show_bug.cgi?id=43521
>
> Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big,
> while OK in AmigaOS 4.1
> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>
> I forward the relevant mail of Joanne, in
>
> https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7
>
> I even have the patch in diff format. And I just checked, the issue is
> still unpatched as of 4.16.3.
>
> ---------- Weitergeleitete Nachricht ----------
>
> Betreff: Re: Partitions: Amiga RDB partition on 2 TB disk way too big,
> while OK in AmigaOS 4.1
> Datum: Montag, 18. Juni 2012, 03:28:48 CEST
> Von: jdow <[email protected]>
> An: Martin Steigerwald <[email protected]>
> Kopie: Geert Uytterhoeven <[email protected]>, linux-
> [email protected], Jens Axboe <[email protected]>, linux-
> [email protected]
>
> On 2012/06/17 14:20, Martin Steigerwald wrote:
>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>> On 2012/06/17 09:36, Geert Uytterhoeven wrote:
>>>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald
>> <[email protected]> wrote:
>>>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>>> | JXFS 64 bit file system
>>>>> |
>>>>> | With AmigaOS 4.x a new file system has been introduced called
>>>>> | JXFS. It is a totally new 64 bit file system that supports
>>>>> | partitions up to 16 TB in size. It is a modern journalling file
>>>>> | system, which means that it reduces data loss if data writes to
>>>>> | the disk are interrupted. It is the fastest and most reliable
>>>>> | file system ever created for AmigaOS.
>>>>>
>>>>> http://www.amigaos.net/content/1/features
>>>>>
>>>>> Well I asked AmigaOS 4 developers about this issue as well. Lets
> see
>>>>> what they say about 2 TB limits.
>>>>
>>>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
>>>> 4096?
>>>>
>>>> block/partitions/amiga.c reads the block size from
>>>> RigidDiskBlock.rdb_BlockBytes,
>>>> but after conversion to 512-byte blocks, all further calculations
> are
>>>> done on "int", so it will overflow for disks larger than 2 TiB.
>>>>
>>>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
>>>> bytes per block,
>>>> so I'll have to get a deeper look into your RDB first...
>> […]
>>> Note that the work I did on the Linux RDB code eons ago took full
>>> cognizance of the physical and virtual block sizes.
>> […]
>>> I've asked Martin for a digital copy of his RDBs and what he thinks
> the
>>> partition(s) should look like. I should also be told whether the disk
>>> is supposed to be solely Amiga OSs or not. I gather it's not.
>>
>> Its all in the bug report. profile-binary.img should be it.
>>
>> Its small so I attach it.
>>
>> Meanwhile I try to get the currently supported maximum disk size out
> from
>> the OS 4 developers. Maybe JXFS is playing tricks that other
> filesystems do
>> not play or simply has a different implementation.
>>
>> Thanks,
>
> I've sent Jens a proposed fix changing these lines in amiga.c.
> ===8<--- was
> struct PartitionBlock *pb;
> int start_sect, nr_sects, blk, part, res = 0;
> int blksize = 1; /* Multiplier for disk block size */
> ===8<--- becomes changing line 338 into lines 338-339
> struct PartitionBlock *pb;
> sector_t start_sect, nr_sects;
> int blk, part, res = 0;
> int blksize = 1; /* Multiplier for disk block size */
> ===8<---
>
> (I'm working without proper tools at the moment or I'd make a real diff.
> Sorry.)
>
> This fix may not be complete. The RDBs contain provisions for describing
> the physical disk block size and how many physical blocks are used in a
> file system's logical blocks. This MAY not work quite right large
> physical
> block sizes.
>
> {^_^} Joanne Dow
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> -------------------------------------------------------------
> […]
>
>> As long as there's someone who can be pestered with bugs, I don't see
>> the need to push their baby out of the tree. I'm a bit more nervous
>> about hfsplus; if it has users and no maintainer, those users are at
>> risk.
>>
>> Perhaps we could advertise on Craigslist or something ... maintainer
>> wanted for LTR.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

2018-04-26 23:59:08

by Finn Thain

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)

On Thu, 26 Apr 2018, Geert Uytterhoeven wrote:

>
> While non-native Linux filesystem support (e.g. affs/isofs/...) could be
> handled by FUSE

Moving to FUSE is a great divide-and-conquer strategy for those who just
want the code to die and don't care about any of the data in that format.

If there is a maintainence burden that can be shared then it should be
shared -- until it can be established that there is no data of value in
that format.

> moving RDB partition support to staging is not an option, as it is the
> only partitioning scheme that Amigas can boot from.
>

Whether or not the original hardware is in use is mostly irrelevant.

As long as the old format is accessible using current hardware, the data
in that format remains accessible (to archivists, to curators, to your
decendents, etc).

> If there are bugs in the RDB parser that people run into, they should be
> fixed. If there are limitations in the RDB format on large disks, that's
> still not a reason to move it to staging (hi msdos partitioning!).
>

--

2018-04-27 01:11:49

by Luis Chamberlain

[permalink] [raw]

Subject: Re: Moving unmaintained filesystems to staging

On Wed, Apr 25, 2018 at 11:11 PM, Pavel Machek <[email protected]> wrote:
> On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
>> Recently ncpfs got moved to staging. Also recently, we had some fuzzer
>> developers report bugs in hfs, which they deem a security hole because
>> Ubuntu attempts to automount an inserted USB device as hfs.
>
> We promise "no-regressions" for code in main repository, no such
> promise for staging. We have quite a lot of code without maintainer.
>
> Moving code to staging means it will get broken -- staging was not
> designed for this. I believe moving anything there is bad idea.
>
> Staging is for ugly code, not for code that needs new maintainter.

Staging is also a hospice for drivers on their way out. Can some of
these be eventually be axed?

Luis

2018-04-27 02:13:21

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)

Hi Geert,

test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
indicate the RDB parser bug is fixed by the patch given there, so if
Martin now submits the patch, all should be well?

(MSDOS partition support is not the only one with limitations - the
SGI partition parser also uses int types. Brings back memories of /me
hacking a large disk into many small partitions because Irix couldn't
digest it whole!)

Going to look into Atari partition format limitations now ...

Cheers,

Michael

On Thu, Apr 26, 2018 at 11:08 PM, Geert Uytterhoeven
<[email protected]> wrote:
> Hi Martin,
>
> CC jdow
>
> On Thu, Apr 26, 2018 at 12:28 PM, Martin Steigerwald
> <[email protected]> wrote:
>> You probably put your stick into a cave with ancient sleeping dragons :)
>>
>> Added in linux-m68k mailing list, as they likely have an opinion on how
>> to treat affs + RDB partition support. Also added in Jens Axboe about
>> patching that RDB support broken with 2 TB or larger harddisks issue
>> which had been in Linux kernel for 6 years while a patch exists that to
>> my testing back then solves the issue.
>
> While non-native Linux filesystem support (e.g. affs/isofs/...) could be
> handled by FUSE, moving RDB partition support to staging is not an option,
> as it is the only partitioning scheme that Amigas can boot from.
>
> If there are bugs in the RDB parser that people run into, they should
> be fixed.
> If there are limitations in the RDB format on large disks, that's still not
> a reason to move it to staging (hi msdos partitioning!).
>
>> Matthew Wilcox - 26.04.18, 04:57:
>>> On Wed, Apr 25, 2018 at 10:30:29PM +0200, David Sterba wrote:
>>> > I had similar toughts some time ago while browsing the fs/
>>> > directory.
>>> > Access to the filesystem images can be reimplemented in FUSE, but
>>> > other than that, I don't think the in-kernel code would be missed.
>>> >
>>> > It's hard to know how many users are there. I was curious eg. about
>>> > bfs, befs, coda or feevxfs, looked at the last commits and searched
>>> > around web if there are any mentions or user community. But as long
>>> > as there's somebody listed in MAINTAINERS, the above are not
>>> > candidates for moving to staging or deletion.
>>>
>>> Yeah, it's pretty sad how few commits some of these filesystems have
>>> had in recent years. One can argue that they're stable and don't need
>>> to be fixed because they aren't broken, but I find it hard to believe
>>> that any of them were better-implemented than ext2 which still sees
>>> regular bugfixes.
>>
>> Regarding affs there is a severe issue which is not in affs itself but
>> in the handling of Rigid Disk Block (RDB) partitions, the Amiga
>> partitioning standard, which is far more advanced than MBR: It overruns
>> for 2 TB or larger drives and then wraps over to the beginning of the
>> drive – I bet you can imagine what happens if you write to an area
>> larger than 2 TB. I learned this with an external 2TB RDB partitioned
>> harddisk back then, which I used for Sam440ep (a kind of successor for
>> old, classic Amiga hardware) backup + some Linux related stuff in
>> another partition.
>>
>> Joanne Dow, a developer who developed hdwrench.library which HDToolBox
>> uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then,
>> but never officially put it officially through upstreaming as I offered
>> to make a good description and upstream it through Jens Axboe.
>>
>> I may take this as a reason to… actually follow through this time,
>> hopefully remembering all the details in order to provide a meaningful
>> patch description – but I think mostly I can do just careful copy and
>> paste. Even tough I believe Joanne Dow´s fix only fixed my bug report
>> 43511, but not 43511 which is more about a safeguarding issue in case of
>> future overflows, I still think it would be good to go in in case affs +
>> RDB stays in their current places.
>>
>> However, in case you move affs to staging, I may be less motivated to do
>> so, but then I suggest you also move RDB partitioning support to
>> staging, cause this is the one that is known to be dangerously badly for
>> 2 TB or larger disks. And yeah, I admit I did not follow through with
>> having that patch upstreamed. Probably I did not want to be responsible
>> in case my description would not have been absolutely accurate or the
>> patch breaks something else. I do not have that 2 TB drive anymore and
>> don´t feel like setting one up in a suitable way in order to go about
>> this patch, but my testing back then was quite elaborate and I still
>> feel pretty confident about it.
>>
>> I totally get your motivation, but I would find it somewhat sad to see
>> the filesystems you mentioned go into staging. However, as I just shown
>> clearly, for the user it may be better, cause there may be unfixed
>> dangerous bugs. FUSE might be an interesting approach, but I bet it will
>> not solve the maintenance issue. If there is no one maintaining it in
>> the kernel, I think its unlikely to find someone adapting it to be a
>> FUSE filesystem and maintaining it. And then I am not aware of FUSE
>> based partitioning support. (And I think think ideally we´d had a
>> microkernel and run all filesystems in userspace processes with a
>> defined set of privileges, but that is simply not Linux as it is.)
>>
>> Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in
>> AmigaOS 4.1
>> https://lkml.org/lkml/2012/6/17/6
>>
>> Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size
>> instead of refusing to use it
>> https://bugzilla.kernel.org/show_bug.cgi?id=43521
>>
>> Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big,
>> while OK in AmigaOS 4.1
>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>
>> I forward the relevant mail of Joanne, in
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7
>>
>> I even have the patch in diff format. And I just checked, the issue is
>> still unpatched as of 4.16.3.
>>
>> ---------- Weitergeleitete Nachricht ----------
>>
>> Betreff: Re: Partitions: Amiga RDB partition on 2 TB disk way too big,
>> while OK in AmigaOS 4.1
>> Datum: Montag, 18. Juni 2012, 03:28:48 CEST
>> Von: jdow <[email protected]>
>> An: Martin Steigerwald <[email protected]>
>> Kopie: Geert Uytterhoeven <[email protected]>, linux-
>> [email protected], Jens Axboe <[email protected]>, linux-
>> [email protected]
>>
>> On 2012/06/17 14:20, Martin Steigerwald wrote:
>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>> On 2012/06/17 09:36, Geert Uytterhoeven wrote:
>>>>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald
>>> <[email protected]> wrote:
>>>>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>>>> | JXFS 64 bit file system
>>>>>> |
>>>>>> | With AmigaOS 4.x a new file system has been introduced called
>>>>>> | JXFS. It is a totally new 64 bit file system that supports
>>>>>> | partitions up to 16 TB in size. It is a modern journalling file
>>>>>> | system, which means that it reduces data loss if data writes to
>>>>>> | the disk are interrupted. It is the fastest and most reliable
>>>>>> | file system ever created for AmigaOS.
>>>>>>
>>>>>> http://www.amigaos.net/content/1/features
>>>>>>
>>>>>> Well I asked AmigaOS 4 developers about this issue as well. Lets
>> see
>>>>>> what they say about 2 TB limits.
>>>>>
>>>>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
>>>>> 4096?
>>>>>
>>>>> block/partitions/amiga.c reads the block size from
>>>>> RigidDiskBlock.rdb_BlockBytes,
>>>>> but after conversion to 512-byte blocks, all further calculations
>> are
>>>>> done on "int", so it will overflow for disks larger than 2 TiB.
>>>>>
>>>>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
>>>>> bytes per block,
>>>>> so I'll have to get a deeper look into your RDB first...
>>> […]
>>>> Note that the work I did on the Linux RDB code eons ago took full
>>>> cognizance of the physical and virtual block sizes.
>>> […]
>>>> I've asked Martin for a digital copy of his RDBs and what he thinks
>> the
>>>> partition(s) should look like. I should also be told whether the disk
>>>> is supposed to be solely Amiga OSs or not. I gather it's not.
>>>
>>> Its all in the bug report. profile-binary.img should be it.
>>>
>>> Its small so I attach it.
>>>
>>> Meanwhile I try to get the currently supported maximum disk size out
>> from
>>> the OS 4 developers. Maybe JXFS is playing tricks that other
>> filesystems do
>>> not play or simply has a different implementation.
>>>
>>> Thanks,
>>
>> I've sent Jens a proposed fix changing these lines in amiga.c.
>> ===8<--- was
>> struct PartitionBlock *pb;
>> int start_sect, nr_sects, blk, part, res = 0;
>> int blksize = 1; /* Multiplier for disk block size */
>> ===8<--- becomes changing line 338 into lines 338-339
>> struct PartitionBlock *pb;
>> sector_t start_sect, nr_sects;
>> int blk, part, res = 0;
>> int blksize = 1; /* Multiplier for disk block size */
>> ===8<---
>>
>> (I'm working without proper tools at the moment or I'd make a real diff.
>> Sorry.)
>>
>> This fix may not be complete. The RDBs contain provisions for describing
>> the physical disk block size and how many physical blocks are used in a
>> file system's logical blocks. This MAY not work quite right large
>> physical
>> block sizes.
>>
>> {^_^} Joanne Dow
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>> -------------------------------------------------------------
>> […]
>>
>>> As long as there's someone who can be pestered with bugs, I don't see
>>> the need to push their baby out of the tree. I'm a bit more nervous
>>> about hfsplus; if it has users and no maintainer, those users are at
>>> risk.
>>>
>>> Perhaps we could advertise on Craigslist or something ... maintainer
>>> wanted for LTR.
>
> Gr{oetje,eeting}s,
>
> Geert
>
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
>
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
> -- Linus Torvalds
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2018-04-27 08:02:30

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)

Geert Uytterhoeven - 26.04.18, 13:08:
> On Thu, Apr 26, 2018 at 12:28 PM, Martin Steigerwald
>
> <[email protected]> wrote:
> > You probably put your stick into a cave with ancient sleeping
> > dragons
> >
> > Added in linux-m68k mailing list, as they likely have an opinion on
> > how to treat affs + RDB partition support. Also added in Jens Axboe
> > about patching that RDB support broken with 2 TB or larger
> > harddisks issue which had been in Linux kernel for 6 years while a
> > patch exists that to my testing back then solves the issue.
[…]
> If there are bugs in the RDB parser that people run into, they should
> be fixed.
> If there are limitations in the RDB format on large disks, that's
> still not a reason to move it to staging (hi msdos partitioning!).

What I ran into was *not* a limitation in the RDB format, but a bug in
the Linux implementation on it. After Joanne Dow´s change the 2 TB disk
was detected and handled properly by Linux. Also AmigaOS 4.x handles
those disks just well and I think also AmigaOS 3.1/3.5/3.9 supported
them, but I am not sure on the details on that, it has been a long time
since I last booted one of my Amiga systems.

Many classic Amiga users may not deal with such large disks, but AmigaOS
4 users probably still, and some of them may run Linux on their PowerPC
motherboards as well.

So I think the issue is worth fixing and am looking into submitting the
fix, which looks pretty straight-forward to me upstream unless someone
bets me to it.

Thanks,
--
Martin

2018-04-27 08:51:32

by jdow

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

On 20180426 16:56, Finn Thain wrote:
> On Thu, 26 Apr 2018, Geert Uytterhoeven wrote:
>
>>
>> While non-native Linux filesystem support (e.g. affs/isofs/...) could be
>> handled by FUSE
>
> Moving to FUSE is a great divide-and-conquer strategy for those who just
> want the code to die and don't care about any of the data in that format.
>
> If there is a maintainence burden that can be shared then it should be
> shared -- until it can be established that there is no data of value in
> that format.
>
>> moving RDB partition support to staging is not an option, as it is the
>> only partitioning scheme that Amigas can boot from.
>>
>
> Whether or not the original hardware is in use is mostly irrelevant.
>
> As long as the old format is accessible using current hardware, the data
> in that format remains accessible (to archivists, to curators, to your
> decendents, etc).
>
>> If there are bugs in the RDB parser that people run into, they should be
>> fixed. If there are limitations in the RDB format on large disks, that's
>> still not a reason to move it to staging (hi msdos partitioning!).
>>

This intrepid cyberunit is inclined to suggest that understanding the RDBs can
go a long way towards defining if there is a bug somewhere and whether it is in
the RDB description or its misuse.

There are some things RDBs can do that REALLY REALLY don't make sense until you
run across the situation which called for it creation. There are two variables
that suggest some blocks at the beginning and the end, respectively, of a
partition are not accessible by the OS. I have used these facilities to
"interleave" partitions and RDBs. I have built a disk which reserved about 128
512 byte blocks for RDBs plus filesystem code (which probably should be
abandoned) which embedded the RDBs describing the partition within the
partition. Then I reserved space at the end of the partition and embedded a
second partition in that space. As absurd as it sounds this had at one time a
decent use case. Disk space was an expensive premium in those days so wasting
space to get nice integer numbers in the disk description, which was phony for a
hard disk in any case, we allowed any numbers and if that went past the end of
the disk we reserved the necessary space so that it would never be used. The
space at the beginning of a partition was needed in any case because a one block
partition signature needed space at the start of the partition. It held the
filesystem's signature, OFS, AFS, SFS, etc.

There is also a good reason for allowing the anchor for the RDBs to start in any
of the first 16 blocks with a recommendation not to use block 0 as other FSs
used that. And we wanted to accommodate at least two different partition
description technologies to work on the disk. My code always placed the RDBs at
block 3.

I hope passing along some of this history will mitigate some fo the feelings
that RDBs are inherently flawed or full of bugs or whatnot. (Full pf security
holes is another story. DriveInit code and filesystem code have worried me from
day one.)

{^_^} Joanne

2018-04-27 08:51:41

by jdow

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

OK guys, I am working from th ememory of a 74 year old with a severe head cold.
The filesystems were always limited to 2 TB since the seek command was signed 32
bit integer number of BYTEs. So block size has nothing to do with this 2TB
issue. That's something else in the descriptions.

The RDBs grew a block size and sector size concept in describing partitions. One
had to be a power of two multiple of the other, as I recall. The relationship
worked two ways. If you had a Fujitsu Magneto Optical drive with 2k physical
sectors you could still allocate space in virtual sectors, block sizes, of 512
bytes. Or you could use much larger blocks than sector size to cut down on the
unallocated blocks list. Of course, all OSs do that these days. With a single
unsigned 16 bit integer your address space would fill up a 32 GB with 512 byte
blocks. We could see this would blow out so we decribed block size and disk size
in blocks with 32 bit integers giving 4 TB of 1 BYTE blocks. Supposing the file
sizes stored are such as to support an 8k block size that means describing a
very large disk should be very easy, even if block size was a 16 bit unsigned
integer.

To carry this forward more accurately I'll have to dig up some old archival data
and see what it actually says. This isn't going to happen when I sit here
sniffling. Maybe in a few days I can dredge up the actual numbers. I seem to
remember at the time that without a 64 bit OS the disk addressing limit was 2
TB, since seeks were signed integers. The disk descriptions could be
considerably larger.

And before I forget there are two features of the RDBs that I heartily recommend
never implementing on Linux. They were good ideas at the time; but, times
changed. The RDBs are capable of storing a filesystem driver and some drive init
code for the plugin disk driver card. That is giving malware authors entirely
goo easy a shot at owning a machine. Martin S., I would strongly suggest that
going forward those two capabilities be removed from the RDB readers in AmigaOS
as well as Linux OS.

I also note that at one time the RDB parsing facility in Linux benefitted from
my hands on the code and handled the block size/sector size issues very nicely.
I had an 18 GB disk I wanted to be able to read on Linux. I seem to remember
somebody at one time had a "this doesn't make any real world sense" event and
had to be reminded that the real world was wider than he envisioned. I don't
know what has happened since then.

{^_^} Joanne Dow

On 20180426 04:08, Geert Uytterhoeven wrote:
> Hi Martin,
>
> CC jdow
>
> On Thu, Apr 26, 2018 at 12:28 PM, Martin Steigerwald
> <[email protected]> wrote:
>> You probably put your stick into a cave with ancient sleeping dragons :)
>>
>> Added in linux-m68k mailing list, as they likely have an opinion on how
>> to treat affs + RDB partition support. Also added in Jens Axboe about
>> patching that RDB support broken with 2 TB or larger harddisks issue
>> which had been in Linux kernel for 6 years while a patch exists that to
>> my testing back then solves the issue.
>
> While non-native Linux filesystem support (e.g. affs/isofs/...) could be
> handled by FUSE, moving RDB partition support to staging is not an option,
> as it is the only partitioning scheme that Amigas can boot from.
>
> If there are bugs in the RDB parser that people run into, they should
> be fixed.
> If there are limitations in the RDB format on large disks, that's still not
> a reason to move it to staging (hi msdos partitioning!).
>
>> Matthew Wilcox - 26.04.18, 04:57:
>>> On Wed, Apr 25, 2018 at 10:30:29PM +0200, David Sterba wrote:
>>>> I had similar toughts some time ago while browsing the fs/
>>>> directory.
>>>> Access to the filesystem images can be reimplemented in FUSE, but
>>>> other than that, I don't think the in-kernel code would be missed.
>>>>
>>>> It's hard to know how many users are there. I was curious eg. about
>>>> bfs, befs, coda or feevxfs, looked at the last commits and searched
>>>> around web if there are any mentions or user community. But as long
>>>> as there's somebody listed in MAINTAINERS, the above are not
>>>> candidates for moving to staging or deletion.
>>>
>>> Yeah, it's pretty sad how few commits some of these filesystems have
>>> had in recent years. One can argue that they're stable and don't need
>>> to be fixed because they aren't broken, but I find it hard to believe
>>> that any of them were better-implemented than ext2 which still sees
>>> regular bugfixes.
>>
>> Regarding affs there is a severe issue which is not in affs itself but
>> in the handling of Rigid Disk Block (RDB) partitions, the Amiga
>> partitioning standard, which is far more advanced than MBR: It overruns
>> for 2 TB or larger drives and then wraps over to the beginning of the
>> drive – I bet you can imagine what happens if you write to an area
>> larger than 2 TB. I learned this with an external 2TB RDB partitioned
>> harddisk back then, which I used for Sam440ep (a kind of successor for
>> old, classic Amiga hardware) backup + some Linux related stuff in
>> another partition.
>>
>> Joanne Dow, a developer who developed hdwrench.library which HDToolBox
>> uses for partitioning in AmigaOS 3.5/3.9, provided a patch back then,
>> but never officially put it officially through upstreaming as I offered
>> to make a good description and upstream it through Jens Axboe.
>>
>> I may take this as a reason to… actually follow through this time,
>> hopefully remembering all the details in order to provide a meaningful
>> patch description – but I think mostly I can do just careful copy and
>> paste. Even tough I believe Joanne Dow´s fix only fixed my bug report
>> 43511, but not 43511 which is more about a safeguarding issue in case of
>> future overflows, I still think it would be good to go in in case affs +
>> RDB stays in their current places.
>>
>> However, in case you move affs to staging, I may be less motivated to do
>> so, but then I suggest you also move RDB partitioning support to
>> staging, cause this is the one that is known to be dangerously badly for
>> 2 TB or larger disks. And yeah, I admit I did not follow through with
>> having that patch upstreamed. Probably I did not want to be responsible
>> in case my description would not have been absolutely accurate or the
>> patch breaks something else. I do not have that 2 TB drive anymore and
>> don´t feel like setting one up in a suitable way in order to go about
>> this patch, but my testing back then was quite elaborate and I still
>> feel pretty confident about it.
>>
>> I totally get your motivation, but I would find it somewhat sad to see
>> the filesystems you mentioned go into staging. However, as I just shown
>> clearly, for the user it may be better, cause there may be unfixed
>> dangerous bugs. FUSE might be an interesting approach, but I bet it will
>> not solve the maintenance issue. If there is no one maintaining it in
>> the kernel, I think its unlikely to find someone adapting it to be a
>> FUSE filesystem and maintaining it. And then I am not aware of FUSE
>> based partitioning support. (And I think think ideally we´d had a
>> microkernel and run all filesystems in userspace processes with a
>> defined set of privileges, but that is simply not Linux as it is.)
>>
>> Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in
>> AmigaOS 4.1
>> https://lkml.org/lkml/2012/6/17/6
>>
>> Bug 43521 - Amiga RDB partitions: truncates miscalculated partition size
>> instead of refusing to use it
>> https://bugzilla.kernel.org/show_bug.cgi?id=43521
>>
>> Bug 43511 - Partitions: Amiga RDB partition on 2 TB disk way too big,
>> while OK in AmigaOS 4.1
>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>
>> I forward the relevant mail of Joanne, in
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=43511#c7
>>
>> I even have the patch in diff format. And I just checked, the issue is
>> still unpatched as of 4.16.3.
>>
>> ---------- Weitergeleitete Nachricht ----------
>>
>> Betreff: Re: Partitions: Amiga RDB partition on 2 TB disk way too big,
>> while OK in AmigaOS 4.1
>> Datum: Montag, 18. Juni 2012, 03:28:48 CEST
>> Von: jdow <[email protected]>
>> An: Martin Steigerwald <[email protected]>
>> Kopie: Geert Uytterhoeven <[email protected]>, linux-
>> [email protected], Jens Axboe <[email protected]>, linux-
>> [email protected]
>>
>> On 2012/06/17 14:20, Martin Steigerwald wrote:
>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>> On 2012/06/17 09:36, Geert Uytterhoeven wrote:
>>>>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald
>>> <[email protected]> wrote:
>>>>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>>>> | JXFS 64 bit file system
>>>>>> |
>>>>>> | With AmigaOS 4.x a new file system has been introduced called
>>>>>> | JXFS. It is a totally new 64 bit file system that supports
>>>>>> | partitions up to 16 TB in size. It is a modern journalling file
>>>>>> | system, which means that it reduces data loss if data writes to
>>>>>> | the disk are interrupted. It is the fastest and most reliable
>>>>>> | file system ever created for AmigaOS.
>>>>>>
>>>>>> http://www.amigaos.net/content/1/features
>>>>>>
>>>>>> Well I asked AmigaOS 4 developers about this issue as well. Lets
>> see
>>>>>> what they say about 2 TB limits.
>>>>>
>>>>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
>>>>> 4096?
>>>>>
>>>>> block/partitions/amiga.c reads the block size from
>>>>> RigidDiskBlock.rdb_BlockBytes,
>>>>> but after conversion to 512-byte blocks, all further calculations
>> are
>>>>> done on "int", so it will overflow for disks larger than 2 TiB.
>>>>>
>>>>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
>>>>> bytes per block,
>>>>> so I'll have to get a deeper look into your RDB first...
>>> […]
>>>> Note that the work I did on the Linux RDB code eons ago took full
>>>> cognizance of the physical and virtual block sizes.
>>> […]
>>>> I've asked Martin for a digital copy of his RDBs and what he thinks
>> the
>>>> partition(s) should look like. I should also be told whether the disk
>>>> is supposed to be solely Amiga OSs or not. I gather it's not.
>>>
>>> Its all in the bug report. profile-binary.img should be it.
>>>
>>> Its small so I attach it.
>>>
>>> Meanwhile I try to get the currently supported maximum disk size out
>> from
>>> the OS 4 developers. Maybe JXFS is playing tricks that other
>> filesystems do
>>> not play or simply has a different implementation.
>>>
>>> Thanks,
>>
>> I've sent Jens a proposed fix changing these lines in amiga.c.
>> ===8<--- was
>> struct PartitionBlock *pb;
>> int start_sect, nr_sects, blk, part, res = 0;
>> int blksize = 1; /* Multiplier for disk block size */
>> ===8<--- becomes changing line 338 into lines 338-339
>> struct PartitionBlock *pb;
>> sector_t start_sect, nr_sects;
>> int blk, part, res = 0;
>> int blksize = 1; /* Multiplier for disk block size */
>> ===8<---
>>
>> (I'm working without proper tools at the moment or I'd make a real diff.
>> Sorry.)
>>
>> This fix may not be complete. The RDBs contain provisions for describing
>> the physical disk block size and how many physical blocks are used in a
>> file system's logical blocks. This MAY not work quite right large
>> physical
>> block sizes.
>>
>> {^_^} Joanne Dow
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>> -------------------------------------------------------------
>> […]
>>
>>> As long as there's someone who can be pestered with bugs, I don't see
>>> the need to push their baby out of the tree. I'm a bit more nervous
>>> about hfsplus; if it has users and no maintainer, those users are at
>>> risk.
>>>
>>> Perhaps we could advertise on Craigslist or something ... maintainer
>>> wanted for LTR.
>
> Gr{oetje,eeting}s,
>
> Geert
>

2018-04-29 19:36:36

by Greg Kroah-Hartman

[permalink] [raw]

Subject: Re: Moving unmaintained filesystems to staging

On Thu, Apr 26, 2018 at 08:11:08AM +0200, Pavel Machek wrote:
> On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
> > Recently ncpfs got moved to staging. Also recently, we had some fuzzer
> > developers report bugs in hfs, which they deem a security hole because
> > Ubuntu attempts to automount an inserted USB device as hfs.
>
> We promise "no-regressions" for code in main repository, no such
> promise for staging. We have quite a lot of code without maintainer.
>
> Moving code to staging means it will get broken -- staging was not
> designed for this. I believe moving anything there is bad idea.
>
> Staging is for ugly code, not for code that needs new maintainter.

Staging is used for getting code _out_ of the kernel tree as well as
_in_. We use it all the time to move code there, see if anyone shows up
in 6-8 months to say "I will fix this!", and if not, we delete it.

Look at what just happened to IRDA in the 4.17-rc1 release as an example
of this.

thanks,

greg k-h

2018-04-29 20:09:27

by Ondrej Zary

[permalink] [raw]

Subject: Re: Moving unmaintained filesystems to staging

On Sunday 29 April 2018 14:07:05 Greg KH wrote:
> On Thu, Apr 26, 2018 at 08:11:08AM +0200, Pavel Machek wrote:
> > On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
> > > Recently ncpfs got moved to staging. Also recently, we had some fuzzer
> > > developers report bugs in hfs, which they deem a security hole because
> > > Ubuntu attempts to automount an inserted USB device as hfs.
> >
> > We promise "no-regressions" for code in main repository, no such
> > promise for staging. We have quite a lot of code without maintainer.
> >
> > Moving code to staging means it will get broken -- staging was not
> > designed for this. I believe moving anything there is bad idea.
> >
> > Staging is for ugly code, not for code that needs new maintainter.
>
> Staging is used for getting code _out_ of the kernel tree as well as
> _in_. We use it all the time to move code there, see if anyone shows up
> in 6-8 months to say "I will fix this!", and if not, we delete it.
>
> Look at what just happened to IRDA in the 4.17-rc1 release as an example
> of this.

Really a "great" example of deleting working code :(

--
Ondrej Zary

2018-04-29 23:40:16

by Greg Kroah-Hartman

[permalink] [raw]

Subject: Re: Moving unmaintained filesystems to staging

On Sun, Apr 29, 2018 at 10:07:26PM +0200, Ondrej Zary wrote:
> On Sunday 29 April 2018 14:07:05 Greg KH wrote:
> > On Thu, Apr 26, 2018 at 08:11:08AM +0200, Pavel Machek wrote:
> > > On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
> > > > Recently ncpfs got moved to staging. Also recently, we had some fuzzer
> > > > developers report bugs in hfs, which they deem a security hole because
> > > > Ubuntu attempts to automount an inserted USB device as hfs.
> > >
> > > We promise "no-regressions" for code in main repository, no such
> > > promise for staging. We have quite a lot of code without maintainer.
> > >
> > > Moving code to staging means it will get broken -- staging was not
> > > designed for this. I believe moving anything there is bad idea.
> > >
> > > Staging is for ugly code, not for code that needs new maintainter.
> >
> > Staging is used for getting code _out_ of the kernel tree as well as
> > _in_. We use it all the time to move code there, see if anyone shows up
> > in 6-8 months to say "I will fix this!", and if not, we delete it.
> >
> > Look at what just happened to IRDA in the 4.17-rc1 release as an example
> > of this.
>
> Really a "great" example of deleting working code :(

What do you mean? The irda code was broken and not working at all.
There were loads of bug reports about it for years, with no developers
or maintainers willing to do the work on it to get it to actually work
again.

If someone does want to step up and do it, great! It's a simple revert
of two git commits and they are back in business.

Dropping code from the tree is not like it is gone for forever. If
someone wants to pick it up, it is trivial to do so. Git is good :)

thanks,

greg k-h

2018-05-01 10:15:06

by Pavel Machek

[permalink] [raw]

Subject: Re: Moving unmaintained filesystems to staging

On Sun 2018-04-29 16:37:37, Greg KH wrote:
> On Sun, Apr 29, 2018 at 10:07:26PM +0200, Ondrej Zary wrote:
> > On Sunday 29 April 2018 14:07:05 Greg KH wrote:
> > > On Thu, Apr 26, 2018 at 08:11:08AM +0200, Pavel Machek wrote:
> > > > On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
> > > > > Recently ncpfs got moved to staging. Also recently, we had some fuzzer
> > > > > developers report bugs in hfs, which they deem a security hole because
> > > > > Ubuntu attempts to automount an inserted USB device as hfs.
> > > >
> > > > We promise "no-regressions" for code in main repository, no such
> > > > promise for staging. We have quite a lot of code without maintainer.
> > > >
> > > > Moving code to staging means it will get broken -- staging was not
> > > > designed for this. I believe moving anything there is bad idea.
> > > >
> > > > Staging is for ugly code, not for code that needs new maintainter.
> > >
> > > Staging is used for getting code _out_ of the kernel tree as well as
> > > _in_. We use it all the time to move code there, see if anyone shows up
> > > in 6-8 months to say "I will fix this!", and if not, we delete it.
> > >
> > > Look at what just happened to IRDA in the 4.17-rc1 release as an example
> > > of this.
> >
> > Really a "great" example of deleting working code :(
>
> What do you mean? The irda code was broken and not working at all.
> There were loads of bug reports about it for years, with no developers
> or maintainers willing to do the work on it to get it to actually work
> again.
>
> If someone does want to step up and do it, great! It's a simple revert
> of two git commits and they are back in business.

> Dropping code from the tree is not like it is gone for forever. If
> someone wants to pick it up, it is trivial to do so.

That is a lie and you know it.

In particular, having code moved to staging means it is going to
bitrot, because it will not be updated with global changes.

Plus coding standards change over time, so if you simply revert,
you'll not be able to simply merge it back.

Plus that revert means bisection is no longer easy/possible to find
the real breakages.

Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Attachments:

(No filename) (2.30 kB)
signature.asc (188.00 B)
Digital signature Download all attachments

2018-05-03 09:19:50

by Pavel Machek

[permalink] [raw]

Subject: Re: Moving unmaintained filesystems to staging

On Thu 2018-04-26 12:36:50, Martin Steigerwald wrote:
> Pavel Machek - 26.04.18, 08:11:
> > On Wed 2018-04-25 08:46:02, Matthew Wilcox wrote:
> > > Recently ncpfs got moved to staging. Also recently, we had some
> > > fuzzer developers report bugs in hfs, which they deem a security
> > > hole because Ubuntu attempts to automount an inserted USB device as
> > > hfs.
> >
> > We promise "no-regressions" for code in main repository, no such
> > promise for staging. We have quite a lot of code without maintainer.
> >
> > Moving code to staging means it will get broken -- staging was not
> > designed for this. I believe moving anything there is bad idea.
> >
> > Staging is for ugly code, not for code that needs new maintainter.
>
> Good point.
>
> Moving things in and out to some "unmaintained" directory… I am not sure
> about that either. I tend to think that moving code around does not
> solve the underlying issue.
>
> Which, according to what I got from Matthew, was that distributors
> enable just about every filesystem they can enable which lead to hfs
> being used for automounting an USB stick formatted with it.

If distro's are shipping it, they are also willing to fix it. So
... no need to drop the code just yet.

Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Attachments:

(No filename) (1.40 kB)
signature.asc (188.00 B)
Digital signature Download all attachments

2018-05-06 01:00:55

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

On Thu, Apr 26, 2018 at 12:45:41PM +0200, John Paul Adrian Glaubitz wrote:

> Exactly. It works fine as is:
>
> root@elgar:~> uname -a
> Linux elgar 4.16.0-rc2-amiga-16784-ga8917fc #650 Mon Mar 5 15:32:52 NZDT 2018 m68k GNU/Linux
> root@elgar:~> mount /dev/sda1 /mnt -taffs
> root@elgar:~> ls -l /mnt | head
> total 0
> drwx------ 1 root root 0 Mar 30 2001 Alt
> -rw------- 1 root root 1352 Mar 27 1997 Alt.info
> drwx------ 1 root root 0 Nov 16 14:39 C
> drwx------ 1 root root 0 Mar 27 1997 CS_Fonts
> drwx------ 1 root root 0 Mar 27 1997 Classes
> -rwx------ 1 root root 1132 Aug 14 1996 Classes.info
> drwx------ 1 root root 0 Feb 10 2004 Commodities
> -rw------- 1 root root 628 Jan 14 2002 Commodities.info
> drwx------ 1 root root 0 Apr 10 1999 CyberTools
> root@elgar:~> mount |grep affs
> /dev/sda1 on /mnt type affs (rw,relatime,bs=512,volume=:)
> root@elgar:~>
>
> There is nothing at the moment that needs fixing.

Funny, that... I'd been going through the damn thing for the
last week or so; open-by-fhandle/nfs export support is completely
buggered. And as for the rest... the least said about the error
handling, the better - something like rename() hitting an IO
error (read one) can not only screw the on-disk data into the
ground, it can do seriously bad things to kernel data structures.

Is there anything resembling fsck for that thing, BTW? Nevermind
the repairs, just the validity checks would be nice...

2018-05-06 07:42:47

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

On Sun, May 06, 2018 at 01:59:51AM +0100, Al Viro wrote:

> > There is nothing at the moment that needs fixing.
>
> Funny, that... I'd been going through the damn thing for the
> last week or so; open-by-fhandle/nfs export support is completely
> buggered. And as for the rest... the least said about the error
> handling, the better - something like rename() hitting an IO
> error (read one) can not only screw the on-disk data into the
> ground, it can do seriously bad things to kernel data structures.

... and while we are at it, consider the following:

mkdir a
mkdir b
touch a/x
ln a/x b

<umount and mount again, to get cold dcache, or just wait for memory
pressure to evict those suckers>

process A: unlink("a/x");
process B: open("b/x");

We had two entries - one in a, another in b; the one in a is ST_FILE one,
the one in b - ST_LINKFILE, with ->original refering to the ST_FILE one.

unlink() needs to kick the entry out of a; since it can't leave an
ST_FILE not included into any directory (and since the file should live on
due to b/x still being there) it ends up removing ST_LINKFILE entry from
b and moving ST_FILE one in its place. That happens in affs_remove_link().

However, open() gets there just as this surgery is about to begin.
It gets to affs_lookup(), which finds the entry for b/x, reads it,
stashes block number of that entry into dentry->d_fsdata, notices
that it's ST_LINKFILE one, picks the block number of ST_FILE one
out of it and proceeds to look up the inode; that doesn't end up
doing any work (the same inode is in icache due to a/x having been
looked up by unlink(2)).

In the meanwhile, since the hash table of b has been unlocked once
we'd done hash lookup, affs_remove_link() can proceed with the
surgery. It *does* try to catch dentries with ->d_fsdata containing
the block number of sacrificed ST_LINKFILE and reset that to block
number of ST_FILE that gets moved in its place. However, it does
so by
static void
affs_fix_dcache(struct inode *inode, u32 entry_ino)
{
struct dentry *dentry;
spin_lock(&inode->i_lock);
hlist_for_each_entry(dentry, &inode->i_dentry, d_u.d_alias) {
if (entry_ino == (u32)(long)dentry->d_fsdata) {
dentry->d_fsdata = (void *)inode->i_ino;
break;
}
}
spin_unlock(&inode->i_lock);
}
i.e. looping through dentries that point to our inode. Except that
*our* dentry isn't attached to inode yet, so we are SOL - it's
left with ->d_fsdata pointing to (destroyed) ST_LINKFILE.

After a while process B closes the file and unlinks it. Take a look
at affs_remove_header() and you'll see how much fun we are in for -
it uses ->d_fsdata to pick the entry to find and remove...

That's an fs corruptor, plain and simple. As far as I had been able
to reconstruct the history, leftover after Roman's locking changes
17 years ago. AFAICS, I'd missed it back then - the races I'd spotted
had been dealt with about a year later (when we started to lock the
victim in vfs_unlink/vfs_rmdir/vfs_rename), but this one went unnoticed...

2018-05-06 08:41:22

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

On 05/06/2018 02:59 AM, Al Viro wrote:
>> There is nothing at the moment that needs fixing.
>
> Funny, that... I'd been going through the damn thing for the
> last week or so; open-by-fhandle/nfs export support is completely
> buggered. And as for the rest... the least said about the error
> handling, the better - something like rename() hitting an IO
> error (read one) can not only screw the on-disk data into the
> ground, it can do seriously bad things to kernel data structures.

Well, yes, there are certainly things in the affs code that can cause
unexpected behavior. But that wasn't my point. My point was that the
purpose for what people are using the affs driver in the kernel, it's
doing its job perfectly fine.

The standard use case is either to run Debian m68k on an Amiga and
access the Amiga partitions through Linux for simple file transfers
between the two operating systems, e.g. copying a new kernel plus
newly generated initrd to the affs boot partition of AmigaOS.

Or, just reading data off an old Amiga harddisk or floppy disk by
either attaching the disk to the Linux machine directly or reading
the contents of a floppy disk with an advanced floppy disk controller
like the Catweasel or KryoFlux.

For both cases, I haven't run into any issues with the affs driver
and I also haven't heard of complaints from anyone in communities
like EAB Amiga, amiga.org, a1k.org etc. That doesn't mean that people
haven't run into issues, of course. I'm just saying I haven't heard
of any.

> Is there anything resembling fsck for that thing, BTW? Nevermind
> the repairs, just the validity checks would be nice...

Not that I know of.

Adrian

PS: Even Matthew Garrett seems to be a Linux/m68k fan:

> https://twitter.com/mjg59/status/945779704329629697

:)

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - [email protected]
`. `' Freie Universitaet Berlin - [email protected]
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

2018-05-06 08:52:54

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

On 04/27/2018 03:26 AM, jdow wrote:
> And before I forget there are two features of the RDBs that I heartily recommend never implementing on Linux. They were good ideas at the time; but, times
> changed. The RDBs are capable of storing a filesystem driver and some drive init code for the plugin disk driver card. That is giving malware authors entirely
> goo easy a shot at owning a machine. Martin S., I would strongly suggest that going forward those two capabilities be removed from the RDB readers in AmigaOS
> as well as Linux OS.

I assume removing the feature for AmigaOS isn't really possible since we don't have
the source code for that, do we?

Also, if I remember correctly, Mac partitions can store filesystem drivers as well
and its actually a feature being used in MacOS. parted received a patch some time
ago to fix the correct handling for storing the filesystem driver in the partition
table.

I would be generally against removing these features as I don't think the security
risk is relevant for the majority of users. The Amiga is a hobbyist machine these
days and AmigaOS has certainly way more on than way to be compromised through
vulnerabilities.

Adrian

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - [email protected]
`. `' Freie Universitaet Berlin - [email protected]
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

2018-05-06 10:11:06

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

John Paul Adrian Glaubitz - 06.05.18, 10:52:
> On 04/27/2018 03:26 AM, jdow wrote:
> > And before I forget there are two features of the RDBs that I
> > heartily recommend never implementing on Linux. They were good
> > ideas at the time; but, times changed. The RDBs are capable of
> > storing a filesystem driver and some drive init code for the plugin
> > disk driver card. That is giving malware authors entirely goo easy
> > a shot at owning a machine. Martin S., I would strongly suggest
> > that going forward those two capabilities be removed from the RDB
> > readers in AmigaOS as well as Linux OS.
>
> I assume removing the feature for AmigaOS isn't really possible since
> we don't have the source code for that, do we?

AmigaOS 4.x does not support loading filesystems from RDB anymore as far
as I know. Meanwhile I am not involved with the AmigaOS development
anymore, but that is the last state I know. Similarily to Linux
filesystems drivers are loaded as "modules" into the kernel. Its also
still possible to load a filesystem as a file from disk, but that does
not work for the filesystem the kernel boots from. The AmigaOS kernel
still decides which of the kernel filesystem to use according to the
DOSType of the partition.

Thanks,
--
Martin

2018-05-06 10:13:05

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Al Viro - 06.05.18, 02:59:
> On Thu, Apr 26, 2018 at 12:45:41PM +0200, John Paul Adrian Glaubitz
wrote:
> > Exactly. It works fine as is:
> >
> > root@elgar:~> uname -a
> > Linux elgar 4.16.0-rc2-amiga-16784-ga8917fc #650 Mon Mar 5 15:32:52
> > NZDT 2018 m68k GNU/Linux root@elgar:~> mount /dev/sda1 /mnt -taffs
> > root@elgar:~> ls -l /mnt | head
> > total 0
> > drwx------ 1 root root 0 Mar 30 2001 Alt
> > -rw------- 1 root root 1352 Mar 27 1997 Alt.info
> > drwx------ 1 root root 0 Nov 16 14:39 C
> > drwx------ 1 root root 0 Mar 27 1997 CS_Fonts
> > drwx------ 1 root root 0 Mar 27 1997 Classes
> > -rwx------ 1 root root 1132 Aug 14 1996 Classes.info
> > drwx------ 1 root root 0 Feb 10 2004 Commodities
> > -rw------- 1 root root 628 Jan 14 2002 Commodities.info
> > drwx------ 1 root root 0 Apr 10 1999 CyberTools
> > root@elgar:~> mount |grep affs
> > /dev/sda1 on /mnt type affs (rw,relatime,bs=512,volume=:)
> > root@elgar:~>
> >
> > There is nothing at the moment that needs fixing.
>
> Funny, that... I'd been going through the damn thing for the
> last week or so; open-by-fhandle/nfs export support is completely
> buggered. And as for the rest... the least said about the error
> handling, the better - something like rename() hitting an IO
> error (read one) can not only screw the on-disk data into the
> ground, it can do seriously bad things to kernel data structures.
>
> Is there anything resembling fsck for that thing, BTW? Nevermind
> the repairs, just the validity checks would be nice...

I am not aware of the fsck command for affs on Linux. There is a
partitioning tool called amiga-fdisk, but for checking a filesystem you
would need to use a tool under AmigaOS.

--
Martin

2018-05-06 20:47:11

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

On Sun, May 06, 2018 at 08:39:55AM +0100, Al Viro wrote:
> On Sun, May 06, 2018 at 01:59:51AM +0100, Al Viro wrote:
>
> > > There is nothing at the moment that needs fixing.
> >
> > Funny, that... I'd been going through the damn thing for the
> > last week or so; open-by-fhandle/nfs export support is completely
> > buggered. And as for the rest... the least said about the error
> > handling, the better - something like rename() hitting an IO
> > error (read one) can not only screw the on-disk data into the
> > ground, it can do seriously bad things to kernel data structures.
>
> ... and while we are at it, consider the following:

[snip]

Another piece of fun: in
affs_add_entry() we have

retval = affs_insert_hash(dir, bh);
mark_buffer_dirty_inode(bh, inode);
affs_unlock_dir(dir);
affs_unlock_link(inode);

d_instantiate(dentry, inode);
done:
affs_brelse(inode_bh);
affs_brelse(bh);
return retval;

and in its callers - things like
error = affs_add_entry(dir, inode, dentry, ST_USERDIR);
if (error) {
clear_nlink(inode);
mark_inode_dirty(inode);
iput(inode);
return error;
}

Guess what happens if we hit affs_insert_hash() failure?
d_instantiate() doesn't do anything to in-core inode refcount -
that's caller's responsibility. affs_new_inode() has allocated
an inode with ->i_count equal to 1; d_instantiate() transfers that
reference into dentry (as ->d_inode). And then, since we'd got
a non-zero error we do hit iput() (same as we would've if an error
happened early enough in affs_add_entry() to bypass d_instantiate()).
That drives ->i_count to 0, getting inode freed (zero link count
when the last in-core reference is dropped means that there's no
point retaining it in icache).
So in that case we end up with struct dentry (still hashed
and available for lookups to pick) that has ->d_inode pointing
to freed memory. Welcome to memory corruption...

I'm fixing that pile of crap (along with the NFS exports
one and, hopefully, rename mess as well). HOWEVER, I am not going
to take over the damn thing - David has violated the 11th
commandment (Thou Shalt Never Volunteer), so he gets to joy of
learning that codebase and taking care of it from now on.

2018-05-06 20:49:45

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Hi Al!

On 05/06/2018 10:46 PM, Al Viro wrote:
> I'm fixing that pile of crap (along with the NFS exports
> one and, hopefully, rename mess as well). HOWEVER, I am not going
> to take over the damn thing - David has violated the 11th
> commandment (Thou Shalt Never Volunteer), so he gets to joy of
> learning that codebase and taking care of it from now on.

Thanks a lot for your stab at this and fixing the worst issues.

Since we're using that code regularly in Debian, your help is highly
appreciated :-). I'm very glad this discussion has lead to the code
being improved!

Again, thank you!
Adrian

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - [email protected]
`. `' Freie Universitaet Berlin - [email protected]
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

2018-05-06 21:33:29

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

On Sun, May 06, 2018 at 09:46:23PM +0100, Al Viro wrote:

> I'm fixing that pile of crap (along with the NFS exports
> one and, hopefully, rename mess as well). HOWEVER, I am not going
> to take over the damn thing - David has violated the 11th
> commandment (Thou Shalt Never Volunteer), so he gets to joy of
> learning that codebase and taking care of it from now on.

Same scenario on link(2) ends up with
* ST_LINKFILE created, inserted into the link chain and left around,
without being present in any hash chain
* target becoming positive hashed dentry, as if link(2) has succeeded,
so dcache lookups will be finding it (for a while)
* unlink(2) on source will have very interesting effects, what with
the attempts to move ST_FILE entry into the directory presumed to
contain ST_LINKFILE one, removing ST_LINKFILE from it at the same time.

2018-05-07 02:17:56

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

On Sun, May 06, 2018 at 10:32:47PM +0100, Al Viro wrote:
> On Sun, May 06, 2018 at 09:46:23PM +0100, Al Viro wrote:
>
> > I'm fixing that pile of crap (along with the NFS exports
> > one and, hopefully, rename mess as well). HOWEVER, I am not going
> > to take over the damn thing - David has violated the 11th
> > commandment (Thou Shalt Never Volunteer), so he gets to joy of
> > learning that codebase and taking care of it from now on.
>
> Same scenario on link(2) ends up with
> * ST_LINKFILE created, inserted into the link chain and left around,
> without being present in any hash chain
> * target becoming positive hashed dentry, as if link(2) has succeeded,
> so dcache lookups will be finding it (for a while)
> * unlink(2) on source will have very interesting effects, what with
> the attempts to move ST_FILE entry into the directory presumed to
> contain ST_LINKFILE one, removing ST_LINKFILE from it at the same time.

Oh, lovely... Looks like that thing wants the hash chains sorted by
block number. affs_insert_hash() doesn't give a toss - it always
adds to the very end of chain.

Maintaining that requirement (and I can understand the rationale -
they don't want too many back-and-forth seeks on directory lookups)
is going to be great fun on rename(), especially for the case when
the target of rename happens to be a primary name for a file with
additional hardlinks.

I think I see how to deal with that sanely, but... ouch.

Incidentally, we'd better verify that hash chains are not looped - as it
is, there's no checks whatsoever, and it *will* happily loop if you
feed it an image with such braindamage. I really hope that no fan of
desktop experience has set the things up for e.g. USB sticks with
that on them being recognized and automounted on insertion...

2018-05-07 02:40:32

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Al,

I don't think there is USB sticks with affs on them as yet. There
isn't even USB host controller support for Amiga hardware (yet).

Last I tried USB on m68k (Atari, 060 accelerator) the desktop
experience was such that I'd rather not repeat that in a hurry (and
that was a simple FAT USB stick).

I see your point regarding the immense practical joke value on any
desktop PC ... my work desktop has the affs module present. Happy to
try this out if someone can provide a sample disk image suitable for
USB flash media.

Cheers,

Michael

On Mon, May 7, 2018 at 2:15 PM, Al Viro <[email protected]> wrote:
> On Sun, May 06, 2018 at 10:32:47PM +0100, Al Viro wrote:
>> On Sun, May 06, 2018 at 09:46:23PM +0100, Al Viro wrote:
>>
>> > I'm fixing that pile of crap (along with the NFS exports
>> > one and, hopefully, rename mess as well). HOWEVER, I am not going
>> > to take over the damn thing - David has violated the 11th
>> > commandment (Thou Shalt Never Volunteer), so he gets to joy of
>> > learning that codebase and taking care of it from now on.
>>
>> Same scenario on link(2) ends up with
>> * ST_LINKFILE created, inserted into the link chain and left around,
>> without being present in any hash chain
>> * target becoming positive hashed dentry, as if link(2) has succeeded,
>> so dcache lookups will be finding it (for a while)
>> * unlink(2) on source will have very interesting effects, what with
>> the attempts to move ST_FILE entry into the directory presumed to
>> contain ST_LINKFILE one, removing ST_LINKFILE from it at the same time.
>
> Oh, lovely... Looks like that thing wants the hash chains sorted by
> block number. affs_insert_hash() doesn't give a toss - it always
> adds to the very end of chain.
>
> Maintaining that requirement (and I can understand the rationale -
> they don't want too many back-and-forth seeks on directory lookups)
> is going to be great fun on rename(), especially for the case when
> the target of rename happens to be a primary name for a file with
> additional hardlinks.
>
> I think I see how to deal with that sanely, but... ouch.
>
> Incidentally, we'd better verify that hash chains are not looped - as it
> is, there's no checks whatsoever, and it *will* happily loop if you
> feed it an image with such braindamage. I really hope that no fan of
> desktop experience has set the things up for e.g. USB sticks with
> that on them being recognized and automounted on insertion...
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2018-05-07 04:55:23

by jdow

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

On 20180506 01:52, John Paul Adrian Glaubitz wrote:
> On 04/27/2018 03:26 AM, jdow wrote:
>> And before I forget there are two features of the RDBs that I heartily recommend never implementing on Linux. They were good ideas at the time; but, times
>> changed. The RDBs are capable of storing a filesystem driver and some drive init code for the plugin disk driver card. That is giving malware authors entirely
>> goo easy a shot at owning a machine. Martin S., I would strongly suggest that going forward those two capabilities be removed from the RDB readers in AmigaOS
>> as well as Linux OS.
>
> I assume removing the feature for AmigaOS isn't really possible since we don't have
> the source code for that, do we?
>
> Also, if I remember correctly, Mac partitions can store filesystem drivers as well
> and its actually a feature being used in MacOS. parted received a patch some time
> ago to fix the correct handling for storing the filesystem driver in the partition
> table.
>
> I would be generally against removing these features as I don't think the security
> risk is relevant for the majority of users. The Amiga is a hobbyist machine these
> days and AmigaOS has certainly way more on than way to be compromised through
> vulnerabilities.
>
> Adrian

You do not necessarily have the source for the device drivers. However the
DriveInit code and the filesystem code get executed by the OS initialization
code. The objection I have to the concept is that it's invisible to the user.
The Linux filesystem code is either compiled into the kernel or is available in
the libraries where it can be monitored at several levels from source code on
up. Within AmigaDOS it can be monitored fairly easily by an AV tool - in theory.
Alas, this is trying to lock the barn door after the barn has burned to the
ground with a clever enough piece of malware. At least AmigaDOS AV tools should
be expected to examine DriveInit and filesystem images on disk in the RDBs for
malware modifications to those blocks. This is a burden Linux should not be
forced to bear. So loading filesystems from RDBs instead of the more usual and
accepted Linux practices should be disabled. And at least a portion of this
discussion is Linux related. That's why I mentioned disabling the feature. While
I cannot see much money in AmigaDOS related malware I can see it in Linux
malware. And there's no real "glory" in launching malware on AmigaDOS. It's too
easy a problem last I knew. "Whoopie, you have just proven you can ride a
tricycle. Can't you do better?" (One could argue that AmigaDOS 1.0 was
self-inflicted malware foisted on marvelous hardware for the era.)

{^_^} Joanne Dow

2018-05-07 07:09:09

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Michael Schmitz - 07.05.18, 04:40:
> Al,
>
> I don't think there is USB sticks with affs on them as yet. There
> isn't even USB host controller support for Amiga hardware (yet).
>
> Last I tried USB on m68k (Atari, 060 accelerator) the desktop
> experience was such that I'd rather not repeat that in a hurry (and
> that was a simple FAT USB stick).

There is USB support available on Amiga since a long time.

On "Classic" Amigas AmigaOS 3.x with Poseidon USB stack + some USB card.

On AmigaOS 4.x built-in. AmigaOS 4.x hardware like Sam boards from Acube
Systems have USB controllers that work out of the bux.

And I am pretty sure, you can also tell it to use Amiga Fast Filesystem
(on Linux affs) on an USB stick. Also you can plug in an external
harddisk with RDB partitions and whatever filesystems you wish.

Thanks,
--
Martin

2018-05-07 20:50:50

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Martin,

On Mon, May 7, 2018 at 7:08 PM, Martin Steigerwald <[email protected]> wrote:
> Michael Schmitz - 07.05.18, 04:40:
>> Al,
>>
>> I don't think there is USB sticks with affs on them as yet. There
>> isn't even USB host controller support for Amiga hardware (yet).
>>
>> Last I tried USB on m68k (Atari, 060 accelerator) the desktop
>> experience was such that I'd rather not repeat that in a hurry (and
>> that was a simple FAT USB stick).
>
> There is USB support available on Amiga since a long time.

Good to hear that. I stand corrected.

> On "Classic" Amigas AmigaOS 3.x with Poseidon USB stack + some USB card.

Haven't seen a Linux driver for that 'some USB card' yet.

> On AmigaOS 4.x built-in. AmigaOS 4.x hardware like Sam boards from Acube
> Systems have USB controllers that work out of the bux.

Forgot about the new (non-m68k) hardware. My focus is somewhat narrow,
on m68k and Linux.

> And I am pretty sure, you can also tell it to use Amiga Fast Filesystem
> (on Linux affs) on an USB stick. Also you can plug in an external
> harddisk with RDB partitions and whatever filesystems you wish.

I already conceded that's possible.

So our problem with the bug Al spotted, and AFFS on USB media are twtofold:

AmigaOS:
Exploitable: yes (unless the AmigaOS AFFS driver detects and mitigates this).
Likelihood: low (as Joanne said there are easier ways to do harm to
these systems)

Linux:
Exploitable: yes, except on hardware that doesn't have USB hardware support.
Likelihood: high

Can we blacklist affs from being autoloaded through udev on USB
storage media discovery?

Cheers,

Michael

2018-05-07 20:58:53

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

On 05/07/2018 10:56 PM, Ingo Jürgensmann wrote:
>> Haven't seen a Linux driver for that 'some USB card' yet.
>
> There is for example RapidRoad (<http://wiki.icomp.de/wiki/RapidRoad>) which
> is an addon for the X-Surf 100 NIC (<http://wiki.icomp.de/wiki/X-Surf-100>).
> Adrian already has this addon module, I will buy one this year as well.

Michael is the one who has started working on the driver, so I think
he knows :-).

Adrian

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - [email protected]
`. `' Freie Universitaet Berlin - [email protected]
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

2018-05-07 21:44:25

by Ingo Jürgensmann

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Am 07.05.2018 um 22:50 schrieb Michael Schmitz <[email protected]>:

>>> I don't think there is USB sticks with affs on them as yet. There
>>> isn't even USB host controller support for Amiga hardware (yet).
>>>
>>> Last I tried USB on m68k (Atari, 060 accelerator) the desktop
>>> experience was such that I'd rather not repeat that in a hurry (and
>>> that was a simple FAT USB stick).
>>
>> There is USB support available on Amiga since a long time.
>
> Good to hear that. I stand corrected.
>
>> On "Classic" Amigas AmigaOS 3.x with Poseidon USB stack + some USB card.
>
> Haven't seen a Linux driver for that 'some USB card' yet.

There is for example RapidRoad (<http://wiki.icomp.de/wiki/RapidRoad>) which is an addon for the X-Surf 100 NIC (<http://wiki.icomp.de/wiki/X-Surf-100>). Adrian already has this addon module, I will buy one this year as well.

--
Ciao... // http://blog.windfluechter.net
Ingo \X/ XMPP: [email protected]

gpg pubkey: http://www.juergensmann.de/ij_public_key.asc

2018-06-24 09:07:45

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)

Hi.

Michael Schmitz - 27.04.18, 04:11:
> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
> indicate the RDB parser bug is fixed by the patch given there, so if
> Martin now submits the patch, all should be well?

Ok, better be honest than having anyone waiting for it:

I do not care enough about this, in order to motivate myself preparing
the a patch from Joanne Dow?s fix.

I am not even using my Amiga boxes anymore, not even the Sam440ep which
I still have in my apartment.

So RDB support in Linux it remains broken for disks larger 2 TB, unless
someone else does.

Thanks.
--
Martin

2018-06-24 11:41:04

by jdow

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn dool.

If memory serves the RDBs think in blocks rather than bytes so it should work up
to 2 gigablocks whatever your block size is. 512 blocks is 2199023255552 bytes.
But that wastes just a WHOLE LOT of disk in block maps. Go up to 4096 or 8192.
The latter is 35 TB.

{^_^}
On 20180624 02:06, Martin Steigerwald wrote:
> Hi.
>
> Michael Schmitz - 27.04.18, 04:11:
>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>> indicate the RDB parser bug is fixed by the patch given there, so if
>> Martin now submits the patch, all should be well?
>
> Ok, better be honest than having anyone waiting for it:
>
> I do not care enough about this, in order to motivate myself preparing
> the a patch from Joanne Dow´s fix.
>
> I am not even using my Amiga boxes anymore, not even the Sam440ep which
> I still have in my apartment.
>
> So RDB support in Linux it remains broken for disks larger 2 TB, unless
> someone else does.
>
> Thanks.
>

2018-06-24 16:18:33

by jdow

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

It's more likely the filesystem unless I didn't get the RDBs up to 3.1 status. I
am pretty damn sure I did as I had an 18 gigabyte disk I was using at the time.
And I made sure it would mount with Linux. Files greater than 2 gigs were sure
to get messed up. But that was not RDBs. The disk had 9 gigs worth of AFS
partitions and 9 gigs worth of SFS partitions. The big SFS partition was most of
the 9 gigabytes. (The hdf file is 8,932,818,944 bytes.) There was a tiny system
partition on it. It used some of Bill Hawes' goodies to overlay some useful paths.

I'm not sure the machine still boots. The disk in question no longer spins up.

{^_^}

On 20180624 02:06, Martin Steigerwald wrote:
> Hi.
>
> Michael Schmitz - 27.04.18, 04:11:
>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>> indicate the RDB parser bug is fixed by the patch given there, so if
>> Martin now submits the patch, all should be well?
>
> Ok, better be honest than having anyone waiting for it:
>
> I do not care enough about this, in order to motivate myself preparing
> the a patch from Joanne Dow´s fix.
>
> I am not even using my Amiga boxes anymore, not even the Sam440ep which
> I still have in my apartment.
>
> So RDB support in Linux it remains broken for disks larger 2 TB, unless
> someone else does.
>
> Thanks.
>

2018-06-25 07:54:37

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)

Hi Martin,

OK, I'll prepare a patch and submit it to linux-block for review. I'll
have to refer to your testing back in 2012 since all I can test is
whether the patch still allows partition tables on small disks to be
recognized at this time (unless Adrian has a 2 TB disk and a SATA-SCSI
bridge to test this properly on).

Cheers,

Michael

Am 24.06.18 um 21:06 schrieb Martin Steigerwald:
> Hi.
>
> Michael Schmitz - 27.04.18, 04:11:
>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>> indicate the RDB parser bug is fixed by the patch given there, so if
>> Martin now submits the patch, all should be well?
> Ok, better be honest than having anyone waiting for it:
>
> I do not care enough about this, in order to motivate myself preparing
> the a patch from Joanne Dow´s fix.
>
> I am not even using my Amiga boxes anymore, not even the Sam440ep which
> I still have in my apartment.
>
> So RDB support in Linux it remains broken for disks larger 2 TB, unless
> someone else does.
>
> Thanks.

2018-06-25 08:27:28

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)

Hi Michael.

Michael Schmitz - 25.06.18, 09:53:
> OK, I'll prepare a patch and submit it to linux-block for review. I'll
> have to refer to your testing back in 2012 since all I can test is
> whether the patch still allows partition tables on small disks to be
> recognized at this time (unless Adrian has a 2 TB disk and a
> SATA-SCSI bridge to test this properly on).

Thank you very much.

I believe the testing I did is still valid.

Feel free to include any parts of the description of the test I made
back then into your patch description as you see it as relevant for it.

Also feel free to include my Tested-By: (probably with a hint the test
was in 2012).

I am not sure I am ready to permanently let go of (some of) the hardware
I still have collected, but at some time I may. I do have SATA-SCSI
bridges. That A4000T I have here probably could be a nice Debian m68k
build host.

Thanks,
Martin

>
> Cheers,
>
> Michael
>
> Am 24.06.18 um 21:06 schrieb Martin Steigerwald:
> > Hi.
> >
> > Michael Schmitz - 27.04.18, 04:11:
> >> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >> indicate the RDB parser bug is fixed by the patch given there, so
> >> if
> >> Martin now submits the patch, all should be well?
> >
> > Ok, better be honest than having anyone waiting for it:
> >
> > I do not care enough about this, in order to motivate myself
> > preparing the a patch from Joanne Dow?s fix.
> >
> > I am not even using my Amiga boxes anymore, not even the Sam440ep
> > which I still have in my apartment.
> >
> > So RDB support in Linux it remains broken for disks larger 2 TB,
> > unless someone else does.
> >
> > Thanks.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k"
> in the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Martin

2018-06-25 08:41:35

by Geert Uytterhoeven

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging? (was: Re: Moving unmaintained filesystems to staging)

Hi Michael,

On Mon, Jun 25, 2018 at 9:53 AM Michael Schmitz <[email protected]> wrote:
> OK, I'll prepare a patch and submit it to linux-block for review. I'll

Thanks a lot!

> have to refer to your testing back in 2012 since all I can test is
> whether the patch still allows partition tables on small disks to be
> recognized at this time (unless Adrian has a 2 TB disk and a SATA-SCSI
> bridge to test this properly on).

Sparse file and loopback mounting with losetup --partscan?

> Am 24.06.18 um 21:06 schrieb Martin Steigerwald:
> > Michael Schmitz - 27.04.18, 04:11:
> >> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >> indicate the RDB parser bug is fixed by the patch given there, so if
> >> Martin now submits the patch, all should be well?
> > Ok, better be honest than having anyone waiting for it:
> >
> > I do not care enough about this, in order to motivate myself preparing
> > the a patch from Joanne Dow´s fix.
> >
> > I am not even using my Amiga boxes anymore, not even the Sam440ep which
> > I still have in my apartment.
> >
> > So RDB support in Linux it remains broken for disks larger 2 TB, unless
> > someone else does.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

2018-06-26 02:25:48

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Joanne,

Martin's boot log (including your patch) says:

Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
Attached SCSI disk

so it's indeed a case of self inflicted damage (RDSK (512) means 512
byte blocks) and can be worked around by using a different block size.

Your memory serves right indeed - blocksize is in 512 bytes units.
I'll still submit a patch to Jens anyway as this may bite others yet.

Cheers,

Michael

On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]> wrote:
> BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn
> dool.
>
> If memory serves the RDBs think in blocks rather than bytes so it should
> work up to 2 gigablocks whatever your block size is. 512 blocks is
> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
> Go up to 4096 or 8192. The latter is 35 TB.
>
> {^_^}
> On 20180624 02:06, Martin Steigerwald wrote:
>>
>> Hi.
>>
>> Michael Schmitz - 27.04.18, 04:11:
>>>
>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>> Martin now submits the patch, all should be well?
>>
>>
>> Ok, better be honest than having anyone waiting for it:
>>
>> I do not care enough about this, in order to motivate myself preparing
>> the a patch from Joanne Dow´s fix.
>>
>> I am not even using my Amiga boxes anymore, not even the Sam440ep which
>> I still have in my apartment.
>>
>> So RDB support in Linux it remains broken for disks larger 2 TB, unless
>> someone else does.
>>
>> Thanks.
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2018-06-26 05:18:36

by jdow

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

As long as it preserves compatibility it should be OK, I suppose. Personally I'd
make any partitioning tool front end gently force the block size towards 8k as
the disk size gets larger. The file systems may also run into 2TB issues that
are not obvious. An unused blocks list will have to go beyond a uint32_t size,
for example. But a block list (OFS for sure, don't remember for the newer AFS)
uses a tad under 1% of the disk all by itself. A block bitmap is not quite so
bad. {^_-}

Just be sure you are aware of all the ramifications when you make a change. I
remember thinking about this for awhile and then determining I REALLY did not
want to think about it as my brain was getting tied into a gordian knot.

{^_^}

On 20180625 19:23, Michael Schmitz wrote:
> Joanne,
>
> Martin's boot log (including your patch) says:
>
> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
> 4)
> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> Attached SCSI disk
>
> so it's indeed a case of self inflicted damage (RDSK (512) means 512
> byte blocks) and can be worked around by using a different block size.
>
> Your memory serves right indeed - blocksize is in 512 bytes units.
> I'll still submit a patch to Jens anyway as this may bite others yet.
>
> Cheers,
>
> Michael
>
>
> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]> wrote:
>> BTW - anybody who uses 512 byte blocks with an Amiga file system is a famn
>> dool.
>>
>> If memory serves the RDBs think in blocks rather than bytes so it should
>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in block maps.
>> Go up to 4096 or 8192. The latter is 35 TB.
>>
>> {^_^}
>> On 20180624 02:06, Martin Steigerwald wrote:
>>>
>>> Hi.
>>>
>>> Michael Schmitz - 27.04.18, 04:11:
>>>>
>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>> Martin now submits the patch, all should be well?
>>>
>>>
>>> Ok, better be honest than having anyone waiting for it:
>>>
>>> I do not care enough about this, in order to motivate myself preparing
>>> the a patch from Joanne Dow´s fix.
>>>
>>> I am not even using my Amiga boxes anymore, not even the Sam440ep which
>>> I still have in my apartment.
>>>
>>> So RDB support in Linux it remains broken for disks larger 2 TB, unless
>>> someone else does.
>>>
>>> Thanks.
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>

2018-06-26 08:04:09

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Michael.

Michael Schmitz - 26.06.18, 04:23:
> Joanne,
>
> Martin's boot log (including your patch) says:
>
> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
> 4)
> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> Attached SCSI disk
>
> so it's indeed a case of self inflicted damage (RDSK (512) means 512
> byte blocks) and can be worked around by using a different block size.

Well I pretty much believe that this was the standard block size of the
tool I used in order to create the RDB. I think it has been Media
Toolbox + the engine behind it, from AmigaOS 4.0 (not 4.1) or so. DOS-
Type JXF points to JXFS, an AmigaOS 4 only filesystem that meanwhile has
been deprecated by AmigaOS upstream.

Maybe HDToolbox + hdwrench.library by Joanne (AmigaOS 3.5/3.9) would
have set it up differently.

Anyway: It works like this in AmigaOS 4 without any issues. With 512
Byte Blocks. I think it is good that Linux does not create data
corruption when writing to disks that work just fine in AmigaOS.
Especially as those using AmigaNG machines like AmigaOne X1000/X5000 or
Acube Sam machines may dual boot AmigaOS and Linux on their machines.

Thanks again for putting together a patch.

Thanks,
--
Martin

2018-06-26 08:13:30

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Joanne.

jdow - 26.06.18, 07:17:
> As long as it preserves compatibility it should be OK, I suppose.
> Personally I'd make any partitioning tool front end gently force the
> block size towards 8k as the disk size gets larger. The file systems
> may also run into 2TB issues that are not obvious. An unused blocks
> list will have to go beyond a uint32_t size, for example. But a block
> list (OFS for sure, don't remember for the newer AFS) uses a tad
> under 1% of the disk all by itself. A block bitmap is not quite so
> bad. {^_-}
>
> Just be sure you are aware of all the ramifications when you make a
> change. I remember thinking about this for awhile and then
> determining I REALLY did not want to think about it as my brain was
> getting tied into a gordian knot.

Heh… :)

Well, all I can say it that it just worked on AmigaOS 4. I did not see
any data corruption in any of the filesystems. Well as far as I have
been able to check. There has been no repair tool for JXFS I think. But
as I migrated the data on it to SFS, I was able to copy everything.

Famous last words.

Well especially the disk size was detected properly and there was no
overflow like on Linux. So I´d say rather have on error less than one
error more.

Of course, it could also be an option to outright *refuse* to detect
such disks with a big fat warning in kernel log that it may unsafe. But
overflowing and thus on writes overwriting existing data is not safe.

I do think it is safe enough, but what I do know about the internals
about RDB (more than the average user certainly, but not as much as you
or some other AmigaOS developer who digged deeper into that).

So in case you´d rather see Linux to refuse to handle disks like that,
that would also be fine with me. Just do not handle them in the broken
way that they are handled in Linux now. I.e. do not deliberately corrupt
things as in "Its dangerous, so let´s overwrite data straight away, so
the user gets it." :)

Anyway, in my opinion RDB still just so much more advanced than MBR and
in some parts even on par with the much later GPT. With some limitations
it is a quite brilliant partition format, if you ask me.

> {^_^}
>
> On 20180625 19:23, Michael Schmitz wrote:
> > Joanne,
> >
> > Martin's boot log (including your patch) says:
> >
> > Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
> > (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
> > spb
> > 4)
> > Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> > Attached SCSI disk
> >
> > so it's indeed a case of self inflicted damage (RDSK (512) means 512
> > byte blocks) and can be worked around by using a different block
> > size.
> >
> > Your memory serves right indeed - blocksize is in 512 bytes units.
> > I'll still submit a patch to Jens anyway as this may bite others
> > yet.
> >
> > Cheers,
> >
> > Michael
> >
> > On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]> wrote:
> >> BTW - anybody who uses 512 byte blocks with an Amiga file system is
> >> a famn dool.
> >>
> >> If memory serves the RDBs think in blocks rather than bytes so it
> >> should work up to 2 gigablocks whatever your block size is. 512
> >> blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of
> >> disk in block maps. Go up to 4096 or 8192. The latter is 35 TB.
> >>
> >> {^_^}
> >>
> >> On 20180624 02:06, Martin Steigerwald wrote:
> >>> Hi.
> >>>
> >>> Michael Schmitz - 27.04.18, 04:11:
> >>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>> indicate the RDB parser bug is fixed by the patch given there, so
> >>>> if
> >>>> Martin now submits the patch, all should be well?
> >>>
> >>> Ok, better be honest than having anyone waiting for it:
> >>>
> >>> I do not care enough about this, in order to motivate myself
> >>> preparing the a patch from Joanne Dow´s fix.
> >>>
> >>> I am not even using my Amiga boxes anymore, not even the Sam440ep
> >>> which I still have in my apartment.
> >>>
> >>> So RDB support in Linux it remains broken for disks larger 2 TB,
> >>> unless someone else does.
> >>>
> >>> Thanks.
[…]
--
Martin

2018-06-26 08:33:37

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Joanne,

I think we all agree that doing 32 bit calculations on 512-byte block
addresses that overflow on disks 2 TB and larger is a bug, causing the
issues Martin reported. Your patch addresses that by using the correct
data type for the calculations (as do other partition parsers that may
have to deal with large disks) and fixes Martin's bug, so appears to be
the right thing to do.

Using 64 bit data types for disks smaller than 2 TB where calculations
don't currently overflow is not expected to cause new issues, other than
enabling use of disk and partitions larger than 2 TB (which may have
ramifications with filesystems on these partitions). So comptibility is
preserved.

Forcing larger block sizes might be a good strategy to avoid overflow
issues in filesystems as well, but I can't see how the block size stored
in the RDB would enforce use of the same block size in filesystems.
We'll have to rely on the filesystem tools to get that right, too. Linux
AFFS does allow block sizes up to 4k (VFS limitation) so this should
allow partitions larger than 2 TB to work already (but I suspect Al Viro
may have found a few issues when he looked at the AFFS code so I won't
say more). Anyway partitioning tools and filesystems are unrelated to
the Linux partition parser code which is all we aim to fix in this patch.

If you feel strongly about unknown ramifications of any filesystems on
partitions larger than 2 TB, say so and I'll have the kernel print a
warning about these partitions.

I'll get this patch tested on Martin's test case image as well as on a
RDB image from a disk known to currently work under Linux (thanks Geert
for the losetup hint). Can't do much more without procuring a working
Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
for tests is a long way away from my home indeed.

Cheers,

Michael

Am 26.06.18 um 17:17 schrieb jdow:
> As long as it preserves compatibility it should be OK, I suppose.
> Personally I'd make any partitioning tool front end gently force the
> block size towards 8k as the disk size gets larger. The file systems
> may also run into 2TB issues that are not obvious. An unused blocks
> list will have to go beyond a uint32_t size, for example. But a block
> list (OFS for sure, don't remember for the newer AFS) uses a tad under
> 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}
>
> Just be sure you are aware of all the ramifications when you make a
> change. I remember thinking about this for awhile and then determining
> I REALLY did not want to think about it as my brain was getting tied
> into a gordian knot.
>
> {^_^}
>
> On 20180625 19:23, Michael Schmitz wrote:
>> Joanne,
>>
>> Martin's boot log (including your patch) says:
>>
>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>> 4)
>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>> Attached SCSI disk
>>
>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>> byte blocks) and can be worked around by using a different block size.
>>
>> Your memory serves right indeed - blocksize is in 512 bytes units.
>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>
>> Cheers,
>>
>> Michael
>>
>>
>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]> wrote:
>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>> a famn
>>> dool.
>>>
>>> If memory serves the RDBs think in blocks rather than bytes so it
>>> should
>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>> block maps.
>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>
>>> {^_^}
>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>
>>>> Hi.
>>>>
>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>
>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>>> Martin now submits the patch, all should be well?
>>>>
>>>>
>>>> Ok, better be honest than having anyone waiting for it:
>>>>
>>>> I do not care enough about this, in order to motivate myself preparing
>>>> the a patch from Joanne Dow´s fix.
>>>>
>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>> which
>>>> I still have in my apartment.
>>>>
>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>> unless
>>>> someone else does.
>>>>
>>>> Thanks.
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-m68k" in
>>> the body of a message to [email protected]
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2018-06-26 08:42:28

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Hi Martin,

Am 26.06.18 um 20:02 schrieb Martin Steigerwald:
> Michael.
>
> Michael Schmitz - 26.06.18, 04:23:
>> Joanne,
>>
>> Martin's boot log (including your patch) says:
>>
>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>> 4)
>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>> Attached SCSI disk
>>
>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>> byte blocks) and can be worked around by using a different block size.
> Well I pretty much believe that this was the standard block size of the
> tool I used in order to create the RDB. I think it has been Media

No offense meant - 512 bytes per block would certainly have been the
default and it certainly wasn't obvious that this needed changing.

> Toolbox + the engine behind it, from AmigaOS 4.0 (not 4.1) or so. DOS-
> Type JXF points to JXFS, an AmigaOS 4 only filesystem that meanwhile has
> been deprecated by AmigaOS upstream.
>
> Maybe HDToolbox + hdwrench.library by Joanne (AmigaOS 3.5/3.9) would
> have set it up differently.
>
> Anyway: It works like this in AmigaOS 4 without any issues. With 512
> Byte Blocks. I think it is good that Linux does not create data
> corruption when writing to disks that work just fine in AmigaOS.
> Especially as those using AmigaNG machines like AmigaOne X1000/X5000 or
> Acube Sam machines may dual boot AmigaOS and Linux on their machines.

That's the goal.

> Thanks again for putting together a patch.

The hard work had been done by Joanne and you six years ago, so no
matter at all. If Jens and the crowd at linux-block point out errors in
review, I take that as a learning experience...

Cheers,

Michael

>
> Thanks,

2018-06-26 09:33:02

by jdow

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

HDToolBox is not mine. That is where the intelligence is needed.

{^_^}

On 20180626 01:02, Martin Steigerwald wrote:
> Michael.
>
> Michael Schmitz - 26.06.18, 04:23:
>> Joanne,
>>
>> Martin's boot log (including your patch) says:
>>
>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>> 4)
>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>> Attached SCSI disk
>>
>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>> byte blocks) and can be worked around by using a different block size.
>
> Well I pretty much believe that this was the standard block size of the
> tool I used in order to create the RDB. I think it has been Media
> Toolbox + the engine behind it, from AmigaOS 4.0 (not 4.1) or so. DOS-
> Type JXF points to JXFS, an AmigaOS 4 only filesystem that meanwhile has
> been deprecated by AmigaOS upstream.
>
> Maybe HDToolbox + hdwrench.library by Joanne (AmigaOS 3.5/3.9) would
> have set it up differently.
>
> Anyway: It works like this in AmigaOS 4 without any issues. With 512
> Byte Blocks. I think it is good that Linux does not create data
> corruption when writing to disks that work just fine in AmigaOS.
> Especially as those using AmigaNG machines like AmigaOne X1000/X5000 or
> Acube Sam machines may dual boot AmigaOS and Linux on their machines.
>
> Thanks again for putting together a patch.
>
> Thanks,
>

2018-06-26 09:47:55

by jdow

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

If it is not backwards compatible I for one would refuse to use it. And if it
still mattered that much to me I'd also generate a reasonable alternative.
Modifying RDBs nay not be even an approximation of a good idea. You'd discover
that as soon as an RDB uint64_t disk is tasted by a uint32_t only system. If it
is for your personal use then you're entirely free to reject my advice and are
probably smart enough to keep it working for yourself.

GPT is probably the right way to go. Preserve the ability to read RDBs for
legacy disks only.

{^_^}

On 20180626 01:31, Michael Schmitz wrote:
> Joanne,
>
> I think we all agree that doing 32 bit calculations on 512-byte block
> addresses that overflow on disks 2 TB and larger is a bug, causing the
> issues Martin reported. Your patch addresses that by using the correct
> data type for the calculations (as do other partition parsers that may
> have to deal with large disks) and fixes Martin's bug, so appears to be
> the right thing to do.
>
> Using 64 bit data types for disks smaller than 2 TB where calculations
> don't currently overflow is not expected to cause new issues, other than
> enabling use of disk and partitions larger than 2 TB (which may have
> ramifications with filesystems on these partitions). So comptibility is
> preserved.
>
> Forcing larger block sizes might be a good strategy to avoid overflow
> issues in filesystems as well, but I can't see how the block size stored
> in the RDB would enforce use of the same block size in filesystems.
> We'll have to rely on the filesystem tools to get that right, too. Linux
> AFFS does allow block sizes up to 4k (VFS limitation) so this should
> allow partitions larger than 2 TB to work already (but I suspect Al Viro
> may have found a few issues when he looked at the AFFS code so I won't
> say more). Anyway partitioning tools and filesystems are unrelated to
> the Linux partition parser code which is all we aim to fix in this patch.
>
> If you feel strongly about unknown ramifications of any filesystems on
> partitions larger than 2 TB, say so and I'll have the kernel print a
> warning about these partitions.
>
> I'll get this patch tested on Martin's test case image as well as on a
> RDB image from a disk known to currently work under Linux (thanks Geert
> for the losetup hint). Can't do much more without procuring a working
> Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
> for tests is a long way away from my home indeed.
>
> Cheers,
>
> Michael
>
> Am 26.06.18 um 17:17 schrieb jdow:
>> As long as it preserves compatibility it should be OK, I suppose.
>> Personally I'd make any partitioning tool front end gently force the
>> block size towards 8k as the disk size gets larger. The file systems
>> may also run into 2TB issues that are not obvious. An unused blocks
>> list will have to go beyond a uint32_t size, for example. But a block
>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>> 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}
>>
>> Just be sure you are aware of all the ramifications when you make a
>> change. I remember thinking about this for awhile and then determining
>> I REALLY did not want to think about it as my brain was getting tied
>> into a gordian knot.
>>
>> {^_^}
>>
>> On 20180625 19:23, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> Martin's boot log (including your patch) says:
>>>
>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>> 4)
>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>> Attached SCSI disk
>>>
>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>> byte blocks) and can be worked around by using a different block size.
>>>
>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>
>>> Cheers,
>>>
>>> Michael
>>>
>>>
>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]> wrote:
>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>> a famn
>>>> dool.
>>>>
>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>> should
>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>> block maps.
>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>
>>>> {^_^}
>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>
>>>>> Hi.
>>>>>
>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>
>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>>>> Martin now submits the patch, all should be well?
>>>>>
>>>>>
>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>
>>>>> I do not care enough about this, in order to motivate myself preparing
>>>>> the a patch from Joanne Dow´s fix.
>>>>>
>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>> which
>>>>> I still have in my apartment.
>>>>>
>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>> unless
>>>>> someone else does.
>>>>>
>>>>> Thanks.
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe
>>>> linux-m68k" in
>>>> the body of a message to [email protected]
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>

2018-06-26 09:48:12

by jdow

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

(Get everybody)
Speak to the high level driver folks about that. The low level stuff is
basically dumb. It tells you what it finds. It tells the rest of the OS (the
file systems) what it found. The amiga-makefs (or whatever it is called) needs
to grow some added intelligence just as HDToolBox needs to grow more
intelligence. The step from RDB uint32_t to RDB uint64_t probably needs a great
deal of thought. Personally I'd be tempted to adopt GUID based partitioning
system. It has already considered this stuff.

{^_^}

On 20180626 01:12, Martin Steigerwald wrote:
> Joanne.
>
> jdow - 26.06.18, 07:17:
>> As long as it preserves compatibility it should be OK, I suppose.
>> Personally I'd make any partitioning tool front end gently force the
>> block size towards 8k as the disk size gets larger. The file systems
>> may also run into 2TB issues that are not obvious. An unused blocks
>> list will have to go beyond a uint32_t size, for example. But a block
>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>> under 1% of the disk all by itself. A block bitmap is not quite so
>> bad. {^_-}
>>
>> Just be sure you are aware of all the ramifications when you make a
>> change. I remember thinking about this for awhile and then
>> determining I REALLY did not want to think about it as my brain was
>> getting tied into a gordian knot.
>
> Heh… :)
>
> Well, all I can say it that it just worked on AmigaOS 4. I did not see
> any data corruption in any of the filesystems. Well as far as I have
> been able to check. There has been no repair tool for JXFS I think. But
> as I migrated the data on it to SFS, I was able to copy everything.
>
> Famous last words.
>
> Well especially the disk size was detected properly and there was no
> overflow like on Linux. So I´d say rather have on error less than one
> error more.
>
> Of course, it could also be an option to outright *refuse* to detect
> such disks with a big fat warning in kernel log that it may unsafe. But
> overflowing and thus on writes overwriting existing data is not safe.
>
> I do think it is safe enough, but what I do know about the internals
> about RDB (more than the average user certainly, but not as much as you
> or some other AmigaOS developer who digged deeper into that).
>
> So in case you´d rather see Linux to refuse to handle disks like that,
> that would also be fine with me. Just do not handle them in the broken
> way that they are handled in Linux now. I.e. do not deliberately corrupt
> things as in "Its dangerous, so let´s overwrite data straight away, so
> the user gets it." :)
>
> Anyway, in my opinion RDB still just so much more advanced than MBR and
> in some parts even on par with the much later GPT. With some limitations
> it is a quite brilliant partition format, if you ask me.
>
>> {^_^}
>>
>> On 20180625 19:23, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> Martin's boot log (including your patch) says:
>>>
>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
>>> spb
>>> 4)
>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>> Attached SCSI disk
>>>
>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>> byte blocks) and can be worked around by using a different block
>>> size.
>>>
>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>> I'll still submit a patch to Jens anyway as this may bite others
>>> yet.
>>>
>>> Cheers,
>>>
>>> Michael
>>>
>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]> wrote:
>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>> a famn dool.
>>>>
>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>> should work up to 2 gigablocks whatever your block size is. 512
>>>> blocks is 2199023255552 bytes. But that wastes just a WHOLE LOT of
>>>> disk in block maps. Go up to 4096 or 8192. The latter is 35 TB.
>>>>
>>>> {^_^}
>>>>
>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>> Hi.
>>>>>
>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>> indicate the RDB parser bug is fixed by the patch given there, so
>>>>>> if
>>>>>> Martin now submits the patch, all should be well?
>>>>>
>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>
>>>>> I do not care enough about this, in order to motivate myself
>>>>> preparing the a patch from Joanne Dow´s fix.
>>>>>
>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>> which I still have in my apartment.
>>>>>
>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>> unless someone else does.
>>>>>
>>>>> Thanks.
> […]
>

2018-06-27 02:33:09

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Joanne,

As far as I have been able to test, the change is backwards compatible
(RDB partitions from an old disk 80 GB disk are still recognized OK).
That"s only been done on an emulator though.

Your advice about the dangers of using RDB disks that would have
failed the current Linux RDB parser on legacy 32 bit systems is well
taken though. Maybe Martin can clarify that for me - was the 2 TB disk
in question ever used on a 32 bit Amiga system?

RDB disk format is meant for legacy use only, so I think we can get
away with printing a big fat warning during boot, advising the user
that the oversize RDB partition(s) scanned are not compatible with
legacy 32 bit AmigaOS. With the proposed fix they will work under both
AmigaOS 4.1 and Linux instead of truncating the first overflowing
partition at disk end and trashing valid partitions that overlap,
which is what Martin was after.

If that still seems too risky, we can make the default behaviour to
bail out once a potential overflow is detected, and allow the user to
override that through a boot parameter. I'd leave that decision up for
the code review on linux-block.

Two more comments: Linux uses 512 byte block sizes for the partition
start and size calculations, so a change of the RDB blocksize to
reduce the block counts stored in the RDB would still result in the
same overflow. And amiga-fdisk is indeed utterly broken and needs
updating (along with probably most legacy m68k partitioners). Adrian
has advertised parted as replacement for the old tools - maybe this
would make a nice test case for parted?

Cheers,

Michael

On Tue, Jun 26, 2018 at 9:45 PM, jdow <[email protected]> wrote:
> If it is not backwards compatible I for one would refuse to use it. And if
> it still mattered that much to me I'd also generate a reasonable
> alternative. Modifying RDBs nay not be even an approximation of a good idea.
> You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t
> only system. If it is for your personal use then you're entirely free to
> reject my advice and are probably smart enough to keep it working for
> yourself.
>
> GPT is probably the right way to go. Preserve the ability to read RDBs for
> legacy disks only.
>
> {^_^}
>
>
> On 20180626 01:31, Michael Schmitz wrote:
>>
>> Joanne,
>>
>> I think we all agree that doing 32 bit calculations on 512-byte block
>> addresses that overflow on disks 2 TB and larger is a bug, causing the
>> issues Martin reported. Your patch addresses that by using the correct
>> data type for the calculations (as do other partition parsers that may
>> have to deal with large disks) and fixes Martin's bug, so appears to be
>> the right thing to do.
>>
>> Using 64 bit data types for disks smaller than 2 TB where calculations
>> don't currently overflow is not expected to cause new issues, other than
>> enabling use of disk and partitions larger than 2 TB (which may have
>> ramifications with filesystems on these partitions). So comptibility is
>> preserved.
>>
>> Forcing larger block sizes might be a good strategy to avoid overflow
>> issues in filesystems as well, but I can't see how the block size stored
>> in the RDB would enforce use of the same block size in filesystems.
>> We'll have to rely on the filesystem tools to get that right, too. Linux
>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>> allow partitions larger than 2 TB to work already (but I suspect Al Viro
>> may have found a few issues when he looked at the AFFS code so I won't
>> say more). Anyway partitioning tools and filesystems are unrelated to
>> the Linux partition parser code which is all we aim to fix in this patch.
>>
>> If you feel strongly about unknown ramifications of any filesystems on
>> partitions larger than 2 TB, say so and I'll have the kernel print a
>> warning about these partitions.
>>
>> I'll get this patch tested on Martin's test case image as well as on a
>> RDB image from a disk known to currently work under Linux (thanks Geert
>> for the losetup hint). Can't do much more without procuring a working
>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
>> for tests is a long way away from my home indeed.
>>
>> Cheers,
>>
>> Michael
>>
>> Am 26.06.18 um 17:17 schrieb jdow:
>>>
>>> As long as it preserves compatibility it should be OK, I suppose.
>>> Personally I'd make any partitioning tool front end gently force the
>>> block size towards 8k as the disk size gets larger. The file systems
>>> may also run into 2TB issues that are not obvious. An unused blocks
>>> list will have to go beyond a uint32_t size, for example. But a block
>>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>>> 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}
>>>
>>> Just be sure you are aware of all the ramifications when you make a
>>> change. I remember thinking about this for awhile and then determining
>>> I REALLY did not want to think about it as my brain was getting tied
>>> into a gordian knot.
>>>
>>> {^_^}
>>>
>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>
>>>> Joanne,
>>>>
>>>> Martin's boot log (including your patch) says:
>>>>
>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>>> 4)
>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>> Attached SCSI disk
>>>>
>>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>>> byte blocks) and can be worked around by using a different block size.
>>>>
>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>>
>>>> Cheers,
>>>>
>>>> Michael
>>>>
>>>>
>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]> wrote:
>>>>>
>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>>> a famn
>>>>> dool.
>>>>>
>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>> should
>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>> block maps.
>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>
>>>>> {^_^}
>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>
>>>>>>
>>>>>> Hi.
>>>>>>
>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>
>>>>>>>
>>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>>>>> Martin now submits the patch, all should be well?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>
>>>>>> I do not care enough about this, in order to motivate myself preparing
>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>
>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>> which
>>>>>> I still have in my apartment.
>>>>>>
>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>> unless
>>>>>> someone else does.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-m68k" in
>>>>> the body of a message to [email protected]
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>>> the body of a message to [email protected]
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2018-06-27 07:24:43

by jdow

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

You allergic to using a GPT solution? It will get away from some of the evils
that RDB has inherent in it because they are also features? (Loading a
filesystem or DriveInit code from RDBs is just asking for a nearly impossible to
remove malware infection.) Furthermore, any 32 bit system that sees an RDSK
block is going to try to translate it. If you add a new RDB format you are going
to get bizarre and probably quite destructive results from the mistake. Fail
safe is a rather good notion, methinks.

Personally I figure this is all rather surreal. 2TG of junk on an Amiga system
seems utterly outlandish to me. You cited another overflow potential. There are
at least three we've identified, I believe. Are you 100% sure there are no more?
The specific one you mention of translating RDB to Linux has a proper solution
in the RDB reader. It should recover such overflow errors in the RDB as it can
with due care and polish. It should flag any other overflow error it detects
within the RDBs and return an error such as to leave the disk unmounted or
mounted read-only if you feel like messing up a poor sod's backups. The simple
solution is to read each of the variables with the nominal RDB size and convert
it to uint64_t before calculating byte indices.

However, consider my inputs as advice from an adult who has seen the Amiga
Elephant so to speak. I am not trying to assert any control. Do as you wish;
but, I would plead with you to avoid ANY chance you can for the user to make a
bonehead stupid move and lose all his treasured disk archives. Doing otherwise
is very poor form.

{o.o} Joanne "Said enough, she has" Dow

On 20180626 18:07, Michael Schmitz wrote:
> Joanne,
>
> As far as I have been able to test, the change is backwards compatible
> (RDB partitions from an old disk 80 GB disk are still recognized OK).
> That"s only been done on an emulator though.
>
> Your advice about the dangers of using RDB disks that would have
> failed the current Linux RDB parser on legacy 32 bit systems is well
> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
> in question ever used on a 32 bit Amiga system?
>
> RDB disk format is meant for legacy use only, so I think we can get
> away with printing a big fat warning during boot, advising the user
> that the oversize RDB partition(s) scanned are not compatible with
> legacy 32 bit AmigaOS. With the proposed fix they will work under both
> AmigaOS 4.1 and Linux instead of truncating the first overflowing
> partition at disk end and trashing valid partitions that overlap,
> which is what Martin was after.
>
> If that still seems too risky, we can make the default behaviour to
> bail out once a potential overflow is detected, and allow the user to
> override that through a boot parameter. I'd leave that decision up for
> the code review on linux-block.
>
> Two more comments: Linux uses 512 byte block sizes for the partition
> start and size calculations, so a change of the RDB blocksize to
> reduce the block counts stored in the RDB would still result in the
> same overflow. And amiga-fdisk is indeed utterly broken and needs
> updating (along with probably most legacy m68k partitioners). Adrian
> has advertised parted as replacement for the old tools - maybe this
> would make a nice test case for parted?
>
> Cheers,
>
> Michael
>
>
>
> On Tue, Jun 26, 2018 at 9:45 PM, jdow <[email protected]> wrote:
>> If it is not backwards compatible I for one would refuse to use it. And if
>> it still mattered that much to me I'd also generate a reasonable
>> alternative. Modifying RDBs nay not be even an approximation of a good idea.
>> You'd discover that as soon as an RDB uint64_t disk is tasted by a uint32_t
>> only system. If it is for your personal use then you're entirely free to
>> reject my advice and are probably smart enough to keep it working for
>> yourself.
>>
>> GPT is probably the right way to go. Preserve the ability to read RDBs for
>> legacy disks only.
>>
>> {^_^}
>>
>>
>> On 20180626 01:31, Michael Schmitz wrote:
>>>
>>> Joanne,
>>>
>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>> addresses that overflow on disks 2 TB and larger is a bug, causing the
>>> issues Martin reported. Your patch addresses that by using the correct
>>> data type for the calculations (as do other partition parsers that may
>>> have to deal with large disks) and fixes Martin's bug, so appears to be
>>> the right thing to do.
>>>
>>> Using 64 bit data types for disks smaller than 2 TB where calculations
>>> don't currently overflow is not expected to cause new issues, other than
>>> enabling use of disk and partitions larger than 2 TB (which may have
>>> ramifications with filesystems on these partitions). So comptibility is
>>> preserved.
>>>
>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>> issues in filesystems as well, but I can't see how the block size stored
>>> in the RDB would enforce use of the same block size in filesystems.
>>> We'll have to rely on the filesystem tools to get that right, too. Linux
>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>> allow partitions larger than 2 TB to work already (but I suspect Al Viro
>>> may have found a few issues when he looked at the AFFS code so I won't
>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>> the Linux partition parser code which is all we aim to fix in this patch.
>>>
>>> If you feel strongly about unknown ramifications of any filesystems on
>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>> warning about these partitions.
>>>
>>> I'll get this patch tested on Martin's test case image as well as on a
>>> RDB image from a disk known to currently work under Linux (thanks Geert
>>> for the losetup hint). Can't do much more without procuring a working
>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to use
>>> for tests is a long way away from my home indeed.
>>>
>>> Cheers,
>>>
>>> Michael
>>>
>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>
>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>> Personally I'd make any partitioning tool front end gently force the
>>>> block size towards 8k as the disk size gets larger. The file systems
>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>> list will have to go beyond a uint32_t size, for example. But a block
>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>>>> 1% of the disk all by itself. A block bitmap is not quite so bad. {^_-}
>>>>
>>>> Just be sure you are aware of all the ramifications when you make a
>>>> change. I remember thinking about this for awhile and then determining
>>>> I REALLY did not want to think about it as my brain was getting tied
>>>> into a gordian knot.
>>>>
>>>> {^_^}
>>>>
>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>
>>>>> Joanne,
>>>>>
>>>>> Martin's boot log (including your patch) says:
>>>>>
>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>>>> 4)
>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>> Attached SCSI disk
>>>>>
>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>>>> byte blocks) and can be worked around by using a different block size.
>>>>>
>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Michael
>>>>>
>>>>>
>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]> wrote:
>>>>>>
>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>>>> a famn
>>>>>> dool.
>>>>>>
>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>> should
>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>> block maps.
>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>
>>>>>> {^_^}
>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi.
>>>>>>>
>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>
>>>>>>>>
>>>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>> indicate the RDB parser bug is fixed by the patch given there, so if
>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>
>>>>>>> I do not care enough about this, in order to motivate myself preparing
>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>
>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>> which
>>>>>>> I still have in my apartment.
>>>>>>>
>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>> unless
>>>>>>> someone else does.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-m68k" in
>>>>>> the body of a message to [email protected]
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>>>> the body of a message to [email protected]
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>

2018-06-27 08:05:43

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Dear Joanne.

jdow - 27.06.18, 08:24:
> You allergic to using a GPT solution? It will get away from some of
> the evils that RDB has inherent in it because they are also features?
> (Loading a filesystem or DriveInit code from RDBs is just asking for
> a nearly impossible to remove malware infection.) Furthermore, any 32
> bit system that sees an RDSK block is going to try to translate it.
> If you add a new RDB format you are going to get bizarre and probably
> quite destructive results from the mistake. Fail safe is a rather
> good notion, methinks.
>
> Personally I figure this is all rather surreal. 2TG of junk on an
> Amiga system seems utterly outlandish to me. You cited another
> overflow potential. There are at least three we've identified, I
> believe. Are you 100% sure there are no more? The specific one you
> mention of translating RDB to Linux has a proper solution in the RDB
> reader. It should recover such overflow errors in the RDB as it can
> with due care and polish. It should flag any other overflow error it
> detects within the RDBs and return an error such as to leave the disk
> unmounted or mounted read-only if you feel like messing up a poor
> sod's backups. The simple solution is to read each of the variables
> with the nominal RDB size and convert it to uint64_t before
> calculating byte indices.
>
> However, consider my inputs as advice from an adult who has seen the
> Amiga Elephant so to speak. I am not trying to assert any control. Do
> as you wish; but, I would plead with you to avoid ANY chance you can
> for the user to make a bonehead stupid move and lose all his
> treasured disk archives. Doing otherwise is very poor form.

I am pretty confident that larger than 2 TiB disks are fully supported
within AmigaOS 4, as I outlined in my other mail.

So with all due respect: I used a larger than 2 TiB disk in AmigaOS 4 in
2012 already *just* fine. I even found I had the same questions back
then, and researched it. Which lead to this official article back then:

http://wiki.amigaos.net/wiki/RDB

I am also pretty sure that AmigaOS still uses RDB as partitioning
format. They support MBR. I don?t think AmigaOS 4.1 supports GPT.
Whether to implement that of course is the decision of AmigaOS 4
development team. I am no longer a member of it since some time.

Linux m68k should already be able to use disks in GPT format, but you
likely won?t be able to read them in AmigaOS, unless there is some third
party support for it meanwhile.

Thanks,
Martin

>
> {o.o} Joanne "Said enough, she has" Dow
>
> On 20180626 18:07, Michael Schmitz wrote:
> > Joanne,
> >
> > As far as I have been able to test, the change is backwards
> > compatible (RDB partitions from an old disk 80 GB disk are still
> > recognized OK). That"s only been done on an emulator though.
> >
> > Your advice about the dangers of using RDB disks that would have
> > failed the current Linux RDB parser on legacy 32 bit systems is well
> > taken though. Maybe Martin can clarify that for me - was the 2 TB
> > disk in question ever used on a 32 bit Amiga system?
> >
> > RDB disk format is meant for legacy use only, so I think we can get
> > away with printing a big fat warning during boot, advising the user
> > that the oversize RDB partition(s) scanned are not compatible with
> > legacy 32 bit AmigaOS. With the proposed fix they will work under
> > both AmigaOS 4.1 and Linux instead of truncating the first
> > overflowing partition at disk end and trashing valid partitions
> > that overlap, which is what Martin was after.
> >
> > If that still seems too risky, we can make the default behaviour to
> > bail out once a potential overflow is detected, and allow the user
> > to
> > override that through a boot parameter. I'd leave that decision up
> > for the code review on linux-block.
> >
> > Two more comments: Linux uses 512 byte block sizes for the partition
> > start and size calculations, so a change of the RDB blocksize to
> > reduce the block counts stored in the RDB would still result in the
> > same overflow. And amiga-fdisk is indeed utterly broken and needs
> > updating (along with probably most legacy m68k partitioners). Adrian
> > has advertised parted as replacement for the old tools - maybe this
> > would make a nice test case for parted?
> >
> > Cheers,
> >
> > Michael
> >
> > On Tue, Jun 26, 2018 at 9:45 PM, jdow <[email protected]> wrote:
> >> If it is not backwards compatible I for one would refuse to use it.
> >> And if it still mattered that much to me I'd also generate a
> >> reasonable alternative. Modifying RDBs nay not be even an
> >> approximation of a good idea. You'd discover that as soon as an
> >> RDB uint64_t disk is tasted by a uint32_t only system. If it is
> >> for your personal use then you're entirely free to reject my
> >> advice and are probably smart enough to keep it working for
> >> yourself.
> >>
> >> GPT is probably the right way to go. Preserve the ability to read
> >> RDBs for legacy disks only.
> >>
> >> {^_^}
> >>
> >> On 20180626 01:31, Michael Schmitz wrote:
> >>> Joanne,
> >>>
> >>> I think we all agree that doing 32 bit calculations on 512-byte
> >>> block
> >>> addresses that overflow on disks 2 TB and larger is a bug, causing
> >>> the issues Martin reported. Your patch addresses that by using
> >>> the correct data type for the calculations (as do other partition
> >>> parsers that may have to deal with large disks) and fixes
> >>> Martin's bug, so appears to be the right thing to do.
> >>>
> >>> Using 64 bit data types for disks smaller than 2 TB where
> >>> calculations don't currently overflow is not expected to cause
> >>> new issues, other than enabling use of disk and partitions larger
> >>> than 2 TB (which may have ramifications with filesystems on these
> >>> partitions). So comptibility is preserved.
> >>>
> >>> Forcing larger block sizes might be a good strategy to avoid
> >>> overflow
> >>> issues in filesystems as well, but I can't see how the block size
> >>> stored in the RDB would enforce use of the same block size in
> >>> filesystems. We'll have to rely on the filesystem tools to get
> >>> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
> >>> limitation) so this should allow partitions larger than 2 TB to
> >>> work already (but I suspect Al Viro may have found a few issues
> >>> when he looked at the AFFS code so I won't say more). Anyway
> >>> partitioning tools and filesystems are unrelated to the Linux
> >>> partition parser code which is all we aim to fix in this patch.
> >>>
> >>> If you feel strongly about unknown ramifications of any
> >>> filesystems on partitions larger than 2 TB, say so and I'll have
> >>> the kernel print a warning about these partitions.
> >>>
> >>> I'll get this patch tested on Martin's test case image as well as
> >>> on a RDB image from a disk known to currently work under Linux
> >>> (thanks Geert for the losetup hint). Can't do much more without
> >>> procuring a working Amiga disk image to use with an emulator,
> >>> sorry. The Amiga I plan to use for tests is a long way away from
> >>> my home indeed.
> >>>
> >>> Cheers,
> >>>
> >>> Michael
> >>>
> >>> Am 26.06.18 um 17:17 schrieb jdow:
> >>>> As long as it preserves compatibility it should be OK, I suppose.
> >>>> Personally I'd make any partitioning tool front end gently force
> >>>> the
> >>>> block size towards 8k as the disk size gets larger. The file
> >>>> systems
> >>>> may also run into 2TB issues that are not obvious. An unused
> >>>> blocks
> >>>> list will have to go beyond a uint32_t size, for example. But a
> >>>> block
> >>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
> >>>> under 1% of the disk all by itself. A block bitmap is not quite
> >>>> so bad. {^_-}
> >>>>
> >>>> Just be sure you are aware of all the ramifications when you make
> >>>> a
> >>>> change. I remember thinking about this for awhile and then
> >>>> determining I REALLY did not want to think about it as my brain
> >>>> was getting tied into a gordian knot.
> >>>>
> >>>> {^_^}
> >>>>
> >>>> On 20180625 19:23, Michael Schmitz wrote:
> >>>>> Joanne,
> >>>>>
> >>>>> Martin's boot log (including your patch) says:
> >>>>>
> >>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512)
> >>>>> sdb1
> >>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
> >>>>> 2 spb
> >>>>> 4)
> >>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> >>>>> Attached SCSI disk
> >>>>>
> >>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
> >>>>> 512
> >>>>> byte blocks) and can be worked around by using a different block
> >>>>> size.
> >>>>>
> >>>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>>> units.
> >>>>> I'll still submit a patch to Jens anyway as this may bite others
> >>>>> yet.
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Michael
> >>>>>
> >>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]>
wrote:
> >>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
> >>>>>> system is
> >>>>>> a famn
> >>>>>> dool.
> >>>>>>
> >>>>>> If memory serves the RDBs think in blocks rather than bytes so
> >>>>>> it
> >>>>>> should
> >>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks
> >>>>>> is
> >>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk
> >>>>>> in
> >>>>>> block maps.
> >>>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>>
> >>>>>> {^_^}
> >>>>>>
> >>>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>>> Hi.
> >>>>>>>
> >>>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>>> test results at
> >>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>>> indicate the RDB parser bug is fixed by the patch given
> >>>>>>>> there, so if
> >>>>>>>> Martin now submits the patch, all should be well?
> >>>>>>>
> >>>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>>
> >>>>>>> I do not care enough about this, in order to motivate myself
> >>>>>>> preparing the a patch from Joanne Dow?s fix.
> >>>>>>>
> >>>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>>> Sam440ep
> >>>>>>> which
> >>>>>>> I still have in my apartment.
> >>>>>>>
> >>>>>>> So RDB support in Linux it remains broken for disks larger 2
> >>>>>>> TB,
> >>>>>>> unless
> >>>>>>> someone else does.
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>
> >>>>>> --
> >>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>> linux-m68k" in
> >>>>>> the body of a message to [email protected]
> >>>>>> More majordomo info at
> >>>>>> http://vger.kernel.org/majordomo-info.html
> >>>>
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe
> >>>> linux-m68k" in the body of a message to
> >>>> [email protected]
> >>>> More majordomo info at
> >>>> http://vger.kernel.org/majordomo-info.html
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe
> >> linux-m68k" in the body of a message to [email protected]
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Martin

2018-06-27 09:02:57

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Joanne,

I'm not at all allergic to avoiding RDB at all cost for new disks. If
AmigaOS 4.1 supports more recent partition formats, all the better. This
is all about supporting use of legacy RDB disks on Linux (though 2 TB
does stretch the definition of 'legacy' a little). My interest in this
is to ensure we can continue to use RDB format disks on m68k Amiga
computers which have no other way to boot Linux from disk.

Not proposing to change the RDB format at all, either. Just trying to
make sure we translate RDB info into Linux 512-byte block offset and
size numbers correctly. The kernel won't modify the RDB at all
(intentionally, that is - with the correct choice of partition sizes,
Martin might well have wiped out his RDB with the current version of the
parser).

The choice of refusing to mount a disk (or mounting read-only) rests
with the VFS drivers alone - AFFS in that case. Not touching any of
that. At partition scan time, we only have the option of making the
partition available (with a warning printed), or refusing to make it
available to the kernel. Once it's made available, all bets are off.

From what Martin writes, his test case RDB was valid and worked as
expected on 32 bit AmigaOS (4.1). Apparently, that version has the
necessary extensions to handle the large offsets resulting from 2 TB
disks. Not sure what safeguards are in place when connecting such a disk
to older versions of AmigaOS, but that is a different matter entirely.

The overflows in partition offset and size are the only ones I can see
in the partition parser - there is no other overflow I've identified. I
just stated that in order to place a partition towards the end of a 2 TB
disk, the offset calculation will overflow regardless of what
combination of rdb->rdb_BlockBytes and sector addresses stored in the
RDB (in units of 512 byte blocks) we use:

blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;

nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 -
be32_to_cpu(pb->pb_Environment[9])) *
be32_to_cpu(pb->pb_Environment[3]) *
be32_to_cpu(pb->pb_Environment[5]) *
blksize;
if (!nr_sects)
continue;
start_sect = be32_to_cpu(pb->pb_Environment[9]) *
be32_to_cpu(pb->pb_Environment[3]) *
be32_to_cpu(pb->pb_Environment[5]) *
blksize;

But in the interest of avoiding any accidental use of a RDB partition
where calculations currently overflow, I'll make the default behaviour
to bail out (instead of using wrong offset or size as we currently do).
Given the 'eat_my_RDB_disk=1' boot option, the user may proceed at their
own risk (though I still can't see what harm should result from now
translating a well formed v4.1 2 TB disk RDB correctly for the first time).

Whether or not Linux correctly handles AFFS filesystems larger than 1 TB
is a matter for VFS experts. Bailing out on nr_sects overflowing ought
to prevent accidental use of AFFS filesystems on RDB disks which I
suppose is the majority of use cases.

Bugs in partitioning tools on Linux are entirely out of scope - the
partitioning tools bypass the partition structure discovered by the
kernel, and work straight on the raw device. No protecting against that.

If you can point out a way to cause data loss with these precautions,
for a disk 2 TB or larger that was partitioned and used on a recent
version or AmigaOS supporting such large disks, I'd consider omitting
the 'eat_my_RDB_disk' boot option, and just bail out as the only safe
option.

Cheers,

Michael

Am 27.06.2018 um 18:24 schrieb jdow:
> You allergic to using a GPT solution? It will get away from some of the
> evils that RDB has inherent in it because they are also features?
> (Loading a filesystem or DriveInit code from RDBs is just asking for a
> nearly impossible to remove malware infection.) Furthermore, any 32 bit
> system that sees an RDSK block is going to try to translate it. If you
> add a new RDB format you are going to get bizarre and probably quite
> destructive results from the mistake. Fail safe is a rather good notion,
> methinks.
>
> Personally I figure this is all rather surreal. 2TG of junk on an Amiga
> system seems utterly outlandish to me. You cited another overflow
> potential. There are at least three we've identified, I believe. Are you
> 100% sure there are no more? The specific one you mention of translating
> RDB to Linux has a proper solution in the RDB reader. It should recover
> such overflow errors in the RDB as it can with due care and polish. It
> should flag any other overflow error it detects within the RDBs and
> return an error such as to leave the disk unmounted or mounted read-only
> if you feel like messing up a poor sod's backups. The simple solution is
> to read each of the variables with the nominal RDB size and convert it
> to uint64_t before calculating byte indices.
>
> However, consider my inputs as advice from an adult who has seen the
> Amiga Elephant so to speak. I am not trying to assert any control. Do as
> you wish; but, I would plead with you to avoid ANY chance you can for
> the user to make a bonehead stupid move and lose all his treasured disk
> archives. Doing otherwise is very poor form.
>
> {o.o} Joanne "Said enough, she has" Dow
>
> On 20180626 18:07, Michael Schmitz wrote:
>> Joanne,
>>
>> As far as I have been able to test, the change is backwards compatible
>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>> That"s only been done on an emulator though.
>>
>> Your advice about the dangers of using RDB disks that would have
>> failed the current Linux RDB parser on legacy 32 bit systems is well
>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>> in question ever used on a 32 bit Amiga system?
>>
>> RDB disk format is meant for legacy use only, so I think we can get
>> away with printing a big fat warning during boot, advising the user
>> that the oversize RDB partition(s) scanned are not compatible with
>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>> partition at disk end and trashing valid partitions that overlap,
>> which is what Martin was after.
>>
>> If that still seems too risky, we can make the default behaviour to
>> bail out once a potential overflow is detected, and allow the user to
>> override that through a boot parameter. I'd leave that decision up for
>> the code review on linux-block.
>>
>> Two more comments: Linux uses 512 byte block sizes for the partition
>> start and size calculations, so a change of the RDB blocksize to
>> reduce the block counts stored in the RDB would still result in the
>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>> updating (along with probably most legacy m68k partitioners). Adrian
>> has advertised parted as replacement for the old tools - maybe this
>> would make a nice test case for parted?
>>
>> Cheers,
>>
>> Michael
>>
>>
>>
>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <[email protected]> wrote:
>>> If it is not backwards compatible I for one would refuse to use it.
>>> And if
>>> it still mattered that much to me I'd also generate a reasonable
>>> alternative. Modifying RDBs nay not be even an approximation of a
>>> good idea.
>>> You'd discover that as soon as an RDB uint64_t disk is tasted by a
>>> uint32_t
>>> only system. If it is for your personal use then you're entirely free to
>>> reject my advice and are probably smart enough to keep it working for
>>> yourself.
>>>
>>> GPT is probably the right way to go. Preserve the ability to read
>>> RDBs for
>>> legacy disks only.
>>>
>>> {^_^}
>>>
>>>
>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>
>>>> Joanne,
>>>>
>>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>>> addresses that overflow on disks 2 TB and larger is a bug, causing the
>>>> issues Martin reported. Your patch addresses that by using the correct
>>>> data type for the calculations (as do other partition parsers that may
>>>> have to deal with large disks) and fixes Martin's bug, so appears to be
>>>> the right thing to do.
>>>>
>>>> Using 64 bit data types for disks smaller than 2 TB where calculations
>>>> don't currently overflow is not expected to cause new issues, other
>>>> than
>>>> enabling use of disk and partitions larger than 2 TB (which may have
>>>> ramifications with filesystems on these partitions). So comptibility is
>>>> preserved.
>>>>
>>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>>> issues in filesystems as well, but I can't see how the block size
>>>> stored
>>>> in the RDB would enforce use of the same block size in filesystems.
>>>> We'll have to rely on the filesystem tools to get that right, too.
>>>> Linux
>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>>> allow partitions larger than 2 TB to work already (but I suspect Al
>>>> Viro
>>>> may have found a few issues when he looked at the AFFS code so I won't
>>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>>> the Linux partition parser code which is all we aim to fix in this
>>>> patch.
>>>>
>>>> If you feel strongly about unknown ramifications of any filesystems on
>>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>>> warning about these partitions.
>>>>
>>>> I'll get this patch tested on Martin's test case image as well as on a
>>>> RDB image from a disk known to currently work under Linux (thanks Geert
>>>> for the losetup hint). Can't do much more without procuring a working
>>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to
>>>> use
>>>> for tests is a long way away from my home indeed.
>>>>
>>>> Cheers,
>>>>
>>>> Michael
>>>>
>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>
>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>> Personally I'd make any partitioning tool front end gently force the
>>>>> block size towards 8k as the disk size gets larger. The file systems
>>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>>> list will have to go beyond a uint32_t size, for example. But a block
>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>>>>> 1% of the disk all by itself. A block bitmap is not quite so bad.
>>>>> {^_-}
>>>>>
>>>>> Just be sure you are aware of all the ramifications when you make a
>>>>> change. I remember thinking about this for awhile and then determining
>>>>> I REALLY did not want to think about it as my brain was getting tied
>>>>> into a gordian knot.
>>>>>
>>>>> {^_^}
>>>>>
>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>
>>>>>> Joanne,
>>>>>>
>>>>>> Martin's boot log (including your patch) says:
>>>>>>
>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>>>>> 4)
>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>> Attached SCSI disk
>>>>>>
>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>>>>> byte blocks) and can be worked around by using a different block
>>>>>> size.
>>>>>>
>>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Michael
>>>>>>
>>>>>>
>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]> wrote:
>>>>>>>
>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>>>>> a famn
>>>>>>> dool.
>>>>>>>
>>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>>> should
>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>> block maps.
>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>
>>>>>>> {^_^}
>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi.
>>>>>>>>
>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>> so if
>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>
>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>> preparing
>>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>>
>>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>>> which
>>>>>>>> I still have in my apartment.
>>>>>>>>
>>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>>> unless
>>>>>>>> someone else does.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>> linux-m68k" in
>>>>>>> the body of a message to [email protected]
>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-m68k" in
>>>>> the body of a message to [email protected]
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>>> the body of a message to [email protected]
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>

2018-06-27 09:23:49

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Michael Schmitz - 27.06.18, 03:07:
> Joanne,
>
> As far as I have been able to test, the change is backwards compatible
> (RDB partitions from an old disk 80 GB disk are still recognized OK).
> That"s only been done on an emulator though.
>
> Your advice about the dangers of using RDB disks that would have
> failed the current Linux RDB parser on legacy 32 bit systems is well
> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
> in question ever used on a 32 bit Amiga system?

Sure! Are there actually *any* 64 bit Amiga systems? I am not completely
sure architecture-wise, but the operating system is still just 32-Bit.

However, on AmigaOS officially since at least 3.5/3.9 there are various
mechanisms to handle 64-bit block numbers for disk devices, called
NSD64, TD64 and scsi direct¹. One of them, NSD 64 in AmigaOS 3.5/3.9, is
part of the operating system. First via SetPatch based update to
Kickstart ROM, but with AmigaOS 4.x its just included.

[1] http://www.amigawiki.de/doku.php?id=de:system:filesystems_limits

(I do not agree with maximum filesystem size limits stated there for
Amiga Fast Filesystem (FFS), but well, for early FFS versions they could
have been true, I am pretty sure that at least with higher block sizes
you can have FFS > 2 GiB in size. And I remember having had JFXS and I
think also SFS2 partitions in AmigaOS 4 with more than 100 GiB.)

> RDB disk format is meant for legacy use only, so I think we can get
> away with printing a big fat warning during boot, advising the user
> that the oversize RDB partition(s) scanned are not compatible with
> legacy 32 bit AmigaOS. With the proposed fix they will work under both
> AmigaOS 4.1 and Linux instead of truncating the first overflowing
> partition at disk end and trashing valid partitions that overlap,
> which is what Martin was after.

They are compatible. For AmigaOS 4.x I am completely sure, cause I
installed and used the disk there. That was the original motivation for
my Linux bug report back then: I used the disk in AmigaOS 4 *just fine*
and it broke in Linux. I used Media Toolbox in order to create the RDB I
posted back then.

Its difficult to find proof for my claims online, but here is an
official one:

http://wiki.amigaos.net/wiki/RDB

Limits of filesystem sizes may of course be different, but here we are
talking about RDB.

Also this confirms that RDB only uses 512 byte block size on AmigaOS 4.

Hmmm, I even created the page back then. Interestingly at about the same
time I made the bug report about the Linux issue. The calculations in
there are not really from me. I bet I copied them over from what other
AmigaOS developers wrote on development mailing lists, at that time.
With their agreement. I think I started my research back then due by my
own question on: Will such a large disk actually work in AmigaOS?

It is all a long time ago…

> If that still seems too risky, we can make the default behaviour to
> bail out once a potential overflow is detected, and allow the user to
> override that through a boot parameter. I'd leave that decision up for
> the code review on linux-block.
>
> Two more comments: Linux uses 512 byte block sizes for the partition
> start and size calculations, so a change of the RDB blocksize to
> reduce the block counts stored in the RDB would still result in the
> same overflow. And amiga-fdisk is indeed utterly broken and needs
> updating (along with probably most legacy m68k partitioners). Adrian
> has advertised parted as replacement for the old tools - maybe this
> would make a nice test case for parted?

I always used AmigaOS based tools to partition in RDB format, cause I
never trusted amiga-fdisk :).

> On Tue, Jun 26, 2018 at 9:45 PM, jdow <[email protected]> wrote:
> > If it is not backwards compatible I for one would refuse to use it.
> > And if it still mattered that much to me I'd also generate a
> > reasonable alternative. Modifying RDBs nay not be even an
> > approximation of a good idea. You'd discover that as soon as an RDB
> > uint64_t disk is tasted by a uint32_t only system. If it is for
> > your personal use then you're entirely free to reject my advice and
> > are probably smart enough to keep it working for yourself.
> >
> > GPT is probably the right way to go. Preserve the ability to read
> > RDBs for legacy disks only.
> >
> > {^_^}
> >
> > On 20180626 01:31, Michael Schmitz wrote:
> >> Joanne,
> >>
> >> I think we all agree that doing 32 bit calculations on 512-byte
> >> block
> >> addresses that overflow on disks 2 TB and larger is a bug, causing
> >> the issues Martin reported. Your patch addresses that by using the
> >> correct data type for the calculations (as do other partition
> >> parsers that may have to deal with large disks) and fixes Martin's
> >> bug, so appears to be the right thing to do.
> >>
> >> Using 64 bit data types for disks smaller than 2 TB where
> >> calculations don't currently overflow is not expected to cause new
> >> issues, other than enabling use of disk and partitions larger than
> >> 2 TB (which may have ramifications with filesystems on these
> >> partitions). So comptibility is preserved.
> >>
> >> Forcing larger block sizes might be a good strategy to avoid
> >> overflow
> >> issues in filesystems as well, but I can't see how the block size
> >> stored in the RDB would enforce use of the same block size in
> >> filesystems. We'll have to rely on the filesystem tools to get
> >> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
> >> limitation) so this should allow partitions larger than 2 TB to
> >> work already (but I suspect Al Viro may have found a few issues
> >> when he looked at the AFFS code so I won't say more). Anyway
> >> partitioning tools and filesystems are unrelated to the Linux
> >> partition parser code which is all we aim to fix in this patch.
> >>
> >> If you feel strongly about unknown ramifications of any filesystems
> >> on partitions larger than 2 TB, say so and I'll have the kernel
> >> print a warning about these partitions.
> >>
> >> I'll get this patch tested on Martin's test case image as well as
> >> on a RDB image from a disk known to currently work under Linux
> >> (thanks Geert for the losetup hint). Can't do much more without
> >> procuring a working Amiga disk image to use with an emulator,
> >> sorry. The Amiga I plan to use for tests is a long way away from
> >> my home indeed.
> >>
> >> Cheers,
> >>
> >> Michael
> >>
> >> Am 26.06.18 um 17:17 schrieb jdow:
> >>> As long as it preserves compatibility it should be OK, I suppose.
> >>> Personally I'd make any partitioning tool front end gently force
> >>> the
> >>> block size towards 8k as the disk size gets larger. The file
> >>> systems
> >>> may also run into 2TB issues that are not obvious. An unused
> >>> blocks
> >>> list will have to go beyond a uint32_t size, for example. But a
> >>> block
> >>> list (OFS for sure, don't remember for the newer AFS) uses a tad
> >>> under 1% of the disk all by itself. A block bitmap is not quite
> >>> so bad. {^_-}
> >>>
> >>> Just be sure you are aware of all the ramifications when you make
> >>> a
> >>> change. I remember thinking about this for awhile and then
> >>> determining I REALLY did not want to think about it as my brain
> >>> was getting tied into a gordian knot.
> >>>
> >>> {^_^}
> >>>
> >>> On 20180625 19:23, Michael Schmitz wrote:
> >>>> Joanne,
> >>>>
> >>>> Martin's boot log (including your patch) says:
> >>>>
> >>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512)
> >>>> sdb1
> >>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
> >>>> spb
> >>>> 4)
> >>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
> >>>> Attached SCSI disk
> >>>>
> >>>> so it's indeed a case of self inflicted damage (RDSK (512) means
> >>>> 512
> >>>> byte blocks) and can be worked around by using a different block
> >>>> size.
> >>>>
> >>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>> units.
> >>>> I'll still submit a patch to Jens anyway as this may bite others
> >>>> yet.
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Michael
> >>>>
> >>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]>
wrote:
> >>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system
> >>>>> is
> >>>>> a famn
> >>>>> dool.
> >>>>>
> >>>>> If memory serves the RDBs think in blocks rather than bytes so
> >>>>> it
> >>>>> should
> >>>>> work up to 2 gigablocks whatever your block size is. 512 blocks
> >>>>> is
> >>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
> >>>>> block maps.
> >>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>
> >>>>> {^_^}
> >>>>>
> >>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>> Hi.
> >>>>>>
> >>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>> test results at
> >>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>> indicate the RDB parser bug is fixed by the patch given there,
> >>>>>>> so if
> >>>>>>> Martin now submits the patch, all should be well?
> >>>>>>
> >>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>
> >>>>>> I do not care enough about this, in order to motivate myself
> >>>>>> preparing the a patch from Joanne Dow´s fix.
> >>>>>>
> >>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>> Sam440ep
> >>>>>> which
> >>>>>> I still have in my apartment.
> >>>>>>
> >>>>>> So RDB support in Linux it remains broken for disks larger 2
> >>>>>> TB,
> >>>>>> unless
> >>>>>> someone else does.
> >>>>>>
> >>>>>> Thanks.
> >>>>>
> >>>>> --
> >>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>> linux-m68k" in
> >>>>> the body of a message to [email protected]
> >>>>> More majordomo info at
> >>>>> http://vger.kernel.org/majordomo-info.html
> >>>
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe
> >>> linux-m68k" in the body of a message to [email protected]
> >>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> > linux-m68k" in the body of a message to [email protected]
> > More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Martin

2018-06-28 03:00:49

by jdow

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

The issue is what happens when one of those disks appears on a 3.1 system.
{^_^}

On 20180627 01:03, Martin Steigerwald wrote:
> Dear Joanne.
>
> jdow - 27.06.18, 08:24:
>> You allergic to using a GPT solution? It will get away from some of
>> the evils that RDB has inherent in it because they are also features?
>> (Loading a filesystem or DriveInit code from RDBs is just asking for
>> a nearly impossible to remove malware infection.) Furthermore, any 32
>> bit system that sees an RDSK block is going to try to translate it.
>> If you add a new RDB format you are going to get bizarre and probably
>> quite destructive results from the mistake. Fail safe is a rather
>> good notion, methinks.
>>
>> Personally I figure this is all rather surreal. 2TG of junk on an
>> Amiga system seems utterly outlandish to me. You cited another
>> overflow potential. There are at least three we've identified, I
>> believe. Are you 100% sure there are no more? The specific one you
>> mention of translating RDB to Linux has a proper solution in the RDB
>> reader. It should recover such overflow errors in the RDB as it can
>> with due care and polish. It should flag any other overflow error it
>> detects within the RDBs and return an error such as to leave the disk
>> unmounted or mounted read-only if you feel like messing up a poor
>> sod's backups. The simple solution is to read each of the variables
>> with the nominal RDB size and convert it to uint64_t before
>> calculating byte indices.
>>
>> However, consider my inputs as advice from an adult who has seen the
>> Amiga Elephant so to speak. I am not trying to assert any control. Do
>> as you wish; but, I would plead with you to avoid ANY chance you can
>> for the user to make a bonehead stupid move and lose all his
>> treasured disk archives. Doing otherwise is very poor form.
>
> I am pretty confident that larger than 2 TiB disks are fully supported
> within AmigaOS 4, as I outlined in my other mail.
>
> So with all due respect: I used a larger than 2 TiB disk in AmigaOS 4 in
> 2012 already *just* fine. I even found I had the same questions back
> then, and researched it. Which lead to this official article back then:
>
> http://wiki.amigaos.net/wiki/RDB
>
> I am also pretty sure that AmigaOS still uses RDB as partitioning
> format. They support MBR. I don´t think AmigaOS 4.1 supports GPT.
> Whether to implement that of course is the decision of AmigaOS 4
> development team. I am no longer a member of it since some time.
>
> Linux m68k should already be able to use disks in GPT format, but you
> likely won´t be able to read them in AmigaOS, unless there is some third
> party support for it meanwhile.
>
> Thanks,
> Martin
>
>>
>> {o.o} Joanne "Said enough, she has" Dow
>>
>> On 20180626 18:07, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> As far as I have been able to test, the change is backwards
>>> compatible (RDB partitions from an old disk 80 GB disk are still
>>> recognized OK). That"s only been done on an emulator though.
>>>
>>> Your advice about the dangers of using RDB disks that would have
>>> failed the current Linux RDB parser on legacy 32 bit systems is well
>>> taken though. Maybe Martin can clarify that for me - was the 2 TB
>>> disk in question ever used on a 32 bit Amiga system?
>>>
>>> RDB disk format is meant for legacy use only, so I think we can get
>>> away with printing a big fat warning during boot, advising the user
>>> that the oversize RDB partition(s) scanned are not compatible with
>>> legacy 32 bit AmigaOS. With the proposed fix they will work under
>>> both AmigaOS 4.1 and Linux instead of truncating the first
>>> overflowing partition at disk end and trashing valid partitions
>>> that overlap, which is what Martin was after.
>>>
>>> If that still seems too risky, we can make the default behaviour to
>>> bail out once a potential overflow is detected, and allow the user
>>> to
>>> override that through a boot parameter. I'd leave that decision up
>>> for the code review on linux-block.
>>>
>>> Two more comments: Linux uses 512 byte block sizes for the partition
>>> start and size calculations, so a change of the RDB blocksize to
>>> reduce the block counts stored in the RDB would still result in the
>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>>> updating (along with probably most legacy m68k partitioners). Adrian
>>> has advertised parted as replacement for the old tools - maybe this
>>> would make a nice test case for parted?
>>>
>>> Cheers,
>>>
>>> Michael
>>>
>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <[email protected]> wrote:
>>>> If it is not backwards compatible I for one would refuse to use it.
>>>> And if it still mattered that much to me I'd also generate a
>>>> reasonable alternative. Modifying RDBs nay not be even an
>>>> approximation of a good idea. You'd discover that as soon as an
>>>> RDB uint64_t disk is tasted by a uint32_t only system. If it is
>>>> for your personal use then you're entirely free to reject my
>>>> advice and are probably smart enough to keep it working for
>>>> yourself.
>>>>
>>>> GPT is probably the right way to go. Preserve the ability to read
>>>> RDBs for legacy disks only.
>>>>
>>>> {^_^}
>>>>
>>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>> Joanne,
>>>>>
>>>>> I think we all agree that doing 32 bit calculations on 512-byte
>>>>> block
>>>>> addresses that overflow on disks 2 TB and larger is a bug, causing
>>>>> the issues Martin reported. Your patch addresses that by using
>>>>> the correct data type for the calculations (as do other partition
>>>>> parsers that may have to deal with large disks) and fixes
>>>>> Martin's bug, so appears to be the right thing to do.
>>>>>
>>>>> Using 64 bit data types for disks smaller than 2 TB where
>>>>> calculations don't currently overflow is not expected to cause
>>>>> new issues, other than enabling use of disk and partitions larger
>>>>> than 2 TB (which may have ramifications with filesystems on these
>>>>> partitions). So comptibility is preserved.
>>>>>
>>>>> Forcing larger block sizes might be a good strategy to avoid
>>>>> overflow
>>>>> issues in filesystems as well, but I can't see how the block size
>>>>> stored in the RDB would enforce use of the same block size in
>>>>> filesystems. We'll have to rely on the filesystem tools to get
>>>>> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
>>>>> limitation) so this should allow partitions larger than 2 TB to
>>>>> work already (but I suspect Al Viro may have found a few issues
>>>>> when he looked at the AFFS code so I won't say more). Anyway
>>>>> partitioning tools and filesystems are unrelated to the Linux
>>>>> partition parser code which is all we aim to fix in this patch.
>>>>>
>>>>> If you feel strongly about unknown ramifications of any
>>>>> filesystems on partitions larger than 2 TB, say so and I'll have
>>>>> the kernel print a warning about these partitions.
>>>>>
>>>>> I'll get this patch tested on Martin's test case image as well as
>>>>> on a RDB image from a disk known to currently work under Linux
>>>>> (thanks Geert for the losetup hint). Can't do much more without
>>>>> procuring a working Amiga disk image to use with an emulator,
>>>>> sorry. The Amiga I plan to use for tests is a long way away from
>>>>> my home indeed.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Michael
>>>>>
>>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>>> Personally I'd make any partitioning tool front end gently force
>>>>>> the
>>>>>> block size towards 8k as the disk size gets larger. The file
>>>>>> systems
>>>>>> may also run into 2TB issues that are not obvious. An unused
>>>>>> blocks
>>>>>> list will have to go beyond a uint32_t size, for example. But a
>>>>>> block
>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>>>>>> under 1% of the disk all by itself. A block bitmap is not quite
>>>>>> so bad. {^_-}
>>>>>>
>>>>>> Just be sure you are aware of all the ramifications when you make
>>>>>> a
>>>>>> change. I remember thinking about this for awhile and then
>>>>>> determining I REALLY did not want to think about it as my brain
>>>>>> was getting tied into a gordian knot.
>>>>>>
>>>>>> {^_^}
>>>>>>
>>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>> Joanne,
>>>>>>>
>>>>>>> Martin's boot log (including your patch) says:
>>>>>>>
>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512)
>>>>>>> sdb1
>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
>>>>>>> 2 spb
>>>>>>> 4)
>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>>> Attached SCSI disk
>>>>>>>
>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
>>>>>>> 512
>>>>>>> byte blocks) and can be worked around by using a different block
>>>>>>> size.
>>>>>>>
>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
>>>>>>> units.
>>>>>>> I'll still submit a patch to Jens anyway as this may bite others
>>>>>>> yet.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Michael
>>>>>>>
>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]>
> wrote:
>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
>>>>>>>> system is
>>>>>>>> a famn
>>>>>>>> dool.
>>>>>>>>
>>>>>>>> If memory serves the RDBs think in blocks rather than bytes so
>>>>>>>> it
>>>>>>>> should
>>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks
>>>>>>>> is
>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk
>>>>>>>> in
>>>>>>>> block maps.
>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>>
>>>>>>>> {^_^}
>>>>>>>>
>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>> Hi.
>>>>>>>>>
>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>> test results at
>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given
>>>>>>>>>> there, so if
>>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>>
>>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>>
>>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>>> preparing the a patch from Joanne Dow´s fix.
>>>>>>>>>
>>>>>>>>> I am not even using my Amiga boxes anymore, not even the
>>>>>>>>> Sam440ep
>>>>>>>>> which
>>>>>>>>> I still have in my apartment.
>>>>>>>>>
>>>>>>>>> So RDB support in Linux it remains broken for disks larger 2
>>>>>>>>> TB,
>>>>>>>>> unless
>>>>>>>>> someone else does.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> linux-m68k" in
>>>>>>>> the body of a message to [email protected]
>>>>>>>> More majordomo info at
>>>>>>>> http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-m68k" in the body of a message to
>>>>>> [email protected]
>>>>>> More majordomo info at
>>>>>> http://vger.kernel.org/majordomo-info.html
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe
>>>> linux-m68k" in the body of a message to [email protected]
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>

2018-06-28 03:20:06

by jdow

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

FFS is limited to 2 GHz file size if you don't want any corruption using
fseek(). Otherwise it can go to 4 GHz sort of safely.
{^_^}

On 20180627 00:57, Martin Steigerwald wrote:
> Michael Schmitz - 27.06.18, 03:07:
>> Joanne,
>>
>> As far as I have been able to test, the change is backwards compatible
>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>> That"s only been done on an emulator though.
>>
>> Your advice about the dangers of using RDB disks that would have
>> failed the current Linux RDB parser on legacy 32 bit systems is well
>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>> in question ever used on a 32 bit Amiga system?
>
> Sure! Are there actually *any* 64 bit Amiga systems? I am not completely
> sure architecture-wise, but the operating system is still just 32-Bit.
>
> However, on AmigaOS officially since at least 3.5/3.9 there are various
> mechanisms to handle 64-bit block numbers for disk devices, called
> NSD64, TD64 and scsi direct¹. One of them, NSD 64 in AmigaOS 3.5/3.9, is
> part of the operating system. First via SetPatch based update to
> Kickstart ROM, but with AmigaOS 4.x its just included.
>
> [1] http://www.amigawiki.de/doku.php?id=de:system:filesystems_limits
>
> (I do not agree with maximum filesystem size limits stated there for
> Amiga Fast Filesystem (FFS), but well, for early FFS versions they could
> have been true, I am pretty sure that at least with higher block sizes
> you can have FFS > 2 GiB in size. And I remember having had JFXS and I
> think also SFS2 partitions in AmigaOS 4 with more than 100 GiB.)
>
>> RDB disk format is meant for legacy use only, so I think we can get
>> away with printing a big fat warning during boot, advising the user
>> that the oversize RDB partition(s) scanned are not compatible with
>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>> partition at disk end and trashing valid partitions that overlap,
>> which is what Martin was after.
>
> They are compatible. For AmigaOS 4.x I am completely sure, cause I
> installed and used the disk there. That was the original motivation for
> my Linux bug report back then: I used the disk in AmigaOS 4 *just fine*
> and it broke in Linux. I used Media Toolbox in order to create the RDB I
> posted back then.
>
> Its difficult to find proof for my claims online, but here is an
> official one:
>
> http://wiki.amigaos.net/wiki/RDB
>
> Limits of filesystem sizes may of course be different, but here we are
> talking about RDB.
>
> Also this confirms that RDB only uses 512 byte block size on AmigaOS 4.
>
> Hmmm, I even created the page back then. Interestingly at about the same
> time I made the bug report about the Linux issue. The calculations in
> there are not really from me. I bet I copied them over from what other
> AmigaOS developers wrote on development mailing lists, at that time.
> With their agreement. I think I started my research back then due by my
> own question on: Will such a large disk actually work in AmigaOS?
>
> It is all a long time ago…
>
>> If that still seems too risky, we can make the default behaviour to
>> bail out once a potential overflow is detected, and allow the user to
>> override that through a boot parameter. I'd leave that decision up for
>> the code review on linux-block.
>>
>> Two more comments: Linux uses 512 byte block sizes for the partition
>> start and size calculations, so a change of the RDB blocksize to
>> reduce the block counts stored in the RDB would still result in the
>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>> updating (along with probably most legacy m68k partitioners). Adrian
>> has advertised parted as replacement for the old tools - maybe this
>> would make a nice test case for parted?
>
> I always used AmigaOS based tools to partition in RDB format, cause I
> never trusted amiga-fdisk :).
>
>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <[email protected]> wrote:
>>> If it is not backwards compatible I for one would refuse to use it.
>>> And if it still mattered that much to me I'd also generate a
>>> reasonable alternative. Modifying RDBs nay not be even an
>>> approximation of a good idea. You'd discover that as soon as an RDB
>>> uint64_t disk is tasted by a uint32_t only system. If it is for
>>> your personal use then you're entirely free to reject my advice and
>>> are probably smart enough to keep it working for yourself.
>>>
>>> GPT is probably the right way to go. Preserve the ability to read
>>> RDBs for legacy disks only.
>>>
>>> {^_^}
>>>
>>> On 20180626 01:31, Michael Schmitz wrote:
>>>> Joanne,
>>>>
>>>> I think we all agree that doing 32 bit calculations on 512-byte
>>>> block
>>>> addresses that overflow on disks 2 TB and larger is a bug, causing
>>>> the issues Martin reported. Your patch addresses that by using the
>>>> correct data type for the calculations (as do other partition
>>>> parsers that may have to deal with large disks) and fixes Martin's
>>>> bug, so appears to be the right thing to do.
>>>>
>>>> Using 64 bit data types for disks smaller than 2 TB where
>>>> calculations don't currently overflow is not expected to cause new
>>>> issues, other than enabling use of disk and partitions larger than
>>>> 2 TB (which may have ramifications with filesystems on these
>>>> partitions). So comptibility is preserved.
>>>>
>>>> Forcing larger block sizes might be a good strategy to avoid
>>>> overflow
>>>> issues in filesystems as well, but I can't see how the block size
>>>> stored in the RDB would enforce use of the same block size in
>>>> filesystems. We'll have to rely on the filesystem tools to get
>>>> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
>>>> limitation) so this should allow partitions larger than 2 TB to
>>>> work already (but I suspect Al Viro may have found a few issues
>>>> when he looked at the AFFS code so I won't say more). Anyway
>>>> partitioning tools and filesystems are unrelated to the Linux
>>>> partition parser code which is all we aim to fix in this patch.
>>>>
>>>> If you feel strongly about unknown ramifications of any filesystems
>>>> on partitions larger than 2 TB, say so and I'll have the kernel
>>>> print a warning about these partitions.
>>>>
>>>> I'll get this patch tested on Martin's test case image as well as
>>>> on a RDB image from a disk known to currently work under Linux
>>>> (thanks Geert for the losetup hint). Can't do much more without
>>>> procuring a working Amiga disk image to use with an emulator,
>>>> sorry. The Amiga I plan to use for tests is a long way away from
>>>> my home indeed.
>>>>
>>>> Cheers,
>>>>
>>>> Michael
>>>>
>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>> Personally I'd make any partitioning tool front end gently force
>>>>> the
>>>>> block size towards 8k as the disk size gets larger. The file
>>>>> systems
>>>>> may also run into 2TB issues that are not obvious. An unused
>>>>> blocks
>>>>> list will have to go beyond a uint32_t size, for example. But a
>>>>> block
>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>>>>> under 1% of the disk all by itself. A block bitmap is not quite
>>>>> so bad. {^_-}
>>>>>
>>>>> Just be sure you are aware of all the ramifications when you make
>>>>> a
>>>>> change. I remember thinking about this for awhile and then
>>>>> determining I REALLY did not want to think about it as my brain
>>>>> was getting tied into a gordian knot.
>>>>>
>>>>> {^_^}
>>>>>
>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>> Joanne,
>>>>>>
>>>>>> Martin's boot log (including your patch) says:
>>>>>>
>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512)
>>>>>> sdb1
>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2
>>>>>> spb
>>>>>> 4)
>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>> Attached SCSI disk
>>>>>>
>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
>>>>>> 512
>>>>>> byte blocks) and can be worked around by using a different block
>>>>>> size.
>>>>>>
>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
>>>>>> units.
>>>>>> I'll still submit a patch to Jens anyway as this may bite others
>>>>>> yet.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Michael
>>>>>>
>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]>
> wrote:
>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system
>>>>>>> is
>>>>>>> a famn
>>>>>>> dool.
>>>>>>>
>>>>>>> If memory serves the RDBs think in blocks rather than bytes so
>>>>>>> it
>>>>>>> should
>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks
>>>>>>> is
>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>> block maps.
>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>
>>>>>>> {^_^}
>>>>>>>
>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>> Hi.
>>>>>>>>
>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>> test results at
>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>> so if
>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>
>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>
>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>> preparing the a patch from Joanne Dow´s fix.
>>>>>>>>
>>>>>>>> I am not even using my Amiga boxes anymore, not even the
>>>>>>>> Sam440ep
>>>>>>>> which
>>>>>>>> I still have in my apartment.
>>>>>>>>
>>>>>>>> So RDB support in Linux it remains broken for disks larger 2
>>>>>>>> TB,
>>>>>>>> unless
>>>>>>>> someone else does.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>> linux-m68k" in
>>>>>>> the body of a message to [email protected]
>>>>>>> More majordomo info at
>>>>>>> http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-m68k" in the body of a message to [email protected]
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-m68k" in the body of a message to [email protected]
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>

2018-06-28 03:51:12

by jdow

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Michael, as long as m68k only supports int fseek( int ) 4 GB * block size is
your HARD limit. Versions that support __int64 fseek64( __int64 ) can work with
larger disks. RDBs could include RDSK and a new set of other blocks that replace
the last two characters with "64" and use __int64 where needed in the various
values. That way a clever disk partitioner could give allow normal (32 bit) RDB
definitions where possible. Then at least SOME of the disk could be supported
AND a very clever filesystem that abstracts very large disks properly could give
access to the whole disk. (Read the RDBs first 32 bits. Then if a filesystem or
driveinit was loaded re-read the RDBs to see if new 64 bit partitions are revealed.

I could be wrong but I do not think RDBs could be safely modified any other way
to work. And, trust me as I bet this is still true, you will need a SERIOUSLY
good bomb shelter on the Moon if you change RDBs. Suppose Joe Amigoid uses it,
and then Joe Amigoid loads Amigados 2.4 because he wants to run a game that
crashes on anything newer. Then Joe got far enough something writes to the disk
and data is corrupted. Note further that Amigoids do NOT, repeat NOT, listen to
facts in such cases. Hell, some of them never listened to facts about an
incident at Jerry Pournelle's place when a 1.1 DPaint session with Kelly Freas
hung and we lost a delightful drawing. Jerry reported it. Amigoids screamed. I
tried to tell them I was there, it was my machine, and 1.1 was, indeed, crap.

{o.o}

On 20180627 02:00, Michael Schmitz wrote:
> Joanne,
>
> I'm not at all allergic to avoiding RDB at all cost for new disks. If AmigaOS
> 4.1 supports more recent partition formats, all the better. This is all about
> supporting use of legacy RDB disks on Linux (though 2 TB does stretch the
> definition of 'legacy' a little). My interest in this is to ensure we can
> continue to use RDB format disks on m68k Amiga computers which have no other way
> to boot Linux from disk.
>
> Not proposing to change the RDB format at all, either. Just trying to make sure
> we translate RDB info into Linux 512-byte block offset and size numbers
> correctly. The kernel won't modify the RDB at all (intentionally, that is - with
> the correct choice of partition sizes, Martin might well have wiped out his RDB
> with the current version of the parser).
>
> The choice of refusing to mount a disk (or mounting read-only) rests with the
> VFS drivers alone - AFFS in that case. Not touching any of that. At partition
> scan time, we only have the option of making the partition available (with a
> warning printed), or refusing to make it available to the kernel. Once it's made
> available, all bets are off.
>
> From what Martin writes, his test case RDB was valid and worked as expected on
> 32 bit AmigaOS (4.1). Apparently, that version has the necessary extensions to
> handle the large offsets resulting from 2 TB disks. Not sure what safeguards are
> in place when connecting such a disk to older versions of AmigaOS, but that is a
> different matter entirely.
>
> The overflows in partition offset and size are the only ones I can see in the
> partition parser - there is no other overflow I've identified. I just stated
> that in order to place a partition towards the end of a 2 TB disk, the offset
> calculation will overflow regardless of what combination of rdb->rdb_BlockBytes
> and sector addresses stored in the RDB (in units of 512 byte blocks) we use:
>
>         blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
>
>
>                 nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 -
>                             be32_to_cpu(pb->pb_Environment[9])) *
>                            be32_to_cpu(pb->pb_Environment[3]) *
>                            be32_to_cpu(pb->pb_Environment[5]) *
>                            blksize;
>                 if (!nr_sects)
>                         continue;
>                 start_sect = be32_to_cpu(pb->pb_Environment[9]) *
>                              be32_to_cpu(pb->pb_Environment[3]) *
>                              be32_to_cpu(pb->pb_Environment[5]) *
>                              blksize;
>
> But in the interest of avoiding any accidental use of a RDB partition where
> calculations currently overflow, I'll make the default behaviour to bail out
> (instead of using wrong offset or size as we currently do). Given the
> 'eat_my_RDB_disk=1' boot option, the user may proceed at their own risk (though
> I still can't see what harm should result from now translating a well formed
> v4.1 2 TB disk RDB correctly for the first time).
>
> Whether or not Linux correctly handles AFFS filesystems larger than 1 TB is a
> matter for VFS experts. Bailing out on nr_sects overflowing ought to prevent
> accidental use of AFFS filesystems on RDB disks which I suppose is the majority
> of use cases.
>
> Bugs in partitioning tools on Linux are entirely out of scope - the partitioning
> tools bypass the partition structure discovered by the kernel, and work straight
> on the raw device. No protecting against that.
>
> If you can point out a way to cause data loss with these precautions, for a disk
> 2 TB or larger that was partitioned and used on a recent version or AmigaOS
> supporting such large disks, I'd consider omitting the 'eat_my_RDB_disk' boot
> option, and just bail out as the only safe option.
>
> Cheers,
>
>     Michael
>
>
> Am 27.06.2018 um 18:24 schrieb jdow:
>> You allergic to using a GPT solution? It will get away from some of the
>> evils that RDB has inherent in it because they are also features?
>> (Loading a filesystem or DriveInit code from RDBs is just asking for a
>> nearly impossible to remove malware infection.) Furthermore, any 32 bit
>> system that sees an RDSK block is going to try to translate it. If you
>> add a new RDB format you are going to get bizarre and probably quite
>> destructive results from the mistake. Fail safe is a rather good notion,
>> methinks.
>>
>> Personally I figure this is all rather surreal. 2TG of junk on an Amiga
>> system seems utterly outlandish to me. You cited another overflow
>> potential. There are at least three we've identified, I believe. Are you
>> 100% sure there are no more? The specific one you mention of translating
>> RDB to Linux has a proper solution in the RDB reader. It should recover
>> such overflow errors in the RDB as it can with due care and polish. It
>> should flag any other overflow error it detects within the RDBs and
>> return an error such as to leave the disk unmounted or mounted read-only
>> if you feel like messing up a poor sod's backups. The simple solution is
>> to read each of the variables with the nominal RDB size and convert it
>> to uint64_t before calculating byte indices.
>>
>> However, consider my inputs as advice from an adult who has seen the
>> Amiga Elephant so to speak. I am not trying to assert any control. Do as
>> you wish; but, I would plead with you to avoid ANY chance you can for
>> the user to make a bonehead stupid move and lose all his treasured disk
>> archives. Doing otherwise is very poor form.
>>
>> {o.o}   Joanne "Said enough, she has" Dow
>>
>> On 20180626 18:07, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> As far as I have been able to test, the change is backwards compatible
>>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>>> That"s only been done on an emulator though.
>>>
>>> Your advice about the dangers of using RDB disks that would have
>>> failed the current Linux RDB parser on legacy 32 bit systems is well
>>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>>> in question ever used on a 32 bit Amiga system?
>>>
>>> RDB disk format is meant for legacy use only, so I think we can get
>>> away with printing a big fat warning during boot, advising the user
>>> that the oversize RDB partition(s) scanned are not compatible with
>>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>>> partition at disk end and trashing valid partitions that overlap,
>>> which is what Martin was after.
>>>
>>> If that still seems too risky, we can make the default behaviour to
>>> bail out once a potential overflow is detected, and allow the user to
>>> override that through a boot parameter. I'd leave that decision up for
>>> the code review on linux-block.
>>>
>>> Two more comments: Linux uses 512 byte block sizes for the partition
>>> start and size calculations, so a change of the RDB blocksize to
>>> reduce the block counts stored in the RDB would still result in the
>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>>> updating (along with probably most legacy m68k partitioners). Adrian
>>> has advertised parted as replacement for the old tools - maybe this
>>> would make a nice test case for parted?
>>>
>>> Cheers,
>>>
>>>    Michael
>>>
>>>
>>>
>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <[email protected]> wrote:
>>>> If it is not backwards compatible I for one would refuse to use it.
>>>> And if
>>>> it still mattered that much to me I'd also generate a reasonable
>>>> alternative. Modifying RDBs nay not be even an approximation of a
>>>> good idea.
>>>> You'd discover that as soon as an RDB uint64_t disk is tasted by a
>>>> uint32_t
>>>> only system. If it is for your personal use then you're entirely free to
>>>> reject my advice and are probably smart enough to keep it working for
>>>> yourself.
>>>>
>>>> GPT is probably the right way to go. Preserve the ability to read
>>>> RDBs for
>>>> legacy disks only.
>>>>
>>>> {^_^}
>>>>
>>>>
>>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>>
>>>>> Joanne,
>>>>>
>>>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>>>> addresses that overflow on disks 2 TB and larger is a bug, causing the
>>>>> issues Martin reported. Your patch addresses that by using the correct
>>>>> data type for the calculations (as do other partition parsers that may
>>>>> have to deal with large disks) and fixes Martin's bug, so appears to be
>>>>> the right thing to do.
>>>>>
>>>>> Using 64 bit data types for disks smaller than 2 TB where calculations
>>>>> don't currently overflow is not expected to cause new issues, other
>>>>> than
>>>>> enabling use of disk and partitions larger than 2 TB (which may have
>>>>> ramifications with filesystems on these partitions). So comptibility is
>>>>> preserved.
>>>>>
>>>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>>>> issues in filesystems as well, but I can't see how the block size
>>>>> stored
>>>>> in the RDB would enforce use of the same block size in filesystems.
>>>>> We'll have to rely on the filesystem tools to get that right, too.
>>>>> Linux
>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>>>> allow partitions larger than 2 TB to work already (but I suspect Al
>>>>> Viro
>>>>> may have found a few issues when he looked at the AFFS code so I won't
>>>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>>>> the Linux partition parser code which is all we aim to fix in this
>>>>> patch.
>>>>>
>>>>> If you feel strongly about unknown ramifications of any filesystems on
>>>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>>>> warning about these partitions.
>>>>>
>>>>> I'll get this patch tested on Martin's test case image as well as on a
>>>>> RDB image from a disk known to currently work under Linux (thanks Geert
>>>>> for the losetup hint). Can't do much more without procuring a working
>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to
>>>>> use
>>>>> for tests is a long way away from my home indeed.
>>>>>
>>>>> Cheers,
>>>>>
>>>>>       Michael
>>>>>
>>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>>
>>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>>> Personally I'd make any partitioning tool front end gently force the
>>>>>> block size towards 8k as the disk size gets larger. The file systems
>>>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>>>> list will have to go beyond a uint32_t size, for example. But a block
>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad under
>>>>>> 1% of the disk all by itself. A block bitmap is not quite so bad.
>>>>>> {^_-}
>>>>>>
>>>>>> Just be sure you are aware of all the ramifications when you make a
>>>>>> change. I remember thinking about this for awhile and then determining
>>>>>> I REALLY did not want to think about it as my brain was getting tied
>>>>>> into a gordian knot.
>>>>>>
>>>>>> {^_^}
>>>>>>
>>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>>
>>>>>>> Joanne,
>>>>>>>
>>>>>>> Martin's boot log (including your patch) says:
>>>>>>>
>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1
>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb
>>>>>>> 4)
>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>>> Attached SCSI disk
>>>>>>>
>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means 512
>>>>>>> byte blocks) and can be worked around by using a different block
>>>>>>> size.
>>>>>>>
>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>>>> I'll still submit a patch to Jens anyway as this may bite others yet.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>>      Michael
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]> wrote:
>>>>>>>>
>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file system is
>>>>>>>> a famn
>>>>>>>> dool.
>>>>>>>>
>>>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>>>> should
>>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>>> block maps.
>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>>
>>>>>>>> {^_^}
>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi.
>>>>>>>>>
>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> test results at https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>>> so if
>>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>>
>>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>>> preparing
>>>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>>>
>>>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>>>> which
>>>>>>>>> I still have in my apartment.
>>>>>>>>>
>>>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>>>> unless
>>>>>>>>> someone else does.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> linux-m68k" in
>>>>>>>> the body of a message to [email protected]
>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-m68k" in
>>>>>> the body of a message to [email protected]
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-m68k" in
>>>> the body of a message to [email protected]
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>

2018-06-28 05:49:17

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Joanne,

Linux on m68k has supported lseek64 (or llseek) for a long time (from
glibc version 2.1 according to what I found). About the only area where
we are limited by 32 bits is the virtual memory size.

I'm not proposing to modify the RDB format definition, though an
extension to store 64 bit offsets separate from the 32 bit ones would be
one way to make certain such disks are safe to use on 3.1 and earlier
versions of AmigaOS. (Another one would be to modify the disk drivers on
older versions to do the offset calculation in 64 bit, and check for
overflow just as we do here. Not sure whether that's feasible. And as
you so eloquently describe, we can't rely on users listening.)

Either way, we need the cooperation of partitioning tool writers to
ensure partition information that is prone to overflows is never stored
in the 32 bit, classic RDB. That appears to have failed already, as
Martin's experience illustrates.

I'm only concerned with fixing the (dangerous) but in the Linux
partition format parser for RDB. Refusing to use any partitions that
will cause havoc on old AmigaOS versions is all I can do to try and get
the users' attention.

Your warning makes me wonder whether the log message should just say
'report this bug to [email protected]' to at least try and
educate any that respond about the dangers of their partitioning scheme
before telling them about the override option. Problem with that is, in
about three years no one will remember any of this ...

Cheers,

Michael

Am 28.06.2018 um 15:44 schrieb jdow:
> Michael, as long as m68k only supports int fseek( int ) 4 GB * block
> size is your HARD limit. Versions that support __int64 fseek64( __int64
> ) can work with larger disks. RDBs could include RDSK and a new set of
> other blocks that replace the last two characters with "64" and use
> __int64 where needed in the various values. That way a clever disk
> partitioner could give allow normal (32 bit) RDB definitions where
> possible. Then at least SOME of the disk could be supported AND a very
> clever filesystem that abstracts very large disks properly could give
> access to the whole disk. (Read the RDBs first 32 bits. Then if a
> filesystem or driveinit was loaded re-read the RDBs to see if new 64 bit
> partitions are revealed.
>
> I could be wrong but I do not think RDBs could be safely modified any
> other way to work. And, trust me as I bet this is still true, you will
> need a SERIOUSLY good bomb shelter on the Moon if you change RDBs.
> Suppose Joe Amigoid uses it, and then Joe Amigoid loads Amigados 2.4
> because he wants to run a game that crashes on anything newer. Then Joe
> got far enough something writes to the disk and data is corrupted. Note
> further that Amigoids do NOT, repeat NOT, listen to facts in such cases.
> Hell, some of them never listened to facts about an incident at Jerry
> Pournelle's place when a 1.1 DPaint session with Kelly Freas hung and we
> lost a delightful drawing. Jerry reported it. Amigoids screamed. I tried
> to tell them I was there, it was my machine, and 1.1 was, indeed, crap.
>
> {o.o}
>
> On 20180627 02:00, Michael Schmitz wrote:
>> Joanne,
>>
>> I'm not at all allergic to avoiding RDB at all cost for new disks. If
>> AmigaOS 4.1 supports more recent partition formats, all the better.
>> This is all about supporting use of legacy RDB disks on Linux (though
>> 2 TB does stretch the definition of 'legacy' a little). My interest in
>> this is to ensure we can continue to use RDB format disks on m68k
>> Amiga computers which have no other way to boot Linux from disk.
>>
>> Not proposing to change the RDB format at all, either. Just trying to
>> make sure we translate RDB info into Linux 512-byte block offset and
>> size numbers correctly. The kernel won't modify the RDB at all
>> (intentionally, that is - with the correct choice of partition sizes,
>> Martin might well have wiped out his RDB with the current version of
>> the parser).
>>
>> The choice of refusing to mount a disk (or mounting read-only) rests
>> with the VFS drivers alone - AFFS in that case. Not touching any of
>> that. At partition scan time, we only have the option of making the
>> partition available (with a warning printed), or refusing to make it
>> available to the kernel. Once it's made available, all bets are off.
>>
>> From what Martin writes, his test case RDB was valid and worked as
>> expected on 32 bit AmigaOS (4.1). Apparently, that version has the
>> necessary extensions to handle the large offsets resulting from 2 TB
>> disks. Not sure what safeguards are in place when connecting such a
>> disk to older versions of AmigaOS, but that is a different matter
>> entirely.
>>
>> The overflows in partition offset and size are the only ones I can see
>> in the partition parser - there is no other overflow I've identified.
>> I just stated that in order to place a partition towards the end of a
>> 2 TB disk, the offset calculation will overflow regardless of what
>> combination of rdb->rdb_BlockBytes and sector addresses stored in the
>> RDB (in units of 512 byte blocks) we use:
>>
>> blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
>>
>>
>> nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 -
>> be32_to_cpu(pb->pb_Environment[9])) *
>> be32_to_cpu(pb->pb_Environment[3]) *
>> be32_to_cpu(pb->pb_Environment[5]) *
>> blksize;
>> if (!nr_sects)
>> continue;
>> start_sect = be32_to_cpu(pb->pb_Environment[9]) *
>> be32_to_cpu(pb->pb_Environment[3]) *
>> be32_to_cpu(pb->pb_Environment[5]) *
>> blksize;
>>
>> But in the interest of avoiding any accidental use of a RDB partition
>> where calculations currently overflow, I'll make the default behaviour
>> to bail out (instead of using wrong offset or size as we currently
>> do). Given the 'eat_my_RDB_disk=1' boot option, the user may proceed
>> at their own risk (though I still can't see what harm should result
>> from now translating a well formed v4.1 2 TB disk RDB correctly for
>> the first time).
>>
>> Whether or not Linux correctly handles AFFS filesystems larger than 1
>> TB is a matter for VFS experts. Bailing out on nr_sects overflowing
>> ought to prevent accidental use of AFFS filesystems on RDB disks which
>> I suppose is the majority of use cases.
>>
>> Bugs in partitioning tools on Linux are entirely out of scope - the
>> partitioning tools bypass the partition structure discovered by the
>> kernel, and work straight on the raw device. No protecting against that.
>>
>> If you can point out a way to cause data loss with these precautions,
>> for a disk 2 TB or larger that was partitioned and used on a recent
>> version or AmigaOS supporting such large disks, I'd consider omitting
>> the 'eat_my_RDB_disk' boot option, and just bail out as the only safe
>> option.
>>
>> Cheers,
>>
>> Michael
>>
>>
>> Am 27.06.2018 um 18:24 schrieb jdow:
>>> You allergic to using a GPT solution? It will get away from some of the
>>> evils that RDB has inherent in it because they are also features?
>>> (Loading a filesystem or DriveInit code from RDBs is just asking for a
>>> nearly impossible to remove malware infection.) Furthermore, any 32 bit
>>> system that sees an RDSK block is going to try to translate it. If you
>>> add a new RDB format you are going to get bizarre and probably quite
>>> destructive results from the mistake. Fail safe is a rather good notion,
>>> methinks.
>>>
>>> Personally I figure this is all rather surreal. 2TG of junk on an Amiga
>>> system seems utterly outlandish to me. You cited another overflow
>>> potential. There are at least three we've identified, I believe. Are you
>>> 100% sure there are no more? The specific one you mention of translating
>>> RDB to Linux has a proper solution in the RDB reader. It should recover
>>> such overflow errors in the RDB as it can with due care and polish. It
>>> should flag any other overflow error it detects within the RDBs and
>>> return an error such as to leave the disk unmounted or mounted read-only
>>> if you feel like messing up a poor sod's backups. The simple solution is
>>> to read each of the variables with the nominal RDB size and convert it
>>> to uint64_t before calculating byte indices.
>>>
>>> However, consider my inputs as advice from an adult who has seen the
>>> Amiga Elephant so to speak. I am not trying to assert any control. Do as
>>> you wish; but, I would plead with you to avoid ANY chance you can for
>>> the user to make a bonehead stupid move and lose all his treasured disk
>>> archives. Doing otherwise is very poor form.
>>>
>>> {o.o} Joanne "Said enough, she has" Dow
>>>
>>> On 20180626 18:07, Michael Schmitz wrote:
>>>> Joanne,
>>>>
>>>> As far as I have been able to test, the change is backwards compatible
>>>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>>>> That"s only been done on an emulator though.
>>>>
>>>> Your advice about the dangers of using RDB disks that would have
>>>> failed the current Linux RDB parser on legacy 32 bit systems is well
>>>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>>>> in question ever used on a 32 bit Amiga system?
>>>>
>>>> RDB disk format is meant for legacy use only, so I think we can get
>>>> away with printing a big fat warning during boot, advising the user
>>>> that the oversize RDB partition(s) scanned are not compatible with
>>>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>>>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>>>> partition at disk end and trashing valid partitions that overlap,
>>>> which is what Martin was after.
>>>>
>>>> If that still seems too risky, we can make the default behaviour to
>>>> bail out once a potential overflow is detected, and allow the user to
>>>> override that through a boot parameter. I'd leave that decision up for
>>>> the code review on linux-block.
>>>>
>>>> Two more comments: Linux uses 512 byte block sizes for the partition
>>>> start and size calculations, so a change of the RDB blocksize to
>>>> reduce the block counts stored in the RDB would still result in the
>>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>>>> updating (along with probably most legacy m68k partitioners). Adrian
>>>> has advertised parted as replacement for the old tools - maybe this
>>>> would make a nice test case for parted?
>>>>
>>>> Cheers,
>>>>
>>>> Michael
>>>>
>>>>
>>>>
>>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <[email protected]> wrote:
>>>>> If it is not backwards compatible I for one would refuse to use it.
>>>>> And if
>>>>> it still mattered that much to me I'd also generate a reasonable
>>>>> alternative. Modifying RDBs nay not be even an approximation of a
>>>>> good idea.
>>>>> You'd discover that as soon as an RDB uint64_t disk is tasted by a
>>>>> uint32_t
>>>>> only system. If it is for your personal use then you're entirely
>>>>> free to
>>>>> reject my advice and are probably smart enough to keep it working for
>>>>> yourself.
>>>>>
>>>>> GPT is probably the right way to go. Preserve the ability to read
>>>>> RDBs for
>>>>> legacy disks only.
>>>>>
>>>>> {^_^}
>>>>>
>>>>>
>>>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>>>
>>>>>> Joanne,
>>>>>>
>>>>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>>>>> addresses that overflow on disks 2 TB and larger is a bug, causing
>>>>>> the
>>>>>> issues Martin reported. Your patch addresses that by using the
>>>>>> correct
>>>>>> data type for the calculations (as do other partition parsers that
>>>>>> may
>>>>>> have to deal with large disks) and fixes Martin's bug, so appears
>>>>>> to be
>>>>>> the right thing to do.
>>>>>>
>>>>>> Using 64 bit data types for disks smaller than 2 TB where
>>>>>> calculations
>>>>>> don't currently overflow is not expected to cause new issues, other
>>>>>> than
>>>>>> enabling use of disk and partitions larger than 2 TB (which may have
>>>>>> ramifications with filesystems on these partitions). So
>>>>>> comptibility is
>>>>>> preserved.
>>>>>>
>>>>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>>>>> issues in filesystems as well, but I can't see how the block size
>>>>>> stored
>>>>>> in the RDB would enforce use of the same block size in filesystems.
>>>>>> We'll have to rely on the filesystem tools to get that right, too.
>>>>>> Linux
>>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>>>>> allow partitions larger than 2 TB to work already (but I suspect Al
>>>>>> Viro
>>>>>> may have found a few issues when he looked at the AFFS code so I
>>>>>> won't
>>>>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>>>>> the Linux partition parser code which is all we aim to fix in this
>>>>>> patch.
>>>>>>
>>>>>> If you feel strongly about unknown ramifications of any
>>>>>> filesystems on
>>>>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>>>>> warning about these partitions.
>>>>>>
>>>>>> I'll get this patch tested on Martin's test case image as well as
>>>>>> on a
>>>>>> RDB image from a disk known to currently work under Linux (thanks
>>>>>> Geert
>>>>>> for the losetup hint). Can't do much more without procuring a working
>>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to
>>>>>> use
>>>>>> for tests is a long way away from my home indeed.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Michael
>>>>>>
>>>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>>>
>>>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>>>> Personally I'd make any partitioning tool front end gently force the
>>>>>>> block size towards 8k as the disk size gets larger. The file systems
>>>>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>>>>> list will have to go beyond a uint32_t size, for example. But a
>>>>>>> block
>>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>>>>>>> under
>>>>>>> 1% of the disk all by itself. A block bitmap is not quite so bad.
>>>>>>> {^_-}
>>>>>>>
>>>>>>> Just be sure you are aware of all the ramifications when you make a
>>>>>>> change. I remember thinking about this for awhile and then
>>>>>>> determining
>>>>>>> I REALLY did not want to think about it as my brain was getting tied
>>>>>>> into a gordian knot.
>>>>>>>
>>>>>>> {^_^}
>>>>>>>
>>>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>>>
>>>>>>>> Joanne,
>>>>>>>>
>>>>>>>> Martin's boot log (including your patch) says:
>>>>>>>>
>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512)
>>>>>>>> sdb1
>>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
>>>>>>>> 2 spb
>>>>>>>> 4)
>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>>>> Attached SCSI disk
>>>>>>>>
>>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
>>>>>>>> 512
>>>>>>>> byte blocks) and can be worked around by using a different block
>>>>>>>> size.
>>>>>>>>
>>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>>>>> I'll still submit a patch to Jens anyway as this may bite others
>>>>>>>> yet.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Michael
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]> wrote:
>>>>>>>>>
>>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
>>>>>>>>> system is
>>>>>>>>> a famn
>>>>>>>>> dool.
>>>>>>>>>
>>>>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>>>>> should
>>>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>>>> block maps.
>>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>>>
>>>>>>>>> {^_^}
>>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi.
>>>>>>>>>>
>>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> test results at
>>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>>>> so if
>>>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>>>
>>>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>>>> preparing
>>>>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>>>>
>>>>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>>>>> which
>>>>>>>>>> I still have in my apartment.
>>>>>>>>>>
>>>>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>>>>> unless
>>>>>>>>>> someone else does.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>> linux-m68k" in
>>>>>>>>> the body of a message to [email protected]
>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>> linux-m68k" in
>>>>>>> the body of a message to [email protected]
>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-m68k" in
>>>>> the body of a message to [email protected]
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>

2018-06-28 07:11:10

by jdow

[permalink] [raw]

Subject: Re: moving affs + RDB partition support to staging?

Anything done to RDBs for Linux must remain 100.000% compatible with existing
Amiga equipment. Otherwise, what's the point of bothering to use RDBs?

Note, as an experiment I had a CrossDOS partition on a hard disk briefly. So
some amazing nonsense is possible. I concluded that for me it was not something
I wanted to do. Alternative techniques worked well enough.

That brings to the fore an interesting question. Why bother with RDBs over 2TB
unless you want a disk with one single partition? This Win10 monster I am using
has a modest BIOS driver partition for the OS and a giant data partition. That
smaller partition would easily work with any RDB/Filesystem combination since
2.0. So there are some good workarounds that are probably "safer" and at least
as flexible as RDBs, one Linux has used for a very long time, too.

With that it is easy in concept to put together a dual boot system with all the
disks and partitions readable if you play the proper games at the filesystem
level and keep individual partitions small. The filesystem would have to take on
some of the attributes of a device driver, though. A 281 TB disk made up of 2 TB
partitions should be possible to create if you think it through. (Note that with
RDBs the CHS disk description is purely an abstraction that has no relationship
to anything inside the disk. The same tricks could exist at the filesystem level.)

As for existing tools screwing up, do you think that is an excuse to perpetuate
the failure?

If we are speaking Linux for m68k, what does it do when confronted with which is
described with a size larger than 32 bits? Before it had everything inside it
that refers to positions within a file by byte index with an __int64 type of
storage did it throw up it's hands, corrupt the disk, or what? If the m68k Linux
is a failure perhaps it doesn't matter what you do for Linuxoids. (I'll stick
with the x64 version in that case. And I never go back to earlier versions for
games since I consider a good compiler is the best game around. It's better than
a blank piece of paper for possibilities.)

As I have said, for the RDB parser fix the famndool thing. Do fix it right in
such a manner that if somebody compiles it against a version with no 64 bit
device code it will throw a proper overflow error and protect the user. Users
are dumb. We like to think of ourselves as smart. Let's try to be smart about
this where we can so fingers can't point back at us rather than the fool that
made some other error.

And do remember, I am merely (and vociferously) advising rather than dictating.
You don't need my approval to proceed. I may want my name noted as an early
contributor only. Meanwhile I spit out ideas as they come to me. One or more of
them might be good. And offering alternatives is better than simply saying "No"
most of the time.

If people are using RDBs for TB level disks I doubt they can remember which is
the left shoe when they are getting dressed in the morning before going out in
the yard to beat some dead horses. Or else maybe they just want to see how far
they can flog the m68k architecture as a mental challenge. In that case, taking
it too seriously could hurt. Note that I am mostly ignoring m68k Linux. People
using that are hard core. People using x86/x64 Linux aren't such hard core
folks. And I bet most of them want to read the disks so they can copy stuff to
Amiga Forever or WinUAE running on other architectures. So TB is not likely to
be an issue for them, either.

{^_^}

On 20180627 22:43, Michael Schmitz wrote:
> Joanne,
>
> Linux on m68k has supported lseek64 (or llseek) for a long time (from glibc
> version 2.1 according to what I found). About the only area where we are limited
> by 32 bits is the virtual memory size.
>
> I'm not proposing to modify the RDB format definition, though an extension to
> store 64 bit offsets separate from the 32 bit ones would be one way to make
> certain such disks are safe to use on 3.1 and earlier versions of AmigaOS.
> (Another one would be to modify the disk drivers on older versions to do the
> offset calculation in 64 bit, and check for overflow just as we do here. Not
> sure whether that's feasible. And as you so eloquently describe, we can't rely
> on users listening.)
>
> Either way, we need the cooperation of partitioning tool writers to ensure
> partition information that is prone to overflows is never stored in the 32 bit,
> classic RDB. That appears to have failed already, as Martin's experience
> illustrates.
>
> I'm only concerned with fixing the (dangerous) but in the Linux partition format
> parser for RDB. Refusing to use any partitions that will cause havoc on old
> AmigaOS versions is all I can do to try and get the users' attention.
>
> Your warning makes me wonder whether the log message should just say 'report
> this bug to [email protected]' to at least try and educate any that
> respond about the dangers of their partitioning scheme before telling them about
> the override option. Problem with that is, in about three years no one will
> remember any of this ...
>
> Cheers,
>
>     Michael
>
>
> Am 28.06.2018 um 15:44 schrieb jdow:
>> Michael, as long as m68k only supports int fseek( int ) 4 GB * block
>> size is your HARD limit. Versions that support __int64 fseek64( __int64
>> ) can work with larger disks. RDBs could include RDSK and a new set of
>> other blocks that replace the last two characters with "64" and use
>> __int64 where needed in the various values. That way a clever disk
>> partitioner could give allow normal (32 bit) RDB definitions where
>> possible. Then at least SOME of the disk could be supported AND a very
>> clever filesystem that abstracts very large disks properly could give
>> access to the whole disk. (Read the RDBs first 32 bits. Then if a
>> filesystem or driveinit was loaded re-read the RDBs to see if new 64 bit
>> partitions are revealed.
>>
>> I could be wrong but I do not think RDBs could be safely modified any
>> other way to work. And, trust me as I bet this is still true, you will
>> need a SERIOUSLY good bomb shelter on the Moon if you change RDBs.
>> Suppose Joe Amigoid uses it, and then Joe Amigoid loads Amigados 2.4
>> because he wants to run a game that crashes on anything newer. Then Joe
>> got far enough something writes to the disk and data is corrupted. Note
>> further that Amigoids do NOT, repeat NOT, listen to facts in such cases.
>> Hell, some of them never listened to facts about an incident at Jerry
>> Pournelle's place when a 1.1 DPaint session with Kelly Freas hung and we
>> lost a delightful drawing. Jerry reported it. Amigoids screamed. I tried
>> to tell them I was there, it was my machine, and 1.1 was, indeed, crap.
>>
>> {o.o}
>>
>> On 20180627 02:00, Michael Schmitz wrote:
>>> Joanne,
>>>
>>> I'm not at all allergic to avoiding RDB at all cost for new disks. If
>>> AmigaOS 4.1 supports more recent partition formats, all the better.
>>> This is all about supporting use of legacy RDB disks on Linux (though
>>> 2 TB does stretch the definition of 'legacy' a little). My interest in
>>> this is to ensure we can continue to use RDB format disks on m68k
>>> Amiga computers which have no other way to boot Linux from disk.
>>>
>>> Not proposing to change the RDB format at all, either. Just trying to
>>> make sure we translate RDB info into Linux 512-byte block offset and
>>> size numbers correctly. The kernel won't modify the RDB at all
>>> (intentionally, that is - with the correct choice of partition sizes,
>>> Martin might well have wiped out his RDB with the current version of
>>> the parser).
>>>
>>> The choice of refusing to mount a disk (or mounting read-only) rests
>>> with the VFS drivers alone - AFFS in that case. Not touching any of
>>> that. At partition scan time, we only have the option of making the
>>> partition available (with a warning printed), or refusing to make it
>>> available to the kernel. Once it's made available, all bets are off.
>>>
>>> From what Martin writes, his test case RDB was valid and worked as
>>> expected on 32 bit AmigaOS (4.1). Apparently, that version has the
>>> necessary extensions to handle the large offsets resulting from 2 TB
>>> disks. Not sure what safeguards are in place when connecting such a
>>> disk to older versions of AmigaOS, but that is a different matter
>>> entirely.
>>>
>>> The overflows in partition offset and size are the only ones I can see
>>> in the partition parser - there is no other overflow I've identified.
>>> I just stated that in order to place a partition towards the end of a
>>> 2 TB disk, the offset calculation will overflow regardless of what
>>> combination of rdb->rdb_BlockBytes and sector addresses stored in the
>>> RDB (in units of 512 byte blocks) we use:
>>>
>>>          blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
>>>
>>>
>>>                  nr_sects = (be32_to_cpu(pb->pb_Environment[10]) + 1 -
>>>                              be32_to_cpu(pb->pb_Environment[9])) *
>>>                             be32_to_cpu(pb->pb_Environment[3]) *
>>>                             be32_to_cpu(pb->pb_Environment[5]) *
>>>                             blksize;
>>>                  if (!nr_sects)
>>>                          continue;
>>>                  start_sect = be32_to_cpu(pb->pb_Environment[9]) *
>>>                               be32_to_cpu(pb->pb_Environment[3]) *
>>>                               be32_to_cpu(pb->pb_Environment[5]) *
>>>                               blksize;
>>>
>>> But in the interest of avoiding any accidental use of a RDB partition
>>> where calculations currently overflow, I'll make the default behaviour
>>> to bail out (instead of using wrong offset or size as we currently
>>> do). Given the 'eat_my_RDB_disk=1' boot option, the user may proceed
>>> at their own risk (though I still can't see what harm should result
>>> from now translating a well formed v4.1 2 TB disk RDB correctly for
>>> the first time).
>>>
>>> Whether or not Linux correctly handles AFFS filesystems larger than 1
>>> TB is a matter for VFS experts. Bailing out on nr_sects overflowing
>>> ought to prevent accidental use of AFFS filesystems on RDB disks which
>>> I suppose is the majority of use cases.
>>>
>>> Bugs in partitioning tools on Linux are entirely out of scope - the
>>> partitioning tools bypass the partition structure discovered by the
>>> kernel, and work straight on the raw device. No protecting against that.
>>>
>>> If you can point out a way to cause data loss with these precautions,
>>> for a disk 2 TB or larger that was partitioned and used on a recent
>>> version or AmigaOS supporting such large disks, I'd consider omitting
>>> the 'eat_my_RDB_disk' boot option, and just bail out as the only safe
>>> option.
>>>
>>> Cheers,
>>>
>>>      Michael
>>>
>>>
>>> Am 27.06.2018 um 18:24 schrieb jdow:
>>>> You allergic to using a GPT solution? It will get away from some of the
>>>> evils that RDB has inherent in it because they are also features?
>>>> (Loading a filesystem or DriveInit code from RDBs is just asking for a
>>>> nearly impossible to remove malware infection.) Furthermore, any 32 bit
>>>> system that sees an RDSK block is going to try to translate it. If you
>>>> add a new RDB format you are going to get bizarre and probably quite
>>>> destructive results from the mistake. Fail safe is a rather good notion,
>>>> methinks.
>>>>
>>>> Personally I figure this is all rather surreal. 2TG of junk on an Amiga
>>>> system seems utterly outlandish to me. You cited another overflow
>>>> potential. There are at least three we've identified, I believe. Are you
>>>> 100% sure there are no more? The specific one you mention of translating
>>>> RDB to Linux has a proper solution in the RDB reader. It should recover
>>>> such overflow errors in the RDB as it can with due care and polish. It
>>>> should flag any other overflow error it detects within the RDBs and
>>>> return an error such as to leave the disk unmounted or mounted read-only
>>>> if you feel like messing up a poor sod's backups. The simple solution is
>>>> to read each of the variables with the nominal RDB size and convert it
>>>> to uint64_t before calculating byte indices.
>>>>
>>>> However, consider my inputs as advice from an adult who has seen the
>>>> Amiga Elephant so to speak. I am not trying to assert any control. Do as
>>>> you wish; but, I would plead with you to avoid ANY chance you can for
>>>> the user to make a bonehead stupid move and lose all his treasured disk
>>>> archives. Doing otherwise is very poor form.
>>>>
>>>> {o.o}   Joanne "Said enough, she has" Dow
>>>>
>>>> On 20180626 18:07, Michael Schmitz wrote:
>>>>> Joanne,
>>>>>
>>>>> As far as I have been able to test, the change is backwards compatible
>>>>> (RDB partitions from an old disk 80 GB disk are still recognized OK).
>>>>> That"s only been done on an emulator though.
>>>>>
>>>>> Your advice about the dangers of using RDB disks that would have
>>>>> failed the current Linux RDB parser on legacy 32 bit systems is well
>>>>> taken though. Maybe Martin can clarify that for me - was the 2 TB disk
>>>>> in question ever used on a 32 bit Amiga system?
>>>>>
>>>>> RDB disk format is meant for legacy use only, so I think we can get
>>>>> away with printing a big fat warning during boot, advising the user
>>>>> that the oversize RDB partition(s) scanned are not compatible with
>>>>> legacy 32 bit AmigaOS. With the proposed fix they will work under both
>>>>> AmigaOS 4.1 and Linux instead of truncating the first overflowing
>>>>> partition at disk end and trashing valid partitions that overlap,
>>>>> which is what Martin was after.
>>>>>
>>>>> If that still seems too risky, we can make the default behaviour to
>>>>> bail out once a potential overflow is detected, and allow the user to
>>>>> override that through a boot parameter. I'd leave that decision up for
>>>>> the code review on linux-block.
>>>>>
>>>>> Two more comments: Linux uses 512 byte block sizes for the partition
>>>>> start and size calculations, so a change of the RDB blocksize to
>>>>> reduce the block counts stored in the RDB would still result in the
>>>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
>>>>> updating (along with probably most legacy m68k partitioners). Adrian
>>>>> has advertised parted as replacement for the old tools - maybe this
>>>>> would make a nice test case for parted?
>>>>>
>>>>> Cheers,
>>>>>
>>>>>    Michael
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <[email protected]> wrote:
>>>>>> If it is not backwards compatible I for one would refuse to use it.
>>>>>> And if
>>>>>> it still mattered that much to me I'd also generate a reasonable
>>>>>> alternative. Modifying RDBs nay not be even an approximation of a
>>>>>> good idea.
>>>>>> You'd discover that as soon as an RDB uint64_t disk is tasted by a
>>>>>> uint32_t
>>>>>> only system. If it is for your personal use then you're entirely
>>>>>> free to
>>>>>> reject my advice and are probably smart enough to keep it working for
>>>>>> yourself.
>>>>>>
>>>>>> GPT is probably the right way to go. Preserve the ability to read
>>>>>> RDBs for
>>>>>> legacy disks only.
>>>>>>
>>>>>> {^_^}
>>>>>>
>>>>>>
>>>>>> On 20180626 01:31, Michael Schmitz wrote:
>>>>>>>
>>>>>>> Joanne,
>>>>>>>
>>>>>>> I think we all agree that doing 32 bit calculations on 512-byte block
>>>>>>> addresses that overflow on disks 2 TB and larger is a bug, causing
>>>>>>> the
>>>>>>> issues Martin reported. Your patch addresses that by using the
>>>>>>> correct
>>>>>>> data type for the calculations (as do other partition parsers that
>>>>>>> may
>>>>>>> have to deal with large disks) and fixes Martin's bug, so appears
>>>>>>> to be
>>>>>>> the right thing to do.
>>>>>>>
>>>>>>> Using 64 bit data types for disks smaller than 2 TB where
>>>>>>> calculations
>>>>>>> don't currently overflow is not expected to cause new issues, other
>>>>>>> than
>>>>>>> enabling use of disk and partitions larger than 2 TB (which may have
>>>>>>> ramifications with filesystems on these partitions). So
>>>>>>> comptibility is
>>>>>>> preserved.
>>>>>>>
>>>>>>> Forcing larger block sizes might be a good strategy to avoid overflow
>>>>>>> issues in filesystems as well, but I can't see how the block size
>>>>>>> stored
>>>>>>> in the RDB would enforce use of the same block size in filesystems.
>>>>>>> We'll have to rely on the filesystem tools to get that right, too.
>>>>>>> Linux
>>>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this should
>>>>>>> allow partitions larger than 2 TB to work already (but I suspect Al
>>>>>>> Viro
>>>>>>> may have found a few issues when he looked at the AFFS code so I
>>>>>>> won't
>>>>>>> say more). Anyway partitioning tools and filesystems are unrelated to
>>>>>>> the Linux partition parser code which is all we aim to fix in this
>>>>>>> patch.
>>>>>>>
>>>>>>> If you feel strongly about unknown ramifications of any
>>>>>>> filesystems on
>>>>>>> partitions larger than 2 TB, say so and I'll have the kernel print a
>>>>>>> warning about these partitions.
>>>>>>>
>>>>>>> I'll get this patch tested on Martin's test case image as well as
>>>>>>> on a
>>>>>>> RDB image from a disk known to currently work under Linux (thanks
>>>>>>> Geert
>>>>>>> for the losetup hint). Can't do much more without procuring a working
>>>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I plan to
>>>>>>> use
>>>>>>> for tests is a long way away from my home indeed.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>>       Michael
>>>>>>>
>>>>>>> Am 26.06.18 um 17:17 schrieb jdow:
>>>>>>>>
>>>>>>>> As long as it preserves compatibility it should be OK, I suppose.
>>>>>>>> Personally I'd make any partitioning tool front end gently force the
>>>>>>>> block size towards 8k as the disk size gets larger. The file systems
>>>>>>>> may also run into 2TB issues that are not obvious. An unused blocks
>>>>>>>> list will have to go beyond a uint32_t size, for example. But a
>>>>>>>> block
>>>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a tad
>>>>>>>> under
>>>>>>>> 1% of the disk all by itself. A block bitmap is not quite so bad.
>>>>>>>> {^_-}
>>>>>>>>
>>>>>>>> Just be sure you are aware of all the ramifications when you make a
>>>>>>>> change. I remember thinking about this for awhile and then
>>>>>>>> determining
>>>>>>>> I REALLY did not want to think about it as my brain was getting tied
>>>>>>>> into a gordian knot.
>>>>>>>>
>>>>>>>> {^_^}
>>>>>>>>
>>>>>>>> On 20180625 19:23, Michael Schmitz wrote:
>>>>>>>>>
>>>>>>>>> Joanne,
>>>>>>>>>
>>>>>>>>> Martin's boot log (including your patch) says:
>>>>>>>>>
>>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512)
>>>>>>>>> sdb1
>>>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res
>>>>>>>>> 2 spb
>>>>>>>>> 4)
>>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb]
>>>>>>>>> Attached SCSI disk
>>>>>>>>>
>>>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512) means
>>>>>>>>> 512
>>>>>>>>> byte blocks) and can be worked around by using a different block
>>>>>>>>> size.
>>>>>>>>>
>>>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes units.
>>>>>>>>> I'll still submit a patch to Jens anyway as this may bite others
>>>>>>>>> yet.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>>      Michael
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]> wrote:
>>>>>>>>>>
>>>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
>>>>>>>>>> system is
>>>>>>>>>> a famn
>>>>>>>>>> dool.
>>>>>>>>>>
>>>>>>>>>> If memory serves the RDBs think in blocks rather than bytes so it
>>>>>>>>>> should
>>>>>>>>>> work up to 2 gigablocks whatever your block size is. 512 blocks is
>>>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk in
>>>>>>>>>> block maps.
>>>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
>>>>>>>>>>
>>>>>>>>>> {^_^}
>>>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi.
>>>>>>>>>>>
>>>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> test results at
>>>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given there,
>>>>>>>>>>>> so if
>>>>>>>>>>>> Martin now submits the patch, all should be well?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ok, better be honest than having anyone waiting for it:
>>>>>>>>>>>
>>>>>>>>>>> I do not care enough about this, in order to motivate myself
>>>>>>>>>>> preparing
>>>>>>>>>>> the a patch from Joanne Dow´s fix.
>>>>>>>>>>>
>>>>>>>>>>> I am not even using my Amiga boxes anymore, not even the Sam440ep
>>>>>>>>>>> which
>>>>>>>>>>> I still have in my apartment.
>>>>>>>>>>>
>>>>>>>>>>> So RDB support in Linux it remains broken for disks larger 2 TB,
>>>>>>>>>>> unless
>>>>>>>>>>> someone else does.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>> linux-m68k" in
>>>>>>>>>> the body of a message to [email protected]
>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> linux-m68k" in
>>>>>>>> the body of a message to [email protected]
>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-m68k" in
>>>>>> the body of a message to [email protected]
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>
>>>
>

2018-06-28 08:23:27

[permalink] [raw]

Subject: Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?)

Dear Joanne.

jdow - 28.06.18, 08:39:
> Anything done to RDBs for Linux must remain 100.000% compatible with
> existing Amiga equipment. Otherwise, what's the point of bothering to
> use RDBs?

Done to, in the sense of written to: Yes. I completely agree. But that
is for amiga-fdisk and parted. And for partitioning tools on native OS.

[…]
> That brings to the fore an interesting question. Why bother with RDBs
> over 2TB unless you want a disk with one single partition? This Win10
> monster I am using has a modest BIOS driver partition for the OS and
> a giant data partition. That smaller partition would easily work with
> any RDB/Filesystem combination since 2.0. So there are some good
> workarounds that are probably "safer" and at least as flexible as
> RDBs, one Linux has used for a very long time, too.

Well, my use case was simple:

I had this 2 TB disk and I choose to share it as a backup disk for Linux
*and* AmigaOS 4.x on that Sam440ep I still have next to me desk here.

As AmigaOS up to my knowledge does not have GPT support and MBR with 2
TB disk is asking for even more trouble and is not supported in any
sensible partitioning tool on AmigaOS and I choose to use Media Toolbox,
I went with RDB as I thought it is nicely supported in Linux, which it
is, apart from the overflowing issue.

So I did it this way.

> As I have said, for the RDB parser fix the famndool thing. Do fix it
> right in such a manner that if somebody compiles it against a version
> with no 64 bit device code it will throw a proper overflow error and
> protect the user. Users are dumb. We like to think of ourselves as

Sure, if the Linux kernel can´t handle it due to missing CONFIG_LBDAF or
so… by all means *bail* out. Without even a kernel option to parse this
anyway. No overflowing. Period. That is what I wrote from the beginning.

> smart. Let's try to be smart about this where we can so fingers can't
> point back at us rather than the fool that made some other error.

I completely agree.

> And do remember, I am merely (and vociferously) advising rather than
> dictating. You don't need my approval to proceed. I may want my name
> noted as an early contributor only. Meanwhile I spit out ideas as
> they come to me. One or more of them might be good. And offering
> alternatives is better than simply saying "No" most of the time.
>
> If people are using RDBs for TB level disks I doubt they can remember
> which is the left shoe when they are getting dressed in the morning
> before going out in the yard to beat some dead horses. Or else maybe

Heh, I remembered my shoes back then. And I informed myself about what I
was doing.

> they just want to see how far they can flog the m68k architecture as
> a mental challenge. In that case, taking it too seriously could hurt.

And well, yes… I wanted to see how far I can get away with it :)

> Note that I am mostly ignoring m68k Linux. People using that are hard
> core. People using x86/x64 Linux aren't such hard core folks. And I
> bet most of them want to read the disks so they can copy stuff to
> Amiga Forever or WinUAE running on other architectures. So TB is not
> likely to be an issue for them, either.

Yes.

Print a warning and be done with it in the RDB parser code.

Put a big fat warning into amiga-fdisk and parted!

Thanks,
Martin

> {^_^}
>
> On 20180627 22:43, Michael Schmitz wrote:
> > Joanne,
> >
> > Linux on m68k has supported lseek64 (or llseek) for a long time
> > (from glibc version 2.1 according to what I found). About the only
> > area where we are limited by 32 bits is the virtual memory size.
> >
> > I'm not proposing to modify the RDB format definition, though an
> > extension to store 64 bit offsets separate from the 32 bit ones
> > would be one way to make certain such disks are safe to use on 3.1
> > and earlier versions of AmigaOS. (Another one would be to modify
> > the disk drivers on older versions to do the offset calculation in
> > 64 bit, and check for overflow just as we do here. Not sure whether
> > that's feasible. And as you so eloquently describe, we can't rely
> > on users listening.)
> >
> > Either way, we need the cooperation of partitioning tool writers to
> > ensure partition information that is prone to overflows is never
> > stored in the 32 bit, classic RDB. That appears to have failed
> > already, as Martin's experience illustrates.
> >
> > I'm only concerned with fixing the (dangerous) but in the Linux
> > partition format parser for RDB. Refusing to use any partitions
> > that will cause havoc on old AmigaOS versions is all I can do to
> > try and get the users' attention.
> >
> > Your warning makes me wonder whether the log message should just say
> > 'report this bug to [email protected]' to at least try and
> > educate any that respond about the dangers of their partitioning
> > scheme before telling them about the override option. Problem with
> > that is, in about three years no one will remember any of this ...
> >
> > Cheers,
> >
> > Michael
> >
> > Am 28.06.2018 um 15:44 schrieb jdow:
> >> Michael, as long as m68k only supports int fseek( int ) 4 GB *
> >> block
> >> size is your HARD limit. Versions that support __int64 fseek64(
> >> __int64 ) can work with larger disks. RDBs could include RDSK and
> >> a new set of other blocks that replace the last two characters
> >> with "64" and use __int64 where needed in the various values. That
> >> way a clever disk partitioner could give allow normal (32 bit) RDB
> >> definitions where possible. Then at least SOME of the disk could
> >> be supported AND a very clever filesystem that abstracts very
> >> large disks properly could give access to the whole disk. (Read
> >> the RDBs first 32 bits. Then if a filesystem or driveinit was
> >> loaded re-read the RDBs to see if new 64 bit partitions are
> >> revealed.
> >>
> >> I could be wrong but I do not think RDBs could be safely modified
> >> any
> >> other way to work. And, trust me as I bet this is still true, you
> >> will need a SERIOUSLY good bomb shelter on the Moon if you change
> >> RDBs. Suppose Joe Amigoid uses it, and then Joe Amigoid loads
> >> Amigados 2.4 because he wants to run a game that crashes on
> >> anything newer. Then Joe got far enough something writes to the
> >> disk and data is corrupted. Note further that Amigoids do NOT,
> >> repeat NOT, listen to facts in such cases. Hell, some of them
> >> never listened to facts about an incident at Jerry Pournelle's
> >> place when a 1.1 DPaint session with Kelly Freas hung and we lost
> >> a delightful drawing. Jerry reported it. Amigoids screamed. I
> >> tried to tell them I was there, it was my machine, and 1.1 was,
> >> indeed, crap.
> >>
> >> {o.o}
> >>
> >> On 20180627 02:00, Michael Schmitz wrote:
> >>> Joanne,
> >>>
> >>> I'm not at all allergic to avoiding RDB at all cost for new disks.
> >>> If
> >>> AmigaOS 4.1 supports more recent partition formats, all the
> >>> better.
> >>> This is all about supporting use of legacy RDB disks on Linux
> >>> (though
> >>> 2 TB does stretch the definition of 'legacy' a little). My
> >>> interest in this is to ensure we can continue to use RDB format
> >>> disks on m68k Amiga computers which have no other way to boot
> >>> Linux from disk.
> >>>
> >>> Not proposing to change the RDB format at all, either. Just trying
> >>> to
> >>> make sure we translate RDB info into Linux 512-byte block offset
> >>> and
> >>> size numbers correctly. The kernel won't modify the RDB at all
> >>> (intentionally, that is - with the correct choice of partition
> >>> sizes,
> >>> Martin might well have wiped out his RDB with the current version
> >>> of
> >>> the parser).
> >>>
> >>> The choice of refusing to mount a disk (or mounting read-only)
> >>> rests
> >>> with the VFS drivers alone - AFFS in that case. Not touching any
> >>> of
> >>> that. At partition scan time, we only have the option of making
> >>> the
> >>> partition available (with a warning printed), or refusing to make
> >>> it
> >>> available to the kernel. Once it's made available, all bets are
> >>> off.
> >>>
> >>> From what Martin writes, his test case RDB was valid and worked
> >>> as
> >>> expected on 32 bit AmigaOS (4.1). Apparently, that version has the
> >>> necessary extensions to handle the large offsets resulting from 2
> >>> TB
> >>> disks. Not sure what safeguards are in place when connecting such
> >>> a
> >>> disk to older versions of AmigaOS, but that is a different matter
> >>> entirely.
> >>>
> >>> The overflows in partition offset and size are the only ones I can
> >>> see in the partition parser - there is no other overflow I've
> >>> identified. I just stated that in order to place a partition
> >>> towards the end of a 2 TB disk, the offset calculation will
> >>> overflow regardless of what combination of rdb->rdb_BlockBytes
> >>> and sector addresses stored in the RDB (in units of 512 byte
> >>> blocks) we use:
> >>>
> >>> blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
> >>>
> >>>
> >>> nr_sects = (be32_to_cpu(pb->pb_Environment[10]) +
> >>> 1 - be32_to_cpu(pb->pb_Environment[9])) *
> >>> be32_to_cpu(pb->pb_Environment[3]) *
> >>> be32_to_cpu(pb->pb_Environment[5]) * blksize;
> >>> if (!nr_sects)
> >>> continue;
> >>> start_sect = be32_to_cpu(pb->pb_Environment[9]) *
> >>> be32_to_cpu(pb->pb_Environment[3]) *
> >>> be32_to_cpu(pb->pb_Environment[5]) *
> >>> blksize;
> >>>
> >>> But in the interest of avoiding any accidental use of a RDB
> >>> partition
> >>> where calculations currently overflow, I'll make the default
> >>> behaviour to bail out (instead of using wrong offset or size as
> >>> we currently do). Given the 'eat_my_RDB_disk=1' boot option, the
> >>> user may proceed at their own risk (though I still can't see what
> >>> harm should result from now translating a well formed v4.1 2 TB
> >>> disk RDB correctly for the first time).
> >>>
> >>> Whether or not Linux correctly handles AFFS filesystems larger
> >>> than 1
> >>> TB is a matter for VFS experts. Bailing out on nr_sects
> >>> overflowing
> >>> ought to prevent accidental use of AFFS filesystems on RDB disks
> >>> which I suppose is the majority of use cases.
> >>>
> >>> Bugs in partitioning tools on Linux are entirely out of scope -
> >>> the
> >>> partitioning tools bypass the partition structure discovered by
> >>> the
> >>> kernel, and work straight on the raw device. No protecting against
> >>> that.
> >>>
> >>> If you can point out a way to cause data loss with these
> >>> precautions,
> >>> for a disk 2 TB or larger that was partitioned and used on a
> >>> recent
> >>> version or AmigaOS supporting such large disks, I'd consider
> >>> omitting
> >>> the 'eat_my_RDB_disk' boot option, and just bail out as the only
> >>> safe
> >>> option.
> >>>
> >>> Cheers,
> >>>
> >>> Michael
> >>>
> >>> Am 27.06.2018 um 18:24 schrieb jdow:
> >>>> You allergic to using a GPT solution? It will get away from some
> >>>> of the evils that RDB has inherent in it because they are also
> >>>> features? (Loading a filesystem or DriveInit code from RDBs is
> >>>> just asking for a nearly impossible to remove malware
> >>>> infection.) Furthermore, any 32 bit system that sees an RDSK
> >>>> block is going to try to translate it. If you add a new RDB
> >>>> format you are going to get bizarre and probably quite
> >>>> destructive results from the mistake. Fail safe is a rather good
> >>>> notion, methinks.
> >>>>
> >>>> Personally I figure this is all rather surreal. 2TG of junk on an
> >>>> Amiga system seems utterly outlandish to me. You cited another
> >>>> overflow potential. There are at least three we've identified, I
> >>>> believe. Are you 100% sure there are no more? The specific one
> >>>> you mention of translating RDB to Linux has a proper solution in
> >>>> the RDB reader. It should recover such overflow errors in the
> >>>> RDB as it can with due care and polish. It should flag any other
> >>>> overflow error it detects within the RDBs and return an error
> >>>> such as to leave the disk unmounted or mounted read-only if you
> >>>> feel like messing up a poor sod's backups. The simple solution
> >>>> is to read each of the variables with the nominal RDB size and
> >>>> convert it to uint64_t before calculating byte indices.
> >>>>
> >>>> However, consider my inputs as advice from an adult who has seen
> >>>> the
> >>>> Amiga Elephant so to speak. I am not trying to assert any
> >>>> control. Do as you wish; but, I would plead with you to avoid
> >>>> ANY chance you can for the user to make a bonehead stupid move
> >>>> and lose all his treasured disk archives. Doing otherwise is
> >>>> very poor form.
> >>>>
> >>>> {o.o} Joanne "Said enough, she has" Dow
> >>>>
> >>>> On 20180626 18:07, Michael Schmitz wrote:
> >>>>> Joanne,
> >>>>>
> >>>>> As far as I have been able to test, the change is backwards
> >>>>> compatible (RDB partitions from an old disk 80 GB disk are
> >>>>> still recognized OK). That"s only been done on an emulator
> >>>>> though.
> >>>>>
> >>>>> Your advice about the dangers of using RDB disks that would have
> >>>>> failed the current Linux RDB parser on legacy 32 bit systems is
> >>>>> well
> >>>>> taken though. Maybe Martin can clarify that for me - was the 2
> >>>>> TB disk in question ever used on a 32 bit Amiga system?
> >>>>>
> >>>>> RDB disk format is meant for legacy use only, so I think we can
> >>>>> get
> >>>>> away with printing a big fat warning during boot, advising the
> >>>>> user
> >>>>> that the oversize RDB partition(s) scanned are not compatible
> >>>>> with
> >>>>> legacy 32 bit AmigaOS. With the proposed fix they will work
> >>>>> under both AmigaOS 4.1 and Linux instead of truncating the
> >>>>> first overflowing partition at disk end and trashing valid
> >>>>> partitions that overlap, which is what Martin was after.
> >>>>>
> >>>>> If that still seems too risky, we can make the default behaviour
> >>>>> to
> >>>>> bail out once a potential overflow is detected, and allow the
> >>>>> user to
> >>>>> override that through a boot parameter. I'd leave that decision
> >>>>> up for the code review on linux-block.
> >>>>>
> >>>>> Two more comments: Linux uses 512 byte block sizes for the
> >>>>> partition
> >>>>> start and size calculations, so a change of the RDB blocksize to
> >>>>> reduce the block counts stored in the RDB would still result in
> >>>>> the
> >>>>> same overflow. And amiga-fdisk is indeed utterly broken and
> >>>>> needs
> >>>>> updating (along with probably most legacy m68k partitioners).
> >>>>> Adrian
> >>>>> has advertised parted as replacement for the old tools - maybe
> >>>>> this
> >>>>> would make a nice test case for parted?
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Michael
> >>>>>
> >>>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <[email protected]>
wrote:
> >>>>>> If it is not backwards compatible I for one would refuse to use
> >>>>>> it.
> >>>>>> And if
> >>>>>> it still mattered that much to me I'd also generate a
> >>>>>> reasonable
> >>>>>> alternative. Modifying RDBs nay not be even an approximation of
> >>>>>> a
> >>>>>> good idea.
> >>>>>> You'd discover that as soon as an RDB uint64_t disk is tasted
> >>>>>> by a
> >>>>>> uint32_t
> >>>>>> only system. If it is for your personal use then you're
> >>>>>> entirely
> >>>>>> free to
> >>>>>> reject my advice and are probably smart enough to keep it
> >>>>>> working for
> >>>>>> yourself.
> >>>>>>
> >>>>>> GPT is probably the right way to go. Preserve the ability to
> >>>>>> read
> >>>>>> RDBs for
> >>>>>> legacy disks only.
> >>>>>>
> >>>>>> {^_^}
> >>>>>>
> >>>>>> On 20180626 01:31, Michael Schmitz wrote:
> >>>>>>> Joanne,
> >>>>>>>
> >>>>>>> I think we all agree that doing 32 bit calculations on
> >>>>>>> 512-byte block
> >>>>>>> addresses that overflow on disks 2 TB and larger is a bug,
> >>>>>>> causing
> >>>>>>> the
> >>>>>>> issues Martin reported. Your patch addresses that by using the
> >>>>>>> correct
> >>>>>>> data type for the calculations (as do other partition parsers
> >>>>>>> that
> >>>>>>> may
> >>>>>>> have to deal with large disks) and fixes Martin's bug, so
> >>>>>>> appears
> >>>>>>> to be
> >>>>>>> the right thing to do.
> >>>>>>>
> >>>>>>> Using 64 bit data types for disks smaller than 2 TB where
> >>>>>>> calculations
> >>>>>>> don't currently overflow is not expected to cause new issues,
> >>>>>>> other
> >>>>>>> than
> >>>>>>> enabling use of disk and partitions larger than 2 TB (which
> >>>>>>> may have
> >>>>>>> ramifications with filesystems on these partitions). So
> >>>>>>> comptibility is
> >>>>>>> preserved.
> >>>>>>>
> >>>>>>> Forcing larger block sizes might be a good strategy to avoid
> >>>>>>> overflow
> >>>>>>> issues in filesystems as well, but I can't see how the block
> >>>>>>> size
> >>>>>>> stored
> >>>>>>> in the RDB would enforce use of the same block size in
> >>>>>>> filesystems.
> >>>>>>> We'll have to rely on the filesystem tools to get that right,
> >>>>>>> too.
> >>>>>>> Linux
> >>>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this
> >>>>>>> should
> >>>>>>> allow partitions larger than 2 TB to work already (but I
> >>>>>>> suspect Al
> >>>>>>> Viro
> >>>>>>> may have found a few issues when he looked at the AFFS code so
> >>>>>>> I
> >>>>>>> won't
> >>>>>>> say more). Anyway partitioning tools and filesystems are
> >>>>>>> unrelated to
> >>>>>>> the Linux partition parser code which is all we aim to fix in
> >>>>>>> this
> >>>>>>> patch.
> >>>>>>>
> >>>>>>> If you feel strongly about unknown ramifications of any
> >>>>>>> filesystems on
> >>>>>>> partitions larger than 2 TB, say so and I'll have the kernel
> >>>>>>> print a
> >>>>>>> warning about these partitions.
> >>>>>>>
> >>>>>>> I'll get this patch tested on Martin's test case image as well
> >>>>>>> as
> >>>>>>> on a
> >>>>>>> RDB image from a disk known to currently work under Linux
> >>>>>>> (thanks
> >>>>>>> Geert
> >>>>>>> for the losetup hint). Can't do much more without procuring a
> >>>>>>> working
> >>>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I
> >>>>>>> plan to
> >>>>>>> use
> >>>>>>> for tests is a long way away from my home indeed.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>>
> >>>>>>> Michael
> >>>>>>>
> >>>>>>> Am 26.06.18 um 17:17 schrieb jdow:
> >>>>>>>> As long as it preserves compatibility it should be OK, I
> >>>>>>>> suppose.
> >>>>>>>> Personally I'd make any partitioning tool front end gently
> >>>>>>>> force the
> >>>>>>>> block size towards 8k as the disk size gets larger. The file
> >>>>>>>> systems
> >>>>>>>> may also run into 2TB issues that are not obvious. An unused
> >>>>>>>> blocks
> >>>>>>>> list will have to go beyond a uint32_t size, for example. But
> >>>>>>>> a
> >>>>>>>> block
> >>>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a
> >>>>>>>> tad
> >>>>>>>> under
> >>>>>>>> 1% of the disk all by itself. A block bitmap is not quite so
> >>>>>>>> bad.
> >>>>>>>> {^_-}
> >>>>>>>>
> >>>>>>>> Just be sure you are aware of all the ramifications when you
> >>>>>>>> make a
> >>>>>>>> change. I remember thinking about this for awhile and then
> >>>>>>>> determining
> >>>>>>>> I REALLY did not want to think about it as my brain was
> >>>>>>>> getting tied
> >>>>>>>> into a gordian knot.
> >>>>>>>>
> >>>>>>>> {^_^}
> >>>>>>>>
> >>>>>>>> On 20180625 19:23, Michael Schmitz wrote:
> >>>>>>>>> Joanne,
> >>>>>>>>>
> >>>>>>>>> Martin's boot log (including your patch) says:
> >>>>>>>>>
> >>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK
> >>>>>>>>> (512)
> >>>>>>>>> sdb1
> >>>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3
> >>>>>>>>> (DOS^C)(res
> >>>>>>>>> 2 spb
> >>>>>>>>> 4)
> >>>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0:
> >>>>>>>>> [sdb]
> >>>>>>>>> Attached SCSI disk
> >>>>>>>>>
> >>>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512)
> >>>>>>>>> means
> >>>>>>>>> 512
> >>>>>>>>> byte blocks) and can be worked around by using a different
> >>>>>>>>> block
> >>>>>>>>> size.
> >>>>>>>>>
> >>>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>>>>>>> units.
> >>>>>>>>> I'll still submit a patch to Jens anyway as this may bite
> >>>>>>>>> others
> >>>>>>>>> yet.
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>>
> >>>>>>>>> Michael
> >>>>>>>>>
> >>>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]>
wrote:
> >>>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
> >>>>>>>>>> system is
> >>>>>>>>>> a famn
> >>>>>>>>>> dool.
> >>>>>>>>>>
> >>>>>>>>>> If memory serves the RDBs think in blocks rather than bytes
> >>>>>>>>>> so it
> >>>>>>>>>> should
> >>>>>>>>>> work up to 2 gigablocks whatever your block size is. 512
> >>>>>>>>>> blocks is
> >>>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of
> >>>>>>>>>> disk in
> >>>>>>>>>> block maps.
> >>>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>>>>>>
> >>>>>>>>>> {^_^}
> >>>>>>>>>>
> >>>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>>>>>>> Hi.
> >>>>>>>>>>>
> >>>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>>>>>>> test results at
> >>>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given
> >>>>>>>>>>>> there,
> >>>>>>>>>>>> so if
> >>>>>>>>>>>> Martin now submits the patch, all should be well?
> >>>>>>>>>>>
> >>>>>>>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>>>>>>
> >>>>>>>>>>> I do not care enough about this, in order to motivate
> >>>>>>>>>>> myself
> >>>>>>>>>>> preparing
> >>>>>>>>>>> the a patch from Joanne Dow´s fix.
> >>>>>>>>>>>
> >>>>>>>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>>>>>>> Sam440ep
> >>>>>>>>>>> which
> >>>>>>>>>>> I still have in my apartment.
> >>>>>>>>>>>
> >>>>>>>>>>> So RDB support in Linux it remains broken for disks larger
> >>>>>>>>>>> 2 TB,
> >>>>>>>>>>> unless
> >>>>>>>>>>> someone else does.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k"
> in the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Martin

2018-06-28 09:58:47

[permalink] [raw]

Subject: Re: Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?)

Changing subject, so that there is at least a chance for someone to find
this discussions with a search engine :)

Joanne,

jdow - 28.06.18, 04:57:
> The issue is what happens when one of those disks appears on a 3.1
> system. {^_^}

That is right, so I think the warning about 64 bit support in native OS
is okay, but that issue already exists *without* Linux.

Remember, I created that RDB with Media Toolbox on AmigaOS 4.0. I did
not even use Linux to create that beast :). If it booms in AmigaOS < 4
without NSD64, TD64 or SCSI direct, that would happen with or without
the warning in Linux, even without the disk ever have been seen by a
Linux kernel.

I´d say the warning about support in native OS does not harm, even when
it is about educating Amiga users who, in case they use that large
drives – and I pretty much bet, some of them do –, better know the
limitations beforehand.

I do think the extra kernel option does not make all that much sense,
but I accept it anyway.

Cause if you argue like that, what would need fixing is AmigaOS < 4
without NSD64, TD64 or SCSI direct, but then that is what NSD64 and TD64
was made for more than 10 years ago.

Of course, if a partitioning tool for Linux ever allows to create such
an RDB, it makes sense to add a big fat warning about that. As… I think
would make sense to have in Media Toolbox and AmigaOS partitioning
tools.

However Linux here just reads the RDB, so I´d personally go with the
warning about support in native OS, but spare myself the extra kernel
option stuff.

It is Michael´s call tough, as he submits the patch. And if he chooses
to be on a safer side than this, that is fine with me.

Thanks,
Martin

> On 20180627 01:03, Martin Steigerwald wrote:
> > Dear Joanne.
> >
> > jdow - 27.06.18, 08:24:
> >> You allergic to using a GPT solution? It will get away from some of
> >> the evils that RDB has inherent in it because they are also
> >> features?
> >> (Loading a filesystem or DriveInit code from RDBs is just asking
> >> for
> >> a nearly impossible to remove malware infection.) Furthermore, any
> >> 32
> >> bit system that sees an RDSK block is going to try to translate it.
> >> If you add a new RDB format you are going to get bizarre and
> >> probably
> >> quite destructive results from the mistake. Fail safe is a rather
> >> good notion, methinks.
> >>
> >> Personally I figure this is all rather surreal. 2TG of junk on an
> >> Amiga system seems utterly outlandish to me. You cited another
> >> overflow potential. There are at least three we've identified, I
> >> believe. Are you 100% sure there are no more? The specific one you
> >> mention of translating RDB to Linux has a proper solution in the
> >> RDB
> >> reader. It should recover such overflow errors in the RDB as it can
> >> with due care and polish. It should flag any other overflow error
> >> it
> >> detects within the RDBs and return an error such as to leave the
> >> disk
> >> unmounted or mounted read-only if you feel like messing up a poor
> >> sod's backups. The simple solution is to read each of the variables
> >> with the nominal RDB size and convert it to uint64_t before
> >> calculating byte indices.
> >>
> >> However, consider my inputs as advice from an adult who has seen
> >> the
> >> Amiga Elephant so to speak. I am not trying to assert any control.
> >> Do
> >> as you wish; but, I would plead with you to avoid ANY chance you
> >> can
> >> for the user to make a bonehead stupid move and lose all his
> >> treasured disk archives. Doing otherwise is very poor form.
> >
> > I am pretty confident that larger than 2 TiB disks are fully
> > supported within AmigaOS 4, as I outlined in my other mail.
> >
> > So with all due respect: I used a larger than 2 TiB disk in AmigaOS
> > 4 in 2012 already *just* fine. I even found I had the same
> > questions back then, and researched it. Which lead to this official
> > article back then:
> >
> > http://wiki.amigaos.net/wiki/RDB
> >
> > I am also pretty sure that AmigaOS still uses RDB as partitioning
> > format. They support MBR. I don´t think AmigaOS 4.1 supports GPT.
> > Whether to implement that of course is the decision of AmigaOS 4
> > development team. I am no longer a member of it since some time.
> >
> > Linux m68k should already be able to use disks in GPT format, but
> > you
> > likely won´t be able to read them in AmigaOS, unless there is some
> > third party support for it meanwhile.
> >
> > Thanks,
> > Martin
> >
> >> {o.o} Joanne "Said enough, she has" Dow
> >>
> >> On 20180626 18:07, Michael Schmitz wrote:
> >>> Joanne,
> >>>
> >>> As far as I have been able to test, the change is backwards
> >>> compatible (RDB partitions from an old disk 80 GB disk are still
> >>> recognized OK). That"s only been done on an emulator though.
> >>>
> >>> Your advice about the dangers of using RDB disks that would have
> >>> failed the current Linux RDB parser on legacy 32 bit systems is
> >>> well
> >>> taken though. Maybe Martin can clarify that for me - was the 2 TB
> >>> disk in question ever used on a 32 bit Amiga system?
> >>>
> >>> RDB disk format is meant for legacy use only, so I think we can
> >>> get
> >>> away with printing a big fat warning during boot, advising the
> >>> user
> >>> that the oversize RDB partition(s) scanned are not compatible with
> >>> legacy 32 bit AmigaOS. With the proposed fix they will work under
> >>> both AmigaOS 4.1 and Linux instead of truncating the first
> >>> overflowing partition at disk end and trashing valid partitions
> >>> that overlap, which is what Martin was after.
> >>>
> >>> If that still seems too risky, we can make the default behaviour
> >>> to
> >>> bail out once a potential overflow is detected, and allow the user
> >>> to
> >>> override that through a boot parameter. I'd leave that decision up
> >>> for the code review on linux-block.
> >>>
> >>> Two more comments: Linux uses 512 byte block sizes for the
> >>> partition
> >>> start and size calculations, so a change of the RDB blocksize to
> >>> reduce the block counts stored in the RDB would still result in
> >>> the
> >>> same overflow. And amiga-fdisk is indeed utterly broken and needs
> >>> updating (along with probably most legacy m68k partitioners).
> >>> Adrian
> >>> has advertised parted as replacement for the old tools - maybe
> >>> this
> >>> would make a nice test case for parted?
> >>>
> >>> Cheers,
> >>>
> >>> Michael
> >>>
> >>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <[email protected]> wrote:
> >>>> If it is not backwards compatible I for one would refuse to use
> >>>> it.
> >>>> And if it still mattered that much to me I'd also generate a
> >>>> reasonable alternative. Modifying RDBs nay not be even an
> >>>> approximation of a good idea. You'd discover that as soon as an
> >>>> RDB uint64_t disk is tasted by a uint32_t only system. If it is
> >>>> for your personal use then you're entirely free to reject my
> >>>> advice and are probably smart enough to keep it working for
> >>>> yourself.
> >>>>
> >>>> GPT is probably the right way to go. Preserve the ability to read
> >>>> RDBs for legacy disks only.
> >>>>
> >>>> {^_^}
> >>>>
> >>>> On 20180626 01:31, Michael Schmitz wrote:
> >>>>> Joanne,
> >>>>>
> >>>>> I think we all agree that doing 32 bit calculations on 512-byte
> >>>>> block
> >>>>> addresses that overflow on disks 2 TB and larger is a bug,
> >>>>> causing
> >>>>> the issues Martin reported. Your patch addresses that by using
> >>>>> the correct data type for the calculations (as do other
> >>>>> partition
> >>>>> parsers that may have to deal with large disks) and fixes
> >>>>> Martin's bug, so appears to be the right thing to do.
> >>>>>
> >>>>> Using 64 bit data types for disks smaller than 2 TB where
> >>>>> calculations don't currently overflow is not expected to cause
> >>>>> new issues, other than enabling use of disk and partitions
> >>>>> larger
> >>>>> than 2 TB (which may have ramifications with filesystems on
> >>>>> these
> >>>>> partitions). So comptibility is preserved.
> >>>>>
> >>>>> Forcing larger block sizes might be a good strategy to avoid
> >>>>> overflow
> >>>>> issues in filesystems as well, but I can't see how the block
> >>>>> size
> >>>>> stored in the RDB would enforce use of the same block size in
> >>>>> filesystems. We'll have to rely on the filesystem tools to get
> >>>>> that right, too. Linux AFFS does allow block sizes up to 4k (VFS
> >>>>> limitation) so this should allow partitions larger than 2 TB to
> >>>>> work already (but I suspect Al Viro may have found a few issues
> >>>>> when he looked at the AFFS code so I won't say more). Anyway
> >>>>> partitioning tools and filesystems are unrelated to the Linux
> >>>>> partition parser code which is all we aim to fix in this patch.
> >>>>>
> >>>>> If you feel strongly about unknown ramifications of any
> >>>>> filesystems on partitions larger than 2 TB, say so and I'll have
> >>>>> the kernel print a warning about these partitions.
> >>>>>
> >>>>> I'll get this patch tested on Martin's test case image as well
> >>>>> as
> >>>>> on a RDB image from a disk known to currently work under Linux
> >>>>> (thanks Geert for the losetup hint). Can't do much more without
> >>>>> procuring a working Amiga disk image to use with an emulator,
> >>>>> sorry. The Amiga I plan to use for tests is a long way away from
> >>>>> my home indeed.
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Michael
> >>>>>
> >>>>> Am 26.06.18 um 17:17 schrieb jdow:
> >>>>>> As long as it preserves compatibility it should be OK, I
> >>>>>> suppose.
> >>>>>> Personally I'd make any partitioning tool front end gently
> >>>>>> force
> >>>>>> the
> >>>>>> block size towards 8k as the disk size gets larger. The file
> >>>>>> systems
> >>>>>> may also run into 2TB issues that are not obvious. An unused
> >>>>>> blocks
> >>>>>> list will have to go beyond a uint32_t size, for example. But a
> >>>>>> block
> >>>>>> list (OFS for sure, don't remember for the newer AFS) uses a
> >>>>>> tad
> >>>>>> under 1% of the disk all by itself. A block bitmap is not quite
> >>>>>> so bad. {^_-}
> >>>>>>
> >>>>>> Just be sure you are aware of all the ramifications when you
> >>>>>> make
> >>>>>> a
> >>>>>> change. I remember thinking about this for awhile and then
> >>>>>> determining I REALLY did not want to think about it as my brain
> >>>>>> was getting tied into a gordian knot.
> >>>>>>
> >>>>>> {^_^}
> >>>>>>
> >>>>>> On 20180625 19:23, Michael Schmitz wrote:
> >>>>>>> Joanne,
> >>>>>>>
> >>>>>>> Martin's boot log (including your patch) says:
> >>>>>>>
> >>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK
> >>>>>>> (512)
> >>>>>>> sdb1
> >>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3
> >>>>>>> (DOS^C)(res
> >>>>>>> 2 spb
> >>>>>>> 4)
> >>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0:
> >>>>>>> [sdb]
> >>>>>>> Attached SCSI disk
> >>>>>>>
> >>>>>>> so it's indeed a case of self inflicted damage (RDSK (512)
> >>>>>>> means
> >>>>>>> 512
> >>>>>>> byte blocks) and can be worked around by using a different
> >>>>>>> block
> >>>>>>> size.
> >>>>>>>
> >>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>>>>> units.
> >>>>>>> I'll still submit a patch to Jens anyway as this may bite
> >>>>>>> others
> >>>>>>> yet.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>>
> >>>>>>> Michael
> >>>>>>>
> >>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]>
> >
> > wrote:
> >>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
> >>>>>>>> system is
> >>>>>>>> a famn
> >>>>>>>> dool.
> >>>>>>>>
> >>>>>>>> If memory serves the RDBs think in blocks rather than bytes
> >>>>>>>> so
> >>>>>>>> it
> >>>>>>>> should
> >>>>>>>> work up to 2 gigablocks whatever your block size is. 512
> >>>>>>>> blocks
> >>>>>>>> is
> >>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of disk
> >>>>>>>> in
> >>>>>>>> block maps.
> >>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>>>>
> >>>>>>>> {^_^}
> >>>>>>>>
> >>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>>>>> Hi.
> >>>>>>>>>
> >>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>>>>> test results at
> >>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>>>>> indicate the RDB parser bug is fixed by the patch given
> >>>>>>>>>> there, so if
> >>>>>>>>>> Martin now submits the patch, all should be well?
> >>>>>>>>>
> >>>>>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>>>>
> >>>>>>>>> I do not care enough about this, in order to motivate myself
> >>>>>>>>> preparing the a patch from Joanne Dow´s fix.
> >>>>>>>>>
> >>>>>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>>>>> Sam440ep
> >>>>>>>>> which
> >>>>>>>>> I still have in my apartment.
> >>>>>>>>>
> >>>>>>>>> So RDB support in Linux it remains broken for disks larger 2
> >>>>>>>>> TB,
> >>>>>>>>> unless
> >>>>>>>>> someone else does.
> >>>>>>>>>
> >>>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>>>> linux-m68k" in
> >>>>>>>> the body of a message to [email protected]
> >>>>>>>> More majordomo info at
> >>>>>>>> http://vger.kernel.org/majordomo-info.html
> >>>>>>
> >>>>>> --
> >>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>> linux-m68k" in the body of a message to
> >>>>>> [email protected]
> >>>>>> More majordomo info at
> >>>>>> http://vger.kernel.org/majordomo-info.html
> >>>>
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe
> >>>> linux-m68k" in the body of a message to [email protected]
> >>>> More majordomo info at
> >>>> http://vger.kernel.org/majordomo-info.html

--
Martin

2018-06-28 10:47:59

[permalink] [raw]

Subject: Amiga RDB partition support for disks >= 2 TB (was: Re: moving affs + RDB partition support to staging?)

Michael Schmitz - 28.06.18, 07:43:
> Joanne,
>
> Linux on m68k has supported lseek64 (or llseek) for a long time (from
> glibc version 2.1 according to what I found). About the only area
> where we are limited by 32 bits is the virtual memory size.
>
> I'm not proposing to modify the RDB format definition, though an
> extension to store 64 bit offsets separate from the 32 bit ones would
> be one way to make certain such disks are safe to use on 3.1 and
> earlier versions of AmigaOS. (Another one would be to modify the disk
> drivers on older versions to do the offset calculation in 64 bit, and
> check for overflow just as we do here. Not sure whether that's
> feasible. And as you so eloquently describe, we can't rely on users
> listening.)

I think that would be up to upstream developers to decide. That would be
AmigaOS developers and/or MorphOS, AROS developers. I could try to at
least point them to this discussion here. Whether they choose to take
the time to look at it? I don´t know. They are developing that stuff in
their spare time.

> Either way, we need the cooperation of partitioning tool writers to
> ensure partition information that is prone to overflows is never
> stored in the 32 bit, classic RDB. That appears to have failed
> already, as Martin's experience illustrates.

Heh :)

But yeah, I´d say the damage is done already.

> The raw, theoretical limit on the maximum device capacity is about
> 2^105 bytes:
>
> 32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512
> bytes/sector for the HD size in struct RigidDiskBlock

http://wiki.amigaos.net/wiki/RDB

Of course that only holds true for as long as >32 bit math is used :).
So yeah.

I wonder how many Amiga users may try to use such a large disk with a
native operating system that does not support this. NSD64 and TD64 are
at least 10 years old, if not older. Just see the dates in:

http://aminet.net/search?query=nsdpatch

http://aminet.net/package/disk/misc/td64patch

Forget about 10 years. 20 years are more accurate. This stuff is
*ancient*. Officially support is in AmigaOS since version 3.5, which has
been released more than 20 years ago as well:

https://en.wikipedia.org/wiki/AmigaOS_version_history#AmigaOS_3.5,_3.9

Even AmigaOS 4.0 is older than 10 years already.

In any case, it is the responsibility of the tool that creates or change
the RDB to take care of warning the user or using a new format.

Here the kernel just reads that which already exists in the wild.

> I'm only concerned with fixing the (dangerous) but in the Linux
> partition format parser for RDB. Refusing to use any partitions that
> will cause havoc on old AmigaOS versions is all I can do to try and
> get the users' attention.

Exactly.

And I do think, that no tool on Linux can create these kind of RDBs, and
if they can… a big fat warning belongs into that tool.

Not even Media Toolbox warns about that currently, so yes… adding an
additional warning in the kernel may be helpful.

However without refusing to parse this, the user may never notice the
warning. So that is an argument for refusing to parse this without a
kernel option.

I still think that is too much hand-holding for the few native OS users
out there. Cause in order to see the warning, the user would have to
stuff the disk into a Linux machine anyway. So the user is still
perfectly able to create such an RDB on AmigaOS 4.x or even AmigaOS
3.5/3.9 or even earlier, and then stuff the disk into a machine with
AmigaOS 1.3 and probably let it go boom there. Now, how many users who
do not know about these limits will ever put the disk into a Linux
machine, receive the warning and then be saved from data destruction? I
bet: None. It is just as unlikely as it can get. :) We are talking about
maybe a few thousand Amiga users here. And there would be even the
question how many of them will try to use disks larger than 2 TiB. I bet
there will be some, but I bet there won´t be many.

So it would be way more important to warn in Media Toolbox, amiga-fdisk,
parted and whatever other partitioning tool users may use.

I´d still leave in the warning anyway, but I think the kernel option… is
over-protecting users.

Anyway your call and I perfectly get it, in case you choose to be on a
100% safe side. You are submitting the patch.

> Your warning makes me wonder whether the log message should just say
> 'report this bug to [email protected]' to at least try and
> educate any that respond about the dangers of their partitioning
> scheme before telling them about the override option. Problem with
> that is, in about three years no one will remember any of this ...
>
> Cheers,
>
> Michael
>
> Am 28.06.2018 um 15:44 schrieb jdow:
> > Michael, as long as m68k only supports int fseek( int ) 4 GB * block
> > size is your HARD limit. Versions that support __int64 fseek64(
> > __int64 ) can work with larger disks. RDBs could include RDSK and a
> > new set of other blocks that replace the last two characters with
> > "64" and use __int64 where needed in the various values. That way a
> > clever disk partitioner could give allow normal (32 bit) RDB
> > definitions where possible. Then at least SOME of the disk could be
> > supported AND a very clever filesystem that abstracts very large
> > disks properly could give access to the whole disk. (Read the RDBs
> > first 32 bits. Then if a filesystem or driveinit was loaded re-read
> > the RDBs to see if new 64 bit partitions are revealed.
> >
> > I could be wrong but I do not think RDBs could be safely modified
> > any
> > other way to work. And, trust me as I bet this is still true, you
> > will need a SERIOUSLY good bomb shelter on the Moon if you change
> > RDBs. Suppose Joe Amigoid uses it, and then Joe Amigoid loads
> > Amigados 2.4 because he wants to run a game that crashes on
> > anything newer. Then Joe got far enough something writes to the
> > disk and data is corrupted. Note further that Amigoids do NOT,
> > repeat NOT, listen to facts in such cases. Hell, some of them never
> > listened to facts about an incident at Jerry Pournelle's place when
> > a 1.1 DPaint session with Kelly Freas hung and we lost a delightful
> > drawing. Jerry reported it. Amigoids screamed. I tried to tell them
> > I was there, it was my machine, and 1.1 was, indeed, crap.
> >
> > {o.o}
> >
> > On 20180627 02:00, Michael Schmitz wrote:
> >> Joanne,
> >>
> >> I'm not at all allergic to avoiding RDB at all cost for new disks.
> >> If
> >> AmigaOS 4.1 supports more recent partition formats, all the better.
> >> This is all about supporting use of legacy RDB disks on Linux
> >> (though
> >> 2 TB does stretch the definition of 'legacy' a little). My interest
> >> in this is to ensure we can continue to use RDB format disks on
> >> m68k Amiga computers which have no other way to boot Linux from
> >> disk.
> >>
> >> Not proposing to change the RDB format at all, either. Just trying
> >> to
> >> make sure we translate RDB info into Linux 512-byte block offset
> >> and
> >> size numbers correctly. The kernel won't modify the RDB at all
> >> (intentionally, that is - with the correct choice of partition
> >> sizes,
> >> Martin might well have wiped out his RDB with the current version
> >> of
> >> the parser).
> >>
> >> The choice of refusing to mount a disk (or mounting read-only)
> >> rests
> >> with the VFS drivers alone - AFFS in that case. Not touching any of
> >> that. At partition scan time, we only have the option of making the
> >> partition available (with a warning printed), or refusing to make
> >> it
> >> available to the kernel. Once it's made available, all bets are
> >> off.
> >>
> >> From what Martin writes, his test case RDB was valid and worked as
> >>
> >> expected on 32 bit AmigaOS (4.1). Apparently, that version has the
> >> necessary extensions to handle the large offsets resulting from 2
> >> TB
> >> disks. Not sure what safeguards are in place when connecting such a
> >> disk to older versions of AmigaOS, but that is a different matter
> >> entirely.
> >>
> >> The overflows in partition offset and size are the only ones I can
> >> see in the partition parser - there is no other overflow I've
> >> identified. I just stated that in order to place a partition
> >> towards the end of a 2 TB disk, the offset calculation will
> >> overflow regardless of what combination of rdb->rdb_BlockBytes and
> >> sector addresses stored in the>>
> >> RDB (in units of 512 byte blocks) we use:
> >> blksize = be32_to_cpu( rdb->rdb_BlockBytes ) / 512;
> >>
> >> nr_sects = (be32_to_cpu(pb->pb_Environment[10]) +
> >> 1 -
> >>
> >> be32_to_cpu(pb->pb_Environment[9])) *
> >>
> >> be32_to_cpu(pb->pb_Environment[3]) *
> >> be32_to_cpu(pb->pb_Environment[5]) *
> >> blksize;
> >>
> >> if (!nr_sects)
> >>
> >> continue;
> >>
> >> start_sect = be32_to_cpu(pb->pb_Environment[9]) *
> >>
> >> be32_to_cpu(pb->pb_Environment[3]) *
> >> be32_to_cpu(pb->pb_Environment[5]) *
> >> blksize;
> >>
> >> But in the interest of avoiding any accidental use of a RDB
> >> partition
> >> where calculations currently overflow, I'll make the default
> >> behaviour to bail out (instead of using wrong offset or size as we
> >> currently do). Given the 'eat_my_RDB_disk=1' boot option, the user
> >> may proceed at their own risk (though I still can't see what harm
> >> should result from now translating a well formed v4.1 2 TB disk
> >> RDB correctly for the first time).
> >>
> >> Whether or not Linux correctly handles AFFS filesystems larger than
> >> 1
> >> TB is a matter for VFS experts. Bailing out on nr_sects overflowing
> >> ought to prevent accidental use of AFFS filesystems on RDB disks
> >> which I suppose is the majority of use cases.
> >>
> >> Bugs in partitioning tools on Linux are entirely out of scope - the
> >> partitioning tools bypass the partition structure discovered by the
> >> kernel, and work straight on the raw device. No protecting against
> >> that.
> >>
> >> If you can point out a way to cause data loss with these
> >> precautions,
> >> for a disk 2 TB or larger that was partitioned and used on a recent
> >> version or AmigaOS supporting such large disks, I'd consider
> >> omitting
> >> the 'eat_my_RDB_disk' boot option, and just bail out as the only
> >> safe
> >> option.
> >>
> >> Cheers,
> >>
> >> Michael
> >>
> >> Am 27.06.2018 um 18:24 schrieb jdow:
> >>> You allergic to using a GPT solution? It will get away from some
> >>> of the evils that RDB has inherent in it because they are also
> >>> features? (Loading a filesystem or DriveInit code from RDBs is
> >>> just asking for a nearly impossible to remove malware infection.)
> >>> Furthermore, any 32 bit system that sees an RDSK block is going
> >>> to try to translate it. If you add a new RDB format you are going
> >>> to get bizarre and probably quite destructive results from the
> >>> mistake. Fail safe is a rather good notion, methinks.
> >>>
> >>> Personally I figure this is all rather surreal. 2TG of junk on an
> >>> Amiga system seems utterly outlandish to me. You cited another
> >>> overflow potential. There are at least three we've identified, I
> >>> believe. Are you 100% sure there are no more? The specific one
> >>> you mention of translating RDB to Linux has a proper solution in
> >>> the RDB reader. It should recover such overflow errors in the RDB
> >>> as it can with due care and polish. It should flag any other
> >>> overflow error it detects within the RDBs and return an error
> >>> such as to leave the disk unmounted or mounted read-only if you
> >>> feel like messing up a poor sod's backups. The simple solution is
> >>> to read each of the variables with the nominal RDB size and
> >>> convert it to uint64_t before calculating byte indices.
> >>>
> >>> However, consider my inputs as advice from an adult who has seen
> >>> the
> >>> Amiga Elephant so to speak. I am not trying to assert any control.
> >>> Do as you wish; but, I would plead with you to avoid ANY chance
> >>> you can for the user to make a bonehead stupid move and lose all
> >>> his treasured disk archives. Doing otherwise is very poor form.
> >>>
> >>> {o.o} Joanne "Said enough, she has" Dow
> >>>
> >>> On 20180626 18:07, Michael Schmitz wrote:
> >>>> Joanne,
> >>>>
> >>>> As far as I have been able to test, the change is backwards
> >>>> compatible (RDB partitions from an old disk 80 GB disk are still
> >>>> recognized OK). That"s only been done on an emulator though.
> >>>>
> >>>> Your advice about the dangers of using RDB disks that would have
> >>>> failed the current Linux RDB parser on legacy 32 bit systems is
> >>>> well
> >>>> taken though. Maybe Martin can clarify that for me - was the 2 TB
> >>>> disk in question ever used on a 32 bit Amiga system?
> >>>>
> >>>> RDB disk format is meant for legacy use only, so I think we can
> >>>> get
> >>>> away with printing a big fat warning during boot, advising the
> >>>> user
> >>>> that the oversize RDB partition(s) scanned are not compatible
> >>>> with
> >>>> legacy 32 bit AmigaOS. With the proposed fix they will work under
> >>>> both AmigaOS 4.1 and Linux instead of truncating the first
> >>>> overflowing partition at disk end and trashing valid partitions
> >>>> that overlap, which is what Martin was after.
> >>>>
> >>>> If that still seems too risky, we can make the default behaviour
> >>>> to
> >>>> bail out once a potential overflow is detected, and allow the
> >>>> user to
> >>>> override that through a boot parameter. I'd leave that decision
> >>>> up for the code review on linux-block.
> >>>>
> >>>> Two more comments: Linux uses 512 byte block sizes for the
> >>>> partition
> >>>> start and size calculations, so a change of the RDB blocksize to
> >>>> reduce the block counts stored in the RDB would still result in
> >>>> the
> >>>> same overflow. And amiga-fdisk is indeed utterly broken and needs
> >>>> updating (along with probably most legacy m68k partitioners).
> >>>> Adrian
> >>>> has advertised parted as replacement for the old tools - maybe
> >>>> this
> >>>> would make a nice test case for parted?
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Michael
> >>>>
> >>>> On Tue, Jun 26, 2018 at 9:45 PM, jdow <[email protected]> wrote:
> >>>>> If it is not backwards compatible I for one would refuse to use
> >>>>> it.
> >>>>> And if
> >>>>> it still mattered that much to me I'd also generate a reasonable
> >>>>> alternative. Modifying RDBs nay not be even an approximation of
> >>>>> a
> >>>>> good idea.
> >>>>> You'd discover that as soon as an RDB uint64_t disk is tasted by
> >>>>> a
> >>>>> uint32_t
> >>>>> only system. If it is for your personal use then you're entirely
> >>>>> free to
> >>>>> reject my advice and are probably smart enough to keep it
> >>>>> working for
> >>>>> yourself.
> >>>>>
> >>>>> GPT is probably the right way to go. Preserve the ability to
> >>>>> read
> >>>>> RDBs for
> >>>>> legacy disks only.
> >>>>>
> >>>>> {^_^}
> >>>>>
> >>>>> On 20180626 01:31, Michael Schmitz wrote:
> >>>>>> Joanne,
> >>>>>>
> >>>>>> I think we all agree that doing 32 bit calculations on 512-byte
> >>>>>> block
> >>>>>> addresses that overflow on disks 2 TB and larger is a bug,
> >>>>>> causing
> >>>>>> the
> >>>>>> issues Martin reported. Your patch addresses that by using the
> >>>>>> correct
> >>>>>> data type for the calculations (as do other partition parsers
> >>>>>> that
> >>>>>> may
> >>>>>> have to deal with large disks) and fixes Martin's bug, so
> >>>>>> appears
> >>>>>> to be
> >>>>>> the right thing to do.
> >>>>>>
> >>>>>> Using 64 bit data types for disks smaller than 2 TB where
> >>>>>> calculations
> >>>>>> don't currently overflow is not expected to cause new issues,
> >>>>>> other
> >>>>>> than
> >>>>>> enabling use of disk and partitions larger than 2 TB (which may
> >>>>>> have
> >>>>>> ramifications with filesystems on these partitions). So
> >>>>>> comptibility is
> >>>>>> preserved.
> >>>>>>
> >>>>>> Forcing larger block sizes might be a good strategy to avoid
> >>>>>> overflow
> >>>>>> issues in filesystems as well, but I can't see how the block
> >>>>>> size
> >>>>>> stored
> >>>>>> in the RDB would enforce use of the same block size in
> >>>>>> filesystems.
> >>>>>> We'll have to rely on the filesystem tools to get that right,
> >>>>>> too.
> >>>>>> Linux
> >>>>>> AFFS does allow block sizes up to 4k (VFS limitation) so this
> >>>>>> should
> >>>>>> allow partitions larger than 2 TB to work already (but I
> >>>>>> suspect Al
> >>>>>> Viro
> >>>>>> may have found a few issues when he looked at the AFFS code so
> >>>>>> I
> >>>>>> won't
> >>>>>> say more). Anyway partitioning tools and filesystems are
> >>>>>> unrelated to
> >>>>>> the Linux partition parser code which is all we aim to fix in
> >>>>>> this
> >>>>>> patch.
> >>>>>>
> >>>>>> If you feel strongly about unknown ramifications of any
> >>>>>> filesystems on
> >>>>>> partitions larger than 2 TB, say so and I'll have the kernel
> >>>>>> print a
> >>>>>> warning about these partitions.
> >>>>>>
> >>>>>> I'll get this patch tested on Martin's test case image as well
> >>>>>> as
> >>>>>> on a
> >>>>>> RDB image from a disk known to currently work under Linux
> >>>>>> (thanks
> >>>>>> Geert
> >>>>>> for the losetup hint). Can't do much more without procuring a
> >>>>>> working
> >>>>>> Amiga disk image to use with an emulator, sorry. The Amiga I
> >>>>>> plan to
> >>>>>> use
> >>>>>> for tests is a long way away from my home indeed.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Michael
> >>>>>>
> >>>>>> Am 26.06.18 um 17:17 schrieb jdow:
> >>>>>>> As long as it preserves compatibility it should be OK, I
> >>>>>>> suppose.
> >>>>>>> Personally I'd make any partitioning tool front end gently
> >>>>>>> force the
> >>>>>>> block size towards 8k as the disk size gets larger. The file
> >>>>>>> systems
> >>>>>>> may also run into 2TB issues that are not obvious. An unused
> >>>>>>> blocks
> >>>>>>> list will have to go beyond a uint32_t size, for example. But
> >>>>>>> a
> >>>>>>> block
> >>>>>>> list (OFS for sure, don't remember for the newer AFS) uses a
> >>>>>>> tad
> >>>>>>> under
> >>>>>>> 1% of the disk all by itself. A block bitmap is not quite so
> >>>>>>> bad.
> >>>>>>> {^_-}
> >>>>>>>
> >>>>>>> Just be sure you are aware of all the ramifications when you
> >>>>>>> make a
> >>>>>>> change. I remember thinking about this for awhile and then
> >>>>>>> determining
> >>>>>>> I REALLY did not want to think about it as my brain was
> >>>>>>> getting tied
> >>>>>>> into a gordian knot.
> >>>>>>>
> >>>>>>> {^_^}
> >>>>>>>
> >>>>>>> On 20180625 19:23, Michael Schmitz wrote:
> >>>>>>>> Joanne,
> >>>>>>>>
> >>>>>>>> Martin's boot log (including your patch) says:
> >>>>>>>>
> >>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK
> >>>>>>>> (512)
> >>>>>>>> sdb1
> >>>>>>>> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3
> >>>>>>>> (DOS^C)(res
> >>>>>>>> 2 spb
> >>>>>>>> 4)
> >>>>>>>> Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0:
> >>>>>>>> [sdb]
> >>>>>>>> Attached SCSI disk
> >>>>>>>>
> >>>>>>>> so it's indeed a case of self inflicted damage (RDSK (512)
> >>>>>>>> means
> >>>>>>>> 512
> >>>>>>>> byte blocks) and can be worked around by using a different
> >>>>>>>> block
> >>>>>>>> size.
> >>>>>>>>
> >>>>>>>> Your memory serves right indeed - blocksize is in 512 bytes
> >>>>>>>> units.
> >>>>>>>> I'll still submit a patch to Jens anyway as this may bite
> >>>>>>>> others
> >>>>>>>> yet.
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>>
> >>>>>>>> Michael
> >>>>>>>>
> >>>>>>>> On Sun, Jun 24, 2018 at 11:40 PM, jdow <[email protected]>
wrote:
> >>>>>>>>> BTW - anybody who uses 512 byte blocks with an Amiga file
> >>>>>>>>> system is
> >>>>>>>>> a famn
> >>>>>>>>> dool.
> >>>>>>>>>
> >>>>>>>>> If memory serves the RDBs think in blocks rather than bytes
> >>>>>>>>> so it
> >>>>>>>>> should
> >>>>>>>>> work up to 2 gigablocks whatever your block size is. 512
> >>>>>>>>> blocks is
> >>>>>>>>> 2199023255552 bytes. But that wastes just a WHOLE LOT of
> >>>>>>>>> disk in
> >>>>>>>>> block maps.
> >>>>>>>>> Go up to 4096 or 8192. The latter is 35 TB.
> >>>>>>>>>
> >>>>>>>>> {^_^}
> >>>>>>>>>
> >>>>>>>>> On 20180624 02:06, Martin Steigerwald wrote:
> >>>>>>>>>> Hi.
> >>>>>>>>>>
> >>>>>>>>>> Michael Schmitz - 27.04.18, 04:11:
> >>>>>>>>>>> test results at
> >>>>>>>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> >>>>>>>>>>> indicate the RDB parser bug is fixed by the patch given
> >>>>>>>>>>> there,
> >>>>>>>>>>> so if
> >>>>>>>>>>> Martin now submits the patch, all should be well?
> >>>>>>>>>>
> >>>>>>>>>> Ok, better be honest than having anyone waiting for it:
> >>>>>>>>>>
> >>>>>>>>>> I do not care enough about this, in order to motivate
> >>>>>>>>>> myself
> >>>>>>>>>> preparing
> >>>>>>>>>> the a patch from Joanne Dow´s fix.
> >>>>>>>>>>
> >>>>>>>>>> I am not even using my Amiga boxes anymore, not even the
> >>>>>>>>>> Sam440ep
> >>>>>>>>>> which
> >>>>>>>>>> I still have in my apartment.
> >>>>>>>>>>
> >>>>>>>>>> So RDB support in Linux it remains broken for disks larger
> >>>>>>>>>> 2 TB,
> >>>>>>>>>> unless
> >>>>>>>>>> someone else does.
> >>>>>>>>>>
> >>>>>>>>>> Thanks.
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>>>>> linux-m68k" in
> >>>>>>>>> the body of a message to [email protected]
> >>>>>>>>> More majordomo info at
> >>>>>>>>> http://vger.kernel.org/majordomo-info.html
> >>>>>>>
> >>>>>>> --
> >>>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>>>> linux-m68k" in
> >>>>>>> the body of a message to [email protected]
> >>>>>>> More majordomo info at
> >>>>>>> http://vger.kernel.org/majordomo-info.html
> >>>>>
> >>>>> --
> >>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>> linux-m68k" in
> >>>>> the body of a message to [email protected]
> >>>>> More majordomo info at
> >>>>> http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-m68k"
> in the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Martin

2018-06-28 11:48:48

by jdow

[permalink] [raw]

Subject: Re: Amiga RDB partition support for disks >= 2 TB

On 20180628 01:16, Martin Steigerwald wrote:
> Dear Joanne.
>
> jdow - 28.06.18, 08:39:
>> Anything done to RDBs for Linux must remain 100.000% compatible with
>> existing Amiga equipment. Otherwise, what's the point of bothering to
>> use RDBs?
>
> Done to, in the sense of written to: Yes. I completely agree. But that
> is for amiga-fdisk and parted. And for partitioning tools on native OS.

Design changes, too.

> […]
>> That brings to the fore an interesting question. Why bother with RDBs
>> over 2TB unless you want a disk with one single partition? This Win10
>> monster I am using has a modest BIOS driver partition for the OS and
>> a giant data partition. That smaller partition would easily work with
>> any RDB/Filesystem combination since 2.0. So there are some good
>> workarounds that are probably "safer" and at least as flexible as
>> RDBs, one Linux has used for a very long time, too.
>
> Well, my use case was simple:
>
> I had this 2 TB disk and I choose to share it as a backup disk for Linux
> *and* AmigaOS 4.x on that Sam440ep I still have next to me desk here.

EEEEEEK! The hair on my neck is standing up straight! Have you heard of SAMBA?
The linux mail server firewall etc machine has an extra 4TB disk on it as a
backup for the other systems, although a piddly 4TB is small when I save the
entire 3G RAID system I have. It's a proof of concept so.... A full backup on a
1gig Ethernet still takes a looooong time. But backing up even an 18GB disk on
an Amiga via 100Base-t isn't too bad. And disk speeds of the era being what they
were it's about all you can do anyway.

{o.o}

2018-06-28 12:18:47

[permalink] [raw]

Subject: Re: Amiga RDB partition support for disks >= 2 TB

Martin Steigerwald - 28.06.18, 13:30:
> jdow - 28.06.18, 12:00:
> > On 20180628 01:16, Martin Steigerwald wrote:
> […]
>
> > >> That brings to the fore an interesting question. Why bother with
> > >> RDBs
> > >> over 2TB unless you want a disk with one single partition? This
> > >> Win10
> > >> monster I am using has a modest BIOS driver partition for the OS
> > >> and
> > >> a giant data partition. That smaller partition would easily work
> > >> with
> > >> any RDB/Filesystem combination since 2.0. So there are some good
> > >> workarounds that are probably "safer" and at least as flexible as
> > >> RDBs, one Linux has used for a very long time, too.
> > >
> > > Well, my use case was simple:
> > >
> > > I had this 2 TB disk and I choose to share it as a backup disk for
> > > Linux *and* AmigaOS 4.x on that Sam440ep I still have next to me
> > > desk here.
> >
> > EEEEEEK! The hair on my neck is standing up straight! Have you heard
> > of SAMBA? The linux mail server firewall etc machine has an extra
> > 4TB
> > disk on it as a backup for the other systems, although a piddly 4TB
> > is small when I save the entire 3G RAID system I have. It's a proof
> > of concept so.... A full backup on a 1gig Ethernet still takes a
> > looooong time. But backing up even an 18GB disk on an Amiga via
> > 100Base-t isn't too bad. And disk speeds of the era being what they
> > were it's about all you can do anyway.
>
> Heh, the thing worked just fine in Amiga OS 4. I got away with it
> without an issue, until I plugged the disk to my Linux laptop and
> wrote data onto the Linux file system. Mind you, I think in that
> partition marked LNX\0 I even created a Linux LVM with pvcreate. Do
> you call that insane? Well it probably is. :)
>
> And as an Amiga user I could just return to you: I clicked it, it did
> not warn, so all is good :)
>
> But yeah, as mentioned I researched the topic before. And I think
> there
> has not even been an overflow within the RDB:
> > The raw, theoretical limit on the maximum device capacity is about
> > 2^105 bytes:
> >
> > 32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512
> > bytes/sector for the HD size in struct RigidDiskBlock
>
> http://wiki.amigaos.net/wiki/RDB_(Amiga_Rigid_Disk_Block)
>
> Confirmed by:
>
> The .ADF (Amiga Disk File) format FAQ:
> http://lclevy.free.fr/adflib/adf_info.html#p6
>
> But what do I write, you know the RDB format :)
>
> So just do the calculation in 96 Bit and you all are set :)

For sectors.

Or

3*32+9 (for 512 bytes per sector) = 105 bits

for bytes.

> Now that is a reason for 128 Bit CPUs :).
>
> Muuhahaha.
--
Martin

2018-06-28 16:37:56