LinuxLists.cc - [PATCH 00/36] Remove UCLINUX from LTP

[permalink] [raw]

Subject: Re: [PATCH 00/36] Remove UCLINUX from LTP

Hi!
> UCLINUX is broken in LTP and nobody really cares. Actually I dare to
> say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX
> from LTP. We have been actively removing UCLINUX from LTP during rewrite
> tests to new LTP API. This removes the rest from the old tests (which
> will be sooner or later rewritten to new API).
>
> Because the patchset is quite big, I did not want to send it to mailing
> lists (but I can do it if you want).

I agree that this should be done, but I'm not sure if we want to get
this in before the January release. I guess that such change would be
safer to merge just after the release so that we have a few months to
actually catch possible problems.

--
Cyril Hrubis
[email protected]

2024-01-03 08:05:22

[permalink] [raw]

Subject: Re: [LTP] [PATCH 00/36] Remove UCLINUX from LTP

Hi!
> > UCLINUX is broken in LTP and nobody really cares. Actually I dare to
> > say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX
> > from LTP. We have been actively removing UCLINUX from LTP during rewrite
> > tests to new LTP API. This removes the rest from the old tests (which
> > will be sooner or later rewritten to new API).
> >
> > Because the patchset is quite big, I did not want to send it to mailing
> > lists (but I can do it if you want).
>
> I agree that this should be done, but I'm not sure if we want to get
> this in before the January release. I guess that such change would be
> safer to merge just after the release so that we have a few months to
> actually catch possible problems.

Looking at the actuall changes it does not look awfuly complex, so maybe
we can try to merge it before the pre-release testing and hopefully
things will not break badly.

--
Cyril Hrubis
[email protected]

2024-01-03 08:40:13

[permalink] [raw]

Subject: Re: [LTP] [PATCH 00/36] Remove UCLINUX from LTP

Hi Cyril,

> Hi!
> > > UCLINUX is broken in LTP and nobody really cares. Actually I dare to
> > > say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX
> > > from LTP. We have been actively removing UCLINUX from LTP during rewrite
> > > tests to new LTP API. This removes the rest from the old tests (which
> > > will be sooner or later rewritten to new API).

> > > Because the patchset is quite big, I did not want to send it to mailing
> > > lists (but I can do it if you want).

> > I agree that this should be done, but I'm not sure if we want to get
> > this in before the January release. I guess that such change would be
> > safer to merge just after the release so that we have a few months to
> > actually catch possible problems.

> Looking at the actuall changes it does not look awfuly complex, so maybe
> we can try to merge it before the pre-release testing and hopefully
> things will not break badly.

Thanks for a quick look. Both ways would work for me, depends on you and others.
Obviously fewer rebasing is better :).

Kind regards,
Petr

2024-01-03 09:46:59

by Geert Uytterhoeven

[permalink] [raw]

Subject: Re: [PATCH 00/36] Remove UCLINUX from LTP

Hi Petr,

CC other uClinux arch lists

On Wed, Jan 3, 2024 at 2:52 AM Petr Vorel <[email protected]> wrote:
> UCLINUX is broken in LTP and nobody really cares. Actually I dare to
> say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX
> from LTP. We have been actively removing UCLINUX from LTP during rewrite
> tests to new LTP API. This removes the rest from the old tests (which
> will be sooner or later rewritten to new API).
>
> Because the patchset is quite big, I did not want to send it to mailing
> lists (but I can do it if you want).
>
> Can you please have look at my fork on gitlab, branch: remove-UCLINUX
> https://gitlab.com/pevik/ltp/-/commits/remove-UCLINUX?ref_type=heads
>
> Build test:
> https://github.com/pevik/ltp/actions/runs/7392470215

Thanks for your series!

I see you only CCed linux-m68k, but AFAIK, uClinux is not restricted
to m68k/coldfire, but also available on arm32, riscv, sh, and xtensa.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

2024-01-03 11:50:18

[permalink] [raw]

Subject: Re: [PATCH 00/36] Remove UCLINUX from LTP

Hi Geert,

> Hi Petr,

> CC other uClinux arch lists

> On Wed, Jan 3, 2024 at 2:52 AM Petr Vorel <[email protected]> wrote:
> > UCLINUX is broken in LTP and nobody really cares. Actually I dare to
> > say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX
> > from LTP. We have been actively removing UCLINUX from LTP during rewrite
> > tests to new LTP API. This removes the rest from the old tests (which
> > will be sooner or later rewritten to new API).

> > Because the patchset is quite big, I did not want to send it to mailing
> > lists (but I can do it if you want).

> > Can you please have look at my fork on gitlab, branch: remove-UCLINUX
> > https://gitlab.com/pevik/ltp/-/commits/remove-UCLINUX?ref_type=heads

> > Build test:
> > https://github.com/pevik/ltp/actions/runs/7392470215

> Thanks for your series!

Thank you for your feedback. May I add your Acked-by: tag to the series when we
agree to merge?

> I see you only CCed linux-m68k, but AFAIK, uClinux is not restricted
> to m68k/coldfire, but also available on arm32, riscv, sh, and xtensa.

Good point, I'll reply to their lists as well.

Kind regards,
Petr

> Gr{oetje,eeting}s,

> Geert

2024-01-03 11:54:49

by Geert Uytterhoeven

[permalink] [raw]

Subject: Re: [PATCH 00/36] Remove UCLINUX from LTP

Hi Petr,

On Wed, Jan 3, 2024 at 12:50 PM Petr Vorel <[email protected]> wrote:
> > On Wed, Jan 3, 2024 at 2:52 AM Petr Vorel <[email protected]> wrote:
> > > UCLINUX is broken in LTP and nobody really cares. Actually I dare to
> > > say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX
> > > from LTP. We have been actively removing UCLINUX from LTP during rewrite
> > > tests to new LTP API. This removes the rest from the old tests (which
> > > will be sooner or later rewritten to new API).
>
> > > Because the patchset is quite big, I did not want to send it to mailing
> > > lists (but I can do it if you want).
>
> > > Can you please have look at my fork on gitlab, branch: remove-UCLINUX
> > > https://gitlab.com/pevik/ltp/-/commits/remove-UCLINUX?ref_type=heads
>
> > > Build test:
> > > https://github.com/pevik/ltp/actions/runs/7392470215
>
> > Thanks for your series!
>
> Thank you for your feedback. May I add your Acked-by: tag to the series when we
> agree to merge?

I am not sure I agree with this series.
Removing support for UCLINUX from LTP is almost a guarantee for
not noticing when more breakage is introduced.

How exactly is UCLINUX broken in LTP?

Thanks!

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

2024-01-03 12:10:24

[permalink] [raw]

Subject: Re: [PATCH 00/36] Remove UCLINUX from LTP

Hi!
> I am not sure I agree with this series.
> Removing support for UCLINUX from LTP is almost a guarantee for
> not noticing when more breakage is introduced.
>
> How exactly is UCLINUX broken in LTP?

As far as we know noone is using it and nobody is maintaing it for a
decade, so it's bitrotting and we do not have manpower to fix it, or
rather we do not want to invest the scarcely limited resources we have
into something that is niche at best. We asked repeatedly if anyone want
to invest time into keeping it alive, but nobody answered the call so
far and I doubt that it will happen at this point.

--
Cyril Hrubis
[email protected]

2024-01-03 12:41:18

[permalink] [raw]

Subject: Re: [PATCH 00/36] Remove UCLINUX from LTP

Hi Geert, Cyril, all,

Geert, first, thank you for Cc all the other lists.
For anybody from those lists, we talk about:
https://lore.kernel.org/ltp/[email protected]/

> Hi!
> > I am not sure I agree with this series.
> > Removing support for UCLINUX from LTP is almost a guarantee for
> > not noticing when more breakage is introduced.

> > How exactly is UCLINUX broken in LTP?

> As far as we know noone is using it and nobody is maintaing it for a
> decade, so it's bitrotting and we do not have manpower to fix it, or
> rather we do not want to invest the scarcely limited resources we have
> into something that is niche at best. We asked repeatedly if anyone want
> to invest time into keeping it alive, but nobody answered the call so
> far and I doubt that it will happen at this point.

Also, UCLINUX was used in tests which used the legacy LTP API, which was buggy.
We slowly rewrite tests into new API [1], which is more reliable and do cleanup
and bug fixes during test rewrites. But because nobody stand to maintain UCLINUX
support, it's not in the new API. Thus we have actively deleted it's support
during the rewrite in past years.

I wonder myself if anybody is even using LTP on UCLINUX platforms. Nearly 25% of
the syscalls tests use fork(), thus will not work on UCLINUX. First tests were
rewritten in 2016 (first release in 20160510) and nobody complained.

All tests C based tests (both new and legacy API):
$ git grep -l -e 'include .tst_test.h' -e 'include .test.h' testcases/ |wc -l
1494

Tests, which use fork(), i.e. not working in UCLINUX:
$ git grep -l '\.forks_child.*1' testcases/ |wc -l
334

Kind regards,
Petr

[1] https://github.com/linux-test-project/ltp/wiki/C-Test-API

2024-01-05 03:44:35

[permalink] [raw]

Subject: Re: [PATCH 00/36] Remove UCLINUX from LTP

On 1/3/24 03:46, Geert Uytterhoeven wrote:
> Hi Petr,
>
> CC other uClinux arch lists
>
> On Wed, Jan 3, 2024 at 2:52 AM Petr Vorel <[email protected]> wrote:
>> UCLINUX is broken in LTP and nobody really cares. Actually I dare to
>> say UCLINUX is dead. Therefore I prepared patchset to remove UCLINUX
>> from LTP. We have been actively removing UCLINUX from LTP during rewrite
>> tests to new LTP API. This removes the rest from the old tests (which
>> will be sooner or later rewritten to new API).
>>
>> Because the patchset is quite big, I did not want to send it to mailing
>> lists (but I can do it if you want).
>>
>> Can you please have look at my fork on gitlab, branch: remove-UCLINUX
>> https://gitlab.com/pevik/ltp/-/commits/remove-UCLINUX?ref_type=heads
>>
>> Build test:
>> https://github.com/pevik/ltp/actions/runs/7392470215
>
> Thanks for your series!
>
> I see you only CCed linux-m68k, but AFAIK, uClinux is not restricted
> to m68k/coldfire, but also available on arm32, riscv, sh, and xtensa.

Do you mean "nommu support", or do you mean the ancient distro Jeff Dionne
stopped maintaining in 2003?

Because I've been doing nommu musl-libc systems for a few years now. Works for me...

Rob

2024-01-05 03:45:41

[permalink] [raw]

Subject: Re: [PATCH 00/36] Remove UCLINUX from LTP

On 1/3/24 06:09, Cyril Hrubis wrote:
> Hi!
>> I am not sure I agree with this series.
>> Removing support for UCLINUX from LTP is almost a guarantee for
>> not noticing when more breakage is introduced.
>>
>> How exactly is UCLINUX broken in LTP?
>
> As far as we know noone is using it and nobody is maintaing it for a
> decade,

Nobody is maintaining "uclinux" because that was a distro, but you can build
nommu support in buildroot and such, and people do.

Rob

2024-01-05 13:15:54

[permalink] [raw]

Subject: Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

Hi all,

[ Cc also automated-testing and buildroot ML
FYI thread started here:
https://lore.kernel.org/ltp/[email protected]/ ]

> On 1/3/24 06:09, Cyril Hrubis wrote:
> > Hi!
> >> I am not sure I agree with this series.
> >> Removing support for UCLINUX from LTP is almost a guarantee for
> >> not noticing when more breakage is introduced.

> >> How exactly is UCLINUX broken in LTP?

> > As far as we know noone is using it and nobody is maintaing it for a
> > decade,

> Nobody is maintaining "uclinux" because that was a distro, but you can build
> nommu support in buildroot and such, and people do.

Right, there are nommu users. Will anybody join LTP development to maintain
nommu support in LTP? The needed work is to add this support to LTP new C API
[1] and use it in the relevant test. There is some implementation in the old
API, I have no idea how well it worked.

If nobody stands for maintaing nommu, we will have to delete it. There is nobody
from the current maintainers who is using LTP on nommu HW (that is the reason why
nommu support have not been implemented in the new API).

Kind regards,
Petr

> Rob

[1] https://github.com/linux-test-project/ltp/wiki/C-Test-API

2024-01-06 03:52:18

[permalink] [raw]

Subject: Re: Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

On 1/5/24 07:11, Petr Vorel wrote:
>> Nobody is maintaining "uclinux" because that was a distro, but you can build
>> nommu support in buildroot and such, and people do.
>
> Right, there are nommu users. Will anybody join LTP development to maintain
> nommu support in LTP? The needed work is to add this support to LTP new C API
> [1] and use it in the relevant test. There is some implementation in the old
> API, I have no idea how well it worked.
>
> If nobody stands for maintaing nommu, we will have to delete it. There is nobody
> from the current maintainers who is using LTP on nommu HW (that is the reason why
> nommu support have not been implemented in the new API).

I'm interested, but overwhelmed. Not sure I've got the spoons to come up to
speed on a new project and give it regular attention just now.

I see you cc'd buildroot (although the message might not go through if you
aren't subscribed, dunno how clogged their moderation queue is these days, and
the cc: list is long enough it might twig anyway). They had a nommu fix go in
earlier this week (commit 98684ba7885b).

That said, qemu supports several nommu platforms and buildroot has defconfigs to
build systems for them:

$ git clone git://buildroot.org/buildroot
$ make help
$ make list-defconfigs | grep qemu
$ make qemu_ppc_bamboo_defconfig
$ make
(time passes...)
>>> host-gettext-tiny 0.3.2 Extracting
gzip -d -c
/home/landley/buildroot/buildroot/dl/gettext-tiny/gettext-tiny-0.3.2.tar.gz |
tar --strip-components=1 -C
/home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2 -xf -
mkdir -p
/home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2/gettext-gnu
xzcat /home/landley/buildroot/buildroot/dl/gettext-tiny/gettext-0.22.4.tar.xz |
tar --strip-components=1 -C
/home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2/gettext-gnu
-xf -
xzcat: /home/landley/buildroot/buildroot/dl/gettext-tiny/gettext-0.22.4.tar.xz:
No such file or directory
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
make: *** [package/pkg-generic.mk:209:
/home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2/.stamp_extracted]
Error 2

Sigh, never build git pull du jour of anything, buildroot's having glitch du
jour. But the point is:

$ grep -rl bamboo board/
board/qemu/ppc-bamboo/readme.txt
$ cat board/qemu/ppc-bamboo/readme.txt
Run the emulation with:

qemu-system-ppc -nographic -M bamboo -kernel output/images/vmlinux -net
nic,model=virtio-net-pci -net user # qemu_ppc_bamboo_defconfig

The login prompt will appear in the terminal that started Qemu
-------------------

In THEORY, once it builds an image (presumably using a tagged release version
rather than expecting "continuous integration" to ever mean anything) you should
be able to launch it with qemu. Assuming the instructions aren't also
bit-rotted. (Or using one of the other nommu boards, I haven't gone through the
whole list to see what they've got. I used to use a nommu arm board, but the
linux kernel broke it when converting everything to device tree and not
regression testing it.)

Buildroot also apparently has an LTP package selectable in menuconfig:

https://github.com/buildroot/buildroot/tree/master/package/ltp-testsuite

But I haven't tried it...

Rob

P.S. I automate qemu testing all the time over in toybox, see testroot.sh under
https://github.com/landley/toybox/tree/master/mkroot for an example.

2024-01-08 08:33:46

by Andrea Cervesato

[permalink] [raw]

Subject: Re: [PATCH 00/36] Remove UCLINUX from LTP

Hi!

My 2 cents. I'm working on refactoring growfiles test which uses UCLINUX
flag.
During its development I had occasion to check UCLINUX support and
(indeed) it seems pretty
broken for LTP, because nobody is maintaining it for a while and such
tests use old API that will
be replaced in any case sooner or later. I agree with other people about
removing it, unless there's
a valid reason to keep it.
Just in case we want to keep it, someone should take care about UCLINUX
support, testing LTP releases for it as well, but it doesn't seem like
something we can do inside the LTP devs team due to the lack of resources.

Regards,
Andrea

On 1/5/24 04:52, Rob Landley wrote:
> On 1/3/24 06:09, Cyril Hrubis wrote:
>> Hi!
>>> I am not sure I agree with this series.
>>> Removing support for UCLINUX from LTP is almost a guarantee for
>>> not noticing when more breakage is introduced.
>>>
>>> How exactly is UCLINUX broken in LTP?
>> As far as we know noone is using it and nobody is maintaing it for a
>> decade,
> Nobody is maintaining "uclinux" because that was a distro, but you can build
> nommu support in buildroot and such, and people do.
>
> Rob

2024-01-08 08:34:51

by Andrea Cervesato

[permalink] [raw]

Subject: Re: [PATCH 00/36] Remove UCLINUX from LTP

2024-01-08 09:04:18

[permalink] [raw]

Subject: Re: Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

Hi Rob, all,

[ Added Niklas Cassel, who is maintainer of qemu_riscv64_nommu_virt_defconfig in
buildroot ]

> On 1/5/24 07:11, Petr Vorel wrote:
> >> Nobody is maintaining "uclinux" because that was a distro, but you can build
> >> nommu support in buildroot and such, and people do.

> > Right, there are nommu users. Will anybody join LTP development to maintain
> > nommu support in LTP? The needed work is to add this support to LTP new C API
> > [1] and use it in the relevant test. There is some implementation in the old
> > API, I have no idea how well it worked.

> > If nobody stands for maintaing nommu, we will have to delete it. There is nobody
> > from the current maintainers who is using LTP on nommu HW (that is the reason why
> > nommu support have not been implemented in the new API).

> I'm interested, but overwhelmed. Not sure I've got the spoons to come up to
> speed on a new project and give it regular attention just now.

> I see you cc'd buildroot (although the message might not go through if you
> aren't subscribed, dunno how clogged their moderation queue is these days, and
> the cc: list is long enough it might twig anyway). They had a nommu fix go in
> earlier this week (commit 98684ba7885b).

> That said, qemu supports several nommu platforms and buildroot has defconfigs to
> build systems for them:

> $ git clone git://buildroot.org/buildroot
> $ make help
> $ make list-defconfigs | grep qemu
> $ make qemu_ppc_bamboo_defconfig
> $ make
> (time passes...)
> >>> host-gettext-tiny 0.3.2 Extracting
> gzip -d -c
> /home/landley/buildroot/buildroot/dl/gettext-tiny/gettext-tiny-0.3.2.tar.gz |
> tar --strip-components=1 -C
> /home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2 -xf -
> mkdir -p
> /home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2/gettext-gnu
> xzcat /home/landley/buildroot/buildroot/dl/gettext-tiny/gettext-0.22.4.tar.xz |
> tar --strip-components=1 -C
> /home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2/gettext-gnu
> -xf -
> xzcat: /home/landley/buildroot/buildroot/dl/gettext-tiny/gettext-0.22.4.tar.xz:
> No such file or directory
> tar: This does not look like a tar archive
> tar: Exiting with failure status due to previous errors
> make: *** [package/pkg-generic.mk:209:
> /home/landley/buildroot/buildroot/output/build/host-gettext-tiny-0.3.2/.stamp_extracted]
> Error 2

> Sigh, never build git pull du jour of anything, buildroot's having glitch du
> jour. But the point is:

> $ grep -rl bamboo board/
> board/qemu/ppc-bamboo/readme.txt
> $ cat board/qemu/ppc-bamboo/readme.txt
> Run the emulation with:

> qemu-system-ppc -nographic -M bamboo -kernel output/images/vmlinux -net
> nic,model=virtio-net-pci -net user # qemu_ppc_bamboo_defconfig

> The login prompt will appear in the terminal that started Qemu
> -------------------

> In THEORY, once it builds an image (presumably using a tagged release version
> rather than expecting "continuous integration" to ever mean anything) you should
> be able to launch it with qemu. Assuming the instructions aren't also
> bit-rotted. (Or using one of the other nommu boards, I haven't gone through the
> whole list to see what they've got. I used to use a nommu arm board, but the
> linux kernel broke it when converting everything to device tree and not
> regression testing it.)

> Buildroot also apparently has an LTP package selectable in menuconfig:

> https://github.com/buildroot/buildroot/tree/master/package/ltp-testsuite

> But I haven't tried it...

I'm the maintainer of the LTP package in buildroot in my private time.
BTW I spent quite a lot of time fixing LTP (and some other system packages,
e.g. nfs-utils) compilation on some old legacy architectures reported via
http://autobuild.buildroot.net/ I've never used in the reality.
But I certainly don't have time to drive nommu support in my private time.
I don't even have an interest, I don't use any nommu device.

And I would not justify to work on nommu in my working paid by SUSE, because
that's not a platform SUSE uses. Lack of resources means that there is a vast
majority of new kernel functionality not being tested. Also with very small
resources it's hard to even fix existing tests broken by changed functionality
in each mainline kernel release.

Therefore nobody who is not involved in nommu will not find a time to support it
in LTP (support does not mean just to add the functionality to the new C API,
but run tests on nommu and fix failing bugs). I suppose nobody is paid to work
on nommu platforms, it would have to be a hobby project, right?

But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to
support him in my free time (review patches, give advices). And if nobody
stands, this patchset which removes the support in the old API will be merged
after next LTP release (in the end of January).

Kind regards,
Petr

> Rob

> P.S. I automate qemu testing all the time over in toybox, see testroot.sh under
> https://github.com/landley/toybox/tree/master/mkroot for an example.

2024-01-08 10:07:45

[permalink] [raw]

Subject: Re: Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

Hi!
> But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to
> support him in my free time (review patches, give advices). And if nobody
> stands, this patchset which removes the support in the old API will be merged
> after next LTP release (in the end of January).

Let me highlight this part, we are eager to help anybody who is willing
to pick the nommu work, but we do not have resources to drive it.

--
Cyril Hrubis
[email protected]

2024-01-09 20:17:57

[permalink] [raw]

Subject: Re: Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

On 1/8/24 03:03, Petr Vorel wrote:
> Hi Rob, all,
>
> [ Added Niklas Cassel, who is maintainer of qemu_riscv64_nommu_virt_defconfig in
> buildroot ]

Hi Niklas!

>> Buildroot also apparently has an LTP package selectable in menuconfig:
>
>> https://github.com/buildroot/buildroot/tree/master/package/ltp-testsuite
>
>> But I haven't tried it...
>
> I'm the maintainer of the LTP package in buildroot in my private time.
> BTW I spent quite a lot of time fixing LTP (and some other system packages,
> e.g. nfs-utils) compilation on some old legacy architectures reported via
> http://autobuild.buildroot.net/ I've never used in the reality.
> But I certainly don't have time to drive nommu support in my private time.
> I don't even have an interest, I don't use any nommu device.

I do, but I've never done much with LTP, and I have my hands full with toybox
and mkroot already.

> Therefore nobody who is not involved in nommu will not find a time to support it
> in LTP (support does not mean just to add the functionality to the new C API,
> but run tests on nommu and fix failing bugs). I suppose nobody is paid to work
> on nommu platforms, it would have to be a hobby project, right?

A bunch of people are paid to work on nommu platforms, and I've worked with them
a bunch, but none of them talk to linux-kernel. They find the culture toxic,
insular, and categorically dismissive of their interests.

For example, cortex-m is a large nommu platform on which vendors support Linux
BSPs, but notice how page 8 of
https://www.microsemi.com/document-portal/doc_view/132181-linux-cortex-m-users-manual
points at a cross compiler toolchain from _2010_ and page 4 says they're booting
a 2.6.33 kernel?

I'm a bit weird in that I try to get CURRENT stuff to work on nommu, and a lot
of people have been happy to consume my work, but getting any of them to post
directly to linux-kernel is like pulling teeth.

> But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to
> support him in my free time (review patches, give advices). And if nobody
> stands, this patchset which removes the support in the old API will be merged
> after next LTP release (in the end of January).

What does the API migration do? Is there a page on it ala OABI vs EABI in arm or
something?

Rob

2024-01-09 22:38:53

by Bird, Tim

[permalink] [raw]

Subject: RE: [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

> -----Original Message-----
> From: [email protected] <[email protected]> On Behalf Of Cyril Hrubis
> Hi!
> > But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to
> > support him in my free time (review patches, give advices). And if nobody
> > stands, this patchset which removes the support in the old API will be merged
> > after next LTP release (in the end of January).
>
> Let me highlight this part, we are eager to help anybody who is willing
> to pick the nommu work, but we do not have resources to drive it.

I have a couple of comments here.

I think it would be good to give a little bit more time to try to find a helper/maintainer
for this. As Rob pointed out, a lot of embedded Linux developers are using very old
kernels (and, if they are using LTP, likely very old versions of LTP). They are also
notorious for not being active on the mailing lists. So this might take some active
outreach to find helpers. (I realize that this thread is part of this
outreach effort). For this reason, I'd like a few more weeks to try to advertise this
need within the embedded Linux community.

I am not using nommu systems myself, so I'm in a similar position as Petr in terms
of it not making much sense for me to be the maintainer. However, having said that,
I have had for a few years now an idea for a background project related to LTP
that might make this a more interesting fit for me. Sony uses NuttX, and is considering
using Zephyr in some of our low-end processor systems. This includes some nommu
systems. For some time now, I have wanted to experiment with using LTP to test
the compatibility of those systems with the Linux system APIs. In full disclosure,
I have no idea if this is a feasible or useful idea or not. But it's something I'd like
to investigate.

I realize that testing non-Linux RTOSes is out-of-scope for LTP. But given that that is
something I would like to do, and that it might be relevant to the Linux nommu tests,
I would humbly request a few weeks to investigate this before the nommu code is removed.
This delay would be to see if it would make sense for me to volunteer to help out with
maintaining this otherwise abandoned code.

I can't promise anything, but I'd like to find out more about:
1) what parts of the current LTP are not supporting nommu (what's currently broken),
2) how much code we're talking about, and
3) what the desired roadmap going forward would be, to continue to support this code.

Thanks,
-- Tim

2024-01-09 23:26:21

[permalink] [raw]

Subject: Re: Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

On 10/1/24 06:24, Rob Landley wrote:
> On 1/8/24 03:03, Petr Vorel wrote:
>> Hi Rob, all,
>>
>> [ Added Niklas Cassel, who is maintainer of qemu_riscv64_nommu_virt_defconfig in
>> buildroot ]
>
> Hi Niklas!
>
>>> Buildroot also apparently has an LTP package selectable in menuconfig:
>>
>>> https://github.com/buildroot/buildroot/tree/master/package/ltp-testsuite
>>
>>> But I haven't tried it...
>>
>> I'm the maintainer of the LTP package in buildroot in my private time.
>> BTW I spent quite a lot of time fixing LTP (and some other system packages,
>> e.g. nfs-utils) compilation on some old legacy architectures reported via
>> http://autobuild.buildroot.net/ I've never used in the reality.
>> But I certainly don't have time to drive nommu support in my private time.
>> I don't even have an interest, I don't use any nommu device.
>
> I do, but I've never done much with LTP, and I have my hands full with toybox
> and mkroot already.
>
>> Therefore nobody who is not involved in nommu will not find a time to support it
>> in LTP (support does not mean just to add the functionality to the new C API,
>> but run tests on nommu and fix failing bugs). I suppose nobody is paid to work
>> on nommu platforms, it would have to be a hobby project, right?
>
> A bunch of people are paid to work on nommu platforms, and I've worked with them
> a bunch, but none of them talk to linux-kernel. They find the culture toxic,
> insular, and categorically dismissive of their interests.

I have been involved in the kernel nommu space for 20 years, and sure, there is
some of that. But mostly spending some time and effort to get involved pays off.
I have seen potential contributors show up with some arrogant attitudes too,
so it cuts both ways here.

The m68k community I have been part of has been nothing but welcoming. The mm
people have tried hard to keep nommu support up-to-date where almost none of them
actually have a vested interest in doing so.

What I have seen is that many companies working in this space just don't want
to spend the time and effort to go mainline. That is a business decision they
make, and that is fine. Heck my work in actual mainline has never really been
paid for by any company and I have sunk a _lot_ of time into it. (Full disclosure
I did get paid to work on early porting and support - just not geting it into
mainline and maintain it there).

> For example, cortex-m is a large nommu platform on which vendors support Linux
> BSPs, but notice how page 8 of
> https://www.microsemi.com/document-portal/doc_view/132181-linux-cortex-m-users-manual
> points at a cross compiler toolchain from _2010_ and page 4 says they're booting
> a 2.6.33 kernel?

Any company/person who follows the route of not working with the linux kernel
community to get their work included is going to inevitably get stuck on older
versions of everything.

> I'm a bit weird in that I try to get CURRENT stuff to work on nommu, and a lot
> of people have been happy to consume my work, but getting any of them to post
> directly to linux-kernel is like pulling teeth.

I regularly test nommu configurations (as in every kernel rc and release) on m68k
and at least every release on other architectures like arm(*) and recently on
riscv as well.

(*) somewhat annoyingly needing a minor patch to run the versatile qemu platform
I like to test with. But hey, that is on me :-)

Regards
Greg

>> But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to
>> support him in my free time (review patches, give advices). And if nobody
>> stands, this patchset which removes the support in the old API will be merged
>> after next LTP release (in the end of January).
>
> What does the API migration do? Is there a page on it ala OABI vs EABI in arm or
> something?
>
> Rob

2024-01-10 04:55:16

[permalink] [raw]

Subject: Re: [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

On 1/9/24 16:37, Bird, Tim wrote:
>> -----Original Message-----
>> From: [email protected] <[email protected]> On Behalf Of Cyril Hrubis
>> Hi!
>> > But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to
>> > support him in my free time (review patches, give advices). And if nobody
>> > stands, this patchset which removes the support in the old API will be merged
>> > after next LTP release (in the end of January).
>>
>> Let me highlight this part, we are eager to help anybody who is willing
>> to pick the nommu work, but we do not have resources to drive it.
>
> I have a couple of comments here.
>
> I think it would be good to give a little bit more time to try to find a helper/maintainer
> for this. As Rob pointed out, a lot of embedded Linux developers are using very old
> kernels (and, if they are using LTP, likely very old versions of LTP). They are also
> notorious for not being active on the mailing lists. So this might take some active
> outreach to find helpers. (I realize that this thread is part of this
> outreach effort). For this reason, I'd like a few more weeks to try to advertise this
> need within the embedded Linux community.

I'd like to point out I have the _interest_ in doing this, and might have some
ability, but what I don't have is the spare bandwidth.

I maintain toybox, which Android is using as the command line for both its
installed systems and its build system (which they call "hermetic" builds,
shipping their own build dependencies). I'm trying to get Android to build under
android, which is a slightly heavier lift than turning busybox into a
development environment capable of building Linux From Scratch starting from
just 7 packages (Linux, busybox, uclibc, make, gcc, binutils, and bash) was.

Which I _succeeded_ at some years ago, by the way:

https://landley.net/aboriginal/about.html

The development work I was doing on busybox was why I wound up maintaining that
project for a bit, and after I got it to work (the project's 1.0 release) other
projects like Alpine and Adelie Linux took it from there.

Unfortunately, this time due to the FSF's spectacular stupidity with GPLv3, the
Android trademark licensing guidelines do not allow adding any GPL code in
userspace beyond what was grandfathered in circa 2007. (Busybox predates
android, yet android does not ship busybox. There's a reason for that.) Even
those grandfathered in ones they've been steadily replacing (rewriting the
bluetooth daemon with an apache licensed version, switching the build from gcc
to llvm, replacing gnu make with kati, and so on...) So I couldn't use ANY
existing gpl code (like busybox could) and largely had to write a new one of
everything. (Android was using a lot of bsd implementations, but have you ever
tried to build the linux kernel with bsd sed or bsd make? Doesn't quite fit.)

Anyway, I'm most of the way done now (see http://landley.net/toybox/status.html
and https://landley.net/toybox/roadmap.html#dev_env) and almost all my toybox
stuff already supports nommu. I'm even writing a bash replacement shell
(handling all the <(bash/{weirdness}/{1..7}) I can manage) that has full nommu
support. To support subshells it does a vfork() and exec of itself, then
marshalls all the local variable and function state across a pipe to the child
instance. (The sending side is at
https://github.com/landley/toybox/blob/master/toys/pending/sh.c#L1360 and the
receiving side at
https://github.com/landley/toybox/blob/master/toys/pending/sh.c#L4146 .)

Speaking of which, I'm still sad that the kernel never implemented a "re-exec
self" that isn't dependent on /proc and doesn't get confused by chroots, but
this is another aspect of "linux-kernel does not care when we bring this stuff
up". The topic comes up from time to time, and some patches have been proposed,
but it has yet to result in a way to do it that I am aware of:

https://lkml.iu.edu/hypermail/linux/kernel/0612.3/0238.html
https://lkml.iu.edu/hypermail/linux/kernel/1709.1/03186.html
https://lkml.iu.edu/hypermail/linux/kernel/2005.2/07206.html

Also, ext4 eventually fixed the ext2/ext3 split, but binfmt_fdpic.c is still a
separate file and not a couple of if() statements in binfmt_elf.c. Sigh...

Anyway...

> I am not using nommu systems myself, so I'm in a similar position as Petr in terms
> of it not making much sense for me to be the maintainer. However, having said that,
> I have had for a few years now an idea for a background project related to LTP
> that might make this a more interesting fit for me. Sony uses NuttX, and is considering
> using Zephyr in some of our low-end processor systems. This includes some nommu
> systems. For some time now, I have wanted to experiment with using LTP to test
> the compatibility of those systems with the Linux system APIs. In full disclosure,
> I have no idea if this is a feasible or useful idea or not. But it's something I'd like
> to investigate.

I've been talking with Rich Felker on IRC about what's involved in porting musl
on top of RTOS du jour. There was a long list of things he said in IRC that I
could try to scrape out of the log if you're interested.

The midipix guy also pointed me at https://midipix.org/sys_sysapi.h and
https://git.midipix.org/mmglue from where he ported musl to Windows.

I also got pointed at
http://lists.landley.net/pipermail/toybox-landley.net/2024-January/029967.html
I.E. https://github.com/apexrtos/apex/blob/master/sys/kern/syscall_table.c from
somebody ELSE who did it for a different RTOS...

> I realize that testing non-Linux RTOSes is out-of-scope for LTP.

What we've basically been discussing is that "the Linux API" is a de-facto
standard somewhere beyond posix, which has been implemented a bunch of different
times now. FreeBSD's Linux emulation layer, Windows Subsystem For Linux, Google
proposed a Linux layer for Fuchsia
(https://9to5google.com/2021/02/12/google-fuchsia-os-android-linux-programs-starnix/),
here's a guy who wrote his own kernel that runs the Linux binaries
(https://github.com/vvaltchev/tilck
I.E. https://www.youtube.com/watch?v=Ce1pMlZO_mI )...

I dunno what subset of the API other operating systems want to support, but
given that Linus is an empty nester now (all three daughters off to college),
idle discussions about his eventual retirement have been quietly going on for a
while now...

> But given that that is
> something I would like to do, and that it might be relevant to the Linux nommu tests,
> I would humbly request a few weeks to investigate this before the nommu code is removed.
> This delay would be to see if it would make sense for me to volunteer to help out with
> maintaining this otherwise abandoned code.

I am interested in helping, but I am overstretched as it is.

My mkroot script builds bootable linux systems and regression tests them under
qemu for a dozen architectures, all in about 350 lines of bash. To make that
work I created an even _more_ compressed kernel config format, microconfig, so
adding support for a new target is... well I recently added or1k support and
everything the script knows about that target is the 3 lines starting at:

https://github.com/landley/toybox/blob/master/mkroot/mkroot.sh#L233

That build script is already mostly self-hosting, in that it first builds a
directory of toybox binaries (line 62) and points the $PATH at those, and then
builds toybox and a Linux kernel under that. Which works, but cheats:

https://github.com/landley/toybox/blob/master/scripts/install.sh#L105

The PENDING= command list is the binaries it symlinks out of the host $PATH, the
ones before the multi-space break are theones I have partial implementations of
in "pending" (but which aren't in defconfig yet because they're not finished),
and the ones after the break are the ones I haven't started writing yet. I.E. I
still need to finish "expr git tr bash sh gzip", and I need to start "awk bison
flex make". But once I've written all of those, I should be able to run
mkroot.sh in a mkroot image.

And THEN I need to get the automated regression test script that makes sure all
the targets still work under qemu
(https://github.com/landley/toybox/blob/master/mkroot/testroot.sh about 100
lines) to also run the toybox test suite in qemu, but that still requires actual
bash to run the test suite, toysh isn't quite finished yet (and I refuse to trim
it because I want toysh to implement all those bash features).

And THEN I need to get it to build Linux From Scratch, which I did back under
aboriginal linux back in the day:

https://landley.net/aboriginal/control-images/

(I have to start over with the current LFS 12.0 stuff because none of those old
packages know what musl is and autoconf is just craptacular. Alas, LFS is full
of gnu packages, and gnu is brittle navel-gazing crap. But that's the best
stress test for my command line stuff handling all the weired evil corner cases
it throws at them. And it's also why both busybox and toybox seds reply to
--version with "this is not gnu sed 9.0" because of STUPID autoconf regexes...)

mkroot's already got the plumbing to add arbitrary extra behavior to the images
as build packages, the
https://github.com/landley/toybox/blob/master/mkroot/packages/tests package is
just a stub for now but I know what I want to have that do once the shell's ready...

And once I've got it building Linux From Scratch, THEN I need to tackle AOSP,
which is its own rant, but luckily its maintainer (Elliott Hughes, the "Android
base OS" maintainer) is the #2 developer on toybox, and I've been discussing
these plans with him for years:

http://lists.landley.net/pipermail/toybox-landley.net/2016-July/024590.html

But he's waiting for me to get through my todo list before actually trying to
add anything like a posix container to Android. It's mostly a security thing,
the ability to create arbitrary code and then execute it is like someone asking
to bring an open flame onto an airplane:

http://lists.landley.net/pipermail/toybox-landley.net/2019-September/026992.html

The other problem is that for historical reasons each app installs as a
different UID and a "posix container" would need to be able to install a UID/GID
_range_ which the container could then remap using container plumbing (and also
lock the hell DOWN using container plumbing, so its ability to trojan the phone
and do evil maid attacks and 37 other bad things was NOT ALLOWED, which is some
design work they seem to be deferring until I wave something otherwise usable at
them)...

(Ok, and he thinks phones are too slow to build android, that nobody does
development on anything less powerful than a 32 processor machine with at least
that many gigs of ram and a fiber connection to the net because nobody at GOOGLE
has less than that. He has boggled at me about this cultural difference before,
skip to about 20:30 in
http://androidbackstage.blogspot.com/2016/07/episode-53-adb-on-adb.html for
example. Anyway, at some point I need to STRIP DOWN the AOSP build so it takes
less than forever. Haven't even opened that can of worms yet, but from a
capability standpoint it's gotta build the whole thing _first_...)

Anyway, the point is this is its own whole ecosystem, which I'd need a staff of
a dozen people to navigate properly, and there's just me. And now that toybox
has users, I get support requests coming in...

> I can't promise anything, but I'd like to find out more about:
> 1) what parts of the current LTP are not supporting nommu (what's currently broken),
> 2) how much code we're talking about, and
> 3) what the desired roadmap going forward would be, to continue to support this code.

I am very interested in these things as well. I would like to help, and am happy
to answer all the questions I can, but caffeine only takes you so far when it
comes to regular commitments...

Rob

2024-01-10 05:41:03

[permalink] [raw]

Subject: Re: Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

On 1/9/24 17:17, Greg Ungerer wrote:
>
> On 10/1/24 06:24, Rob Landley wrote:
>> On 1/8/24 03:03, Petr Vorel wrote:
>>> Hi Rob, all,
>>>
>>> [ Added Niklas Cassel, who is maintainer of qemu_riscv64_nommu_virt_defconfig in
>>> buildroot ]
>>
>> Hi Niklas!
>>
>>>> Buildroot also apparently has an LTP package selectable in menuconfig:
>>>
>>>> https://github.com/buildroot/buildroot/tree/master/package/ltp-testsuite
>>>
>>>> But I haven't tried it...
>>>
>>> I'm the maintainer of the LTP package in buildroot in my private time.
>>> BTW I spent quite a lot of time fixing LTP (and some other system packages,
>>> e.g. nfs-utils) compilation on some old legacy architectures reported via
>>> http://autobuild.buildroot.net/ I've never used in the reality.
>>> But I certainly don't have time to drive nommu support in my private time.
>>> I don't even have an interest, I don't use any nommu device.
>>
>> I do, but I've never done much with LTP, and I have my hands full with toybox
>> and mkroot already.
>>
>>> Therefore nobody who is not involved in nommu will not find a time to support it
>>> in LTP (support does not mean just to add the functionality to the new C API,
>>> but run tests on nommu and fix failing bugs). I suppose nobody is paid to work
>>> on nommu platforms, it would have to be a hobby project, right?
>>
>> A bunch of people are paid to work on nommu platforms, and I've worked with them
>> a bunch, but none of them talk to linux-kernel. They find the culture toxic,
>> insular, and categorically dismissive of their interests.
>
> I have been involved in the kernel nommu space for 20 years, and sure, there is
> some of that. But mostly spending some time and effort to get involved pays off.
> I have seen potential contributors show up with some arrogant attitudes too,
> so it cuts both ways here.
>
> The m68k community I have been part of has been nothing but welcoming. The mm
> people have tried hard to keep nommu support up-to-date where almost none of them
> actually have a vested interest in doing so.
>
> What I have seen is that many companies working in this space just don't want
> to spend the time and effort to go mainline.

Sometimes they don't bother. Sometimes there's a language barrier. Sometimes
they can't get anything newer than 2.6 working because that's the BSP they were
given so what's the point of trying to engage upstream? Sometimes they think
it's their upstream vendor's responsibility. Sometimes they poke their head up
and get it bit off ala http://landley.net/notes-2008.html#11-12-2008 and then
that serves as a warning to others for generations. Sometimes the company's
legal department thinks it's a terrible idea to attract attention from people
like the SFLC. And sometimes...

The SmartFusion 2 project I was doing on cortex-m was for a project that was to
be launched into space on a NASA rocket, and thus fell under ITAR export
regulations (as the entire US space program has since 1996 due to
https://en.wikipedia.org/wiki/International_Traffic_in_Arms_Regulations#:~:text=Intelsat
) and my manager explained it to me as:

"If I buy a screwdriver from Home Depot, it's just a screwdriver. If I use it to
turn a screw on a spacecraft it is now a munition and cannot be discussed with
non-US persons".

The stupid linked above was:

1) cryptography was categorized as a munition until Bill Clinton relaxed it via
executive order, because 56 bit https was preventing anybody from trusting the
web with their credit card data.

2) Before that, in 1996, china wanted to launch a satellite into space with
crypto stuff so they negotiated with the USA to get some cryptographic hardware
which was delivered/installed under armed guard and escorted to the launch pad.

3) The rocket blew up on launch, scattering debris over a chinese city (becuase
of COURSE china's rocket launches go over large population centers). The crypto
hardware was never recovered.

4) It became a scandal. Congress freaked. And somehow in the scuffle ITAR export
regulations were extended from cryptography to the entire US space program.

5) The US space program dried up and blew away, as engineers had to choose
between "I can work on space stuff" and "I can have any sort of professional
network online". (Because who online is a "non-US person"? That includes canada.
You can't discuss ITAR subjects with canadians. Or foreign nationals living in
the USA. You couldn't ask Alan Cox a question about an ITAR project.)

So that's a whole category that stays very quiet about what they do, and whose
legal analysis of the GPL is "we're making 3 of these and shooting them into
space, if you retrieve one from space and demand source code from us we will
forward you to the relevant federal agencies, and there's a nonzero chance
you'll be on a black site in Diego Garcia within 24 hours where they will figure
out how you did that. Or maybe you'll get the code. Who knows?"

Me, I try to avoid that kind of contract...

> That is a business decision they
> make, and that is fine. Heck my work in actual mainline has never really been
> paid for by any company and I have sunk a _lot_ of time into it. (Full disclosure
> I did get paid to work on early porting and support - just not geting it into
> mainline and maintain it there).

The thing is if you post something _once_ it gets ignored, and if you follow-up
long enough for it to go in (which often takes years), it will then get ripped
out again a few years later because "we never hear from anybody who uses this".

Engaging with the community is signing up for an ongoing commitment.

>> For example, cortex-m is a large nommu platform on which vendors support Linux
>> BSPs, but notice how page 8 of
>> https://www.microsemi.com/document-portal/doc_view/132181-linux-cortex-m-users-manual
>> points at a cross compiler toolchain from _2010_ and page 4 says they're booting
>> a 2.6.33 kernel?
>
> Any company/person who follows the route of not working with the linux kernel
> community to get their work included is going to inevitably get stuck on older
> versions of everything.

I fight hard to get current versions of everything to work on all my supported
targets. This requires regular regression testing, and I maintain a pile of
patches that I post here periodically but I fully admit will probably never go in:

https://lkml.iu.edu/hypermail/linux/kernel/2302.2/05594.html

(Sigh, now that 6.7 is out I should post the new round...)

People who want to use my kernels as a source are welcome to do so (and I've
seen my patches quietly show up in other projects), but getting upstream to
actually _fix_ anything? Every one of those patches had a link to the previous
time it was posted to the list and ignored.

I mean literally, the first of those patches teaches the makefile to autodetect
whether $PREFIX-cc is gcc or llvm and just do the right thing, and I was told
that they actively didn't want it to:

https://lkml.iu.edu/hypermail/linux/kernel/2302.2/07184.html

That is modern linux-kernel development.

>> I'm a bit weird in that I try to get CURRENT stuff to work on nommu, and a lot
>> of people have been happy to consume my work, but getting any of them to post
>> directly to linux-kernel is like pulling teeth.
>
> I regularly test nommu configurations (as in every kernel rc and release) on m68k
> and at least every release on other architectures like arm(*) and recently on
> riscv as well.

Sigh, I should start caring about riscv. I added or1k support, I should do
riscv. (Except I did or1k because I found it in actual hardware, the Orange Pi
3b's power controller is an or1k asic so I needed an or1k toolchain to build
some of u-boot's firmware or else the board couldn't reboot, and there was a
qemu-system-or1k already, which turned into adding it to mkroot via a long
https://lore.kernel.org/openrisc/ZX1xbs_AGdgLgcx7@antec/ thread with its
developers. Alas I still can't get qemu to exit (I.E. virtually reboot or power
off), apparently I need to reinstall my laptop to have a new enough version of
python 3 to build a newer qemu with. It's on the todo list...)

I still have a hard time considering riscv anything other than open source's
version of Itanium. Promises of ubiquity, but even a 28 nanometer mask is still
6 figures before you run any wafers and your mask build process is sucking in
all the black box libraries the fab can sell you, so what does "open" really get
you here? Cortex-m got cheap when the superh patents expired so Arm didn't have
to pay royalties to hitachi (renesas?) for the thumb instruction set anymore,
and they belt those suckers en masse amortizing the up-front costs over ENORMOUS
volume.

And yes, j-core was trying to fix the closed source library and toolchain issues
back when I was still working with them. Among other things fishing
Google/skywater's openlane toolchain build out of their magic docker and
reproducing it under a vanilla debootstrap, ala
https://github.com/j-core/openlane-vhdl-build (As with most corporate
clusterfscks, once you dig far enough it turns out you can throw over 90% of it
out...)

But these days I'm trying to get toybox to 1.0...

> (*) somewhat annoyingly needing a minor patch to run the versatile qemu platform
> I like to test with. But hey, that is on me :-)

I would very much like to add more nommu targets to mkroot, can I get your
build/config info offline? (I tried fishing configs out of buildroot a couple
years ago, but after the THIRD one where the secret was "use very old versions
of packages, the current stuff is broken"... And the problems were things like
"the conversion to device tree deleted a huge chunk of this infrastructure", not
simple fixes.)

> Regards
> Greg

Rob

2024-01-10 13:36:53

[permalink] [raw]

Subject: Re: Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

Hi Rob, all,

> On 1/8/24 03:03, Petr Vorel wrote:
> > Hi Rob, all,

> > [ Added Niklas Cassel, who is maintainer of qemu_riscv64_nommu_virt_defconfig in
> > buildroot ]

> Hi Niklas!

> >> Buildroot also apparently has an LTP package selectable in menuconfig:

> >> https://github.com/buildroot/buildroot/tree/master/package/ltp-testsuite

> >> But I haven't tried it...

> > I'm the maintainer of the LTP package in buildroot in my private time.
> > BTW I spent quite a lot of time fixing LTP (and some other system packages,
> > e.g. nfs-utils) compilation on some old legacy architectures reported via
> > http://autobuild.buildroot.net/ I've never used in the reality.
> > But I certainly don't have time to drive nommu support in my private time.
> > I don't even have an interest, I don't use any nommu device.

> I do, but I've never done much with LTP, and I have my hands full with toybox
> and mkroot already.

Understand.

> > Therefore nobody who is not involved in nommu will not find a time to support it
> > in LTP (support does not mean just to add the functionality to the new C API,
> > but run tests on nommu and fix failing bugs). I suppose nobody is paid to work
> > on nommu platforms, it would have to be a hobby project, right?

> A bunch of people are paid to work on nommu platforms, and I've worked with them
> a bunch, but none of them talk to linux-kernel. They find the culture toxic,
> insular, and categorically dismissive of their interests.

> For example, cortex-m is a large nommu platform on which vendors support Linux
> BSPs, but notice how page 8 of
> https://www.microsemi.com/document-portal/doc_view/132181-linux-cortex-m-users-manual
> points at a cross compiler toolchain from _2010_ and page 4 says they're booting
> a 2.6.33 kernel?

> I'm a bit weird in that I try to get CURRENT stuff to work on nommu, and a lot
> of people have been happy to consume my work, but getting any of them to post
> directly to linux-kernel is like pulling teeth.

Interesting, thanks for sharing this. BTW I'm not saying anybody is using nommu,
but I wonder if anybody really test it with LTP. And if yes, I wonder why we
don't have reports about tests broken in new API.

> > But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to
> > support him in my free time (review patches, give advices). And if nobody
> > stands, this patchset which removes the support in the old API will be merged
> > after next LTP release (in the end of January).

> What does the API migration do? Is there a page on it ala OABI vs EABI in arm or
> something?

New C API is documented at our wiki: the API for using in the tests [1]
and the library itself [2]. (We also have shell API, but we can ignore it for
nommu.)

All files in lib/ directory which include tst_test.h are part of new C API. Main
file is lib/tst_test.c. LTP tests, which has been rewritten to new API include
tst_test.h, they are in testcases/ directory. Library has it's own tests (for
testing regression in in lib/newlib_tests/*.c.

The reason why Cyril wrote in 2016 new C API was that the old API was buggy
(tests randomly fails). Tests which are still using the old API (there is
ongoing rewrite) include test.h. The old API is not much documented.

Feel free to ask any more question.

Kind regards,
Petr

[1] https://github.com/linux-test-project/ltp/wiki/C-Test-API
[2] https://github.com/linux-test-project/ltp/tree/master/lib

> Rob

2024-01-10 14:18:46

[permalink] [raw]

Subject: Re: [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

Hi Tim, all,

> > -----Original Message-----
> > From: [email protected] <[email protected]> On Behalf Of Cyril Hrubis
> > Hi!
> > > But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to
> > > support him in my free time (review patches, give advices). And if nobody
> > > stands, this patchset which removes the support in the old API will be merged
> > > after next LTP release (in the end of January).

> > Let me highlight this part, we are eager to help anybody who is willing
> > to pick the nommu work, but we do not have resources to drive it.

> I have a couple of comments here.

> I think it would be good to give a little bit more time to try to find a helper/maintainer
> for this. As Rob pointed out, a lot of embedded Linux developers are using very old
> kernels (and, if they are using LTP, likely very old versions of LTP). They are also
> notorious for not being active on the mailing lists. So this might take some active
> outreach to find helpers. (I realize that this thread is part of this
> outreach effort). For this reason, I'd like a few more weeks to try to advertise this
> need within the embedded Linux community.

Thank you.

> I am not using nommu systems myself, so I'm in a similar position as Petr in terms
> of it not making much sense for me to be the maintainer. However, having said that,
> I have had for a few years now an idea for a background project related to LTP
> that might make this a more interesting fit for me. Sony uses NuttX, and is considering
> using Zephyr in some of our low-end processor systems. This includes some nommu
> systems. For some time now, I have wanted to experiment with using LTP to test
> the compatibility of those systems with the Linux system APIs. In full disclosure,
> I have no idea if this is a feasible or useful idea or not. But it's something I'd like
> to investigate.

> I realize that testing non-Linux RTOSes is out-of-scope for LTP. But given that that is
> something I would like to do, and that it might be relevant to the Linux nommu tests,
> I would humbly request a few weeks to investigate this before the nommu code is removed.
> This delay would be to see if it would make sense for me to volunteer to help out with
> maintaining this otherwise abandoned code.

> I can't promise anything, but I'd like to find out more about:
> 1) what parts of the current LTP are not supporting nommu (what's currently broken),
The new C API, I described it in my reply to Rob:
https://lore.kernel.org/ltp/20240110133358.GB1698252@pevik/

But I don't know whether the code in the old API was even working,
whole old API suffered with random failures, that was one of the reasons to
write a new one from the scratch.

> 2) how much code we're talking about, and

There was FORK_OR_VFORK(), which would probably in the new API call vfork() for
nommu targets (tst_old_flush() is probably not needed in the new API).

There is a special handling of getopts in lib/parse_opts.c + -C param for it.
One would have to integrate these two functions from lib/self_exec.c to the new
API (and port them to use new API via tst_test.h with #define
TST_NO_DEFAULT_MAIN):

void maybe_run_child(void (*child)(), const char *fmt, ...);
int self_exec(const char *argv0, const char *fmt, ...);

char *child_args is somehow integrated to lib/tst_test.c via -C arg, I haven't
found what uses that option.

There is m4, that would be usable (m4/ltp-nommu-linux.m4).

Various tests and testsuites were not compiled for nommu (e.g. capget).

There is MAP_PRIVATE_EXCEPT_UCLINUX constant to avoid using MAP_PRIVATE on
uClinux, who knows if this is relevant on nommu?

> 3) what the desired roadmap going forward would be, to continue to support this code.

All LTP tests are being rewritten to use new API since 2016 (new API was
introduced in 20160510), thus we are loosing the support with old API going
away. Sure, I can hold on this patchset and we continue removing the
functionality tests manually. But sooner or later it's gone.

One can check files which had special handling in the old API:

$ git grep -l UCLINUX 20160126 -- testcases/ | wc -l
173

What is supported now:

$ git grep -l UCLINUX -- testcases/ |wc -l
55

=> We have now removed nearly 2/3 of it (this means we're arguing about 1/3 of
the tests which initially somehow supported nommu).

Kind regards,
Petr

> Thanks,
> -- Tim

> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#1271): https://lists.yoctoproject.org/g/automated-testing/message/1271
> Mute This Topic: https://lists.yoctoproject.org/mt/103541824/3616762
> Group Owner: [email protected]
> Unsubscribe: https://lists.yoctoproject.org/g/automated-testing/unsub [[email protected]]
> -=-=-=-=-=-=-=-=-=-=-=-

2024-01-10 14:46:49

[permalink] [raw]

Subject: Re: Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

On 10/1/24 15:47, Rob Landley wrote:
> On 1/9/24 17:17, Greg Ungerer wrote:
>> On 10/1/24 06:24, Rob Landley wrote:
>>> I'm a bit weird in that I try to get CURRENT stuff to work on nommu, and a lot
>>> of people have been happy to consume my work, but getting any of them to post
>>> directly to linux-kernel is like pulling teeth.
>>
>> I regularly test nommu configurations (as in every kernel rc and release) on m68k
>> and at least every release on other architectures like arm(*) and recently on
>> riscv as well.
>
> Sigh, I should start caring about riscv. I added or1k support, I should do
> riscv. (Except I did or1k because I found it in actual hardware, the Orange Pi
> 3b's power controller is an or1k asic so I needed an or1k toolchain to build
> some of u-boot's firmware or else the board couldn't reboot, and there was a
> qemu-system-or1k already, which turned into adding it to mkroot via a long
> https://lore.kernel.org/openrisc/ZX1xbs_AGdgLgcx7@antec/ thread with its
> developers. Alas I still can't get qemu to exit (I.E. virtually reboot or power
> off), apparently I need to reinstall my laptop to have a new enough version of
> python 3 to build a newer qemu with. It's on the todo list...)
>
> I still have a hard time considering riscv anything other than open source's
> version of Itanium. Promises of ubiquity, but even a 28 nanometer mask is still
> 6 figures before you run any wafers and your mask build process is sucking in
> all the black box libraries the fab can sell you, so what does "open" really get
> you here? Cortex-m got cheap when the superh patents expired so Arm didn't have
> to pay royalties to hitachi (renesas?) for the thumb instruction set anymore,
> and they belt those suckers en masse amortizing the up-front costs over ENORMOUS
> volume.
>
> And yes, j-core was trying to fix the closed source library and toolchain issues
> back when I was still working with them. Among other things fishing
> Google/skywater's openlane toolchain build out of their magic docker and
> reproducing it under a vanilla debootstrap, ala
> https://github.com/j-core/openlane-vhdl-build (As with most corporate
> clusterfscks, once you dig far enough it turns out you can throw over 90% of it
> out...)
>
> But these days I'm trying to get toybox to 1.0...
>
>> (*) somewhat annoyingly needing a minor patch to run the versatile qemu platform
>> I like to test with. But hey, that is on me :-)
>
> I would very much like to add more nommu targets to mkroot, can I get your
> build/config info offline? (I tried fishing configs out of buildroot a couple
> years ago, but after the THIRD one where the secret was "use very old versions
> of packages, the current stuff is broken"... And the problems were things like
> "the conversion to device tree deleted a huge chunk of this infrastructure", not
> simple fixes.)

Maybe getting a little off-topic here, but I'll just send links here.
Who knows it might be useful to others.

Recently I have been experimenting with minimal builds, this is a bunch of
scripts, configs and a couple of patches I currently have:

https://github.com/gregungerer/simple-linux

Mostly the kernel builds use the architecture defconfigs, but for armnommu
versatile it was easier to use a dedicated config and patch:

https://github.com/gregungerer/simple-linux/blob/master/configs/linux-6.6-armnommu-versatile.config
https://github.com/gregungerer/simple-linux/blob/master/patches/linux-6.6-armnommu-versatile.patch

Anyway the scripting uses the newest package versions of everything
(binutils, gcc, linux, uClibc, busybox).

Regards
Greg

2024-01-10 18:16:51

[permalink] [raw]

Subject: Re: Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

On 1/10/24 07:33, Petr Vorel wrote:
>> I'm a bit weird in that I try to get CURRENT stuff to work on nommu, and a lot
>> of people have been happy to consume my work, but getting any of them to post
>> directly to linux-kernel is like pulling teeth.
>
> Interesting, thanks for sharing this. BTW I'm not saying anybody is using nommu,
> but I wonder if anybody really test it with LTP. And if yes, I wonder why we
> don't have reports about tests broken in new API.

I don't expect a lot of nommu users are aware you ever _could_ run LTP on nommu.

But I'd like to get nommu more regularly supported. You _should_ be able to
build a musl-linux userspace with busybox or toybox and be able to build a
recognizable system (even an alpine-alike) which could then get the basic
plumbing regression tested on qemu even without access to nommu hardware.

>> > But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to
>> > support him in my free time (review patches, give advices). And if nobody
>> > stands, this patchset which removes the support in the old API will be merged
>> > after next LTP release (in the end of January).
>
>> What does the API migration do? Is there a page on it ala OABI vs EABI in arm or
>> something?
>
> New C API is documented at our wiki: the API for using in the tests [1]
> and the library itself [2]. (We also have shell API, but we can ignore it for
> nommu.)

I'm writing a bash-compatible shell, which (thanks to Elliott forwarding
questions) has involved surprisingly long threads with the bash maintainer about
weird corner cases neither the man page nor my testing made clear:

http://lists.landley.net/pipermail/toybox-landley.net/2023-July/029631.html

(Alas I try NOT to involve him because when I bring stuff up he keeps FIXING
BASH which from my point of view just makes it a moving target...)

Anyway, running the shell API on nommu doesn't seem out of the question, but
probably not any time soon. (The fact the shell isn't finished yet is one of the
big REASONS I haven't got enough time to take on LTP. That and I haven't started
writing "awk" and "make" yet". And I need to cycle back to
https://landley.net/notes-2023.html#12-10-2023 . And after that debian, ala
https://peertube.debian.social/w/chzkKrMvEczG7qQyjbMKPr and
https://peertube.debian.social/w/45XroN9CnbYLNLKQH3GD9F . And follow up on
https://lists.gnu.org/archive/html/coreutils/2023-08/msg00009.html . And...)

> All files in lib/ directory which include tst_test.h are part of new C API. Main
> file is lib/tst_test.c.

safe_fork(), safe_clone(), fork_testrun()...

> LTP tests, which has been rewritten to new API include
> tst_test.h, they are in testcases/ directory. Library has it's own tests (for
> testing regression in in lib/newlib_tests/*.c.

Library meaning... libc? Or does LTP have a library?

> The reason why Cyril wrote in 2016 new C API was that the old API was buggy
> (tests randomly fails). Tests which are still using the old API (there is
> ongoing rewrite) include test.h. The old API is not much documented.
>
> Feel free to ask any more question.

My standard questions are "what does success look like" and "how do I reproduce
the problem".

For the first: if there previously was nommu support in LTP, what's the last
version that's known to work? Is there an existing build/test setup that can be
reproduced?

For the second... If I try to run LTP on sh2eb (my current nommu test board)
with the current LTP... do I get a build break? Additional test failures at
runtime? You talk about "removing nommu support", but... what's the current
status? (A subset of tests still use the old api...?)

Yes I need to read https://github.com/linux-test-project/ltp/wiki/C-Test-API but
I also need to know how to build LTP from source. I'm looking at the README's
list of "autoconf, automake, m4, pkgconf / pkg-config" and wincing
significantly. (What does gnu/autoconf DO here? Disable tests? I never
understand why anybody uses that giant hairball of complexity. Half of cross
compiling is figuring out how to lie to autoconf, and my normal workaround for
that is to bootstrap a target system and build natively, but while I've gotten
gcc to run natively on nommu systems, I never _tried_ gnu/autoconf.
Bootstrapping some subset of LFS on a nommu system so it has the dependencies
LFS needs to natively build seems like the long way 'round...

(I am not the right guy for "make it work the easy way". I am the guy who will
step on every land mine between here and there. I code by debugging an empty
screen. If I don't start from "known working" setup... it would take a while.)

Rob

2024-01-10 19:17:27

[permalink] [raw]

Subject: Re: [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

On 1/10/24 08:14, Petr Vorel wrote:
> There is MAP_PRIVATE_EXCEPT_UCLINUX constant to avoid using MAP_PRIVATE on
> uClinux, who knows if this is relevant on nommu?

MAP_PRIVATE creates a copy-on-write mapping, and doing copy-on-write requires an
MMU. (You mark it read only in the page table, take a fault when they try to
write, the fault handler allocates a new physical page, copies the old contents
to it, marks it writeable, and returns allowing the write to complete to the new
page.)

On NOMMU you can MAP_SHARED and MAP_ANON, but not MAP_PRIVATE.

Swap is implemented kind of similarly, except when you recycle pages you mark
them as neither readable nor writeable in the page table, schedule the page's
contents to be written to disk, suspend the process so the scheduler can go run
something else, and then when you get the I/O completion interrupt you mark the
page free so whatever else needed a page can use it. And then when the process
tries to access the page the fault handler reverses the process, allocating a
new physical page and load in the contents back in while the process is
suspended waiting for that to finish. Can't do that without an MMU either.

>> 3) what the desired roadmap going forward would be, to continue to support this code.
>
> All LTP tests are being rewritten to use new API since 2016 (new API was
> introduced in 20160510), thus we are loosing the support with old API going
> away. Sure, I can hold on this patchset and we continue removing the
> functionality tests manually. But sooner or later it's gone.

You can't fork() on nommu because copies of the mappings have different
addresses, meaning any pointers in the copied mappings would point into the OLD
mappings (belonging to the parent process), and fixing them up is 100%
equivalent to the "garbage collection in C" problem. (It's AI-complete. Of the
C3PO kind, not the "autocorrect with syntax checking" kind.) People get hung up
on the "it would be very inefficient to do that because no copy-on-write"
problem and miss the "the child couldn't FUNCTION because its pointer variables
all contain parent addresses" problem.

So instead vfork() creates a child with all the same memory mappings (a bit like
a thread) and freezes the parent process until that child discards those
mappings, either by calling exec() or _exit(). (At which point the parent gets
un-suspended.)

The child can do quite a lot of setup before calling exec, it already has its
own filehandle table for example, but it has to be really careful about MEMORY.
Anything it writes to global variables the parent will see, any changes to the
heap persist in the parent, and anything it writes to local variables the parent
MAY see. (Systems have historically differed about whether the vfork() child
gets a new stack like a thread would, or keeps using the parent's mapping since
the new stack would be quickly discarded anyway. If you call into a new setup()
function after vfork() it doesn't matter much either way, but do NOT return from
the function that called vfork(): either your new stack hasn't got anything to
return to or you'll corrupt the parent's stack by overwriting its return address
so when the parent exits from its current function it jumps to la-la land.)

The OTHER fun thing about nommu is you can't run conventional ELF binaries,
because everything is linked at fixed address. So you might be able to run ONE
instance of the program as your init task, assuming those addresses were
available even then, but as soon as you try to run a second one it's a conflict.

The quick and dirty work around is to make PIE binaries, which can relocate
everything into available space, which works but doesn't scale. The problem with
ELF PIE is that everything is linked contiguously from a single base pointer,
meaning your text, rodata, data, and bss segments are all one linear blob. So if
you run two instances of bash, you've loaded two copies of the test and the
rodoata. This fills up your memory fast.

AND PIE requires contiguous memory, which nommu is bad at providing because it
has no page tables to remap stuff. With an mmu it can coalesce scattered
physical pages into a virtually contiguous range, but without an mmu you can
have plenty of memory free but in tiny chunks, none big enough to satisfy an
allocation request.

So they invented FDPIC, which is ELF with FOUR base pointers. Each major section
(rodata, text, data, and bss) has its own base pointer, so you need to find
smaller chunks of memory to load them into (and thus it can work on a more
fragmented system), AND it means that two instances of the same program can
share the read-only sections (rodata and text) so you only need new copies of
the writeable segments (data and bss. And the heap. And the stack.)

(The earlier binflt format is an a.out variant with 4 base pointers. FDPIC is
the ELF version of the same idea. Since a.out went bye-bye binflt is obsolete,
but not everybody's moved off it yet because so many nommu people are still
using 2.6 or even earlier, and also using gcc 3.x or 2.x toolchains.)

Oh, the OTHER thing is none of this is deferred allocation, it's all up front.
On systems with mmu you can populate large empty virtual mappings that read as
zeroed but it's actually redundant copy-on-write instances of the zero page, and
when you write to them it takes a soft fault and the fault handler allocates the
page you dirtied when you dirty it. On nommu, if you want a 4 megabyte mapping
you have to find 4 contiguous megabyte and allocate it immediately, or else the
mmap() or malloc() returns failure. (On systems with mmu malloc() almost never
returns NULL, because you've got virtual address space coming out of your ears
and if you ACTUALLY run out of memory that's happens way later, the OOM killer
triggers long after malloc() returned success. But on a nommu system, malloc()
returns NULL all the time, even if you THINK you have enough memory, because
what's left is too fragmented to contain a free chunk THAT BIG...)

This impacts the stack. On MMU Linux, the default stack size is 8 megs but it's
seldom all used. On nommu linux, that would be RIDICULOUS because A) it would
always be allocated to its full size right up front, B) you'd need contiguous
memory for it. So instead you set the default stack size when building the
linker (you can also set it on the ld command line), and common sizes range from
8k to maybe 256k depending on what you expect to be running. Toybox tries not to
use more than 16k stack, although I usually test it with 32k on nommu. (There's
no guard page to tell you when you went off the edge, because no MMU so no page
faults, but you can check that the stack page at end-16k is still clean at exit
if you like. Some nommu hardware has range registers, but Linux has never
supported them that I'm aware of.)

There's not THAT much to learn about NOMMU. It could all be covered in an hour
presentation at a conference, I expect?

> One can check files which had special handling in the old API:
>
> $ git grep -l UCLINUX 20160126 -- testcases/ | wc -l
> 173
>
> What is supported now:
>
> $ git grep -l UCLINUX -- testcases/ |wc -l
> 55

UCLINUX is a long-dead distro. Linaro died in the dot-com crash and its founder
Jeff Dionne moved to Japan for his next gig and never came back. On the way out
he handed uclinux off to someone else, who didn't do a lot of work maintaining
it. Most of the actual support went "upstream" into various packages (linux and
busybox and gcc and so on) before the handoff, so you didn't NEED uclinux anymore.

The real nail in the coffin is the inheritor of uclinux never migrated it off
CVS, and then the disk holding the CVS archive crashed with no backup. He came
out with a couple more releases after that by monkey-patching the last release's
filesystem, but the writing was on the wall and it rolled to a stop.

I did a triage of its last release (from 2014) as part of my toybox roadmap:

https://landley.net/toybox/roadmap.html#uclinux

> => We have now removed nearly 2/3 of it (this means we're arguing about 1/3 of
> the tests which initially somehow supported nommu).

I'd like to get more tests supporting nommu. Possibly the approach here is just
get ANYTHING working with the new api, and then whack-a-mole more tests in as we go.

Other than lacking fork(), restricted mmap(), different executable packaging,
smaller stack size, having to actually test the return from malloc(), no page
faults if you follow a wild pointer, and the complete lack of swap space...

Unless I missed something, it's otherwise normal Linux? (People comfortable with
threads can still do all their thread tricks on nommu systems. And the Turtle
board I'm using is an SMP nommu system, they do exist. :)

Rob

2024-01-10 21:17:50

by Niklas Cassel

[permalink] [raw]

Subject: Re: [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

On Wed, Jan 10, 2024 at 01:23:51PM -0600, Rob Landley wrote:
> UCLINUX is a long-dead distro. Linaro died in the dot-com crash and its founder
> Jeff Dionne moved to Japan for his next gig and never came back. On the way out
> he handed uclinux off to someone else, who didn't do a lot of work maintaining
> it. Most of the actual support went "upstream" into various packages (linux and
> busybox and gcc and so on) before the handoff, so you didn't NEED uclinux anymore.

s/Linaro/Lineo/

Kind regards,
Niklas

2024-01-10 22:33:47

[permalink] [raw]

Subject: Re: Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

> On 1/10/24 07:33, Petr Vorel wrote:
> >> I'm a bit weird in that I try to get CURRENT stuff to work on nommu, and a lot
> >> of people have been happy to consume my work, but getting any of them to post
> >> directly to linux-kernel is like pulling teeth.

> > Interesting, thanks for sharing this. BTW I'm not saying anybody is using nommu,
> > but I wonder if anybody really test it with LTP. And if yes, I wonder why we
> > don't have reports about tests broken in new API.

> I don't expect a lot of nommu users are aware you ever _could_ run LTP on nommu.

> But I'd like to get nommu more regularly supported. You _should_ be able to
> build a musl-linux userspace with busybox or toybox and be able to build a
> recognizable system (even an alpine-alike) which could then get the basic
> plumbing regression tested on qemu even without access to nommu hardware.

> >> > But as I said, if anybody from nommu decides to maintain it in LTP, I'll try to
> >> > support him in my free time (review patches, give advices). And if nobody
> >> > stands, this patchset which removes the support in the old API will be merged
> >> > after next LTP release (in the end of January).

> >> What does the API migration do? Is there a page on it ala OABI vs EABI in arm or
> >> something?

> > New C API is documented at our wiki: the API for using in the tests [1]
> > and the library itself [2]. (We also have shell API, but we can ignore it for
> > nommu.)

> I'm writing a bash-compatible shell, which (thanks to Elliott forwarding
> questions) has involved surprisingly long threads with the bash maintainer about
> weird corner cases neither the man page nor my testing made clear:

> http://lists.landley.net/pipermail/toybox-landley.net/2023-July/029631.html

> (Alas I try NOT to involve him because when I bring stuff up he keeps FIXING
> BASH which from my point of view just makes it a moving target...)

> Anyway, running the shell API on nommu doesn't seem out of the question, but
> probably not any time soon. (The fact the shell isn't finished yet is one of the
> big REASONS I haven't got enough time to take on LTP. That and I haven't started
> writing "awk" and "make" yet". And I need to cycle back to
> https://landley.net/notes-2023.html#12-10-2023 . And after that debian, ala
> https://peertube.debian.social/w/chzkKrMvEczG7qQyjbMKPr and
> https://peertube.debian.social/w/45XroN9CnbYLNLKQH3GD9F . And follow up on
> https://lists.gnu.org/archive/html/coreutils/2023-08/msg00009.html . And...)

> > All files in lib/ directory which include tst_test.h are part of new C API. Main
> > file is lib/tst_test.c.

> safe_fork(), safe_clone(), fork_testrun()...

> > LTP tests, which has been rewritten to new API include
> > tst_test.h, they are in testcases/ directory. Library has it's own tests (for
> > testing regression in in lib/newlib_tests/*.c.

> Library meaning... libc? Or does LTP have a library?

Yes, LTP has a library (lib/libltp.a). That's what I meant here and in all my
text. So far I did not mention anything libc specific.

> > The reason why Cyril wrote in 2016 new C API was that the old API was buggy
> > (tests randomly fails). Tests which are still using the old API (there is
> > ongoing rewrite) include test.h. The old API is not much documented.

> > Feel free to ask any more question.

> My standard questions are "what does success look like" and "how do I reproduce
> the problem".

> For the first: if there previously was nommu support in LTP, what's the last
> version that's known to work? Is there an existing build/test setup that can be
> reproduced?

I have no idea whether it worked. Best would be to ask Mike Frysinger (the
author of m4/ltp-nommu-linux.m4). The code was added 14 years ago, even before
all of the current maintainers were involved.

> For the second... If I try to run LTP on sh2eb (my current nommu test board)
> with the current LTP... do I get a build break? Additional test failures at
> runtime? You talk about "removing nommu support", but... what's the current
> status? (A subset of tests still use the old api...?)

Yes, subset of the tests which use the old API (git grep UCLINUX).

> Yes I need to read https://github.com/linux-test-project/ltp/wiki/C-Test-API but
> I also need to know how to build LTP from source. I'm looking at the README's
> list of "autoconf, automake, m4, pkgconf / pkg-config" and wincing
> significantly. (What does gnu/autoconf DO here? Disable tests? I never
> understand why anybody uses that giant hairball of complexity. Half of cross
> compiling is figuring out how to lie to autoconf, and my normal workaround for
> that is to bootstrap a target system and build natively, but while I've gotten
> gcc to run natively on nommu systems, I never _tried_ gnu/autoconf.
> Bootstrapping some subset of LFS on a nommu system so it has the dependencies
> LFS needs to natively build seems like the long way 'round...

Well, one day we might migrate to use something else (meson?), but until then
autoconf + m4 + pkgconf is used (instead of automake there is LTP custom
system). This was written in 2009 and nobody plans to change it (well, Andrea
played with meson [1] [2]). But we got far away from the original topic :).

Kind regards,
Petr

[1] https://github.com/acerv/ltp-core
[2] https://github.com/acerv/ltp-testcases

> (I am not the right guy for "make it work the easy way". I am the guy who will
> step on every land mine between here and there. I code by debugging an empty
> screen. If I don't start from "known working" setup... it would take a while.)

> Rob

2024-01-11 00:00:26

[permalink] [raw]

Subject: Re: [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

On 11/1/24 07:17, Niklas Cassel wrote:
> On Wed, Jan 10, 2024 at 01:23:51PM -0600, Rob Landley wrote:
>> UCLINUX is a long-dead distro. Linaro died in the dot-com crash and its founder
>> Jeff Dionne moved to Japan for his next gig and never came back. On the way out
>> he handed uclinux off to someone else, who didn't do a lot of work maintaining
>> it. Most of the actual support went "upstream" into various packages (linux and
>> busybox and gcc and so on) before the handoff, so you didn't NEED uclinux anymore.
>
> s/Linaro/Lineo/

Lineo was not founded by Jeff Dionne, see https://en.wikipedia.org/wiki/Lineo
for its genisys. Maybe you are thinking of RT-Control.

Greg

2024-01-11 02:26:32

[permalink] [raw]

Subject: Re: [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

Sorry Rob, but I think you are wrong about a number of things here.
I know, I was there at the coal face so to speak during the early
years of uClinux and right up today. I feel I have to correct some of
the things you claim.

On 11/1/24 05:23, Rob Landley wrote:
> On 1/10/24 08:14, Petr Vorel wrote:
>> There is MAP_PRIVATE_EXCEPT_UCLINUX constant to avoid using MAP_PRIVATE on
>> uClinux, who knows if this is relevant on nommu?
>
> MAP_PRIVATE creates a copy-on-write mapping, and doing copy-on-write requires an
> MMU. (You mark it read only in the page table, take a fault when they try to
> write, the fault handler allocates a new physical page, copies the old contents
> to it, marks it writeable, and returns allowing the write to complete to the new
> page.)
>
> On NOMMU you can MAP_SHARED and MAP_ANON, but not MAP_PRIVATE.
>
> Swap is implemented kind of similarly, except when you recycle pages you mark
> them as neither readable nor writeable in the page table, schedule the page's
> contents to be written to disk, suspend the process so the scheduler can go run
> something else, and then when you get the I/O completion interrupt you mark the
> page free so whatever else needed a page can use it. And then when the process
> tries to access the page the fault handler reverses the process, allocating a
> new physical page and load in the contents back in while the process is
> suspended waiting for that to finish. Can't do that without an MMU either.
>
>>> 3) what the desired roadmap going forward would be, to continue to support this code.
>>
>> All LTP tests are being rewritten to use new API since 2016 (new API was
>> introduced in 20160510), thus we are loosing the support with old API going
>> away. Sure, I can hold on this patchset and we continue removing the
>> functionality tests manually. But sooner or later it's gone.
>
> You can't fork() on nommu because copies of the mappings have different
> addresses, meaning any pointers in the copied mappings would point into the OLD
> mappings (belonging to the parent process), and fixing them up is 100%
> equivalent to the "garbage collection in C" problem. (It's AI-complete. Of the
> C3PO kind, not the "autocorrect with syntax checking" kind.) People get hung up
> on the "it would be very inefficient to do that because no copy-on-write"
> problem and miss the "the child couldn't FUNCTION because its pointer variables
> all contain parent addresses" problem.
>
> So instead vfork() creates a child with all the same memory mappings (a bit like
> a thread) and freezes the parent process until that child discards those
> mappings, either by calling exec() or _exit(). (At which point the parent gets
> un-suspended.)

Just to be clear, vfork is not a uClinux invention. It dates way back to the
early BSD UNIX days. It just so happens that it fits in very nicely with
the no-MMU model of the world.

> The child can do quite a lot of setup before calling exec, it already has its
> own filehandle table for example, but it has to be really careful about MEMORY.
> Anything it writes to global variables the parent will see, any changes to the
> heap persist in the parent, and anything it writes to local variables the parent
> MAY see. (Systems have historically differed about whether the vfork() child
> gets a new stack like a thread would, or keeps using the parent's mapping since
> the new stack would be quickly discarded anyway. If you call into a new setup()
> function after vfork() it doesn't matter much either way, but do NOT return from
> the function that called vfork(): either your new stack hasn't got anything to
> return to or you'll corrupt the parent's stack by overwriting its return address
> so when the parent exits from its current function it jumps to la-la land.)
>
> The OTHER fun thing about nommu is you can't run conventional ELF binaries,
> because everything is linked at fixed address. So you might be able to run ONE
> instance of the program as your init task, assuming those addresses were
> available even then, but as soon as you try to run a second one it's a conflict.
>
> The quick and dirty work around is to make PIE binaries, which can relocate
> everything into available space, which works but doesn't scale. The problem with
> ELF PIE is that everything is linked contiguously from a single base pointer,
> meaning your text, rodata, data, and bss segments are all one linear blob. So if
> you run two instances of bash, you've loaded two copies of the test and the
> rodoata. This fills up your memory fast.
>
> AND PIE requires contiguous memory, which nommu is bad at providing because it
> has no page tables to remap stuff. With an mmu it can coalesce scattered
> physical pages into a virtually contiguous range, but without an mmu you can
> have plenty of memory free but in tiny chunks, none big enough to satisfy an
> allocation request.
>
> So they invented FDPIC, which is ELF with FOUR base pointers. Each major section
> (rodata, text, data, and bss) has its own base pointer, so you need to find
> smaller chunks of memory to load them into (and thus it can work on a more
> fragmented system), AND it means that two instances of the same program can
> share the read-only sections (rodata and text) so you only need new copies of
> the writeable segments (data and bss. And the heap. And the stack.)

It is worth noting that to run PIE style ELF binaries on no-MMU systems you
actually use the FDPIC ELF loader - not the traditional linux ELF loader (binfmt_elf).
Using this setup is going to become much easier now that uClibc has native
support for generating a relevant library and ld.so for noMMU linux (certainly
for m68k, arm and riscv at least) - as of version 1.0.45.

> (The earlier binflt format is an a.out variant with 4 base pointers. FDPIC is
> the ELF version of the same idea. Since a.out went bye-bye binflt is obsolete,
> but not everybody's moved off it yet because so many nommu people are still
> using 2.6 or even earlier, and also using gcc 3.x or 2.x toolchains.)

Flat format (binfmt_flat) is in no way related to a.out. It is not derived from
it and shares nothing with it. A.out being removed from the kernel in no way affects
flat format. It doesn't magically make it obsolete. It isn't going anywhere.

I don't see how this is related to versions of kernel or gcc or toolchains either.
Flat format works on every kernel up to todays v6.7. The conversion tool works
with the latest gcc and binutils (gcc-13.2 and bintuils-2.41 as of today) and many
versions going back for the best part of 20 years. Sure that elf2flt tool has been
forked a lot it has a spotty history of getting updated. But, hey, as of today it
looks pretty up to date across most architectures.

> Oh, the OTHER thing is none of this is deferred allocation, it's all up front.
> On systems with mmu you can populate large empty virtual mappings that read as
> zeroed but it's actually redundant copy-on-write instances of the zero page, and
> when you write to them it takes a soft fault and the fault handler allocates the
> page you dirtied when you dirty it. On nommu, if you want a 4 megabyte mapping
> you have to find 4 contiguous megabyte and allocate it immediately, or else the
> mmap() or malloc() returns failure. (On systems with mmu malloc() almost never
> returns NULL, because you've got virtual address space coming out of your ears
> and if you ACTUALLY run out of memory that's happens way later, the OOM killer
> triggers long after malloc() returned success. But on a nommu system, malloc()
> returns NULL all the time, even if you THINK you have enough memory, because
> what's left is too fragmented to contain a free chunk THAT BIG...)
>
> This impacts the stack. On MMU Linux, the default stack size is 8 megs but it's
> seldom all used. On nommu linux, that would be RIDICULOUS because A) it would
> always be allocated to its full size right up front, B) you'd need contiguous
> memory for it. So instead you set the default stack size when building the
> linker (you can also set it on the ld command line), and common sizes range from
> 8k to maybe 256k depending on what you expect to be running. Toybox tries not to
> use more than 16k stack, although I usually test it with 32k on nommu. (There's
> no guard page to tell you when you went off the edge, because no MMU so no page
> faults, but you can check that the stack page at end-16k is still clean at exit
> if you like. Some nommu hardware has range registers, but Linux has never
> supported them that I'm aware of.)
>
> There's not THAT much to learn about NOMMU. It could all be covered in an hour
> presentation at a conference, I expect?
>
>> One can check files which had special handling in the old API:
>>
>> $ git grep -l UCLINUX 20160126 -- testcases/ | wc -l
>> 173
>>
>> What is supported now:
>>
>> $ git grep -l UCLINUX -- testcases/ |wc -l
>> 55
>
> UCLINUX is a long-dead distro. Linaro died in the dot-com crash and its founder
> Jeff Dionne moved to Japan for his next gig and never came back. On the way out
> he handed uclinux off to someone else, who didn't do a lot of work maintaining
> it. Most of the actual support went "upstream" into various packages (linux and
> busybox and gcc and so on) before the handoff, so you didn't NEED uclinux anymore.

Why do you keep claiming that "UCLINUX" (why all caps?) is a distro?
That is just not the case. uClinux has been used as an umbrella term for
patches, tools, packages relating to running Linux on a CPU with no MMU.

There was a build package called uclinux-dist that you might consider a distro.
But it has always been cleanly named as "uclinux-dist". For a long time that
was the goto starting point for anyone wanting to use Linux with no-MMU.
The collection of patches to the toolchains, kernel, libraries and application
packages was a pretty high mountain to climb to get started in those early
days (late 90's and early 2000's). But through the work of a _lot_ of people
much of that change has made its way into relevant packages across the board
(from gcc to kernel to creation of uClibc, etc).

But lets face it once no-MMU support was integrated into mainline linux as off
v2.5.46 it is really now just "Linux". The no-MMU support outgrew that trade marked
term. But the name has stubbornly stuck around.

> The real nail in the coffin is the inheritor of uclinux never migrated it off
> CVS, and then the disk holding the CVS archive crashed with no backup. He came
> out with a couple more releases after that by monkey-patching the last release's
> filesystem, but the writing was on the wall and it rolled to a stop.

No, the public facing CVS was never the master sources for the uclinux-dist.
The public facing CVS was only ever a copy of the tar ball releases.
Its loss was annoying to some who used it but had no real impact on the
development. Its development model was kind of "internal" and published
at random points in time. Today that is what probably most people would
call the "vendor" distro model.

That work was slowing down due to fact that it just wasn't really
needed any more. There are way more popular build systems (eg build-root)
for building complete firmware images from scratch.

> I did a triage of its last release (from 2014) as part of my toybox roadmap:

No, the last release was in 2016, see it archived at:
https://sourceforge.net/projects/uclinux/files/uClinux%20Stable/

But that is all kind of archeological now, relegated to history.

Regards
Greg

2024-01-11 09:24:35

by Niklas Cassel

[permalink] [raw]

Subject: Re: [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

On Thu, Jan 11, 2024 at 10:00:11AM +1000, Greg Ungerer wrote:
>
> On 11/1/24 07:17, Niklas Cassel wrote:
> > On Wed, Jan 10, 2024 at 01:23:51PM -0600, Rob Landley wrote:
> > > UCLINUX is a long-dead distro. Linaro died in the dot-com crash and its founder
> > > Jeff Dionne moved to Japan for his next gig and never came back. On the way out
> > > he handed uclinux off to someone else, who didn't do a lot of work maintaining
> > > it. Most of the actual support went "upstream" into various packages (linux and
> > > busybox and gcc and so on) before the handoff, so you didn't NEED uclinux anymore.
> >
> > s/Linaro/Lineo/
>
> Lineo was not founded by Jeff Dionne, see https://en.wikipedia.org/wiki/Lineo
> for its genisys. Maybe you are thinking of RT-Control.

Yes, Jeff cofounded Rt-Control which later merged with Lineo.

My main point is that Linaro is a completely different company,
which did not "die in the dot-com crash", like stated above.

Kind regards,
Niklas

2024-01-11 13:12:08

by Geert Uytterhoeven

[permalink] [raw]

Subject: Re: [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

Hi Rob,

On Wed, Jan 10, 2024 at 8:17 PM Rob Landley <[email protected]> wrote:
> You can't fork() on nommu because copies of the mappings have different
> addresses, meaning any pointers in the copied mappings would point into the OLD
> mappings (belonging to the parent process), and fixing them up is 100%
> equivalent to the "garbage collection in C" problem. (It's AI-complete. Of the
> C3PO kind, not the "autocorrect with syntax checking" kind.) People get hung up
> on the "it would be very inefficient to do that because no copy-on-write"
> problem and miss the "the child couldn't FUNCTION because its pointer variables
> all contain parent addresses" problem.

Actually you can implement fork(), if you teach the compiler to use
separate stacks for return addresses and data:
- The first stack would contain only absolute addresses, to be
relocated after copying,
- The second stack would contain integers and relative pointers
(see FDPIC below), which do not need relocation after copying.

> The OTHER fun thing about nommu is you can't run conventional ELF binaries,
> because everything is linked at fixed address. So you might be able to run ONE
> instance of the program as your init task, assuming those addresses were
> available even then, but as soon as you try to run a second one it's a conflict.
>
> The quick and dirty work around is to make PIE binaries, which can relocate
> everything into available space, which works but doesn't scale. The problem with
> ELF PIE is that everything is linked contiguously from a single base pointer,
> meaning your text, rodata, data, and bss segments are all one linear blob So if
> you run two instances of bash, you've loaded two copies of the test and the
> rodoata. This fills up your memory fast.
>
> AND PIE requires contiguous memory, which nommu is bad at providing because it
> has no page tables to remap stuff. With an mmu it can coalesce scattered
> physical pages into a virtually contiguous range, but without an mmu you can
> have plenty of memory free but in tiny chunks, none big enough to satisfy an
> allocation request.
>
> So they invented FDPIC, which is ELF with FOUR base pointers. Each major section
> (rodata, text, data, and bss) has its own base pointer, so you need to find
> smaller chunks of memory to load them into (and thus it can work on a more
> fragmented system), AND it means that two instances of the same program can
> share the read-only sections (rodata and text) so you only need new copies of
> the writeable segments (data and bss. And the heap. And the stack.)

Or Amiga LoadSeg() relocatable binaries and shared libraries ;-)
As this supported splitting code, data, and bss in lots of smaller
hunks, it could counter fragmented memory quite well.

BTW, can't you run and thus test nommu-binaries under normal Linux, too?

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68korg

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

2024-01-11 13:23:44

[permalink] [raw]

Subject: Re: [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

On 11/1/24 23:11, Geert Uytterhoeven wrote:
> Hi Rob,
>
> On Wed, Jan 10, 2024 at 8:17 PM Rob Landley <[email protected]> wrote:
>> You can't fork() on nommu because copies of the mappings have different
>> addresses, meaning any pointers in the copied mappings would point into the OLD
>> mappings (belonging to the parent process), and fixing them up is 100%
>> equivalent to the "garbage collection in C" problem. (It's AI-complete. Of the
>> C3PO kind, not the "autocorrect with syntax checking" kind.) People get hung up
>> on the "it would be very inefficient to do that because no copy-on-write"
>> problem and miss the "the child couldn't FUNCTION because its pointer variables
>> all contain parent addresses" problem.
>
> Actually you can implement fork(), if you teach the compiler to use
> separate stacks for return addresses and data:
> - The first stack would contain only absolute addresses, to be
> relocated after copying,
> - The second stack would contain integers and relative pointers
> (see FDPIC below), which do not need relocation after copying.
>
>> The OTHER fun thing about nommu is you can't run conventional ELF binaries,
>> because everything is linked at fixed address. So you might be able to run ONE
>> instance of the program as your init task, assuming those addresses were
>> available even then, but as soon as you try to run a second one it's a conflict.
>>
>> The quick and dirty work around is to make PIE binaries, which can relocate
>> everything into available space, which works but doesn't scale. The problem with
>> ELF PIE is that everything is linked contiguously from a single base pointer,
>> meaning your text, rodata, data, and bss segments are all one linear blob. So if
>> you run two instances of bash, you've loaded two copies of the test and the
>> rodoata. This fills up your memory fast.
>>
>> AND PIE requires contiguous memory, which nommu is bad at providing because it
>> has no page tables to remap stuff. With an mmu it can coalesce scattered
>> physical pages into a virtually contiguous range, but without an mmu you can
>> have plenty of memory free but in tiny chunks, none big enough to satisfy an
>> allocation request.
>>
>> So they invented FDPIC, which is ELF with FOUR base pointers. Each major section
>> (rodata, text, data, and bss) has its own base pointer, so you need to find
>> smaller chunks of memory to load them into (and thus it can work on a more
>> fragmented system), AND it means that two instances of the same program can
>> share the read-only sections (rodata and text) so you only need new copies of
>> the writeable segments (data and bss. And the heap. And the stack.)
>
> Or Amiga LoadSeg() relocatable binaries and shared libraries ;-)
> As this supported splitting code, data, and bss in lots of smaller
> hunks, it could counter fragmented memory quite well.
>
> BTW, can't you run and thus test nommu-binaries under normal Linux, too?

Yes, you can. The flat format loader can be built for MMU arm and m68k Linux.
It will happily load and run flat format binaries on normal VM Linux.
I test that often on m68k (on ColdFire platforms).

Regards
Greg

2024-01-12 20:10:34

[permalink] [raw]

Subject: Re: [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

On 1/10/24 20:25, Greg Ungerer wrote:
> Sorry Rob, but I think you are wrong about a number of things here.

Happy to be corrected. I learned most of this stuff by people pointing things
out I didn't know. But I _have_ been trying to collect this info for about 15
years now...

> I know, I was there at the coal face so to speak during the early
> years of uClinux and right up today. I feel I have to correct some of
> the things you claim.

I only heard about the politics long after the fact, and the stories have a lot
of elisions because what happens in Utah sadly does NOT stay in Utah.

I didn't get involved in busybox until either 2002 or 2003 depending how you
want to count it. After the dot com crash, around the time of the SCO trial.
Erik Andersen having once worked there was about as relevant as Linus' day job
at Transmeta, or busybox having started life years earlier as Debian's answer to
Red Hat Nash before being abandoned for 3 years before Erik picked it up. I
heard that this stuff was used in uClinux, but its use in linksys routers was
far more immediately relevant.

Shortly after the dot-com crash Ray Noorda went senile and got elder abused into
allowing the SCO lawsuits run by the canopy group (his personal money managers,
gone rogue), by which time Erik Andersen and Jeff Dionne and Tim Bird and so on
had all gone on to other employers. Tim heading to Sony, Jeff moved to Japan to
do hardware and leaving uClinux behind, and Erik continued busybox and uclibc as
personal projects from a server connected to the DSL line in his basement.

By the time I showed up uclinux never came up on the mailing list, the IRC
channel, or in private email. I was introduced to busybox+uclibc by tomsrtbt,
and the highest profile consumer of Erik's output was Linksys routers.

My goal with busybox was making it work in a development environment, initially
to replace most of the the 110 megabytes of gnu bloatware in Linux From Scratch
with the 1.7 megabytes of tomsrtbt to free up some of the 700 megabyte budget of
knoppix CDs. Making that actually _work_ took long enough to accomplish that
"knoppix" stopped being "the bootable CD" and every distro had one now (usually
as their install CD: try it live then install from the desktop you'd booted into
if you want to keep it).

After I finally got it all working (https://landley.net/aboriginal/news.html
either the 1.0 release rebuilding itself under itself on September 5, 2010 or
the 1.1 release building Linux From Scratch 6.7 under the result on October 2,
2011 depending how you want to measure), I moved on to focusing on toybox
development instead (trying to get it into Android so THAT could become
self-hosting and provide a real development environment with a USB
keyboard+mouse and chromecast to a big TV), and other people took over creating
Adelie Linux and Alpine Linux and so on based on busybox instead of gnu tools.

I didn't get into nommu development full-time until 2014 when Jeff Dionne hired
me to work on his j-core project. I'd heard the name, but never spoken to him
before he reached out to me by email, then phone interview, then flew me to Japan.

While working for Jeff I asked various computer history questions because it's a
hobby of mine, but there's been a pandemic between then and now. I wasn't there,
the details of the answers I _did_ get are fuzzy by now, and there were some...
stories. Let's just say a canadian who moved to Utah and then moved to LITERALLY
THE OTHER SIDE OF THE PLANET may may have seen some stuff. I've also bumped into
Tim Bird on and off since 2006 (through CELF/ELC and the Linux Jamboree at
Nakano Sun Plaza in Tokyo and
https://elinux.org/CELF_Project_Proposal/Combine_tcg_with_tcc and so on) and
he's never spoken of Lineo once.

The work I did helped kill the old uclinux distro, although I didn't realize it
at the time. The various package patches it had carried were already pushed
upstream into linux and gcc and so on, and busybox worked as a nommu userspace.
By making "build a busybox+uclibc system with a vanilla kernel" easier to do and
more powerful by itself, uclinux became less relevant. Various people sent me
nommu fixes when I was maintaining busybox, which was like sending me fixes for
running busybox on linksys (or openwrt) routers. I didn't have that hardware,
but I knew the _general_ theory and tried not to break it. Don't do unaligned
access, be careful of endianness, and here's what's necessary for nommu:

https://git.busybox.net/busybox/commit/?id=b1b3cee831bc
https://git.busybox.net/busybox/commit/?id=03628c8f75ba
https://git.busybox.net/busybox/commit/?id=b21837714a37

When buildroot metastasized from uclibc's test suite (what packages build under
uclibc? Let's regression test them and slot them together into a testable root
filesystem) into a new distro (the first post to its mailing list in 2006 is
from me because I abused my busybox maintainer root access to the shared server
to set up a third list, because buildroot discussion was smothering uclibc
development on that list), buildroot became the logical base point for nommu
systems. If you wanted a busybox+uclibc root filesystem, buildroot provided an
up to date actively developed one with build recipes for hundreds of packages,
and a busy community where you could ask questions.

Meaning there was already no reason to install uclinux in 2006. It had been
replaced by either a simple busybox+uclibc root filesystem, or buildroot.

So "me noticing this area in 2002", "me accidentally helping finish off uclinux
in passing in 2006", and "me getting involved in nommu development in 2014
because Jeff Dionne flew me to Japan and personally explained it to me" does not
give me any direct experience with what went on at lineo before that, true. Just
like I never visited transmeta.

>> You can't fork() on nommu because copies of the mappings have different
>> addresses, meaning any pointers in the copied mappings would point into the OLD
>> mappings (belonging to the parent process), and fixing them up is 100%
>> equivalent to the "garbage collection in C" problem. (It's AI-complete. Of the
>> C3PO kind, not the "autocorrect with syntax checking" kind.) People get hung up
>> on the "it would be very inefficient to do that because no copy-on-write"
>> problem and miss the "the child couldn't FUNCTION because its pointer variables
>> all contain parent addresses" problem.
>>
>> So instead vfork() creates a child with all the same memory mappings (a bit like
>> a thread) and freezes the parent process until that child discards those
>> mappings, either by calling exec() or _exit(). (At which point the parent gets
>> un-suspended.)
>
> Just to be clear, vfork is not a uClinux invention.

I never said vfork() was a uClinux invention. I'm describing how Linux does
nommu. Doing nommu on linux hasn't had anything to do with uclinux for 20 years.

Confusing "uclinux" with "nommu" is like confusing "knoppix" with "Linux Live
CD". We've moved on a bit since one magic distro pioneered the technique literal
decades ago, there are other options now.

I first looked at uClinux itself in 2015 because Jeff Dionne asked me to see if
there was anything left to salvage out of it, resulting in
https://github.com/landley/toybox/commit/469d7f11b66d and the answer "no, there
isn't" nine years ago.

> It dates way back to the
> early BSD UNIX days.

I know.

> It just so happens that it fits in very nicely with
> the no-MMU model of the world.

Ken and Dennis' ported Unix from a PDP-7 to a PDP-11/20, which didn't have an
MMU. Dennis wrote about it here:

https://www.bell-labs.com/usr/dmr/www/odd.html

The PDP-11/45 the upgraded to did, and the system call sematics changed quite a
bit in those early years before stabilizing with Unix v7 (largely because AT&T
stopped allowing Bell Labs releases to go out to the public after that one, so
v8, v9, and v10 saw almost no use, and then they started over with Plan 9.)

So yeah I knew about BSD coming up with the name, but I thought "vfork()" was
just "what fork() used to do circa Unix v3 or similar"?

I was trying to explain why vfork() is still in use NOW. It's not just nommu
either, it's also useful when forking from heavily threaded processes, because
normal fork() will freeze all the threads in the process causing a latency spike
and potentially quite large allocation because in that case it breaks all
copy-on-write up front rather than trying to untangle the layers of sharing. But
vfork() will only freeze the one thread that called it, and doesn't allocate new
memory before the exec() or _exit() of the new child process. (You may hit a vma
lock if other threads try to tear down the mappings, I haven't tried. But if you
DON'T do something stupid, you don't get a latency spike on the other threads.)

>> So they invented FDPIC,

"They" being the blackfin devs, I think.

>> which is ELF with FOUR base pointers. Each major section
>> (rodata, text, data, and bss) has its own base pointer, so you need to find
>> smaller chunks of memory to load them into (and thus it can work on a more
>> fragmented system), AND it means that two instances of the same program can
>> share the read-only sections (rodata and text) so you only need new copies of
>> the writeable segments (data and bss. And the heap. And the stack.)
>
> It is worth noting that to run PIE style ELF binaries on no-MMU systems you
> actually use the FDPIC ELF loader - not the traditional linux ELF loader (binfmt_elf).

Yup. Although that's mostly because binfmt_elf refuses to build without
CONFIG_MMU due to some stupid #ifdefs and assumptions.

My understanding (mostly from making puppy eyes at Rich Felker) is that the
bimfmt_fdpic loader can load conventional ELF binaries just fine, the way the
ext4 driver can mount ext3 or ext2. Unfortunately, my attempts to build
bimfmt_fdpic on i386 and arm64 and mips and powerpc and so on to TRY that hit
the same "kconfig and the #ifdefs really dowanna allow this" nonsense.

I note that qemu's application emulation handles conventional ELF, static PIE,
and fdpic pretty much interchangeably. THEY didn't see the need to have two
redundant loaders doing the same thing, with only one understanding some fairly
minor extensions. But the linux-kernel community seems to segregate out nommu
support and quarrantine it, for no obvious reason. (And then make the nommu
stuff build ONLY in one mode and the with-mmu stuff build ONLY in the other mode
and NEVER THE TWAIN SHALL MEET, again for no obvious reason.)

> Using this setup is going to become much easier now that uClibc has native
> support for generating a relevant library and ld.so for noMMU linux (certainly
> for m68k, arm and riscv at least) - as of version 1.0.45.

I haven't followed buildroot's uclibc fork since uclibc.org died, I switched to
musl instead.

>> (The earlier binflt format is an a.out variant with 4 base pointers. FDPIC is
>> the ELF version of the same idea. Since a.out went bye-bye binflt is obsolete,
>> but not everybody's moved off it yet because so many nommu people are still
>> using 2.6 or even earlier, and also using gcc 3.x or 2.x toolchains.)
>
> Flat format (binfmt_flat) is in no way related to a.out. It is not derived from
> it and shares nothing with it. A.out being removed from the kernel in no way affects
> flat format. It doesn't magically make it obsolete. It isn't going anywhere.

*shrug* I'll take your word for it.

Back when I tried to make it work circa
https://landley.net/notes-2014.html#07-12-2014 the binflt plumbing was so
bit-rotted that A) there was no official repository for it, B) it was
implemented as a weird post-processing step where you ran an "elf2flt" binary
but it could not actually digest a normal ELF file, instead you made a SPECIAL
elf file with various compiler flags, which the postprocessing tool then
converted to a flat file.

(This is another variant of "embedded devs get something to work and then keep
using the old version FOREVER". I could download 10 year old binflt toolchains
that produced binaries, but nobody seemed to have built a NEW one in years.)

So I shoehorned basic elf2flt support into my build system:

https://github.com/landley/aboriginal/commit/50d139530dd4

And then I asked Yoshinori Sato what version _he'd_ used when adding h8300
support to Linux (since that was nommu) and switched to his version, which he'd
fixed a lot of stuff in and which worked with newer compiler versions and had
commits less than a year old in the tree:

https://github.com/landley/aboriginal/commit/247ee063028f

And around that time I emailed various people who had once been interested in
this, and they started a new binflt tree (merging the various trees I'd found)
and switched buildroot over to use it. But I didn't track it much beyond that
because instead Jeff paid Rich Felker to add fdpic support to musl and we
switched j-core over to that instead.

What little I know about binflt comes from reading the code and from Jeff Dionne
trying to explain it to me back in 2014 and 2015, and mostly he was concerned
that you had to use older versions of gcc (despite sato-san's upgrades) because
current versions of gcc could do some pointer range thing that broke design
assumptions in elf2flt, which apparently weren't fixable because information had
already been discarded when the .o file was generated, so the thing consuming
that output didn't have all the information it needed to link. (You'd have to
ask Jeff, I never quite wrapped my head around what the specific issue was and
it's a decade ago now. My impression was you'd _mostly_ get lucky and it would
usually work with modern gcc, but you rolled the dice with every code change...)

> I don't see how this is related to versions of kernel or gcc or toolchains either.
> Flat format works on every kernel up to todays v6.7. The conversion tool works
> with the latest gcc and binutils (gcc-13.2 and bintuils-2.41 as of today)

Again, I'll take your word for it.

> and many
> versions going back for the best part of 20 years. Sure that elf2flt tool has been
> forked a lot it has a spotty history of getting updated. But, hey, as of today it
> looks pretty up to date across most architectures.

Um, yes. Check your back email for a message from me September 4, 2015, title
"elf2flt upstream repo". You were cc'd on it.

I created a website that tried to act as a uclinux.org replacement for the
social aspect of "where devs can learn about nommu development for linux, and
come together to talk about it":

https://web.archive.org/web/20160425072007/http://nommu.org/

Alas, I was busy with a half-dozen other things and handed it off to "official
web people" when the company got some, and I wasn't supposed to do "community
outreach" anymore because we had specific staff for that now. The discussion on
the mailing list about elf2flt petered out, but it DID result in people setting
up a new repo and merging some forks and buildroot switching over to the new one.

Sadly, archive.org only spidered the TOP page, not the actual months:

https://web.archive.org/web/20160416102445/http://lists.nommu.org/pipermail/nommu/

I can probably fish it out of my old mbox files if anybody cares. But the point
is, elf2flt got maintained again (not by me, I didn't have the expertise), and I
switched over to fdpic anyway.

Alas reliance of fdpic means that trying to do cortex-m in mkroot, I was stuck
using https://github.com/mickael-guene/fdpic_manifest and waiting for it to go
upstream, which is _why_ I was doing the static pie toolchain for that
architecture until such time as gcc+binutils+linux caught up with the group. (I
think I saw arm fdpic support go into the kernel. Rich said that musl should
work, last I asked him. I haven't checked upstream gcc or binutils recently,
mostly because Rich won't update musl-cross-make and redoing my
https://github.com/landley/toybox/blob/master/scripts/mcm-buildall.sh to build
without it is nontrivial. For one thing I don't think
https://github.com/richfelker/musl-cross-make/blob/master/patches/binutils-2.33.1/0001-j2.diff
ever went upstream either...)

>> UCLINUX is a long-dead distro. Linaro died in the dot-com crash and its founder
>> Jeff Dionne moved to Japan for his next gig and never came back. On the way out
>> he handed uclinux off to someone else, who didn't do a lot of work maintaining
>> it. Most of the actual support went "upstream" into various packages (linux and
>> busybox and gcc and so on) before the handoff, so you didn't NEED uclinux anymore.
>
> Why do you keep claiming that "UCLINUX" (why all caps?) is a distro?

Because it was. Full of obsolete crap, which had its last release in 2014.

> That is just not the case. uClinux has been used as an umbrella term for
> patches, tools, packages relating to running Linux on a CPU with no MMU.

Which is why so many people are convinced nommu linux is dead because
http://uclinux.org/ was long-dead before it went down. Because of exactly that
confusion.

> There was a build package called uclinux-dist that you might consider a distro.
> But it has always been cleanly named as "uclinux-dist".

Cleanly. Uh-huh:

https://web.archive.org/web/20150219205915/http://www.uclinux.org/

https://web.archive.org/web/20181202164812/http://www.uclinux.org/description/

Here are two entries from cut and pasted from
https://web.archive.org/web/20181204132729/http://www.uclinux.org/status/

12 April 2000
The release of uClinux version 2.0.38.1pre7 announced today. Download the .diff NOW!

10 January 2000
The latest release of uClinux, 2.0.38.1pre5, is announced today. The uClinux CD
contains the popular uClinux System Builder Kit. This new version supports a
broad spectrum of chip sets. Order the CD!

> For a long time that
> was the goto starting point for anyone wanting to use Linux with no-MMU.
> The collection of patches to the toolchains, kernel, libraries and application
> packages was a pretty high mountain to climb to get started in those early
> days (late 90's and early 2000's). But through the work of a _lot_ of people
> much of that change has made its way into relevant packages across the board
> (from gcc to kernel to creation of uClibc, etc).

It was introduced to me as a distro at the first ELC I attended in 2006. (I
attended as the new busybox maintainer, there were two "uclinux developers" who
wanted to say hi, we spoke for about 30 seconds. One of them wore a t-shirt with
"uclinux" on it?)

As I said, I never used it. I just heard about it. You're the first person to
tell me it _didn't_ consider itself a distro.

> But lets face it once no-MMU support was integrated into mainline linux as off
> v2.5.46 it is really now just "Linux". The no-MMU support outgrew that trade marked
> term. But the name has stubbornly stuck around.

My successor as busybox maintainer was already ripping _uclinux_ #defines out of
that codebase a dozen years ago.

https://git.busybox.net/busybox/commit/?id=153fcaa6c171

>> The real nail in the coffin is the inheritor of uclinux never migrated it off
>> CVS, and then the disk holding the CVS archive crashed with no backup. He came
>> out with a couple more releases after that by monkey-patching the last release's
>> filesystem, but the writing was on the wall and it rolled to a stop.
>
> No, the public facing CVS was never the master sources for the uclinux-dist.
> The public facing CVS was only ever a copy of the tar ball releases.

All I know is I emailed Michael Durrant in February 2015 to see if I could get a
copy of the repo (preparatory for doing my triage of what was actually IN
uclinux for Jeff) and he replied:

> Are you are talking about the dead cvs.uclinux.org (CVS) machine or the
> the uClibc website? (http://www.uclibc.org/)
>
> If so .. that machine is very very dead .. nothing from the harddrive was
> salvageable. I will look and see if we have a CD or DVD backup but it is very
> very doubtful.
>
> I did replace the http://mailman.uclinux.org machine a few years ago after it
> had a catastrophic fail.

I sent a follow-up email but didn't get a further reply.

Thank you for clarifying that the stuff I was interested in was apparently never
tracked under source control in the first place.

> That work was slowing down due to fact that it just wasn't really
> needed any more. There are way more popular build systems (eg build-root)
> for building complete firmware images from scratch.

Agreed, yes.

>> I did a triage of its last release (from 2014) as part of my toybox roadmap:
>
> No, the last release was in 2016, see it archived at:
> https://sourceforge.net/projects/uclinux/files/uClinux%20Stable/

They did another one after my triage then. I hadn't noticed.

In early 2015 I went to
https://web.archive.org/web/20150219205915/http://www.uclinux.org/ which had a
cvs archive link on the left (which was dead, hence the email exchange), a page
on bflt in the same nav bar (first link on that page went to the dead CVS
archive, the beyondlogic page was 404, and vapier's stuff was still there but
several years old, I tracked down the newer forks later), and then the "http
download" link at the end of the list led to
https://web.archive.org/web/20150206052300/http://www.uclinux.org/pub/uClinux/
the newest date in which was 2004.

The resulting first impression was "this project is VERY DEAD". And then my
email exchange with the contact info link got back "cvs archive died with no
backup" and no reply to a follow-up email.

> But that is all kind of archeological now, relegated to history.

Agreed.

> Regards
> Greg

Rob

2024-01-12 20:11:52

[permalink] [raw]

Subject: Re: [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

On 1/11/24 03:21, Niklas Cassel wrote:
>> > s/Linaro/Lineo/
>>
>> Lineo was not founded by Jeff Dionne, see https://en.wikipedia.org/wiki/Lineo
>> for its genisys. Maybe you are thinking of RT-Control.
>
> Yes, Jeff cofounded Rt-Control which later merged with Lineo.
>
> My main point is that Linaro is a completely different company,
> which did not "die in the dot-com crash", like stated above.

Sorry, I typo uclibc/uclinux all the time too. (In extreme cases I swap
busybox/toybox, which is personally embarassing...)

Rob

2024-01-14 13:01:18

[permalink] [raw]

Subject: Re: [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

Hey Rob,

Reading your emails can be exhausting :-)

On 13/1/24 06:16, Rob Landley wrote:
> On 1/10/24 20:25, Greg Ungerer wrote:
>> Sorry Rob, but I think you are wrong about a number of things here.
>
> Happy to be corrected. I learned most of this stuff by people pointing things
> out I didn't know. But I _have_ been trying to collect this info for about 15
> years now...
>
>> I know, I was there at the coal face so to speak during the early
>> years of uClinux and right up today. I feel I have to correct some of
>> the things you claim.
>
> I only heard about the politics long after the fact, and the stories have a lot
> of elisions because what happens in Utah sadly does NOT stay in Utah.

I came to Lineo through the Moreton Bay acquisition, and I worked with
Jeff, Tim and Erik. Actually I had been collaborating with Jeff when still at
RT-control before that as well. Lineo was a wild ride, to say the least.

> I didn't get involved in busybox until either 2002 or 2003 depending how you
> want to count it. After the dot com crash, around the time of the SCO trial.
> Erik Andersen having once worked there was about as relevant as Linus' day job
> at Transmeta, or busybox having started life years earlier as Debian's answer to
> Red Hat Nash before being abandoned for 3 years before Erik picked it up. I
> heard that this stuff was used in uClinux, but its use in linksys routers was
> far more immediately relevant.
>
> Shortly after the dot-com crash Ray Noorda went senile and got elder abused into
> allowing the SCO lawsuits run by the canopy group (his personal money managers,
> gone rogue), by which time Erik Andersen and Jeff Dionne and Tim Bird and so on
> had all gone on to other employers. Tim heading to Sony, Jeff moved to Japan to
> do hardware and leaving uClinux behind, and Erik continued busybox and uclibc as
> personal projects from a server connected to the DSL line in his basement.
>
> By the time I showed up uclinux never came up on the mailing list, the IRC
> channel, or in private email. I was introduced to busybox+uclibc by tomsrtbt,
> and the highest profile consumer of Erik's output was Linksys routers.
>
> My goal with busybox was making it work in a development environment, initially
> to replace most of the the 110 megabytes of gnu bloatware in Linux From Scratch
> with the 1.7 megabytes of tomsrtbt to free up some of the 700 megabyte budget of
> knoppix CDs. Making that actually _work_ took long enough to accomplish that
> "knoppix" stopped being "the bootable CD" and every distro had one now (usually
> as their install CD: try it live then install from the desktop you'd booted into
> if you want to keep it).
>
> After I finally got it all working (https://landley.net/aboriginal/news.html
> either the 1.0 release rebuilding itself under itself on September 5, 2010 or
> the 1.1 release building Linux From Scratch 6.7 under the result on October 2,
> 2011 depending how you want to measure), I moved on to focusing on toybox
> development instead (trying to get it into Android so THAT could become
> self-hosting and provide a real development environment with a USB
> keyboard+mouse and chromecast to a big TV), and other people took over creating
> Adelie Linux and Alpine Linux and so on based on busybox instead of gnu tools.
>
> I didn't get into nommu development full-time until 2014 when Jeff Dionne hired
> me to work on his j-core project. I'd heard the name, but never spoken to him
> before he reached out to me by email, then phone interview, then flew me to Japan.
>
> While working for Jeff I asked various computer history questions because it's a
> hobby of mine, but there's been a pandemic between then and now. I wasn't there,
> the details of the answers I _did_ get are fuzzy by now, and there were some...
> stories. Let's just say a canadian who moved to Utah and then moved to LITERALLY
> THE OTHER SIDE OF THE PLANET may may have seen some stuff. I've also bumped into
> Tim Bird on and off since 2006 (through CELF/ELC and the Linux Jamboree at
> Nakano Sun Plaza in Tokyo and
> https://elinux.org/CELF_Project_Proposal/Combine_tcg_with_tcc and so on) and
> he's never spoken of Lineo once.
>
> The work I did helped kill the old uclinux distro, although I didn't realize it

I think you are giving yourself a little bit too much credit here...

> at the time. The various package patches it had carried were already pushed
> upstream into linux and gcc and so on, and busybox worked as a nommu userspace.
> By making "build a busybox+uclibc system with a vanilla kernel" easier to do and
> more powerful by itself, uclinux became less relevant. Various people sent me
> nommu fixes when I was maintaining busybox, which was like sending me fixes for
> running busybox on linksys (or openwrt) routers. I didn't have that hardware,
> but I knew the _general_ theory and tried not to break it. Don't do unaligned
> access, be careful of endianness, and here's what's necessary for nommu:

Surely this is the end goal though?
Build systems should not have to carry patches in a perfect world.

> https://git.busybox.net/busybox/commit/?id=b1b3cee831bc
> https://git.busybox.net/busybox/commit/?id=03628c8f75ba
> https://git.busybox.net/busybox/commit/?id=b21837714a37
>
> When buildroot metastasized from uclibc's test suite (what packages build under
> uclibc? Let's regression test them and slot them together into a testable root
> filesystem) into a new distro (the first post to its mailing list in 2006 is
> from me because I abused my busybox maintainer root access to the shared server
> to set up a third list, because buildroot discussion was smothering uclibc
> development on that list), buildroot became the logical base point for nommu
> systems. If you wanted a busybox+uclibc root filesystem, buildroot provided an
> up to date actively developed one with build recipes for hundreds of packages,
> and a busy community where you could ask questions.
>
> Meaning there was already no reason to install uclinux in 2006. It had been
> replaced by either a simple busybox+uclibc root filesystem, or buildroot.
>
> So "me noticing this area in 2002", "me accidentally helping finish off uclinux
> in passing in 2006", and "me getting involved in nommu development in 2014
> because Jeff Dionne flew me to Japan and personally explained it to me" does not
> give me any direct experience with what went on at lineo before that, true. Just
> like I never visited transmeta.
>
>>> You can't fork() on nommu because copies of the mappings have different
>>> addresses, meaning any pointers in the copied mappings would point into the OLD
>>> mappings (belonging to the parent process), and fixing them up is 100%
>>> equivalent to the "garbage collection in C" problem. (It's AI-complete. Of the
>>> C3PO kind, not the "autocorrect with syntax checking" kind.) People get hung up
>>> on the "it would be very inefficient to do that because no copy-on-write"
>>> problem and miss the "the child couldn't FUNCTION because its pointer variables
>>> all contain parent addresses" problem.
>>>
>>> So instead vfork() creates a child with all the same memory mappings (a bit like
>>> a thread) and freezes the parent process until that child discards those
>>> mappings, either by calling exec() or _exit(). (At which point the parent gets
>>> un-suspended.)
>>
>> Just to be clear, vfork is not a uClinux invention.
>
> I never said vfork() was a uClinux invention. I'm describing how Linux does
> nommu. Doing nommu on linux hasn't had anything to do with uclinux for 20 years.
>
> Confusing "uclinux" with "nommu" is like confusing "knoppix" with "Linux Live
> CD". We've moved on a bit since one magic distro pioneered the technique literal
> decades ago, there are other options now.
>
> I first looked at uClinux itself in 2015 because Jeff Dionne asked me to see if
> there was anything left to salvage out of it, resulting in
> https://github.com/landley/toybox/commit/469d7f11b66d and the answer "no, there
> isn't" nine years ago.
>
>> It dates way back to the
>> early BSD UNIX days.
>
> I know.
>
>> It just so happens that it fits in very nicely with
>> the no-MMU model of the world.
>
> Ken and Dennis' ported Unix from a PDP-7 to a PDP-11/20, which didn't have an
> MMU. Dennis wrote about it here:
>
> https://www.bell-labs.com/usr/dmr/www/odd.html
>
> The PDP-11/45 the upgraded to did, and the system call sematics changed quite a
> bit in those early years before stabilizing with Unix v7 (largely because AT&T
> stopped allowing Bell Labs releases to go out to the public after that one, so
> v8, v9, and v10 saw almost no use, and then they started over with Plan 9.)
>
> So yeah I knew about BSD coming up with the name, but I thought "vfork()" was
> just "what fork() used to do circa Unix v3 or similar"?
>
> I was trying to explain why vfork() is still in use NOW. It's not just nommu
> either, it's also useful when forking from heavily threaded processes, because
> normal fork() will freeze all the threads in the process causing a latency spike
> and potentially quite large allocation because in that case it breaks all
> copy-on-write up front rather than trying to untangle the layers of sharing. But
> vfork() will only freeze the one thread that called it, and doesn't allocate new
> memory before the exec() or _exit() of the new child process. (You may hit a vma
> lock if other threads try to tear down the mappings, I haven't tried. But if you
> DON'T do something stupid, you don't get a latency spike on the other threads.)
>
>>> So they invented FDPIC,
>
> "They" being the blackfin devs, I think.

No, I believe it was David Howells that did it originally for the Fujitsu FRV architecture.

>>> which is ELF with FOUR base pointers. Each major section
>>> (rodata, text, data, and bss) has its own base pointer, so you need to find
>>> smaller chunks of memory to load them into (and thus it can work on a more
>>> fragmented system), AND it means that two instances of the same program can
>>> share the read-only sections (rodata and text) so you only need new copies of
>>> the writeable segments (data and bss. And the heap. And the stack.)
>>
>> It is worth noting that to run PIE style ELF binaries on no-MMU systems you
>> actually use the FDPIC ELF loader - not the traditional linux ELF loader (binfmt_elf).
>
> Yup. Although that's mostly because binfmt_elf refuses to build without
> CONFIG_MMU due to some stupid #ifdefs and assumptions.
>
> My understanding (mostly from making puppy eyes at Rich Felker) is that the
> bimfmt_fdpic loader can load conventional ELF binaries just fine, the way the
> ext4 driver can mount ext3 or ext2. Unfortunately, my attempts to build
> bimfmt_fdpic on i386 and arm64 and mips and powerpc and so on to TRY that hit
> the same "kconfig and the #ifdefs really dowanna allow this" nonsense.
>
> I note that qemu's application emulation handles conventional ELF, static PIE,
> and fdpic pretty much interchangeably. THEY didn't see the need to have two
> redundant loaders doing the same thing, with only one understanding some fairly
> minor extensions. But the linux-kernel community seems to segregate out nommu
> support and quarrantine it, for no obvious reason. (And then make the nommu
> stuff build ONLY in one mode and the with-mmu stuff build ONLY in the other mode
> and NEVER THE TWAIN SHALL MEET, again for no obvious reason.)>
>> Using this setup is going to become much easier now that uClibc has native
>> support for generating a relevant library and ld.so for noMMU linux (certainly
>> for m68k, arm and riscv at least) - as of version 1.0.45.
>
> I haven't followed buildroot's uclibc fork since uclibc.org died, I switched to
> musl instead.
>
>>> (The earlier binflt format is an a.out variant with 4 base pointers. FDPIC is
>>> the ELF version of the same idea. Since a.out went bye-bye binflt is obsolete,
>>> but not everybody's moved off it yet because so many nommu people are still
>>> using 2.6 or even earlier, and also using gcc 3.x or 2.x toolchains.)
>>
>> Flat format (binfmt_flat) is in no way related to a.out. It is not derived from
>> it and shares nothing with it. A.out being removed from the kernel in no way affects
>> flat format. It doesn't magically make it obsolete. It isn't going anywhere.
>
> *shrug* I'll take your word for it.
>
> Back when I tried to make it work circa
> https://landley.net/notes-2014.html#07-12-2014 the binflt plumbing was so
> bit-rotted that A) there was no official repository for it, B) it was
> implemented as a weird post-processing step where you ran an "elf2flt" binary
> but it could not actually digest a normal ELF file, instead you made a SPECIAL
> elf file with various compiler flags, which the postprocessing tool then
> converted to a flat file.
>
> (This is another variant of "embedded devs get something to work and then keep
> using the old version FOREVER". I could download 10 year old binflt toolchains
> that produced binaries, but nobody seemed to have built a NEW one in years.)
>
> So I shoehorned basic elf2flt support into my build system:
>
> https://github.com/landley/aboriginal/commit/50d139530dd4
>
> And then I asked Yoshinori Sato what version _he'd_ used when adding h8300
> support to Linux (since that was nommu) and switched to his version, which he'd
> fixed a lot of stuff in and which worked with newer compiler versions and had
> commits less than a year old in the tree:
>
> https://github.com/landley/aboriginal/commit/247ee063028f
>
> And around that time I emailed various people who had once been interested in
> this, and they started a new binflt tree (merging the various trees I'd found)
> and switched buildroot over to use it. But I didn't track it much beyond that
> because instead Jeff paid Rich Felker to add fdpic support to musl and we
> switched j-core over to that instead.
>
> What little I know about binflt comes from reading the code and from Jeff Dionne
> trying to explain it to me back in 2014 and 2015, and mostly he was concerned
> that you had to use older versions of gcc (despite sato-san's upgrades) because
> current versions of gcc could do some pointer range thing that broke design
> assumptions in elf2flt, which apparently weren't fixable because information had
> already been discarded when the .o file was generated, so the thing consuming
> that output didn't have all the information it needed to link. (You'd have to
> ask Jeff, I never quite wrapped my head around what the specific issue was and
> it's a decade ago now. My impression was you'd _mostly_ get lucky and it would
> usually work with modern gcc, but you rolled the dice with every code change...)
>
>> I don't see how this is related to versions of kernel or gcc or toolchains either.
>> Flat format works on every kernel up to todays v6.7. The conversion tool works
>> with the latest gcc and binutils (gcc-13.2 and bintuils-2.41 as of today)
>
> Again, I'll take your word for it.
>
>> and many
>> versions going back for the best part of 20 years. Sure that elf2flt tool has been
>> forked a lot it has a spotty history of getting updated. But, hey, as of today it
>> looks pretty up to date across most architectures.
>
> Um, yes. Check your back email for a message from me September 4, 2015, title
> "elf2flt upstream repo". You were cc'd on it.
>
> I created a website that tried to act as a uclinux.org replacement for the
> social aspect of "where devs can learn about nommu development for linux, and
> come together to talk about it":
>
> https://web.archive.org/web/20160425072007/http://nommu.org/
>
> Alas, I was busy with a half-dozen other things and handed it off to "official
> web people" when the company got some, and I wasn't supposed to do "community
> outreach" anymore because we had specific staff for that now. The discussion on
> the mailing list about elf2flt petered out, but it DID result in people setting
> up a new repo and merging some forks and buildroot switching over to the new one.
>
> Sadly, archive.org only spidered the TOP page, not the actual months:
>
> https://web.archive.org/web/20160416102445/http://lists.nommu.org/pipermail/nommu/
>
> I can probably fish it out of my old mbox files if anybody cares. But the point
> is, elf2flt got maintained again (not by me, I didn't have the expertise), and I
> switched over to fdpic anyway.

My point is that the elf2flt tool was very widely forked. At the very least
pretty much every architecture port has its own, and in many cases many vendors
had their own as well. So, yeah, its development and progress got messy.

But I did get involved in that effort to get elf2flt centered and restored
to some semblance of being maintained. The git hub repository is now the main focus,
https://github.com/uclinux-dev/elf2flt. It is maintained and pretty much up to
date. So I think we can notch that up as a win.

Of course the flat format itself has issues, it is far from perfect. But for the
most part it works fine for its intended purpose. elf-fdpic is superior in many
respects, but requires much more invasive compiler support - that takes a lot
more effort to create for a new architecture.

> Alas reliance of fdpic means that trying to do cortex-m in mkroot, I was stuck
> using https://github.com/mickael-guene/fdpic_manifest and waiting for it to go
> upstream, which is _why_ I was doing the static pie toolchain for that
> architecture until such time as gcc+binutils+linux caught up with the group. (I
> think I saw arm fdpic support go into the kernel. Rich said that musl should
> work, last I asked him. I haven't checked upstream gcc or binutils recently,
> mostly because Rich won't update musl-cross-make and redoing my
> https://github.com/landley/toybox/blob/master/scripts/mcm-buildall.sh to build
> without it is nontrivial. For one thing I don't think
> https://github.com/richfelker/musl-cross-make/blob/master/patches/binutils-2.33.1/0001-j2.diff
> ever went upstream either...)

Elf-fdpic on ARM works fine, I test it regularly. You don't need any patches
for current versions of binutils, gcc, or linux kernel. Oh, I test with uClibc-ng,
don't know about other libraries.

And PIE style ELF works at least on ARM, M68K, RISCV and XTENSA nommu configurations.
Again with uClibc-ng at least.

>>> UCLINUX is a long-dead distro. Linaro died in the dot-com crash and its founder
>>> Jeff Dionne moved to Japan for his next gig and never came back. On the way out
>>> he handed uclinux off to someone else, who didn't do a lot of work maintaining
>>> it. Most of the actual support went "upstream" into various packages (linux and
>>> busybox and gcc and so on) before the handoff, so you didn't NEED uclinux anymore.
>>
>> Why do you keep claiming that "UCLINUX" (why all caps?) is a distro?
>
> Because it was. Full of obsolete crap, which had its last release in 2014.
>
>> That is just not the case. uClinux has been used as an umbrella term for
>> patches, tools, packages relating to running Linux on a CPU with no MMU.
>
> Which is why so many people are convinced nommu linux is dead because
> http://uclinux.org/ was long-dead before it went down. Because of exactly that
> confusion.

I am very confused by this. Who thinks nommu linux is dead?

I am continually surprised by how many people still use it.

>> There was a build package called uclinux-dist that you might consider a distro.
>> But it has always been cleanly named as "uclinux-dist".
>
> Cleanly. Uh-huh:
>
> https://web.archive.org/web/20150219205915/http://www.uclinux.org/
>
> https://web.archive.org/web/20181202164812/http://www.uclinux.org/description/
>
> Here are two entries from cut and pasted from
> https://web.archive.org/web/20181204132729/http://www.uclinux.org/status/
>
> 12 April 2000
> The release of uClinux version 2.0.38.1pre7 announced today. Download the .diff NOW!
>
> 10 January 2000
> The latest release of uClinux, 2.0.38.1pre5, is announced today. The uClinux CD
> contains the popular uClinux System Builder Kit. This new version supports a
> broad spectrum of chip sets. Order the CD!

More than anything they point out the continual overloading of the term and
general confusion on what the term actually covers.

It is trivial to find examples of its use that are clearly not related to a distribution.
Grep for it in the Linux kernel sources one, there is plenty, eg:

mm/nommu.c: * handle mapping creation for uClinux
arch/m68k/include/asm/flat.h: * flat.h -- uClinux flat-format executables
fs/Kconfig.binfmt: Support uClinux FLAT format binaries.
fs/binfmt_elf_fdpic.c: * - on uClinux the holes between may actually be filled with system
..

Despite what you think my experience is that most people simply equate
uClinux with no-mmu linux. And I for one am pretty comfortable with that.

>> For a long time that
>> was the goto starting point for anyone wanting to use Linux with no-MMU.
>> The collection of patches to the toolchains, kernel, libraries and application
>> packages was a pretty high mountain to climb to get started in those early
>> days (late 90's and early 2000's). But through the work of a _lot_ of people
>> much of that change has made its way into relevant packages across the board
>> (from gcc to kernel to creation of uClibc, etc).
>
> It was introduced to me as a distro at the first ELC I attended in 2006. (I
> attended as the new busybox maintainer, there were two "uclinux developers" who
> wanted to say hi, we spoke for about 30 seconds. One of them wore a t-shirt with
> "uclinux" on it?)
>
> As I said, I never used it. I just heard about it. You're the first person to
> tell me it _didn't_ consider itself a distro.
>
>> But lets face it once no-MMU support was integrated into mainline linux as off
>> v2.5.46 it is really now just "Linux". The no-MMU support outgrew that trade marked
>> term. But the name has stubbornly stuck around.
>
> My successor as busybox maintainer was already ripping _uclinux_ #defines out of
> that codebase a dozen years ago.
>
> https://git.busybox.net/busybox/commit/?id=153fcaa6c171
>
>>> The real nail in the coffin is the inheritor of uclinux never migrated it off
>>> CVS, and then the disk holding the CVS archive crashed with no backup. He came
>>> out with a couple more releases after that by monkey-patching the last release's
>>> filesystem, but the writing was on the wall and it rolled to a stop.
>>
>> No, the public facing CVS was never the master sources for the uclinux-dist.
>> The public facing CVS was only ever a copy of the tar ball releases.
>
> All I know is I emailed Michael Durrant in February 2015 to see if I could get a
> copy of the repo (preparatory for doing my triage of what was actually IN
> uclinux for Jeff) and he replied:
>
>> Are you are talking about the dead cvs.uclinux.org (CVS) machine or the
>> the uClibc website? (http://www.uclibc.org/)
>>
>> If so .. that machine is very very dead .. nothing from the harddrive was
>> salvageable. I will look and see if we have a CD or DVD backup but it is very
>> very doubtful.
>>
>> I did replace the http://mailman.uclinux.org machine a few years ago after it
>> had a catastrophic fail.
>
> I sent a follow-up email but didn't get a further reply.
>
> Thank you for clarifying that the stuff I was interested in was apparently never
> tracked under source control in the first place.

I didn't say it was not tracked under source control, it was.
But that was an internal company vendor tree. The external public facing CVS
tree on uclinux.org was simply not a copy of that internal working tree.
It was not much more than a dump of the tar ball releases (and occasionally
direct fix updates).

>> That work was slowing down due to fact that it just wasn't really
>> needed any more. There are way more popular build systems (eg build-root)
>> for building complete firmware images from scratch.
>
> Agreed, yes.
>
>>> I did a triage of its last release (from 2014) as part of my toybox roadmap:
>>
>> No, the last release was in 2016, see it archived at:
>> https://sourceforge.net/projects/uclinux/files/uClinux%20Stable/
>
> They did another one after my triage then. I hadn't noticed.
>
> In early 2015 I went to
> https://web.archive.org/web/20150219205915/http://www.uclinux.org/ which had a
> cvs archive link on the left (which was dead, hence the email exchange), a page
> on bflt in the same nav bar (first link on that page went to the dead CVS
> archive, the beyondlogic page was 404, and vapier's stuff was still there but
> several years old, I tracked down the newer forks later), and then the "http
> download" link at the end of the list led to
> https://web.archive.org/web/20150206052300/http://www.uclinux.org/pub/uClinux/
> the newest date in which was 2004.
>
> The resulting first impression was "this project is VERY DEAD". And then my
> email exchange with the contact info link got back "cvs archive died with no
> backup" and no reply to a follow-up email.
>
>> But that is all kind of archeological now, relegated to history.
>
> Agreed.
>
>> Regards
>> Greg
>
> Rob

Regards
Greg

2024-01-15 13:50:21

by Waldemar Brodkorb

[permalink] [raw]

Subject: Re: [Buildroot] [Automated-testing] Call for nommu LTP maintainer [was: Re: [PATCH 00/36] Remove UCLINUX from LTP]

Hi,

I want to clarify some things of the point of view of uClibc-ng
support.

There is support for following configurations for noMMU targets:
ARM:
- FLAT with Linuxthreads supported (for Qemu you need a Linux patch)
- FDPIC with NPTL supported (NPTL works only on real hardware not in Qemu)
- ELF with Thread support not working

M68k:
- FLAT with Linuxthreads supported
- ELF with Thread support not working

RISCV64:
- FLAT without Thread support
- ELF with Thread support not working

RISCV32:
- FLAT without Thread support, needs a small Linux Kernel patch

SH2/J2:
- FLAT with Linuxthreads supported

Xtensa:
- FLAT with Linuxthreads supported

There are some obsolete architectures supported by uClibc-ng, but
no longer supported by Linux:

Blackfin:
- FLAT with Linuxthreads supported
- FDPIC

H8300:
- FLAT with Linuxthreads supported

C6X:
- DSBT

LM32:
- FLAT

LTP requires NPTL to work, so the only testable platform is ARM with
FDPIC right now.
Unfortunately LTP 20230929 needs fork for some files:

RANLIB libltp.a
/home/wbx/arm/toolchain_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/usr/lib/gcc/arm-openadk-uclinuxfdpiceabi/13.2.0/../../../../arm-openadk-uclinuxfdpiceabi/bin/ld: ../../lib/libltp.a(tst_res.o): in function `tst_fork':
/home/wbx/arm/build_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/w-ltp-20230929-1/ltp-full-20230929/lib/tst_res.c:430:(.text+0x952): undefined reference to `fork'
/home/wbx/arm/toolchain_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/usr/lib/gcc/arm-openadk-uclinuxfdpiceabi/13.2.0/../../../../arm-openadk-uclinuxfdpiceabi/bin/ld: ../../lib/libltp.a(tst_test.o): in function `fork_testrun':
/home/wbx/arm/build_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/w-ltp-20230929-1/ltp-full-20230929/lib/tst_test.c:1597:(.text+0xf4e): undefined reference to `fork'
/home/wbx/arm/toolchain_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/usr/lib/gcc/arm-openadk-uclinuxfdpiceabi/13.2.0/../../../../arm-openadk-uclinuxfdpiceabi/bin/ld: ../../lib/libltp.a(tst_test.o): in function `safe_fork':
/home/wbx/arm/build_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/w-ltp-20230929-1/ltp-full-20230929/lib/tst_test.c:435:(.text+0x345c): undefined reference to `fork'
collect2: error: ld returned 1 exit status
gmake[8]: *** [../../include/mk/rules.mk:45: test01] Error 1
gmake[7]: *** [../include/mk/generic_trunk_target.inc:108: all] Error 2
gmake[6]: *** [Makefile:94: lib-all] Error 2
gmake[5]: *** [/home/wbx/arm/mk/pkg-bottom.mk:141: /home/wbx/arm/build_st-stm32f746g_uclibc-ng_cortex_m7_soft_eabi_thumb_nommu/w-ltp-20230929-1/ltp-full-20230929/.build_done] Error 2
gmake[4]: *** [Makefile:61: ltp-compile] Error 2
gmake[3]: *** [mk/build.mk:221: package/compile] Error 2
gmake[2]: *** [/home/wbx/arm/mk/build.mk:176: world] Error 2

So there is really work to be done to make the existing code work on noMMU.

best regards
Waldemar

2024-01-15 14:23:38