Hi all,
We have several embedded systems where pass console= or console=null
in production to disable the console.
Later we check for this in user space: in our inittab we check if fd0
is "associated with a terminal" (test -t 0); if so, we are in
development mode and we open a debug shell; otherwise (console
disabled) we just start the application.
Recently [1] this behaviour has changed and now if we pass console= or
console=null, the new ttynull driver is used. This breaks the check we
were doing (test -t 0 always true now).
Is there a way to get the previous behaviour? If not, is there an easy
way for userspace to detect whether the console device is a "real" tty
or ttynull (other than trying to parse the kernel boot args, which
would be a bit fragile).
Thank you,
(If possible, please CC me in any replies)
[1]: https://lore.kernel.org/lkml/X%2FcDG%2FxCCzSWW2cd@alley/t/
--
Guillermo Rodriguez Garcia
[email protected]
On 3/16/23 03:29, Guillermo Rodriguez Garcia wrote:
> Hi all,
>
> We have several embedded systems where pass console= or console=null
> in production to disable the console.
>
> Later we check for this in user space: in our inittab we check if fd0
> is "associated with a terminal" (test -t 0); if so, we are in
> development mode and we open a debug shell; otherwise (console
> disabled) we just start the application.
>
> Recently [1] this behaviour has changed and now if we pass console= or
> console=null, the new ttynull driver is used. This breaks the check we
> were doing (test -t 0 always true now).
>
> Is there a way to get the previous behaviour? If not, is there an easy
> way for userspace to detect whether the console device is a "real" tty
> or ttynull (other than trying to parse the kernel boot args, which
> would be a bit fragile).
>
> Thank you,
>
> (If possible, please CC me in any replies)
>
> [1]: https://lore.kernel.org/lkml/X%2FcDG%2FxCCzSWW2cd@alley/t/
>
Let me know if/when you find a solution. In ChromeOS we have to carry
reverts of commit 48021f981308 ("printk: handle blank console arguments
passed in.") and commit 3cffa06aeef7 ("printk/console: Allow to disable
console output by using console="" or console=null") to handle the
same problem (the above mentioned commit didn't work and had odd side
effects).
Guenter
On Thu 2023-03-16 11:29:26, Guillermo Rodriguez Garcia wrote:
> Hi all,
>
> We have several embedded systems where pass console= or console=null
> in production to disable the console.
>
> Later we check for this in user space: in our inittab we check if fd0
> is "associated with a terminal" (test -t 0); if so, we are in
> development mode and we open a debug shell; otherwise (console
> disabled) we just start the application.
>
> Recently [1] this behaviour has changed and now if we pass console= or
> console=null, the new ttynull driver is used. This breaks the check we
> were doing (test -t 0 always true now).
>
> [1]: https://lore.kernel.org/lkml/X%2FcDG%2FxCCzSWW2cd@alley/t/
This is actually exactly the problem that the change tried to solve.
Some systems failed to boot when there was no console and they tried
to write something at stdout.
> Is there a way to get the previous behaviour? If not, is there an easy
> way for userspace to detect whether the console device is a "real" tty
> or ttynull (other than trying to parse the kernel boot args, which
> would be a bit fragile).
A solution would be to check that /proc/consoles has ttynull as the
only registred console, for example:
grep -q ttynull /proc/consoles && test `cat /proc/consoles | wc -l` -eq 1
Would this work for you, please?
Best Regards,
Petr
El jue, 16 mar 2023 a las 14:40, Guenter Roeck (<[email protected]>) escribió:
>
> On 3/16/23 03:29, Guillermo Rodriguez Garcia wrote:
> > Hi all,
> >
> > We have several embedded systems where pass console= or console=null
> > in production to disable the console.
> >
> > Later we check for this in user space: in our inittab we check if fd0
> > is "associated with a terminal" (test -t 0); if so, we are in
> > development mode and we open a debug shell; otherwise (console
> > disabled) we just start the application.
> >
> > Recently [1] this behaviour has changed and now if we pass console= or
> > console=null, the new ttynull driver is used. This breaks the check we
> > were doing (test -t 0 always true now).
> >
> > Is there a way to get the previous behaviour? If not, is there an easy
> > way for userspace to detect whether the console device is a "real" tty
> > or ttynull (other than trying to parse the kernel boot args, which
> > would be a bit fragile).
> >
> > Thank you,
> >
> > (If possible, please CC me in any replies)
> >
> > [1]: https://lore.kernel.org/lkml/X%2FcDG%2FxCCzSWW2cd@alley/t/
> >
>
> Let me know if/when you find a solution. In ChromeOS we have to carry
> reverts of commit 48021f981308 ("printk: handle blank console arguments
> passed in.") and commit 3cffa06aeef7 ("printk/console: Allow to disable
> console output by using console="" or console=null") to handle the
> same problem (the above mentioned commit didn't work and had odd side
> effects).
The kernel seems to specifically check for "" (empty string) or "null"
and in this case uses the ttynull driver. But apparently we can use
any other invalid string (e.g "invalid") to get the previous behaviour
(no console).
I am still checking if this has other side effects.
Best regards,
Guillermo Rodriguez Garcia
[email protected]
Hi Petr,
El vie, 17 mar 2023 a las 13:51, Petr Mladek (<[email protected]>) escribió:
>
> On Thu 2023-03-16 11:29:26, Guillermo Rodriguez Garcia wrote:
> > Hi all,
> >
> > We have several embedded systems where pass console= or console=null
> > in production to disable the console.
> >
> > Later we check for this in user space: in our inittab we check if fd0
> > is "associated with a terminal" (test -t 0); if so, we are in
> > development mode and we open a debug shell; otherwise (console
> > disabled) we just start the application.
> >
> > Recently [1] this behaviour has changed and now if we pass console= or
> > console=null, the new ttynull driver is used. This breaks the check we
> > were doing (test -t 0 always true now).
> >
> > [1]: https://lore.kernel.org/lkml/X%2FcDG%2FxCCzSWW2cd@alley/t/
>
> This is actually exactly the problem that the change tried to solve.
> Some systems failed to boot when there was no console and they tried
> to write something at stdout.
Well, I did not have any problem before this change.
I understand that some systems had a problem (many others didn't, and
setting console= or console=null has been standard practice for a long
time). Since this change in behaviour could (and did) break things in
user space, perhaps it should have been made opt-in...
>
> > Is there a way to get the previous behaviour? If not, is there an easy
> > way for userspace to detect whether the console device is a "real" tty
> > or ttynull (other than trying to parse the kernel boot args, which
> > would be a bit fragile).
>
> A solution would be to check that /proc/consoles has ttynull as the
> only registred console, for example:
>
> grep -q ttynull /proc/consoles && test `cat /proc/consoles | wc -l` -eq 1
>
> Would this work for you, please?
I was trying to avoid something like this as it feels like userspace
now needs to have too much knowledge of what the kernel is doing
internally, however I guess this is the best option.
The suggested check seems to work but now I am seeing a different
(related?) issue: when I try to reboot from the console (using
busybox's reboot command), the system reboots normally if I am using a
"normal" console (e.g. console=ttyXXX), however when using
console=ttynull, the command takes 20-25 seconds to complete. I am not
sure why this would happen; if I understand correctly, for userspace
ttynull is just like a regular tty driver, so why would there be a
difference?
Any ideas?
Thanks,
Guillermo Rodriguez Garcia
[email protected]
On Wed, Mar 22, 2023 at 09:23:28AM +0100, Guillermo Rodriguez Garcia wrote:
> El jue, 16 mar 2023 a las 14:40, Guenter Roeck (<[email protected]>) escribi?:
> >
> > On 3/16/23 03:29, Guillermo Rodriguez Garcia wrote:
> > > Hi all,
> > >
> > > We have several embedded systems where pass console= or console=null
> > > in production to disable the console.
> > >
> > > Later we check for this in user space: in our inittab we check if fd0
> > > is "associated with a terminal" (test -t 0); if so, we are in
> > > development mode and we open a debug shell; otherwise (console
> > > disabled) we just start the application.
> > >
> > > Recently [1] this behaviour has changed and now if we pass console= or
> > > console=null, the new ttynull driver is used. This breaks the check we
> > > were doing (test -t 0 always true now).
> > >
> > > Is there a way to get the previous behaviour? If not, is there an easy
> > > way for userspace to detect whether the console device is a "real" tty
> > > or ttynull (other than trying to parse the kernel boot args, which
> > > would be a bit fragile).
> > >
> > > Thank you,
> > >
> > > (If possible, please CC me in any replies)
> > >
> > > [1]: https://lore.kernel.org/lkml/X%2FcDG%2FxCCzSWW2cd@alley/t/
> > >
> >
> > Let me know if/when you find a solution. In ChromeOS we have to carry
> > reverts of commit 48021f981308 ("printk: handle blank console arguments
> > passed in.") and commit 3cffa06aeef7 ("printk/console: Allow to disable
> > console output by using console="" or console=null") to handle the
> > same problem (the above mentioned commit didn't work and had odd side
> > effects).
>
> The kernel seems to specifically check for "" (empty string) or "null"
> and in this case uses the ttynull driver. But apparently we can use
> any other invalid string (e.g "invalid") to get the previous behaviour
> (no console).
>
> I am still checking if this has other side effects.
>
Interesting. I'll check if that works for ChromeOS; that would be better
than having to carry the reverts.
Thanks,
Guenter
> Best regards,
>
> Guillermo Rodriguez Garcia
> [email protected]
On Wed, Mar 22, 2023 at 09:33:32AM +0100, Guillermo Rodriguez Garcia wrote:
> Hi Petr,
>
> El vie, 17 mar 2023 a las 13:51, Petr Mladek (<[email protected]>) escribi?:
> >
> > On Thu 2023-03-16 11:29:26, Guillermo Rodriguez Garcia wrote:
> > > Hi all,
> > >
> > > We have several embedded systems where pass console= or console=null
> > > in production to disable the console.
> > >
> > > Later we check for this in user space: in our inittab we check if fd0
> > > is "associated with a terminal" (test -t 0); if so, we are in
> > > development mode and we open a debug shell; otherwise (console
> > > disabled) we just start the application.
> > >
> > > Recently [1] this behaviour has changed and now if we pass console= or
> > > console=null, the new ttynull driver is used. This breaks the check we
> > > were doing (test -t 0 always true now).
> > >
> > > [1]: https://lore.kernel.org/lkml/X%2FcDG%2FxCCzSWW2cd@alley/t/
> >
> > This is actually exactly the problem that the change tried to solve.
> > Some systems failed to boot when there was no console and they tried
> > to write something at stdout.
>
> Well, I did not have any problem before this change.
>
> I understand that some systems had a problem (many others didn't, and
> setting console= or console=null has been standard practice for a long
> time). Since this change in behaviour could (and did) break things in
> user space, perhaps it should have been made opt-in...
>
> >
> > > Is there a way to get the previous behaviour? If not, is there an easy
> > > way for userspace to detect whether the console device is a "real" tty
> > > or ttynull (other than trying to parse the kernel boot args, which
> > > would be a bit fragile).
> >
> > A solution would be to check that /proc/consoles has ttynull as the
> > only registred console, for example:
> >
> > grep -q ttynull /proc/consoles && test `cat /proc/consoles | wc -l` -eq 1
> >
> > Would this work for you, please?
>
> I was trying to avoid something like this as it feels like userspace
> now needs to have too much knowledge of what the kernel is doing
> internally, however I guess this is the best option.
>
> The suggested check seems to work but now I am seeing a different
> (related?) issue: when I try to reboot from the console (using
> busybox's reboot command), the system reboots normally if I am using a
> "normal" console (e.g. console=ttyXXX), however when using
> console=ttynull, the command takes 20-25 seconds to complete. I am not
> sure why this would happen; if I understand correctly, for userspace
> ttynull is just like a regular tty driver, so why would there be a
> difference?
>
I think we may have seen something similar, causing us to revert the
upstream patches in ChromeOS. We did not try to find out what exactly
caused the problems.
Guenter
El mié, 22 mar 2023 a las 16:05, Guenter Roeck (<[email protected]>) escribió:
>
> On Wed, Mar 22, 2023 at 09:33:32AM +0100, Guillermo Rodriguez Garcia wrote:
> > Hi Petr,
> >
> > El vie, 17 mar 2023 a las 13:51, Petr Mladek (<[email protected]>) escribió:
> > >
> > > On Thu 2023-03-16 11:29:26, Guillermo Rodriguez Garcia wrote:
> > > > Hi all,
> > > >
> > > > We have several embedded systems where pass console= or console=null
> > > > in production to disable the console.
> > > >
> > > > Later we check for this in user space: in our inittab we check if fd0
> > > > is "associated with a terminal" (test -t 0); if so, we are in
> > > > development mode and we open a debug shell; otherwise (console
> > > > disabled) we just start the application.
> > > >
> > > > Recently [1] this behaviour has changed and now if we pass console= or
> > > > console=null, the new ttynull driver is used. This breaks the check we
> > > > were doing (test -t 0 always true now).
> > > >
> > > > [1]: https://lore.kernel.org/lkml/X%2FcDG%2FxCCzSWW2cd@alley/t/
> > >
> > > This is actually exactly the problem that the change tried to solve.
> > > Some systems failed to boot when there was no console and they tried
> > > to write something at stdout.
> >
> > Well, I did not have any problem before this change.
> >
> > I understand that some systems had a problem (many others didn't, and
> > setting console= or console=null has been standard practice for a long
> > time). Since this change in behaviour could (and did) break things in
> > user space, perhaps it should have been made opt-in...
> >
> > >
> > > > Is there a way to get the previous behaviour? If not, is there an easy
> > > > way for userspace to detect whether the console device is a "real" tty
> > > > or ttynull (other than trying to parse the kernel boot args, which
> > > > would be a bit fragile).
> > >
> > > A solution would be to check that /proc/consoles has ttynull as the
> > > only registred console, for example:
> > >
> > > grep -q ttynull /proc/consoles && test `cat /proc/consoles | wc -l` -eq 1
> > >
> > > Would this work for you, please?
> >
> > I was trying to avoid something like this as it feels like userspace
> > now needs to have too much knowledge of what the kernel is doing
> > internally, however I guess this is the best option.
> >
> > The suggested check seems to work but now I am seeing a different
> > (related?) issue: when I try to reboot from the console (using
> > busybox's reboot command), the system reboots normally if I am using a
> > "normal" console (e.g. console=ttyXXX), however when using
> > console=ttynull, the command takes 20-25 seconds to complete. I am not
> > sure why this would happen; if I understand correctly, for userspace
> > ttynull is just like a regular tty driver, so why would there be a
> > difference?
> >
>
> I think we may have seen something similar, causing us to revert the
> upstream patches in ChromeOS. We did not try to find out what exactly
> caused the problems.
If you have any further details on this that would be very helpful,
even if you didn't get to pinpoint the actual cause of the problem.
Thanks,
Guillermo Rodriguez Garcia
[email protected]
Hi Linus,
I need an advice whether to revert the commit 3cffa06aeef7ece30f
("printk/console: Allow to disable console output by using
console="" or console=null").
On Wed 2023-03-22 09:33:32, Guillermo Rodriguez Garcia wrote:
> Hi Petr,
>
> El vie, 17 mar 2023 a las 13:51, Petr Mladek (<[email protected]>) escribi?:
> >
> > On Thu 2023-03-16 11:29:26, Guillermo Rodriguez Garcia wrote:
> > > Hi all,
> > >
> > > We have several embedded systems where pass console= or console=null
> > > in production to disable the console.
> > >
> > > Later we check for this in user space: in our inittab we check if fd0
> > > is "associated with a terminal" (test -t 0); if so, we are in
> > > development mode and we open a debug shell; otherwise (console
> > > disabled) we just start the application.
> > >
> > > Recently [1] this behaviour has changed and now if we pass console= or
> > > console=null, the new ttynull driver is used. This breaks the check we
> > > were doing (test -t 0 always true now).
> > >
> > > [1]: https://lore.kernel.org/lkml/X%2FcDG%2FxCCzSWW2cd@alley/t/
> >
> > This is actually exactly the problem that the change tried to solve.
> > Some systems failed to boot when there was no console and they tried
> > to write something at stdout.
>
> Well, I did not have any problem before this change.
>
> I understand that some systems had a problem (many others didn't, and
> setting console= or console=null has been standard practice for a long
> time). Since this change in behaviour could (and did) break things in
> user space, perhaps it should have been made opt-in...
Historically, a single and invalid console=<name> kernel parameter
might cause that no console gets registered. As a result, there
is no stdin/stdout/stderr for the init process.
From the code perspective, the <name> is considered a preferred
console. No console is registered by default because there is
the preferred console. But an invalid <name> never matches.
The non-existing stdin/stdout/stderr caused Oopses reported during boot, see
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/
It would have been great to debug the root of the Oopses.
But it was not trivial. And it looked like the fallback
to ttynull made sense even if the kernel did not crash:
1. The init process might expect a working stdin/stderr/stdout.
The POSIX standard says:
At program start-up, three streams are predefined and need not be
opened explicitly: standard input (for reading conventional input),
standard output (for writing conventional output), and standard error
(for writing diagnostic output). When opened, the standard error
stream is not fully buffered; the standard input and standard output
streams are fully buffered if and only if the stream can be determined
not to refer to an interactive device.
2. Also the comment in init/main.c is rather clear:
/* Open /dev/console, for stdin/stdout/stderr, this should never fail */
void __init console_on_rootfs(void)
This is why we implemented the fallback to ttynull. But it causes
regressions on Chromebooks. They use:
+ console="" to disable consoles on production systems.
(better performance, nobody reads the console anyway).
+ "test -f 0" to detect whether the Chromebook is in
debugging mode.
The regression is:
+ "test -f 0" does not longer work.
+ ttynull delays reboot on busybox by 20-25sec from unknown
reasons
From this POV, the commit breaks userspace and should be reverted.
On the other hand, I think that having stdin/stdout/stderr
might be required by POSIX.
Also note that the commit was accepted in v5.11-rc1. So the regression
is 2 years old. The revert might be seen as a regression as well.
My opinion:
I would prefer to keep the commit because it goes in the right
direction. And the revert would cause regression for the original
reporter.
The regression did not reach normal users. Chromebook developers
reverted it in their tree, see
https://lore.kernel.org/all/[email protected]/
So, we could find a better solution for them.
What is your opinion, please?
> > > Is there a way to get the previous behaviour? If not, is there an easy
> > > way for userspace to detect whether the console device is a "real" tty
> > > or ttynull (other than trying to parse the kernel boot args, which
> > > would be a bit fragile).
> >
> > A solution would be to check that /proc/consoles has ttynull as the
> > only registred console, for example:
> >
> > grep -q ttynull /proc/consoles && test `cat /proc/consoles | wc -l` -eq 1
> >
> > Would this work for you, please?
>
> I was trying to avoid something like this as it feels like userspace
> now needs to have too much knowledge of what the kernel is doing
> internally, however I guess this is the best option.
Fair enough. It is true the "ttynull" is an implementation detail.
I wonder if the console is the only detail that might be used
to detect if the system is in debug mode.
> The suggested check seems to work but now I am seeing a different
> (related?) issue: when I try to reboot from the console (using
> busybox's reboot command), the system reboots normally if I am using a
> "normal" console (e.g. console=ttyXXX), however when using
> console=ttynull, the command takes 20-25 seconds to complete. I am not
> sure why this would happen; if I understand correctly, for userspace
> ttynull is just like a regular tty driver, so why would there be a
> difference?
This looks like a timeout. It is hard to say who is waiting on what.
Anyway, the ttynull driver is super trivial, see
drivers/tty/ttynull.c.
Best Regards,
Petr
Hi Petr,
El vie, 24 mar 2023 a las 16:08, Petr Mladek (<[email protected]>) escribió:
>
> Hi Linus,
>
> I need an advice whether to revert the commit 3cffa06aeef7ece30f
> ("printk/console: Allow to disable console output by using
> console="" or console=null").
>
> On Wed 2023-03-22 09:33:32, Guillermo Rodriguez Garcia wrote:
> > Hi Petr,
> >
> > El vie, 17 mar 2023 a las 13:51, Petr Mladek (<[email protected]>) escribió:
> > >
> > > On Thu 2023-03-16 11:29:26, Guillermo Rodriguez Garcia wrote:
> > > > Hi all,
> > > >
> > > > We have several embedded systems where pass console= or console=null
> > > > in production to disable the console.
> > > >
> > > > Later we check for this in user space: in our inittab we check if fd0
> > > > is "associated with a terminal" (test -t 0); if so, we are in
> > > > development mode and we open a debug shell; otherwise (console
> > > > disabled) we just start the application.
> > > >
> > > > Recently [1] this behaviour has changed and now if we pass console= or
> > > > console=null, the new ttynull driver is used. This breaks the check we
> > > > were doing (test -t 0 always true now).
> > > >
> > > > [1]: https://lore.kernel.org/lkml/X%2FcDG%2FxCCzSWW2cd@alley/t/
> > >
> > > This is actually exactly the problem that the change tried to solve.
> > > Some systems failed to boot when there was no console and they tried
> > > to write something at stdout.
> >
> > Well, I did not have any problem before this change.
> >
> > I understand that some systems had a problem (many others didn't, and
> > setting console= or console=null has been standard practice for a long
> > time). Since this change in behaviour could (and did) break things in
> > user space, perhaps it should have been made opt-in...
>
> Historically, a single and invalid console=<name> kernel parameter
> might cause that no console gets registered. As a result, there
> is no stdin/stdout/stderr for the init process.
>
> From the code perspective, the <name> is considered a preferred
> console. No console is registered by default because there is
> the preferred console. But an invalid <name> never matches.
>
> The non-existing stdin/stdout/stderr caused Oopses reported during boot, see
> https://lore.kernel.org/all/[email protected]/
> https://lore.kernel.org/all/[email protected]/
>
> It would have been great to debug the root of the Oopses.
> But it was not trivial. And it looked like the fallback
> to ttynull made sense even if the kernel did not crash:
>
> 1. The init process might expect a working stdin/stderr/stdout.
> The POSIX standard says:
>
> At program start-up, three streams are predefined and need not be
> opened explicitly: standard input (for reading conventional input),
> standard output (for writing conventional output), and standard error
> (for writing diagnostic output). When opened, the standard error
> stream is not fully buffered; the standard input and standard output
> streams are fully buffered if and only if the stream can be determined
> not to refer to an interactive device.
>
> 2. Also the comment in init/main.c is rather clear:
>
> /* Open /dev/console, for stdin/stdout/stderr, this should never fail */
> void __init console_on_rootfs(void)
>
>
> This is why we implemented the fallback to ttynull. But it causes
> regressions on Chromebooks. They use:
>
> + console="" to disable consoles on production systems.
> (better performance, nobody reads the console anyway).
>
> + "test -f 0" to detect whether the Chromebook is in
> debugging mode.
FWIW, in my case (not a Chromebook) I was not using test -f, but test
-t in order to check whether fd 0 was "a terminal".
(In hindsight perhaps this worked because when the console is invalid,
there is no fd 0 at all).
[...]
>
> The regression did not reach normal users. Chromebook developers
This is not entirely correct; the regression not only affected
Chromebooks. I suspect it affected many embedded systems where setting
console="" or console=null is very common. For example, in U-Boot,
when you use the "silent" parameter to suppress log output (which is
common practice in production), the console parameter is automatically
set to "" (empty string). When the meaning of console="" changed, this
also caused side-effects that had to be fixed in U-Boot; see
https://github.com/u-boot/u-boot/commit/ba9aa40bb387385b8ef8b6594661a97ddcb8d04c
My point is that perhaps only Chromebook developers complained here,
but many other users were probably affected (specially on embedded
systems).
BR,
Guillermo
--
Guillermo Rodriguez Garcia
[email protected]
On Fri, Mar 24, 2023 at 8:08 AM Petr Mladek <[email protected]> wrote:
>
> I need an advice whether to revert the commit 3cffa06aeef7ece30f
> ("printk/console: Allow to disable console output by using
> console="" or console=null").
I think it's too late to revert it by now. It sounds like people who
were affected are either chrome developers that already changed their
setup, or embedded people who similarly have already reacted to it.
This went into 5.11, it's ancient history by now, so reverting it
would likely just cause more issues.
Linus
On Sun, Mar 26, 2023 at 04:20:16PM -0700, Linus Torvalds wrote:
> On Fri, Mar 24, 2023 at 8:08 AM Petr Mladek <[email protected]> wrote:
> >
> > I need an advice whether to revert the commit 3cffa06aeef7ece30f
> > ("printk/console: Allow to disable console output by using
> > console="" or console=null").
>
> I think it's too late to revert it by now. It sounds like people who
> were affected are either chrome developers that already changed their
> setup, or embedded people who similarly have already reacted to it.
>
Autually, our (non-)solution was to revert the patches in question in
all ChromeOS branches.
> This went into 5.11, it's ancient history by now, so reverting it
> would likely just cause more issues.
>
Agreed. Maybe we'll find a better solution at some point, but for now
we'll just keep the reverts in ChromeOS.
Guenter