Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
In-Reply-To: <0b3a9bd0-3046-cdab-cfee-0ca45ee64e8d@landley.net>
References: <b33da177-a836-40eb-25c8-8134da83c63d@landley.net>
 <CA+55aFyw74DcPygS=SB0d-Fufz3j73zTVp2UXUUOUt4=1_He=Q@mail.gmail.com>
 <CA+55aFyWWMipF2sLdcUbz2JwMQ0YRK5YAHTy8HrUPevMF+6XZA@mail.gmail.com> <0b3a9bd0-3046-cdab-cfee-0ca45ee64e8d@landley.net>
From:   enh <enh@google.com>
Date:   Fri, 3 Nov 2017 17:03:13 -0700
Message-ID: <CAJgzZorNiMYz+a6r9pO_zRbWX-TQOGAgdQdmQu7abZ_gUQtbrQ@mail.gmail.com>
Subject: Re: [Toybox] Regression: commit da029c11e6b1 broke toybox xargs.
To:     Rob Landley <rob@landley.net>
Cc:     Linus Torvalds <torvalds@linux-foundation.org>, enh@gmail.com,
        Kees Cook <keescook@chromium.org>, toybox@lists.landley.net,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Fri, Nov 3, 2017 at 4:58 PM, Rob Landley <rob@landley.net> wrote:
> On 11/02/2017 10:40 AM, Linus Torvalds wrote:
>> On Wed, Nov 1, 2017 at 9:28 PM, Linus Torvalds
>> <torvalds@linux-foundation.org> wrote:
>>>
>>> Behavior changed. Things that test particular limits will get different
>>> results. That's not breakage.
>>>
>>> Did an actual user application or script break?
>
> Only due to getting the limit wrong. The actual failure's in the android
> internal bugzilla I've never been able to read:
>
> http://lists.landley.net/pipermail/toybox-landley.net/2017-September/009167.html

there was nothing secret or interesting in the original bug report.
just someone trying to use find/xargs on the whole file system. here's
a copy & paste:

marlin:/ # find / /data -xdev -type f ! -name \*.jpg -exec grep -Fl
belkin.a {} +
find: exec grep: Argument list too long

marlin:/ # find / /data -xdev -type f ! -name \*.jpg -print | xargs
grep -Fl belkin.a
xargs: exec grep: Argument list too long

> But it boils down to "got the limit wrong, the exec failed after the
> fork(), dynamic recovery from which is awkward so I'm trying to figure
> out the right limit".
>
>> Ahh. I should have read that email more carefully. If xargs broke,
>> that _will_ break actual scripts, yes. Do you actually set the stack
>> limit to insane values? Anybody using toybox really shouldn't be doing
>> 32MB stacks.
>
> Toybox is the default command line of android since M, which went 64 bit
> in L, and the Pixel 2 phone has 4 gigs of ram. My goal with toybox is to
> turn android into a self-hosting development environment no longer
> cross-compiled from a PC (http://landley.net/talks/celf-2013.txt) so I'm
> trying to implement a command line that can run the entire AOSP build.
>
> I.E. I have no idea what people will do with it, and try not to get in
> their way.
>
> My problem here is it's hard to figure out what exec size the limit
> _is_. There's a sysconf(_SC_ARG_MAX) which bionic and glibc are
> currently returning as stack_limit/4, which is now too big and exec()
> will error out after the fork. Musl is returning the 131072 limit from
> 2011-ish, meaning "/bin/echo $(printf '%0*d' 131071)" works but
> "printf '%0*d' 131071 | xargs" fails, an inconsistency I was trying to
> avoid. Maybe I don't have that luxury...
>
> Each argument has its own limit separate from the argv+envp total limit,
> but there's only one "size" you can query through sysconf, so the
> querying API is insufficient at the design level.
>
> Meanwhile under bash you can allocate and dirty 256 megabytes from the
> command line with:
>
>   echo $(printf '%0*d' $((1<<28)))
>
> Because it's a shell builtin so there's no actual exec. (And if
> https://sourceware.org/bugzilla/show_bug.cgi?id=17829 ever gets fixed
> it'll go back to allowing INT_MAX.)
>
> Posix is its usual helpful self, read conservatively
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/xargs.html
> says to break the line at 2048 bytes.
>
>> So I still do wonder if this actually breaks anything real, or just a
>> test-suite or something?
>
> I've cc'd Elliott, who would know. (He's the Android base os userspace
> maintainer, he knows everything. Or can at least decode
> http://b/65818597 .)
>
> But this just broke my _fix_, not the earlier deployed stuff. I removed
> the size measuring code when the 131072 limit went away, the bug was
> there's a new limit I need to not hit, I tried to figure out what the
> limit is now, confirmed that the various libc implementations don't
> agree, then the actual kernel limit changed again while I was looking at it.
>
>>                    Linus
>
> Should I just go back to hardwiring in 131072? It's no _less_ arbitrary
> than 10 megs, and it sounds like getting it _right_ is unachievable.
>
> Thanks,
>
> Rob
>
>
> _______________________________________________
> Toybox mailing list
> Toybox@lists.landley.net
> http://lists.landley.net/listinfo.cgi/toybox-landley.net


-- 
Elliott Hughes - http://who/enh - http://jessies.org/~enh/
Android native code/tools questions? Mail me/drop by/add me as a reviewer.

From 1583091369137192079@xxx Fri Nov 03 23:59:39 +0000 2017
X-GM-THRID: 1582908659593594930
X-Gmail-Labels: Inbox,Category Forums,HistoricalUnread