Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Wed, 23 Jan 2019 21:32:16 -0500
From:   Joel Fernandes <joel@joelfernandes.org>
To:     Daniel Colascione <dancol@google.com>
Cc:     Karim Yaghmour <karim.yaghmour@opersys.com>,
        "H. Peter Anvin" <hpa@zytor.com>,
        Greg KH <gregkh@linuxfoundation.org>,
        Christoph Hellwig <hch@infradead.org>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>, ast@kernel.org,
        atish patra <atishp04@gmail.com>,
        Borislav Petkov <bp@alien8.de>, Ingo Molnar <mingo@redhat.com>,
        Jan Kara <jack@suse.cz>, Jonathan Corbet <corbet@lwn.net>,
        Kees Cook <keescook@chromium.org>, kernel-team@android.com,
        "open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
        Manoj Rao <linux@manojrajarao.com>,
        Masahiro Yamada <yamada.masahiro@socionext.com>,
        Paul McKenney <paulmck@linux.vnet.ibm.com>,
        "Peter Zijlstra (Intel)" <peterz@infradead.org>,
        Randy Dunlap <rdunlap@infradead.org>, rostedt@goodmis.org,
        Thomas Gleixner <tglx@linutronix.de>,
        "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" <x86@kernel.org>,
        yhs@fb.com
Subject: Re: [RFC] Provide in-kernel headers for making it easy to extend the
 kernel
Message-ID: <20190124023216.GA75730@google.com>
References: <20190119102800.GB17723@infradead.org>
 <20190119103606.GA17400@kroah.com>
 <8BD4CB7A-944C-4EC5-A198-1D85C9EE28D6@zytor.com>
 <20190120161003.GB23827@google.com>
 <A40C22DF-B43F-44BA-8148-FC0881087315@zytor.com>
 <20190121014553.GD23827@google.com>
 <BC774ED4-C0FC-4AA6-9BAF-36930A661D9B@zytor.com>
 <20190122133901.GA189736@google.com>
 <117d2f96-b0e9-2376-69b7-836fa0c52539@opersys.com>
 <CAKOZuesWgo3CZj49aUChX_-dpijCrVLxgVZChyLLmOWXCLQf-Q@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAKOZuesWgo3CZj49aUChX_-dpijCrVLxgVZChyLLmOWXCLQf-Q@mail.gmail.com>
User-Agent: Mutt/1.10.1 (2018-07-13)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Wed, Jan 23, 2019 at 02:37:47PM -0800, Daniel Colascione wrote:
> On Wed, Jan 23, 2019 at 1:29 PM Karim Yaghmour
> <karim.yaghmour@opersys.com> wrote:
[...]
> > Personally I advocated a more aggressive approach with Joel in private:
> > just put the darn headers straight into the kernel image, it's the
> > *only* artifact we're sure will follow the Android device whatever
> > happens to it (like built-in ftrace).
> 
> I was thinking along similar lines. Ordinarily, we make loadable
> kernel modules. What we kind of want here is a non-loadable kernel
> module --- or a non-loadable section in the kernel image proper. I'm
> not familiar with early-stage kernel loader operation: I know it's
> possible to crease discardable sections in the kernel image, but can
> we create sections that are never slurped into memory in the first
> place? If not, maybe loading and immediately discarding the header
> section is good enough.

I am happy to see if I can shrink it down further. Especially using xz and
stripping all comments period. I am optimistic this can be brought down
further to a point where it would make sense to everyone to build it into the
kernel. Lets see.

Last time I stripped comments, it went down by ~40%. What I haven't tried is
doing this *with* xz compression. I am also open to brainstorming what else
can be stripped.

OTOH the reason I didn't focus much on size is: modules are pretty much
universal and I'm confident of wide spread use of this feature for Android-based
products and not needing of "chasing headers" if we modularize it, since
Android's project treble has modularized things and modules are now default
enabled. Putting headers into a module lets us enjoy the ride there.

I am quite hosed for the next week or so to work on this, but I should be
able to get back to it after.

cheers,

 - Joel


> 
> > To that end, I even had some crazy
> > ideas on how to compress the headers even further than with std
> > compression algorithms -- here's a snippet from an email I sent Joel
> > some time back detailing such a hack:
> > > Since C headers have fairly constrained semantics and since the types of semantics generally used to name structs, etc. in the Linux kernel are well established, we can likely devise a very customized compression algorithm for the purpose.
> 
> Would such a thing really do better than LZMA? LZMA already has very
> clever techniques for eliminating long-range redundancies in
> compressible text, including redundancies at the sub-byte level. I can
> certainly understand the benefit of stripping comments, since removing
> comments really does decrease the total amount of information the
> compressor has to preserve, but I'm not sure how much the encoding
> scheme you propose below would help, since it reminds me of the
> encoding scheme that LZMA would discover automatically.
> 
> > Whether such craziness makes sense or is adopted or not isn't mine to
> > chart, but I certainly can't see eBPF reaching the same mass deployment
> > ftrace has within the Android ecosystem until there's a way to use it
> > without having to chase kernel headers independently of kernel images.
> > There are "too many clicks" involved and someone somewhere will drop the
> > ball if it's not glued to the kernel in some way shape or form. Any
> > solution that solves this is one I'd love to hear about.
> 
> I agree. There definitely needs to be a "just collect a damn trace"
> button that works on any device, and for this button to work and
> incorporate eBPF, the system needs to be able to describe itself.