Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Sat, 19 Jan 2019 18:25:03 -0500
From:   Joel Fernandes <joel@joelfernandes.org>
To:     Daniel Colascione <dancol@google.com>
Cc:     Greg KH <gregkh@linuxfoundation.org>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>, ast@kernel.org,
        atishp04@gmail.com, Borislav Petkov <bp@alien8.de>,
        "H. Peter Anvin" <hpa@zytor.com>, Ingo Molnar <mingo@redhat.com>,
        Jan Kara <jack@suse.cz>, Jonathan Corbet <corbet@lwn.net>,
        karim.yaghmour@opersys.com, Kees Cook <keescook@chromium.org>,
        kernel-team@android.com,
        "open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
        Manoj Rao <linux@manojrajarao.com>,
        Masahiro Yamada <yamada.masahiro@socionext.com>,
        Paul McKenney <paulmck@linux.vnet.ibm.com>,
        "Peter Zijlstra (Intel)" <peterz@infradead.org>,
        Randy Dunlap <rdunlap@infradead.org>, rostedt@goodmis.org,
        Thomas Gleixner <tglx@linutronix.de>,
        "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" <x86@kernel.org>,
        yhs@fb.com
Subject: Re: [RFC] Provide in-kernel headers for making it easy to extend the
 kernel
Message-ID: <20190119232503.GA149403@google.com>
References: <20190118225543.86996-1-joel@joelfernandes.org>
 <20190119082532.GA9004@kroah.com>
 <20190119162754.GC231277@google.com>
 <CAKOZuet+7a4jdox6rebMN=kJtDn5RUYzvNjFh+Pzn2UkuP0Y8Q@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAKOZuet+7a4jdox6rebMN=kJtDn5RUYzvNjFh+Pzn2UkuP0Y8Q@mail.gmail.com>
User-Agent: Mutt/1.10.1 (2018-07-13)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Sat, Jan 19, 2019 at 12:43:35PM -0500, Daniel Colascione wrote:
> On Sat, Jan 19, 2019 at 11:27 AM Joel Fernandes <joel@joelfernandes.org> wrote:
> >
> > On Sat, Jan 19, 2019 at 09:25:32AM +0100, Greg KH wrote:
> > > On Fri, Jan 18, 2019 at 05:55:43PM -0500, Joel Fernandes wrote:
> > > > --- /dev/null
> > > > +++ b/kernel/kheaders.c
> 
> Thanks a ton for this work. It'll make it much easier to do cool
> things with BPF.

You're welcome, thanks.

> One question: I can imagine wanting to probe
> structures that are defined, not in headers, but in random
> implementation files. Would it be possible to optionally include *all*
> kernel source files?

That would be prohibitively too large to justify keeping it in memory, even
compressed. Arguably, such structures should be moved into include/ if
modules or whatever is extending the kernel like eBPF needs them.

> If not, what about a hash, so we could at least
> do precise correlation between a candidate local tree and what's
> actually on device?

That would make a tool too difficult to write wouldn't it, since they you have to
correlate every possible hash and keep updating eBPF tools with new hashes -
probably not scalable. I think what you want is to use the kernel version to
assume what such internal structures look like although that's still not
robust.

> BTW, I'm not sure that the magic constants you've defined are long
> enough.  I'd feel more comfortable with two UUIDs (16 bytes each).

Ok, I'll expand it.

> I'd also strongly consider LZMA compression: xz -9 on the kernel
> headers (with comments) brings the size down to 5MB, compared to the
> 7MB I get for gzip -9. Considering that this feature is optional, I
> think it's okay to introduce a dependency on widespread modern
> compression tools. (For comparison, bzip2 -9 gets us 6MB.)

Ok, I'll look into LZMA. Thanks for checking the compression sizes.

- Joel