Received: by 2002:ad5:474a:0:0:0:0:0 with SMTP id i10csp5305158imu; Sun, 20 Jan 2019 08:01:17 -0800 (PST) X-Google-Smtp-Source: ALg8bN4P4+5A34hAtW4YqfPkEZppsFN1ybbT1E837gwxtvTZh6K36fsfxi4lzG3qEtmJEH9D4Q9Y X-Received: by 2002:a17:902:2a89:: with SMTP id j9mr27022299plb.296.1548000077157; Sun, 20 Jan 2019 08:01:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1548000077; cv=none; d=google.com; s=arc-20160816; b=PA+difO6y1jGSSgw4AbgZ9qybSboGgCoWV0z/Msik/BwQnLJfwxvtOGjELpfXjLPRn Jysh+e5MTIXGoNGVCMCY+HsVJJKmvbFpLJ25V8y9Z+a+9InN8U3qtDDlETUCOzhfATU6 FokA/CkJVriWx/VkeSyyN5CS2sJMheAsCGYQ+m0f+bjKdsYaQ9K5wWY9UobmywADUKbY Jzt3fdFps1y1sWpoXGafNdKJoUZD2YXkjiVc6UPJql90YJ8Oa79EVor5feU/v/qhgqUA ZRgsdXQQ1yvmymZo4wtZae0oWy7YFNblNDZAX6WB4XcJRCMYLiNqkJVBxK4pBZkmh7lG D6QA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:dkim-signature; bh=jTmIghGjwE8iHAb3Dh6gyTw5bykolHFzGqZYxXOr2j4=; b=y8kbaP87k0eJuTxx/ZNYqppL/kRsb7A5u2bDItV5avNSjVBxYg9cN2X7t5RbDMITdM +RlQHFdNtP1zI0HjMoaGi39DDPi2n1mycNm3IHIhtKK1uIk+Vjjqxdy/09rghNdhG5Sl vKyEfEa0wjuvNiFcQqagEa5UW0drKpaf1r7O1WQnkItglTisT6CjNiF+oDFZcpAKjSsq wIzH6QpoLH+Uve77Pa+afHKY/stVj/DQqtuw7Y4N60uC960VwMSVrCkNIpveCR74aweR g0ABFofPmzDP4h/yifxzgmBvXwE7Xtszu4Ytteh8YiGG7dQvBdxnqKjZLHx1C+hyfePh 3xxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b="v/l9nPGT"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 88si10295797plb.288.2019.01.20.08.01.01; Sun, 20 Jan 2019 08:01:17 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b="v/l9nPGT"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726524AbfATP6m (ORCPT + 99 others); Sun, 20 Jan 2019 10:58:42 -0500 Received: from mail-qt1-f194.google.com ([209.85.160.194]:43225 "EHLO mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725908AbfATP6l (ORCPT ); Sun, 20 Jan 2019 10:58:41 -0500 Received: by mail-qt1-f194.google.com with SMTP id i7so20518793qtj.10 for ; Sun, 20 Jan 2019 07:58:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=jTmIghGjwE8iHAb3Dh6gyTw5bykolHFzGqZYxXOr2j4=; b=v/l9nPGTlkHu6FuY5yAwwmDsblcdJb5KPsIbmx7UFTrLehP1ann8DWpXqMPg0hj1QM 8IyvYAh22vUnNQkdlhuR7ZqPufE4+31/zcOC528V1UG0DLtZ/J00sW5fBgy+eQwfl0Nw JlowaEMaZyxnRBY2jZBpfGaxzLB9mbe3F3UDQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=jTmIghGjwE8iHAb3Dh6gyTw5bykolHFzGqZYxXOr2j4=; b=VgtQvSdLHSmeH2b654KKPzmzzxVB+hhu0BrgIYPbGHlQJgemKzLJls1/6Gp9Gz6dx+ xC0J8koZKKixqfG8nXWi+IQBbTXfizQHk7mwqWBZr6YNquuyIL/O+EyYG3VwsgLFZa3g hnX0+NdqjUgmRv+9yrVr+36ewidz0Bm6f7lWv3Ug31l26oSMJN16ziBrmcz8x82ZFhUg b1vga2hcCwAlkqL+mqY7DBjfsWu/V7kxLGzYF48UidQo69EEfotDZPCJKuH5bp283hp4 tU68piLtcqC40+HVZPFMZphIzw6GUBZJDaW8raseCSXNaFUM2Sm+nhlrnwpXHQrQYoYJ O6OQ== X-Gm-Message-State: AJcUukfzPrHpqegz5SwMl4kg5Z2fy2BODhaHTM9kZ+Y803WCtrCYc5/5 ecGvyca3IPGazSwPRrF0lTpkiA== X-Received: by 2002:ac8:1712:: with SMTP id w18mr23293243qtj.76.1547999920270; Sun, 20 Jan 2019 07:58:40 -0800 (PST) Received: from localhost ([2620:0:1004:1100:cca9:fccc:8667:9bdc]) by smtp.gmail.com with ESMTPSA id a17sm55244827qth.93.2019.01.20.07.58.39 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 20 Jan 2019 07:58:39 -0800 (PST) Date: Sun, 20 Jan 2019 10:58:38 -0500 From: Joel Fernandes To: hpa@zytor.com Cc: Daniel Colascione , Greg KH , linux-kernel , Andrew Morton , ast@kernel.org, atishp04@gmail.com, Borislav Petkov , Ingo Molnar , Jan Kara , Jonathan Corbet , karim.yaghmour@opersys.com, Kees Cook , kernel-team@android.com, "open list:DOCUMENTATION" , Manoj Rao , Masahiro Yamada , Paul McKenney , "Peter Zijlstra (Intel)" , Randy Dunlap , rostedt@goodmis.org, Thomas Gleixner , "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" , yhs@fb.com Subject: Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel Message-ID: <20190120155838.GA23827@google.com> References: <20190118225543.86996-1-joel@joelfernandes.org> <20190119082532.GA9004@kroah.com> <20190119162754.GC231277@google.com> <20190119232503.GA149403@google.com> <78AACAF1-8EBF-4DF3-BE94-5B14E78BA791@zytor.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <78AACAF1-8EBF-4DF3-BE94-5B14E78BA791@zytor.com> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, Jan 19, 2019 at 03:44:48PM -0800, hpa@zytor.com wrote: > On January 19, 2019 3:25:03 PM PST, Joel Fernandes wrote: > >On Sat, Jan 19, 2019 at 12:43:35PM -0500, Daniel Colascione wrote: > >> On Sat, Jan 19, 2019 at 11:27 AM Joel Fernandes > > wrote: > >> > > >> > On Sat, Jan 19, 2019 at 09:25:32AM +0100, Greg KH wrote: > >> > > On Fri, Jan 18, 2019 at 05:55:43PM -0500, Joel Fernandes wrote: > >> > > > --- /dev/null > >> > > > +++ b/kernel/kheaders.c > >> > >> Thanks a ton for this work. It'll make it much easier to do cool > >> things with BPF. > > > >You're welcome, thanks. > > > >> One question: I can imagine wanting to probe > >> structures that are defined, not in headers, but in random > >> implementation files. Would it be possible to optionally include > >*all* > >> kernel source files? > > > >That would be prohibitively too large to justify keeping it in memory, > >even > >compressed. Arguably, such structures should be moved into include/ if > >modules or whatever is extending the kernel like eBPF needs them. > > > >> If not, what about a hash, so we could at least > >> do precise correlation between a candidate local tree and what's > >> actually on device? > > > >That would make a tool too difficult to write wouldn't it, since they > >you have to > >correlate every possible hash and keep updating eBPF tools with new > >hashes - > >probably not scalable. I think what you want is to use the kernel > >version to > >assume what such internal structures look like although that's still > >not > >robust. > > > >> BTW, I'm not sure that the magic constants you've defined are long > >> enough. I'd feel more comfortable with two UUIDs (16 bytes each). > > > >Ok, I'll expand it. > > > >> I'd also strongly consider LZMA compression: xz -9 on the kernel > >> headers (with comments) brings the size down to 5MB, compared to the > >> 7MB I get for gzip -9. Considering that this feature is optional, I > >> think it's okay to introduce a dependency on widespread modern > >> compression tools. (For comparison, bzip2 -9 gets us 6MB.) > > > >Ok, I'll look into LZMA. Thanks for checking the compression sizes. > > > >- Joel > > Don't use lzma, use xz if you are going to do something. Ok, sounds good. > However, it seems unlikely to me that someone not willing to spend the space in the filesystem will spend unswappable kernel memory. > > It would seem that a far saner way to do this is to use inittmpfs or perhaps an auxiliary "ktmpfs" so it can at least be swapped out if you have swap. But this is already possible with the proposed solution, you would load the module, extract it into a tmpfs, and unload the module. TMPFS pages can already be swapped. thanks, - Joel