Received: by 2002:ad5:474a:0:0:0:0:0 with SMTP id i10csp1480331imu; Wed, 23 Jan 2019 18:34:06 -0800 (PST) X-Google-Smtp-Source: ALg8bN4brCW5JW/q/NmiDVxB9W4nG8I4Q0IgM2wF6bNH37i2h4K8aHFidWoZ8ckRAGOy59ZfaVN9 X-Received: by 2002:a63:193:: with SMTP id 141mr4322461pgb.136.1548297246051; Wed, 23 Jan 2019 18:34:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1548297246; cv=none; d=google.com; s=arc-20160816; b=m9oBoJyMUtf2t/VvhGxVh7F4xCmO51i+NrzeJ83Ki1WqoiToUHAC840g224FNIXbdq PR+bcpCd8+/Dr9MW47+XOhYhcDJNRbyoHD6xYtPFaBAt/F42arS9wZgTJhfqjaFy3bKD nhaIgzXBeC7zK4wEUR5E6wFp8KHC0QaYw9uXbr46Bzlu5DljtvL+eIcKnTBQZ8nRtkHe E1HVLKdboMDy7/EcI34qF7v4tvjDFfrCSBiV30EvWu/sLX12D+YAXRMC82MMpm2Xtbap 2+tS1X9xBsTVOBvubks5tsT97ZbtR+1CpxhfLytvyZeqV5phsuT7mPjkczwhcvHkBkZC 1Pbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:dkim-signature; bh=CGuClkjsH6Nm5oWhdq5m1OKE4/YFrdge0arIdi6haBM=; b=BklNKWnSXA4ce1j5E26m8TUdP8QkSwfl8qIJfY4ZL38bole8vE9W/5uG1w4IJBL4ar WPWLqzty5aWj+DRSPZB6p8l3C1sE88QQbvMRjUdFIuOsNxC6DRwgg+Snzmy302qrj8Q8 kCwE9AYiO1WBUdKzRITT6p9R6O5sAyLFfxhZ2UOlLLiU0c/D/xRPBRzaUW8cik9eGVje Y8BAYUo2wBHzhItGadpd/089jE5JW9N8OCQbkIas2G430HMv8pUqYcPKAUJYYdW9Mvb1 S43gB1E8/D67dbBhjRt7Z7KMHQKi7JAyyS4qAkAUgb/azyvleVcOdrWrAQL3SH9sK+Xb VF8Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=MTUFYL4u; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id v75si19804250pfd.157.2019.01.23.18.33.49; Wed, 23 Jan 2019 18:34:06 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=MTUFYL4u; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727096AbfAXCcU (ORCPT + 99 others); Wed, 23 Jan 2019 21:32:20 -0500 Received: from mail-qt1-f194.google.com ([209.85.160.194]:37551 "EHLO mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726517AbfAXCcT (ORCPT ); Wed, 23 Jan 2019 21:32:19 -0500 Received: by mail-qt1-f194.google.com with SMTP id t33so4901906qtt.4 for ; Wed, 23 Jan 2019 18:32:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=CGuClkjsH6Nm5oWhdq5m1OKE4/YFrdge0arIdi6haBM=; b=MTUFYL4uz20ZkJdQhBGq81oRQWYM/a5LaWZ3I2DMb0GPKDgW3mOaYzvDCEXEC6ECvI RQEiAEm/QayLuH9k2zjTSD2J4QfCvJ4VmXSfnISUcdMH0bpVanUC1/u7ofTELrIdCUtx 7yUACNXLezQ7OAFylAuhiAP2YzBLqdjQX5fY8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=CGuClkjsH6Nm5oWhdq5m1OKE4/YFrdge0arIdi6haBM=; b=q94KLGOK1Df+B8Gw/XCoADSQud7tU1OXJ+FmVHXRT4dlDgvnHK+kFH481C+YRKUin8 MwbUbF77xmlDe4qdQ3JfYza5ebmwCnRqAlmtFRbnSqSzgSep1IHc6b/bptQ2ZwX4x/la 78/aPIi+QcRJB/vl/WK+MSLhvAu/GmmyvvxxWYt5XCkMYT347WqWAeIbIYABFLoR/8BH 4KJZmKfyeDEpdIhH6vgw1l4iP7MkXRbYsjAozM8QitTYMPGS2DnnAtrT1gUdflBfGn9F 7vHFA4GuQPfTNMcfal9fXLmGFDeYQAlPgvI1mi/1/QjH1FmT/Evc/i3OHxRL4x0z5HT7 Fiaw== X-Gm-Message-State: AJcUuke0+xY7unAXeWKG2kNVz0fS2r3tF+u5cluWzZ//Tl8pRLN/w6kz bmBP56INoXlMeiI3Gn/iDfwnPw== X-Received: by 2002:a0c:a144:: with SMTP id d62mr4370503qva.50.1548297138371; Wed, 23 Jan 2019 18:32:18 -0800 (PST) Received: from localhost ([2620:0:1004:1100:cca9:fccc:8667:9bdc]) by smtp.gmail.com with ESMTPSA id a17sm73484791qtk.82.2019.01.23.18.32.17 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Jan 2019 18:32:17 -0800 (PST) Date: Wed, 23 Jan 2019 21:32:16 -0500 From: Joel Fernandes To: Daniel Colascione Cc: Karim Yaghmour , "H. Peter Anvin" , Greg KH , Christoph Hellwig , linux-kernel , Andrew Morton , ast@kernel.org, atish patra , Borislav Petkov , Ingo Molnar , Jan Kara , Jonathan Corbet , Kees Cook , kernel-team@android.com, "open list:DOCUMENTATION" , Manoj Rao , Masahiro Yamada , Paul McKenney , "Peter Zijlstra (Intel)" , Randy Dunlap , rostedt@goodmis.org, Thomas Gleixner , "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" , yhs@fb.com Subject: Re: [RFC] Provide in-kernel headers for making it easy to extend the kernel Message-ID: <20190124023216.GA75730@google.com> References: <20190119102800.GB17723@infradead.org> <20190119103606.GA17400@kroah.com> <8BD4CB7A-944C-4EC5-A198-1D85C9EE28D6@zytor.com> <20190120161003.GB23827@google.com> <20190121014553.GD23827@google.com> <20190122133901.GA189736@google.com> <117d2f96-b0e9-2376-69b7-836fa0c52539@opersys.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jan 23, 2019 at 02:37:47PM -0800, Daniel Colascione wrote: > On Wed, Jan 23, 2019 at 1:29 PM Karim Yaghmour > wrote: [...] > > Personally I advocated a more aggressive approach with Joel in private: > > just put the darn headers straight into the kernel image, it's the > > *only* artifact we're sure will follow the Android device whatever > > happens to it (like built-in ftrace). > > I was thinking along similar lines. Ordinarily, we make loadable > kernel modules. What we kind of want here is a non-loadable kernel > module --- or a non-loadable section in the kernel image proper. I'm > not familiar with early-stage kernel loader operation: I know it's > possible to crease discardable sections in the kernel image, but can > we create sections that are never slurped into memory in the first > place? If not, maybe loading and immediately discarding the header > section is good enough. I am happy to see if I can shrink it down further. Especially using xz and stripping all comments period. I am optimistic this can be brought down further to a point where it would make sense to everyone to build it into the kernel. Lets see. Last time I stripped comments, it went down by ~40%. What I haven't tried is doing this *with* xz compression. I am also open to brainstorming what else can be stripped. OTOH the reason I didn't focus much on size is: modules are pretty much universal and I'm confident of wide spread use of this feature for Android-based products and not needing of "chasing headers" if we modularize it, since Android's project treble has modularized things and modules are now default enabled. Putting headers into a module lets us enjoy the ride there. I am quite hosed for the next week or so to work on this, but I should be able to get back to it after. cheers, - Joel > > > To that end, I even had some crazy > > ideas on how to compress the headers even further than with std > > compression algorithms -- here's a snippet from an email I sent Joel > > some time back detailing such a hack: > > > Since C headers have fairly constrained semantics and since the types of semantics generally used to name structs, etc. in the Linux kernel are well established, we can likely devise a very customized compression algorithm for the purpose. > > Would such a thing really do better than LZMA? LZMA already has very > clever techniques for eliminating long-range redundancies in > compressible text, including redundancies at the sub-byte level. I can > certainly understand the benefit of stripping comments, since removing > comments really does decrease the total amount of information the > compressor has to preserve, but I'm not sure how much the encoding > scheme you propose below would help, since it reminds me of the > encoding scheme that LZMA would discover automatically. > > > Whether such craziness makes sense or is adopted or not isn't mine to > > chart, but I certainly can't see eBPF reaching the same mass deployment > > ftrace has within the Android ecosystem until there's a way to use it > > without having to chase kernel headers independently of kernel images. > > There are "too many clicks" involved and someone somewhere will drop the > > ball if it's not glued to the kernel in some way shape or form. Any > > solution that solves this is one I'd love to hear about. > > I agree. There definitely needs to be a "just collect a damn trace" > button that works on any device, and for this button to work and > incorporate eBPF, the system needs to be able to describe itself.