Add a text explaining how to quickly build a kernel, as that's something
users will often have to do when they want to report an issue or test
proposed fixes. This is a huge and frightening task for quite a few
users these days, as many rely on pre-compiled kernels and have never
built their own. They find help on quite a few websites explaining the
process in various ways, but those howtos often omit important details
or make things too hard for the 'quickly build just for testing' case
that 'localmodconfig' is really useful for. Hence give users something
at hand to guide them, as that makes it easier for them to help with
testing, debugging, and fixing the kernel.
To keep the complexity at bay, the document explicitly focuses on how to
compile the kernel on commodity distributions running on commodity
hardware. People that deal with less common distributions or hardware
will often know their way around already anyway.
The text describes a few oddities of Arch and Debian that were found by
the author and a few volunteers that tested the described procedure.
There are likely more such quirks that need to be covered as well as a
few things the author will have missed -- but one has to start
somewhere.
The document heavily uses anchors and links to them, which makes things
slightly harder to read in the source tree. But the intended target
audience is way more likely to read rendered versions of this text on
pages like docs.kernel.org anyway -- and there those anchors and links
allow easy jumps to the reference section and back, which makes the
document a lot easier to work with for the intended target audience.
Aspects relevant for bisection were left out on purpose, as that is a
related, but in the end different use case. The rough plan is to have a
second document with a similar style to cover bisection. The idea is to
reuse a few bits from this document and link quite often to entries in
the reference section with the help of the anchors in this text.
Signed-off-by: Thorsten Leemhuis <[email protected]>
---
Hi! Above patch description covers nearly everything already, there is
just one thing where I'm unsure about the approach and would like to
hear other opinions on:
The text currently describes two approaches to retrieve Linux' sources
using git: the regular clone with linux-stable as a remote and a shallow
clone with just one branch from linux-stable. The shallow clone approach
is a little bit more tricky to describe and handle, but downloads way
less data – and thus is a lot quicker, unless you have a really really
quick link to the internet (which in some parts of the world is hard to
come by). That's why I wonder if the text should switch to making the
shallow clone with selected stable branches the default. What do you
think, dear reader?
Ciao, Thorsten
P.S.: a rendered version of this text is temporarily available at:
https://www.leemhuis.info/files/misc/How%20to%20quickly%20build%20a%20Linux%20kernel%20%E2%80%94%20The%20Linux%20Kernel%20documentation.html
---
.../admin-guide/howto-quickly-build-linux.rst | 903 ++++++++++++++++++
Documentation/admin-guide/index.rst | 1 +
MAINTAINERS | 1 +
3 files changed, 905 insertions(+)
create mode 100644 Documentation/admin-guide/howto-quickly-build-linux.rst
diff --git a/Documentation/admin-guide/howto-quickly-build-linux.rst b/Documentation/admin-guide/howto-quickly-build-linux.rst
new file mode 100644
index 000000000000..d1ca349774a1
--- /dev/null
+++ b/Documentation/admin-guide/howto-quickly-build-linux.rst
@@ -0,0 +1,903 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
+.. [see the bottom of this file for redistribution information]
+
+===================================
+How to quickly build a Linux kernel
+===================================
+
+This guide explains how to build Linux kernels that are ideal for testing
+purposes, but perfectly fine for day-to-day use, too.
+
+The essence of the process (aka 'TL;DR')
+========================================
+
+*[If you are new to compiling Linux, ignore this TLDR and head over to the next
+section below: it contains a step-by-step guide, which is more detailed, but
+still brief and easy to follow; that guide and its accompanying reference
+section also mention alternatives, pitfalls, and additional aspects, all of
+which might be relevant for you.]*
+
+If your platform uses techniques like Secure Boot, prepare the system to permit
+starting self-compiled Linux kernels; install compilers and everything else
+needed for building Linux; make sure to have 15 Gigabyte free space in your home
+directory. Now run the following commands to download the sources of Linux,
+which you then use to configure, build and install your own kernel::
+
+ mkdir ~/linux/ ~/linux/sources ~/linux/build
+ git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git \
+ ~/linux/sources
+ cd ~/linux/sources/
+ # hint: if you don't want to build any stable or longterm kernels, skip the
+ # next two commands
+ git remote add linux-stable \
+ https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
+ git fetch --all
+ # hint: if you want to apply patches, do it at this point (see below for details)
+ # hint: at this point it's recommended to tag your build (see below for details)
+ yes "" | make O=~/linux/build/ localmodconfig
+ # hint: at this point you might want or have to adjust the build configuration
+ # (see below for details)
+ # note: if you are running a Debian kernel, you'll have to adjust the configuration
+ # at this point (see below)
+ make -j $(nproc --all) O=~/linux/build/
+ command -v installkernel && sudo make O=~/linux/build/ modules_install install
+ # note: on quite a few distributions the last command does the trick; on others,
+ # like Arch Linux, it's not enough (see below)
+
+Step-by-step guide
+==================
+
+Compiling your own Linux kernel is easy in principle. There are various ways to
+do it. Which of them actually work and is the best depends on the circumstances.
+
+This guide describes a way perfectly suited for those who want to quickly
+install Linux from sources without being bothered by complicated details; the
+goal is to cover everything typically needed on mainstream Linux distributions
+running on commodity PC or server hardware.
+
+The described approach is great for testing purposes, for example to try a
+proposed fix or check if the latest codebase still contains a problem.
+Nonetheless, kernels built this way are also totally fine for day-to-day use
+while at the same time being easy to keep up to date.
+
+The following steps describe the important aspects of the process; a
+comprehensive reference section later explains each of them in more detail. It
+sometimes also describes alternative approaches, pitfalls, as well as errors
+that might occur at a particular point -- and how to then get things rolling
+again.
+
+.. _backup_sbs:
+
+ * Create a fresh backup and put system repair and restore tools at hand, just
+ to be prepared for the unlikely case that something might go sideways.
+
+ [:ref:`details<backup>`]
+
+.. _secureboot_sbs:
+
+ * On platforms with 'Secure Boot' or similar techniques, prepare everything to
+ ensure the system will permit your self-compiled kernel to boot later. The
+ quickest and easiest way to achieve this on commodity x86 systems is to turn
+ such techniques off in the BIOS setup utility; alternatively remove their
+ restrictions through a process initiated by ``mokutil --disable-validation``.
+
+ [:ref:`details<secureboot>`]
+
+.. _buildrequires_sbs:
+
+ * Install all software required to build a Linux kernel. Often you will need:
+ 'bc', 'binutils' ('ld' et al.), 'bison', 'flex', 'gcc', 'git', 'openssl',
+ 'pahole', 'perl', and the development headers for 'libelf' and 'openssl'. The
+ reference section shows how to quickly install those on various Linux
+ distributions.
+
+ [:ref:`details<buildrequires>`]
+
+.. _diskspace_sbs:
+
+ * Ensure to have enough free space for building and installing Linux. For the
+ latter 150 Megabyte in /lib/ and 100 in /boot/ are a safe bet. For storing
+ sources and build artifacts 15 Gigabyte in your home directory should
+ typically suffice. If you have less available, be sure to check the reference
+ section for the steps about downloading the Linux sources and adjusting your
+ kernels build configuration: they mention tricks that reduce the amount of
+ required space in /home/ to around 3,5 Gigabyte.
+
+ [:ref:`details<diskspace>`]
+
+.. _directories_sbs:
+
+ * Create directories for sources and build artifacts::
+
+ mkdir ~/linux/ ~/linux/sources ~/linux/build
+
+ [:ref:`details<directories>`]
+
+.. _sources_sbs:
+
+ * Retrieve the sources of the Linux version you intend to build; then change
+ into the directory holding them, as all further commands in this guide are
+ meant to be executed from there.
+
+ If you plan to only build one particular kernel version, download its source
+ archive from https://kernel.org; afterwards extract its content to '~/linux/'
+ and change into the directory created during extraction.
+
+ In most other situations your best choice is to fetch the sources using git::
+
+ git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git \
+ ~/linux/sources
+ cd ~/linux/sources/
+
+ Now you can check out any mainline release with a command like
+ ``git checkout --detach v6.1``; pre-release like v6.2-rc2 work, too.
+ Specifying 'origin/master' will hand you the latest mainline code, which is
+ automatically checked out by the above 'git clone […]' command.
+
+ In case you want to build a stable or longterm kernel, run this, too::
+
+ git remote add linux-stable \
+ https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
+ git fetch --all
+
+ Afterwards you can use a command like ``git checkout --detach v6.1.5`` to
+ access any such release.
+
+ This flexibility comes at a cost, as cloning the sources like this will
+ download quite a bit of data: roundabout 2,5 Gigabyte for mainline and 1,8
+ for stable/longterm releases as of early 2022. See the reference section for
+ a somewhat more complicated approach which is also using git, but downloads
+ only slightly more data than downloading a compressed tarball would.
+
+ [:ref:`details<sources>`]
+
+.. _patching_sbs:
+
+ * In case you want to apply a kernel patch, do so now. Often a command like
+ this will do the trick::
+
+ patch -p1 < ../proposed-fix.patch
+
+ If the '-p1' is actually needed, depends on how the patch was created; in
+ case it doesn't apply thus try without it.
+
+ If you cloned the sources with git and anything goes sideways, run ``git
+ reset --hard`` to undo any changes to the sources.
+
+ [:ref:`details<patching>`]
+
+.. _tagging_sbs:
+
+ * If you patched your kernel or already have that kernel version installed,
+ better tag your kernel by extending its release name (the one 'uname -r'
+ prints when executed under a kernel)::
+
+ sed -i 's!^EXTRAVERSION =.*$!&-proposed-fix!' Makefile
+
+ [:ref:`details<tagging>`]
+
+ .. _configuration_sbs:
+
+ * Create the build configuration for your kernel based on an existing
+ configuration.
+
+ If you already prepared such a '.config' file yourself, copy it to
+ ~/linux/build/ and run ``make O=~/linux/build/ olddefconfig``.
+
+ Use the same command, if your distribution or somebody else already tailored
+ your running kernel to your or your hardware's needs: the make target
+ olddefconfig will then try to use that kernel's .config as base.
+
+ Using this make target is fine for everybody else, too -- but you often can
+ save a lot of time by using this command instead::
+
+ yes "" | make O=~/linux/build/ localmodconfig
+
+ This will also try to pick your distribution's kernel as base, but then
+ disable modules for any features apparently superfluous for your setup. This
+ will later reduce the compile time enormously, especially if you are running
+ an universal kernel from a mainstream Linux distribution.
+
+ There is a catch: localmodconfig will disable kernel features you haven't
+ actually utilized through some action or program since you booted the system.
+ You can reduce or even eliminate that risk by using tricks outlined in the
+ reference section; for quick testing purposes that risk is often negligible,
+ but it's an aspect you want to keep in mind in case your kernel behaves
+ oddly.
+
+ [:ref:`details<configuration>`]
+
+.. _configmods_sbs:
+
+ * Check if you might want to or have to adjust some kernel configuration
+ options:
+
+ * Evaluate how you want to handle debug symbols. Enable them, if you later
+ might need to decode a stack trace found for example in a 'panic', 'Oops',
+ 'warning', or 'BUG'; on the other hand disable them, if you are short on
+ storage space or prefer a smaller kernel binary. See the reference section
+ for details on how to do either. If neither applies, it'll likely be fine
+ to simply not bother with this. [:ref:`details<configmods_debugsymbols>`]
+
+ * Are you running Debian? Then to avoid known problems you need to perform
+ additional adjustments explained in the reference section.
+ [:ref:`details<configmods_distros>`].
+
+.. _build_sbs:
+
+ * Build the image and the modules of your kernel::
+
+ make -j $(nproc --all) O=~/linux/build/
+
+ [:ref:`details<build>`]
+
+.. _install_sbs:
+
+ * Now install your kernel::
+
+ command -v installkernel && sudo make O=~/linux/build/ modules_install install
+
+ Often all left for you to do afterwards is a ``reboot``, as many commodity
+ Linux distributions will create an initramfs (also known as initrd) and an
+ entry for your kernel in your boot-loader's configuration when you execute
+ above command; but on some distributions you have to take care of these two
+ steps manually for reasons the reference section explains.
+
+ On a few distributions like Arch Linux and its derivatives the above command
+ does nothing at all; in that case you have to manually install your kernel,
+ as outlined in the reference section.
+
+ [:ref:`details<install>`]
+
+.. _another_sbs:
+
+ * To later build another kernel you need similar, but sometimes slightly
+ different commands.
+
+ In case you cloned the sources using git and want to build the latest
+ mainline codebase or update to a version released in between? Then first
+ update your git clone::
+
+ git fetch --all
+
+ To build another version or codebase you will have to first discard any local
+ modification, for example if you applied a patch or a tag as advertised; only
+ then you can switch to the code you are interested in::
+
+ git reset --hard
+ git checkout origin/master
+
+ At this point you might want to patch or tag the sources again, as explained
+ above. Afterwards adjust the build configuration to the new environment and
+ build your next kernel::
+
+ # reminder: if you want to apply patches, do it at this point
+ # reminder: at this point it's recommended to tag your build
+ make O=~/linux/build/ olddefconfig
+ make -j $(nproc --all) O=~/linux/build/
+
+ Install this kernel as described earlier; most of the time this command thus
+ will do the trick::
+
+ command -v installkernel && sudo make O=~/linux/build/ modules_install install
+
+ [:ref:`details<another>`]
+
+.. _uninstall_sbs:
+
+ * All parts of your installed kernel are normally identifiable by its release
+ name, which make them easy to locate and remove later. When doing so, ensure
+ to not remove the kernel you are running, as that might render your system
+ unbootable.
+
+ Start by deleting the directory holding your kernel's modules, which is named
+ after its release name -- '6.0.1-foobar' in the following example::
+
+ sudo rm -rf /lib/modules/6.9.1-foobar
+
+ Now try the following command, which on some distributions will delete all
+ other kernel files installed and remove the kernel's entry from the boot
+ loader configuration::
+
+ command -v kernel-install && sudo kernel-install -v remove 6.0.1-foobar
+
+ If that command doesn't output anything or fails, see the reference section;
+ do the same if any files named '*6.0.1-foobar*' remain in /boot/.
+
+ [:ref:`details<uninstall>`]
+
+.. _submit_improvements:
+
+Did you run into trouble following any of the above steps that is not cleared up
+by the reference section below? Or do you have ideas how to improve the text?
+Then please take a moment of your time and let the maintainer of this document
+know by email (Thorsten Leemhuis <[email protected]>), ideally while CCing the
+Linux docs mailing list ([email protected]). Such feedback is vital to
+improve this document further, which is in everybody's interest, as it will
+enable more people to master the task described here.
+
+Reference section for the step-by-step guide
+============================================
+
+This section holds additional information for each of the steps in the above
+guide.
+
+.. _backup:
+
+Prepare for emergencies
+-----------------------
+
+ *Create a fresh backup and put system repair and restore tools at hand*
+ [:ref:`... <backup_sbs>`]
+
+Remember, you are dealing with computers, which sometimes do unexpected things
+-- especially if you fiddle with crucial parts like the kernel of an operating
+system. That's what you are about to do in this process. Hence, better prepare
+for something going sideways, even if that shouldn't happen.
+
+[:ref:`back to step-by-step guide <backup_sbs>`]
+
+.. _secureboot:
+
+Dealing with techniques like Secure Boot
+----------------------------------------
+
+ *On platforms with 'Secure Boot' or similar techniques, prepare everything to
+ ensure the system will permit your self-compiled kernel to boot later.*
+ [:ref:`... <secureboot_sbs>`]
+
+Many modern systems allow only certain operating systems to start; they thus by
+default will reject booting self-compiled kernels.
+
+You ideally deal with this by making your platform trust your self-built kernels
+with the help of a certificate and signing. How to do that is not described
+here, as it requires various steps that would take the text too far away from
+its purpose; but you'll find many pages on the web that explain this in detail.
+
+Temporarily disabling solutions like Secure Boot is another way to make your own
+Linux boot. On commodity x86 systems it's possible to do this in the BIOS Setup
+utility; the steps to do so are not described here, as they greatly vary between
+machines.
+
+On mainstream x86 Linux distributions there is a third and universal option:
+disable all Secure Boot restrictions for your Linux environment. You can
+initiate this process by running ``mokutil --disable-validation``; this will
+tell you to create a one-time password, which is safe to write down. Now
+restart; right after your BIOS performed all self-tests the bootloader Shim will
+show a blue box with a message 'Press any key to perform MOK management'. Hit
+some key before the countdown exposes. This will open a menu and choose 'Change
+Secure Boot state' there. Shim's 'MokManager' will now ask you to enter three
+randomly chosen characters from the one-time password specified earlier. Once
+you provided them, confirm that you really want to disable the validation.
+Afterwards, permit MokManager to reboot the machine.
+
+[:ref:`back to step-by-step guide <secureboot_sbs>`]
+
+.. _buildrequires:
+
+Install build requirements
+--------------------------
+
+ *Install all software required to build a Linux kernel.*
+ [:ref:`...<buildrequires_sbs>`]
+
+The kernel is pretty stand-alone, but besides tools like the compiler you'll
+sometimes need a few libraries to build one. How to install everything needed
+depends on your Linux distribution and the configuration of the kernel you are
+about to build.
+
+Here are a few examples what you typically need on some mainstream
+distributions:
+
+ * Debian, Ubuntu, and derivatives::
+
+ sudo apt install bc binutils bison dwarves flex gcc git make openssl \
+ pahole perl-base libssl-dev libelf-dev
+
+ * Fedora and derivatives::
+
+ sudo dnf install binutils /usr/include/{libelf.h,openssl/pkcs7.h} \
+ /usr/bin/{bc,bison,flex,gcc,git,openssl,make,perl,pahole}
+
+ * openSUSE and derivatives::
+
+ sudo zypper install bc binutils bison dwarves flex gcc git make perl-base \
+ openssl openssl-devel libelf-dev
+
+In case you wonder why these lists include openssl and its development headers:
+they are needed for the Secure Boot support, which many distributions enable in
+their kernel configuration for x86 machines.
+
+Sometimes you'll need tools for compression formats like bzip2, gzip, lz4, lzma,
+lzo, xz, or zstd as well.
+
+You might need additional libraries and their development headers in case you
+perform tasks not covered in this guide. For example, zlib will be needed when
+building kernel tools from the tools/ directory; make targets like 'menuconfig'
+or 'xconfig' will require development headers for ncurses or Qt.
+
+[:ref:`back to step-by-step guide <buildrequires_sbs>`]
+
+.. _diskspace:
+
+Space requirements
+------------------
+
+ *Ensure to have enough free space for building and installing Linux.*
+ [:ref:`... <diskspace_sbs>`]
+
+The numbers mentioned are rough estimates with a big extra charge to be on the
+safe side, so often you will need less.
+
+If you have space constraints, remember to read the reference section for the
+steps :ref:`Downloading the Linux sources<sources>` and :ref:`Debug symbols in
+the section about configuration adjustments' <configmods>`: both mention tricks
+that reduce the consumed disk space by quite a lot.
+
+[:ref:`back to step-by-step guide <diskspace_sbs>`]
+
+
+.. _directories:
+
+Create dedicated directories
+----------------------------
+
+ *Create directories for sources and build artifacts:*
+ [:ref:`...<directories_sbs>`]
+
+You could build your kernel directly in the source tree, but it is good practice
+to keep it clean by storing the build artifacts separately. That's why this
+guide recommends creating a directory for the latter, which you hence must
+specify in later commands. Most of the time this is done by passing
+'O=~/linux/build/' to make.
+
+[:ref:`back to step-by-step guide <directories_sbs>`]
+
+.. _sources:
+
+Download the sources
+--------------------
+
+ *Retrieve the sources of the Linux version you intend to build.*
+ [:ref:`...<sources_sbs>`]
+
+Fetching the complete stable git repository as explained in the step-by-step
+guide above is easy and offers the most flexibility, but also downloads quite a
+bit of data (more than 4 Gigabyte as of late 2022). If you want something
+lighter, you instead might want to create a 'shallow clone' that omits some of
+the history and only contains stable series you are interested in::
+
+ git clone --no-checkout \
+ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git \
+ --depth 1 -b v6.0 ~/linux/sources/
+ cd ~/linux/sources/
+ git remote set-branches --add origin master
+ git remote add -t linux-6.1.y linux-stable \
+ https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
+ git fetch --all --shallow-exclude=v6.0
+
+The initial command creates a clone with the sources of just Linux v6.0; this
+downloads just a little more data than retrieving a xz-compressed tarball of
+that version would (~230 vs 204 Megabyte); afterwards the master branch is added
+to the list of tracked branches, which allows you to retrieve anything state of
+mainline since Linux 6.0 was released. The linux-stable repository is then added
+as a remote which just tracks the linux-6.1.y branch. In early 2022 as of Linux
+v6.2-rc4 the last two steps downloaded ~95 and ~2 Megabyte.
+
+With this tree you can check out the latest mainline code using ``git checkout
+--detach origin/master``. All versions and individual changes released between
+6.0 and the latest 6.1.y version are at hand, too -- this tree thus also allows
+you to bisect a problem introduced in that range.
+
+Later run above 'git fetch [...]' command again, when you want to retrieve the
+latest mainline codebase or access 6.1.y versions released in between. If you
+want to switch to a release from a newer stable series (say Linux v6.2.y), you
+first need to add its remote branch with a command like ``git remote
+set-branches --add linux-stable linux-6.2.y``; once you run 'git fetch [...]'
+again you'll be able to check out versions from that series.
+
+[:ref:`back to step-by-step guide <sources_sbs>`]
+
+.. _patching:
+
+Patch the sources (optional)
+----------------------------
+
+ *In case you want to apply a kernel patch, do so now.*
+ [:ref:`...<patching_sbs>`]
+
+This is the point where you might want to patch your kernel -- for example when
+a developer proposed a fix and asked you to check if it helps. The step-by-step
+guide already explains everything crucial here.
+
+[:ref:`back to step-by-step guide <patching_sbs>`]
+
+.. _tagging:
+
+Tagging this kernel build (optional, often wise)
+------------------------------------------------
+
+ *If you patched your kernel or already have that kernel version installed,
+ better tag your kernel by extending its release name:*
+ [:ref:`...<tagging_sbs>`]
+
+Tagging your kernel will help avoid confusion later, especially when you patched
+your kernel. Adding an individual tag will also ensure the kernel's image and
+its modules are installed in parallel to any existing kernels.
+
+The step-by-step guide uses a sed command to create a descriptive tag to your
+kernel's release name by adding it to the EXTRAVERSION line at the start of the
+main Makefile. To remove the tag later, simply delete it; if you cloned the
+sources with git, you can do this by discarding any changes to the Makefile by
+running ``git checkout Makefile``.
+
+[:ref:`back to step-by-step guide <tagging_sbs>`]
+
+.. _configuration:
+
+Define the build configuration for your kernel
+----------------------------------------------
+
+ *Create the build configuration for your kernel based on an existing
+ configuration.* [:ref:`... <configuration_sbs>`]
+
+There are various aspects for this steps that require a more careful
+explanation:
+
+Pitfalls when using another configuration file as base
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Make targets like localmodconfig and olddefconfig share a few common snares you
+want to be aware of:
+
+ * These targets will reuse a kernel build configuration in your build directory
+ (e.g. '~/linux/build/.config'), if one exists. In case you want to start from
+ scratch you thus need to delete it.
+
+ * The make targets try to find the configuration for your running kernel
+ automatically, but might choose poorly. A line like '# using defaults found in
+ /boot/config-6.0.7-250.fc36.x86_64' or 'using config:
+ '/boot/config-6.0.7-250.fc36.x86_64' tells you which file they picked. If that
+ is not the intended one, simply store it as '~/linux/build/.config' before
+ using these make targets.
+
+ * Unexpected things might happen if you try to use a config file prepared for
+ one kernel (say v6.0) on an older generation (say v5.15). In that case you
+ might want to use a configuration as base which your distribution utilized when
+ they used that or an slightly older kernel version.
+
+Influencing the configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The make target 'olddefconfig' and the ``yes "" |`` used when utilizing
+localmodconfig will set any undefined build options to their default value. This
+among others will disable many kernel features that were introduced after your
+base kernel was released.
+
+If you want to set these configurations options manually, use 'oldconfig'
+instead of 'olddefconfig' or omit the ``yes "" |`` when utilizing
+localmodconfig. Then for each undefined configuration option you will be asked
+how to proceed. In case you are unsure what to answer, simply hit 'enter' to
+apply the default value.
+
+Big pitfall when using localmodconfig
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As explained briefly in the step-by-step guide already: with localmodconfig it
+can easily happen that your self-built kernel will lack modules for tasks you
+didn't perform before utilizing this make target. That's because those tasks
+require kernel modules that are normally autoloaded when you perform that task
+for the first time; if you didn't perform that task at least once before using
+localmodonfig, the latter will thus assume these modules are superfluous and
+disable them.
+
+You can try to avoid this by performing typical tasks that often will autoload
+additional kernel modules: start a VM, establish VPN connections, loop-mount a
+CD/DVD ISO, mount network shares (CIFS, NFS, …), and connect all external
+devices (2FA keys, headsets, webcams, …) as well as storage devices with file
+systems you otherwise don't utilize (btrfs, ext4, FAT, NTFS, XFS, …). But it's
+hard to think of everything that might be needed -- even kernel developers often
+forget one thing or another at this point.
+
+Don't let that risk bother you, especially when compiling a kernel only for
+testing purposes: everything typically crucial will be there. And if you forget
+something important you can turn on a missing feature later and quickly run the
+commands to compile and install a better kernel.
+
+But if you plan to build and use self-built kernels regularly, you might want to
+reduce the risk by recording which modules your system loads over the course of
+a few weeks. You can automate this with `modprobed-db
+<https://github.com/graysky2/modprobed-db>`_. Afterwards use 'LSMOD=<path>' to
+point localmodconfig to the list of modules modprobed-db noticed being used::
+
+ yes "" | make O=~/linux/build/ LSMOD="${HOME}"/.config/modprobed.db localmodconfig
+
+Remote building with localmodconfig
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to use 'localmodconfig' to build a kernel for another machine, run
+``lsmod > lsmod_foo-machine`` on it and transfer that file to your build host.
+Now point the build scripts to the file like this: ``yes "" | make
+O=~/linux/build/ LSMOD=~/lsmod_foo-machine localmodconfig``. Note, in this case
+you likely want to copy a base kernel configuration from the other machine over
+as well and place it as .config in your build directory.
+
+[:ref:`back to step-by-step guide <configuration_sbs>`]
+
+.. _configmods:
+
+Adjust build configuration
+--------------------------
+
+ *Check if you might want to or have to adjust some kernel configuration
+ options:*
+
+Depending on your needs you at this point might want or have to adjust some
+kernel configuration options.
+
+.. _configmods_debugsymbols:
+
+Debug symbols
+~~~~~~~~~~~~~
+
+ *Evaluate how you want to handle debug symbols.*
+ [:ref:`...<configmods_sbs>`]
+
+Most users don't need to care about this, it's often fine to leave everything as
+it is; but you should take a closer look at this, if you might need to decode a
+stack trace or want to reduce space consumption.
+
+Having debug symbols available can be important when your kernel throws a
+'panic', 'Oops', 'warning', or 'BUG' later when running, as then you will be
+able to find the exact place where the problem occurred in the code. But
+collecting and embedding the needed debug information takes time and consumes
+quite a bit of space: in late 2022 the build artifacts for a typical x86 kernel
+configured with localmodconfig consumed around 5 Gigabyte of space with debug
+symbols, but less than 1 when they were disabled. The resulting kernel image and
+the modules are bigger as well, which increases load times.
+
+Hence, if you want a small kernel and are unlikely to decode a stack trace
+later, you might want to disable debug symbols to avoid above downsides::
+
+ ./scripts/config --file ~/linux/build/.config -d DEBUG_INFO \
+ -d DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -d DEBUG_INFO_DWARF4 \
+ -d DEBUG_INFO_DWARF5 -e CONFIG_DEBUG_INFO_NONE
+ make O=~/linux/build/ olddefconfig
+
+You on the other hand definitely want to enable them, if there is a decent
+chance that you need to decode a stack trace later (as explained by 'Decode
+failure messages' in Documentation/admin-guide/tainted-kernels.rst in more
+detail)::
+
+ ./scripts/config --file ~/linux/build/.config -d DEBUG_INFO_NONE -e DEBUG_KERNEL
+ -e DEBUG_INFO -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT -e KALLSYMS -e KALLSYMS_ALL
+ make O=~/linux/build/ olddefconfig
+
+Note, many mainstream distributions enable debug symbols in their kernel
+configurations -- make targets like localmodconfig and olddefconfig thus will
+often pick that setting up.
+
+[:ref:`back to step-by-step guide <configmods_sbs>`]
+
+.. _configmods_distros:
+
+Distro specific adjustments
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ *Are you running* [:ref:`... <configmods_sbs>`]
+
+The following sections help you to avoid build problems that are known to occur
+when following this guide on a few commodity distributions.
+
+**Debian:**
+
+ * Remove a stale reference to a certificate file that would cause your build to
+ fail::
+
+ ./scripts/config --file ~/linux/build/.config --set-str SYSTEM_TRUSTED_KEYS ''
+
+ Alternatively, download the needed certificate and make that configuration
+ option point to it, as `the Debian handbook explains in more detail
+ <https://debian-handbook.info/browse/stable/sect.kernel-compilation.html>`_ .
+
+[:ref:`back to step-by-step guide <configmods_sbs>`]
+
+.. _build:
+
+Build your kernel
+-----------------
+
+ *Build the image and the modules of your kernel* [:ref:`... <build_sbs>`]
+
+A lot can go wrong at this stage that will abort the build process. Sometimes
+this is caused by problems on your side, which you'll often be able to fix
+quickly; sometimes though the problem lies in the code and can only be fixed by
+a developer.
+
+A close examination of the failure messages coupled with some research on the
+internet will often tell you which of the two it is. Before doing this, restart
+the build process like this::
+
+ make O=~/linux/build/ V=1
+
+The 'V=1' activates verbose output, which might be needed to see the actual
+error. To make it easier to spot, this command also omits the '-j $(nproc
+--all)' used earlier to utilize every CPU core in the system for the job. But
+this parallelism results in some clutter when failures occur, which makes the
+actual error message harder to spot.
+
+Once the failure happens again, try to find the most crucial line describing the
+problem. Then search the internet for the most important and non-generic section
+of that line (say 4 to 8 words); avoid or remove anything that looks remotely
+system-specific, like your username or local path names like '~/linux/build/'.
+First try your regular internet search engine with that string, afterwards
+search Linux kernel mailing lists via `lore.kernel.org/all/
+<https://lore.kernel.org/all/>`_.
+
+This most of the time will find something that will explain what's wrong; quite
+often one of the hits will provide a solution for your problem, too. If you
+don't find anything that matches your problem, try again from a different angle
+by modifying your search terms or using another line from the printed error
+messages.
+
+In the end, most trouble you are to run into has likely been encountered and
+reported by others already. That includes issues where the cause is not your
+system, but lies the code. If you run into one of those, you might thus find a
+solution (e.g. a patch) or workaround for your problem, too.
+
+[:ref:`back to step-by-step guide <build_sbs>`]
+
+.. _install:
+
+Install your kernel
+-------------------
+
+ *Now install your kernel* [:ref:`... <install_sbs>`]
+
+What you need to do after executing the command in the step-by-step guide
+depends on the existence and the implementation of an 'installkernel'
+executable. Many commodity Linux distributions ship such a kernel installer in
+'/sbin/' that does everything needed, hence there is nothing left for you except
+rebooting. But some distributions contain an installkernel that does only part
+of the job -- and a few lack it completely and leave all the work to you.
+
+If 'installkernel' is found, the kernel's build system will delegate the actual
+installation of your kernel's image and related files to this executable. On
+almost all Linux distributions it will store the image as '/boot/vmlinuz-<your
+kernel's release name>' and put a 'System.map-<your kernel's release name>'
+alongside it. Your kernel will thus be installed in parallel to any existing
+ones, unless you already have one with exactly the same release name.
+Installkernel on many distributions will afterwards generate an 'initramfs'
+(often also called 'initrd'), which commodity distributions rely on for booting;
+hence be sure to keep the order of the two make targets used in the step-by-step
+guide, as things will go sideways if you install your kernel's image before its
+modules. Often installkernel will then add your kernel to the boot loader
+configuration, too. You have to take care of one or both of these tasks
+yourself, if your distributions installkernel doesn't handle them.
+
+A few distributions like Arch Linux and its derivatives totally lack an
+installkernel executable. On those just install the modules using the kernel's
+build system and then install the image and the System.map file manually::
+
+ sudo make O=~/linux/build/ modules_install
+ sudo install -m 0600 ~/linux/build/$(make O=~/linux/build/ -s image_name) \
+ /boot/vmlinuz-$(make O=~/linux/build/ -s kernelrelease)
+ sudo install -m 0600 ~/linux/build/System.map \
+ /boot/System.map-$(make O=~/linux/build/ -s kernelrelease)
+
+If your distribution boots with the help of an initramfs, now generate one for
+your kernel using the tools your distribution provides for this process.
+Afterwards add your kernel to your boot-loader configuration and reboot.
+
+[:ref:`back to step-by-step guide <install_sbs>`]
+
+.. _another:
+
+Another round later
+-------------------
+
+ *To later build another kernel you need similar, but sometimes slightly
+ different commands* [:ref:`... <another_sbs>`]
+
+The process to build later kernels is similar, but at some points slightly
+different. You for example don't want to use 'localmodconfig' for succeeding
+kernel builds, as you already created a trimmed down configuration you want to
+use from now on. Hence instead just use 'oldconfig' or 'olddefconfig' to adjust
+your build configurations to the needs of the kernel version you are about to
+build.
+
+If you created a shallow-clone with git, remember what the :ref:`section that
+explained the setup described in more detail <sources>`: you need to use a
+slightly different 'git fetch' command and when switching to another series need
+to add an additional remote branch.
+
+[:ref:`back to step-by-step guide <another_sbs>`]
+
+.. _uninstall:
+
+Uninstall the kernel later
+--------------------------
+
+ *All parts of your installed kernel are identifiable by its release name and
+ thus easy to remove later.* [:ref:`... <uninstall_sbs>`]
+
+Don't worry installing your kernel manually and thus bypassing your
+distribution's packaging system will totally mess up your machine: all parts of
+your kernel are easy to remove later, as files are stored in two places only and
+normally identifiable by the kernel's release name.
+
+One of the two places is a directory in /lib/modules/, which holds the modules
+for each installed kernel. This directory is named after the kernel's release
+name; hence, to remove all modules for one of your kernels, simply remove its
+modules directory in /lib/modules/.
+
+The other place is /boot/, where typically one to five files will be placed
+during installation of a kernel. All of them usually contain the release name in
+their file name, but how many files and their name depends somewhat on your
+distribution's installkernel executable (:ref:`see above <install>`) and its
+initramfs generator. On some distributions the 'kernel-install' command
+mentioned in the step-by-step guide will remove all of these files for you --
+and the entry for your kernel in the boot-loader configuration at the same time,
+too. On others you have to take care of these steps yourself. The following
+command should interactively remove the two main files of a kernel with the
+release name '6.0.1-foobar'::
+
+ rm -i /boot/{System.map,vmlinuz}-6.0.1-foobar
+
+Now remove the belonging initramfs, which often will be called something like
+'/boot/initramfs-6.0.1-foobar.img' or '/boot/initrd.img-6.0.1-foobar'.
+Afterwards check for other files in /boot/ that have '6.0.1-foobar' in their
+name and delete them as well. Now remove the kernel from your boot-loader's
+configuration.
+
+Note, be very careful with wildcards like '*' when deleting files or directories
+for kernels manually: you might accidentally remove files of a 6.0.11 kernel
+when all you want is to remove 6.0 or 6.0.1.
+
+[:ref:`back to step-by-step guide <uninstall_sbs>`]
+
+.. _faq:
+
+FAQ
+===
+
+Why doesn't this 'how-to' work on my system?
+--------------------------------------------
+
+As initially stated, this guide is "designed to cover everything typically
+needed [to build a kernel] on mainstream Linux distributions running on
+commodity PC or server hardware". The outlined approach despite this should work
+on many other setups as well. But trying to cover every possible use-case in one
+guide would defeat its purpose, as without such a focus you'd need dozens or
+hundreds of constructs along the lines of "in case you are having <insert
+machine or distro>, you at this point have to do <this and that>
+<instead|additionally>". Each of which would make the text longer, more
+complicated, and harder to follow.
+
+That being said: this of course is a balancing act. Hence, if you think an
+additional use-case is worth describing, suggest it to the maintainers of this
+document (:ref:`see above <submit_improvements>`).
+
+
+..
+ end-of-content
+..
+ This document is maintained by Thorsten Leemhuis <[email protected]>. If
+ you spot a typo or small mistake, feel free to let him know directly and
+ he'll fix it. You are free to do the same in a mostly informal way if you
+ want to contribute changes to the text -- but for copyright reasons please CC
+ [email protected] and 'sign-off' your contribution as
+ Documentation/process/submitting-patches.rst explains in the section 'Sign
+ your work - the Developer's Certificate of Origin'.
+..
+ This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
+ of the file. If you want to distribute this text under CC-BY-4.0 only,
+ please use 'The Linux kernel development community' for author attribution
+ and link this as source:
+ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/howto-quickly-build-linux.rst
+..
+ Note: Only the content of this RST file as found in the Linux kernel sources
+ is available under CC-BY-4.0, as versions of this text that were processed
+ (for example by the kernel's build system) might contain content taken from
+ files which use a more restrictive license.
+
diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst
index 5bfafcbb9562..e11cd4bcfb9c 100644
--- a/Documentation/admin-guide/index.rst
+++ b/Documentation/admin-guide/index.rst
@@ -37,6 +37,7 @@ problems and bugs in particular.
reporting-issues
reporting-regressions
security-bugs
+ howto-quickly-build-linux
bug-hunting
bug-bisect
tainted-kernels
diff --git a/MAINTAINERS b/MAINTAINERS
index 7f86d02cb427..3d6da1f76026 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6329,6 +6329,7 @@ DOCUMENTATION REPORTING ISSUES
M: Thorsten Leemhuis <[email protected]>
L: [email protected]
S: Maintained
+F: Documentation/admin-guide/howto-quickly-build-linux.rst
F: Documentation/admin-guide/reporting-issues.rst
DOCUMENTATION SCRIPTS
base-commit: a16abc68e66c86ca9d80c4a04a2103ef7ed1a40f
--
2.39.1
On Wed, Feb 01, 2023 at 12:52:30PM +0100, Thorsten Leemhuis wrote:
> Add a text explaining how to quickly build a kernel, as that's something
> users will often have to do when they want to report an issue or test
> proposed fixes. This is a huge and frightening task for quite a few
> users these days, as many rely on pre-compiled kernels and have never
> built their own. They find help on quite a few websites explaining the
> process in various ways, but those howtos often omit important details
> or make things too hard for the 'quickly build just for testing' case
> that 'localmodconfig' is really useful for. Hence give users something
> at hand to guide them, as that makes it easier for them to help with
> testing, debugging, and fixing the kernel.
First off, this is great, thanks for doing this.
One minor comment, to prevent people from "overloading" the
git.kernel.org systems:
> +.. _sources_sbs:
> +
> + * Retrieve the sources of the Linux version you intend to build; then change
> + into the directory holding them, as all further commands in this guide are
> + meant to be executed from there.
> +
> + If you plan to only build one particular kernel version, download its source
> + archive from https://kernel.org; afterwards extract its content to '~/linux/'
> + and change into the directory created during extraction.
> +
> + In most other situations your best choice is to fetch the sources using git::
> +
> + git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git \
> + ~/linux/sources
> + cd ~/linux/sources/
Ideally you should never do a "full clone from scratch" like this, as it
takes up loads of server resources. The "best" way to do this is to
download the kernel git bundle, and then pull and resolve the remaining
bits. It's explained, with a script to do the work for you, here:
https://kernel.org/best-way-to-do-linux-clones-for-your-ci.html
Using that will save you time (the CI bundles are mirrored around the
world), and greatly reduce the server load (which is already high
enough.)
thanks,
greg k-h
On Wed, Feb 01, 2023 at 01:42:29PM +0100, Greg KH wrote:
> On Wed, Feb 01, 2023 at 12:52:30PM +0100, Thorsten Leemhuis wrote:
> > Add a text explaining how to quickly build a kernel, as that's something
> > users will often have to do when they want to report an issue or test
> > proposed fixes. This is a huge and frightening task for quite a few
> > users these days, as many rely on pre-compiled kernels and have never
> > built their own. They find help on quite a few websites explaining the
> > process in various ways, but those howtos often omit important details
> > or make things too hard for the 'quickly build just for testing' case
> > that 'localmodconfig' is really useful for. Hence give users something
> > at hand to guide them, as that makes it easier for them to help with
> > testing, debugging, and fixing the kernel.
>
> First off, this is great, thanks for doing this.
>
> One minor comment, to prevent people from "overloading" the
> git.kernel.org systems:
>
> > +.. _sources_sbs:
> > +
> > + * Retrieve the sources of the Linux version you intend to build; then change
> > + into the directory holding them, as all further commands in this guide are
> > + meant to be executed from there.
> > +
> > + If you plan to only build one particular kernel version, download its source
> > + archive from https://kernel.org; afterwards extract its content to '~/linux/'
> > + and change into the directory created during extraction.
> > +
> > + In most other situations your best choice is to fetch the sources using git::
> > +
> > + git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git \
> > + ~/linux/sources
> > + cd ~/linux/sources/
>
> Ideally you should never do a "full clone from scratch" like this, as it
> takes up loads of server resources. The "best" way to do this is to
> download the kernel git bundle, and then pull and resolve the remaining
> bits. It's explained, with a script to do the work for you, here:
>
> https://kernel.org/best-way-to-do-linux-clones-for-your-ci.html
Oops, here's the full steps involved:
https://www.kernel.org/cloning-linux-from-a-bundle.html
the first link above has a script that does it all for you, but you
probably just want to copy the steps at this last link instead.
thanks,
greg k-h
On 01.02.23 13:44, Greg KH wrote:
> On Wed, Feb 01, 2023 at 01:42:29PM +0100, Greg KH wrote:
>> On Wed, Feb 01, 2023 at 12:52:30PM +0100, Thorsten Leemhuis wrote:
>>> Add a text explaining how to quickly build a kernel, as that's something
>>> users will often have to do when they want to report an issue or test
>>> proposed fixes. This is a huge and frightening task for quite a few
>>> users these days, as many rely on pre-compiled kernels and have never
>>> built their own. They find help on quite a few websites explaining the
>>> process in various ways, but those howtos often omit important details
>>> or make things too hard for the 'quickly build just for testing' case
>>> that 'localmodconfig' is really useful for. Hence give users something
>>> at hand to guide them, as that makes it easier for them to help with
>>> testing, debugging, and fixing the kernel.
>>
>> First off, this is great, thanks for doing this.
Thx, feels good to hear.
Bisection is next on my todo list once this matured...
>> One minor comment, to prevent people from "overloading" the
>> git.kernel.org systems:
>>
>>> +.. _sources_sbs:
>>> +
>>> + * Retrieve the sources of the Linux version you intend to build; then change
>>> + into the directory holding them, as all further commands in this guide are
>>> + meant to be executed from there.
>>> +
>>> + If you plan to only build one particular kernel version, download its source
>>> + archive from https://kernel.org; afterwards extract its content to '~/linux/'
>>> + and change into the directory created during extraction.
>>> +
>>> + In most other situations your best choice is to fetch the sources using git::
>>> +
>>> + git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git \
>>> + ~/linux/sources
>>> + cd ~/linux/sources/
>>
>> Ideally you should never do a "full clone from scratch" like this, as it
>> takes up loads of server resources. The "best" way to do this is to
>> download the kernel git bundle, and then pull and resolve the remaining
>> bits. It's explained, with a script to do the work for you, here:
>>
>> https://kernel.org/best-way-to-do-linux-clones-for-your-ci.html
>
> Oops, here's the full steps involved:
> https://www.kernel.org/cloning-linux-from-a-bundle.html
> the first link above has a script that does it all for you, but you
> probably just want to copy the steps at this last link instead.
Great idea, thx for bringing this up -- now that you mention it, I
remember those pages and the script again... :-/
Sadly my "after cloning Linus tree, add the stable tree as remote and
fetch everything" approach would will still create a lot of load. I
could use the script with the stable git repo, as that includes
mainline; but I noticed it sometimes is not fully up2date. At least I
once noticed it was quite a few hours (or maybe even a day?) behind. Is
that normal? I assume you should know.
Anyway: maybe "a little bit behind" isn't something that's much of a
problem for this document.
Ciao, Thorsten
On Wed, Feb 01, 2023 at 02:22:07PM +0100, Thorsten Leemhuis wrote:
> On 01.02.23 13:44, Greg KH wrote:
> > On Wed, Feb 01, 2023 at 01:42:29PM +0100, Greg KH wrote:
> >> On Wed, Feb 01, 2023 at 12:52:30PM +0100, Thorsten Leemhuis wrote:
> >>> Add a text explaining how to quickly build a kernel, as that's something
> >>> users will often have to do when they want to report an issue or test
> >>> proposed fixes. This is a huge and frightening task for quite a few
> >>> users these days, as many rely on pre-compiled kernels and have never
> >>> built their own. They find help on quite a few websites explaining the
> >>> process in various ways, but those howtos often omit important details
> >>> or make things too hard for the 'quickly build just for testing' case
> >>> that 'localmodconfig' is really useful for. Hence give users something
> >>> at hand to guide them, as that makes it easier for them to help with
> >>> testing, debugging, and fixing the kernel.
> >>
> >> First off, this is great, thanks for doing this.
>
> Thx, feels good to hear.
>
> Bisection is next on my todo list once this matured...
>
> >> One minor comment, to prevent people from "overloading" the
> >> git.kernel.org systems:
> >>
> >>> +.. _sources_sbs:
> >>> +
> >>> + * Retrieve the sources of the Linux version you intend to build; then change
> >>> + into the directory holding them, as all further commands in this guide are
> >>> + meant to be executed from there.
> >>> +
> >>> + If you plan to only build one particular kernel version, download its source
> >>> + archive from https://kernel.org; afterwards extract its content to '~/linux/'
> >>> + and change into the directory created during extraction.
> >>> +
> >>> + In most other situations your best choice is to fetch the sources using git::
> >>> +
> >>> + git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git \
> >>> + ~/linux/sources
> >>> + cd ~/linux/sources/
> >>
> >> Ideally you should never do a "full clone from scratch" like this, as it
> >> takes up loads of server resources. The "best" way to do this is to
> >> download the kernel git bundle, and then pull and resolve the remaining
> >> bits. It's explained, with a script to do the work for you, here:
> >>
> >> https://kernel.org/best-way-to-do-linux-clones-for-your-ci.html
> >
> > Oops, here's the full steps involved:
> > https://www.kernel.org/cloning-linux-from-a-bundle.html
> > the first link above has a script that does it all for you, but you
> > probably just want to copy the steps at this last link instead.
>
> Great idea, thx for bringing this up -- now that you mention it, I
> remember those pages and the script again... :-/
>
> Sadly my "after cloning Linus tree, add the stable tree as remote and
> fetch everything" approach would will still create a lot of load.
Not really, nothing like doing a "full" clone of the original repo.
It's a much smaller working set that needs to be handled that way on the
server side.
I do this (as do many others) for CI systems, and the primary workload
is on the client resolving the bundle, that can take a bit of time, but
that's all client side, not server.
> I
> could use the script with the stable git repo, as that includes
> mainline; but I noticed it sometimes is not fully up2date. At least I
> once noticed it was quite a few hours (or maybe even a day?) behind. Is
> that normal? I assume you should know.
I think it's generated daily, so yes, it will be a bit out of date, but
that's fine.
> Anyway: maybe "a little bit behind" isn't something that's much of a
> problem for this document.
Adding the remote and doing a pull is good and recommended, so it's fine
if it's a day or so out of date.
thanks,
greg k-h
[adding Konstantin and Greg to the list of recipients]
On 01.02.23 12:52, Thorsten Leemhuis wrote:
> Add a text explaining how to quickly build a kernel, as that's something
> users will often have to do when they want to report an issue or test
> proposed fixes. This is a huge and frightening task for quite a few
> users these days, as many rely on pre-compiled kernels and have never
> built their own. They find help on quite a few websites explaining the
> process in various ways, but those howtos often omit important details
> or make things too hard for the 'quickly build just for testing' case
> that 'localmodconfig' is really useful for. Hence give users something
> at hand to guide them, as that makes it easier for them to help with
> testing, debugging, and fixing the kernel.
Side note: after feedback on social media I'll likely switch to a title
like "how to quickly configure & build a trimmed-down Linux kernel", as
some people from the current title assumed this would be about things
like ccache. I'll also likely will switch to using a localversion file
in the buildroot instead of modifying the EXTRAVERSION in the top-level
makefile (but I haven't actually tried it yet).
> [...]
>
> The text currently describes two approaches to retrieve Linux' sources
> using git: the regular clone with linux-stable as a remote and a shallow
> clone with just one branch from linux-stable. The shallow clone approach
> is a little bit more tricky to describe and handle, but downloads way
> less data – and thus is a lot quicker, unless you have a really really
> quick link to the internet (which in some parts of the world is hard to
> come by). That's why I wonder if the text should switch to making the
> shallow clone with selected stable branches the default. What do you
> think, dear reader?
So, I looked into what Greg suggested (e.g.
https://kernel.org/best-way-to-do-linux-clones-for-your-ci.html and
https://www.kernel.org/cloning-linux-from-a-bundle.html
). Assuming users have a up2date git (afaics 2.38+) I could use commands
like this in my text:
curl -L
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/clone.bundle
-o ~/linux/linux-stable.git.bundle
git clone --bundle-uri=linux-stable.git.bundle
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
~/linux/sources
rm ~/linux/linix-stable.git.bundle
This took roundabout 16 minutes with my 100 Mbit cable internet
connection (~9 min for the download, 7 for the clone [the machine used
is somewhat old]) and downloads & stores ~4,5 GByte data (without checkout).
[side note: using
"--bundle-uri=https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/clone.bundle"
does not work (due to the redirect? whatever) -- but that might be
unwise anyway in case the download is interrupted]
Then I tried creating a shallow clone like this:
git clone
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
--depth 1 -b v6.1
git remote set-branches --add origin master
git fetch --all --shallow-exclude=v6.1
git remote add -t linux-6.1.y linux-stable
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
git fetch --all --shallow-exclude=v6.1
This took only roundabout 2 minutes and downloads & stores ~512 MByte
data (without checkout).
Not totally sure, but the shallow clone somehow feels more appropriate
for the use case (reminder, there is a "quickly" in the document title),
even if such a clone is less flexible (e.g. users have to manually add
stable branches they are interested it; and they need to be careful when
using git fetch).
That's why I now strongly consider using the shallow clone method by
default in v2 of this text. Or does that also create a lot of load on
the servers? Or are there other strong reason why using a shallow clone
might be a bad idea for this use case?
Ciao, Thorsten
> [...]
> +.. _sources_sbs:
> +
> + * Retrieve the sources of the Linux version you intend to build; then change
> + into the directory holding them, as all further commands in this guide are
> + meant to be executed from there.
> +
> + If you plan to only build one particular kernel version, download its source
> + archive from https://kernel.org; afterwards extract its content to '~/linux/'
> + and change into the directory created during extraction.
> +
> + In most other situations your best choice is to fetch the sources using git::
> +
> + git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git \
> + ~/linux/sources
> + cd ~/linux/sources/
> +
> + Now you can check out any mainline release with a command like
> + ``git checkout --detach v6.1``; pre-release like v6.2-rc2 work, too.
> + Specifying 'origin/master' will hand you the latest mainline code, which is
> + automatically checked out by the above 'git clone […]' command.
> +
> + In case you want to build a stable or longterm kernel, run this, too::
> +
> + git remote add linux-stable \
> + https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> + git fetch --all
> +
> + Afterwards you can use a command like ``git checkout --detach v6.1.5`` to
> + access any such release.
> +
> + This flexibility comes at a cost, as cloning the sources like this will
> + download quite a bit of data: roundabout 2,5 Gigabyte for mainline and 1,8
> + for stable/longterm releases as of early 2022. See the reference section for
> + a somewhat more complicated approach which is also using git, but downloads
> + only slightly more data than downloading a compressed tarball would.
> +
> + [:ref:`details<sources>`]
> +
> [...]
> +
> +.. _sources:
> +
> +Download the sources
> +--------------------
> +
> + *Retrieve the sources of the Linux version you intend to build.*
> + [:ref:`...<sources_sbs>`]
> +
> +Fetching the complete stable git repository as explained in the step-by-step
> +guide above is easy and offers the most flexibility, but also downloads quite a
> +bit of data (more than 4 Gigabyte as of late 2022). If you want something
> +lighter, you instead might want to create a 'shallow clone' that omits some of
> +the history and only contains stable series you are interested in::
> +
> + git clone --no-checkout \
> + https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git \
> + --depth 1 -b v6.0 ~/linux/sources/
> + cd ~/linux/sources/
> + git remote set-branches --add origin master
> + git remote add -t linux-6.1.y linux-stable \
> + https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> + git fetch --all --shallow-exclude=v6.0
> +
> +The initial command creates a clone with the sources of just Linux v6.0; this
> +downloads just a little more data than retrieving a xz-compressed tarball of
> +that version would (~230 vs 204 Megabyte); afterwards the master branch is added
> +to the list of tracked branches, which allows you to retrieve anything state of
> +mainline since Linux 6.0 was released. The linux-stable repository is then added
> +as a remote which just tracks the linux-6.1.y branch. In early 2022 as of Linux
> +v6.2-rc4 the last two steps downloaded ~95 and ~2 Megabyte.
> +
> +With this tree you can check out the latest mainline code using ``git checkout
> +--detach origin/master``. All versions and individual changes released between
> +6.0 and the latest 6.1.y version are at hand, too -- this tree thus also allows
> +you to bisect a problem introduced in that range.
> +
> +Later run above 'git fetch [...]' command again, when you want to retrieve the
> +latest mainline codebase or access 6.1.y versions released in between. If you
> +want to switch to a release from a newer stable series (say Linux v6.2.y), you
> +first need to add its remote branch with a command like ``git remote
> +set-branches --add linux-stable linux-6.2.y``; once you run 'git fetch [...]'
> +again you'll be able to check out versions from that series.
> +
> +[:ref:`back to step-by-step guide <sources_sbs>`]
> +
On Thu, Feb 02, 2023 at 12:15:36PM +0100, Linux kernel regression tracking (Thorsten Leemhuis) wrote:
> [adding Konstantin and Greg to the list of recipients]
>
> On 01.02.23 12:52, Thorsten Leemhuis wrote:
> > Add a text explaining how to quickly build a kernel, as that's something
> > users will often have to do when they want to report an issue or test
> > proposed fixes. This is a huge and frightening task for quite a few
> > users these days, as many rely on pre-compiled kernels and have never
> > built their own. They find help on quite a few websites explaining the
> > process in various ways, but those howtos often omit important details
> > or make things too hard for the 'quickly build just for testing' case
> > that 'localmodconfig' is really useful for. Hence give users something
> > at hand to guide them, as that makes it easier for them to help with
> > testing, debugging, and fixing the kernel.
>
> Side note: after feedback on social media I'll likely switch to a title
> like "how to quickly configure & build a trimmed-down Linux kernel", as
> some people from the current title assumed this would be about things
> like ccache. I'll also likely will switch to using a localversion file
> in the buildroot instead of modifying the EXTRAVERSION in the top-level
> makefile (but I haven't actually tried it yet).
>
> > [...]
> >
> > The text currently describes two approaches to retrieve Linux' sources
> > using git: the regular clone with linux-stable as a remote and a shallow
> > clone with just one branch from linux-stable. The shallow clone approach
> > is a little bit more tricky to describe and handle, but downloads way
> > less data – and thus is a lot quicker, unless you have a really really
> > quick link to the internet (which in some parts of the world is hard to
> > come by). That's why I wonder if the text should switch to making the
> > shallow clone with selected stable branches the default. What do you
> > think, dear reader?
>
> So, I looked into what Greg suggested (e.g.
> https://kernel.org/best-way-to-do-linux-clones-for-your-ci.html and
> https://www.kernel.org/cloning-linux-from-a-bundle.html
> ). Assuming users have a up2date git (afaics 2.38+) I could use commands
> like this in my text:
>
> curl -L
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/clone.bundle
> -o ~/linux/linux-stable.git.bundle
> git clone --bundle-uri=linux-stable.git.bundle
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> ~/linux/sources
> rm ~/linux/linix-stable.git.bundle
>
> This took roundabout 16 minutes with my 100 Mbit cable internet
> connection (~9 min for the download, 7 for the clone [the machine used
> is somewhat old]) and downloads & stores ~4,5 GByte data (without checkout).
>
> [side note: using
> "--bundle-uri=https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/clone.bundle"
> does not work (due to the redirect? whatever) -- but that might be
> unwise anyway in case the download is interrupted]
>
>
> Then I tried creating a shallow clone like this:
>
> git clone
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> --depth 1 -b v6.1
> git remote set-branches --add origin master
> git fetch --all --shallow-exclude=v6.1
> git remote add -t linux-6.1.y linux-stable
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> git fetch --all --shallow-exclude=v6.1
>
> This took only roundabout 2 minutes and downloads & stores ~512 MByte
> data (without checkout).
>
>
> Not totally sure, but the shallow clone somehow feels more appropriate
> for the use case (reminder, there is a "quickly" in the document title),
> even if such a clone is less flexible (e.g. users have to manually add
> stable branches they are interested it; and they need to be careful when
> using git fetch).
>
> That's why I now strongly consider using the shallow clone method by
> default in v2 of this text. Or does that also create a lot of load on
> the servers? Or are there other strong reason why using a shallow clone
> might be a bad idea for this use case?
I think Konstantin answered your question already on a social network
based on the server load question.
For the "will this work for testing", sure, a shallow clone should work
just fine, if no one has to use 'git bisect' to go back further than the
version you originally clone. Hopefully that's not a normal thing.
thanks,
greg k-h
On 02.02.23 15:27, Greg KH wrote:
> On Thu, Feb 02, 2023 at 12:15:36PM +0100, Linux kernel regression tracking (Thorsten Leemhuis) wrote:
>> [adding Konstantin and Greg to the list of recipients]
>>
>> On 01.02.23 12:52, Thorsten Leemhuis wrote:
>>> Add a text explaining how to quickly build a kernel, as that's something
>>> users will often have to do when they want to report an issue or test
>>> proposed fixes. This is a huge and frightening task for quite a few
>>> users these days, as many rely on pre-compiled kernels and have never
>>> built their own. They find help on quite a few websites explaining the
>>> process in various ways, but those howtos often omit important details
>>> or make things too hard for the 'quickly build just for testing' case
>>> that 'localmodconfig' is really useful for. Hence give users something
>>> at hand to guide them, as that makes it easier for them to help with
>>> testing, debugging, and fixing the kernel.
>>
>> Side note: after feedback on social media I'll likely switch to a title
>> like "how to quickly configure & build a trimmed-down Linux kernel", as
>> some people from the current title assumed this would be about things
>> like ccache. I'll also likely will switch to using a localversion file
>> in the buildroot instead of modifying the EXTRAVERSION in the top-level
>> makefile (but I haven't actually tried it yet).
>>
>>> [...]
>>>
>>> The text currently describes two approaches to retrieve Linux' sources
>>> using git: the regular clone with linux-stable as a remote and a shallow
>>> clone with just one branch from linux-stable. The shallow clone approach
>>> is a little bit more tricky to describe and handle, but downloads way
>>> less data – and thus is a lot quicker, unless you have a really really
>>> quick link to the internet (which in some parts of the world is hard to
>>> come by). That's why I wonder if the text should switch to making the
>>> shallow clone with selected stable branches the default. What do you
>>> think, dear reader?
>>
>> So, I looked into what Greg suggested (e.g.
>> https://kernel.org/best-way-to-do-linux-clones-for-your-ci.html and
>> https://www.kernel.org/cloning-linux-from-a-bundle.html
>> ). Assuming users have a up2date git (afaics 2.38+) I could use commands
>> like this in my text:
>>
>> curl -L
>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/clone.bundle
>> -o ~/linux/linux-stable.git.bundle
>> git clone --bundle-uri=linux-stable.git.bundle
>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
>> ~/linux/sources
>> rm ~/linux/linix-stable.git.bundle
>>
>> This took roundabout 16 minutes with my 100 Mbit cable internet
>> connection (~9 min for the download, 7 for the clone [the machine used
>> is somewhat old]) and downloads & stores ~4,5 GByte data (without checkout).
>>
>> [side note: using
>> "--bundle-uri=https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/clone.bundle"
>> does not work (due to the redirect? whatever) -- but that might be
>> unwise anyway in case the download is interrupted]
>>
>>
>> Then I tried creating a shallow clone like this:
>>
>> git clone
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>> --depth 1 -b v6.1
>> git remote set-branches --add origin master
>> git fetch --all --shallow-exclude=v6.1
>> git remote add -t linux-6.1.y linux-stable
>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
>> git fetch --all --shallow-exclude=v6.1
>>
>> This took only roundabout 2 minutes and downloads & stores ~512 MByte
>> data (without checkout).
>>
>>
>> Not totally sure, but the shallow clone somehow feels more appropriate
>> for the use case (reminder, there is a "quickly" in the document title),
>> even if such a clone is less flexible (e.g. users have to manually add
>> stable branches they are interested it; and they need to be careful when
>> using git fetch).
>>
>> That's why I now strongly consider using the shallow clone method by
>> default in v2 of this text. Or does that also create a lot of load on
>> the servers? Or are there other strong reason why using a shallow clone
>> might be a bad idea for this use case?
>
> I think Konstantin answered your question already on a social network
> based on the server load question.
Yup, he did. For the record, this is what he wrote:
```
it [a shallow clone] is pretty expensive on the server side, but it is
only really a problem when a bunch of shallow clones are performed at
once (e.g. when a CI farm does it across 50 nodes or something). When
that happens, it overwhelms the server.
If an actual human being does a single shallow clone it's not a big deal.
```
> For the "will this work for testing", sure, a shallow clone should work
> just fine, if no one has to use 'git bisect' to go back further than the
> version you originally clone. Hopefully that's not a normal thing.
Yeah, that's something I have in mind already. But seems it's not much
of a problem, as one apparently can access older versions (including
tags) by deepening a shallow clone with a command like "git fetch origin
--shallow-exclude=v5.19" these days.
Ciao, Thorsten
On Thu, Feb 02, 2023 at 12:15:36PM +0100, Linux kernel regression tracking (Thorsten Leemhuis) wrote:
> Then I tried creating a shallow clone like this:
>
> git clone
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> --depth 1 -b v6.1
> git remote set-branches --add origin master
> git fetch --all --shallow-exclude=v6.1
> git remote add -t linux-6.1.y linux-stable
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> git fetch --all --shallow-exclude=v6.1
>
> This took only roundabout 2 minutes and downloads & stores ~512 MByte
> data (without checkout).
Can we also include the option of just downloading the tarball, if it's a
released version? That's the fastest and most lightweight option 100% of the
time. :)
> Not totally sure, but the shallow clone somehow feels more appropriate
> for the use case (reminder, there is a "quickly" in the document title),
> even if such a clone is less flexible (e.g. users have to manually add
> stable branches they are interested it; and they need to be careful when
> using git fetch).
>
> That's why I now strongly consider using the shallow clone method by
> default in v2 of this text. Or does that also create a lot of load on
> the servers? Or are there other strong reason why using a shallow clone
> might be a bad idea for this use case?
As I mentioned elsewhere, this is only a problem when it's done in batch mode
by CI systems. A full clone uses pregenerated pack files and is very cheap,
because it's effectively a sendfile operation. A shallow clone requires
generating a brand new pack, compressing it, and then keeping it around in
memory for the duration of the clone process. Not a big deal when a few humans
here and there do it, but when 50 CI nodes do it all at once, it effectively
becomes a DDoS. :)
-K
On 02.02.23 16:08, Konstantin Ryabitsev wrote:
> On Thu, Feb 02, 2023 at 12:15:36PM +0100, Linux kernel regression tracking (Thorsten Leemhuis) wrote:
>> Then I tried creating a shallow clone like this:
>>
>> git clone
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>> --depth 1 -b v6.1
>> git remote set-branches --add origin master
>> git fetch --all --shallow-exclude=v6.1
>> git remote add -t linux-6.1.y linux-stable
>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
>> git fetch --all --shallow-exclude=v6.1
>>
>> This took only roundabout 2 minutes and downloads & stores ~512 MByte
>> data (without checkout).
>
> Can we also include the option of just downloading the tarball, if it's a
> released version? That's the fastest and most lightweight option 100% of the
> time. :)
Don't worry, that was in there and will stay in there:
+ If you plan to only build one particular kernel version, download
its source
+ archive from https://kernel.org; afterwards extract its content to
'~/linux/'
+ and change into the directory created during extraction.
>> Not totally sure, but the shallow clone somehow feels more appropriate
>> for the use case (reminder, there is a "quickly" in the document title),
>> even if such a clone is less flexible (e.g. users have to manually add
>> stable branches they are interested it; and they need to be careful when
>> using git fetch).
>>
>> That's why I now strongly consider using the shallow clone method by
>> default in v2 of this text. Or does that also create a lot of load on
>> the servers? Or are there other strong reason why using a shallow clone
>> might be a bad idea for this use case?
>
> As I mentioned elsewhere, this is only a problem when it's done in batch mode
> by CI systems. A full clone uses pregenerated pack files and is very cheap,
> because it's effectively a sendfile operation. A shallow clone requires
> generating a brand new pack, compressing it, and then keeping it around in
> memory for the duration of the clone process. Not a big deal when a few humans
> here and there do it, but when 50 CI nodes do it all at once, it effectively
> becomes a DDoS. :)
Thx again for your insights, much appreciated.
Ciao, Thorsten
On Thu, 02 Feb 2023, Thorsten Leemhuis <[email protected]> wrote:
> On 02.02.23 16:08, Konstantin Ryabitsev wrote:
>> On Thu, Feb 02, 2023 at 12:15:36PM +0100, Linux kernel regression tracking (Thorsten Leemhuis) wrote:
>>> Then I tried creating a shallow clone like this:
>>>
>>> git clone
>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>> --depth 1 -b v6.1
>>> git remote set-branches --add origin master
>>> git fetch --all --shallow-exclude=v6.1
>>> git remote add -t linux-6.1.y linux-stable
>>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
>>> git fetch --all --shallow-exclude=v6.1
>>>
>>> This took only roundabout 2 minutes and downloads & stores ~512 MByte
>>> data (without checkout).
>>
>> Can we also include the option of just downloading the tarball, if it's a
>> released version? That's the fastest and most lightweight option 100% of the
>> time. :)
>
> Don't worry, that was in there and will stay in there:
>
> + If you plan to only build one particular kernel version, download
> its source
> + archive from https://kernel.org; afterwards extract its content to
> '~/linux/'
> + and change into the directory created during extraction.
The trouble is, if this is for someone who needs to try kernels for
debugging, a typical idea is to ask them to revert something or apply a
patch. All the guides for that will be 'git revert' and 'git am'. Bisect
is right up there on the list too. And then they'll first grab a tarball
and fail, then do a shallow copy and fail, and then finally get a full
one... :p
BR,
Jani.
>>> Not totally sure, but the shallow clone somehow feels more appropriate
>>> for the use case (reminder, there is a "quickly" in the document title),
>>> even if such a clone is less flexible (e.g. users have to manually add
>>> stable branches they are interested it; and they need to be careful when
>>> using git fetch).
>>>
>>> That's why I now strongly consider using the shallow clone method by
>>> default in v2 of this text. Or does that also create a lot of load on
>>> the servers? Or are there other strong reason why using a shallow clone
>>> might be a bad idea for this use case?
>>
>> As I mentioned elsewhere, this is only a problem when it's done in batch mode
>> by CI systems. A full clone uses pregenerated pack files and is very cheap,
>> because it's effectively a sendfile operation. A shallow clone requires
>> generating a brand new pack, compressing it, and then keeping it around in
>> memory for the duration of the clone process. Not a big deal when a few humans
>> here and there do it, but when 50 CI nodes do it all at once, it effectively
>> becomes a DDoS. :)
>
> Thx again for your insights, much appreciated.
>
> Ciao, Thorsten
--
Jani Nikula, Intel Open Source Graphics Center
On 03.02.23 10:44, Jani Nikula wrote:
> On Thu, 02 Feb 2023, Thorsten Leemhuis <[email protected]> wrote:
>> On 02.02.23 16:08, Konstantin Ryabitsev wrote:
>>> On Thu, Feb 02, 2023 at 12:15:36PM +0100, Linux kernel regression tracking (Thorsten Leemhuis) wrote:
>>>> Then I tried creating a shallow clone like this:
>>>>
>>>> git clone
>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>>>> --depth 1 -b v6.1
>>>> git remote set-branches --add origin master
>>>> git fetch --all --shallow-exclude=v6.1
>>>> git remote add -t linux-6.1.y linux-stable
>>>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
>>>> git fetch --all --shallow-exclude=v6.1
>>>>
>>>> This took only roundabout 2 minutes and downloads & stores ~512 MByte
>>>> data (without checkout).
>>>
>>> Can we also include the option of just downloading the tarball, if it's a
>>> released version? That's the fastest and most lightweight option 100% of the
>>> time. :)
>>
>> Don't worry, that was in there and will stay in there:
>>
>> + If you plan to only build one particular kernel version, download
>> its source
>> + archive from https://kernel.org; afterwards extract its content to
>> '~/linux/'
>> + and change into the directory created during extraction.
>
> The trouble is, if this is for someone who needs to try kernels for
> debugging, a typical idea is to ask them to revert something or apply a
> patch. All the guides for that will be 'git revert' and 'git am'. Bisect
> is right up there on the list too. And then they'll first grab a tarball
> and fail,
Yeah, those are the reasons why I don't like the tarball approach too
much myself. Guess I should point them out in the text to make readers
aware of them...
> then do a shallow copy and fail,
The new test I wrote (still a draft) will suggest to use a recent
release as base, hence bisection or reverting a patch will be possible.
And if the range turns out to be to shallow, there is still "git fetch
--shallow-exclude=v6.1" to deepen it, which should avoid...
> and then finally get a full one... :p
...this scenario -- at least unless I missed anything.
Ciao, Thorsten
>>>> Not totally sure, but the shallow clone somehow feels more appropriate
>>>> for the use case (reminder, there is a "quickly" in the document title),
>>>> even if such a clone is less flexible (e.g. users have to manually add
>>>> stable branches they are interested it; and they need to be careful when
>>>> using git fetch).
>>>>
>>>> That's why I now strongly consider using the shallow clone method by
>>>> default in v2 of this text. Or does that also create a lot of load on
>>>> the servers? Or are there other strong reason why using a shallow clone
>>>> might be a bad idea for this use case?
>>>
>>> As I mentioned elsewhere, this is only a problem when it's done in batch mode
>>> by CI systems. A full clone uses pregenerated pack files and is very cheap,
>>> because it's effectively a sendfile operation. A shallow clone requires
>>> generating a brand new pack, compressing it, and then keeping it around in
>>> memory for the duration of the clone process. Not a big deal when a few humans
>>> here and there do it, but when 50 CI nodes do it all at once, it effectively
>>> becomes a DDoS. :)
>>
>> Thx again for your insights, much appreciated.
>>
>> Ciao, Thorsten
>
Quick update, in case anyone wants an early look at the reworked
sections that likely will soon be posted as part of a v2:
On 02.02.23 12:15, Linux kernel regression tracking (Thorsten Leemhuis)
wrote:
> [adding Konstantin and Greg to the list of recipients]
>
> On 01.02.23 12:52, Thorsten Leemhuis wrote:
>> Add a text explaining how to quickly build a kernel, as that's something
>> users will often have to do when they want to report an issue or test
>> proposed fixes. This is a huge and frightening task for quite a few
>> users these days, as many rely on pre-compiled kernels and have never
>> built their own. They find help on quite a few websites explaining the
>> process in various ways, but those howtos often omit important details
>> or make things too hard for the 'quickly build just for testing' case
>> that 'localmodconfig' is really useful for. Hence give users something
>> at hand to guide them, as that makes it easier for them to help with
>> testing, debugging, and fixing the kernel.
>
> Side note: after feedback on social media I'll likely switch to a title
> like "how to quickly configure & build a trimmed-down Linux kernel", as
> some people from the current title assumed this would be about things
> like ccache.
I for now settled for "How to quickly build a trimmed Linux kernel". I
wonder if "adapt" or "attuned" might be better than "trimmed". I'd
appreciate any input from native speakers here.
> I'll also likely will switch to using a localversion file
> in the buildroot instead of modifying the EXTRAVERSION in the top-level
> makefile (but I haven't actually tried it yet).
The section in the step by step guide now reads:
```
* If you patched your kernel or already have a kernel of the same
version installed, better tag the one you are about to build by
extending its release name::
echo "-proposed_fix" > ~/linux/build/localversion
If you build Linux 6.2-rc4 and later execute ``uname -r`` while
running that kernel, it will print '6.2-rc4-proposed_fix'.
```
>> [...]
>>
>> The text currently describes two approaches to retrieve Linux' sources
>> using git: the regular clone with linux-stable as a remote and a shallow
>> clone with just one branch from linux-stable. [...]
>
> [...]>
> That's why I now strongly consider using the shallow clone method by
> default in v2 of this text.
TWIMC, I went down that route and took a bit of back and forth to figure
out a good approach.
The segment in the step-by-step guide now looks like this:
```
* Retrieve the sources of the Linux version you intend to build; then
change into the directory holding them, as all further commands in this
guide are meant to be executed from there.
*[Note, the following paragraphs describe how to retrieve the sources
by partially cloning the Linux stable repository. The reference section
explains two alternatives to such a 'shallow clone': packaged archives
and a full clone. Use the latter, if downloading a lot of data does not
bother you, as that will avoid some quirks of shallow clones the
reference section explains.]*
Execute the following command to retrieve a fresh mainline codebase::
git clone --no-checkput --depth 1 -b master \
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git\
~/linux/sources/
cd ~/linux/sources/
If you want to access recent mainline releases and pre-releases,
deepen you clone's history to the oldest version you are interested in::
git fetch --shallow-exclude=v6.1
In case you want to access a stable/longterm release (say v6.1.5),
simply add the branch holding that series; afterwards fetch the history
up to the mainline version that started the series (v6.1) or is older::
git remote set-branches --add origin linux-6.1.y
git fetch --shallow-exclude=v6.1
Now check out the code you are interested in. If you just performed
the initial clone, you will be able to check out a fresh mainline
codebase ('origin/master') -- it is ideal for checking whether
developers already addressed an issue you face::
git checkout --detach origin/master
If you deepened your clone, you instead of 'origin/master' can
specify the version you deepened to; any later releases like 'v6.2' or
pre-release like 'v6.3-rc1' will work, too. Stable or longterm versions
like 'v6.1.5' work just the same, if you added the appropriate
stable/longterm branches as described.
```
The corresponding segment in the reference section got somewhat long and
now looks like this:
```
Download the sources
--------------------
*Retrieve the sources of the Linux version you intend to build.*
[:ref:`...<sources_sbs>`]
The step-by-step guide outlines how to retrieve Linux' sources using a
shallow git clone. There is more to tell about this method and two
alternate ways worth describing: packaged archives and a full git clone.
And the aspects 'wouldn't it be wiser to use a proper pre-release than
the latest mainline code' and 'how to get an even fresher mainline
codebase' need elaboration, too.
Unexpected aspects of shallow clones
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The step-by-step guide uses a shallow clone, as it's the best solution
for most of this document's target audience. There are a few quirks of
this approach worth mentioning:
* This document in most places uses 'git fetch' with
'--shallow-exclude=' to specify the tag of the earliest version you care
about. You alternatively can use the parameter '--shallow-since=' to
specify an absolute (say '2023-07-15') or relative ('12 months') date to
define the depth of the history you want to download.
* When running 'git fetch', remember to always specify the oldest
version or time you care about as shown in the step-by-step guide.
Otherwise you'll risk downloading nearly the entire git history, which
will consume quite a bit of time and bandwidth while also stressing the
servers.
Note, you don't have to use the same version/date all the time, but
when you start using a different one git will deepen or flatten the
history to the specified point. That allows you to retrieve versions you
initially thought you did not need -- or it will discard the sources of
older versions, when you want to free up some disk space. The latter
will happen automatically when using '--shallow-since=' with a relative
date.
* Be warned, when deepening the history you might encounter an error
like 'fatal: error in object: unshallow
1234567890abcdef1234567890abcdef12345678'. In that case run ``git repack
-d`` and try again.
* In case you want to revert changes from a certain version (say Linux
6.3) or perform a bisection ending on it (v6.2..v6.3), better tell 'git
fetch' to retrieve objects up to three versions earlier (e.g. 6.0):
``git describe`` will then be able to describe most commits just like it
would in a full tree.
Downloading the sources using a packages archive
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
People new to compiling Linux often assume downloading an archive via
the front-page of https://kernel.org is the best approach to retrieve
Linux' sources. It actually can be, if you are certain to build just one
particular kernel version without changing any code. Thing is: you might
be sure this will be the case, but in practice it then often will turn
out to be a wrong assumption.
That's because when reporting or debugging an issue developers will
often ask to give an earlier or later version a try. They also might
suggest temporarily undoing a commit with 'git revert …' or provide
various patches to try. Sometimes reporters will also be asked to use
'git bisect' to find the change causing a problem. These things rely on
git or are a lot easier and quicker to handle with it.
Additionally, a shallow clone also doesn't add much overhead: when git
is creating one for just the latest mainline codebase it retrieves only
a little more data than downloading a packaged archive would.
A shallow clone therefore is often the better choice. If you
nevertheless want to use a packaged source archive, download one via
kernel.org; afterwards extract its content to '~/linux/' and change to
the directory created during extraction. The rest of the step-by-step
guide will work just fine, apart from things that rely on git -- but
this mainly concerns the section on successive builds of other versions.
Downloading the sources using a full git clone
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If downloading and storing a lot of data (~4,4 Gigabyte as of early
2023) is nothing that bothers you, instead of a shallow clone perform a
full git clone instead. They are a little easier to handle and will have
the complete development history at hand at any time::
curl -L
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/clone.bundle
-o ~/linux/linux-stable.git.bundle
git clone clone.bundle ~/linux/sources/
rm ~/linux/linux-stable.git.bundle
cd ~/linux/sources/
git remote set-url origin
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
git fetch --all
git checkout --detach origin/master
Proper pre-releases (RCs) vs. latest mainline
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When cloning the sources using git and checking out origin/master, you
often will retrieve a codebase that is somewhere between the latest and
the next release or pre-release. This almost always is the code you want
when giving mainline a shot: pre-releases like v6.1-rc5 are in no way
special, as they don't get any significant extra testing before being
published.
There is one exception: you might want to stick to the latest mainline
release (say v6.1) before its successor's first pre-release (v6.2-rc1)
is out. That's because compiler errors and other problems are more
likely to occur during this time, as mainline then is in its 'merge
window': a usually two week long phase, in which the bulk of the changes
for the next release is merged.
Avoiding the mainline lag
~~~~~~~~~~~~~~~~~~~~~~~~~
The explanations for both the shallow clone and the full clone both
retrieve the code from the Linux stable git repository. That makes
things simpler for this document's audience, as it allows easy access to
both mainline and stable/longterm releases. This approach has just one
downside:
Changes merged into the mainline repository are only synced to the
master branch of the Linux stable repository every few hours. This lag
most of the time is not something to worry about; but in case you really
need the latest code, just add the mainline repo as additional remote
and checkout the code from there::
git remote add mainline
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git fetch --all
git checkout --detach mainline/master
```
Ciao, Thorsten
On Fri, Feb 10, 2023 at 12:38:07PM +0100, Thorsten Leemhuis wrote:
> Downloading the sources using a full git clone
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> If downloading and storing a lot of data (~4,4 Gigabyte as of early
> 2023) is nothing that bothers you, instead of a shallow clone perform a
> full git clone instead.
As a minor segue, we know we can dramatically reduce the size of the stable
repo (and the stable clone.bundle) if we drop all branches and tags for old
EOL releases and move them to a separate repo (e.g. stable-eol.git).
I have suggested this in the past, but it's not a trivial change:
- it requires a fair amount of git work
- it may disrupt some of the build infra around the world still building EOL
kernels (for whatever reason)
In my preliminary tests, we easily dropped the size from 4GB to under 3GB.
-K
Hi!
> +If your platform uses techniques like Secure Boot, prepare the system to permit
> +starting self-compiled Linux kernels; install compilers and everything else
> +needed for building Linux; make sure to have 15 Gigabyte free space in your home
> +directory. Now run the following commands to download the sources of Linux,
> +which you then use to configure, build and install your own
> kernel::
15GB is quite a lot. Maybe the shallow clone should be given another
chance?
> + mkdir ~/linux/ ~/linux/sources ~/linux/build
> + git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git \
> + ~/linux/sources
> + cd ~/linux/sources/
> + # hint: if you don't want to build any stable or longterm kernels, skip the
> + # next two commands
> + git remote add linux-stable \
> + https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> + git fetch --all
> + # hint: if you want to apply patches, do it at this point (see below for details)
> + # hint: at this point it's recommended to tag your build (see below for details)
> + yes "" | make O=~/linux/build/ localmodconfig
> + # hint: at this point you might want or have to adjust the build configuration
> + # (see below for details)
> + # note: if you are running a Debian kernel, you'll have to adjust the configuration
> + # at this point (see below)
> + make -j $(nproc --all) O=~/linux/build/
Is the complexity of using O= worth it for one-off kernel builds?
Best regards,
Pavel
--
People of Russia, stop Putin before his war on Ukraine escalates.
On 25.02.23 10:17, Pavel Machek wrote:
>> +If your platform uses techniques like Secure Boot, prepare the system to permit
>> +starting self-compiled Linux kernels; install compilers and everything else
>> +needed for building Linux; make sure to have 15 Gigabyte free space in your home
>> +directory. Now run the following commands to download the sources of Linux,
>> +which you then use to configure, build and install your own
>> kernel::
>
> 15GB is quite a lot. Maybe the shallow clone should be given another
> chance?
It got and made it, see v2:
https://lore.kernel.org/all/8cfcf069d48c1b8d7b83aafe0132f8dad0f1d0ea.1676400947.git.linux@leemhuis.info/
>> + mkdir ~/linux/ ~/linux/sources ~/linux/build
>> + git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git \
>> + ~/linux/sources
>> + cd ~/linux/sources/
>> + # hint: if you don't want to build any stable or longterm kernels, skip the
>> + # next two commands
>> + git remote add linux-stable \
>> + https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
>> + git fetch --all
>> + # hint: if you want to apply patches, do it at this point (see below for details)
>> + # hint: at this point it's recommended to tag your build (see below for details)
>> + yes "" | make O=~/linux/build/ localmodconfig
>> + # hint: at this point you might want or have to adjust the build configuration
>> + # (see below for details)
>> + # note: if you are running a Debian kernel, you'll have to adjust the configuration
>> + # at this point (see below)
>> + make -j $(nproc --all) O=~/linux/build/
>
> Is the complexity of using O= worth it for one-off kernel builds?
That is a good question. I seemed like the right thing to do when I
started writing this guide (maybe just because I've been doing it that
way for ages), but now that you ask I'm unsure myself.
/me wonders how others feel about it
Ciao, Thorsten
On 25.02.23 10:29, Thorsten Leemhuis wrote:
> On 25.02.23 10:17, Pavel Machek wrote:
>
>>> + mkdir ~/linux/ ~/linux/sources ~/linux/build
>>> + git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git \
>>> + ~/linux/sources
>>> + cd ~/linux/sources/
>>> + # hint: if you don't want to build any stable or longterm kernels, skip the
>>> + # next two commands
>>> + git remote add linux-stable \
>>> + https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
>>> + git fetch --all
>>> + # hint: if you want to apply patches, do it at this point (see below for details)
>>> + # hint: at this point it's recommended to tag your build (see below for details)
>>> + yes "" | make O=~/linux/build/ localmodconfig
>>> + # hint: at this point you might want or have to adjust the build configuration
>>> + # (see below for details)
>>> + # note: if you are running a Debian kernel, you'll have to adjust the configuration
>>> + # at this point (see below)
>>> + make -j $(nproc --all) O=~/linux/build/
>>
>> Is the complexity of using O= worth it for one-off kernel builds?
>
> That is a good question. I seemed like the right thing to do when I
> started writing this guide (maybe just because I've been doing it that
> way for ages), but now that you ask I'm unsure myself.
>
> /me wonders how others feel about it
/me should have pondered with the idea for more time before writing aboe
mail, because:
Having thought about this some more, I decided to drop it, the
complexity really is not worth it.
Many thx for the suggestion!
Ciao, Thorsten