Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
From:   jim.cromie@gmail.com
Date:   Thu, 15 Jul 2021 12:02:33 -0600
Message-ID: <CAJfuBxwZbrAuzvfHRxLHhh8a9nva7fR4ZiLc7NECspVZ0gx2ow@mail.gmail.com>
Subject: subordinate trees ? we have %G-space, we should add %K-space
To:     git@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

hi git people,

managing .configs, over time, and across work-trees, is an ongoing burden.
I wonder if/how git could help.

I imagine a subordinate/secondary tree,
which basically has only 1 tracked file ( .config ) in it.
Auto-commit on .config ( perhaps as 1st step of make ) would mean no
changes are missed

A tertiary tree could hold all of ./builds/*
or really any set of files that are ignored by the primary tree's
.gitignore file
its purpose would be to track build products; all *.o etc
its kind-of a superset of what might be found in your .ccache,
(it also includes *.cmd.o etc, and all other intermediate outputs)
but it is attributable to a TooChainCompilationTransferFn(.config, HEAD)

Since HEAD already has a sha*, we just need one for .config,
the secondary tree keeps that for us.

the tertiary tree doesnt REALLY need to keep *all* the build products,
it just needs to keep track of them (just the sha*, since they are
reproducible),
but every commit (to either source, or to .config) should result in 1 or more
build products changing.  It might be useful for QA purposes to predict and
verify what products should change.

Autocommit here means that make *anything* has known inputs,
so if results are useful/reportable, they have known inputs (no -dirty)

Also, it might be handy to be able to diff any 2 objdumps/readelfs of foo.o,
which would be easier with a cryptographically known DB


Heres how I think it could get leveraged for linux itself

We have %G space, expressed in the kernel name:
  5.14.0-rc1-lm1-00003-g4a7356173ef9-dirty

Lets add %K space, to extend that:
  5.14.0-rc1-lm1-00003-g4a7356173ef9-dirty-k6173ef96173

now posit a kverstr("%D-%G-%K") producing the 2nd name above.
    %T - tag :  "5.14.0-rc1"
    %L - localversion : "-lm1"
    %C - count : "00003", %C{5} default
    %G - HEAD-sha : "-g4a7356173ef9-dirty"
    %K - sha(.config) : "-k6c173efc961c", obviously 'c' is taken.
    %D - desc : "%T-%L-%C"

%L is seductive, but ultimately a nuisance; mine above is a
super-imaginitive "-localmod-1" abbreviation.

In contrast, %K would give:

- updates every time the .config is tweaked.

- each arch make defconfig yields single %Kval
  or a few (1 per salt-val)

- normalization done by make oldconfig narrows %K-space
  some %Kvals are "impossible" (aka: hand altered or fuzzed .configs)

- many .configs will get "well-known" %Kvals
  bots will report probs with %Gs against specific %Ks

- all well-known configs: defconfig, allmodconfig, etc...
  get unique %Ks, maybe become well-known
  esp as %K{5} which is short enough to recognize
  %Ks could be annotated
  %Ks change slowly as **/KBuild.* composition changes
  traceable to commits in %G

With (imagined) Git extensions:

1. subordinate trees, in same src/work-dir:

a configs subtree would have only .config file checked in, and
probably should auto-commit on changes.

ls ./.gitinclude.*

These actively define inclusions of files in the named subtree, rather
than ignores.  They dont see files that are not explicitly ignored by
primary tree.

so ./.gitinclude.1.configs:
# with
.config

could always, actively, track and commit changes to .config,
`history|grep config | tail -n5`, automatically added to commit-log,
would help (a little) explain why.  A string serialization of the
CONFIG_* changes, would completely characterize the .config diff, and

and ./.gitinclude.3.builds
# with
./builds/

captures all build product files written under ./builds/* into a
single subordinate tree.  This tree can only contain files ignored by
.gitignore (ie the primary/source tree)

obviously this will include all of ./**/*.o, which is essentially
a %G+%K versioned (and clean) ccache result set.

./builds/{*}   # distinct build-product (%G+%K dependent) sub-trees

./builds/foo # foo
./builds/bar

not really sure how the 2ndary tree (.config tracker) should play with
worktrees,
or even with multiple ./builds/ subdirs, each with a .config
those subdirs are our %K-space-explored,
ie those configs which we have at least tried

make randconfig is a rather low-quality way to expand the tested %K-space
but with %K handled a bit better, we could at least track how much of
the space gets tested this way.

ok, now Im meandering.