Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230407203456.27141-1-quic_johmoo@quicinc.com> <20230407203456.27141-2-quic_johmoo@quicinc.com>
In-Reply-To: <20230407203456.27141-2-quic_johmoo@quicinc.com>
From:   Masahiro Yamada <masahiroy@kernel.org>
Date:   Mon, 10 Apr 2023 19:03:05 +0900
Message-ID: <CAK7LNAQQmoyUx+0Jk3c7iqY20KokrHEOPwHNb2doZOOA8RWBDA@mail.gmail.com>
Subject: Re: [PATCH v5 1/2] check-uapi: Introduce check-uapi.sh
To:     John Moon <quic_johmoo@quicinc.com>
Cc:     Nathan Chancellor <nathan@kernel.org>,
        Nick Desaulniers <ndesaulniers@google.com>,
        Nicolas Schier <nicolas@fjasle.eu>,
        linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org,
        linux-arm-msm@vger.kernel.org,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Randy Dunlap <rdunlap@infradead.org>,
        Arnd Bergmann <arnd@arndb.de>,
        Bjorn Andersson <andersson@kernel.org>,
        Todd Kjos <tkjos@google.com>,
        Matthias Maennich <maennich@google.com>,
        Giuliano Procida <gprocida@google.com>,
        kernel-team@android.com, libabigail@sourceware.org,
        Jordan Crouse <jorcrous@amazon.com>,
        Trilok Soni <quic_tsoni@quicinc.com>,
        Satya Durga Srinivasu Prabhala <quic_satyap@quicinc.com>,
        Elliot Berman <quic_eberman@quicinc.com>,
        Guru Das Srinagesh <quic_gurus@quicinc.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Sat, Apr 8, 2023 at 5:35=E2=80=AFAM John Moon <quic_johmoo@quicinc.com> =
wrote:
>
> While the kernel community has been good at maintaining backwards
> compatibility with kernel UAPIs, it would be helpful to have a tool
> to check if a commit introduces changes that break backwards
> compatibility.
>
> To that end, introduce check-uapi.sh: a simple shell script that
> checks for changes to UAPI headers using libabigail.
>
> libabigail is "a framework which aims at helping developers and
> software distributors to spot some ABI-related issues like interface
> incompatibility in ELF shared libraries by performing a static
> analysis of the ELF binaries at hand."
>
> The script uses one of libabigail's tools, "abidiff", to compile the
> changed header before and after the commit to detect any changes.
>
> abidiff "compares the ABI of two shared libraries in ELF format. It
> emits a meaningful report describing the differences between the two
> ABIs."
>
> The script also includes the ability to check the compatibility of
> all UAPI headers across commits. This allows developers to inspect
> the stability of the UAPIs over time.
>
> Signed-off-by: John Moon <quic_johmoo@quicinc.com>


BTW, is there anybody (except the submitters) who loves this tool?
(or anybody who has ever evaluated this?)

I am the only person who pointed out something.


I put some more comments from users' point of view.


[1] The number of lines of the error log.


According to this tool, it looks like we broke a lot of UAPI
headers in the previous MW (between v6.2 and v6.3-rc1).

Not only in the previous MW, we always broke so many UAPI
headers, this script says.


The following command outputs 2380 lines.


$ ./scripts/check-uapi.sh -q -b v6.3-rc1 -p v6.2 2>&1  | wc
   2380    9438  104953


I do not know how many are real breakages, and
how many are false positives.

Anyway, I will not check the 2380 lines of the error log.


And, please note that the help message explains '-q' is the *quiet* mode.
It is the opposite. This is super noisy.


[2] Be careful!


While testing the patch submissions of this,
I messed up my repository multiple times.

The script takes some time because it builds many objects
internally.

However, once this script starts running, you must not hit Ctrl-C.
If you do it, your repository will be sprinkled with a ton
of untracked files.

Apply this patch, and run "./scripts/check-uapi.sh -p v6.0"
and hit Ctrl-C.

Repeat it a couple of times, and "git status" will show you
something horrible.


You will never know when git is checking out a commit
because this script hides it by 'git checkout --quiet'.


So, this tool should show a caveat at least.


'git checkout' should not be hidden, or
maybe a caveat message should be shown.


CAVEAT
This tool runs 'git checkout' a couple of times internally.
If you interrupt it, your worktree might be messed up.


--
Best Regards
Masahiro Yamada