Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <20191028151427.31612-1-jeyu@kernel.org> <20191028151427.31612-4-jeyu@kernel.org>
 <CAK7LNAQ8WDQ3PX8iwQNj5eNQADFMnKNVo3+uC8dt3rCPWK8a-w@mail.gmail.com> <20191030161726.GA13413@linux-8ccs>
In-Reply-To: <20191030161726.GA13413@linux-8ccs>
From:   Masahiro Yamada <yamada.masahiro@socionext.com>
Date:   Thu, 31 Oct 2019 21:27:10 +0900
Message-ID: <CAK7LNAR5vJExV9L+vO491V=ZRfJGiwDPd5rsZJe-4T5Dz1eh1A@mail.gmail.com>
Subject: Re: [PATCH 4/4] scripts/nsdeps: make sure to pass all module source
 files to spatch
To:     Jessica Yu <jeyu@kernel.org>
Cc:     Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Matthias Maennich <maennich@google.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Thu, Oct 31, 2019 at 1:17 AM Jessica Yu <jeyu@kernel.org> wrote:
>
> +++ Masahiro Yamada [29/10/19 21:57 +0900]:
> >On Tue, Oct 29, 2019 at 12:14 AM Jessica Yu <jeyu@kernel.org> wrote:
> >>
> >> The nsdeps script passes a list of the module source files to
> >> generate_deps_for_ns() as a space delimited string named $mod_source_f=
iles,
> >> which then passes it to spatch. But since $mod_source_files is not enc=
ased
> >> in quotes, each source file in that string is treated as a separate sh=
ell
> >> function argument (as $2, $3, $4, etc.).  However, the spatch invocati=
on
> >> only refers to $2, so only the first file out of $mod_source_files is
> >> processed by spatch.
> >>
> >> This causes problems (namely, the MODULE_IMPORT_NS() statement doesn't
> >> get inserted) when a module is composed of many source files and the
> >> "main" module file containing the MODULE_LICENSE() statement is not th=
e
> >> first file listed in $mod_source_files. Fix this by encasing
> >> $mod_source_files in quotes so that the entirety of the string is
> >> treated as a single argument and can be referred to as $2.
> >>
> >> Signed-off-by: Jessica Yu <jeyu@kernel.org>
> >> ---
> >>  scripts/nsdeps | 2 +-
> >>  1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/scripts/nsdeps b/scripts/nsdeps
> >> index 9ddcd5cb96b1..5055b059a81b 100644
> >> --- a/scripts/nsdeps
> >> +++ b/scripts/nsdeps
> >> @@ -36,7 +36,7 @@ generate_deps() {
> >>                                               | sed -E "s%(^|\s)([^/][=
^ ]*)%\1$srctree/\2%g"`
> >>         for ns in `cat $ns_deps_file`; do
> >>                 echo "Adding namespace $ns to module $mod_name (if nee=
ded)."
> >> -               generate_deps_for_ns $ns $mod_source_files
> >> +               generate_deps_for_ns $ns "$mod_source_files"
> >>                 # sort the imports
> >>                 for source_file in $mod_source_files; do
> >>                         sed '/MODULE_IMPORT_NS/Q' $source_file > ${sou=
rce_file}.tmp
> >
> >I think this change is correct, but
> >did you succeed in nsdeps for composite modules
> >with this patch only?
> >
> >I think the following is needed too:
> >
> >
> >diff --git a/scripts/nsdeps b/scripts/nsdeps
> >index dda6fbac016e..5a23ea616446 100644
> >--- a/scripts/nsdeps
> >+++ b/scripts/nsdeps
> >@@ -31,9 +31,9 @@ generate_deps() {
> >        local mod_file=3D`echo $@ | sed -e 's/\.ko/\.mod/'`
> >        local ns_deps_file=3D`echo $@ | sed -e 's/\.ko/\.ns_deps/'`
> >        if [ ! -f "$ns_deps_file" ]; then return; fi
> >-       local mod_source_files=3D`cat $mod_file | sed -n 1p             =
         \
> >+       local mod_source_files=3D"`cat $mod_file | sed -n 1p
> >         \
> >                                              | sed -e 's/\.o/\.c/g'    =
       \
> >-                                             | sed "s|[^ ]* *|${srctree=
}/&|g"`
> >+                                             | sed "s|[^ ]* *|${srctree=
}/&|g"`"
> >        for ns in `cat $ns_deps_file`; do
> >                echo "Adding namespace $ns to module $mod_name (if neede=
d)."
> >                generate_deps_for_ns $ns $mod_source_files
> >
> >
> >Without this, a module that consists of two files
> >will be expanded to:
> >
> >local mod_source_files=3Dsource1.c source2.c
>
> Yes, I was able to have nsdeps work for composite modules with just my
> patch. Without this patch applied, the script produces the following
> expansion of the generate_deps_for_ns call, (I just added a test
> namespace MODULE):
>
> Adding namespace MODULE to module fs/nfs/nfs.ko.
> + generate_deps_for_ns MODULE /tmp/ppyu/linux/fs/nfs/client.c /tmp/ppyu/l=
inux/fs/nfs/dir.c /tmp/ppyu/linux/fs/nfs/file.c /tmp/ppyu/linux/fs/nfs/getr=
oot.c /tmp/ppyu/linux/fs/nfs/inode.c /tmp/ppyu/linux/fs/nfs/super.c /tmp/pp=
yu/linux/fs/nfs/io.c /tmp/ppyu/linux/fs/nfs/direct.c /tmp/ppyu/linux/fs/nfs=
/pagelist.c /tmp/ppyu/linux/fs/nfs/read.c /tmp/ppyu/linux/fs/nfs/symlink.c =
/tmp/ppyu/linux/fs/nfs/unlink.c /tmp/ppyu/linux/fs/nfs/write.c /tmp/ppyu/li=
nux/fs/nfs/namespace.c /tmp/ppyu/linux/fs/nfs/mount_clnt.c /tmp/ppyu/linux/=
fs/nfs/nfstrace.c /tmp/ppyu/linux/fs/nfs/export.c /tmp/ppyu/linux/fs/nfs/sy=
sfs.c /tmp/ppyu/linux/fs/nfs/sysctl.c /tmp/ppyu/linux/fs/nfs/fscache.c /tmp=
/ppyu/linux/fs/nfs/fscache-index.c
> + /usr/bin/spatch --very-quiet --in-place --sp-file /tmp/ppyu/linux/scrip=
ts/coccinelle/misc/add_namespace.cocci -D ns=3DMODULE /tmp/ppyu/linux/fs/nf=
s/client.c
>
> So only the first file got included in the spatch invocation. But the
> spatch call gets fixed with all the files when quotes are added in the
> call to generate_deps_for_ns.
>
> But we need to include your change anyway, to make the script more
> robust.

Hmm.
With this patch only, I see "bad variable name" error.


masahiro@grover:~/ref/linux$ make -j8  nsdeps
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  Building modules, stage 2.
  MODPOST 20 modules
WARNING: module nfs uses symbol foo from namespace USB_STORAGE, but
does not import it.
  Building modules, stage 2.
  MODPOST 20 modules
./scripts/nsdeps: 34: local: ./fs/nfs/dir.c: bad variable name
make: *** [Makefile;1689: nsdeps] Error 2


> It would probably prevent more shell script related bugs in
> the future (Like [1]). I can respin this patch only while the other
> ones are superceded by your patchset.
>
> [1] https://unix.stackexchange.com/a/131767

Anyway.

Is this patch aiming for v5.4 (i.e. fixes) or v5.5-rc1 ?

If you touch the mod_source_files line,
we will have a conflict because
https://patchwork.kernel.org/patch/11217839/
is touching the same line.

How should we organize the patch order?

--
Best Regards
Masahiro Yamada