From: Keith Owens <kaos@ocs.com.au>
To: "Justin T. Gibbs" <gibbs@scsiguy.com>
Cc: linux-kernel@vger.kernel.org
Subject: Re: [patch] 2.4.19-pre6 standardize {aic7xxx,aicasm}/Makefile 
In-Reply-To: Your message of "Sun, 07 Apr 2002 22:27:40 CST."
             <200204080427.g384Re999479@aslan.scsiguy.com> 
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Date: Wed, 10 Apr 2002 12:41:43 +1000
Message-ID: <2317.1018406503@kao2.melbourne.sgi.com>
Sender: linux-kernel-owner@vger.kernel.org

On Sun, 07 Apr 2002 22:27:40 -0600, 
"Justin T. Gibbs" <gibbs@scsiguy.com> wrote:
>kaos wrote
>>Having multiple conglomerates gets messy, especially if you allow a
>>mixture of built in and modular selection and if it is possible for
>>everything to be a module with no built in stubs.  The generic case
>>looks like this
>
>Since this is the case, and the Makefile will return to this format
>as soon as the U320 driver is released, can we come up with an
>interrim Makefile that assumes the new driver will show up shortly?

I renew my standing offer that goes back to June 2001.  If you agree to

* use the standard Linux kernel build methods
* stop shipping files and overwriting them at run time
* make the decision about firmware generation automatic instead of
  manual
* remove the BSD'isms from the Linux aic7xxx Makefiles

then I will happily write clean aic7xxx makefiles and support them.
But if you insist on doing it your own way that does not match the
Linux kernel build, then I am afraid that you are on your own.


Why shipping files and overwritten them at build time is a bad idea,
and why relying on a manual switch to regenerate files is the lazy
approach.  From kbuild-2.5/Documentation/kbuild/kbuild-2.5.txt.

SHIPPING GENERATED FILES

  In general it is bad practice to include generated files in the kernel
  source tree.  The only exceptions are when the generating code is not in
  the kernel or when the generating code is in the kernel but it requires
  additional tools which not all users have installed.  An example of a
  generated file without the code to regenerate it is firmware for a network
  driver.  An example of a generation process that requires extra tools is
  drivers/char/defkeymap.c which requires the loadkeys program.

  There is no justification for shipping any other generated file as part of
  the kernel.  This is especially true if the user is always expected to
  regenerate the file, even more so if the generated data depends upon the
  user's .config.  Under these circumstances it is dangerous to ship a
  generated file because it can be used when it does not reflect the user's
  kernel.

  When a generated file has no kernel support to regenerate it, the file can
  be shipped as is.  The question of whether including generated files in the
  kernel without the underlying code is a violation of the kernel license is
  outside the scope of this document.

  For generated files where the code is included but not all users can run
  it, you must be careful about when the generated file is rebuilt.  make
  looks at the time stamp of the generating and generated files to decide if
  the output needs to be regenerated.  This works for the person making the
  changes because they control when the files are updated.

  Time stamps do not work when the change is issued to the rest of the world
  as a diff.  After patch has run you cannot guarantee which of the updated
  files has the more recent time.  The patch program does not preserve time
  stamps (it must not) so the time stamps on the changed files depend on the
  order of the entries in the patch set.  There is no defined order for
  entries in a patch set, it depends on the program that generated the
  difference listing.  GNU diff generates patches in alphabetical order but
  only when it is given an entire directory.  When diff is invoked from a
  source repository tool it is typically called once for each file and the
  file order is controlled by the repository tool.  Even if your repository
  generates the patch in the correct order, that order can change when the
  patch is merged into the kernel.

  Since you cannot rely on time stamps to decide if a generated and shipped
  file needs to be regenerated, you must use another mechanism for this case.
  There are two basic possibilities, manual or automatic (otherwise known as
  lazy or correct).

  In the manual case you ignore changes to the generated or generating files
  until the user performs some manual operation, such as selecting
  CONFIG_REGENERATE_foo.  This is easy to implement but it is also the lazy
  approach, making the end user do extra work to save the maintainer doing
  anything.  This goes against the whole idea of *automatically* deciding
  what needs to be rebuilt, instead it relies on human intervention.

  In the automatic case the maintainer has to do more work but the make rules
  can automatically determine if the end user has changed any of the related
  files and regenerate automatically.  This is the correct approach.  In the
  absence of reliable time stamps you need another method to check if the
  generating and generated files are in sync or not.  The obvious method is a
  checksum of files.

  Given a generated file 'foo' and a script to generate it 'foo-gen' which
  not all users can run because they may not have the required software
  installed.

    foo_files := $(srcfile foo-gen) $(srcfile foo.out_shipped)
    $(objfile foo_sum.d): $(srcfile foo_sum) $(foo_files)
            $(KBUILD_QUIET)(set -e;
	      mkdir -p $(objdir); \
              sed -e 's:$$[ABD-Z][^$$]*\$$::g' $(foo_files) > \
                $(objfile .tmp_foo_files); \
              ($(MD5SUM) -c $(srcfile foo_sum) >/dev/null 2>&1 && \
                echo S OK || echo F) > $@; \
              rm $(objfile .tmp_foo_files); \
            )

    user_command(foo.out
            ($(objfile foo_sum.d))
            ([ "`cat $<`" = "S OK" ] &&
              (cp $(srcfile $(@F)_shipped) $@) ||
              (set -ex ; $(srcfile foo-gen) > $(objfile foo.out)))
            ()
            )

    # Exception: this writes to the source tree.  The target must be
    # explicitly requested.
    .PHONY: foo_sum_shipped
    foo_sum_shipped: $(objfile foo.out)
            @cmp -s $(objfile foo.out) $(srcfile foo.out_shipped) || \
                 mv $(objfile foo.out) $(srcfile foo.out_shipped)
            @( \
             set -e; \
             sed -e 's:$$[ABD-Z][^$$]*\$$::g' $(foo_files) > \
                $(objfile .tmp_foo_files); \
              $(MD5SUM) $(objfile .tmp_foo_files) > $(objfile .tmp_$(@F)) && \
              cmp -s $(objfile .tmp_$(@F)) $(srcfile foo_sum) || \
                (echo Updating checksum for foo; \
                 mv $(objfile .tmp_$(@F)) $(srcfile foo_sum)); \
            )
            @rm -f $(objfile foo_sum.d) $(objfile .tmp_$(@F)) \
              $(objfile .tmp_foo_files)

    ifsel(CONFIG_foo)
      all_sum_shipped: foo_sum_shipped
    endif

  That is boiler plate code which can be used for any shipped files, by
  globally replacing 'foo-gen' and 'foo'.  To initialize the sequence, touch
  $(srcfile foo_sum) and $(srcfile foo.out_shipped) then make
  foo_sum_shipped.  Thereafter any real change to the foo files will
  regenerate foo, ignoring any spurious time stamp differences caused by
  patch.  It removes all need for special configuration options, if anybody
  changes the input files then kbuild automatically rebuilds foo when it is
  used.  Users who do not change the input files get the shipped version when
  foo is required.

    foo_files       - List all the source files, including the generating
                      code and the shipped version of foo.  The list does not
                      include foo_sum.
    foo_sum         - Checksum over $(foo_files), i.e. the master copies.
                      The checksum ignores RCS id and date strings.
    foo_sum.d       - Built on the first make and when any of the input files
                      are updated after the first make.  foo_sum.d either
                      contains 'S OK' (sum is valid) or 'F' (failed).
    user_command    - Creates the working copy of foo, either from the
                      shipped copy (sum is valid) or by running foo-gen (sum
                      does not match, a real change has occurred).  Watch the
                      parenthesis in the user_command, in particular the use
                      of foo-gen requires () around the set of commands.
    foo_sum_shipped - Takes the latest version of foo, generating it if
                      necessary and writes it back to the source tree as
                      foo.out_shipped.  Generates a new checksum
                      automatically.  Note: This command must be manually
                      specified by the foo maintainer before shipping a new
                      version of foo.

  For a concrete example, see defkeymap.c in drivers/char/Makefile.in.

  Because this is boiler plate code, it has been automated.

    shipped(prefix
             (generated files)
             (generating files)
             (commands to create all the generated files)
             (CONFIG dependencies to select all_sum_shipped)
           )

  The first field is the prefix for variables and for _sum files.  It must be
  globally unique in the kbuild files.

  The generated files are bare names, without '/', without '_shipped', no
  $(objfile) or $(srcfile) is allowed, in fact no $(variables) are accepted.
  These names are used to create the user_command() and the 'cp' commands,
  one for each generated file.  Because user_command() only takes one target,
  all the other generated files are defined as side_effect()s of the last
  target, you must list the generated files in the order that they are
  created by your commands.

  The generating files are bare names and list the source files used to
  generate the targets.  For example, scripts, tables, lex or yacc code.

  The commands define how you create the target files if the shipped versions
  of the targets are no longer valid, i.e. the checksum has failed.  I
  recommend that the first command is 'set -ex;'.

  The config dependencies are zero or more configs to check.  If any are
  selected then all_sum_shipped is extended with foo_sum_shipped.

  The code above shrinks to these few lines.

    shipped(foo
             (foo.out)
             (foo-gen)
             (set -ex ; $(srcfile foo-gen) > $(objfile foo.out))
             (CONFIG_foo)
           )

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/