spdxcheck -v output is just providing basic statistics, but lacks per
directory statistics.
Finding files without SPDX identifiers is cumbersome with spdxcheck, though
it has all the information required.
The exclude of files and directories is hardcoded in the script which makes
it hard to maintain and the information cannot be accessed by external tools.
The following series addresses this by adding:
1) Directory statistics
Incomplete directories: SPDX in Files
./ : 6 of 13 46%
./Documentation : 4096 of 8451 48%
./arch : 13476 of 16402 82%
./block : 100 of 101 99%
./certs : 11 of 14 78%
./crypto : 145 of 176 82%
./drivers : 24682 of 30745 80%
2) The ability to show files without SPDX
Files without SPDX:
./kernel/cpu.c
./kernel/kmod.c
./kernel/relay.c
3) A file based handling for exclude patterns
Thanks,
tglx
On Mon, May 16 2022 at 15:14, Max Mehl wrote:
> Thank you for picking up the effort to add license (and perhaps also
> copyright) info to all files in the Kernel.
Adding copyright notices retroactively is not going to happen
ever. That's just impossible.
>> The exclude of files and directories is hardcoded in the script which makes
>> it hard to maintain and the information cannot be accessed by external tools.
>
> Unfortunately, excluding files (i.e. not adding machine-readable
> license/copyright information to it) would also block reaching full
> compliance with the REUSE best practices. Have you considered making
> them available under GPL-2.0-only or a license similar to public domain
> [^2]?
The LICENSE directory is already handled by spdxcheck as the license
information is read from there. And no, we cannot add a GPL-2.0-only
identifier to all of the files under the LICENSE directory for obvious
reasons.
license-rules.rst is not longer a problem as all incarnations have a
proper SPDX identifier today.
Thanks,
tglx
Thank you for picking up the effort to add license (and perhaps also
copyright) info to all files in the Kernel. This, as you know, is also
working towards making the whole repo REUSE compliant [^1].
~ Thomas Gleixner [2022-05-16 12:27 +0200]:
> Finding files without SPDX identifiers is cumbersome with spdxcheck, though
> it has all the information required.
>
> The exclude of files and directories is hardcoded in the script which makes
> it hard to maintain and the information cannot be accessed by external tools.
Unfortunately, excluding files (i.e. not adding machine-readable
license/copyright information to it) would also block reaching full
compliance with the REUSE best practices. Have you considered making
them available under GPL-2.0-only or a license similar to public domain
[^2]?
Regarding false-positives, e.g. in license-rules.rst, you could use the
brand-new feature that allows to ignore blocks of code (to be released
later this week) [^3]. I am aware that spdxcheck would not be able to
detect this, but using the REUSE helper tool [^4] could also be a
solution to scan for missing files.
Best,
Max
[^1]: https://reuse.software
[^2]: https://reuse.software/faq/#exclude-file
[^3]: https://github.com/fsfe/reuse-docs/pull/104/files
[^4]: https://github.com/fsfe/reuse-tool
--
Max Mehl - Programme Manager -- Free Software Foundation Europe
Contact and information: https://fsfe.org/about/mehl -- @mxmehl
The FSFE is a charity that empowers users to control technology
On Mon, May 16 2022 at 20:52, Thomas Gleixner wrote:
> On Mon, May 16 2022 at 15:14, Max Mehl wrote:
>> Thank you for picking up the effort to add license (and perhaps also
>> copyright) info to all files in the Kernel.
>
> Adding copyright notices retroactively is not going to happen
> ever. That's just impossible.
>
>>> The exclude of files and directories is hardcoded in the script which makes
>>> it hard to maintain and the information cannot be accessed by external tools.
>>
>> Unfortunately, excluding files (i.e. not adding machine-readable
>> license/copyright information to it) would also block reaching full
>> compliance with the REUSE best practices. Have you considered making
>> them available under GPL-2.0-only or a license similar to public domain
>> [^2]?
>
> The LICENSE directory is already handled by spdxcheck as the license
> information is read from there. And no, we cannot add a GPL-2.0-only
> identifier to all of the files under the LICENSE directory for obvious
> reasons.
>
> license-rules.rst is not longer a problem as all incarnations have a
> proper SPDX identifier today.
There is also an argument to be made whether we really need to have SPDX
identifiers on trivial files:
#include <someheader.h>
<EOF>
Such files are not copyrightable by any means. So what's the value of
doubling the line count to add an SPDX identifier? Just to make nice
statistics?
Thanks,
tglx
~ Thomas Gleixner [2022-05-16 20:59 +0200]:
> On Mon, May 16 2022 at 20:52, Thomas Gleixner wrote:
>>> Unfortunately, excluding files (i.e. not adding machine-readable
>>> license/copyright information to it) would also block reaching full
>>> compliance with the REUSE best practices. Have you considered making
>>> them available under GPL-2.0-only or a license similar to public domain
>>> [^2]?
>>
>> The LICENSE directory is already handled by spdxcheck as the license
>> information is read from there. And no, we cannot add a GPL-2.0-only
>> identifier to all of the files under the LICENSE directory for obvious
>> reasons.
Absolutely. REUSE obviously also ignores this directory, as well as
e.g. zero-length files, symlinks, submodules, or .git directory.
> There is also an argument to be made whether we really need to have SPDX
> identifiers on trivial files:
>
> #include <someheader.h>
> <EOF>
>
> Such files are not copyrightable by any means. So what's the value of
> doubling the line count to add an SPDX identifier? Just to make nice
> statistics?
We agree that such files are not copyrightable. But where is the
threshold? Lines of code? Creativity? Number of used functions? And how
to embed this threshold in tooling? So instead of fuzzy exclusion of
such files in tools like spdxcheck or REUSE, it makes sense to treat
them as every other file with the cost of adding two comment lines.
This clear-cut rule eases maintaining and growing the effort you and
others did because developers would know exactly what to add to a new
file (license + copyright) without requiring looking up the thresholds
or a manual review by maintainers who can interpret them.
Best,
Max
--
Max Mehl - Programme Manager -- Free Software Foundation Europe
Contact and information: https://fsfe.org/about/mehl -- @mxmehl
The FSFE is a charity that empowers users to control technology
On Tue, May 17 2022 at 10:25, Max Mehl wrote:
> ~ Thomas Gleixner [2022-05-16 20:59 +0200]:
>> There is also an argument to be made whether we really need to have SPDX
>> identifiers on trivial files:
>>
>> #include <someheader.h>
>> <EOF>
>>
>> Such files are not copyrightable by any means. So what's the value of
>> doubling the line count to add an SPDX identifier? Just to make nice
>> statistics?
>
> We agree that such files are not copyrightable. But where is the
> threshold? Lines of code? Creativity? Number of used functions? And how
> to embed this threshold in tooling? So instead of fuzzy exclusion of
> such files in tools like spdxcheck or REUSE, it makes sense to treat
> them as every other file with the cost of adding two comment lines.
>
> This clear-cut rule eases maintaining and growing the effort you and
> others did because developers would know exactly what to add to a new
> file (license + copyright) without requiring looking up the thresholds
> or a manual review by maintainers who can interpret them.
Seriously no. I'm outright refusing to add my copyright to a trivial
file with one or two includes or a silly comment like '/* empty because */.
There is nothing copyrightable there.
I'm not going to make myself a fool just to make tools happy, which can
figure out on their own whether there is reasonable content in the vast
majority of cases.
Also you need some exclude rules in any case. Why?
- How do you tell a tool that a file is generated, e.g. in the kernel
the default configuration files?
Yes, the file content depends on human input to the generator tool,
but I'm looking forward for the explanation how this is
copyrightable especially with multiple people updating this file
over time where some of the updates are just done by invoking the
generator tool itself.
- How do you tell a tool that a file contains licensing documentation?
Go and look what license scanners make out of all the various
license-rules.rst files.
- ....
Do all scanners have to grow heuristics for ignoring the content past
the topmost SPDX License identifier in certain files or for figuring
out what might be generated content?
You also might need to add information about binary blobs, which
obviously cannot be part of the binary blobs themself.
The exclude rules I added are lazy and mostly focussed on spdxcheck, but
I'm happy to make them more useful and let them carry information about
the nature of the exclude or morph them into a general scanner info
which also contains binary blob info and other helpful information. But
that needs a larger discussion about the format and rules for such a
file.
That said, I'm all for clear cut rules, but rules just for the rules
sake are almost as bad as no rules at all.
As always you have to apply common sense and look at the bigger picture
and come up with solutions which are practicable, enforcable and useful
for the larger eco-system.
Your goal of having SPDX ids and copyright notices in every file of a
project is honorable, but impractical for various reasons.
See above.
Aside of that you cannot replace a full blown license scanner by REUSE
even if your project is SPDX and Copyright notice clean at the top level
of a file. You still need to verify that there is no other information
in a 'clean' file which might be contradicting or supplemental. You
cannot add all of this functionality to REUSE or whatever.
Thanks,
tglx
On 5/17/22 3:43 PM, Thomas Gleixner wrote:
> On Tue, May 17 2022 at 10:25, Max Mehl wrote:
>> ~ Thomas Gleixner [2022-05-16 20:59 +0200]:
>>> There is also an argument to be made whether we really need to have SPDX
>>> identifiers on trivial files:
>>>
>>> #include <someheader.h>
>>> <EOF>
>>>
>>> Such files are not copyrightable by any means. So what's the value of
>>> doubling the line count to add an SPDX identifier? Just to make nice
>>> statistics?
>> We agree that such files are not copyrightable. But where is the
>> threshold? Lines of code? Creativity? Number of used functions? And how
>> to embed this threshold in tooling? So instead of fuzzy exclusion of
>> such files in tools like spdxcheck or REUSE, it makes sense to treat
>> them as every other file with the cost of adding two comment lines.
>>
>> This clear-cut rule eases maintaining and growing the effort you and
>> others did because developers would know exactly what to add to a new
>> file (license + copyright) without requiring looking up the thresholds
>> or a manual review by maintainers who can interpret them.
> Seriously no. I'm outright refusing to add my copyright to a trivial
> file with one or two includes or a silly comment like '/* empty because */.
>
> There is nothing copyrightable there.
>
> I'm not going to make myself a fool just to make tools happy, which can
> figure out on their own whether there is reasonable content in the vast
> majority of cases.
>
> Also you need some exclude rules in any case. Why?
>
> - How do you tell a tool that a file is generated, e.g. in the kernel
> the default configuration files?
>
> Yes, the file content depends on human input to the generator tool,
> but I'm looking forward for the explanation how this is
> copyrightable especially with multiple people updating this file
> over time where some of the updates are just done by invoking the
> generator tool itself.
>
> - How do you tell a tool that a file contains licensing documentation?
>
> Go and look what license scanners make out of all the various
> license-rules.rst files.
>
> - ....
>
> Do all scanners have to grow heuristics for ignoring the content past
> the topmost SPDX License identifier in certain files or for figuring
> out what might be generated content?
>
> You also might need to add information about binary blobs, which
> obviously cannot be part of the binary blobs themself.
>
> The exclude rules I added are lazy and mostly focussed on spdxcheck, but
> I'm happy to make them more useful and let them carry information about
> the nature of the exclude or morph them into a general scanner info
> which also contains binary blob info and other helpful information. But
> that needs a larger discussion about the format and rules for such a
> file.
>
> That said, I'm all for clear cut rules, but rules just for the rules
> sake are almost as bad as no rules at all.
>
> As always you have to apply common sense and look at the bigger picture
> and come up with solutions which are practicable, enforcable and useful
> for the larger eco-system.
>
> Your goal of having SPDX ids and copyright notices in every file of a
> project is honorable, but impractical for various reasons.
>
> See above.
>
> Aside of that you cannot replace a full blown license scanner by REUSE
> even if your project is SPDX and Copyright notice clean at the top level
> of a file. You still need to verify that there is no other information
> in a 'clean' file which might be contradicting or supplemental. You
> cannot add all of this functionality to REUSE or whatever.
>
Max, Thomas,
I think the discussion here is hitting upon the "inconvenience" of the
lack of black/white rules in the law (as to what is copyrightable)
versus the convenience of downstream recipients of code who want to be
sure they have proper rights (which mixes in the guidance/rules of
Reuse, tooling, etc.).
I think some rules in terms of files that are clearly not copyrightable
can be implemented in various tooling (hopefully, with the guidance of a
lawyer steeped in copyright law), and I agree that putting a license (by
way of an SPDX identifier or any other way for that matter) on such
files is neither a good use of time nor a good idea (from the
perspective of being inaccurate as to the need for a license and thus
sending the wrong impression). That being said, there will not be a way
to make clear cut rules for everything, without involving a judge.
Sorry! That's just how the law works (and we actually often don't want
black/white lines in the law, actually).
I can see a policy of, "when it's not clear (as to copyrightability),
then add a license", though.
Thanks,
Jilayne
On Mon, May 23 2022 at 10:11, J. Lovejoy wrote:
> On 5/17/22 3:43 PM, Thomas Gleixner wrote:
> I think the discussion here is hitting upon the "inconvenience" of the
> lack of black/white rules in the law (as to what is copyrightable)
> versus the convenience of downstream recipients of code who want to be
> sure they have proper rights (which mixes in the guidance/rules of
> Reuse, tooling, etc.).
Correct.
> I think some rules in terms of files that are clearly not copyrightable
> can be implemented in various tooling (hopefully, with the guidance of a
> lawyer steeped in copyright law), and I agree that putting a license (by
> way of an SPDX identifier or any other way for that matter) on such
> files is neither a good use of time nor a good idea (from the
> perspective of being inaccurate as to the need for a license and thus
> sending the wrong impression). That being said, there will not be a way
> to make clear cut rules for everything, without involving a judge.
> Sorry! That's just how the law works (and we actually often don't want
> black/white lines in the law, actually).
>
> I can see a policy of, "when it's not clear (as to copyrightability),
> then add a license", though.
No argument here, but trivial things like an include which file includes
another include file are pretty clear IMO and we really should make our
mind up on those. Even a header file which contains a single function
declaration is questionable at best, but yes it's hard to put a hard
line on those.
Thanks,
tglx