Hi Greg,
(Cc-ing oss-security because I think more people there might be
interested than people subscribed to [email protected] and I didn't want to
cross-post to multiple lists)
Up until last month someone had been managing a linuxkernelcves[1][2]
site, but it's somehow gone without a trace (DNS emptied, no message I
could see announcing it anywhere)
[1] https://www.linuxkernelcves.com
[2] https://github.com/nluedtke/linux_kernel_cves
With the new vulns[3] repo I thought I could do similar search there,
but while there are scripts to search by commit ID or by CVE I don't see
anything allowing search for issues affecting a given stable release.
[3] https://git.kernel.org/pub/scm/linux/security/vulns.git/
My motivation here is double:
- We notify our users of notable CVEs fixed on every update to encourage
them to upgrade every time (it's sad, but in the embedded world not
updating is still the norm despite our efforts to make upgrades as
painless as possible... New regulations are coming so hopefully that
will slowly improve, but as of now such motivations help)
- I'm currently not watching patches entering newer stable branches as
closely, so if there are any new CVEs not fixed in the latest 5.10 I'd
like to check if some impact us and will help with backports as possible
(we're a small company so my time is limited, but might as well give
back when I can)
The information is there in the json files, so it's just a matter of
writing some scripts to check them, but I can't believe there's none so
I probably have missed something.
Does someone have such a script that'd list the latest CVEs for a given
tree?
Thanks,
--
Dominique Martinet | Asmadeus
On Wed, May 29, 2024 at 09:53:48AM +0900, Dominique Martinet wrote:
>
> With the new vulns[3] repo I thought I could do similar search there,
> but while there are scripts to search by commit ID or by CVE I don't see
> anything allowing search for issues affecting a given stable release.
>
> [3] https://git.kernel.org/pub/scm/linux/security/vulns.git/
True, we don't have that yet, but with the scripts in there, it should
be easy to knock this up (hint, pass the id to scripts/cve_search) if
you need it.
> My motivation here is double:
> - We notify our users of notable CVEs fixed on every update to encourage
> them to upgrade every time (it's sad, but in the embedded world not
> updating is still the norm despite our efforts to make upgrades as
> painless as possible... New regulations are coming so hopefully that
> will slowly improve, but as of now such motivations help)
The issue is, CVEs are assigned usually long _AFTER_ the stable release
has happened. So if you want to do this type of report for the latest
stable release, it will look like there are no CVEs. But if you wait a
few weeks, suddenly that old release will have many CVEs assigned to
them.
This is just due to the process we currently have where we review each
commit in the stable releases to determine if a CVE should be assigned
or not. Obviously this takes time and we are running a few weeks behind
the current releases.
So you would have to run the script a lot, to keep it up to date, which
is why a "how many CVEs are listed in the latest release" isn't really
going to be all that valuable to your users.
> - I'm currently not watching patches entering newer stable branches as
> closely, so if there are any new CVEs not fixed in the latest 5.10 I'd
> like to check if some impact us and will help with backports as possible
> (we're a small company so my time is limited, but might as well give
> back when I can)
That would be great, for where we know, we list when a vulnerability was
added to the tree, and where it was fixed. That can leave many branches
still vulnerable where we have not fixed the issue yet. One example
would be CVE-2024-26629.
You can see these in our repo by just doing:
git grep "5\.10" | grep introduced | grep -v fixed
But note that for some issues, we don't have the information for when
they are introduced, so if they are not fixed in the 5.10 branch, does
that mean the branch _is_ vulnerable, or is not? One example of a "is
not" might be CVE-2024-35867 as we think the code isn't present in 5.10,
but we don't have an automated way of determining that. So that would
take more work than just a simple grep of the tree.
> The information is there in the json files, so it's just a matter of
> writing some scripts to check them, but I can't believe there's none so
> I probably have missed something.
>
> Does someone have such a script that'd list the latest CVEs for a given
> tree?
How about something as simple as the following to see what is in
5.10.101:
for id in $(git log --format="%H" v5.10.100..v5.10.101); do
cve=$(cve_search ${id})
cve_found=$?
if [[ "${cve_found}" == "0" ]]; then
echo "${cve} is in range"
fi
done
Note, typed in email client, not tested, use at your own risk...
hope this helps,
greg k-h
Greg Kroah-Hartman wrote on Wed, May 29, 2024 at 09:23:50PM +0200:
> > The information is there in the json files, so it's just a matter of
> > writing some scripts to check them, but I can't believe there's none so
> > I probably have missed something.
> >
> > Does someone have such a script that'd list the latest CVEs for a given
> > tree?
>
> How about something as simple as the following to see what is in
> 5.10.101:
>
> for id in $(git log --format="%H" v5.10.100..v5.10.101); do
> cve=$(cve_search ${id})
> cve_found=$?
> if [[ "${cve_found}" == "0" ]]; then
(pedantic: `if cve=$(cve_search "$id"); then` is a bit simpler/failproof)
> echo "${cve} is in range"
> fi
> done
That's roughly what I had done earlier this week (handpicking the
commits that could impact our users), but this doesn't address my second
point as it won't catch any new CVE introduced before that tree that
wasn't fixed.
(also probably a bit more efficient to go by version tag since we have
the info in the json, more below)
> > My motivation here is double:
> > - We notify our users of notable CVEs fixed on every update to encourage
> > them to upgrade every time (it's sad, but in the embedded world not
> > updating is still the norm despite our efforts to make upgrades as
> > painless as possible... New regulations are coming so hopefully that
> > will slowly improve, but as of now such motivations help)
>
> The issue is, CVEs are assigned usually long _AFTER_ the stable release
> has happened. So if you want to do this type of report for the latest
> stable release, it will look like there are no CVEs. But if you wait a
> few weeks, suddenly that old release will have many CVEs assigned to
> them.
>
> This is just due to the process we currently have where we review each
> commit in the stable releases to determine if a CVE should be assigned
> or not. Obviously this takes time and we are running a few weeks behind
> the current releases.
>
> So you would have to run the script a lot, to keep it up to date, which
> is why a "how many CVEs are listed in the latest release" isn't really
> going to be all that valuable to your users.
Right; I don't need this to be 100% complete -- as long as a couple of
issues turn up it's probably good enough motivation.
In practice just listing a bunch of numbers probably won't change the
way people think, so I'm taking the time to briefly describe potential
impacts (what component, very broad trigger conditions e.g. network
packet or local access, likely risk if exploited e.g. RCE, memory
leak...); so ultimately it requires looking at things in more details
than I have time to check for all CVEs and will likely keep checking a
few "juicy" ones...
But it's a very good point, we should check again regularly and update
that list if some new bad thing stands out.
> > - I'm currently not watching patches entering newer stable branches as
> > closely, so if there are any new CVEs not fixed in the latest 5.10 I'd
> > like to check if some impact us and will help with backports as possible
> > (we're a small company so my time is limited, but might as well give
> > back when I can)
>
> That would be great, for where we know, we list when a vulnerability was
> added to the tree, and where it was fixed. That can leave many branches
> still vulnerable where we have not fixed the issue yet. One example
> would be CVE-2024-26629.
>
> You can see these in our repo by just doing:
> git grep "5\.10" | grep introduced | grep -v fixed
I didn't think of checking the mails, that's certainly easier to grep
than json as it's line-oriented.
It's going to take a bit more of processing to check not just bugs that
were backported in the stable trees, but things introduced in earlier
kernels... Someting like this?
rg -l 'Issue introduced in ([234]\.[0-9]* |5\.[0-9] |5\.10\.[0-9]* )' | sort > introduced_before_5.10
xargs rg -l 'fixed in 5\.10' < introduced_before_5.10 | sort > fixed_in_5.10
comm -3 introduced_before_5.10 fixed_in_5.10 |tail
cve/published/2024/CVE-2024-35844.mbox
cve/published/2024/CVE-2024-35904.mbox
cve/published/2024/CVE-2024-35951.mbox
cve/published/2024/CVE-2024-35971.mbox
cve/published/2024/CVE-2024-36009.mbox
cve/published/2024/CVE-2024-36013.mbox
grep 'Issue introdu' cve/published/2024/CVE-2024-35971.mbox
Issue introduced in 5.8 with commit 797047f875b5 and fixed in 6.1.87 with commit 492337a4fbd1
Issue introduced in 5.8 with commit 797047f875b5 and fixed in 6.6.28 with commit cba376eb036c
Issue introduced in 5.8 with commit 797047f875b5 and fixed in 6.8.7 with commit 49d5d70538b6
Issue introduced in 5.8 with commit 797047f875b5 and fixed in 6.9 with commit be0384bf599c
The regex is a bit too manual to make a generic search script, and that
feels very kludgy (at least mbox files do look like they get updated
together with json), but that can be enough for my local needs for now.
I was thinking something more along the line of parsing all the json
files for containers.cna.affected by release version item (versionType
!= git);
It should be possible for a given stable tag to check if a given CVE
applies or not immediately so it would be a matter of making this a bit
more searchable -- probably make a reverse index with all the edges for
faster search and keep appending new CVEs as they pop up.
But it's a bit more work, so I'll gratefully take the grep mailboxes
version for now :)
> But note that for some issues, we don't have the information for when
> they are introduced, so if they are not fixed in the 5.10 branch, does
> that mean the branch _is_ vulnerable, or is not? One example of a "is
> not" might be CVE-2024-35867 as we think the code isn't present in 5.10,
> but we don't have an automated way of determining that. So that would
> take more work than just a simple grep of the tree.
Yes, these (basically patches without a Fixes:) will always be
problematic; basically would need to also check all mails with "Fixed in
[newer versions]" without the "Issue introduced in" text.
Capitalization on Fixed can differentiate this so something like this?
rg -l 'Issue introduced in ([234]\.[0-9]* |5\.[0-9] |5\.10\.[0-9]* )|Fixed in'
This is bringing up the list of candidates from 101 to 543 so the
initial check is going to be "fun"; I'll probably stick to the first
variant for now...
Also, in this case the json properly defaults to affected so a search by
json as suggested above would also turn them up for further inspection.
Well, either way we won't get around the problem that much manual work
is still required; as said in my first mail my time is quite limited but
I'll try to check a few from time to time.
Ideally we'll want to limit duplicating this work for other
downstreams... So:
- get more people to look at these
- if unaffected (e.g. CVE-2024-35867 you singled out above as not
affecting 5.10), report it so the reference files can be updated
- if affected backport patch, so it can be fixed and the refernece
file can also be updated.
- less work for everyone else!
But finding volunteers for that kind of work might not be quite as easy
as I make it sound like :)
Cheers,
--
Dominique Martinet | Asmadeus
On Thu, May 30, 2024 at 01:45:39PM +0900, Dominique Martinet wrote:
> Greg Kroah-Hartman wrote on Wed, May 29, 2024 at 09:23:50PM +0200:
> > > The information is there in the json files, so it's just a matter of
> > > writing some scripts to check them, but I can't believe there's none so
> > > I probably have missed something.
> > >
> > > Does someone have such a script that'd list the latest CVEs for a given
> > > tree?
> >
> > How about something as simple as the following to see what is in
> > 5.10.101:
> >
> > for id in $(git log --format="%H" v5.10.100..v5.10.101); do
> > cve=$(cve_search ${id})
> > cve_found=$?
> > if [[ "${cve_found}" == "0" ]]; then
> (pedantic: `if cve=$(cve_search "$id"); then` is a bit simpler/failproof)
Very true, I do not claim to be a "robust" bash programmer at all :)
> > echo "${cve} is in range"
> > fi
> > done
>
> That's roughly what I had done earlier this week (handpicking the
> commits that could impact our users), but this doesn't address my second
> point as it won't catch any new CVE introduced before that tree that
> wasn't fixed.
True.
> (also probably a bit more efficient to go by version tag since we have
> the info in the json, more below)
Yeah, but the json files have their own issues, more below...
> > > My motivation here is double:
> > > - We notify our users of notable CVEs fixed on every update to encourage
> > > them to upgrade every time (it's sad, but in the embedded world not
> > > updating is still the norm despite our efforts to make upgrades as
> > > painless as possible... New regulations are coming so hopefully that
> > > will slowly improve, but as of now such motivations help)
> >
> > The issue is, CVEs are assigned usually long _AFTER_ the stable release
> > has happened. So if you want to do this type of report for the latest
> > stable release, it will look like there are no CVEs. But if you wait a
> > few weeks, suddenly that old release will have many CVEs assigned to
> > them.
> >
> > This is just due to the process we currently have where we review each
> > commit in the stable releases to determine if a CVE should be assigned
> > or not. Obviously this takes time and we are running a few weeks behind
> > the current releases.
> >
> > So you would have to run the script a lot, to keep it up to date, which
> > is why a "how many CVEs are listed in the latest release" isn't really
> > going to be all that valuable to your users.
>
> Right; I don't need this to be 100% complete -- as long as a couple of
> issues turn up it's probably good enough motivation.
>
> In practice just listing a bunch of numbers probably won't change the
> way people think, so I'm taking the time to briefly describe potential
> impacts (what component, very broad trigger conditions e.g. network
> packet or local access, likely risk if exploited e.g. RCE, memory
> leak...); so ultimately it requires looking at things in more details
> than I have time to check for all CVEs and will likely keep checking a
> few "juicy" ones...
> But it's a very good point, we should check again regularly and update
> that list if some new bad thing stands out.
Great. Only you know your use cases, which is why we do not offer up
any "grading" of kernel CVEs as Linux is used in so many different ways.
> > > - I'm currently not watching patches entering newer stable branches as
> > > closely, so if there are any new CVEs not fixed in the latest 5.10 I'd
> > > like to check if some impact us and will help with backports as possible
> > > (we're a small company so my time is limited, but might as well give
> > > back when I can)
> >
> > That would be great, for where we know, we list when a vulnerability was
> > added to the tree, and where it was fixed. That can leave many branches
> > still vulnerable where we have not fixed the issue yet. One example
> > would be CVE-2024-26629.
> >
> > You can see these in our repo by just doing:
> > git grep "5\.10" | grep introduced | grep -v fixed
>
> I didn't think of checking the mails, that's certainly easier to grep
> than json as it's line-oriented.
> It's going to take a bit more of processing to check not just bugs that
> were backported in the stable trees, but things introduced in earlier
> kernels... Someting like this?
>
> rg -l 'Issue introduced in ([234]\.[0-9]* |5\.[0-9] |5\.10\.[0-9]* )' | sort > introduced_before_5.10
> xargs rg -l 'fixed in 5\.10' < introduced_before_5.10 | sort > fixed_in_5.10
> comm -3 introduced_before_5.10 fixed_in_5.10 |tail
> cve/published/2024/CVE-2024-35844.mbox
> cve/published/2024/CVE-2024-35904.mbox
> cve/published/2024/CVE-2024-35951.mbox
> cve/published/2024/CVE-2024-35971.mbox
> cve/published/2024/CVE-2024-36009.mbox
> cve/published/2024/CVE-2024-36013.mbox
> grep 'Issue introdu' cve/published/2024/CVE-2024-35971.mbox
> Issue introduced in 5.8 with commit 797047f875b5 and fixed in 6.1.87 with commit 492337a4fbd1
> Issue introduced in 5.8 with commit 797047f875b5 and fixed in 6.6.28 with commit cba376eb036c
> Issue introduced in 5.8 with commit 797047f875b5 and fixed in 6.8.7 with commit 49d5d70538b6
> Issue introduced in 5.8 with commit 797047f875b5 and fixed in 6.9 with commit be0384bf599c
>
>
> The regex is a bit too manual to make a generic search script, and that
> feels very kludgy (at least mbox files do look like they get updated
> together with json), but that can be enough for my local needs for now.
The mbox files do get updated along with the json, but please, let's not
parse mbox files, that was a bad example I gave here, sorry.
> I was thinking something more along the line of parsing all the json
> files for containers.cna.affected by release version item (versionType
> != git);
That might be good, but really, we already have the needed information
here with the tool that creates all of this 'dyad', in the scripts/
directory. The output of that should be _much_ easier to parse:
$ ./scripts/dyad be0384bf599c
# ./scripts/dyad version: efdbc505ff2f
# getting vulnerable:fixed pairs for git id be0384bf599cf1eb8d337517feeb732d71f75a6f
5.8:797047f875b5463719cc70ba213eb691d453c946:6.1.87:492337a4fbd1421b42df684ee9b34be2a2722540
5.8:797047f875b5463719cc70ba213eb691d453c946:6.6.28:cba376eb036c2c20077b41d47b317d8218fe754f
5.8:797047f875b5463719cc70ba213eb691d453c946:6.8.7:49d5d70538b6b8f2a3f8f1ac30c1f921d4a0929b
5.8:797047f875b5463719cc70ba213eb691d453c946:6.9:be0384bf599cf1eb8d337517feeb732d71f75a6f
That tool generates a list of "vulnerable:fixed" pairs of version and
git ids. I have thought about checking in that output into the git repo
as odds are that would be easier than forcing you to regenerate it all
the time.
Here you see that the 5.10.y branch does not have a fix yet, and might
be easier than parsing the json files (which also show this), unless you
have a good json parser (i.e. something other than just bash.)
This output also catches where we introduce, and then fix, the issue in
the same release. dyad will show this, but as the issue never was in a
public release, CVE will not let us list it as that isn't relevent
there. But it IS relevent for those that might cherry-pick random
commits.
> It should be possible for a given stable tag to check if a given CVE
> applies or not immediately so it would be a matter of making this a bit
> more searchable -- probably make a reverse index with all the edges for
> faster search and keep appending new CVEs as they pop up.
Sure, that would be great. Usually when a commit is NOT in an older
branch, that means it either did not apply, or it might be queued up for
the next release. Or sometimes, the stable developers just didn't know
it needed to be backported at the time. We probably should just sweep
the current ids and catch that last issue now to make it easier for
others going forward. I'll add that to my long todo list...
> But it's a bit more work, so I'll gratefully take the grep mailboxes
> version for now :)
Agreed, or again, look at how dyad works, that might be simpler too, at
the expense that you have to compute it all the time, and it isn't the
quickest tool around (it's in bash and has not been optimized at all,
sorry.)
> Ideally we'll want to limit duplicating this work for other
> downstreams... So:
> - get more people to look at these
> - if unaffected (e.g. CVE-2024-35867 you singled out above as not
> affecting 5.10), report it so the reference files can be updated
Yes, if we "know" when a specific fix actually is introduced, we can add
that to our tools to properly catch and mark the entries. That
information would be great to have.
> - if affected backport patch, so it can be fixed and the refernece
> file can also be updated.
> - less work for everyone else!
>
> But finding volunteers for that kind of work might not be quite as easy
> as I make it sound like :)
Too true, there are very few of us actually working on this type of
thing, despite all of the people actually relying on it :(
Any help is greatly appreciated.
I will call out, we have had help from many developers from SuSE in
reviewing the current CVE entries and helping find duplicates and issues
we shouldn't have marked as such. And we have help from other
developers at other companies in doing reviews and seeing if commits
should, or should not, get CVE entries. All of that help is greatly
appreciated and we can always use more.
thanks,
greg k-h