alloc_percpu() may return NULL, which means chan->buf may be set to
NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we
dereference an invalid pointer:
BUG: Unable to handle kernel data access at 0x7dae0000
Faulting instruction address: 0xc0000000003f3fec
...
NIP [c0000000003f3fec] relay_open+0x29c/0x600
LR [c0000000003f3fc0] relay_open+0x270/0x600
Call Trace:
[c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
[c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
[c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
[c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
[c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
[c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
[c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
[c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68
Check if alloc_percpu returns NULL. Because we can readily catch and
handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.
This was found by syzkaller both on x86 and powerpc, and the reproducer
it found on powerpc is capable of hitting the issue as an unprivileged
user.
Fixes: 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
Reported-by: [email protected]
Reported-by: [email protected]
Reported-by: [email protected]
Reported-by: [email protected]
Cc: Akash Goel <[email protected]>
Cc: Andrew Donnellan <[email protected]> # syzkaller-ppc64
Cc: [email protected] # v4.10+
Signed-off-by: Daniel Axtens <[email protected]>
--
There's a syz reproducer on the powerpc syzbot that eventually hits
the bug, but it can take up to an hour or so before it keels over on a
kernel with all the syzkaller debugging on, and even longer on a
production kernel. I have been able to reproduce it once on a stock
Ubuntu 5.0 ppc64le kernel.
I will ask MITRE for a CVE - while only the process doing the syscall
gets killed, it gets killed while holding the relay_channels_mutex,
so it blocks all future relay activity.
---
kernel/relay.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/kernel/relay.c b/kernel/relay.c
index ade14fb7ce2e..a376cc6b54ec 100644
--- a/kernel/relay.c
+++ b/kernel/relay.c
@@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
if (!chan)
return NULL;
- chan->buf = alloc_percpu(struct rchan_buf *);
+ chan->buf = alloc_percpu_gfp(struct rchan_buf *,
+ GFP_KERNEL | __GFP_NOWARN);
+ if (!chan->buf) {
+ kfree(chan);
+ return NULL;
+ }
+
chan->version = RELAYFS_CHANNEL_VERSION;
chan->n_subbufs = n_subbufs;
chan->subbuf_size = subbuf_size;
--
2.20.1
Daniel Axtens <[email protected]> writes:
> alloc_percpu() may return NULL, which means chan->buf may be set to
> NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we
> dereference an invalid pointer:
>
> BUG: Unable to handle kernel data access at 0x7dae0000
> Faulting instruction address: 0xc0000000003f3fec
> ...
> NIP [c0000000003f3fec] relay_open+0x29c/0x600
> LR [c0000000003f3fc0] relay_open+0x270/0x600
> Call Trace:
> [c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
> [c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
> [c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
> [c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
> [c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
> [c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
> [c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
> [c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68
>
> Check if alloc_percpu returns NULL. Because we can readily catch and
> handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.
>
> This was found by syzkaller both on x86 and powerpc, and the reproducer
> it found on powerpc is capable of hitting the issue as an unprivileged
> user.
>
> Fixes: 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
> Reported-by: [email protected]
> Reported-by: [email protected]
> Reported-by: [email protected]
> Reported-by: [email protected]
> Cc: Akash Goel <[email protected]>
> Cc: Andrew Donnellan <[email protected]> # syzkaller-ppc64
> Cc: [email protected] # v4.10+
> Signed-off-by: Daniel Axtens <[email protected]>
...
> diff --git a/kernel/relay.c b/kernel/relay.c
> index ade14fb7ce2e..a376cc6b54ec 100644
> --- a/kernel/relay.c
> +++ b/kernel/relay.c
> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
> if (!chan)
> return NULL;
>
> - chan->buf = alloc_percpu(struct rchan_buf *);
> + chan->buf = alloc_percpu_gfp(struct rchan_buf *,
> + GFP_KERNEL | __GFP_NOWARN);
> + if (!chan->buf) {
> + kfree(chan);
> + return NULL;
> + }
> +
> chan->version = RELAYFS_CHANNEL_VERSION;
> chan->n_subbufs = n_subbufs;
> chan->subbuf_size = subbuf_size;
This looks right to me. The kfree + direct return is correct, there's
nothing else that needs tear down in this function.
I think I'm 50/50 on the __GFP_NOWARN. We're only asking for 8 bytes per
cpu, and if that fails the system is pretty sick, so a warning could be
helpful. There's also logic in the percpu allocator to limit the number
of warnings printed. But see what others think.
Reviewed-by: Michael Ellerman <[email protected]>
cheers
On 29/11/19 3:59 pm, Michael Ellerman wrote:
>> diff --git a/kernel/relay.c b/kernel/relay.c
>> index ade14fb7ce2e..a376cc6b54ec 100644
>> --- a/kernel/relay.c
>> +++ b/kernel/relay.c
>> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
>> if (!chan)
>> return NULL;
>>
>> - chan->buf = alloc_percpu(struct rchan_buf *);
>> + chan->buf = alloc_percpu_gfp(struct rchan_buf *,
>> + GFP_KERNEL | __GFP_NOWARN);
>> + if (!chan->buf) {
>> + kfree(chan);
>> + return NULL;
>> + }
>> +
>> chan->version = RELAYFS_CHANNEL_VERSION;
>> chan->n_subbufs = n_subbufs;
>> chan->subbuf_size = subbuf_size;
>
> This looks right to me. The kfree + direct return is correct, there's
> nothing else that needs tear down in this function.
>
> I think I'm 50/50 on the __GFP_NOWARN. We're only asking for 8 bytes per
> cpu, and if that fails the system is pretty sick, so a warning could be
> helpful. There's also logic in the percpu allocator to limit the number
> of warnings printed. But see what others think.
mpe was wondering why we didn't see a message printed from the percpu
allocator - the answer appears to be that we hit this case when the
process is killed while the percpu allocator is waiting for
pcpu_alloc_mutex, in which case it bails out without printing a warning.
It looks to me like that case doesn't warrant a warning message, while a
failing allocation for other reasons should probably get a warning.
But whatever, otherwise this patch looks good to me. I've told our
powerpc syzbot to test it.
Reviewed-by: Andrew Donnellan <[email protected]>
--
Andrew Donnellan OzLabs, ADL Canberra
[email protected] IBM Australia Limited
Daniel Axtens <[email protected]> writes:
> --
>
> There's a syz reproducer on the powerpc syzbot that eventually hits
> the bug, but it can take up to an hour or so before it keels over on a
> kernel with all the syzkaller debugging on, and even longer on a
> production kernel. I have been able to reproduce it once on a stock
> Ubuntu 5.0 ppc64le kernel.
>
> I will ask MITRE for a CVE - while only the process doing the syscall
> gets killed, it gets killed while holding the relay_channels_mutex,
> so it blocks all future relay activity.
CVE-2019-19462 has been assigned.
Regards,
Daniel
> ---
> kernel/relay.c | 8 +++++++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/relay.c b/kernel/relay.c
> index ade14fb7ce2e..a376cc6b54ec 100644
> --- a/kernel/relay.c
> +++ b/kernel/relay.c
> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
> if (!chan)
> return NULL;
>
> - chan->buf = alloc_percpu(struct rchan_buf *);
> + chan->buf = alloc_percpu_gfp(struct rchan_buf *,
> + GFP_KERNEL | __GFP_NOWARN);
> + if (!chan->buf) {
> + kfree(chan);
> + return NULL;
> + }
> +
> chan->version = RELAYFS_CHANNEL_VERSION;
> chan->n_subbufs = n_subbufs;
> chan->subbuf_size = subbuf_size;
> --
> 2.20.1
On Fri, Nov 29, 2019 at 12:37:45PM +1100, Daniel Axtens wrote:
> alloc_percpu() may return NULL, which means chan->buf may be set to
> NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we
> dereference an invalid pointer:
>
> BUG: Unable to handle kernel data access at 0x7dae0000
> Faulting instruction address: 0xc0000000003f3fec
> ...
> NIP [c0000000003f3fec] relay_open+0x29c/0x600
> LR [c0000000003f3fc0] relay_open+0x270/0x600
> Call Trace:
> [c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
> [c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
> [c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
> [c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
> [c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
> [c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
> [c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
> [c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68
>
> Check if alloc_percpu returns NULL. Because we can readily catch and
> handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.
>
> This was found by syzkaller both on x86 and powerpc, and the reproducer
> it found on powerpc is capable of hitting the issue as an unprivileged
> user.
>
> Fixes: 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
> Reported-by: [email protected]
> Reported-by: [email protected]
> Reported-by: [email protected]
> Reported-by: [email protected]
> Cc: Akash Goel <[email protected]>
> Cc: Andrew Donnellan <[email protected]> # syzkaller-ppc64
> Cc: [email protected] # v4.10+
> Signed-off-by: Daniel Axtens <[email protected]>
>
So there is a CVE now, but it appears that the patch went nowhere.
Are there any plans to actually apply it ?
Thanks,
Guenter
> --
>
> There's a syz reproducer on the powerpc syzbot that eventually hits
> the bug, but it can take up to an hour or so before it keels over on a
> kernel with all the syzkaller debugging on, and even longer on a
> production kernel. I have been able to reproduce it once on a stock
> Ubuntu 5.0 ppc64le kernel.
>
> I will ask MITRE for a CVE - while only the process doing the syscall
> gets killed, it gets killed while holding the relay_channels_mutex,
> so it blocks all future relay activity.
> ---
> kernel/relay.c | 8 +++++++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/relay.c b/kernel/relay.c
> index ade14fb7ce2e..a376cc6b54ec 100644
> --- a/kernel/relay.c
> +++ b/kernel/relay.c
> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
> if (!chan)
> return NULL;
>
> - chan->buf = alloc_percpu(struct rchan_buf *);
> + chan->buf = alloc_percpu_gfp(struct rchan_buf *,
> + GFP_KERNEL | __GFP_NOWARN);
> + if (!chan->buf) {
> + kfree(chan);
> + return NULL;
> + }
> +
> chan->version = RELAYFS_CHANNEL_VERSION;
> chan->n_subbufs = n_subbufs;
> chan->subbuf_size = subbuf_size;
> --
> 2.20.1
>
Guenter Roeck <[email protected]> writes:
> On Fri, Nov 29, 2019 at 12:37:45PM +1100, Daniel Axtens wrote:
>> alloc_percpu() may return NULL, which means chan->buf may be set to
>> NULL. In that case, when we do *per_cpu_ptr(chan->buf, ...), we
>> dereference an invalid pointer:
>>
>> BUG: Unable to handle kernel data access at 0x7dae0000
>> Faulting instruction address: 0xc0000000003f3fec
>> ...
>> NIP [c0000000003f3fec] relay_open+0x29c/0x600
>> LR [c0000000003f3fc0] relay_open+0x270/0x600
>> Call Trace:
>> [c000000054353a70] [c0000000003f3fb4] relay_open+0x264/0x600 (unreliable)
>> [c000000054353b00] [c000000000451764] __blk_trace_setup+0x254/0x600
>> [c000000054353bb0] [c000000000451b78] blk_trace_setup+0x68/0xa0
>> [c000000054353c10] [c0000000010da77c] sg_ioctl+0x7bc/0x2e80
>> [c000000054353cd0] [c000000000758cbc] do_vfs_ioctl+0x13c/0x1300
>> [c000000054353d90] [c000000000759f14] ksys_ioctl+0x94/0x130
>> [c000000054353de0] [c000000000759ff8] sys_ioctl+0x48/0xb0
>> [c000000054353e20] [c00000000000bcd0] system_call+0x5c/0x68
>>
>> Check if alloc_percpu returns NULL. Because we can readily catch and
>> handle this situation, switch to alloc_cpu_gfp and pass in __GFP_NOWARN.
>>
>> This was found by syzkaller both on x86 and powerpc, and the reproducer
>> it found on powerpc is capable of hitting the issue as an unprivileged
>> user.
>>
>> Fixes: 017c59c042d0 ("relay: Use per CPU constructs for the relay channel buffer pointers")
>> Reported-by: [email protected]
>> Reported-by: [email protected]
>> Reported-by: [email protected]
>> Reported-by: [email protected]
>> Cc: Akash Goel <[email protected]>
>> Cc: Andrew Donnellan <[email protected]> # syzkaller-ppc64
>> Cc: [email protected] # v4.10+
>> Signed-off-by: Daniel Axtens <[email protected]>
>>
>
> So there is a CVE now, but it appears that the patch went nowhere.
> Are there any plans to actually apply it ?
I sent a v2 that addresses some review comments, I guess if anything is
applied it will be that.
Daniel
>
> Thanks,
> Guenter
>
>> --
>>
>> There's a syz reproducer on the powerpc syzbot that eventually hits
>> the bug, but it can take up to an hour or so before it keels over on a
>> kernel with all the syzkaller debugging on, and even longer on a
>> production kernel. I have been able to reproduce it once on a stock
>> Ubuntu 5.0 ppc64le kernel.
>>
>> I will ask MITRE for a CVE - while only the process doing the syscall
>> gets killed, it gets killed while holding the relay_channels_mutex,
>> so it blocks all future relay activity.
>> ---
>> kernel/relay.c | 8 +++++++-
>> 1 file changed, 7 insertions(+), 1 deletion(-)
>>
>> diff --git a/kernel/relay.c b/kernel/relay.c
>> index ade14fb7ce2e..a376cc6b54ec 100644
>> --- a/kernel/relay.c
>> +++ b/kernel/relay.c
>> @@ -580,7 +580,13 @@ struct rchan *relay_open(const char *base_filename,
>> if (!chan)
>> return NULL;
>>
>> - chan->buf = alloc_percpu(struct rchan_buf *);
>> + chan->buf = alloc_percpu_gfp(struct rchan_buf *,
>> + GFP_KERNEL | __GFP_NOWARN);
>> + if (!chan->buf) {
>> + kfree(chan);
>> + return NULL;
>> + }
>> +
>> chan->version = RELAYFS_CHANNEL_VERSION;
>> chan->n_subbufs = n_subbufs;
>> chan->subbuf_size = subbuf_size;
>> --
>> 2.20.1
>>