From: Zhang Zhengming <[email protected]>
There is a crash in relay_file_read, as the var from
point to the end of last subbuf.
The oops looks something like:
pc : __arch_copy_to_user+0x180/0x310
lr : relay_file_read+0x20c/0x2c8
Call trace:
__arch_copy_to_user+0x180/0x310
full_proxy_read+0x68/0x98
vfs_read+0xb0/0x1d0
ksys_read+0x6c/0xf0
__arm64_sys_read+0x20/0x28
el0_svc_common.constprop.3+0x84/0x108
do_el0_svc+0x74/0x90
el0_svc+0x1c/0x28
el0_sync_handler+0x88/0xb0
el0_sync+0x148/0x180
We get the condition by analyzing the vmcore:
1). The last produced byte and last consumed byte
both at the end of the last subbuf
2). A softirq who will call function(e.g __blk_add_trace)
to write relay buffer occurs when an program calling
function relay_file_read_avail.
relay_file_read
relay_file_read_avail
relay_file_read_consume(buf, 0, 0);
//interrupted by softirq who will write subbuf
....
return 1;
//read_start point to the end of the last subbuf
read_start = relay_file_read_start_pos
//avail is equal to subsize
avail = relay_file_read_subbuf_avail
//from points to an invalid memory address
from = buf->start + read_start
//system is crashed
copy_to_user(buffer, from, avail)
Signed-off-by: Zhang Zhengming <[email protected]>
Reviewed-by: Zhao Lei <[email protected]>
Reviewed-by: Zhou Kete <[email protected]>
---
kernel/relay.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/kernel/relay.c b/kernel/relay.c
index 9aa70ae..a80fa01 100644
--- a/kernel/relay.c
+++ b/kernel/relay.c
@@ -989,7 +989,8 @@ static size_t relay_file_read_start_pos(struct rchan_buf *buf)
size_t subbuf_size = buf->chan->subbuf_size;
size_t n_subbufs = buf->chan->n_subbufs;
size_t consumed = buf->subbufs_consumed % n_subbufs;
- size_t read_pos = consumed * subbuf_size + buf->bytes_consumed;
+ size_t read_pos = (consumed * subbuf_size + buf->bytes_consumed)
+ % (n_subbufs * subbuf_size);
read_subbuf = read_pos / subbuf_size;
padding = buf->padding[read_subbuf];
--
2.17.1
On Wed, 19 Apr 2023 12:02:03 +0800 zhangzhengming <[email protected]> wrote:
> From: Zhang Zhengming <[email protected]>
>
> There is a crash in relay_file_read, as the var from
> point to the end of last subbuf.
> The oops looks something like:
> pc : __arch_copy_to_user+0x180/0x310
> lr : relay_file_read+0x20c/0x2c8
> Call trace:
> __arch_copy_to_user+0x180/0x310
> full_proxy_read+0x68/0x98
> vfs_read+0xb0/0x1d0
> ksys_read+0x6c/0xf0
> __arm64_sys_read+0x20/0x28
> el0_svc_common.constprop.3+0x84/0x108
> do_el0_svc+0x74/0x90
> el0_svc+0x1c/0x28
> el0_sync_handler+0x88/0xb0
> el0_sync+0x148/0x180
>
> We get the condition by analyzing the vmcore:
> 1). The last produced byte and last consumed byte
> both at the end of the last subbuf
> 2). A softirq who will call function(e.g __blk_add_trace)
> to write relay buffer occurs when an program calling
> function relay_file_read_avail.
> relay_file_read
> relay_file_read_avail
> relay_file_read_consume(buf, 0, 0);
> //interrupted by softirq who will write subbuf
> ....
> return 1;
> //read_start point to the end of the last subbuf
> read_start = relay_file_read_start_pos
> //avail is equal to subsize
> avail = relay_file_read_subbuf_avail
> //from points to an invalid memory address
> from = buf->start + read_start
> //system is crashed
> copy_to_user(buffer, from, avail)
Thanks. Hopefully Pengcheng Yang and Jens Axboe can comment.
> --- a/kernel/relay.c
> +++ b/kernel/relay.c
> @@ -989,7 +989,8 @@ static size_t relay_file_read_start_pos(struct rchan_buf *buf)
> size_t subbuf_size = buf->chan->subbuf_size;
> size_t n_subbufs = buf->chan->n_subbufs;
> size_t consumed = buf->subbufs_consumed % n_subbufs;
> - size_t read_pos = consumed * subbuf_size + buf->bytes_consumed;
> + size_t read_pos = (consumed * subbuf_size + buf->bytes_consumed)
> + % (n_subbufs * subbuf_size);
>
> read_subbuf = read_pos / subbuf_size;
> padding = buf->padding[read_subbuf];
I'm thinking we should backport this into earlier kernels and that the
commit we're fixing is
Fixes: 341a7213e5c1 ("kernel/relay.c: fix read_pos error when multiple readers")
On 4/19/23 3:03?PM, Andrew Morton wrote:
> On Wed, 19 Apr 2023 12:02:03 +0800 zhangzhengming <[email protected]> wrote:
>
>> From: Zhang Zhengming <[email protected]>
>>
>> There is a crash in relay_file_read, as the var from
>> point to the end of last subbuf.
>> The oops looks something like:
>> pc : __arch_copy_to_user+0x180/0x310
>> lr : relay_file_read+0x20c/0x2c8
>> Call trace:
>> __arch_copy_to_user+0x180/0x310
>> full_proxy_read+0x68/0x98
>> vfs_read+0xb0/0x1d0
>> ksys_read+0x6c/0xf0
>> __arm64_sys_read+0x20/0x28
>> el0_svc_common.constprop.3+0x84/0x108
>> do_el0_svc+0x74/0x90
>> el0_svc+0x1c/0x28
>> el0_sync_handler+0x88/0xb0
>> el0_sync+0x148/0x180
>>
>> We get the condition by analyzing the vmcore:
>> 1). The last produced byte and last consumed byte
>> both at the end of the last subbuf
>> 2). A softirq who will call function(e.g __blk_add_trace)
>> to write relay buffer occurs when an program calling
>> function relay_file_read_avail.
>> relay_file_read
>> relay_file_read_avail
>> relay_file_read_consume(buf, 0, 0);
>> //interrupted by softirq who will write subbuf
>> ....
>> return 1;
>> //read_start point to the end of the last subbuf
>> read_start = relay_file_read_start_pos
>> //avail is equal to subsize
>> avail = relay_file_read_subbuf_avail
>> //from points to an invalid memory address
>> from = buf->start + read_start
>> //system is crashed
>> copy_to_user(buffer, from, avail)
>
> Thanks. Hopefully Pengcheng Yang and Jens Axboe can comment.
Patch looks good to me, but that doesn't necessarily say much. I never
did much relayfs hacking, and the bits I did was probably almost 20
years ago at this point when I wrote blktrace...
--
Jens Axboe
On April 20, 2023 5:04 AM, Andrew Morton wrote:
> On Wed, 19 Apr 2023 12:02:03 +0800 zhangzhengming <[email protected]> wrote:
>
>> From: Zhang Zhengming <[email protected]>
>>
>> There is a crash in relay_file_read, as the var from
>> point to the end of last subbuf.
>> The oops looks something like:
>> pc : __arch_copy_to_user+0x180/0x310
>> lr : relay_file_read+0x20c/0x2c8
>> Call trace:
>> __arch_copy_to_user+0x180/0x310
>> full_proxy_read+0x68/0x98
>> vfs_read+0xb0/0x1d0
>> ksys_read+0x6c/0xf0
>> __arm64_sys_read+0x20/0x28
>> el0_svc_common.constprop.3+0x84/0x108
>> do_el0_svc+0x74/0x90
>> el0_svc+0x1c/0x28
>> el0_sync_handler+0x88/0xb0
>> el0_sync+0x148/0x180
>>
>> We get the condition by analyzing the vmcore:
>> 1). The last produced byte and last consumed byte
>> both at the end of the last subbuf
>> 2). A softirq who will call function(e.g __blk_add_trace)
>> to write relay buffer occurs when an program calling
>> function relay_file_read_avail.
>> relay_file_read
>> relay_file_read_avail
>> relay_file_read_consume(buf, 0, 0);
>> //interrupted by softirq who will write subbuf
>> ....
>> return 1;
>> //read_start point to the end of the last subbuf
>> read_start = relay_file_read_start_pos
>> //avail is equal to subsize
>> avail = relay_file_read_subbuf_avail
>> //from points to an invalid memory address
>> from = buf->start + read_start
>> //system is crashed
>> copy_to_user(buffer, from, avail)
>
> Thanks. Hopefully Pengcheng Yang and Jens Axboe can comment.
This patch looks good to me.
Reviewed-by: Pengcheng Yang <[email protected]>
>
> I'm thinking we should backport this into earlier kernels and that the
> commit we're fixing is
>
> Fixes: 341a7213e5c1 ("kernel/relay.c: fix read_pos error when multiple readers")
I suggest starting backport with this tag:
Fixes: 8d62fdebdaf9 ("relay file read: start-pos fix")