Greetings,
Kernel bound workloads seem to trigger the below for whatever reason.
?I only see this when beating up NFS. ?There was a kworker wakeup
latency issue, but with a bandaid applied to fix that up, I can still
trigger this.
[ 1313.811031] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
[ 1313.811035] swiotlb: coherent allocation failed for device 0000:01:00.0 size=2097152
[ 1313.811038] CPU: 6 PID: 3026 Comm: Xorg Tainted: G E 4.15.0.g1291a0d5-master #355
[ 1313.811040] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[ 1313.811041] Call Trace:
[ 1313.811049] dump_stack+0x7c/0xb6
[ 1313.811053] swiotlb_alloc_coherent+0x13f/0x150
[ 1313.811060] ttm_dma_pool_alloc_new_pages+0x106/0x3c0 [ttm]
[ 1313.811066] ttm_dma_pool_get_pages+0x10a/0x1e0 [ttm]
[ 1313.811070] ttm_dma_populate+0x21f/0x2f0 [ttm]
[ 1313.811075] ttm_tt_bind+0x2f/0x60 [ttm]
[ 1313.811079] ttm_bo_handle_move_mem+0x51f/0x580 [ttm]
[ 1313.811084] ? ttm_bo_handle_move_mem+0x5/0x580 [ttm]
[ 1313.811088] ttm_bo_validate+0x10c/0x120 [ttm]
[ 1313.811092] ? ttm_bo_validate+0x5/0x120 [ttm]
[ 1313.811106] ? drm_mode_setcrtc+0x20e/0x540 [drm]
[ 1313.811109] ttm_bo_init_reserved+0x290/0x490 [ttm]
[ 1313.811114] ttm_bo_init+0x52/0xb0 [ttm]
[ 1313.811141] ? nv10_bo_put_tile_region+0x60/0x60 [nouveau]
[ 1313.811163] nouveau_bo_new+0x465/0x5e0 [nouveau]
[ 1313.811184] ? nv10_bo_put_tile_region+0x60/0x60 [nouveau]
[ 1313.811203] nouveau_gem_new+0x66/0x110 [nouveau]
[ 1313.811223] ? nouveau_gem_new+0x110/0x110 [nouveau]
[ 1313.811241] nouveau_gem_ioctl_new+0x48/0xc0 [nouveau]
[ 1313.811249] drm_ioctl_kernel+0x64/0xb0 [drm]
[ 1313.811257] drm_ioctl+0x2a4/0x360 [drm]
[ 1313.811276] ? nouveau_gem_new+0x110/0x110 [nouveau]
[ 1313.811285] ? drm_ioctl+0x5/0x360 [drm]
[ 1313.811304] nouveau_drm_ioctl+0x50/0xb0 [nouveau]
[ 1313.811308] do_vfs_ioctl+0x90/0x690
[ 1313.811311] ? do_vfs_ioctl+0x5/0x690
[ 1313.811313] SyS_ioctl+0x3b/0x70
[ 1313.811316] entry_SYSCALL_64_fastpath+0x1f/0x91
[ 1313.811320] RIP: 0033:0x7f3234746227
[ 1313.811321] RSP: 002b:00007ffc3ace0408 EFLAGS: 00003246 ORIG_RAX: 0000000000000010
[ 1313.811324] RAX: ffffffffffffffda RBX: 00000000025515d0 RCX: 00007f3234746227
[ 1313.811325] RDX: 00007ffc3ace0460 RSI: 00000000c0306480 RDI: 000000000000000b
[ 1313.811326] RBP: 0000000000824120 R08: 0000000002548f80 R09: 00000000025490d0
[ 1313.811328] R10: 0000000000000000 R11: 0000000000003246 R12: 000000000000093d
[ 1313.811329] R13: 0000000002aff74c R14: 0000000000824150 R15: 0000000000000000
On 12/18/17 7:06 PM, Mike Galbraith wrote:
> Greetings,
>
> Kernel bound workloads seem to trigger the below for whatever reason.
> ?I only see this when beating up NFS. ?There was a kworker wakeup
> latency issue, but with a bandaid applied to fix that up, I can still
> trigger this.
Hi,
i have seen this one as well with my system, but i could not find an
easy way to trigger it for bisecting purpose. If you can trigger it
conveniently, a bisect would be nice!
Greetings,
Tobias
>
> [ 1313.811031] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> [ 1313.811035] swiotlb: coherent allocation failed for device 0000:01:00.0 size=2097152
> [ 1313.811038] CPU: 6 PID: 3026 Comm: Xorg Tainted: G E 4.15.0.g1291a0d5-master #355
> [ 1313.811040] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
> [ 1313.811041] Call Trace:
> [ 1313.811049] dump_stack+0x7c/0xb6
> [ 1313.811053] swiotlb_alloc_coherent+0x13f/0x150
> [ 1313.811060] ttm_dma_pool_alloc_new_pages+0x106/0x3c0 [ttm]
> [ 1313.811066] ttm_dma_pool_get_pages+0x10a/0x1e0 [ttm]
> [ 1313.811070] ttm_dma_populate+0x21f/0x2f0 [ttm]
> [ 1313.811075] ttm_tt_bind+0x2f/0x60 [ttm]
> [ 1313.811079] ttm_bo_handle_move_mem+0x51f/0x580 [ttm]
> [ 1313.811084] ? ttm_bo_handle_move_mem+0x5/0x580 [ttm]
> [ 1313.811088] ttm_bo_validate+0x10c/0x120 [ttm]
> [ 1313.811092] ? ttm_bo_validate+0x5/0x120 [ttm]
> [ 1313.811106] ? drm_mode_setcrtc+0x20e/0x540 [drm]
> [ 1313.811109] ttm_bo_init_reserved+0x290/0x490 [ttm]
> [ 1313.811114] ttm_bo_init+0x52/0xb0 [ttm]
> [ 1313.811141] ? nv10_bo_put_tile_region+0x60/0x60 [nouveau]
> [ 1313.811163] nouveau_bo_new+0x465/0x5e0 [nouveau]
> [ 1313.811184] ? nv10_bo_put_tile_region+0x60/0x60 [nouveau]
> [ 1313.811203] nouveau_gem_new+0x66/0x110 [nouveau]
> [ 1313.811223] ? nouveau_gem_new+0x110/0x110 [nouveau]
> [ 1313.811241] nouveau_gem_ioctl_new+0x48/0xc0 [nouveau]
> [ 1313.811249] drm_ioctl_kernel+0x64/0xb0 [drm]
> [ 1313.811257] drm_ioctl+0x2a4/0x360 [drm]
> [ 1313.811276] ? nouveau_gem_new+0x110/0x110 [nouveau]
> [ 1313.811285] ? drm_ioctl+0x5/0x360 [drm]
> [ 1313.811304] nouveau_drm_ioctl+0x50/0xb0 [nouveau]
> [ 1313.811308] do_vfs_ioctl+0x90/0x690
> [ 1313.811311] ? do_vfs_ioctl+0x5/0x690
> [ 1313.811313] SyS_ioctl+0x3b/0x70
> [ 1313.811316] entry_SYSCALL_64_fastpath+0x1f/0x91
> [ 1313.811320] RIP: 0033:0x7f3234746227
> [ 1313.811321] RSP: 002b:00007ffc3ace0408 EFLAGS: 00003246 ORIG_RAX: 0000000000000010
> [ 1313.811324] RAX: ffffffffffffffda RBX: 00000000025515d0 RCX: 00007f3234746227
> [ 1313.811325] RDX: 00007ffc3ace0460 RSI: 00000000c0306480 RDI: 000000000000000b
> [ 1313.811326] RBP: 0000000000824120 R08: 0000000002548f80 R09: 00000000025490d0
> [ 1313.811328] R10: 0000000000000000 R11: 0000000000003246 R12: 000000000000093d
> [ 1313.811329] R13: 0000000002aff74c R14: 0000000000824150 R15: 0000000000000000
On Mon, 2017-12-18 at 20:01 +0100, Tobias Klausmann wrote:
> On 12/18/17 7:06 PM, Mike Galbraith wrote:
> > Greetings,
> >
> > Kernel bound workloads seem to trigger the below for whatever reason.
> > ?I only see this when beating up NFS. ?There was a kworker wakeup
> > latency issue, but with a bandaid applied to fix that up, I can still
> > trigger this.
>
>
> Hi,
>
> i have seen this one as well with my system, but i could not find an
> easy way to trigger it for bisecting purpose. If you can trigger it
> conveniently, a bisect would be nice!
Workload permitting. ?To reproduce, mount your box NFS, cd to somewhere
the NFS mount, and just do bonnie -s <memory size>. ?There, maybe
you'll beat me to it. ?I hope so, I have multiple kernels doing the
annoying "baby birds in a nest" thing at me literally endlessly :)
-Mike
On 2017-12-18 08:01 PM, Tobias Klausmann wrote:
> On 12/18/17 7:06 PM, Mike Galbraith wrote:
>> Greetings,
>>
>> Kernel bound workloads seem to trigger the below for whatever reason.
>> I only see this when beating up NFS. There was a kworker wakeup
>> latency issue, but with a bandaid applied to fix that up, I can still
>> trigger this.
>
>
> Hi,
>
> i have seen this one as well with my system, but i could not find an
> easy way to trigger it for bisecting purpose. If you can trigger it
> conveniently, a bisect would be nice!
I'm seeing this (with the amdgpu and radeon drivers) when restic takes a
backup, creating memory pressure. I happen to have just finished
bisecting, the result is:
648bc3574716400acc06f99915815f80d9563783 is the first bad commit
commit 648bc3574716400acc06f99915815f80d9563783
Author: Christian König <[email protected]>
Date: Thu Jul 6 09:59:43 2017 +0200
drm/ttm: add transparent huge page support for DMA allocations v2
Try to allocate huge pages when it makes sense.
v2: fix comment and use ifdef
--
Earthling Michel Dänzer | http://www.amd.com
Libre software enthusiast | Mesa and X developer
On 2017-12-19 11:37 AM, Michel Dänzer wrote:
> On 2017-12-18 08:01 PM, Tobias Klausmann wrote:
>> On 12/18/17 7:06 PM, Mike Galbraith wrote:
>>> Greetings,
>>>
>>> Kernel bound workloads seem to trigger the below for whatever reason.
>>> I only see this when beating up NFS. There was a kworker wakeup
>>> latency issue, but with a bandaid applied to fix that up, I can still
>>> trigger this.
>>
>>
>> Hi,
>>
>> i have seen this one as well with my system, but i could not find an
>> easy way to trigger it for bisecting purpose. If you can trigger it
>> conveniently, a bisect would be nice!
>
> I'm seeing this (with the amdgpu and radeon drivers) when restic takes a
> backup, creating memory pressure. I happen to have just finished
> bisecting, the result is:
>
> 648bc3574716400acc06f99915815f80d9563783 is the first bad commit
> commit 648bc3574716400acc06f99915815f80d9563783
> Author: Christian König <[email protected]>
> Date: Thu Jul 6 09:59:43 2017 +0200
>
> drm/ttm: add transparent huge page support for DMA allocations v2
>
> Try to allocate huge pages when it makes sense.
>
> v2: fix comment and use ifdef
>
>
BTW, I haven't noticed any bad effects other than the dmesg splats, so
maybe it's just noise about transient failures for which there is a
proper fallback in place.
--
Earthling Michel Dänzer | http://www.amd.com
Libre software enthusiast | Mesa and X developer
Am 19.12.2017 um 11:39 schrieb Michel Dänzer:
> On 2017-12-19 11:37 AM, Michel Dänzer wrote:
>> On 2017-12-18 08:01 PM, Tobias Klausmann wrote:
>>> On 12/18/17 7:06 PM, Mike Galbraith wrote:
>>>> Greetings,
>>>>
>>>> Kernel bound workloads seem to trigger the below for whatever reason.
>>>> I only see this when beating up NFS. There was a kworker wakeup
>>>> latency issue, but with a bandaid applied to fix that up, I can still
>>>> trigger this.
>>>
>>> Hi,
>>>
>>> i have seen this one as well with my system, but i could not find an
>>> easy way to trigger it for bisecting purpose. If you can trigger it
>>> conveniently, a bisect would be nice!
>> I'm seeing this (with the amdgpu and radeon drivers) when restic takes a
>> backup, creating memory pressure. I happen to have just finished
>> bisecting, the result is:
>>
>> 648bc3574716400acc06f99915815f80d9563783 is the first bad commit
>> commit 648bc3574716400acc06f99915815f80d9563783
>> Author: Christian König <[email protected]>
>> Date: Thu Jul 6 09:59:43 2017 +0200
>>
>> drm/ttm: add transparent huge page support for DMA allocations v2
>>
>> Try to allocate huge pages when it makes sense.
>>
>> v2: fix comment and use ifdef
>>
>>
> BTW, I haven't noticed any bad effects other than the dmesg splats, so
> maybe it's just noise about transient failures for which there is a
> proper fallback in place.
Yeah, I think that is exactly what happens here.
We try to allocate a huge page, but fail and so fall back to using
multiple 4k pages instead.
Going to send out a patch to suppress the warning.
Thanks for bisecting this,
Christian.
On Tue, Dec 19, 2017 at 8:45 AM, Christian König
<[email protected]> wrote:
> Am 19.12.2017 um 11:39 schrieb Michel Dänzer:
>>
>> On 2017-12-19 11:37 AM, Michel Dänzer wrote:
>>>
>>> On 2017-12-18 08:01 PM, Tobias Klausmann wrote:
>>>>
>>>> On 12/18/17 7:06 PM, Mike Galbraith wrote:
>>>>>
>>>>> Greetings,
>>>>>
>>>>> Kernel bound workloads seem to trigger the below for whatever reason.
>>>>> I only see this when beating up NFS. There was a kworker wakeup
>>>>> latency issue, but with a bandaid applied to fix that up, I can still
>>>>> trigger this.
>>>>
>>>>
>>>> Hi,
>>>>
>>>> i have seen this one as well with my system, but i could not find an
>>>> easy way to trigger it for bisecting purpose. If you can trigger it
>>>> conveniently, a bisect would be nice!
>>>
>>> I'm seeing this (with the amdgpu and radeon drivers) when restic takes a
>>> backup, creating memory pressure. I happen to have just finished
>>> bisecting, the result is:
>>>
>>> 648bc3574716400acc06f99915815f80d9563783 is the first bad commit
>>> commit 648bc3574716400acc06f99915815f80d9563783
>>> Author: Christian König <[email protected]>
>>> Date: Thu Jul 6 09:59:43 2017 +0200
>>>
>>> drm/ttm: add transparent huge page support for DMA allocations v2
>>>
>>> Try to allocate huge pages when it makes sense.
>>>
>>> v2: fix comment and use ifdef
>>>
>>>
>> BTW, I haven't noticed any bad effects other than the dmesg splats, so
>> maybe it's just noise about transient failures for which there is a
>> proper fallback in place.
>
>
> Yeah, I think that is exactly what happens here.
>
> We try to allocate a huge page, but fail and so fall back to using multiple
> 4k pages instead.
>
> Going to send out a patch to suppress the warning.
Hi Christian,
Did you ever send out such a patch? I didn't see one on the list, but
perhaps I missed it. One definitely hasn't made it upstream yet. (I
just hit the issue myself with Linus's tree from last night.)
Thanks,
-ilia
On Sun, 2017-12-31 at 13:27 -0500, Ilia Mirkin wrote:
> On Tue, Dec 19, 2017 at 8:45 AM, Christian K?nig
> <[email protected]> wrote:
> > Am 19.12.2017 um 11:39 schrieb Michel D?nzer:
> >>
> >> On 2017-12-19 11:37 AM, Michel D?nzer wrote:
> >>>
> >>> On 2017-12-18 08:01 PM, Tobias Klausmann wrote:
> >>>>
> >>>> On 12/18/17 7:06 PM, Mike Galbraith wrote:
> >>>>>
> >>>>> Greetings,
> >>>>>
> >>>>> Kernel bound workloads seem to trigger the below for whatever reason.
> >>>>> I only see this when beating up NFS. There was a kworker wakeup
> >>>>> latency issue, but with a bandaid applied to fix that up, I can still
> >>>>> trigger this.
> >>>>
> >>>>
> >>>> Hi,
> >>>>
> >>>> i have seen this one as well with my system, but i could not find an
> >>>> easy way to trigger it for bisecting purpose. If you can trigger it
> >>>> conveniently, a bisect would be nice!
> >>>
> >>> I'm seeing this (with the amdgpu and radeon drivers) when restic takes a
> >>> backup, creating memory pressure. I happen to have just finished
> >>> bisecting, the result is:
> >>>
> >>> 648bc3574716400acc06f99915815f80d9563783 is the first bad commit
> >>> commit 648bc3574716400acc06f99915815f80d9563783
> >>> Author: Christian K?nig <[email protected]>
> >>> Date: Thu Jul 6 09:59:43 2017 +0200
> >>>
> >>> drm/ttm: add transparent huge page support for DMA allocations v2
> >>>
> >>> Try to allocate huge pages when it makes sense.
> >>>
> >>> v2: fix comment and use ifdef
> >>>
> >>>
> >> BTW, I haven't noticed any bad effects other than the dmesg splats, so
> >> maybe it's just noise about transient failures for which there is a
> >> proper fallback in place.
> >
> >
> > Yeah, I think that is exactly what happens here.
> >
> > We try to allocate a huge page, but fail and so fall back to using multiple
> > 4k pages instead.
> >
> > Going to send out a patch to suppress the warning.
>
> Hi Christian,
>
> Did you ever send out such a patch? I didn't see one on the list, but
> perhaps I missed it. One definitely hasn't made it upstream yet. (I
> just hit the issue myself with Linus's tree from last night.)
Actually, that wants a bit more methinks, because while the stack dump
goes away, you still get spammed, it just comes in smaller chunks.
-Mike
On Sun, Dec 31, 2017 at 3:53 PM, Mike Galbraith <[email protected]> wrote:
> On Sun, 2017-12-31 at 13:27 -0500, Ilia Mirkin wrote:
>> On Tue, Dec 19, 2017 at 8:45 AM, Christian König
>> <[email protected]> wrote:
>> > Am 19.12.2017 um 11:39 schrieb Michel Dänzer:
>> >>
>> >> On 2017-12-19 11:37 AM, Michel Dänzer wrote:
>> >>>
>> >>> On 2017-12-18 08:01 PM, Tobias Klausmann wrote:
>> >>>>
>> >>>> On 12/18/17 7:06 PM, Mike Galbraith wrote:
>> >>>>>
>> >>>>> Greetings,
>> >>>>>
>> >>>>> Kernel bound workloads seem to trigger the below for whatever reason.
>> >>>>> I only see this when beating up NFS. There was a kworker wakeup
>> >>>>> latency issue, but with a bandaid applied to fix that up, I can still
>> >>>>> trigger this.
>> >>>>
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>> i have seen this one as well with my system, but i could not find an
>> >>>> easy way to trigger it for bisecting purpose. If you can trigger it
>> >>>> conveniently, a bisect would be nice!
>> >>>
>> >>> I'm seeing this (with the amdgpu and radeon drivers) when restic takes a
>> >>> backup, creating memory pressure. I happen to have just finished
>> >>> bisecting, the result is:
>> >>>
>> >>> 648bc3574716400acc06f99915815f80d9563783 is the first bad commit
>> >>> commit 648bc3574716400acc06f99915815f80d9563783
>> >>> Author: Christian König <[email protected]>
>> >>> Date: Thu Jul 6 09:59:43 2017 +0200
>> >>>
>> >>> drm/ttm: add transparent huge page support for DMA allocations v2
>> >>>
>> >>> Try to allocate huge pages when it makes sense.
>> >>>
>> >>> v2: fix comment and use ifdef
>> >>>
>> >>>
>> >> BTW, I haven't noticed any bad effects other than the dmesg splats, so
>> >> maybe it's just noise about transient failures for which there is a
>> >> proper fallback in place.
>> >
>> >
>> > Yeah, I think that is exactly what happens here.
>> >
>> > We try to allocate a huge page, but fail and so fall back to using multiple
>> > 4k pages instead.
>> >
>> > Going to send out a patch to suppress the warning.
>>
>> Hi Christian,
>>
>> Did you ever send out such a patch? I didn't see one on the list, but
>> perhaps I missed it. One definitely hasn't made it upstream yet. (I
>> just hit the issue myself with Linus's tree from last night.)
>
> Actually, that wants a bit more methinks, because while the stack dump
> goes away, you still get spammed, it just comes in smaller chunks.
OK, well this has to either be fixed or reverted. Right now it's
complaining all the time for me after like a day of uptime.
-ilia
Am 01.01.2018 um 19:08 schrieb Ilia Mirkin:
> On Sun, Dec 31, 2017 at 3:53 PM, Mike Galbraith <[email protected]> wrote:
>> On Sun, 2017-12-31 at 13:27 -0500, Ilia Mirkin wrote:
>>> On Tue, Dec 19, 2017 at 8:45 AM, Christian König
>>> <[email protected]> wrote:
>>>> Am 19.12.2017 um 11:39 schrieb Michel Dänzer:
>>>>> On 2017-12-19 11:37 AM, Michel Dänzer wrote:
>>>>>> On 2017-12-18 08:01 PM, Tobias Klausmann wrote:
>>>>>>> On 12/18/17 7:06 PM, Mike Galbraith wrote:
>>>>>>>> Greetings,
>>>>>>>>
>>>>>>>> Kernel bound workloads seem to trigger the below for whatever reason.
>>>>>>>> I only see this when beating up NFS. There was a kworker wakeup
>>>>>>>> latency issue, but with a bandaid applied to fix that up, I can still
>>>>>>>> trigger this.
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> i have seen this one as well with my system, but i could not find an
>>>>>>> easy way to trigger it for bisecting purpose. If you can trigger it
>>>>>>> conveniently, a bisect would be nice!
>>>>>> I'm seeing this (with the amdgpu and radeon drivers) when restic takes a
>>>>>> backup, creating memory pressure. I happen to have just finished
>>>>>> bisecting, the result is:
>>>>>>
>>>>>> 648bc3574716400acc06f99915815f80d9563783 is the first bad commit
>>>>>> commit 648bc3574716400acc06f99915815f80d9563783
>>>>>> Author: Christian König <[email protected]>
>>>>>> Date: Thu Jul 6 09:59:43 2017 +0200
>>>>>>
>>>>>> drm/ttm: add transparent huge page support for DMA allocations v2
>>>>>>
>>>>>> Try to allocate huge pages when it makes sense.
>>>>>>
>>>>>> v2: fix comment and use ifdef
>>>>>>
>>>>>>
>>>>> BTW, I haven't noticed any bad effects other than the dmesg splats, so
>>>>> maybe it's just noise about transient failures for which there is a
>>>>> proper fallback in place.
>>>>
>>>> Yeah, I think that is exactly what happens here.
>>>>
>>>> We try to allocate a huge page, but fail and so fall back to using multiple
>>>> 4k pages instead.
>>>>
>>>> Going to send out a patch to suppress the warning.
>>> Hi Christian,
>>>
>>> Did you ever send out such a patch? I didn't see one on the list, but
>>> perhaps I missed it. One definitely hasn't made it upstream yet. (I
>>> just hit the issue myself with Linus's tree from last night.)
>> Actually, that wants a bit more methinks, because while the stack dump
>> goes away, you still get spammed, it just comes in smaller chunks.
> OK, well this has to either be fixed or reverted. Right now it's
> complaining all the time for me after like a day of uptime.
I've already send out a patch to Konrad Rzeszutek Wilk and he wanted to
queue that up.
But there is another warning I'm currently working on, just didn't had
time to during the holidays.
Regards,
Christian.