Dear RT Folks,
This is the RT stable review cycle of patch v4.4.162-rt176-rc1.
Please scream at me if I messed something up. Please test the patches
too.
The -rc release will be uploaded to kernel.org and will be deleted
when the final release is out. This is just a review release (or
release candidate).
The pre-releases will not be pushed to the git repository, only the
final release is.
If all goes well, this patch will be converted to the next main
release on 2018-11-05.
Enjoy,
-- Daniel
Daniel Wagner (2):
x86/kconfig: Fall back to ticket spinlocks
Linux 4.4.162-rt176-rc1
arch/x86/Kconfig | 1 -
localversion-rt | 2 +-
2 files changed, 1 insertion(+), 2 deletions(-)
--
2.14.4
From: Daniel Wagner <[email protected]>
v4.4.162-rt176-rc1 stable review patch.
If anyone has any objections, please let me know.
-----------
Sebastian writes:
"""
We reproducibly observe cache line starvation on a Core2Duo E6850 (2
cores), a i5-6400 SKL (4 cores) and on a NXP LS2044A ARM Cortex-A72 (4
cores).
The problem can be triggered with a v4.9-RT kernel by starting
cyclictest -S -p98 -m -i2000 -b 200
and as "load"
stress-ng --ptrace 4
The reported maximal latency is usually less than 60us. If the problem
triggers then values around 400us, 800us or even more are reported. The
upperlimit is the -i parameter.
Reproduction with 4.9-RT is almost immediate on Core2Duo, ARM64 and SKL,
but it took 7.5 hours to trigger on v4.14-RT on the Core2Duo.
Instrumentation show always the picture:
CPU0 CPU1
=> do_syscall_64 => do_syscall_64
=> SyS_ptrace => syscall_slow_exit_work
=> ptrace_check_attach => ptrace_do_notify / rt_read_unlock
=> wait_task_inactive rt_spin_lock_slowunlock()
-> while task_running() __rt_mutex_unlock_common()
/ check_task_state() mark_wakeup_next_waiter()
| raw_spin_lock_irq(&p->pi_lock); raw_spin_lock(¤t->pi_lock);
| . .
| raw_spin_unlock_irq(&p->pi_lock); .
\ cpu_relax() .
- .
*IRQ* <lock acquired>
In the error case we observe that the while() loop is repeated more than
5000 times which indicates that the pi_lock can be acquired. CPU1 on the
other side does not make progress waiting for the same lock with interrupts
disabled.
This continues until an IRQ hits CPU0. Once CPU0 starts processing the IRQ
the other CPU is able to acquire pi_lock and the situation relaxes.
"""
This matches with the observeration for v4.4-rt on a Core2Duo E6850:
CPU 0:
- no progress for a very long time in rt_mutex_dequeue_pi):
stress-n-1931 0d..11 5060.891219: function: __try_to_take_rt_mutex
stress-n-1931 0d..11 5060.891219: function: rt_mutex_dequeue
stress-n-1931 0d..21 5060.891220: function: rt_mutex_enqueue_pi
stress-n-1931 0....2 5060.891220: signal_generate: sig=17 errno=0 code=262148 comm=stress-ng-ptrac pid=1928 grp=1 res=1
stress-n-1931 0d..21 5060.894114: function: rt_mutex_dequeue_pi
stress-n-1931 0d.h11 5060.894115: local_timer_entry: vector=239
CPU 1:
- IRQ at 5060.894114 on CPU 1 followed by the IRQ on CPU 0
stress-n-1928 1....0 5060.891215: sys_enter: NR 101 (18, 78b, 0, 0, 17, 788)
stress-n-1928 1d..11 5060.891216: function: __try_to_take_rt_mutex
stress-n-1928 1d..21 5060.891216: function: rt_mutex_enqueue_pi
stress-n-1928 1d..21 5060.891217: function: rt_mutex_dequeue_pi
stress-n-1928 1....1 5060.891217: function: rt_mutex_adjust_prio
stress-n-1928 1d..11 5060.891218: function: __rt_mutex_adjust_prio
stress-n-1928 1d.h10 5060.894114: local_timer_entry: vector=239
Backporting all qspinlock related patches is very likely to introduce
regressions. Therefore, the recommended solution by PeterZ and Thomas
is to drop back to ticket spinlocks for v4.4.
Cc: Sebastian Andrzej Siewior <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Signed-off-by: Daniel Wagner <[email protected]>
---
arch/x86/Kconfig | 1 -
1 file changed, 1 deletion(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 6df130a37d41..f00cab581e2d 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -42,7 +42,6 @@ config X86
select ARCH_USE_BUILTIN_BSWAP
select ARCH_USE_CMPXCHG_LOCKREF if X86_64
select ARCH_USE_QUEUED_RWLOCKS
- select ARCH_USE_QUEUED_SPINLOCKS
select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH
select ARCH_WANTS_DYNAMIC_TASK_STRUCT
select ARCH_WANT_FRAME_POINTERS
--
2.14.4
v4.4.162-rt176-rc1 stable review patch.
If anyone has any objections, please let me know.
-----------
Signed-off-by: Daniel Wagner <[email protected]>
---
localversion-rt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/localversion-rt b/localversion-rt
index 8ead11a7cbc7..c3267d6bdd98 100644
--- a/localversion-rt
+++ b/localversion-rt
@@ -1 +1 @@
--rt175
+-rt176-rc1
--
2.14.4
On 2018-10-29 21:16:16 [+0100], Daniel Wagner wrote:
> From: Daniel Wagner <[email protected]>
>
> v4.4.162-rt176-rc1 stable review patch.
> If anyone has any objections, please let me know.
I though that we are going to route this via Greg/stable for v4.4?
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -42,7 +42,6 @@ config X86
> select ARCH_USE_BUILTIN_BSWAP
> select ARCH_USE_CMPXCHG_LOCKREF if X86_64
> select ARCH_USE_QUEUED_RWLOCKS
> - select ARCH_USE_QUEUED_SPINLOCKS
> select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH
> select ARCH_WANTS_DYNAMIC_TASK_STRUCT
> select ARCH_WANT_FRAME_POINTERS
> --
> 2.14.4
On 30/10/2018 00:12, Sebastian Andrzej Siewior wrote:
> On 2018-10-29 21:16:16 [+0100], Daniel Wagner wrote:
>> From: Daniel Wagner <[email protected]>
>>
>> v4.4.162-rt176-rc1 stable review patch.
>> If anyone has any objections, please let me know.
>
> I though that we are going to route this via Greg/stable for v4.4?
ah yes... I should not send out patches after a long day of work...
next try