Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753843AbbBZN4l (ORCPT ); Thu, 26 Feb 2015 08:56:41 -0500 Received: from www.linutronix.de ([62.245.132.108]:49055 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753420AbbBZN4j (ORCPT ); Thu, 26 Feb 2015 08:56:39 -0500 Date: Thu, 26 Feb 2015 14:56:30 +0100 From: Sebastian Andrzej Siewior To: Steven Rostedt Cc: Thavatchai Makphaibulchoke , Thavatchai Makphaibulchoke , linux-kernel@vger.kernel.org, mingo@redhat.com, tglx@linutronix.de, linux-rt-users@vger.kernel.org Subject: Re: [PATCH 3.14.25-rt22 1/2] rtmutex Real-Time Linux: Fixing kernel BUG at kernel/locking/rtmutex.c:997! Message-ID: <20150226135630.GD12992@linutronix.de> References: <1424395866-81589-1-git-send-email-tmac@hp.com> <1424395866-81589-2-git-send-email-tmac@hp.com> <20150223133719.2b7c604e@gandalf.local.home> <54EBC2DB.3050904@hp.com> <20150223195743.546b2ef0@grimm.local.home> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20150223195743.546b2ef0@grimm.local.home> X-Key-Id: 97C4700B X-Key-Fingerprint: 09E2 D1F3 9A3A FF13 C3D3 961C 0688 1C1E 97C4 700B User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2934 Lines: 60 * Steven Rostedt | 2015-02-23 19:57:43 [-0500]: >On Mon, 23 Feb 2015 17:16:27 -0700 >Thavatchai Makphaibulchoke wrote: >> If I'm not mistaken, another reason could also be due to the rate of the >> timer interrupt, in the case that the mutex is highly contested IH could >> stall the non-real-time requester for a long time, even to the point of >> the cpu is perceived as hung. > >Perhaps we should just have trylocks fail if there are other waiters. >As it wont slow down the high priority task. And doing that would >probably even help other rt tasks that are blocked on the lock waiting >for a release. Why make those tasks wait more if even a higher priority >task is simply doing a trylock and can safely fail it. At least we >could do that if the task trying to get the lock is a interrupt. What happened so far? The events I remember: - we gained FULL_NO_HZ - people started to isolate CPUs and push their work / RT tasks there - it has been noticed that the kernel is raising the timer softirq even there is nothing going on once the softirq was started. - tglx came with a patch which could go mainline if solves the problem. - this patch did not make its way upstream (yet) and I pulled it into -RT. Isn't this a problem in mainline, too? Why isn't there anyone screaming? - we had dead locks because the inner-lock of the sleeping was not safe to be used from hardirq context. #1 - we had boxes freezing on startup and not making progress due to missed irq_work wakeups. #2 - we had a deadlock splat on UP because the trylock failed. This was fixed by only allowing this feature on SMP (since it only makes sense with isolated CPUs). #3 - Now since the rtmutex rework we have dead lock splats which BUG()s the systems. #4 The four problems we had/have so far are -RT specific but still plainfull when I think back. rtmutex wasn't made to be accessed from hardirq context. That is where we use the rawlocks. One problem that we still have and Peter pointer out around #1 is about owner boosting if the lock is held in hardirq context and the wrong owner is recorded. This problem was ignored so far. Using a fake task as you suggest in irq context and ignoring would somehow fix the boosting problem and avoid the deadlock we see now. I am not sure if we want keep doing that. The only reason why we grab the lock in the first place was to check if there is a timer pending and we run on the isolated CPU. It should not matter for the other CPUs, right? So instead going further that road, what about storing base->next_timer someplace so it can be obtained via atomic_read() for the isolated CPUs? >-- Steve Sebastian -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/