Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Fri, 15 Jun 2018 12:41:10 -0400 (EDT)
From:   Mikulas Patocka <mpatocka@redhat.com>
To:     Steven Rostedt <rostedt@goodmis.org>
cc:     Alan Stern <stern@rowland.harvard.edu>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ming Lei <ming.lei@redhat.com>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        USB list <linux-usb@vger.kernel.org>,
        Kernel development list <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] usb: don't offload isochronous urb completions to
 ksoftirq
In-Reply-To: <20180614183546.7e04024c@gandalf.local.home>
Message-ID: <alpine.LRH.2.02.1806151222200.5648@file01.intranet.prod.int.rdu2.redhat.com>
References: <Pine.LNX.4.44L0.1806131442230.1476-100000@iolanthe.rowland.org> <alpine.LRH.2.02.1806131521540.26196@file01.intranet.prod.int.rdu2.redhat.com> <20180613183145.17564d82@gandalf.local.home> <alpine.LRH.2.02.1806141809500.6435@file01.intranet.prod.int.rdu2.redhat.com>
 <20180614183546.7e04024c@gandalf.local.home>
User-Agent: Alpine 2.02 (LRH 1266 2009-07-14)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


On Thu, 14 Jun 2018, Steven Rostedt wrote:

> On Thu, 14 Jun 2018 18:23:11 -0400 (EDT)
> Mikulas Patocka <mpatocka@redhat.com> wrote:
> 
> > I don't think it's so easy. The kernel 2.4 and below did this. And the 
> > problem was that if there's a storm of network packets, the softirq code 
> > would cause lockup of the whole machine. In order to not lockup the 
> > machine - somewhere in the 2.4 cycle - the ksoftirqd thread was 
> > introduced.
> > 
> > If you have CONFIG_PREEMPT_RT_FULL and you call softirqs in the interrupt 
> > thread, you could only stall the interrupt thread. If you do the same 
> > thing without CONFIG_PREEMPT_RT_FULL, you stall the whole CPU.
> > 
> 
> Note, PREEMPT_RT also uses ksoftirqd too. Although we may set it to RT
> prio 1. It is triggered if the softirq itself raises a softirq of the
> same kind, and then the work is passed off to the ksoftirqd.

I think the major problem (in the upstream kernel) with softirq latency is 
this:
static inline void invoke_softirq(void)
{
        if (ksoftirqd_running())
                return;

It means that if any piece of code kicks ksoftirq, no tasklets are 
processed in the irq context at all.

So, the ehci callbacks will be offloaded to ksoftirqd (no matter how small 
they are) and this causes audio skipping. Could this be changed, so that 
it processes tasklets submitted with "tasklet_hi_schedule" in irq context 
even if ksoftirqd is running?

It's not easy - __do_softirq lacks any locking - so it can't run 
concurrently in process context and in irq context.

> Causing the interrupt thread to stall (or by going into a loop of
> softirqs) is likely to lock up the CPU on RT too, as interrupt threads
> are usually run at priority 50, which will keep normal threads from
> running on that CPU.
> 
> -- Steve

BTW. when I subject the machine to a ping flood (ping -f), the 
non-realtime kernel (with the patch to avoid offloading ehci urb 
callbacks) performs better than the real-time kernel.

With the real-time kernel, all the networking work is done in the thread 
"irq/12-eth0", that has (by default) priority -51, it consumes 30% CPU 
time and causes sound skipping. I can avoid the skipping by lowering the 
priority of "irq/12-eth0".

With non-realtime kernel, no such problem during ping flood exists.

Mikulas