Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [PATCH 4.18 000/123] 4.18.6-stable review
To:     Linus Torvalds <torvalds@linux-foundation.org>
Cc:     Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Shuah Khan <shuah@kernel.org>, patches@kernelci.org,
        Ben Hutchings <ben.hutchings@codethink.co.uk>,
        lkft-triage@lists.linaro.org, stable <stable@vger.kernel.org>
References: <20180903165719.499675257@linuxfoundation.org>
 <20180904162434.GA16396@roeck-us.net> <20180905090110.GC30538@kroah.com>
 <7d4d11ab-c769-44b4-0037-d1be7f45e2c8@roeck-us.net>
 <CA+55aFzpnXYHNeog==-aWYO=V_67viGrMah1ZSdcDqcRO214-w@mail.gmail.com>
From:   Guenter Roeck <linux@roeck-us.net>
Message-ID: <ec9bdd81-8dcd-38fa-0d96-cd7b1a550e92@roeck-us.net>
Date:   Sat, 8 Sep 2018 20:58:24 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.9.1
MIME-Version: 1.0
In-Reply-To: <CA+55aFzpnXYHNeog==-aWYO=V_67viGrMah1ZSdcDqcRO214-w@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 09/05/2018 10:01 AM, Linus Torvalds wrote:
> On Wed, Sep 5, 2018 at 8:34 AM Guenter Roeck <linux@roeck-us.net> wrote:
>>
>> On 09/05/2018 02:01 AM, Greg Kroah-Hartman wrote:
>>>> ---
>>>> [ 9990.754641] watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [kworker/5:1:155]
>>>> [ 9990.762601] RIP: 0010:smp_call_function_many+0x208/0x270
>>>> [ 9990.762601] Code: e8 0d d1 77 00 3b 05 cb f0 24 01 0f 83 86 fe ff ff 48 63 d0 49 8b 0c 24 48 03 0c d5 00 f7 11 a7 8b 51 18 83 e2 01 74 0a f3 90 <8b> 51 18 83 e2 01 75 f6 eb c7 0f b6 4d d0 4c 89 f2 4c 89 ee 44 89
> 
> It's stuck in this loop:
> 
>     loop:
>          pause
>          mov    0x18(%rcx),%edx
>          and    $0x1,%edx
>          jne    loop
> 
> which is csd_lock_wait().
> 
> Judging by the offset in smp_call_function_many(), it's the final one
> (there's two: the other one is part of "csd_lock()"). But that's just
> a guess.
> 
> Anyway, it means that we're waiting for another CPU to finish
> processing an IPI - either a previous one we sent asynchronously (if
> it's the earlier csd_lock() case) or the TLB IPI we just sent and
> we're waiting for completion of.
> 
>> Not tested, but I see it in v4.17.19 and in v4.18.6-rc2. Turns out it is
>> related to heavy load, not to suspend/resume. At this point I suspect that
>> it may be an AMD/Ryzen specific problem - it looks like it disappears if I
>> add "kernel.randomize_va_space = 0" to /etc/sysctl.conf. No idea if it is a
>> CPU bug or some AMD specific code problem. I'll try to analyze it further.
> 
> Ouch. Some IPI sending/receiving problem would be very very painful to
> debug if it's hw related.
> 

Turns out this is a well known problem with Ryzen CPUs:

https://bugzilla.kernel.org/show_bug.cgi?id=196683

Guenter