Received: by 2002:a25:6193:0:0:0:0:0 with SMTP id v141csp406954ybb; Fri, 20 Mar 2020 01:05:41 -0700 (PDT) X-Google-Smtp-Source: ADFU+vt5+BXWsApYQuiK8Ttxq6/rJjeXSqJIexX4sNG8iCOFBQeKCEzdgt9pAUIJphAZYHOGf87B X-Received: by 2002:a05:6808:43:: with SMTP id v3mr5584876oic.59.1584691541246; Fri, 20 Mar 2020 01:05:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1584691541; cv=none; d=google.com; s=arc-20160816; b=V2MDmi9JZxVmPMCbGy0nLF+FIEt87ybPipjiQxqsNFDiABC0TtQ5qionMSOCMCBs1I diezRzrPBHXEIRj8ocmznpzQcq71gjW7D5rVR0lf+e/7GXPOh1rbqqKVF5SK2XIQT5XA NbpKyF5deY2EgAiF/zsDhEHTuHiEKC3HWsn3CtcVLQsws5j+/0mXkX4OjZtiqTP2njar C8WyByzqIWRohBI2LNPDwQGAGMmLZAWOGXy2+OFWUclKnaL9TzK62aEgVEFD5+l9xIXr RPfFgJIrcRvW0jc2qi2DOrIxqCDQwQvFqeSwyfD/OoDShPqq/YY34kuw3TaFdo5q5ait kipw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:autocrypt:openpgp:from:references:cc:to:subject :ironport-sdr:ironport-sdr; bh=o+OviVy99Gj5wxCuvypNBFVr8BE1QISTdhny/DgYGaw=; b=j0A20Gn/7Vj+ZZxK5Ux56ZgJ4Rak67K0bd3ZP8y49+4O7D/bbSjXO2C4be25M6thAf ioZm1aHRuDwYvc3EorvtYmABpLZ7Gmz/db1Eas7F3jWuvC6MgpdMDg337KFrI1wf/HIX q9PmFRNvUR7aFlNcASAbfG5xX3NrOWHAzVC32sjyhW85hcK8VOZf29lnZAXQ0ZBEYM7z 6bt7hiieF/tEo0MUioqTLzrLB8ZDyqbhLbZX8l0pR0YlGQ3Rhld9lY7MFFxmAZHsNSTk sbdeuutRj1/gU/V/e4j9qULCeAtaej6p1IAQAEQvDllHyxHkm3HBUIzCYGu3TpWqYIT8 l8LQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 96si2623824otn.253.2020.03.20.01.05.28; Fri, 20 Mar 2020 01:05:41 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726821AbgCTIFA (ORCPT + 99 others); Fri, 20 Mar 2020 04:05:00 -0400 Received: from mga09.intel.com ([134.134.136.24]:44276 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726654AbgCTIE7 (ORCPT ); Fri, 20 Mar 2020 04:04:59 -0400 IronPort-SDR: UNVRl/GOOzZqE4wgxa4aRIsjACsMNH0gv5fir14nNqwA1uv949MbX+KEIqa+4W9j5zV4sb/arb XO2wFixgBRzw== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Mar 2020 01:04:59 -0700 IronPort-SDR: ovVxiJIUF16KpUVou9unsRe35fIh0X5WQMpcIXeGfYTOypfMU2EP3gviUE0YqkENq4UecBnU3j tLLp7rHiAJhw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,283,1580803200"; d="scan'208";a="245420251" Received: from mattu-haswell.fi.intel.com (HELO [10.237.72.170]) ([10.237.72.170]) by orsmga003.jf.intel.com with ESMTP; 20 Mar 2020 01:04:56 -0700 Subject: Re: MSI interrupt for xhci still lost on 5.6-rc6 after cpu hotplug To: Evan Green Cc: x86@kernel.org, linux-pci , Thomas Gleixner , LKML , Bjorn Helgaas , "Ghorai, Sukumar" , "Amara, Madhusudanarao" , "Nandamuri, Srikanth" References: <806c51fa-992b-33ac-61a9-00a606f82edb@linux.intel.com> From: Mathias Nyman Openpgp: preference=signencrypt Autocrypt: addr=mathias.nyman@linux.intel.com; prefer-encrypt=mutual; keydata= mQINBFMB0ccBEADd+nZnZrFDsIjQtclVz6OsqFOQ6k0nQdveiDNeBuwyFYykkBpaGekoHZ6f lH4ogPZzQ+pzoJEMlRGXc881BIggKMCMH86fYJGfZKWdfpg9O6mqSxyEuvBHKe9eZCBKPvoC L2iwygtO8TcXXSCynvXSeZrOwqAlwnxWNRm4J2ikDck5S5R+Qie0ZLJIfaId1hELofWfuhy+ tOK0plFR0HgVVp8O7zWYT2ewNcgAzQrRbzidA3LNRfkL7jrzyAxDapuejuK8TMrFQT/wW53e uegnXcRJaibJD84RUJt+mJrn5BvZ0MYfyDSc1yHVO+aZcpNr+71yZBQVgVEI/AuEQ0+p9wpt O9Wt4zO2KT/R5lq2lSz1MYMJrtfFRKkqC6PsDSB4lGSgl91XbibK5poxrIouVO2g9Jabg04T MIPpVUlPme3mkYHLZUsboemRQp5/pxV4HTFR0xNBCmsidBICHOYAepCzNmfLhfo1EW2Uf+t4 L8IowAaoURKdgcR2ydUXjhACVEA/Ldtp3ftF4hTQ46Qhba/p4MUFtDAQ5yeA5vQVuspiwsqB BoL/298+V119JzM998d70Z1clqTc8fiGMXyVnFv92QKShDKyXpiisQn2rrJVWeXEIVoldh6+ J8M3vTwzetnvIKpoQdSFJ2qxOdQ8iYRtz36WYl7hhT3/hwkHuQARAQABtCdNYXRoaWFzIE55 bWFuIDxtYXRoaWFzLm55bWFuQGdtYWlsLmNvbT6JAjsEEwECACUCGwMGCwkIBwMCBhUIAgkK CwQWAgMBAh4BAheABQJTAeo1AhkBAAoJEFiDn/uYk8VJOdIP/jhA+RpIZ7rdUHFIYkHEKzHw tkwrJczGA5TyLgQaI8YTCTPSvdNHU9Rj19mkjhUO/9MKvwfoT2RFYqhkrtk0K92STDaBNXTL JIi4IHBqjXOyJ/dPADU0xiRVtCHWkBgjEgR7Wihr7McSdVpgupsaXhbZjXXgtR/N7PE0Wltz hAL2GAnMuIeJyXhIdIMLb+uyoydPCzKdH6znfu6Ox76XfGWBCqLBbvqPXvk4oH03jcdt+8UG 2nfSeti/To9ANRZIlSKGjddCGMa3xzjtTx9ryf1Xr0MnY5PeyNLexpgHp93sc1BKxKKtYaT0 lR6p0QEKeaZ70623oB7Sa2Ts4IytqUVxkQKRkJVWeQiPJ/dZYTK5uo15GaVwufuF8VTwnMkC 4l5X+NUYNAH1U1bpRtlT40aoLEUhWKAyVdowxW4yGCP3nL5E69tZQQgsag+OnxBa6f88j63u wxmOJGNXcwCerkCb+wUPwJzChSifFYmuV5l89LKHgSbv0WHSN9OLkuhJO+I9fsCNvro1Y7dT U/yq4aSVzjaqPT3yrnQkzVDxrYT54FLWO1ssFKAOlcfeWzqrT9QNcHIzHMQYf5c03Kyq3yMI Xi91hkw2uc/GuA2CZ8dUD3BZhUT1dm0igE9NViE1M7F5lHQONEr7MOCg1hcrkngY62V6vh0f RcDeV0ISwlZWuQINBFMB0ccBEACXKmWvojkaG+kh/yipMmqZTrCozsLeGitxJzo5hq9ev31N 2XpPGx4AGhpccbco63SygpVN2bOd0W62fJJoxGohtf/g0uVtRSuK43OTstoBPqyY/35+VnAV oA5cnfvtdx5kQPIL6LRcxmYKgN4/3+A7ejIxbOrjWFmbWCC+SgX6mzHHBrV0OMki8R+NnrNa NkUmMmosi7jBSKdoi9VqDqgQTJF/GftvmaZHqgmVJDWNrCv7UiorhesfIWPt1O/AIk9luxlE dHwkx5zkWa9CGYvV6LfP9BznendEoO3qYZ9IcUlW727Le80Q1oh69QnHoI8pODDBBTJvEq1h bOWcPm/DsNmDD8Rwr/msRmRyIoxjasFi5WkM/K/pzujICKeUcNGNsDsEDJC5TCmRO/TlvCvm 0X+vdfEJRZV6Z+QFBflK1asUz9QHFre5csG8MyVZkwTR9yUiKi3KiqQdaEu+LuDD2CGF5t68 xEl66Y6mwfyiISkkm3ETA4E8rVZP1rZQBBm83c5kJEDvs0A4zrhKIPTcI1smK+TWbyVyrZ/a mGYDrZzpF2N8DfuNSqOQkLHIOL3vuOyx3HPzS05lY3p+IIVmnPOEdZhMsNDIGmVorFyRWa4K uYjBP/W3E5p9e6TvDSDzqhLoY1RHfAIadM3I8kEx5wqco67VIgbIHHB9DbRcxQARAQABiQIf BBgBAgAJBQJTAdHHAhsMAAoJEFiDn/uYk8VJb7AQAK56tgX8V1Wa6RmZDmZ8dmBC7W8nsMRz PcKWiDSMIvTJT5bygMy1lf7gbHXm7fqezRtSfXAXr/OJqSA8LB2LWfThLyuuCvrdNsQNrI+3 D+hjHJjhW/4185y3EdmwwHcelixPg0X9EF+lHCltV/w29Pv3PiGDkoKxJrnOpnU6jrwiBebz eAYBfpSEvrCm4CR4hf+T6MdCs64UzZnNt0nxL8mLCCAGmq1iks9M4bZk+LG36QjCKGh8PDXz 9OsnJmCggptClgjTa7pO6040OW76pcVrP2rZrkjo/Ld/gvSc7yMO/m9sIYxLIsR2NDxMNpmE q/H7WO+2bRG0vMmsndxpEYS4WnuhKutoTA/goBEhtHu1fg5KC+WYXp9wZyTfeNPrL0L8F3N1 BCEYefp2JSZ/a355X6r2ROGSRgIIeYjAiSMgGAZMPEVsdvKsYw6BH17hDRzltNyIj5S0dIhb Gjynb3sXforM/GVbr4mnuxTdLXQYlj2EJ4O4f0tkLlADT7podzKSlSuZsLi2D+ohKxtP3U/r 42i8PBnX2oAV0UIkYk7Oel/3hr0+BP666SnTls9RJuoXc7R5XQVsomqXID6GmjwFQR5Wh/RE IJtkiDAsk37cfZ9d1kZ2gCQryTV9lmflSOB6AFZkOLuEVSC5qW8M/s6IGDfYXN12YJaZPptJ fiD/ Message-ID: Date: Fri, 20 Mar 2020 10:07:26 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 19.3.2020 22.24, Evan Green wrote: > On Wed, Mar 18, 2020 at 12:23 PM Mathias Nyman > wrote: >> >> Hi >> >> I can reproduce the lost MSI interrupt issue on 5.6-rc6 which includes >> the "Plug non-maskable MSI affinity race" patch. >> >> I can see this on a couple platforms, I'm running a script that first generates >> a lot of usb traffic, and then in a busyloop sets irq affinity and turns off >> and on cpus: >> >> for i in 1 3 5 7; do >> echo "1" > /sys/devices/system/cpu/cpu$i/online >> done >> echo "A" > "/proc/irq/*/smp_affinity" >> echo "A" > "/proc/irq/*/smp_affinity" >> echo "F" > "/proc/irq/*/smp_affinity" >> for i in 1 3 5 7; do >> echo "0" > /sys/devices/system/cpu/cpu$i/online >> done >> >> I added some very simple debugging but I don't really know what to look for. >> xhci interrupts (122) just stop after a setting msi affinity, it survived many >> similar msi_set_affinity() calls before this. >> >> I'm not that familiar with the inner workings of this, but I'll be happy to >> help out with adding debugging and testing patches. > > How quickly are you able to reproduce this when you run your script? Less than a minute > Does reverting Thomas' patch make it repro faster? Can you send the > output of lspci -vvv for the xhci device? It's roughly the same after reverting "6f1a4891a592 x86/apic/msi: Plug non-maskable MSI affinity race" # lspci -vvv -d :a3af 00:14.0 USB controller: Intel Corporation Device a3af (prog-if 30 [XHCI]) Subsystem: Intel Corporation Device 7270 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR-