Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752691AbdL0Tei (ORCPT ); Wed, 27 Dec 2017 14:34:38 -0500 Received: from mail-bn3nam01on0061.outbound.protection.outlook.com ([104.47.33.61]:4288 "EHLO NAM01-BN3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752175AbdL0Tee (ORCPT ); Wed, 27 Dec 2017 14:34:34 -0500 Date: Wed, 27 Dec 2017 19:34:15 +0000 From: Jayachandran C To: Robin Murphy Cc: Tomasz Nowicki , joro@8bytes.org, will.deacon@arm.com, lorenzo.pieralisi@arm.com, bhelgaas@google.com, Jayachandran.Nair@cavium.com, Ganapatrao.Kulkarni@cavium.com, linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-acpi@vger.kernel.org, linux-pci@vger.kernel.org, mw@semihalf.com, stable@vger.kernel.org Subject: Re: [PATCH V1 0/1] Fix kernel panic caused by device ID duplication presented to the IOMMU Message-ID: <20171227193414.GA17016@localhost> References: <1513696436-31834-1-git-send-email-tomasz.nowicki@caviumnetworks.com> <7f7fca76-88f8-6ee7-c402-fe4300c62253@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <7f7fca76-88f8-6ee7-c402-fe4300c62253@arm.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Originating-IP: [50.233.148.156] X-ClientProxiedBy: MWHPR20CA0041.namprd20.prod.outlook.com (2603:10b6:300:ed::27) To SN2PR0701MB1069.namprd07.prod.outlook.com (2a01:111:e400:3420::28) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f099dd7b-319d-44f5-daef-08d54d60d9cb X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(5600026)(4604075)(2017052603307)(7153060);SRVR:SN2PR0701MB1069; X-Microsoft-Exchange-Diagnostics: 1;SN2PR0701MB1069;3:jV2EP/XOBZWHBBFG4za1xmAVcxmUNPmDuNVy4Y67M6EZrlqPmAegXMPbODi+mYzu5Q2KVGfcF6+mbPUa7Pl9MhSfg66zc5NDFYCQ052IMgasgolP1jxjizg0ApOevIsN5QfiGTYSMglmPgIbtQ/E3LeStD0yAmAJquM/7EZG2Y873baNbiTzK21Ja2A25MYkV9qgHq5+/y/2QRtWB7wsr8tMEvWL8uWHRkzpBAdP9G4jYxWbSrYq0RqE4kPVZREC;25:LMIXjjITBdeSSoC9BzCF3l2zDGGC3NOt3NTwMDHYq2SORe2/xsGFkyeY1reXjZqGM/Y0mE+126yoMokkDBHloJDo3bDxzjkJL2nXNiC/Agh1XU606bS67+yxsAliTCQlAjCX5OKiD+Ln+PhhmlmtobO6H3KgD2D9dFrNSyy6W29Tr7MMZ3MIf/ZzpvOU3jE0cN30UxAoSZ14VgCHJvx1uYizNEV66SWMpiJEtYsjy/uyFPDi+7LLSwaki5kid62SAHZZuQLAQc6I7TndCPr3LA1nGZnLUwKOJyiMr/k+ZdU/Elcr1GqElGWVV5tKbRgVHBdI9hzBuALX2tMNGTEmJClv/moFS7/RhT5y6/Aq4q8=;31:6NYK3HEjDoq0cZESF2kmDNK4mKQr7oaT8w7Bgq/pOTKwdbKjJgt+pMLFdKfXzbFrBFRcoG9mksHwVNcbKxdfxbVBmX068THx2MrPIHqxqXFlCUqn/a6jskLeqmxQAAWX2xIuArOjSIudal4Q1V1ZR9415jxJIkSzieoiCEUF7cNji217vnsAMX1hLuab7HEitwIFkFitdWlRP9zsyGOw+eq5JSVZbihRez1WF0lwQz0= X-MS-TrafficTypeDiagnostic: SN2PR0701MB1069: Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jayachandran.Nair@cavium.com; X-Microsoft-Exchange-Diagnostics: 1;SN2PR0701MB1069;20:jzuifYofuom6EqP2n5/yLIhzb9n/4LZX24Cf8nepu0RiopjrTUVgUdQGtUhj0/2tSiisSMzsa5O3bx7i3fBv5DwrF+O7QwlpZCDsIrX7b8qwqglNRycgccgfjLBNHzZ//EIeeW3N5nQU7jXlvr0GzNXwEypWR1HQGqdWzkKSwSZiUkVCM5UIx9IMM6D2Z9aI0g0YgJFUVmwtSHLRGYejS9qfinXt6QjskVk8tuYhpUcJyuSL2NtrTEQn29ZfFICoATufkIP7aSTQSgT1ojmljoDhIxyhf9Ylq/GaB9wsYJ6YfZwPbztg9mr4NCEXtx2qPRddNuGq8Y33f89M+MAO4p4qFDJwUNDtiGoXE55hGisHtXuMeIjCc+qXgA891eeuE4ybD20uOIBEZUqMEOOPkcmMYvsaqqYsWmNuxmu1Xg8N9VDpiUe4D+RDeVHwD6Ma6dc8+8gae63W3RE+2E1ZPK2CJqO2XrWsEggV+nLSiY28WSIwoVVwK0kXOmQFofaOt0EkM3j+yapCOIAU2R5bvI6vLJsNOGSt+838y+wWfAoHsD4pfe5SPpE68JAxwVu+dCeYHVrMJjZ0tqzLfWbzYrJ9Wn79UzBMJIVAthCbRNU=;4:0vb6vS1KOgrNT65T4iBrqMvfKq55sWZiwR5f18tZS9YYICKYGP/+tEwvm1skOaTvr3iB892HmhRTNsZI19UtmutzXtAMBYmsdudPlyzTrULclAW8EQ0ecfgJpjpo0YrFDt82CA+ZzlH5wQ/rf0tXcvT3DpSJSla3XjD6MUNXnDVXLTU4PsaeRp494fs9KJ27YAu7c7A9HsFph3skUNIO2gepXmFZag0e4FneWhAmk4F8djAtOrBpjsgn9XZkHH4FK2NphJPh0kjb7HTNd1iHrLUh/Gl4S3Ip0oU1jpB4L7EQkyiWnPTBiCzCBx0pKSf3 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(131327999870524); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040470)(2401047)(8121501046)(5005006)(93006095)(3231023)(944501075)(10201501046)(3002001)(6041268)(20161123562045)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123558120)(6072148)(201708071742011);SRVR:SN2PR0701MB1069;BCL:0;PCL:0;RULEID:(100000803101)(100110400095);SRVR:SN2PR0701MB1069; X-Forefront-PRVS: 0534947130 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6069001)(39380400002)(366004)(396003)(39850400004)(376002)(346002)(24454002)(189003)(199004)(33656002)(386003)(6116002)(53546011)(2906002)(3846002)(1076002)(23726003)(53936002)(25786009)(4326008)(7416002)(6246003)(105586002)(68736007)(42882006)(76506005)(2950100002)(8676002)(6916009)(106356001)(6666003)(47776003)(5890100001)(5660300001)(81166006)(81156014)(66066001)(50466002)(8936002)(33716001)(45080400002)(229853002)(97736004)(551934003)(72206003)(7736002)(58126008)(478600001)(16586007)(316002)(59450400001)(52116002)(9686003)(55016002)(16526018)(575784001)(33896004)(305945005)(76176011)(6496006)(83506002)(18370500001);DIR:OUT;SFP:1101;SCL:1;SRVR:SN2PR0701MB1069;H:localhost;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;SN2PR0701MB1069;23:m9TIBxeTgzX/T31r44JMZxAA6R/tfKt4z58teRn?= =?us-ascii?Q?gKIpR2MzQIczKsKRGDQdRGI5zm4/aQln/2u57ieuZcKf4jI8+69kC+efEctd?= =?us-ascii?Q?YQeS/Ghb3VbqEF61dkryYUIuu3CQvJJMaDjoUpEQOT5IhCDGBubvdmQp9vid?= =?us-ascii?Q?+eCaUy66Kx1Ivc/37Cp1/RZ57X8M4qLEwKc0k1KstVHJq1d2fUlbhuj6OLt1?= =?us-ascii?Q?ZvqwZ/RRKKFSAm99aHu0HZPtecsumCwgvCi27mXVdMZpAe3dl1VZuU3fLZ5c?= =?us-ascii?Q?QqZ/S7RAJjLILEcPlYalXVpBHm/7n2PizdafEg5Ve+B1YzQbU69kaZOlndQw?= =?us-ascii?Q?Euj9pCJizKPBgDiOjiR0tzM+8LvylefFqCL7lUZbsf/cFE+xvuKf0EHg89AO?= =?us-ascii?Q?EsKL95navIMNlD1OJqN6/bHcFtJu7KyGrbjoWBcJXTfhZX36PAHZyRZZ8yqU?= =?us-ascii?Q?MVcTZGSKyB2zYxNOKVQ16HmdC8Aj5CQHCvFGUOBwese8ax9o7+ztUrJ9/pMY?= =?us-ascii?Q?n0VpMB3AApTpl1SqqU/oPwowX/KuQnDur/R21nMjY1JvA3FKTBb7ut5uMK3y?= =?us-ascii?Q?KleVG1Er9wzN2knX/Lg6hxdHniiqoTrwjm/hrXLrh1Kasx0ENuvy7Wh1w+ce?= =?us-ascii?Q?tExTZd7hIjwblNfgpryHW0PYCn9ATEJPJOy7z/r1yRjVx4UR+J0ihcAdE2gG?= =?us-ascii?Q?bXgB949EBJqwWdwrbnJ/Q2ElFsuvaK3MAYUwy8pJKU8ud3vNklQS8pSxC72Q?= =?us-ascii?Q?5RP13/gEh+Xb3PL2x6EvOayXe0I7EQsbDoDKwSi8qeNAdo2tx6C/b1kEtJKO?= =?us-ascii?Q?/hZ6RwT4JJlVcL3StEJ8+/yJT6DXiMx3GYcVR6yKrU/+e3ctouSJAL+uVrLf?= =?us-ascii?Q?e+mPba4k02mFQxWpDaBu3zZqpM7Lh879WmMcoBjG7ouUOR/h6etug3ukqJE8?= =?us-ascii?Q?cuxcM/V7OAy/XcQ1FBIy28WDU3n4qEX4g1QBvGZ+fz693KlSvKDoNmnux5o3?= =?us-ascii?Q?nW4FaV26MmLu4WGjoUFSAxNvjXlbYSh70/35rx/IuIjioTkqJkL8n4qSHGNd?= =?us-ascii?Q?adPcQHlbdnrxCetXKCI8N5R+/7dzN/mJ+k0jIx3IVfEGjigT7YZ4wh1hpoox?= =?us-ascii?Q?DheJOAyXweXVc7c7flWXZnJ6QFe2gW3IO2pVm9uAhS3AHwr/GFAVTHoPDBww?= =?us-ascii?Q?3x4Qd2v3Y8CUtqzSNBbukmo1hrdvRuSaT166vhoB4bwUZp7m1KYQqRgLdY4K?= =?us-ascii?Q?PN3uB4YXoqraI73i4Wlco/h5NPfEUdAtFsKyYOF3Sk20mdzuC72KuRwRzk1t?= =?us-ascii?Q?TbzxTSmPcl2fMERDlq2zXnMX1ot0www9y+up4Cw8GIkOoC1oijIx7pC7cFmd?= =?us-ascii?Q?JTCJG94XBHWQ9fiAioFu5kK905XMOuQ+1/Jo3G0JhTE9Vpi4p4kQx/P8qtqu?= =?us-ascii?Q?NbRtoYisoIHsvkEiAWAPudOdj3xqr200AGc58UT/ycTXL8PaNOmuA?= X-Microsoft-Exchange-Diagnostics: 1;SN2PR0701MB1069;6:aUca9fzqBAtt4gvSIM4SU4xYadWsEkEKMw1alY/p0ijJk4MtbZ8aruVqItwO69fEJPitiot4j42i1TlQscokukD9zdje9vtFv6+s8zyX1kbtXeLwevrbJmWf2DwkEEJJ+wYA78wwVela9OcKf1jsEW+lovuKw3c3UpGy85QCPiD3b016WtFgvkjaWTKvqxkm8mxgBaoKw5N0UmLJ0XbBCrxFYjU1wzJfXL7dMpL5TGrUnAqNES7Fw+oHktHN2g3xun5+/Bda7WwMNx4hBeG4J5ek80SiafzhRnED4piAWy26NtsabPn5hpJzNwCJ6Um49e/Zi8YUgJMGAhCTKdq/6+omUJdT9srgV3EyH+ENJ6A=;5:fst0eu0bRt+LsvX9W+3KNhCGgk/DbM/evcd9uD2KfWV211L8qmu5KuVckz+MjKAUw8EJz4axRvrgJYFzhjLF+82nmKXM89fdPP7Tdc9bloGNe1vYvb+97jHVnrvJdLxN7x5BkNX55EN/+osGwpZTC1+ZrTGRUZDDpO68coGV9nI=;24:LQf/o9hCdaN3YedbTuXgpKSi85091YQHmzmf0r15krMK9thE6MTTbDQxhcg/D9MRgf2jiev2QD8v+pJvIK7O83cNxHCiQ5nfHlrPBsgMS7A=;7:90M6T+YmeuvyNerSXOAQGb7sMiip2cIkUCJNxNPalgvRF5BvLF5FsKkwRqlFcs2ZT5T32bfT7/UQPzGMaht13H4Qo4KCYUv74ZUFvukgn07CyVQIraHUbhwDVWxUkSTxsJzwxeU/7hevn7AEgoGB27segHRmGXe667XyoW/zUWRN6kqrLawlMueIScurHOB/jx9g9ziz3cJLeh7yrfRisiN4cB0mT3eISiIECzXRaXxIuLtQf03PhbtHNoYFeREz SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Dec 2017 19:34:29.6664 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f099dd7b-319d-44f5-daef-08d54d60d9cb X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN2PR0701MB1069 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 6357 Lines: 135 Hi Robin, On Tue, Dec 19, 2017 at 04:34:46PM +0000, Robin Murphy wrote: > Hi Tomasz, > > On 19/12/17 15:13, Tomasz Nowicki wrote: > >Here is my lspci output of ThunderX2 for which I am observing kernel panic coming from > >SMMUv3 driver -> arm_smmu_write_strtab_ent() -> BUG_ON(ste_live): > > > ># lspci -vt > >-[0000:00]-+-00.0-[01-1f]--+ [...] > > + [...] > > \-00.0-[1e-1f]----00.0-[1f]----00.0 ASPEED Technology, Inc. ASPEED Graphics Family > > > >ASP device -> 1f:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family > >PCI-Express to PCI/PCI-X Bridge -> 1e:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge > >While setting up ASP device SID in IORT dirver: > >iort_iommu_configure() -> pci_for_each_dma_alias() > >we need to walk up and iterate over each device which alias transaction from > >downstream devices. > > > >AST device (1f:00.0) gets BDF=0x1f00 and corresponding SID=0x1f00 from IORT. > >Bridge (1e:00.0) is the first alias. Following PCI Express to PCI/PCI-X Bridge > >spec: PCIe-to-PCI/X bridges alias transactions from downstream devices using > >the subordinate bus number. For bridge (1e:00.0), the subordinate is equal > >to 0x1f. This gives BDF=0x1f00 and SID=1f00 which is the same as downstream > >device. So it is possible to have two identical SIDs. The question is what we > >should do about such case. Presented patch prevents from registering the same > >ID so that SMMUv3 is not complaining later on. > > Ooh, subtle :( There is logic in arm_smmu_attach_device() to tolerate > grouped devices aliasing to the same ID, but I guess I overlooked the > distinction of a device sharing an alias ID with itself. I'm not sure > I really like trying to work around this in generic code, since > fwspec->ids is essentially opaque data in a driver-specific format - in > theory a driver is free to encode a single logical ID into multiple > fwspec elements (I think I did that in an early draft of SMMUv2 SMR > support), at which point this approach might corrupt things massively. > > Does the (untested) diff below suffice? > > Robin. > > ----->8-----diff --git a/drivers/iommu/arm-smmu-v3.c > b/drivers/iommu/arm-smmu-v3.c > index f122071688fd..d8a730d83401 100644 > --- a/drivers/iommu/arm-smmu-v3.c > +++ b/drivers/iommu/arm-smmu-v3.c > @@ -1731,7 +1731,7 @@ static __le64 *arm_smmu_get_step_for_sid(struct > arm_smmu_device *smmu, u32 sid) > > static void arm_smmu_install_ste_for_dev(struct iommu_fwspec *fwspec) > { > - int i; > + int i, j; > struct arm_smmu_master_data *master = fwspec->iommu_priv; > struct arm_smmu_device *smmu = master->smmu; > > @@ -1739,6 +1739,13 @@ static void arm_smmu_install_ste_for_dev(struct > iommu_fwspec *fwspec) > u32 sid = fwspec->ids[i]; > __le64 *step = arm_smmu_get_step_for_sid(smmu, sid); > > + /* Bridged PCI devices may end up with duplicated IDs */ > + for (j = 0; j < i; j++) > + if (fwspec->ids[j] == sid) > + break; > + if (j < i) > + continue; > + > arm_smmu_write_strtab_ent(smmu, sid, step, &master->ste); > } > } Here is another tested by if you need one more: Tested-by: Jayachandran C. This fixes the crash below seen in ThunderX2 boards with Aspeed BMC (when booted without iommu.passthrough). Since this is a regression that breaks bootup on the platform, can you consider submitting this for the 4.15 cycle? [ 84.729351] ------------[ cut here ]------------ [ 84.729354] kernel BUG at /home/ubuntu/linux/drivers/iommu/arm-smmu-v3.c:1201! [ 84.729358] Internal error: Oops - BUG: 0 [#1] SMP [ 84.729360] Modules linked in: ast(+) ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops qede(+) drm q ed bnx2x(+) mpt3sas raid_class scsi_transport_sas sdhci_acpi mdio sdhci libcrc32c gpio_xlp [ 84.729375] CPU: 190 PID: 1725 Comm: systemd-udevd Not tainted 4.15.0-rc5 #37 [ 84.729376] Hardware name: Cavium Inc. Unknown/Unknown, BIOS 1.0 11/08/2017 [ 84.729379] pstate: 80400009 (Nzcv daif +PAN -UAO) [ 84.729388] pc : arm_smmu_write_strtab_ent+0x1f0/0x1f8 [ 84.729391] lr : arm_smmu_install_ste_for_dev+0x70/0xc8 [ 84.729392] sp : ffff00001f6e3730 [ 84.729393] x29: ffff00001f6e3730 x28: ffff809ecc0d2268 [ 84.729395] x27: 0000000000000018 x26: ffff00001f6e3910 [ 84.729397] x25: ffff809ecc0d2238 x24: ffff809ecdc94158 [ 84.729398] x23: 0000000000000018 x22: 0000000000001d00 [ 84.729400] x21: ffff809ecdc90018 x20: ffff809ecb5ef288 [ 84.729401] x19: ffff80beca480000 x18: ffff0000093b8c08 [ 84.729402] x17: ffff000008aeab70 x16: ffff00001f6e3a20 [ 84.729404] x15: 0000000000000000 x14: dead000000000100 [ 84.729405] x13: dead000000000200 x12: 0000000000000020 [ 84.729407] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f [ 84.729408] x9 : 0000000000000000 x8 : 0000000000000000 [ 84.729410] x7 : 0000000000000100 x6 : 0000000000000015 [ 84.729411] x5 : 0000000000000015 x4 : 0000000000000002 [ 84.729413] x3 : ffff809ecb5ef288 x2 : 0000000000000001 [ 84.729414] x1 : ffff000008691e30 x0 : ffff809ecc0d2238 [ 84.729418] Process systemd-udevd (pid: 1725, stack limit = 0x000000002c585821) [ 84.729420] Call trace: [ 84.729423] arm_smmu_write_strtab_ent+0x1f0/0x1f8 [ 84.729425] arm_smmu_install_ste_for_dev+0x70/0xc8 [ 84.729426] arm_smmu_attach_dev+0x100/0x2f8 [ 84.729431] __iommu_attach_device+0x54/0xe0 [ 84.729433] iommu_group_add_device+0x150/0x428 [ 84.729435] iommu_group_get_for_dev+0x84/0x180 [ 84.729436] arm_smmu_add_device+0x138/0x240 [ 84.729445] iort_iommu_configure+0x138/0x188 [ 84.729452] acpi_dma_configure+0x3c/0x80 [ 84.729456] dma_configure+0xb0/0xe0 [ 84.729462] driver_probe_device+0x1f0/0x4a8 [ 84.729464] __driver_attach+0x124/0x128 [ 84.729466] bus_for_each_dev+0x70/0xb0 [ 84.729467] driver_attach+0x30/0x40 [ 84.729469] bus_add_driver+0x248/0x2b8 [ 84.729471] driver_register+0x68/0x100 [ 84.729478] __pci_register_driver+0x5c/0x70 [ 84.729488] ast_init+0x30/0x1000 [ast] [ 84.729494] do_one_initcall+0x5c/0x168 [ 84.729501] do_init_module+0x64/0x1f4 [ 84.729502] load_module+0x1e64/0x21e8 [ 84.729503] SyS_finit_module+0x108/0x118 [ 84.729505] el0_svc_naked+0x20/0x24 [ 84.729508] Code: 34ffff82 d4210000 d2800120 17ffffd8 (d4210000) Thanks, JC.