Received: by 2002:a25:ab43:0:0:0:0:0 with SMTP id u61csp5377083ybi; Wed, 12 Jun 2019 01:15:59 -0700 (PDT) X-Google-Smtp-Source: APXvYqyOkmE2Egf79dfFXd6PGGns0Cn8sioFQ4LmCm0osiCJMYQUfJOHCpFsdBcOfOZFVrZXRkfD X-Received: by 2002:a17:902:4481:: with SMTP id l1mr81984562pld.121.1560327359805; Wed, 12 Jun 2019 01:15:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560327359; cv=none; d=google.com; s=arc-20160816; b=HSb4eYHn2IrFxxOvOyzsioRrsZskhKUfPVqdEsfiIFAbSA3mlInQQlqazO0wk1lULJ Uu/GuzFz6lhQFOd0OgXImCF4dv6n/zBST8kWWIevo+BBtNW03m656vBWfOTdohxDDv7V sY2ROJdNB7OyuOLvVDdhxHGe0XhHs6ZmABBZioPwDhXPjh5faHiJnrHjLwyykZV4Tm2B u4lyHdyLO0C2oBhnevh2iGUuQ2Er3epny6RZOstCxvCaDpKGz0gnxMSz0lJ+BMGgWUhA nhe5CgCR5Xjfh9gLsiwRidUVMkSdHUEqr9IhASepNT7aoVo30P3HIEFqgXIw+Su5b5hW 36eA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=lKi9fDU0Lxa/G5VIecJ7zcekWrVtK+7i/1xldaoGnKM=; b=TAi2IxE7o6nJ4Pg/+86hQo0cBVAY+VHJ2urOMuvMxP6/3c7y84bUkRDYx6csd/y+As pYDqzmu5k07I2h0koDE9RO78SFIMeMcYTAkXHeohmwWrMQBeLtqO+jI8k4wf3G7TV3bJ 2t6RgC4UZqnW3asusvBSs9mICgFQwzoSuI3Q2s1ZNoQyMjn0JZb77+5+MSHrfElqOSHq UqRb6grIHo4340Opk88NWD8RZ9cKrOKEE0elba0j2BKXlDfXu4ZlDLjzc8CTaalgC0Uj HIfxHMC9VbpFRZcorYlT0p9JGWpAJmFHmqfnGavq0coqRDjhNhVGP/eHpTxKdlUiaSM+ 4zRg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=eyDsg02x; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x12si9832660pgx.314.2019.06.12.01.15.43; Wed, 12 Jun 2019 01:15:59 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=eyDsg02x; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731018AbfFLG0M (ORCPT + 99 others); Wed, 12 Jun 2019 02:26:12 -0400 Received: from mail-vs1-f67.google.com ([209.85.217.67]:38216 "EHLO mail-vs1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726010AbfFLG0M (ORCPT ); Wed, 12 Jun 2019 02:26:12 -0400 Received: by mail-vs1-f67.google.com with SMTP id k9so386830vso.5 for ; Tue, 11 Jun 2019 23:26:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=lKi9fDU0Lxa/G5VIecJ7zcekWrVtK+7i/1xldaoGnKM=; b=eyDsg02xExXKcTNUxPVULiXqWMYD105i/SYss3Q7V7I4w3D6V8UOwCLq2ZoHI3mfQg OTOZo+12wHolv7tBGv3sdkmuSCqRtGRtsH/CXaw+WsTDMgkJfbsIxMXGQHVukIrCMeIp WMJtP4K8mh4CfnD/OzA0LsD4n5zUmMJFuZbqIliJxj+/ysyNeX8d0WMZJkgvyDtB53cj r8AAjqQ2NGyvklSU8TGfeou44Ndc8sZVnQxufRmi9ZUNRSE31V2H7I8bSiLHmCPd18cI IzURSSp1riW/tb78SPfnulXCdtyqSevY9l/dVLxcqTLtox2ZZPsTDydfn6bTGUfVyk/U s8xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=lKi9fDU0Lxa/G5VIecJ7zcekWrVtK+7i/1xldaoGnKM=; b=oGUp/1YCtsc2dayZuFucHQpVA8VonfwIWWJGBtZwZVG3UAvlXZjeOHdC+asa5tweom WeIyZCvrU72BEB8dR8+1L0u+k4Y4X7v0xf+mGNEWeweoocfqfpZq+WJkR/z3ujlqPcPt iqJuoiscHUlUJl9rNisD2lAmqyspuq2RiPZDIELiCbRZqD/1XkShlxwGpxC7uEWBiqyh bG+qN65PJRHaoyoasS9JA/bV+2OM+4cT/IIwyxM83vyb3gsa5+6QS+4afrO3J87tGqfH LEBbkzUQ+3XN7QO1dwRrU+x64bKzqParWSPz5m2zp1HkEwdlpOS9QDXiiSutTbvj9JWQ bt/A== X-Gm-Message-State: APjAAAXNxnO4Qyj5RvTe2AVfWBnHYjZLiPPSVgszJlL76T4L/LdL3vQA EUw7IvOAjnyw3jdiFmVGDmp4JhqWoYZ/Ao4gfu0= X-Received: by 2002:a67:e3d5:: with SMTP id k21mr28292559vsm.172.1560320771385; Tue, 11 Jun 2019 23:26:11 -0700 (PDT) MIME-Version: 1.0 References: <20190611092144.11194-1-oded.gabbay@gmail.com> <20190611095857.GB24058@kroah.com> <20190611151753.GA11404@infradead.org> <20190611152655.GA3972@kroah.com> In-Reply-To: From: Oded Gabbay Date: Wed, 12 Jun 2019 09:25:45 +0300 Message-ID: Subject: Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9 To: Benjamin Herrenschmidt Cc: Greg KH , linuxppc-dev@ozlabs.org, Christoph Hellwig , "Linux-Kernel@Vger. Kernel. Org" , Oliver OHalloran , Russell Currey Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jun 12, 2019 at 1:53 AM Benjamin Herrenschmidt wrote: > > On Tue, 2019-06-11 at 20:22 +0300, Oded Gabbay wrote: > > > > > So, to summarize: > > > If I call pci_set_dma_mask with 48, then it fails on POWER9. However, > > > in runtime, I don't know if its POWER9 or not, so upon failure I will > > > call it again with 32, which makes our device pretty much unusable. > > > If I call pci_set_dma_mask with 64, and do the dedicated configuration > > > in Goya's PCIe controller, then it won't work on x86-64, because bit > > > 59 will be set and the host won't like it (I checked it). In addition, > > > I might get addresses above 50 bits, which my device can't generate. > > > > > > I hope this makes things more clear. Now, please explain to me how I > > > can call pci_set_dma_mask without any regard to whether I run on > > > x86-64 or POWER9, considering what I wrote above ? > > > > > > Thanks, > > > Oded > > > > Adding ppc mailing list. > > You can't. Your device is broken. Devices that don't support DMAing to > the full 64-bit deserve to be added to the trash pile. > Hmm... right know they are added to customers data-centers but what do I know ;) > As a result, getting it to work will require hacks. Some GPUs have > similar issues and require similar hacks, it's unfortunate. > > Added a couple of guys on CC who might be able to help get those hacks > right. Thanks :) > > It's still very fishy .. the idea is to detect the case where setting a > 64-bit mask will give your system memory mapped at a fixed high address > (1 << 59 in our case) and program that in your chip in the "Fixed high > bits" register that you seem to have (also make sure it doesn't affect > MSIs or it will break them). MSI-X are working. The set of bit 59 doesn't apply to MSI-X transactions (AFAICS from the PCIe controller spec we have). > > This will only work as long as all of the system memory can be > addressed at an offset from that fixed address that itself fits your > device addressing capabilities (50 bits in this case). It may or may > not be the case but there's no way to check since the DMA mask logic > won't really apply. Understood. In the specific system we are integrated to, that is the case - we have less then 48 bits. But, as you pointed out, it is not a generic solution but with my H/W I can't give a generic fit-all solution for POWER9. I'll settle for the best that I can do. > > You might want to consider fixing your HW in the next iteration... This > is going to bite you when x86 increases the max physical memory for > example, or on other architectures. Understood and taken care of. > > Cheers, > Ben. > > > >