Received: by 2002:a25:ab43:0:0:0:0:0 with SMTP id u61csp5403761ybi; Wed, 12 Jun 2019 01:50:19 -0700 (PDT) X-Google-Smtp-Source: APXvYqwr10pmbEMvMYz1co8K0UWE6bAZiUSE2tEIKY4JBZfzU9ze+qecnubFE/ucfR7vFcE2UGWu X-Received: by 2002:a17:902:24d:: with SMTP id 71mr82750557plc.166.1560329419380; Wed, 12 Jun 2019 01:50:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560329419; cv=none; d=google.com; s=arc-20160816; b=mZtHW9v70MDi9VliHiKuXR+WZ5d7ht2PCc6DXbGjzedc0oc/DColov/pntpGpn6mPd 52sG1K5oDkzXCG0JRcmYxCtbyXs8JuCGg5qa8/CP/WwjD7mjvbvoJeudZEy5QXTLxAcc A2zR+hddIobS5N58K9D7jHc+hdB65LRqW1KYH4r1lK8kOgG02B9eKgbkfN7Po7I26+U5 b+LBhfR0c0eu+36DrwJ+2Ka1LCQKBZ6MF4Q6UYpxZJxX2s3dpmeDn62WT3bvlq6Lfem7 N6pjQTv7pGbXvqk3BPCBa+n4EsrwrvLhKsf5ADL07FUegn3Cyqep+0ycVXkVvl3uxlA2 FGYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:date:cc:to:from:subject:message-id; bh=7eZxfdJBw1YMLGfm17J4A1BtIxJl4dlr/LlLWFkHEsc=; b=pAu4OM3l1VCt9rYmiJpOJIfktvUeGLarLgiIH5tKvn/SPY+Xg8UdYyONPcEMr1uJoA E66o2yItusI0eWTh8fp/j+PEvvOIEpfUWlvID59Dm3cbo851aUdwV8vz69k4oI7SnRrg N+VijtFwbwsz81JU62enJUFPLBW/9nURVM/YzmB6K11Wr45H9ZpTzGS3HOjeYiiQ7n2S Q1r+Y1YcOIqRQpvER0OdTgDg8N91TvnPVU+M9iA5zH6YkuJUM6pLjSXPAkv9Yl8djt+q hMq2GdD4RNmsJOHNNoIkYOK4xFgJRDl66E2MUIi2fUlUz66p0G9aqCSB/Y5cPtCFsed7 aarQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f34si15647925plf.258.2019.06.12.01.50.04; Wed, 12 Jun 2019 01:50:19 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2436459AbfFLITH (ORCPT + 99 others); Wed, 12 Jun 2019 04:19:07 -0400 Received: from gate.crashing.org ([63.228.1.57]:33437 "EHLO gate.crashing.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730984AbfFLITH (ORCPT ); Wed, 12 Jun 2019 04:19:07 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by gate.crashing.org (8.14.1/8.14.1) with ESMTP id x5C8Iqah008654; Wed, 12 Jun 2019 03:18:53 -0500 Message-ID: <8fa52f5f1455b9f3d81c71517ed499b93b357044.camel@kernel.crashing.org> Subject: Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9 From: Benjamin Herrenschmidt To: Oded Gabbay Cc: Greg KH , linuxppc-dev@ozlabs.org, Christoph Hellwig , "Linux-Kernel@Vger. Kernel. Org" , Oliver OHalloran , Russell Currey Date: Wed, 12 Jun 2019 18:18:52 +1000 In-Reply-To: References: <20190611092144.11194-1-oded.gabbay@gmail.com> <20190611095857.GB24058@kroah.com> <20190611151753.GA11404@infradead.org> <20190611152655.GA3972@kroah.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5-0ubuntu0.18.04.1 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 2019-06-12 at 09:25 +0300, Oded Gabbay wrote: > > > You can't. Your device is broken. Devices that don't support DMAing to > > the full 64-bit deserve to be added to the trash pile. > > > > Hmm... right know they are added to customers data-centers but what do I know ;) Well, some customers don't know they are being sold a lemon :) > > As a result, getting it to work will require hacks. Some GPUs have > > similar issues and require similar hacks, it's unfortunate. > > > > Added a couple of guys on CC who might be able to help get those hacks > > right. > > Thanks :) > > > > It's still very fishy .. the idea is to detect the case where setting a > > 64-bit mask will give your system memory mapped at a fixed high address > > (1 << 59 in our case) and program that in your chip in the "Fixed high > > bits" register that you seem to have (also make sure it doesn't affect > > MSIs or it will break them). > > MSI-X are working. The set of bit 59 doesn't apply to MSI-X > transactions (AFAICS from the PCIe controller spec we have). Ok. > > This will only work as long as all of the system memory can be > > addressed at an offset from that fixed address that itself fits your > > device addressing capabilities (50 bits in this case). It may or may > > not be the case but there's no way to check since the DMA mask logic > > won't really apply. > > Understood. In the specific system we are integrated to, that is the > case - we have less then 48 bits. But, as you pointed out, it is not a > generic solution but with my H/W I can't give a generic fit-all > solution for POWER9. I'll settle for the best that I can do. > > > > > You might want to consider fixing your HW in the next iteration... This > > is going to bite you when x86 increases the max physical memory for > > example, or on other architectures. > > Understood and taken care of. Cheers, Ben. > > > > Cheers, > > Ben. > > > > > > > >