Received: by 2002:a25:6193:0:0:0:0:0 with SMTP id v141csp1402536ybb; Wed, 1 Apr 2020 23:26:50 -0700 (PDT) X-Google-Smtp-Source: APiQypI9YIdEynqMmJTxvSL+kuJ6nPCJVkhkDguECRQZ73pvU8pAg+537zllUC/jE+QxgAg4at2B X-Received: by 2002:a9d:7747:: with SMTP id t7mr1151441otl.96.1585808810220; Wed, 01 Apr 2020 23:26:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1585808810; cv=none; d=google.com; s=arc-20160816; b=k5LCih+KuugUPTWpSan6gVpr+I2qJaN50fivc9RgRNQRAR9C3TDwJ9BeGgxcZxCtfq Fr9ENc0P12pIap1FnZjB+XmSr+76irbk/CFmKYClHgnhUOFvMInGFLrB5mVW7hQzuTAa HkXrz0p+rpajm8awR5tGunUo45ndX6q+FV2ZAqIjOiGRusH3llVej1iQJgoRHAaci1ok ft1k6f9zuiNDLwik7CG+yHtr9CrDd0nVlqBksxYT2yLxgQRyLlb2jnmYeWIGNmanC2Zt VcaX4SqU3+u1/iKbxYfYqjX80kmwWniM0vTE3bwMuUhMH0wEDUMq2vAvbN4wmmIV9kLv DNsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date:from :references:cc:to:subject; bh=GBVKq3bOnBe07sZ5Iu5cvDxMfeEuwuq7LRU7PtAkpoQ=; b=lxkNThluUSfE3/QwJXtwkPVB7euCIuP3elsHcGM0dR3IGO8U2DQ2iTrZSsZIszOcWA Dh8Bc4QaWBsZlfBAC8yGDwLTL+2wL71CB8DKMbBOVgQj3W/ZU0U0XyvtCnJA/nGkWZGE N2xFzAygaLptr1uVXPjBOnwITjJuWOBMaGNH30Z83PcOC7JsU7bTAnQgzFyCCrLd9Q9a L0hBxFEj7DuZ9YNLuUH4jMftJyH4kzHnuIjcWSmVXAdM1VVjrtvRJY+2eqjK5cPUm/fB ADz6VD9p7Ud1HrGMgJJaV+vNR08em8+1GRemGqpiG+twI0mHciwFoC5Sq2XOeKWcCsgx i3Qw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h27si1966701ots.293.2020.04.01.23.26.35; Wed, 01 Apr 2020 23:26:50 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728661AbgDBGVu (ORCPT + 99 others); Thu, 2 Apr 2020 02:21:50 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:32032 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727746AbgDBGVt (ORCPT ); Thu, 2 Apr 2020 02:21:49 -0400 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 03265FLi100851 for ; Thu, 2 Apr 2020 02:21:48 -0400 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0b-001b2d01.pphosted.com with ESMTP id 304gst2b4g-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 02 Apr 2020 02:21:48 -0400 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 2 Apr 2020 07:21:31 +0100 Received: from b06avi18878370.portsmouth.uk.ibm.com (9.149.26.194) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 2 Apr 2020 07:21:23 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 0326LbiA43254262 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Apr 2020 06:21:37 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D87214C050; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 314D74C046; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Thu, 2 Apr 2020 06:21:37 +0000 (GMT) Received: from [9.102.43.12] (unknown [9.102.43.12]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 67DCAA0130; Thu, 2 Apr 2020 17:21:30 +1100 (AEDT) Subject: Re: [PATCH v4 06/25] ocxl: Tally up the LPC memory on a link & allow it to be mapped To: Dan Williams , "Alastair D'Silva" Cc: "Aneesh Kumar K . V" , "Oliver O'Halloran" , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Frederic Barrat , Arnd Bergmann , Greg Kroah-Hartman , Vishal Verma , Dave Jiang , Ira Weiny , Andrew Morton , Mauro Carvalho Chehab , "David S. Miller" , Rob Herring , Anton Blanchard , Krzysztof Kozlowski , Mahesh Salgaonkar , Madhavan Srinivasan , =?UTF-8?Q?C=c3=a9dric_Le_Goater?= , Anju T Sudhakar , Hari Bathini , Thomas Gleixner , Greg Kurz , Nicholas Piggin , Masahiro Yamada , Alexey Kardashevskiy , Linux Kernel Mailing List , linuxppc-dev , linux-nvdimm , Linux MM References: <20200327071202.2159885-1-alastair@d-silva.org> <20200327071202.2159885-7-alastair@d-silva.org> From: Andrew Donnellan Date: Thu, 2 Apr 2020 17:21:34 +1100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 x-cbid: 20040206-4275-0000-0000-000003B8050E X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 20040206-4276-0000-0000-000038CD59BE Message-Id: <0e188ea7-1845-c9ca-a18f-4f331f31b07c@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-01_04:2020-03-31,2020-04-01 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 spamscore=0 malwarescore=0 suspectscore=0 priorityscore=1501 lowpriorityscore=0 phishscore=0 clxscore=1015 mlxlogscore=999 mlxscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004020050 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 1/4/20 7:48 pm, Dan Williams wrote: > On Sun, Mar 29, 2020 at 10:53 PM Alastair D'Silva wrote: >> >> OpenCAPI LPC memory is allocated per link, but each link supports >> multiple AFUs, and each AFU can have LPC memory assigned to it. > > Is there an OpenCAPI primer to decode these objects and their > associations that I can reference? There isn't presently a primer that I think addresses these questions nicely (to my knowledge - Fred might have something he can link to?) - there are the specs published by the OpenCAPI Consortium at https://opencapi.org but they're really for hardware implementers. We should probably expand what's currently documented in Documentation/userspace-api/accelerators/ocxl.rst generally, and this series should probably update that to include details on LPC. To explain the specific objects here: - A "link" is a point-to-point link between the host CPU, and a single OpenCAPI card. (We don't currently support cards making use of multiple links for increased bandwidth, though that is supported from a hardware point of view.) - On POWER9, each link appears as a separate PCI domain, with a single bus, and the card appears as a single device. - A device can have up to 8 functions, per PCI. - An Attached Functional Unit (AFU) is the abstraction for a particular application function. Each PCI function defines the number of AFUs it has through a set of OpenCAPI-specific DVSECs, max 64 per function. The ocxl driver handles AFU discovery. - On the host side, LPC memory is mapped by setting a single BAR for the whole link, but on the device side, LPC memory is requested on a per-AFU basis, through an AFU descriptor that is exposed through the aforementioned DVSECs. Hence the need to loop through the AFUs and get the total required LPC memory to work out the correct BAR value. -- Andrew Donnellan OzLabs, ADL Canberra ajd@linux.ibm.com IBM Australia Limited