Received: by 10.223.185.116 with SMTP id b49csp5255266wrg; Tue, 27 Feb 2018 10:13:26 -0800 (PST) X-Google-Smtp-Source: AG47ELsglFQ11HdDNZT2XS65gzpKQlB2oVD1v624SDNXxUUBs9T33od+6ktQFp5Xmsf+69V9h5L6 X-Received: by 2002:a17:902:1a2:: with SMTP id b31-v6mr6816976plb.203.1519755206271; Tue, 27 Feb 2018 10:13:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1519755206; cv=none; d=google.com; s=arc-20160816; b=TfAl/teRwuPhp/Wrj9PF2a5nxpdCEylXzzqdnMu+XwJ8NnYuVFBkL6r5kiN/7slfIm LJh/oqPao8OOSxitpoaZzxbmKbak/hVLRWltt8AGZBN0UShrtFnukaPmyWd0z/Hp0KSz JmtbTsTZktvShzncuCUwG8TS+OyaEydc/YxMxBSqigH8xgztv4/FPHR7hNm+IJal7vNW xjV+0mmo4j9zoRzAWMPIdtvfunhgBprGkX1tq/hIjgKoLLgVk+tzvSaJiPdCGskowSzD pVNjwP/s5YWd5Gvd/lZWOdXsjXHxC7I/sy5QhYe/h3yaJ5WPEEZcqs6T5qSyRsN70Jw0 yV2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:content-language :content-transfer-encoding:in-reply-to:mime-version:user-agent:date :from:references:cc:to:subject:arc-authentication-results; bh=pvsAYzpptjvF/IAI4IQpBCXaQgrHcJV+LGzIB+7JHoc=; b=HxFxa7BpfHq9kp6eheTpZH2GlAez9vdFwwFU6l8fdNYe6OrqsI5HIiUPl+sqJJR8f4 pqlk+uSbzfrWI9bFPdxxZfFZivDKbrGOljrixuKeycqpD81Lg78UuNeF7ouRM+k8qCQR 8eAXsvBgABiyR6v1UvQ5ASrgmB3yvT3t8seziNr4d2Fy3ytlkYaYS0a0JzdgSKVm2J0x +ck43JSZX1yw8s8zLtVPM+QRZtbxuv5+aQ12bmg7vuOjtWIfqvy4zbF3sm179pPXNVkD oTr3WDzQHWhz6V8ExKzEt8p0e3lIFCdxsF3TLs9BB3FGVgtsrkLbphrEDgdMzoknEXyc wQNA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a18-v6si712921plm.773.2018.02.27.10.13.11; Tue, 27 Feb 2018 10:13:26 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751587AbeB0SMc (ORCPT + 99 others); Tue, 27 Feb 2018 13:12:32 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:50204 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751126AbeB0SMa (ORCPT ); Tue, 27 Feb 2018 13:12:30 -0500 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w1RICBfP064461 for ; Tue, 27 Feb 2018 13:12:30 -0500 Received: from e32.co.us.ibm.com (e32.co.us.ibm.com [32.97.110.150]) by mx0a-001b2d01.pphosted.com with ESMTP id 2gdcccg3ff-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Tue, 27 Feb 2018 13:12:29 -0500 Received: from localhost by e32.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 27 Feb 2018 11:12:28 -0700 Received: from b03cxnp08028.gho.boulder.ibm.com (9.17.130.20) by e32.co.us.ibm.com (192.168.1.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 27 Feb 2018 11:12:26 -0700 Received: from b03ledav005.gho.boulder.ibm.com (b03ledav005.gho.boulder.ibm.com [9.17.130.236]) by b03cxnp08028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w1RICOhA8913238; Tue, 27 Feb 2018 11:12:24 -0700 Received: from b03ledav005.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 87D25BE044; Tue, 27 Feb 2018 11:12:24 -0700 (MST) Received: from oc8043147753.ibm.com (unknown [9.60.75.238]) by b03ledav005.gho.boulder.ibm.com (Postfix) with ESMTP id 5CE85BE039; Tue, 27 Feb 2018 11:12:22 -0700 (MST) Subject: Re: [PATCH v2 15/15] s390: doc: detailed specifications for AP virtualization To: Cornelia Huck Cc: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, freude@de.ibm.com, schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com, borntraeger@de.ibm.com, kwankhede@nvidia.com, bjsdjshi@linux.vnet.ibm.com, pbonzini@redhat.com, alex.williamson@redhat.com, pmorel@linux.vnet.ibm.com, alifm@linux.vnet.ibm.com, mjrosato@linux.vnet.ibm.com, jjherne@linux.vnet.ibm.com, thuth@redhat.com, pasic@linux.vnet.ibm.com, fiuczy@linux.vnet.ibm.com, buendgen@de.ibm.com References: <1519741693-17440-1-git-send-email-akrowiak@linux.vnet.ibm.com> <1519741693-17440-16-git-send-email-akrowiak@linux.vnet.ibm.com> <20180227165702.67b171db.cohuck@redhat.com> From: Tony Krowiak Date: Tue, 27 Feb 2018 13:12:21 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: <20180227165702.67b171db.cohuck@redhat.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-TM-AS-GCONF: 00 x-cbid: 18022718-0004-0000-0000-000013BA17F7 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00008601; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000254; SDB=6.00995936; UDB=6.00506275; IPR=6.00775260; MB=3.00019764; MTD=3.00000008; XFM=3.00000015; UTC=2018-02-27 18:12:28 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18022718-0005-0000-0000-000086431E01 Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-02-27_09:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1802270224 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 02/27/2018 10:57 AM, Cornelia Huck wrote: > On Tue, 27 Feb 2018 09:28:13 -0500 > Tony Krowiak wrote: > >> This patch provides documentation describing the AP architecture and >> design concepts behind the virtualization of AP devices. It also >> includes an example of how to configure AP devices for exclusive >> use of KVM guests. >> >> Signed-off-by: Tony Krowiak >> --- >> Documentation/s390/vfio-ap.txt | 514 ++++++++++++++++++++++++++++++++++++++++ >> MAINTAINERS | 1 + >> 2 files changed, 515 insertions(+), 0 deletions(-) >> create mode 100644 Documentation/s390/vfio-ap.txt >> >> diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt >> new file mode 100644 >> index 0000000..c599f30 >> --- /dev/null >> +++ b/Documentation/s390/vfio-ap.txt >> @@ -0,0 +1,514 @@ >> +Introduction: >> +============ >> +The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised >> +of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards. >> +The AP devices provide cryptographic functions to all CPUs assigned to a >> +linux system running in an IBM Z system LPAR. >> + >> +The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap >> +is to make AP cards available to KVM guests using the VFIO mediated device >> +framework. > Maybe drop a sentence in here that this makes heavy usage of the s390 > virtualization facilities, which do the heavy lifting? Will do. > >> + >> +AP Architectural Overview: >> +========================= >> +To facilitate the comprehension of the design, let's start with some >> +definitions: >> + >> +* AP adapter >> + >> + An AP adapter is an IBM Z adapter card that can perform cryptographic >> + functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters >> + assigned to the LPAR in which a linux host is running will be available to >> + the linux host. Each adapter is identified by a number from 0 to 255. When >> + installed, an AP adapter is accessed by AP instructions executed by any CPU. >> + >> +* AP domain >> + >> + An adapter is partitioned into domains. Each domain can be thought of as >> + a set of hardware registers for processing AP instructions. An adapter can >> + hold up to 256 domains. Each domain is identified by a number from 0 to 255. >> + Domains can be further classified into two types: >> + >> + * Usage domains are domains that can be accessed directly to process AP >> + commands >> + >> + * Control domains are domains that are accessed indirectly by AP >> + commands sent to a usage domain to control or change the domain, for >> + example; to set a secure private key for the domain. >> + >> +* AP Queue >> + >> + An AP queue is the means by which an AP command-request message is sent to an >> + AP usage domain inside a specific AP. An AP queue is identified by a tuple >> + comprised of an AP adapter ID (APID) and an AP queue index (APQI). The >> + APQI corresponds to a given usage domain number within the adapter. This tuple >> + forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP >> + instructions include a field containing the APQN to identify the AP queue to >> + which the AP command-request message is to be sent for processing. >> + >> +* AP Instructions: >> + >> + There are three AP instructions: >> + >> + * NQAP: to enqueue an AP command-request message to a queue >> + * DQAP: to dequeue an AP command-reply message from a queue >> + * PQAP: to administer the queues > Do you also want to explain how these entities show up on the ap bus in > Linux? It might make the explanations further down easier to > understand. (Is there any document for the ap bus you could point to?) I am not aware of any documentation for the AP bus, but I can provide a few sentences to explain how AP devices are managed by the AP bus. > >> + >> +Start Interpretive Execution (SIE) Instruction: >> +============================================== > Call this "AP and SIE" or so? You're not trying to explain the whole > SIE architecture :) Okay > > >> +Let's now see how AP instructions are interpreted by the hardware. > (...) > >> +Reserve APQNs for exclusive use of KVM guests >> +--------------------------------------------- >> +The following block diagram illustrates the mechanism by which APQNs are >> +reserved: >> + >> + +------------------+ >> + remove | | unbind >> + +------------------->+ cex4queue driver +<-----------+ >> + | | | | >> + | +------------------+ | >> + | | >> + | | >> + | | >> ++--------+---------+ register +------------------+ +-----+------+ >> +| +<---------+ | bind | | >> +| ap_bus | | vfio_ap driver +<-----+ admin | >> +| +--------->+ | | | >> ++------------------+ probe +---+--------+-----+ +------------+ >> + | | >> + create | | store APQN >> + | | >> + v v >> + +---+--------+-----+ >> + | | >> + | matrix device | >> + | | >> + +------------------+ >> + > Thank you for including diagrams, these are really helpful. You are welcomed > > (...) > >> +Initialize the CPU model feature for AP >> +--------------------------------------- >> +This design exploits a feature of the SIE architecture called interpretive >> +execution (IE). When IE is enabled for a KVM guest, the AP instructions >> +executed in the guest will be interpreted by the firmware and the commands >> +contained therein will be passed directly through to an AP device assigned to >> +the linux host. In order to enable interpretive execution for a KVM guest, SIE >> +must have access to the AP facilities installed on the linux host. A new CPU >> +model feature is introduced by this design to indicate that the guest will >> +directly access the host AP facilities. This feature will be enabled by the >> +kernel only if the AP facilities are installed on the linux host. This feature >> +must be turned turned on for the guest in order to ac: >> + >> + /usr/bin/qemu-system-s390x ... -cpu xxx,ap=on >> + >> + Where xxx is the CPU model being used. >> + >> +If the CPU model feature is not enabled by the kernel, QEMU >> +will fail and report that the feature is not supported. > The cpu model interface is supposed to be user space agnostic, although > it is only used by QEMU in practice. Mark this as an example more > clearly? Will do > > (...) > > I have not looked at this in detail (will probably come back to this > later), but this looks like a useful document. Excellent, I'm glad to hear that. It took some time to put together. >