Received: by 2002:a05:6a10:f347:0:0:0:0 with SMTP id d7csp563553pxu; Fri, 4 Dec 2020 09:47:18 -0800 (PST) X-Google-Smtp-Source: ABdhPJxDKfSl6Es9WhnRINorc6SNrPpAP9wqjc8XV/KpkvpyjoR81AO4tJUj37/x96qR0tmY6vFY X-Received: by 2002:a17:906:43c5:: with SMTP id j5mr8447172ejn.474.1607104037761; Fri, 04 Dec 2020 09:47:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1607104037; cv=none; d=google.com; s=arc-20160816; b=G+QOQmPcBWCwSFUzArgac8/vJgcDWiABc1eNfLEGRE9ecKw5f/WBUzwGqi4E2MdNxV bjwHqu+KIaSt8oIKRochNttrVPEqXqY7q3Omk6ZpyeCpFx8n28GRq1ZP/jY/rhblqUsU PdQ87hHmLejBBOvegT4cOd0MvYx7FSkDwkSQJ0Cfwdzj+YjoWGiv8/yhnE+t822Uuji7 qY9lWE+cQkEElD2yvWOdviJI4wTaRIhHags86t1SG92ceOCFk4H0TAqQc+liqygvnED0 bGixMYrCqs9wgDBCkpyoM/X8WtHns5wsJ5YyLGY1p2Rw0zquBL+jUBnWJZ6CynLyWegk tCBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :in-reply-to:references:thread-topic:message-id:cc:to:from:subject :date:user-agent; bh=yuS2Mko+ZnPMuHk9VTAM93nMFmS914qPe1aI6/A5EcA=; b=IOxgRXNW4hRfXkUMQmuHmUVcvgPSbGdnhvBiPPmXxdt0RebaQw5l+K95XURVHY7GBP zAtVOP1X/yj3sfSWuX4qoyLgn2ye0RIj+Wr9Rdh3d4J2GqSYGwxYhqGrA3Hm8cgRelyJ PoEKvIXKbs17tQzAd4N96DsBWC0Z6oZsq0dEI/JT67+L5HHnxknsEdXbvfq0o4wUDvGv anqH3ugaISAcoJA1+yqiQpIX6uV4ttbo4Li7cgXbkqsZNN+G0xFBzZjInjk6xK4tb+jj 8aOkx5BddDt4uJWBqVBd56mdNcYhDOYodWuR8ue0rMz/NWdvQEsKZQnZ4LHjHA+o7GiP KCfw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id l4si3237678edt.49.2020.12.04.09.46.53; Fri, 04 Dec 2020 09:47:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731411AbgLDRmw convert rfc822-to-8bit (ORCPT + 99 others); Fri, 4 Dec 2020 12:42:52 -0500 Received: from server.avery-design.com ([198.57.169.184]:58476 "EHLO server.avery-design.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729645AbgLDRmv (ORCPT ); Fri, 4 Dec 2020 12:42:51 -0500 Received: from ool-944ab965.dyn.optonline.net ([148.74.185.101]:52265 helo=[192.168.1.180]) by server.avery-design.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1klF1E-0006B3-Jn; Fri, 04 Dec 2020 17:37:20 +0000 User-Agent: Microsoft-MacOutlook/16.43.20110804 Date: Fri, 04 Dec 2020 12:40:03 -0500 Subject: Re: [RFC PATCH 0/9] CXL 2.0 Support From: Chris Browy To: Ben Widawsky CC: , , , , , , , , , Message-ID: Thread-Topic: [RFC PATCH 0/9] CXL 2.0 Support References: In-Reply-To: Mime-version: 1.0 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: 8BIT X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server.avery-design.com X-AntiAbuse: Original Domain - vger.kernel.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - avery-design.com X-Get-Message-Sender-Via: server.avery-design.com: authenticated_id: cbrowy@avery-design.com X-Authenticated-Sender: server.avery-design.com: cbrowy@avery-design.com X-Source: X-Source-Args: X-Source-Dir: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Ben, Trying to bring up the environment using the latest developments as follows: 1. Linux kernel baseline version is cloned using      git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git    Using master branch.  Merged the 9 CXL linux kernel patches manually and built kernel 2. QEMU baseline version is cloned using      git clone https://gitlab.com/bwidawsk/qemu.git 3. UEFI baseline is cloned using      git clone https://github.com/tianocore/edk2.git    Using master and built 4. Now can run qemu as follows:      The qcow2 we use is based on Ubuntu 20.10 with updated with kernel from 1) above      QEMU command:      sudo qemu-system-x86_64 -nic \      user,hostfwd=tcp::2222-:22,hostfwd=tcp::1234-:1234 -machine \      type=pc-q35-4.0,hmat=on,accel=kvm -enable-kvm -cpu host -smp \      6,cores=6,threads=1,sockets=1 -m 8G -boot order=d -k 'en-us' -vga virtio \      -drive file=/home/chris/Downloads/AQCXL/ubuntu_20.qcow,format=qcow2 -drive \      if=pflash,format=raw,readonly,file=/home/chris/OVMF_CODE.fd \      -drive if=pflash,format=raw,file=/home/chris/OVMF_VARS.fd \      -object memory-backend-file,id=cxl-mem1,share,mem-path=/tmp/cxl-test/cxl,size=512M \      -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0,len-window-base=1,\      window-base[0]=0x4c0000000,memdev[0]=cxl-mem1 \      -device cxl-rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0  \      -device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M  2>&1 | tee -a \      /home/chris/Downloads/AQCXL/log/qemu.log    The qemu options are derived from looking at the tests/qtests/cxl-test.c    along with the -hmat=on which seemed to make sense.    The system boots and lspci -vvv shows the CXL device is enumerated.  But    no DOE capability register for CDAT access though (see below).  Otherwise the    DVSEC registers are present.    acpidump indicates the CXL0 and CXLM devices but no SRAT or HMAT tables are    in the dump which is curious. 35:00.0 Memory controller [0502]: Intel Corporation Device 0d93 (rev 01) (prog-if 10)     Subsystem: Red Hat, Inc. Device 1100     Physical Slot: 0     Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-     Capabilities: [138 v1] Designated Vendor-Specific     Kernel driver in use: cxl_mem Questions/Comments: ------------------- 1. Linux   a. Is there a gitlab for the linux kernel patches for CXL?  This would      facilitate review and code modifications. 2. UEFI (edk2 from tianocore)   a. seems to only support CXL 1.1 which means only method #1 (Device      option ROM) of Coherent Device Attribute Table_1.02 spec      for CDAT handling is possible now.      Does device option ROM need to be added to QEMU CXL setup?      Can we add a CXL 1.1 emulated device?   b. lspci doesn’t show the existence of the DOE extended capability register      in the CXL CT3D (needed to support method #2).  Are there more patches? 3. Do you have example user programs to share or better yet the CXL 2.0    Sec 14.3.6.1 Application Layer/ Transaction layer test for CXL.mem? 4. What are the userspace system APIs for targeting CXL HDM address domain?    Usually you can mmap a SPA if you know how to look it up. Best Regards, Chris Browy