Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Mon, 19 Oct 2020 12:25:56 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Tom Lendacky <thomas.lendacky@amd.com>
CC:     <x86@kernel.org>, <linux-kernel@vger.kernel.org>,
        <linux-arch@vger.kernel.org>, <linux-mm@kvack.org>,
        <kvm@vger.kernel.org>,
        Radim =?utf-8?B?S3LEjW3DocWZ?= <rkrcmar@redhat.com>,
        Arnd Bergmann <arnd@arndb.de>,
        Matt Fleming <matt@codeblueprint.co.uk>,
        "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com>,
        Andrey Ryabinin <aryabinin@virtuozzo.com>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        "Andy Lutomirski" <luto@kernel.org>,
        "H. Peter Anvin" <hpa@zytor.com>,
        "Paolo Bonzini" <pbonzini@redhat.com>,
        Alexander Potapenko <glider@google.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Dmitry Vyukov <dvyukov@google.com>,
        Rik van Riel <riel@redhat.com>,
        Larry Woodman <lwoodman@redhat.com>,
        "Dave Young" <dyoung@redhat.com>,
        Toshimitsu Kani <toshi.kani@hpe.com>,
        "Michael S. Tsirkin" <mst@redhat.com>,
        Brijesh Singh <brijesh.singh@amd.com>
Subject: AMD SME encrpytion and PCI BAR pages to user space
Message-ID: <20201019152556.GA560082@nvidia.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from mlx.ziepe.ca (156.34.48.30) by MN2PR05CA0054.namprd05.prod.outlook.com (2603:10b6:208:236::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3499.8 via Frontend Transport; Mon, 19 Oct 2020 15:25:57 +0000
Received: from jgg by mlx with local (Exim 4.94)        (envelope-from <jgg@nvidia.com>)        id 1kUX2q-002LsA-MN; Mon, 19 Oct 2020 12:25:56 -0300
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1;
        t=1603121072; bh=EAD4YhYy4w/iIVhyD1aD9WKwwuU5PyaeU+HlrQwSGwI=;
        h=ARC-Seal:ARC-Message-Signature:ARC-Authentication-Results:Date:
         From:To:CC:Subject:Message-ID:Content-Type:Content-Disposition:
         X-ClientProxiedBy:MIME-Version:
         X-MS-Exchange-MessageSentRepresentingType;
        b=a9wjNQx/jaO6toRrCHDO33yzuLunPbL4estPTGud0blfahf3VGVP20iAccliJWmmL
         nUFtUyvw7dzs6hzTtRo8vH+7mjE75JM7GcPDWApHAojhIp6euvyKfyMpIQvjcizAJf
         zIwcuieZ+RjigrmYERMW0Ht/CQGaLx/iP/x/NiNbu07KZOHY3o7GzX2ikAMLQFDvGA
         DjYzUGn0fWRlfA7REZBc38O8dA5jq7vWq1O+kDtl9fpUYNEpnvHK7rvrbLDLSBN+1S
         rXdRg5gc+P0tFs/RhKfFbMDYlpg8Z8NcYWuEzdofha9d4iyZyAt8Ft3qIVwoJdjVhy
         SGgGdYyjHoh3A==
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Tom,

We've found a bug where systems that have the AMD SME turned on are
not able to run RDMA work loads. It seems the kernel is automatically
encrypting VMA's pointing at PCI BAR memory created by
io_remap_pfn_range() - adding a prot_decrypted() causes things to
start working.

To me this is surprising, before I go adding random prot_decrypted()
into the RDMA subsystem can you confirm this is actually how things
are expected to work?

Is RDMA missing something? I don't see anything special in VFIO for
instance and the two are very similar - does VFIO work with SME, eg
DPDK or something unrelated to virtualization?
     
Is there a reason not to just add prot_decrypted() to
io_remap_pfn_range()? Is there use cases where a caller actually wants
encrypted io memory?

I saw your original patch series edited a few drivers this way, but
not nearly enough. So I feel like I'm missing something.. Does vfio
work with SME? I couldn't find any sign of it calling prot_decrypted()
either?

(BTW, I don't have any AMD SME systems to test on here, I'm getting
 this bug report from deployed system, running a distro kernel)

Thanks,
Jason