DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3C51E2080C
Subject: Re: [PATCH v1 4/4] xprtrdma: Plant XID in on-the-wire RDMA offset
 (FRWR)
To:     Jason Gunthorpe <jgg@ziepe.ca>
Cc:     "Mora, Jorge" <Jorge.Mora@netapp.com>,
        Chuck Lever <chuck.lever@oracle.com>,
        Olga Kornievskaia <aglo@umich.edu>,
        linux-rdma <linux-rdma@vger.kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
References: <6592845E-0136-4D42-8426-3E2A0BB5FFE7@oracle.com>
 <CAN-5tyEw5iRaA6mCp5DeNv1DMkFGAqtvq1PcRSiFZ6PQyJpDUA@mail.gmail.com>
 <4A94F1A9-96A4-4A2F-8617-AF0E2380D0C1@oracle.com>
 <CAN-5tyGU85PJe4Uoocw5NkT8YiYSF8wTWTgg4HycAgKpyHSfiw@mail.gmail.com>
 <9BD3F7C9-96BF-4555-A901-5E82E2CF9E28@oracle.com>
 <CAN-5tyELSi-Vtf1W3CSFzynbgVW7rcyJnfHmcb6Ehs0NEUiy4A@mail.gmail.com>
 <4EE34B64-0BEB-439A-B2A2-D77673D4CF70@oracle.com>
 <5EA42399-05C2-40D3-A5CA-7B40971AEC33@netapp.com>
 <20181119224615.GM4890@ziepe.ca>
 <d6e65bdf-6cdd-1a23-36de-71aeb57a22e0@talpey.com>
 <20181120030916.GQ4890@ziepe.ca>
From:   Tom Talpey <tom@talpey.com>
Message-ID: <8fd325fa-290a-6381-80d8-e6ac20acffe8@talpey.com>
Date:   Mon, 19 Nov 2018 22:25:37 -0500
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.9.1
MIME-Version: 1.0
In-Reply-To: <20181120030916.GQ4890@ziepe.ca>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

On 11/19/2018 10:09 PM, Jason Gunthorpe wrote:
> On Mon, Nov 19, 2018 at 09:45:24PM -0500, Tom Talpey wrote:
>> On 11/19/2018 5:46 PM, Jason Gunthorpe wrote:
>>> On Mon, Nov 19, 2018 at 09:42:56PM +0000, Mora, Jorge wrote:
>>>> Hello Chuck,
>>>>
>>>> I am confused, is it the whole purpose of RDMA is to place the data
>>>> directly into the memory location given by the virtual address or
>>>> offset? What you are saying is that this offset is not the actual
>>>> memory address and so the driver must map this offset to the actual
>>>> address?
>>>
>>> All RDMA on the wire addresses are 'virtual' and always go through
>>> translation hardware to reach a physical memory address. Even if you
>>> have a virtual address that matches your physical address there will
>>> still be translation hardware to enforce security.
>>>
>>> The direct placement refers to not copying data, not to using physical
>>> memory addresses throughout the protocol design.
>>>
>>> So it is better to use a virtual address that doesn't disclose any
>>> information about the host's configuration to the network, such as
>>> physical locations of kernel memory.
>>
>> Agreed. And, this is a very subtle but very powerful aspect of FRWR
>> registration, that the high-order bits of the "virtual address" can
>> basically be any value of the requestor's (client's) choice.
>>
>> Note, the low-order bits, up to the maximum offset used by the upper
>> layer (e.g. a few MB for NFS), must be reserved in order to allow RDMA
>> segments to iterate over the range of bytes in the region. That's only
>> 24 or so bits out of the available 64, here however.
> 
> Why does it depend on protocol?

Because the maximum payload being RDMA's is an upper layer matter. The
idea here is to provide a certain number of "invariant" bits in the
offset / virtual address. Chuck proposes reserving just 32 bits, leaving
32 bits of range, quite comfy.

> So long as the adaptor can compute VA + Offset = Physical the upper
> bits can be changed.

Absolutely, but if the offset walks into the high-order invariant bits,
they won't be invariant any longer. It would still work, but it would
defeat the wireshark trick that Chuck wants to play.

> For many adaptors that means PAGE_SIZE or huge
> page size level alignment..

I'm not sure I understand this. The FRWR hides the native page size, by
providing the page frames in the local work request. Their size isn't
known to the protocol.

Tom.