DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2D05220659
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\))
Subject: Re: [PATCH v1 4/4] xprtrdma: Plant XID in on-the-wire RDMA offset
 (FRWR)
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <CAN-5tyEw5iRaA6mCp5DeNv1DMkFGAqtvq1PcRSiFZ6PQyJpDUA@mail.gmail.com>
Date:   Mon, 19 Nov 2018 13:18:47 -0500
Cc:     linux-rdma <linux-rdma@vger.kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <4A94F1A9-96A4-4A2F-8617-AF0E2380D0C1@oracle.com>
References: <20181119153707.10832.42881.stgit@manet.1015granger.net>
 <20181119154607.10832.92558.stgit@manet.1015granger.net>
 <CAN-5tyG26W+TjTsjpTVvF3RJOGCnD8L79dCbRqKTm5F1V3fuxw@mail.gmail.com>
 <6592845E-0136-4D42-8426-3E2A0BB5FFE7@oracle.com>
 <CAN-5tyEw5iRaA6mCp5DeNv1DMkFGAqtvq1PcRSiFZ6PQyJpDUA@mail.gmail.com>
To:     Olga Kornievskaia <aglo@umich.edu>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk


> On Nov 19, 2018, at 1:08 PM, Olga Kornievskaia <aglo@umich.edu> wrote:
>=20
> On Mon, Nov 19, 2018 at 12:59 PM Chuck Lever <chuck.lever@oracle.com> =
wrote:
>>=20
>>=20
>>=20
>>> On Nov 19, 2018, at 12:47 PM, Olga Kornievskaia <aglo@umich.edu> =
wrote:
>>>=20
>>> On Mon, Nov 19, 2018 at 10:46 AM Chuck Lever =
<chuck.lever@oracle.com> wrote:
>>>>=20
>>>> Place the associated RPC transaction's XID in the upper 32 bits of
>>>> each RDMA segment's rdma_offset field. These bits are currently
>>>> always zero.
>>>>=20
>>>> There are two reasons to do this:
>>>>=20
>>>> - The R_key only has 8 bits that are different from registration to
>>>> registration. The XID adds more uniqueness to each RDMA segment to
>>>> reduce the likelihood of a software bug on the server reading from
>>>> or writing into memory it's not supposed to.
>>>>=20
>>>> - On-the-wire RDMA Read and Write operations do not otherwise carry
>>>> any identifier that matches them up to an RPC. The XID in the
>>>> upper 32 bits will act as an eye-catcher in network captures.
>>>=20
>>> Is this just an "eye-catcher" or do you have plans to use it in
>>> wireshark? If the latter, then can we really do that? while a linux
>>> implementation may do that, other (or even possibly future linux)
>>> implementation might not do this. Can we justify changing the
>>> wireshark logic for it?
>>=20
>> No plans to change the wireshark RPC-over-RDMA dissector.
>> That would only be a valid thing to do if adding the XID
>> were made part of the RPC-over-RDMA protocol via an RFC.
>=20
> Agreed. Can you also help me understand the proposal (as I'm still
> trying to figure why it is useful).
>=20
> You are proposing to modify the RDMA segments's RDMA offset field (I
> see top 6bits are indeed always 0). I don't see how adding that helps
> an RDMA read/write message which does not have an "offset" field in it
> be matched to a particular RPC. I don't believe we have (had) any
> issues matching the initial RC Send only that contains the RDMA_MSG to
> the RPC.

The ULP has access to only the low order 8 bits of the R_key. The
upper 24 bits are fixed for each MR. So for any given MR, there are
only 256 unique R_key values. That means the same R_key will appear
again quickly on the wire.

The 64-bit offset field is set by the ULP, and can be essentially
any arbitrary value. Most kernel ULPs use the iova of the registered
memory. We only need the lower 32 bits for that.

The purpose of adding junk to the offset is to make the offset
unique to that RPC transaction, just like the R_key is. This helps
make the RDMA segment co-ordinates (handle, length, offset) more
unique and thus harder to spoof.

We could use random numbers in that upper 32 bits, but we have
something more handy: the RPC's XID.

Now when you look at an RDMA Read or Write, the top 32 bits in each
RDMA segment's offset match the XID of the RPC transaction that the
RDMA operations go with. This is really a secondary benefit to the
uniquifying effect above.


--
Chuck Lever