Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
References: <20191230102942.18395-1-jinpuwang@gmail.com> <20191230102942.18395-15-jinpuwang@gmail.com>
 <1ad8b279-1a45-1d70-39c7-acd42f28abca@acm.org>
In-Reply-To: <1ad8b279-1a45-1d70-39c7-acd42f28abca@acm.org>
From:   Jinpu Wang <jinpu.wang@cloud.ionos.com>
Date:   Tue, 7 Jan 2020 15:48:57 +0100
Message-ID: <CAMGffEnzoo7Y-Bh4F7ZDONr1kE3U20Btr=763rQyiYu=+YMosA@mail.gmail.com>
Subject: Re: [PATCH v6 14/25] rtrs: a bit of documentation
To:     Bart Van Assche <bvanassche@acm.org>
Cc:     Jack Wang <jinpuwang@gmail.com>, linux-block@vger.kernel.org,
        linux-rdma@vger.kernel.org, Jens Axboe <axboe@kernel.dk>,
        Christoph Hellwig <hch@infradead.org>,
        Sagi Grimberg <sagi@grimberg.me>,
        Leon Romanovsky <leon@kernel.org>,
        Doug Ledford <dledford@redhat.com>,
        Danil Kipnis <danil.kipnis@cloud.ionos.com>, rpenyaev@suse.de,
        linux-kernel@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Dec 31, 2019 at 12:19 AM Bart Van Assche <bvanassche@acm.org> wrote:
>
> On 2019-12-30 02:29, Jack Wang wrote:
> > diff --git a/drivers/infiniband/ulp/rtrs/README b/drivers/infiniband/ulp/rtrs/README
>
> Other kernel driver documentation exists under the Documentation/
> directory. Should this README file perhaps be moved to a subdirectory of
> the Documentation/ directory?
I did check most of the drivers are in the drivers directory eg:
find ./ -name README
./fs/reiserfs/README
./fs/qnx4/README
./fs/qnx6/README
./fs/cramfs/README
./Documentation/ABI/README
./Documentation/virt/kvm/devices/README
./README
./tools/usb/usbip/README
./tools/virtio/ringtest/README
./tools/virtio/virtio-trace/README
./tools/power/pm-graph/README
./tools/power/cpupower/README
./tools/memory-model/README
./tools/memory-model/scripts/README
./tools/memory-model/litmus-tests/README
./tools/testing/vsock/README
./tools/testing/ktest/examples/README
./tools/testing/selftests/ftrace/README
./tools/testing/selftests/arm64/signal/README
./tools/testing/selftests/arm64/README
./tools/testing/selftests/android/ion/README
./tools/testing/selftests/zram/README
./tools/testing/selftests/livepatch/README
./tools/testing/selftests/net/forwarding/README
./tools/testing/selftests/futex/README
./tools/testing/selftests/tc-testing/README
./tools/thermal/tmon/README
./tools/build/tests/ex/empty2/README
./tools/perf/tests/attr/README
./tools/perf/pmu-events/README
./tools/perf/scripts/perl/Perf-Trace-Util/README
./tools/io_uring/README
./net/decnet/README
./scripts/ksymoops/README
./scripts/selinux/README
./arch/powerpc/boot/README
./arch/m68k/q40/README
./arch/m68k/ifpsp060/README
./arch/m68k/fpsp040/README
./arch/parisc/math-emu/README
./arch/x86/math-emu/README
./drivers/bcma/README
./drivers/char/mwave/README
./drivers/staging/nvec/README
./drivers/staging/wlan-ng/README
./drivers/staging/axis-fifo/README
./drivers/staging/fbtft/README
./drivers/staging/fsl-dpaa2/ethsw/README
./drivers/staging/goldfish/README
./drivers/staging/gs_fpgaboot/README
./drivers/staging/comedi/drivers/ni_routing/README
./drivers/net/wireless/marvell/mwifiex/README
./drivers/net/wireless/marvell/libertas/README

>
> > +****************************
> > +InfiniBand Transport (RTRS)
> > +****************************
>
> The abbreviation does not match the full title. Do you agree that this
> is confusing?
>
> > +RTRS is used by the RNBD (Infiniband Network Block Device) modules.
>
> Is RNBD an RDMA or an InfiniBand network block device?
will fix.
>
> > +
> > +==================
> > +Transport protocol
> > +==================
> > +
> > +Overview
> > +--------
> > +An established connection between a client and a server is called rtrs
> > +session. A session is associated with a set of memory chunks reserved on the
> > +server side for a given client for rdma transfer. A session
> > +consists of multiple paths, each representing a separate physical link
> > +between client and server. Those are used for load balancing and failover.
> > +Each path consists of as many connections (QPs) as there are cpus on
> > +the client.
> > +
> > +When processing an incoming rdma write or read request rtrs client uses memory
>
> A quote from
> https://linuxplumbersconf.org/event/4/contributions/367/attachments/331/555/LPC_2019_RMDA_MC_IBNBD_IBTRS_Upstreaming.pdf:
> "Only RDMA writes with immediate". Has the wire protocol perhaps been
> changed such that both RDMA reads and writes are used? I haven't found
> any references to RDMA reads in the "IO path" section in this file. Did
> I perhaps overlook something?
>
> Thanks,
>
> Bart.
We do not use RDMA_READ, only RDMA_WRITE/RDMA_WRITE_WITH_IMM/SEND_WITH_IMM
SEND_WITH_IMM was used only when always_invalidate=Y.
Will extend the document.

Thanks Bart.