by Arnaud POULIQUEN

[permalink] [raw]

Subject: Re: [PATCH 3/3] rpmsg: virtio_rpmsg_bus: get buffer size from config space

Hello Xiang,

Similar mechanism has been proposed by Loic 2 years ago (link to the
series here https://lkml.org/lkml/2017/3/28/349).

Did you see them? Regarding history, patches seem just on hold...

Main differences (except interesting RX/TX size split) seems that you
- don't use the virtio_config_ops->get
- define a new feature VIRTIO_RPMSG_F_NS.

Regards
Arnaud

On 1/31/19 4:41 PM, Xiang Xiao wrote:
> 512 bytes isn't always suitable for all case, let firmware
> maker decide the best value from resource table.
> enable by VIRTIO_RPMSG_F_BUFSZ feature bit.
>
> Signed-off-by: Xiang Xiao <[email protected]>
> ---
> drivers/rpmsg/virtio_rpmsg_bus.c | 50 +++++++++++++++++++++++++--------------
> include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++
> 2 files changed, 56 insertions(+), 18 deletions(-)
> create mode 100644 include/uapi/linux/virtio_rpmsg.h
>
> diff --git a/drivers/rpmsg/virtio_rpmsg_bus.c b/drivers/rpmsg/virtio_rpmsg_bus.c
> index 59c4554..049dd97 100644
> --- a/drivers/rpmsg/virtio_rpmsg_bus.c
> +++ b/drivers/rpmsg/virtio_rpmsg_bus.c
> @@ -16,6 +16,7 @@
> #include <linux/virtio.h>
> #include <linux/virtio_ids.h>
> #include <linux/virtio_config.h>
> +#include <linux/virtio_rpmsg.h>
> #include <linux/scatterlist.h>
> #include <linux/dma-mapping.h>
> #include <linux/slab.h>
> @@ -38,7 +39,8 @@
> * @sbufs: kernel address of tx buffers
> * @num_rbufs: total number of buffers for rx
> * @num_sbufs: total number of buffers for tx
> - * @buf_size: size of one rx or tx buffer
> + * @rbuf_size: size of one rx buffer
> + * @sbuf_size: size of one tx buffer
> * @last_sbuf: index of last tx buffer used
> * @rbufs_dma: dma base addr of rx buffers
> * @sbufs_dma: dma base addr of tx buffers
> @@ -61,7 +63,8 @@ struct virtproc_info {
> void *rbufs, *sbufs;
> unsigned int num_rbufs;
> unsigned int num_sbufs;
> - unsigned int buf_size;
> + unsigned int rbuf_size;
> + unsigned int sbuf_size;
> int last_sbuf;
> dma_addr_t rbufs_dma;
> dma_addr_t sbufs_dma;
> @@ -73,9 +76,6 @@ struct virtproc_info {
> struct rpmsg_endpoint *ns_ept;
> };
>
> -/* The feature bitmap for virtio rpmsg */
> -#define VIRTIO_RPMSG_F_NS 0 /* RP supports name service notifications */
> -
> /**
> * struct rpmsg_hdr - common header for all rpmsg messages
> * @src: source address
> @@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)
>
> /* either pick the next unused tx buffer */
> if (vrp->last_sbuf < vrp->num_sbufs)
> - ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++;
> + ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;
> /* or recycle a used one */
> else
> ret = virtqueue_get_buf(vrp->svq, &len);
> @@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,
> * messaging), or to improve the buffer allocator, to support
> * variable-length buffer sizes.
> */
> - if (len > vrp->buf_size - sizeof(struct rpmsg_hdr)) {
> + if (len > vrp->sbuf_size - sizeof(struct rpmsg_hdr)) {
> dev_err(dev, "message is too big (%d)\n", len);
> return -EMSGSIZE;
> }
> @@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
> * We currently use fixed-sized buffers, so trivially sanitize
> * the reported payload length.
> */
> - if (len > vrp->buf_size ||
> + if (len > vrp->rbuf_size ||
> msg->len > (len - sizeof(struct rpmsg_hdr))) {
> dev_warn(dev, "inbound msg too big: (%d, %d)\n", len, msg->len);
> return -EINVAL;
> @@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
> dev_warn(dev, "msg received with no recipient\n");
>
> /* publish the real size of the buffer */
> - rpmsg_sg_init(&sg, msg, vrp->buf_size);
> + rpmsg_sg_init(&sg, msg, vrp->rbuf_size);
>
> /* add the buffer back to the remote processor's virtqueue */
> err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, msg, GFP_KERNEL);
> @@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)
> else
> vrp->num_sbufs = MAX_RPMSG_NUM_BUFS;
>
> - vrp->buf_size = MAX_RPMSG_BUF_SIZE;
> + /* try to get buffer size from config space */
> + if (virtio_has_feature(vdev, VIRTIO_RPMSG_F_BUFSZ)) {
> + /* note: virtio_rpmsg_config is defined from remote view */
> + virtio_cread(vdev, struct virtio_rpmsg_config,
> + txbuf_size, &vrp->rbuf_size);
> + virtio_cread(vdev, struct virtio_rpmsg_config,
> + rxbuf_size, &vrp->sbuf_size);
> + }
> +
> + /* use the default if resource table doesn't provide one */
> + if (!vrp->rbuf_size)
> + vrp->rbuf_size = MAX_RPMSG_BUF_SIZE;
> + if (!vrp->sbuf_size)
> + vrp->sbuf_size = MAX_RPMSG_BUF_SIZE;
>
> /* allocate coherent memory for the buffers */
> vrp->rbufs = dma_alloc_coherent(vdev->dev.parent->parent,
> - vrp->num_rbufs * vrp->buf_size,
> + vrp->num_rbufs * vrp->rbuf_size,
> &vrp->rbufs_dma, GFP_KERNEL);
> if (!vrp->rbufs) {
> err = -ENOMEM;
> @@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
> vrp->rbufs, &vrp->rbufs_dma);
>
> vrp->sbufs = dma_alloc_coherent(vdev->dev.parent->parent,
> - vrp->num_sbufs * vrp->buf_size,
> + vrp->num_sbufs * vrp->sbuf_size,
> &vrp->sbufs_dma, GFP_KERNEL);
> if (!vrp->sbufs) {
> err = -ENOMEM;
> @@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)
> /* set up the receive buffers */
> for (i = 0; i < vrp->num_rbufs; i++) {
> struct scatterlist sg;
> - void *cpu_addr = vrp->rbufs + i * vrp->buf_size;
> + void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size;
>
> - rpmsg_sg_init(&sg, cpu_addr, vrp->buf_size);
> + rpmsg_sg_init(&sg, cpu_addr, vrp->rbuf_size);
>
> err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, cpu_addr,
> GFP_KERNEL);
> @@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev)
>
> free_sbufs:
> dma_free_coherent(vdev->dev.parent->parent,
> - vrp->num_sbufs * vrp->buf_size,
> + vrp->num_sbufs * vrp->sbuf_size,
> vrp->sbufs, vrp->sbufs_dma);
> free_rbufs:
> dma_free_coherent(vdev->dev.parent->parent,
> - vrp->num_rbufs * vrp->buf_size,
> + vrp->num_rbufs * vrp->rbuf_size,
> vrp->rbufs, vrp->rbufs_dma);
> vqs_del:
> vdev->config->del_vqs(vrp->vdev);
> @@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev)
> vdev->config->del_vqs(vrp->vdev);
>
> dma_free_coherent(vdev->dev.parent->parent,
> - vrp->num_sbufs * vrp->buf_size,
> + vrp->num_sbufs * vrp->sbuf_size,
> vrp->sbufs, vrp->sbufs_dma);
> dma_free_coherent(vdev->dev.parent->parent,
> - vrp->num_rbufs * vrp->buf_size,
> + vrp->num_rbufs * vrp->rbuf_size,
> vrp->rbufs, vrp->rbufs_dma);
>
> kfree(vrp);
> @@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {
>
> static unsigned int features[] = {
> VIRTIO_RPMSG_F_NS,
> + VIRTIO_RPMSG_F_BUFSZ,
> };
>
> static struct virtio_driver virtio_ipc_driver = {
> diff --git a/include/uapi/linux/virtio_rpmsg.h b/include/uapi/linux/virtio_rpmsg.h
> new file mode 100644
> index 0000000..24fa0dd
> --- /dev/null
> +++ b/include/uapi/linux/virtio_rpmsg.h
> @@ -0,0 +1,24 @@
> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> +/*
> + * Copyright (C) Pinecone Inc. 2019
> + * Copyright (C) Xiang Xiao <[email protected]>
> + */
> +
> +#ifndef _UAPI_LINUX_VIRTIO_RPMSG_H
> +#define _UAPI_LINUX_VIRTIO_RPMSG_H
> +
> +#include <linux/types.h>
> +
> +/* The feature bitmap for virtio rpmsg */
> +#define VIRTIO_RPMSG_F_NS 0 /* RP supports name service notifications */
> +#define VIRTIO_RPMSG_F_BUFSZ 2 /* RP get buffer size from config space */
> +
> +struct virtio_rpmsg_config {
> + /* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */
> + __u32 txbuf_size;
> + __u32 rxbuf_size;
> + __u32 reserved[14]; /* Reserve for the future use */
> + /* Put the customize config here */
> +} __attribute__((packed));
> +
> +#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */
>

2019-05-09 12:39:25

On Wed 05 Jun 00:33 PDT 2019, Arnaud Pouliquen wrote:

> Hi Bjorn,
>
> On 6/5/19 6:34 AM, Bjorn Andersson wrote:
> > On Thu 31 Jan 07:41 PST 2019, Xiang Xiao wrote:
> >
> >> Hi,
> >> This series enhance the buffer allocation by:
> >> 1.Support the different buffer number in rx/tx direction
> >> 2.Get the individual rx/tx buffer size from config space
> >>
> >> Here is the related OpenAMP change:
> >> https://github.com/OpenAMP/open-amp/pull/155
> >>
> >
> > This looks pretty reasonable, but can you confirm that it's possible to
> > use new firmware with an old Linux kernel when introducing this?
> >
> >
> > But ever since we discussed Loic's similar proposal earlier I've been
> > questioning if the fixed buffer size isn't just an artifact of how we
> > preallocate our buffers. The virtqueue seems to support arbitrary sizes
> > of buffers and I see that the receive function in OpenAMP has been fixed
> > to put back the buffer of the size that was received, rather than 512
> > bytes. So it seems like Linux would be able to send whatever size
> > messages to OpenAMP it would handle it.
> >
> > The question is if we could do the same in the other direction, perhaps
> > by letting the OpenAMP side do it's message allocation when it's
> > sending, rather than Linux pushing inbufs to be filled by the remote.
>
> IMHO, both could be useful and could be not correlated.
> On-the fly buffer allocation seems more efficient but needs an
> allocator.This can be a generic allocator (with a va to da) for system
> where large amount of memories are accessible from both side.
>
> Now what about system with small shared memory? In this case you have to
> deal with a limited/optimized memory chunk. To avoid memory
> fragmentation the allocator should have a pre-reserved buffers pool(so
> similar to existing implementation). This serie seems useful to optimize
> the size of the pre-reserved pool.
>

Having an implementation that uses small fixed size buffers seems very
reasonable and I'm in favour of making the message size configurable.

I would however prefer to have this implemented in a way where the
remote side should not be receiving messages in a way that's based on
the remote side's allocation parameters.

I don't think this series prevents the introduction of such isolation,
but it would render this code unnecessary.

Regards,
Bjorn

2023-10-24 08:10:03

by Arnaud POULIQUEN

[permalink] [raw]

Subject: RE: [PATCH 0/3] Enhance virtio rpmsg bus driver buffer allocation

Hello Divin,

This development is currently on hold, but several people seem interested in it.
I suggest that you reply directly to the email thread to continue the discussion and address the remoteproc mailing list

Regards,
Arnaud

>From: Divin Raj <[email protected]>
>Sent: Monday, October 23, 2023 12:44 PM
>To: [email protected]; Xiang Xiao <[email protected]>; [email protected]; Ohad Ben Cohen <[email protected]>; Bjorn Andersson <[email protected]>; Arnaud >POULIQUEN <[email protected]>; xiang xiao <[email protected]>
>Cc: [email protected]; [email protected]
>Subject: Re: [PATCH 0/3] Enhance virtio rpmsg bus driver buffer allocation

>Hello all,
>I am reaching out with reference to the patch discussed here: https://lore.kernel.org/all/CAH2Cfb-sv3SAL8bcczC-Dc3_r58MYZCS7s7zGtn1Qfo3mmBqVg@mail.gmail.com/
>I've been keenly following the developments around enhancing buffer allocation strategies, especially those focused on dynamic buffer sizing and the considerations for systems under varying memory >constraints. This work is highly relevant to several projects I am involved in, and I am quite interested in its progression. May I kindly request an update on the current phase of these initiatives? >Additionally, I am eager to know if there would be an opportunity for me to contribute to enhancing the patch, possibly by working on improvements or assisting in verification processes.
>Furthermore, if there are any condensed resources, summaries, or specific threads that encapsulate recent advancements or discussions on this topic, I would be grateful to receive directions to them.
>I appreciate everyone's dedicated efforts and invaluable contributions to this area of development. Looking forward to the updates.
>Regards Divin
>IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not >disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

ST Restricted