Here is the reconnect support in vduse
Kernel will allocate pages for reconnection.
Userspace needs to use mmap to map the memory to userspace and use these pages to
save the reconnect information.
test passd in vduse+dpdk-testpmd
change in V2
1. Address the comments from v1
2. Add the document for reconnect process
change in V3
1. Move the vdpa_vq_state to the uAPI. vduse will use this to synchronize the vq info between the kernel and userspace app.
2. Add a new ioctl VDUSE_DEV_GET_CONFIG. userspace app use this to get config space
3. Rewrite the commit message.
4. Only save the address for the page address and remove the index.
5. remove the ioctl VDUSE_GET_RECONNECT_INFO, userspace app will use uAPI VDUSE_RECONNCT_MMAP_SIZE to mmap
6. Rewrite the document for the reconnect process to make it clearer.
change in v4
1. Change the number of map pages to VQ numbers. UserSpace APP can define and maintain the structure for saving reconnection information in userspace. The kernel will not maintain this information.
2. Rewrite the document for the reconnect process to make it clearer.
3. add the new ioctl for VDUSE_DEV_GET_CONFIG/VDUSE_DEV_GET_STATUS
change in V5
1. update the Documentation for vduse reconnection
2. merge the patch from Dan Carpenter <[email protected]> vduse: Fix off by one in vduse_dev_mmap()
3. fix the wrong comment in the code
Signed-off-by: Cindy Lu <[email protected]>
Cindy Lu (5):
vduse: Add new ioctl VDUSE_DEV_GET_CONFIG
vduse: Add new ioctl VDUSE_DEV_GET_STATUS
vduse: Add function to get/free the pages for reconnection
vduse: Add file operation for mmap
Documentation: Add reconnect process for VDUSE
Documentation/userspace-api/vduse.rst | 41 +++++++++
drivers/vdpa/vdpa_user/vduse_dev.c | 125 ++++++++++++++++++++++++++
include/uapi/linux/vduse.h | 5 ++
3 files changed, 171 insertions(+)
--
2.43.0
The ioctl VDUSE_DEV_GET_STATUS is used by the Userspace App
to get the device status
Signed-off-by: Cindy Lu <[email protected]>
---
drivers/vdpa/vdpa_user/vduse_dev.c | 7 +++++++
include/uapi/linux/vduse.h | 2 ++
2 files changed, 9 insertions(+)
diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
index ab246da27616..ef3c9681941e 100644
--- a/drivers/vdpa/vdpa_user/vduse_dev.c
+++ b/drivers/vdpa/vdpa_user/vduse_dev.c
@@ -1389,6 +1389,13 @@ static long vduse_dev_ioctl(struct file *file, unsigned int cmd,
ret = 0;
break;
}
+
+ case VDUSE_DEV_GET_STATUS:
+ /*
+ * Returns the status read from device
+ */
+ ret = put_user(dev->status, (u8 __user *)argp);
+ break;
default:
ret = -ENOIOCTLCMD;
break;
diff --git a/include/uapi/linux/vduse.h b/include/uapi/linux/vduse.h
index 125d7529d91b..f501173a9d69 100644
--- a/include/uapi/linux/vduse.h
+++ b/include/uapi/linux/vduse.h
@@ -353,4 +353,6 @@ struct vduse_dev_response {
/* get device configuration space */
#define VDUSE_DEV_GET_CONFIG _IOR(VDUSE_BASE, 0x1b, struct vduse_config_data)
+#define VDUSE_DEV_GET_STATUS _IOR(VDUSE_BASE, 0x1c, __u8)
+
#endif /* _UAPI_VDUSE_H_ */
--
2.43.0
Add a document explaining the reconnect process, including what the
Userspace App needs to do and how it works with the kernel.
Signed-off-by: Cindy Lu <[email protected]>
---
Documentation/userspace-api/vduse.rst | 41 +++++++++++++++++++++++++++
1 file changed, 41 insertions(+)
diff --git a/Documentation/userspace-api/vduse.rst b/Documentation/userspace-api/vduse.rst
index bdb880e01132..7faa83462e78 100644
--- a/Documentation/userspace-api/vduse.rst
+++ b/Documentation/userspace-api/vduse.rst
@@ -231,3 +231,44 @@ able to start the dataplane processing as follows:
after the used ring is filled.
For more details on the uAPI, please see include/uapi/linux/vduse.h.
+
+HOW VDUSE devices reconnection works
+------------------------------------
+1. What is reconnection?
+
+ When the userspace application loads, it should establish a connection
+ to the vduse kernel device. Sometimes,the userspace application exists,
+ and we want to support its restart and connect to the kernel device again
+
+2. How can I support reconnection in a userspace application?
+
+2.1 During initialization, the userspace application should first verify the
+ existence of the device "/dev/vduse/vduse_name".
+ If it doesn't exist, it means this is the first-time for connection. goto step 2.2
+ If it exists, it means this is a reconnection, and we should goto step 2.3
+
+2.2 Create a new VDUSE instance with ioctl(VDUSE_CREATE_DEV) on
+ /dev/vduse/control.
+ When ioctl(VDUSE_CREATE_DEV) is called, kernel allocates memory for
+ the reconnect information. The total memory size is PAGE_SIZE*vq_mumber.
+
+2.3 Check if the information is suitable for reconnect
+ If this is reconnection :
+ Before attempting to reconnect, The userspace application needs to use the
+ ioctl(VDUSE_DEV_GET_CONFIG, VDUSE_DEV_GET_STATUS, VDUSE_DEV_GET_FEATURES...)
+ to get the information from kernel.
+ Please review the information and confirm if it is suitable to reconnect.
+
+2.4 Userspace application needs to mmap the memory to userspace
+ The userspace application requires mapping one page for every vq. These pages
+ should be used to save vq-related information during system running. Additionally,
+ the application must define its own structure to store information for reconnection.
+
+2.5 Completed the initialization and running the application.
+ While the application is running, it is important to store relevant information
+ about reconnections in mapped pages. When calling the ioctl VDUSE_VQ_GET_INFO to
+ get vq information, it's necessary to check whether it's a reconnection. If it is
+ a reconnection, the vq-related information must be get from the mapped pages.
+
+2.6 When the Userspace application exits, it is necessary to unmap all the
+ pages for reconnection
--
2.43.0
On Fri, Apr 12, 2024 at 9:31 PM Cindy Lu <[email protected]> wrote:
>
> Add a document explaining the reconnect process, including what the
> Userspace App needs to do and how it works with the kernel.
>
> Signed-off-by: Cindy Lu <[email protected]>
> ---
> Documentation/userspace-api/vduse.rst | 41 +++++++++++++++++++++++++++
> 1 file changed, 41 insertions(+)
>
> diff --git a/Documentation/userspace-api/vduse.rst b/Documentation/userspace-api/vduse.rst
> index bdb880e01132..7faa83462e78 100644
> --- a/Documentation/userspace-api/vduse.rst
> +++ b/Documentation/userspace-api/vduse.rst
> @@ -231,3 +231,44 @@ able to start the dataplane processing as follows:
> after the used ring is filled.
>
> For more details on the uAPI, please see include/uapi/linux/vduse.h.
> +
> +HOW VDUSE devices reconnection works
> +------------------------------------
> +1. What is reconnection?
> +
> + When the userspace application loads, it should establish a connection
> + to the vduse kernel device. Sometimes,the userspace application exists,
I guess you meant "exists"? If yes, it should be better to say "exits
unexpectedly"
> + and we want to support its restart and connect to the kernel device again
> +
> +2. How can I support reconnection in a userspace application?
Better to say "How reconnection is supported"?
> +
> +2.1 During initialization, the userspace application should first verify the
> + existence of the device "/dev/vduse/vduse_name".
> + If it doesn't exist, it means this is the first-time for connection. goto step 2.2
> + If it exists, it means this is a reconnection, and we should goto step 2.3
> +
> +2.2 Create a new VDUSE instance with ioctl(VDUSE_CREATE_DEV) on
> + /dev/vduse/control.
> + When ioctl(VDUSE_CREATE_DEV) is called, kernel allocates memory for
> + the reconnect information. The total memory size is PAGE_SIZE*vq_mumber.
I think we need to mention that this should be part of the previous
"VDUSE devices are created as follows"?
> +
> +2.3 Check if the information is suitable for reconnect
> + If this is reconnection :
> + Before attempting to reconnect, The userspace application needs to use the
> + ioctl(VDUSE_DEV_GET_CONFIG, VDUSE_DEV_GET_STATUS, VDUSE_DEV_GET_FEATURES...)
> + to get the information from kernel.
> + Please review the information and confirm if it is suitable to reconnect.
Need to define "review" here and how to decide if it is not suitable
to reconnect.
> +
> +2.4 Userspace application needs to mmap the memory to userspace
> + The userspace application requires mapping one page for every vq. These pages
> + should be used to save vq-related information during system running.
Not a native speaker, but it looks better with
"should be used by the userspace to store virtqueue specific information".
> Additionally,
> + the application must define its own structure to store information for reconnection.
> +
> +2.5 Completed the initialization and running the application.
> + While the application is running, it is important to store relevant information
> + about reconnections in mapped pages.
I think we need some link/code examples to demonstrate what needs to be stored.
> When calling the ioctl VDUSE_VQ_GET_INFO to
> + get vq information, it's necessary to check whether it's a reconnection.
Better with some examples of codes.
> If it is
> + a reconnection, the vq-related information must be get from the mapped pages.
> +
> +2.6 When the Userspace application exits, it is necessary to unmap all the
> + pages for reconnection
This seems to be unnecessary, for example there could be an unexpected exit.
Thanks
> --
> 2.43.0
>
On Tue, Apr 16, 2024 at 11:46 AM Jason Wang <[email protected]> wrote:
>
> On Fri, Apr 12, 2024 at 9:31 PM Cindy Lu <[email protected]> wrote:
> >
> > Add a document explaining the reconnect process, including what the
> > Userspace App needs to do and how it works with the kernel.
> >
> > Signed-off-by: Cindy Lu <[email protected]>
> > ---
> > Documentation/userspace-api/vduse.rst | 41 +++++++++++++++++++++++++++
> > 1 file changed, 41 insertions(+)
> >
> > diff --git a/Documentation/userspace-api/vduse.rst b/Documentation/userspace-api/vduse.rst
> > index bdb880e01132..7faa83462e78 100644
> > --- a/Documentation/userspace-api/vduse.rst
> > +++ b/Documentation/userspace-api/vduse.rst
> > @@ -231,3 +231,44 @@ able to start the dataplane processing as follows:
> > after the used ring is filled.
> >
> > For more details on the uAPI, please see include/uapi/linux/vduse.h.
> > +
> > +HOW VDUSE devices reconnection works
> > +------------------------------------
> > +1. What is reconnection?
> > +
> > + When the userspace application loads, it should establish a connection
> > + to the vduse kernel device. Sometimes,the userspace application exists,
>
> I guess you meant "exists"? If yes, it should be better to say "exits
> unexpectedly"
>
> > + and we want to support its restart and connect to the kernel device again
> > +
> > +2. How can I support reconnection in a userspace application?
>
> Better to say "How reconnection is supported"?
>
> > +
> > +2.1 During initialization, the userspace application should first verify the
> > + existence of the device "/dev/vduse/vduse_name".
> > + If it doesn't exist, it means this is the first-time for connection. goto step 2.2
> > + If it exists, it means this is a reconnection, and we should goto step 2.3
> > +
> > +2.2 Create a new VDUSE instance with ioctl(VDUSE_CREATE_DEV) on
> > + /dev/vduse/control.
> > + When ioctl(VDUSE_CREATE_DEV) is called, kernel allocates memory for
> > + the reconnect information. The total memory size is PAGE_SIZE*vq_mumber.
>
> I think we need to mention that this should be part of the previous
> "VDUSE devices are created as follows"?
>
> > +
> > +2.3 Check if the information is suitable for reconnect
> > + If this is reconnection :
> > + Before attempting to reconnect, The userspace application needs to use the
> > + ioctl(VDUSE_DEV_GET_CONFIG, VDUSE_DEV_GET_STATUS, VDUSE_DEV_GET_FEATURES...)
> > + to get the information from kernel.
> > + Please review the information and confirm if it is suitable to reconnect.
>
> Need to define "review" here and how to decide if it is not suitable
> to reconnect.
>
> > +
> > +2.4 Userspace application needs to mmap the memory to userspace
> > + The userspace application requires mapping one page for every vq. These pages
> > + should be used to save vq-related information during system running.
>
> Not a native speaker, but it looks better with
>
> "should be used by the userspace to store virtqueue specific information".
>
> > Additionally,
> > + the application must define its own structure to store information for reconnection.
> > +
> > +2.5 Completed the initialization and running the application.
> > + While the application is running, it is important to store relevant information
> > + about reconnections in mapped pages.
>
> I think we need some link/code examples to demonstrate what needs to be stored.
>
> > When calling the ioctl VDUSE_VQ_GET_INFO to
> > + get vq information, it's necessary to check whether it's a reconnection.
>
> Better with some examples of codes.
>
> > If it is
> > + a reconnection, the vq-related information must be get from the mapped pages.
> > +
> > +2.6 When the Userspace application exits, it is necessary to unmap all the
> > + pages for reconnection
>
> This seems to be unnecessary, for example there could be an unexpected exit.
>
> Thanks
>
Thanks Jason, I will send a new version
Thanks
Cindy
> > --
> > 2.43.0
> >
>