2024-05-27 22:09:12

by Michael Grzeschik

[permalink] [raw]
Subject: [PATCH v5 0/3] usb: gadget: 9pfs transport

This series is adding support to mount usb hostside exported 9pfs
filesystems via the usb gadget interface. It also includes a simple tool
(p9_fwd.py) to translate an tcp 9pfs transport and reuse it via the usb
interface.

+--------------------------+ | +--------------------------+
| 9PFS mounting client | | | 9PFS exporting server |
SW | | | | |
| (this:trans_usbg) | | |(e.g. diod or nfs-ganesha)|
+-------------^------------+ | +-------------^------------+
| | |
| | +------v------+
| | | p9_fwd.py |
| | +------^------+
| | |
------------------|------------------------------------|-------------
| | |
+-------------v------------+ | +-------------v------------+
| | | | |
HW | USB Device Controller <---------> USB Host Controller |
| | | | |
+--------------------------+ | +--------------------------+

The USB host exports a filesystem, while the gadget on the USB device
side makes it mountable.

Diod (9pfs server) and the forwarder are on the development host, where
the root filesystem is actually stored. The gadget is initialized during
boot (or later) on the embedded board. Then the forwarder will find it
on the USB bus and start forwarding requests.

In this case the 9p requests come from the device and are handled by the
host. The reason is that USB device ports are normally not available on
PCs, so a connection in the other direction would not work.

One use-case is to use it as an alternative to NFS root booting during
the development of embedded Linux devices.

Signed-off-by: Michael Grzeschik <[email protected]>
---
Changes in v5:
- fixed lockup in mount -> remount -> monut scenario
- improved p9_fwd transport script with more options
- Link to v4: https://lore.kernel.org/r/[email protected]

Changes in v4:
- reworked the naming scheme to be set by the configfs instance
- added conn_cancel function to properly stop the transfers
- ensured that umount -f will work even when the host side has crahed
- added all the review feedback from Andrzej Pietrasiewicz
- Link to v3: https://lore.kernel.org/r/[email protected]

Changes in v3:
- dropped patch "usb: gadget: legacy: add 9pfs multi gadget" as discussed with gregkh
- Link to v2: https://lore.kernel.org/r/[email protected]

Changes in v2:
- improved the commit messages
- introduced an patch to move the header u_f.h to include/linux/usb to compile usb gadget functions treewide
- moved usbg gadget function to net/9p/
- adderessed several comments in function driver, like the cleanup path and kbuild errors
- improved the documentation in Documentation/filesystems/9p.rst
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Michael Grzeschik (3):
usb: gadget: function: move u_f.h to include/linux/usb/func_utils.h
net/9p/usbg: Add new usb gadget function transport
tools: usb: p9_fwd: add usb gadget packet forwarder script

Documentation/filesystems/9p.rst | 58 +-
drivers/usb/gadget/configfs.c | 2 +-
drivers/usb/gadget/function/f_fs.c | 2 +-
drivers/usb/gadget/function/f_hid.c | 2 +-
drivers/usb/gadget/function/f_loopback.c | 2 +-
drivers/usb/gadget/function/f_midi.c | 2 +-
drivers/usb/gadget/function/f_midi2.c | 2 +-
drivers/usb/gadget/function/f_sourcesink.c | 2 +-
drivers/usb/gadget/u_f.c | 2 +-
.../gadget/u_f.h => include/linux/usb/func_utils.h | 2 +-
net/9p/Kconfig | 6 +
net/9p/Makefile | 4 +
net/9p/trans_usbg.c | 974 +++++++++++++++++++++
tools/usb/p9_fwd.py | 243 +++++
14 files changed, 1293 insertions(+), 10 deletions(-)
---
base-commit: 2bfcfd584ff5ccc8bb7acde19b42570414bf880b
change-id: 20240116-ml-topic-u9p-895274530eb1

Best regards,
--
Michael Grzeschik <[email protected]>



2024-05-27 22:09:19

by Michael Grzeschik

[permalink] [raw]
Subject: [PATCH v5 1/3] usb: gadget: function: move u_f.h to include/linux/usb/func_utils.h

We move the func_utils.h header to include/linux/usb to be
able to compile function drivers outside of the
drivers/usb/gadget/function directory.

Signed-off-by: Michael Grzeschik <[email protected]>

---
v4 -> v5:
- renamed to func_utils.h
v3 -> v4: -
v2 -> v3: -
v1 -> v2:
- new introduced patch
---
drivers/usb/gadget/configfs.c | 2 +-
drivers/usb/gadget/function/f_fs.c | 2 +-
drivers/usb/gadget/function/f_hid.c | 2 +-
drivers/usb/gadget/function/f_loopback.c | 2 +-
drivers/usb/gadget/function/f_midi.c | 2 +-
drivers/usb/gadget/function/f_midi2.c | 2 +-
drivers/usb/gadget/function/f_sourcesink.c | 2 +-
drivers/usb/gadget/u_f.c | 2 +-
drivers/usb/gadget/u_f.h => include/linux/usb/func_utils.h | 2 +-
9 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/drivers/usb/gadget/configfs.c b/drivers/usb/gadget/configfs.c
index ce3cfa1f36f51..e3cda6e0cd0d2 100644
--- a/drivers/usb/gadget/configfs.c
+++ b/drivers/usb/gadget/configfs.c
@@ -8,8 +8,8 @@
#include <linux/usb/composite.h>
#include <linux/usb/gadget_configfs.h>
#include <linux/usb/webusb.h>
+#include <linux/usb/func_utils.h>
#include "configfs.h"
-#include "u_f.h"
#include "u_os_desc.h"

int check_user_usb_string(const char *name,
diff --git a/drivers/usb/gadget/function/f_fs.c b/drivers/usb/gadget/function/f_fs.c
index 1f21459b1188c..f06e6c4974a6a 100644
--- a/drivers/usb/gadget/function/f_fs.c
+++ b/drivers/usb/gadget/function/f_fs.c
@@ -33,6 +33,7 @@
#include <linux/usb/ccid.h>
#include <linux/usb/composite.h>
#include <linux/usb/functionfs.h>
+#include <linux/usb/func_utils.h>

#include <linux/aio.h>
#include <linux/kthread.h>
@@ -40,7 +41,6 @@
#include <linux/eventfd.h>

#include "u_fs.h"
-#include "u_f.h"
#include "u_os_desc.h"
#include "configfs.h"

diff --git a/drivers/usb/gadget/function/f_hid.c b/drivers/usb/gadget/function/f_hid.c
index 2db01e03bfbf0..0b7684c4a5b8b 100644
--- a/drivers/usb/gadget/function/f_hid.c
+++ b/drivers/usb/gadget/function/f_hid.c
@@ -16,8 +16,8 @@
#include <linux/wait.h>
#include <linux/sched.h>
#include <linux/usb/g_hid.h>
+#include <linux/usb/func_utils.h>

-#include "u_f.h"
#include "u_hid.h"

#define HIDG_MINORS 4
diff --git a/drivers/usb/gadget/function/f_loopback.c b/drivers/usb/gadget/function/f_loopback.c
index 17ac6ace0cffa..8fce2c2fd18f7 100644
--- a/drivers/usb/gadget/function/f_loopback.c
+++ b/drivers/usb/gadget/function/f_loopback.c
@@ -14,9 +14,9 @@
#include <linux/module.h>
#include <linux/err.h>
#include <linux/usb/composite.h>
+#include <linux/usb/func_utils.h>

#include "g_zero.h"
-#include "u_f.h"

/*
* LOOPBACK FUNCTION ... a testing vehicle for USB peripherals,
diff --git a/drivers/usb/gadget/function/f_midi.c b/drivers/usb/gadget/function/f_midi.c
index 20c6fbd94f32d..9f8f3a334b579 100644
--- a/drivers/usb/gadget/function/f_midi.c
+++ b/drivers/usb/gadget/function/f_midi.c
@@ -33,8 +33,8 @@
#include <linux/usb/gadget.h>
#include <linux/usb/audio.h>
#include <linux/usb/midi.h>
+#include <linux/usb/func_utils.h>

-#include "u_f.h"
#include "u_midi.h"

MODULE_AUTHOR("Ben Williamson");
diff --git a/drivers/usb/gadget/function/f_midi2.c b/drivers/usb/gadget/function/f_midi2.c
index ec8cd7c7bbfc1..830aa075c130f 100644
--- a/drivers/usb/gadget/function/f_midi2.c
+++ b/drivers/usb/gadget/function/f_midi2.c
@@ -18,8 +18,8 @@
#include <linux/usb/gadget.h>
#include <linux/usb/audio.h>
#include <linux/usb/midi-v2.h>
+#include <linux/usb/func_utils.h>

-#include "u_f.h"
#include "u_midi2.h"

struct f_midi2;
diff --git a/drivers/usb/gadget/function/f_sourcesink.c b/drivers/usb/gadget/function/f_sourcesink.c
index 2edbd9b510d60..c9f38f24a92aa 100644
--- a/drivers/usb/gadget/function/f_sourcesink.c
+++ b/drivers/usb/gadget/function/f_sourcesink.c
@@ -13,10 +13,10 @@
#include <linux/device.h>
#include <linux/module.h>
#include <linux/usb/composite.h>
+#include <linux/usb/func_utils.h>
#include <linux/err.h>

#include "g_zero.h"
-#include "u_f.h"

/*
* SOURCE/SINK FUNCTION ... a primary testing vehicle for USB peripheral
diff --git a/drivers/usb/gadget/u_f.c b/drivers/usb/gadget/u_f.c
index 6aea1ecb39999..9f90665827945 100644
--- a/drivers/usb/gadget/u_f.c
+++ b/drivers/usb/gadget/u_f.c
@@ -8,7 +8,7 @@
* Author: Andrzej Pietrasiewicz <[email protected]>
*/

-#include "u_f.h"
+#include <linux/usb/func_utils.h>
#include <linux/usb/ch9.h>

struct usb_request *alloc_ep_req(struct usb_ep *ep, size_t len)
diff --git a/drivers/usb/gadget/u_f.h b/include/linux/usb/func_utils.h
similarity index 99%
rename from drivers/usb/gadget/u_f.h
rename to include/linux/usb/func_utils.h
index e313c3b8dcb19..9f2a32c765260 100644
--- a/drivers/usb/gadget/u_f.h
+++ b/include/linux/usb/func_utils.h
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
/*
- * u_f.h
+ * usbf_utils.h
*
* Utility definitions for USB functions
*

--
2.39.2


2024-05-27 22:09:33

by Michael Grzeschik

[permalink] [raw]
Subject: [PATCH v5 3/3] tools: usb: p9_fwd: add usb gadget packet forwarder script

This patch is adding an small python tool to forward 9pfs requests
from the USB gadget to an existing 9pfs TCP server. Since currently all
9pfs servers lack support for the usb transport this tool is an useful
helper to get started.

Refer the Documentation section "USBG Example" in
Documentation/filesystems/9p.rst on how to use it.

Signed-off-by: Jan Luebbe <[email protected]>
Signed-off-by: Michael Grzeschik <[email protected]>

---
v4 -> v5:
- updated documentation for new subcommands list/connect
- run ruff format
- make vid and pid parameterized
- add list as subcommand to scan for devices
- move connect to extra subcommand
v3 -> v4: -
v2 -> v3: -
v1 -> v2:
- added usbg 9pfs detailed instructions to 9p.rst doc
---
Documentation/filesystems/9p.rst | 41 +++++++
tools/usb/p9_fwd.py | 243 +++++++++++++++++++++++++++++++++++++++
2 files changed, 284 insertions(+)

diff --git a/Documentation/filesystems/9p.rst b/Documentation/filesystems/9p.rst
index 10cf79dc287f8..2cc85f3e8659f 100644
--- a/Documentation/filesystems/9p.rst
+++ b/Documentation/filesystems/9p.rst
@@ -67,6 +67,47 @@ To mount a 9p FS on a USB Host accessible via the gadget as root filesystem::
where <device> is the tag associated by the usb gadget transport.
It is defined by the configfs instance name.

+USBG Example
+============
+
+The USB host exports a filesystem, while the gadget on the USB device
+side makes it mountable.
+
+Diod (9pfs server) and the forwarder are on the development host, where
+the root filesystem is actually stored. The gadget is initialized during
+boot (or later) on the embedded board. Then the forwarder will find it
+on the USB bus and start forwarding requests.
+
+In this case the 9p requests come from the device and are handled by the
+host. The reason is that USB device ports are normally not available on
+PCs, so a connection in the other direction would not work.
+
+When using the usbg transport, for now there is no native usb host
+service capable to handle the requests from the gadget driver. For
+this we have to use the extra python tool p9_fwd.py from tools/usb.
+
+Just start the 9pfs capable network server like diod/nfs-ganesha e.g.:
+
+ $ diod -f -n -d 0 -S -l 0.0.0.0:9999 -e $PWD
+
+Optionaly scan your bus if there are more then one usbg gadgets to find their path:
+
+ $ python $kernel_dir/tools/usb/p9_fwd.py list
+
+ Bus | Addr | Manufacturer | Product | ID | Path
+ --- | ---- | ---------------- | ---------------- | --------- | ----
+ 2 | 67 | unknown | unknown | 1d6b:0109 | 2-1.1.2
+ 2 | 68 | unknown | unknown | 1d6b:0109 | 2-1.1.3
+
+Then start the python transport:
+
+ $ python $kernel_dir/tools/usb/p9_fwd.py --path 2-1.1.2 connect -p 9999
+
+After that the gadget driver can be used as described above.
+
+One use-case is to use it as an alternative to NFS root booting during
+the development of embedded Linux devices.
+
Options
=======

diff --git a/tools/usb/p9_fwd.py b/tools/usb/p9_fwd.py
new file mode 100755
index 0000000000000..7bedefce75c7b
--- /dev/null
+++ b/tools/usb/p9_fwd.py
@@ -0,0 +1,243 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+import argparse
+import errno
+import logging
+import socket
+import struct
+import time
+
+import usb.core
+import usb.util
+
+
+def path_from_usb_dev(dev):
+ """Takes a pyUSB device as argument and returns a string.
+ The string is a Path representation of the position of the USB device on the USB bus tree.
+
+ This path is used to find a USB device on the bus or all devices connected to a HUB.
+ The path is made up of the number of the USB controller followed be the ports of the HUB tree."""
+ if dev.port_numbers:
+ dev_path = ".".join(str(i) for i in dev.port_numbers)
+ return f"{dev.bus}-{dev_path}"
+ return ""
+
+
+HEXDUMP_FILTER = "".join(chr(x).isprintable() and chr(x) or "." for x in range(128)) + "." * 128
+
+
+class Forwarder:
+ @staticmethod
+ def _log_hexdump(data):
+ if not logging.root.isEnabledFor(logging.TRACE):
+ return
+ L = 16
+ for c in range(0, len(data), L):
+ chars = data[c : c + L]
+ dump = " ".join(f"{x:02x}" for x in chars)
+ printable = "".join(HEXDUMP_FILTER[x] for x in chars)
+ line = f"{c:08x} {dump:{L*3}s} |{printable:{L}s}|"
+ logging.root.log(logging.TRACE, "%s", line)
+
+ def __init__(self, server, vid, pid, path):
+ self.stats = {
+ "c2s packets": 0,
+ "c2s bytes": 0,
+ "s2c packets": 0,
+ "s2c bytes": 0,
+ }
+ self.stats_logged = time.monotonic()
+
+ def find_filter(dev):
+ dev_path = path_from_usb_dev(dev)
+ if path is not None:
+ return dev_path == path
+ return True
+
+ dev = usb.core.find(idVendor=vid, idProduct=pid, custom_match=find_filter)
+ if dev is None:
+ raise ValueError("Device not found")
+
+ logging.info(f"found device: {dev.bus}/{dev.address} located at {path_from_usb_dev(dev)}")
+
+ # dev.set_configuration() is not necessary since g_multi has only one
+ usb9pfs = None
+ # g_multi adds 9pfs as last interface
+ cfg = dev.get_active_configuration()
+ for intf in cfg:
+ # we have to detach the usb-storage driver from multi gadget since
+ # stall option could be set, which will lead to spontaneous port
+ # resets and our transfers will run dead
+ if intf.bInterfaceClass == 0x08:
+ if dev.is_kernel_driver_active(intf.bInterfaceNumber):
+ dev.detach_kernel_driver(intf.bInterfaceNumber)
+
+ if intf.bInterfaceClass == 0xFF and intf.bInterfaceSubClass == 0xFF and intf.bInterfaceProtocol == 0x09:
+ usb9pfs = intf
+ if usb9pfs is None:
+ raise ValueError("Interface not found")
+
+ logging.info(f"claiming interface:\n{usb9pfs}")
+ usb.util.claim_interface(dev, usb9pfs.bInterfaceNumber)
+ ep_out = usb.util.find_descriptor(
+ usb9pfs,
+ custom_match=lambda e: usb.util.endpoint_direction(e.bEndpointAddress) == usb.util.ENDPOINT_OUT,
+ )
+ assert ep_out is not None
+ ep_in = usb.util.find_descriptor(
+ usb9pfs,
+ custom_match=lambda e: usb.util.endpoint_direction(e.bEndpointAddress) == usb.util.ENDPOINT_IN,
+ )
+ assert ep_in is not None
+ logging.info("interface claimed")
+
+ self.ep_out = ep_out
+ self.ep_in = ep_in
+ self.dev = dev
+
+ # create and connect socket
+ self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ self.s.connect(server)
+
+ logging.info("connected to server")
+
+ def c2s(self):
+ """forward a request from the USB client to the TCP server"""
+ data = None
+ while data is None:
+ try:
+ logging.log(logging.TRACE, "c2s: reading")
+ data = self.ep_in.read(self.ep_in.wMaxPacketSize)
+ except usb.core.USBTimeoutError:
+ logging.log(logging.TRACE, "c2s: reading timed out")
+ continue
+ except usb.core.USBError as e:
+ if e.errno == errno.EIO:
+ logging.debug("c2s: reading failed with %s, retrying", repr(e))
+ time.sleep(0.5)
+ continue
+ logging.error("c2s: reading failed with %s, aborting", repr(e))
+ raise
+ size = struct.unpack("<I", data[:4])[0]
+ while len(data) < size:
+ data += self.ep_in.read(size - len(data))
+ logging.log(logging.TRACE, "c2s: writing")
+ self._log_hexdump(data)
+ self.s.send(data)
+ logging.debug("c2s: forwarded %i bytes", size)
+ self.stats["c2s packets"] += 1
+ self.stats["c2s bytes"] += size
+
+ def s2c(self):
+ """forward a response from the TCP server to the USB client"""
+ logging.log(logging.TRACE, "s2c: reading")
+ data = self.s.recv(4)
+ size = struct.unpack("<I", data[:4])[0]
+ while len(data) < size:
+ data += self.s.recv(size - len(data))
+ logging.log(logging.TRACE, "s2c: writing")
+ self._log_hexdump(data)
+ while data:
+ written = self.ep_out.write(data)
+ assert written > 0
+ data = data[written:]
+ if size % self.ep_out.wMaxPacketSize == 0:
+ logging.log(logging.TRACE, "sending zero length packet")
+ self.ep_out.write(b"")
+ logging.debug("s2c: forwarded %i bytes", size)
+ self.stats["s2c packets"] += 1
+ self.stats["s2c bytes"] += size
+
+ def log_stats(self):
+ logging.info("statistics:")
+ for k, v in self.stats.items():
+ logging.info(f" {k+':':14s} {v}")
+
+ def log_stats_interval(self, interval=5):
+ if (time.monotonic() - self.stats_logged) < interval:
+ return
+
+ self.log_stats()
+ self.stats_logged = time.monotonic()
+
+
+def try_get_usb_str(dev, name):
+ try:
+ with open(f"/sys/bus/usb/devices/{dev.bus}-{dev.address}/{name}") as f:
+ return f.read().strip()
+ except FileNotFoundError:
+ return None
+
+
+def list_usb(args):
+ vid, pid = [int(x, 16) for x in args.id.split(":", 1)]
+
+ print("Bus | Addr | Manufacturer | Product | ID | Path")
+ print("--- | ---- | ---------------- | ---------------- | --------- | ----")
+ for dev in usb.core.find(find_all=True, idVendor=vid, idProduct=pid):
+ path = path_from_usb_dev(dev) or ""
+ manufacturer = try_get_usb_str(dev, "manufacturer") or "unknown"
+ product = try_get_usb_str(dev, "product") or "unknown"
+ print(
+ f"{dev.bus:3} | {dev.address:4} | {manufacturer:16} | {product:16} | {dev.idVendor:04x}:{dev.idProduct:04x} | {path:18}"
+ )
+
+
+def connect(args):
+ vid, pid = [int(x, 16) for x in args.id.split(":", 1)]
+
+ f = Forwarder(server=(args.server, args.port), vid=vid, pid=pid, path=args.path)
+
+ try:
+ while True:
+ f.c2s()
+ f.s2c()
+ f.log_stats_interval()
+ finally:
+ f.log_stats()
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Forward 9PFS requests from USB to TCP",
+ )
+
+ parser.add_argument("--id", type=str, default="1d6b:0109", help="vid:pid of target device")
+ parser.add_argument("--path", type=str, default="", help="path of target device")
+ parser.add_argument("-v", "--verbose", action="count", default=0)
+
+ subparsers = parser.add_subparsers()
+ subparsers.required = True
+ subparsers.dest = "command"
+
+ parser_list = subparsers.add_parser("list", help="List all connected 9p gadgets")
+ parser_list.set_defaults(func=list_usb)
+
+ parser_connect = subparsers.add_parser(
+ "connect", help="Forward messages between the usb9pfs gadget and the 9p server"
+ )
+ parser_connect.set_defaults(func=connect)
+ connect_group = parser_connect.add_argument_group()
+ connect_group.required = True
+ parser_connect.add_argument("-s", "--server", type=str, default="127.0.0.1", help="server hostname")
+ parser_connect.add_argument("-p", "--port", type=int, default=564, help="server port")
+
+ args = parser.parse_args()
+
+ logging.TRACE = logging.DEBUG - 5
+ logging.addLevelName(logging.TRACE, "TRACE")
+
+ if args.verbose >= 2:
+ level = logging.TRACE
+ elif args.verbose:
+ level = logging.DEBUG
+ else:
+ level = logging.INFO
+ logging.basicConfig(level=level, format="%(asctime)-15s %(levelname)-8s %(message)s")
+
+ args.func(args)
+
+
+if __name__ == "__main__":
+ main()

--
2.39.2


2024-05-27 22:09:52

by Michael Grzeschik

[permalink] [raw]
Subject: [PATCH v5 2/3] net/9p/usbg: Add new usb gadget function transport

Add the new gadget function for 9pfs transport. This function is
defining an simple 9pfs transport interface that consists of one in and
one out endpoint. The endpoints transmit and receive the 9pfs protocol
payload when mounting a 9p filesystem over usb.

Signed-off-by: Michael Grzeschik <[email protected]>

---
v4 -> v5:
- remove instance from instance list on free
- bail out if no req was queued in disable
- free requests on disable
- use irqsave/restore in other functions then the completion handlers
- simplify the completion handlers
- reset tx_req pointer to NULL after the status update
- add unbind function
- use the right lock when protecting 9p requets
- refactor giveback of tx_request to extra function clear_tx
- enable endpoints once on set_alt
- only disable endpoints on unbind
- reset request pointers on free
- remove req_lock which is not necessary since only one transfer can
be queued for now
v3 -> v4:
- implemented conn_cancel
- moved from mount_tag to device name like ffs does
- renamed function_list to usb9pfs_instance
- renamed usbg_function_list to usb9pfs_instance_list
- renamed rx/tx_fill to queue_rx/tx
- added use of retvals
- added locking comment over usb9pfs_transmit
- using if_else instead of two pathed switch calls
- fixed return values of completion handler
- using void pointer as parameter in rx_header
- added a missed req_put in rx_header
- removed extra disable function but call disable_ep directly
- repaired several return values
- remove the suspicious disable / enable path in usbg_create
v2 -> v3: -
v1 -> v2:
- move the driver to net/9p/trans_usbg.c
- fixed the commit title
- fix %zu usage in p9_debug for size_t type
- define functions static that are only used localy
- return 0 in alloc_requests
- use id_alloc_max instead of ida_simple_get
- call ida_free in usb9pfs_free_func
- call kfree for usb9pfs->tag and usb9pfs in usb9pfs_free_func
- add MODULE_ALIAS_9P to load this module automatic when mounting
- improved the documentation
- added depends on USB_GADGET in Kconfig
---
Documentation/filesystems/9p.rst | 17 +-
net/9p/Kconfig | 6 +
net/9p/Makefile | 4 +
net/9p/trans_usbg.c | 974 +++++++++++++++++++++++++++++++++++++++
4 files changed, 1000 insertions(+), 1 deletion(-)

diff --git a/Documentation/filesystems/9p.rst b/Documentation/filesystems/9p.rst
index 1e0e0bb6fdf91..10cf79dc287f8 100644
--- a/Documentation/filesystems/9p.rst
+++ b/Documentation/filesystems/9p.rst
@@ -48,11 +48,25 @@ For server running on QEMU host with virtio transport::

mount -t 9p -o trans=virtio <mount_tag> /mnt/9

-where mount_tag is the tag associated by the server to each of the exported
+where mount_tag is the tag generated by the server to each of the exported
mount points. Each 9P export is seen by the client as a virtio device with an
associated "mount_tag" property. Available mount tags can be
seen by reading /sys/bus/virtio/drivers/9pnet_virtio/virtio<n>/mount_tag files.

+USBG Usage
+==========
+
+To mount a 9p FS on a USB Host accessible via the gadget at runtime::
+
+ mount -t 9p -o trans=usbg,aname=/path/to/fs <device> /mnt/9
+
+To mount a 9p FS on a USB Host accessible via the gadget as root filesystem::
+
+ root=<device> rootfstype=9p rootflags=trans=usbg,cache=loose,uname=root,access=0,dfltuid=0,dfltgid=0,aname=/path/to/rootfs
+
+where <device> is the tag associated by the usb gadget transport.
+It is defined by the configfs instance name.
+
Options
=======

@@ -68,6 +82,7 @@ Options
virtio connect to the next virtio channel available
(from QEMU with trans_virtio module)
rdma connect to a specified RDMA channel
+ usbg connect to a specified usb gadget channel
======== ============================================

uname=name user name to attempt mount as on the remote server. The
diff --git a/net/9p/Kconfig b/net/9p/Kconfig
index bcdab9c23b402..2d7e596e22c3f 100644
--- a/net/9p/Kconfig
+++ b/net/9p/Kconfig
@@ -40,6 +40,12 @@ config NET_9P_XEN
This builds support for a transport for 9pfs between
two Xen domains.

+config NET_9P_USBG
+ bool "9P USB Gadget Transport"
+ depends on USB_GADGET
+ help
+ This builds support for a transport for 9pfs over
+ usb gadget.

config NET_9P_RDMA
depends on INET && INFINIBAND && INFINIBAND_ADDR_TRANS
diff --git a/net/9p/Makefile b/net/9p/Makefile
index 1df9b344c30bd..22794a451c3f7 100644
--- a/net/9p/Makefile
+++ b/net/9p/Makefile
@@ -4,6 +4,7 @@ obj-$(CONFIG_NET_9P_FD) += 9pnet_fd.o
obj-$(CONFIG_NET_9P_XEN) += 9pnet_xen.o
obj-$(CONFIG_NET_9P_VIRTIO) += 9pnet_virtio.o
obj-$(CONFIG_NET_9P_RDMA) += 9pnet_rdma.o
+obj-$(CONFIG_NET_9P_USBG) += 9pnet_usbg.o

9pnet-objs := \
mod.o \
@@ -23,3 +24,6 @@ obj-$(CONFIG_NET_9P_RDMA) += 9pnet_rdma.o

9pnet_rdma-objs := \
trans_rdma.o \
+
+9pnet_usbg-objs := \
+ trans_usbg.o \
diff --git a/net/9p/trans_usbg.c b/net/9p/trans_usbg.c
new file mode 100644
index 0000000000000..3a84097256837
--- /dev/null
+++ b/net/9p/trans_usbg.c
@@ -0,0 +1,974 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * trans_usbg.c - USB peripheral usb9pfs configuration driver and transport.
+ *
+ * Copyright (C) 2024 Michael Grzeschik <[email protected]>
+ */
+
+/* Gadget usb9pfs only needs two bulk endpoints, and will use the usb9pfs
+ * transport to mount host exported filesystem via usb gadget.
+ */
+
+/* +--------------------------+ | +--------------------------+
+ * | 9PFS mounting client | | | 9PFS exporting server |
+ * SW | | | | |
+ * | (this:trans_usbg) | | |(e.g. diod or nfs-ganesha)|
+ * +-------------^------------+ | +-------------^------------+
+ * | | |
+ * ------------------|------------------------------------|-------------
+ * | | |
+ * +-------------v------------+ | +-------------v------------+
+ * | | | | |
+ * HW | USB Device Controller <---------> USB Host Controller |
+ * | | | | |
+ * +--------------------------+ | +--------------------------+
+ */
+
+#include <linux/slab.h>
+#include <linux/kernel.h>
+#include <linux/device.h>
+#include <linux/module.h>
+#include <linux/err.h>
+#include <linux/usb/composite.h>
+#include <linux/usb/func_utils.h>
+
+#include <net/9p/9p.h>
+#include <net/9p/client.h>
+#include <net/9p/transport.h>
+
+#define DEFAULT_BUFLEN 16384
+
+struct f_usb9pfs {
+ struct p9_client *client;
+
+ struct p9_req_t *p9_tx_req;
+
+ struct list_head tx_req_list;
+
+ /* 9p request lock for en/dequeue */
+ spinlock_t lock;
+
+ struct usb_request *in_req;
+ struct usb_request *out_req;
+
+ struct usb_ep *in_ep;
+ struct usb_ep *out_ep;
+
+ unsigned int buflen;
+
+ struct usb_function function;
+};
+
+static inline struct f_usb9pfs *func_to_usb9pfs(struct usb_function *f)
+{
+ return container_of(f, struct f_usb9pfs, function);
+}
+
+struct f_usb9pfs_opts {
+ struct usb_function_instance func_inst;
+ unsigned int buflen;
+
+ struct f_usb9pfs_dev *dev;
+
+ /* Read/write access to configfs attributes is handled by configfs.
+ *
+ * This is to protect the data from concurrent access by read/write
+ * and create symlink/remove symlink.
+ */
+ struct mutex lock;
+ int refcnt;
+};
+
+struct f_usb9pfs_dev {
+ struct f_usb9pfs *usb9pfs;
+ struct f_usb9pfs_opts *opts;
+ char tag[41];
+ bool inuse;
+
+ struct list_head usb9pfs_instance;
+};
+
+static DEFINE_MUTEX(usb9pfs_lock);
+static struct list_head usbg_instance_list;
+
+static int usb9pfs_queue_tx(struct f_usb9pfs *usb9pfs, struct usb_request *req,
+ gfp_t gfp_flags)
+{
+ struct usb_composite_dev *cdev = usb9pfs->function.config->cdev;
+ int ret = -ENOMEM;
+
+ if (!(usb9pfs->p9_tx_req->tc.size % usb9pfs->in_ep->maxpacket))
+ req->zero = 1;
+
+ req->buf = usb9pfs->p9_tx_req->tc.sdata;
+ req->length = usb9pfs->p9_tx_req->tc.size;
+
+ dev_dbg(&cdev->gadget->dev, "%s usb9pfs send --> %d/%d, zero: %d\n",
+ usb9pfs->in_ep->name, req->actual, req->length, req->zero);
+
+ ret = usb_ep_queue(usb9pfs->in_ep, req, gfp_flags);
+
+ dev_dbg(&cdev->gadget->dev, "tx submit --> %d\n", ret);
+
+ return ret;
+}
+
+static int usb9pfs_queue_rx(struct f_usb9pfs *usb9pfs, struct usb_request *req,
+ gfp_t gfp_flags)
+{
+ struct usb_composite_dev *cdev = usb9pfs->function.config->cdev;
+ int ret = -ENOMEM;
+
+ ret = usb_ep_queue(usb9pfs->out_ep, req, gfp_flags);
+
+ dev_dbg(&cdev->gadget->dev, "rx submit --> %d\n", ret);
+
+ return ret;
+}
+
+static int usb9pfs_transmit(struct f_usb9pfs *usb9pfs)
+{
+ struct p9_req_t *p9_req = NULL;
+ unsigned long flags;
+ int ret = 0;
+
+ spin_lock_irqsave(&usb9pfs->lock, flags);
+
+ if (usb9pfs->p9_tx_req) {
+ spin_unlock_irqrestore(&usb9pfs->lock, flags);
+ return -EBUSY;
+ }
+
+ p9_req = list_first_entry_or_null(&usb9pfs->tx_req_list,
+ struct p9_req_t, req_list);
+ if (!p9_req) {
+ spin_unlock_irqrestore(&usb9pfs->lock, flags);
+ return -ENOENT;
+ }
+
+ list_del(&p9_req->req_list);
+
+ usb9pfs->p9_tx_req = p9_req;
+
+ p9_req_get(usb9pfs->p9_tx_req);
+
+ ret = usb9pfs_queue_tx(usb9pfs, usb9pfs->in_req, GFP_ATOMIC);
+
+ spin_unlock_irqrestore(&usb9pfs->lock, flags);
+
+ return ret;
+}
+
+static void usb9pfs_tx_complete(struct usb_ep *ep, struct usb_request *req)
+{
+ struct f_usb9pfs *usb9pfs = ep->driver_data;
+ struct usb_composite_dev *cdev = usb9pfs->function.config->cdev;
+ int ret = 0;
+
+ if (req->status) {
+ dev_err(&cdev->gadget->dev, "%s usb9pfs complete --> %d, %d/%d\n",
+ ep->name, req->status, req->actual, req->length);
+ return;
+ }
+
+ /* reset zero packages */
+ req->zero = 0;
+
+ dev_dbg(&cdev->gadget->dev, "%s usb9pfs complete --> %d, %d/%d\n",
+ ep->name, req->status, req->actual, req->length);
+
+ WRITE_ONCE(usb9pfs->p9_tx_req->status, REQ_STATUS_SENT);
+
+ p9_req_put(usb9pfs->client, usb9pfs->p9_tx_req);
+
+ ret = usb9pfs_queue_rx(usb9pfs, usb9pfs->out_req, GFP_ATOMIC);
+ if (ret)
+ return;
+
+ return;
+}
+
+static struct p9_req_t *usb9pfs_rx_header(struct f_usb9pfs *usb9pfs, void *buf)
+{
+ struct p9_req_t *p9_rx_req;
+ struct p9_fcall rc;
+ int ret;
+
+ /* start by reading header */
+ rc.sdata = buf;
+ rc.offset = 0;
+ rc.capacity = P9_HDRSZ;
+ rc.size = P9_HDRSZ;
+
+ p9_debug(P9_DEBUG_TRANS, "mux %p got %zu bytes\n", usb9pfs,
+ rc.capacity - rc.offset);
+
+ ret = p9_parse_header(&rc, &rc.size, NULL, NULL, 0);
+ if (ret) {
+ p9_debug(P9_DEBUG_ERROR,
+ "error parsing header: %d\n", ret);
+ return NULL;
+ }
+
+ p9_debug(P9_DEBUG_TRANS,
+ "mux %p pkt: size: %d bytes tag: %d\n",
+ usb9pfs, rc.size, rc.tag);
+
+ p9_rx_req = p9_tag_lookup(usb9pfs->client, rc.tag);
+ if (!p9_rx_req || p9_rx_req->status != REQ_STATUS_SENT) {
+ p9_debug(P9_DEBUG_ERROR, "Unexpected packet tag %d\n", rc.tag);
+ return NULL;
+ }
+
+ if (rc.size > p9_rx_req->rc.capacity) {
+ p9_debug(P9_DEBUG_ERROR,
+ "requested packet size too big: %d for tag %d with capacity %zd\n",
+ rc.size, rc.tag, p9_rx_req->rc.capacity);
+ p9_req_put(usb9pfs->client, p9_rx_req);
+ return NULL;
+ }
+
+ if (!p9_rx_req->rc.sdata) {
+ p9_debug(P9_DEBUG_ERROR,
+ "No recv fcall for tag %d (req %p), disconnecting!\n",
+ rc.tag, p9_rx_req);
+ p9_req_put(usb9pfs->client, p9_rx_req);
+ return NULL;
+ }
+
+ return p9_rx_req;
+}
+
+static void usb9pfs_rx_complete(struct usb_ep *ep, struct usb_request *req)
+{
+ struct f_usb9pfs *usb9pfs = ep->driver_data;
+ struct usb_composite_dev *cdev = usb9pfs->function.config->cdev;
+ struct p9_req_t *p9_rx_req;
+ unsigned long flags;
+
+ if (req->status) {
+ dev_err(&cdev->gadget->dev, "%s usb9pfs complete --> %d, %d/%d\n",
+ ep->name, req->status, req->actual, req->length);
+ return;
+ }
+
+ p9_rx_req = usb9pfs_rx_header(usb9pfs, req->buf);
+ if (!p9_rx_req)
+ return;
+
+ memcpy(p9_rx_req->rc.sdata, req->buf, req->actual);
+
+ p9_rx_req->rc.size = req->actual;
+
+ p9_client_cb(usb9pfs->client, p9_rx_req, REQ_STATUS_RCVD);
+ p9_req_put(usb9pfs->client, p9_rx_req);
+
+ spin_lock_irqsave(&usb9pfs->lock, flags);
+ usb9pfs->p9_tx_req = NULL;
+
+ spin_unlock_irqrestore(&usb9pfs->lock, flags);
+
+ usb9pfs_transmit(usb9pfs);
+
+ return;
+}
+
+static void disable_ep(struct usb_composite_dev *cdev, struct usb_ep *ep)
+{
+ int value;
+
+ value = usb_ep_disable(ep);
+ if (value < 0)
+ dev_info(&cdev->gadget->dev,
+ "disable %s --> %d\n", ep->name, value);
+}
+
+static void disable_usb9pfs(struct f_usb9pfs *usb9pfs)
+{
+ struct usb_composite_dev *cdev =
+ usb9pfs->function.config->cdev;
+
+ if (usb9pfs->in_req) {
+ usb_ep_free_request(usb9pfs->in_ep, usb9pfs->in_req);
+ usb9pfs->in_req = NULL;
+ }
+
+ if (usb9pfs->out_req) {
+ usb_ep_free_request(usb9pfs->out_ep, usb9pfs->out_req);
+ usb9pfs->out_req = NULL;
+ }
+
+ disable_ep(cdev, usb9pfs->in_ep);
+ disable_ep(cdev, usb9pfs->out_ep);
+ dev_dbg(&cdev->gadget->dev, "%s disabled\n",
+ usb9pfs->function.name);
+}
+
+static int alloc_requests(struct usb_composite_dev *cdev,
+ struct f_usb9pfs *usb9pfs)
+{
+ int ret = 0;
+
+ usb9pfs->in_req = usb_ep_alloc_request(usb9pfs->in_ep, GFP_ATOMIC);
+ if (!usb9pfs->in_req) {
+ ret = -ENOENT;
+ goto fail;
+ }
+
+ usb9pfs->out_req = alloc_ep_req(usb9pfs->out_ep, usb9pfs->buflen);
+ if (!usb9pfs->out_req) {
+ ret = -ENOENT;
+ goto fail_in;
+ }
+
+ usb9pfs->in_req->complete = usb9pfs_tx_complete;
+ usb9pfs->out_req->complete = usb9pfs_rx_complete;
+
+ /* length will be set in complete routine */
+ usb9pfs->in_req->context = usb9pfs;
+ usb9pfs->out_req->context = usb9pfs;
+
+ return ret;
+
+fail_in:
+ usb_ep_free_request(usb9pfs->in_ep, usb9pfs->in_req);
+fail:
+ return ret;
+}
+
+static int enable_endpoint(struct usb_composite_dev *cdev,
+ struct f_usb9pfs *usb9pfs, struct usb_ep *ep)
+{
+ int ret;
+
+ ret = config_ep_by_speed(cdev->gadget, &usb9pfs->function, ep);
+ if (ret)
+ return ret;
+
+ ret = usb_ep_enable(ep);
+ if (ret < 0)
+ return ret;
+
+ ep->driver_data = usb9pfs;
+
+ return ret;
+}
+
+static int
+enable_usb9pfs(struct usb_composite_dev *cdev, struct f_usb9pfs *usb9pfs)
+{
+ int result = 0;
+
+ result = enable_endpoint(cdev, usb9pfs, usb9pfs->in_ep);
+ if (result)
+ goto out;
+
+ result = enable_endpoint(cdev, usb9pfs, usb9pfs->out_ep);
+ if (result)
+ goto disable_in;
+
+ result = alloc_requests(cdev, usb9pfs);
+ if (result)
+ goto disable_out;
+
+ dev_dbg(&cdev->gadget->dev, "%s enabled\n", usb9pfs->function.name);
+ return 0;
+
+disable_out:
+ usb_ep_disable(usb9pfs->out_ep);
+disable_in:
+ usb_ep_disable(usb9pfs->in_ep);
+out:
+ return result;
+}
+
+static int p9_usbg_create(struct p9_client *client, const char *devname, char *args)
+{
+ struct usb_composite_dev *cdev;
+ struct f_usb9pfs_dev *dev;
+ struct f_usb9pfs_dev *tmp;
+ struct f_usb9pfs *usb9pfs;
+ struct usb_function *f;
+ int ret = -ENOENT;
+ int found = 0;
+
+ if (!devname)
+ return -EINVAL;
+
+ mutex_lock(&usb9pfs_lock);
+ list_for_each_entry_safe(dev, tmp, &usbg_instance_list, usb9pfs_instance) {
+ if (!strncmp(devname, dev->tag, strlen(devname))) {
+ if (!dev->inuse) {
+ dev->inuse = true;
+ found = 1;
+ break;
+ }
+ ret = -EBUSY;
+ break;
+ }
+ }
+ mutex_unlock(&usb9pfs_lock);
+
+ if (!found) {
+ pr_err("no channels available for device %s\n", devname);
+ return ret;
+ }
+
+ usb9pfs = dev->usb9pfs;
+ if (!usb9pfs)
+ return -EINVAL;
+
+ INIT_LIST_HEAD(&usb9pfs->tx_req_list);
+
+ spin_lock_init(&usb9pfs->lock);
+
+ f = &usb9pfs->function;
+ cdev = f->config->cdev;
+
+ client->trans = (void *)usb9pfs;
+ client->status = Connected;
+ usb9pfs->client = client;
+
+ client->trans_mod->maxsize = usb9pfs->buflen;
+
+ return 0;
+}
+
+static void usb9pfs_clear_tx(struct f_usb9pfs *usb9pfs)
+{
+ struct p9_req_t *req;
+ unsigned long flags;
+
+ spin_lock_irqsave(&usb9pfs->lock, flags);
+
+ req = usb9pfs->p9_tx_req;
+ if (!req) {
+ spin_unlock_irqrestore(&usb9pfs->lock, flags);
+ return;
+ }
+
+ if (!req->t_err)
+ req->t_err = -ECONNRESET;
+
+ p9_client_cb(usb9pfs->client, req, REQ_STATUS_ERROR);
+
+ usb9pfs->p9_tx_req = NULL;
+
+ spin_unlock_irqrestore(&usb9pfs->lock, flags);
+}
+
+static void p9_usbg_close(struct p9_client *client)
+{
+ struct f_usb9pfs *usb9pfs;
+ struct f_usb9pfs_dev *dev;
+ struct f_usb9pfs_opts *opts;
+
+ if (!client)
+ return;
+
+ usb9pfs = client->trans;
+ if (!usb9pfs)
+ return;
+
+ client->status = Disconnected;
+
+ usb9pfs_clear_tx(usb9pfs);
+
+ opts = container_of(usb9pfs->function.fi,
+ struct f_usb9pfs_opts, func_inst);
+
+ dev = opts->dev;
+
+ mutex_lock(&usb9pfs_lock);
+ dev->inuse = false;
+ mutex_unlock(&usb9pfs_lock);
+}
+
+static int p9_usbg_request(struct p9_client *client, struct p9_req_t *p9_req)
+{
+ struct f_usb9pfs *usb9pfs = client->trans;
+ unsigned long flags;
+
+ if (client->status != Connected)
+ return -EBUSY;
+
+ spin_lock_irqsave(&usb9pfs->lock, flags);
+ list_add_tail(&p9_req->req_list, &usb9pfs->tx_req_list);
+ spin_unlock_irqrestore(&usb9pfs->lock, flags);
+
+ return usb9pfs_transmit(usb9pfs);
+}
+
+static int p9_usbg_cancel(struct p9_client *client, struct p9_req_t *req)
+{
+ struct f_usb9pfs *usb9pfs = client->trans;
+ unsigned long flags;
+ int ret = 1;
+
+ p9_debug(P9_DEBUG_TRANS, "client %p req %p\n", client, req);
+
+ spin_lock_irqsave(&usb9pfs->lock, flags);
+
+ if (req->status == REQ_STATUS_UNSENT) {
+ list_del(&req->req_list);
+ WRITE_ONCE(req->status, REQ_STATUS_FLSHD);
+ p9_req_put(client, req);
+ ret = 0;
+ }
+ spin_unlock_irqrestore(&usb9pfs->lock, flags);
+
+ return ret;
+}
+
+static struct p9_trans_module p9_usbg_trans = {
+ .name = "usbg",
+ .create = p9_usbg_create,
+ .close = p9_usbg_close,
+ .request = p9_usbg_request,
+ .cancel = p9_usbg_cancel,
+ .owner = THIS_MODULE,
+};
+
+/*-------------------------------------------------------------------------*/
+
+#define USB_PROTOCOL_9PFS 0x09
+
+static struct usb_interface_descriptor usb9pfs_intf = {
+ .bLength = sizeof(usb9pfs_intf),
+ .bDescriptorType = USB_DT_INTERFACE,
+
+ .bNumEndpoints = 2,
+ .bInterfaceClass = USB_CLASS_VENDOR_SPEC,
+ .bInterfaceSubClass = USB_SUBCLASS_VENDOR_SPEC,
+ .bInterfaceProtocol = USB_PROTOCOL_9PFS,
+
+ /* .iInterface = DYNAMIC */
+};
+
+/* full speed support: */
+
+static struct usb_endpoint_descriptor fs_usb9pfs_source_desc = {
+ .bLength = USB_DT_ENDPOINT_SIZE,
+ .bDescriptorType = USB_DT_ENDPOINT,
+
+ .bEndpointAddress = USB_DIR_IN,
+ .bmAttributes = USB_ENDPOINT_XFER_BULK,
+};
+
+static struct usb_endpoint_descriptor fs_usb9pfs_sink_desc = {
+ .bLength = USB_DT_ENDPOINT_SIZE,
+ .bDescriptorType = USB_DT_ENDPOINT,
+
+ .bEndpointAddress = USB_DIR_OUT,
+ .bmAttributes = USB_ENDPOINT_XFER_BULK,
+};
+
+static struct usb_descriptor_header *fs_usb9pfs_descs[] = {
+ (struct usb_descriptor_header *)&usb9pfs_intf,
+ (struct usb_descriptor_header *)&fs_usb9pfs_sink_desc,
+ (struct usb_descriptor_header *)&fs_usb9pfs_source_desc,
+ NULL,
+};
+
+/* high speed support: */
+
+static struct usb_endpoint_descriptor hs_usb9pfs_source_desc = {
+ .bLength = USB_DT_ENDPOINT_SIZE,
+ .bDescriptorType = USB_DT_ENDPOINT,
+
+ .bmAttributes = USB_ENDPOINT_XFER_BULK,
+ .wMaxPacketSize = cpu_to_le16(512),
+};
+
+static struct usb_endpoint_descriptor hs_usb9pfs_sink_desc = {
+ .bLength = USB_DT_ENDPOINT_SIZE,
+ .bDescriptorType = USB_DT_ENDPOINT,
+
+ .bmAttributes = USB_ENDPOINT_XFER_BULK,
+ .wMaxPacketSize = cpu_to_le16(512),
+};
+
+static struct usb_descriptor_header *hs_usb9pfs_descs[] = {
+ (struct usb_descriptor_header *)&usb9pfs_intf,
+ (struct usb_descriptor_header *)&hs_usb9pfs_source_desc,
+ (struct usb_descriptor_header *)&hs_usb9pfs_sink_desc,
+ NULL,
+};
+
+/* super speed support: */
+
+static struct usb_endpoint_descriptor ss_usb9pfs_source_desc = {
+ .bLength = USB_DT_ENDPOINT_SIZE,
+ .bDescriptorType = USB_DT_ENDPOINT,
+
+ .bmAttributes = USB_ENDPOINT_XFER_BULK,
+ .wMaxPacketSize = cpu_to_le16(1024),
+};
+
+static struct usb_ss_ep_comp_descriptor ss_usb9pfs_source_comp_desc = {
+ .bLength = USB_DT_SS_EP_COMP_SIZE,
+ .bDescriptorType = USB_DT_SS_ENDPOINT_COMP,
+ .bMaxBurst = 0,
+ .bmAttributes = 0,
+ .wBytesPerInterval = 0,
+};
+
+static struct usb_endpoint_descriptor ss_usb9pfs_sink_desc = {
+ .bLength = USB_DT_ENDPOINT_SIZE,
+ .bDescriptorType = USB_DT_ENDPOINT,
+
+ .bmAttributes = USB_ENDPOINT_XFER_BULK,
+ .wMaxPacketSize = cpu_to_le16(1024),
+};
+
+static struct usb_ss_ep_comp_descriptor ss_usb9pfs_sink_comp_desc = {
+ .bLength = USB_DT_SS_EP_COMP_SIZE,
+ .bDescriptorType = USB_DT_SS_ENDPOINT_COMP,
+ .bMaxBurst = 0,
+ .bmAttributes = 0,
+ .wBytesPerInterval = 0,
+};
+
+static struct usb_descriptor_header *ss_usb9pfs_descs[] = {
+ (struct usb_descriptor_header *)&usb9pfs_intf,
+ (struct usb_descriptor_header *)&ss_usb9pfs_source_desc,
+ (struct usb_descriptor_header *)&ss_usb9pfs_source_comp_desc,
+ (struct usb_descriptor_header *)&ss_usb9pfs_sink_desc,
+ (struct usb_descriptor_header *)&ss_usb9pfs_sink_comp_desc,
+ NULL,
+};
+
+/* function-specific strings: */
+static struct usb_string strings_usb9pfs[] = {
+ [0].s = "usb9pfs input to output",
+ { } /* end of list */
+};
+
+static struct usb_gadget_strings stringtab_usb9pfs = {
+ .language = 0x0409, /* en-us */
+ .strings = strings_usb9pfs,
+};
+
+static struct usb_gadget_strings *usb9pfs_strings[] = {
+ &stringtab_usb9pfs,
+ NULL,
+};
+
+/*-------------------------------------------------------------------------*/
+
+static int usb9pfs_func_bind(struct usb_configuration *c,
+ struct usb_function *f)
+{
+ struct f_usb9pfs *usb9pfs = func_to_usb9pfs(f);
+ struct f_usb9pfs_opts *opts;
+ struct usb_composite_dev *cdev = c->cdev;
+ int ret;
+ int id;
+
+ /* allocate interface ID(s) */
+ id = usb_interface_id(c, f);
+ if (id < 0)
+ return id;
+ usb9pfs_intf.bInterfaceNumber = id;
+
+ id = usb_string_id(cdev);
+ if (id < 0)
+ return id;
+ strings_usb9pfs[0].id = id;
+ usb9pfs_intf.iInterface = id;
+
+ /* allocate endpoints */
+ usb9pfs->in_ep = usb_ep_autoconfig(cdev->gadget,
+ &fs_usb9pfs_source_desc);
+ if (!usb9pfs->in_ep)
+ goto autoconf_fail;
+
+ usb9pfs->out_ep = usb_ep_autoconfig(cdev->gadget,
+ &fs_usb9pfs_sink_desc);
+ if (!usb9pfs->out_ep)
+ goto autoconf_fail;
+
+ /* support high speed hardware */
+ hs_usb9pfs_source_desc.bEndpointAddress =
+ fs_usb9pfs_source_desc.bEndpointAddress;
+ hs_usb9pfs_sink_desc.bEndpointAddress =
+ fs_usb9pfs_sink_desc.bEndpointAddress;
+
+ /* support super speed hardware */
+ ss_usb9pfs_source_desc.bEndpointAddress =
+ fs_usb9pfs_source_desc.bEndpointAddress;
+ ss_usb9pfs_sink_desc.bEndpointAddress =
+ fs_usb9pfs_sink_desc.bEndpointAddress;
+
+ ret = usb_assign_descriptors(f, fs_usb9pfs_descs, hs_usb9pfs_descs,
+ ss_usb9pfs_descs, ss_usb9pfs_descs);
+ if (ret)
+ return ret;
+
+ opts = container_of(f->fi, struct f_usb9pfs_opts, func_inst);
+ opts->dev->usb9pfs = usb9pfs;
+
+ dev_dbg(&cdev->gadget->dev, "%s speed %s: IN/%s, OUT/%s\n",
+ (gadget_is_superspeed(c->cdev->gadget) ? "super" :
+ (gadget_is_dualspeed(c->cdev->gadget) ? "dual" : "full")),
+ f->name, usb9pfs->in_ep->name, usb9pfs->out_ep->name);
+
+ return 0;
+
+autoconf_fail:
+ ERROR(cdev, "%s: can't autoconfigure on %s\n",
+ f->name, cdev->gadget->name);
+ return -ENODEV;
+}
+
+static void usb9pfs_func_unbind(struct usb_configuration *c,
+ struct usb_function *f)
+{
+ struct f_usb9pfs *usb9pfs = func_to_usb9pfs(f);
+
+ disable_usb9pfs(usb9pfs);
+}
+
+static void usb9pfs_free_func(struct usb_function *f)
+{
+ struct f_usb9pfs *usb9pfs = func_to_usb9pfs(f);
+ struct f_usb9pfs_opts *opts;
+
+ kfree(usb9pfs);
+
+ opts = container_of(f->fi, struct f_usb9pfs_opts, func_inst);
+
+ mutex_lock(&opts->lock);
+ opts->refcnt--;
+ mutex_unlock(&opts->lock);
+
+ usb_free_all_descriptors(f);
+}
+
+static int usb9pfs_set_alt(struct usb_function *f,
+ unsigned int intf, unsigned int alt)
+{
+ struct f_usb9pfs *usb9pfs = func_to_usb9pfs(f);
+ struct usb_composite_dev *cdev = f->config->cdev;
+
+ return enable_usb9pfs(cdev, usb9pfs);
+}
+
+static void usb9pfs_disable(struct usb_function *f)
+{
+ struct f_usb9pfs *usb9pfs = func_to_usb9pfs(f);
+
+ usb9pfs_clear_tx(usb9pfs);
+}
+
+static struct usb_function *usb9pfs_alloc(struct usb_function_instance *fi)
+{
+ struct f_usb9pfs_opts *usb9pfs_opts;
+ struct f_usb9pfs *usb9pfs;
+
+ usb9pfs = kzalloc(sizeof(*usb9pfs), GFP_KERNEL);
+ if (!usb9pfs)
+ return ERR_PTR(-ENOMEM);
+
+ usb9pfs_opts = container_of(fi, struct f_usb9pfs_opts, func_inst);
+
+ mutex_lock(&usb9pfs_opts->lock);
+ usb9pfs_opts->refcnt++;
+ mutex_unlock(&usb9pfs_opts->lock);
+
+ usb9pfs->buflen = usb9pfs_opts->buflen;
+
+ usb9pfs->function.name = "usb9pfs";
+ usb9pfs->function.bind = usb9pfs_func_bind;
+ usb9pfs->function.unbind = usb9pfs_func_unbind;
+ usb9pfs->function.set_alt = usb9pfs_set_alt;
+ usb9pfs->function.disable = usb9pfs_disable;
+ usb9pfs->function.strings = usb9pfs_strings;
+
+ usb9pfs->function.free_func = usb9pfs_free_func;
+
+ return &usb9pfs->function;
+}
+
+static inline struct f_usb9pfs_opts *to_f_usb9pfs_opts(struct config_item *item)
+{
+ return container_of(to_config_group(item), struct f_usb9pfs_opts,
+ func_inst.group);
+}
+
+static inline struct f_usb9pfs_opts *fi_to_f_usb9pfs_opts(struct usb_function_instance *fi)
+{
+ return container_of(fi, struct f_usb9pfs_opts, func_inst);
+}
+
+static void usb9pfs_attr_release(struct config_item *item)
+{
+ struct f_usb9pfs_opts *usb9pfs_opts = to_f_usb9pfs_opts(item);
+
+ usb_put_function_instance(&usb9pfs_opts->func_inst);
+}
+
+static struct configfs_item_operations usb9pfs_item_ops = {
+ .release = usb9pfs_attr_release,
+};
+
+static ssize_t f_usb9pfs_opts_buflen_show(struct config_item *item, char *page)
+{
+ struct f_usb9pfs_opts *opts = to_f_usb9pfs_opts(item);
+ int result;
+
+ mutex_lock(&opts->lock);
+ result = sprintf(page, "%d\n", opts->buflen);
+ mutex_unlock(&opts->lock);
+
+ return result;
+}
+
+static ssize_t f_usb9pfs_opts_buflen_store(struct config_item *item,
+ const char *page, size_t len)
+{
+ struct f_usb9pfs_opts *opts = to_f_usb9pfs_opts(item);
+ int ret;
+ u32 num;
+
+ mutex_lock(&opts->lock);
+ if (opts->refcnt) {
+ ret = -EBUSY;
+ goto end;
+ }
+
+ ret = kstrtou32(page, 0, &num);
+ if (ret)
+ goto end;
+
+ opts->buflen = num;
+ ret = len;
+end:
+ mutex_unlock(&opts->lock);
+ return ret;
+}
+
+CONFIGFS_ATTR(f_usb9pfs_opts_, buflen);
+
+static struct configfs_attribute *usb9pfs_attrs[] = {
+ &f_usb9pfs_opts_attr_buflen,
+ NULL,
+};
+
+static const struct config_item_type usb9pfs_func_type = {
+ .ct_item_ops = &usb9pfs_item_ops,
+ .ct_attrs = usb9pfs_attrs,
+ .ct_owner = THIS_MODULE,
+};
+
+static struct f_usb9pfs_dev *_usb9pfs_do_find_dev(const char *tag)
+{
+ struct f_usb9pfs_dev *usb9pfs_dev;
+ struct f_usb9pfs_dev *tmp;
+
+ if (!tag)
+ return NULL;
+
+ list_for_each_entry_safe(usb9pfs_dev, tmp, &usbg_instance_list, usb9pfs_instance) {
+ if (strcmp(usb9pfs_dev->tag, tag) == 0)
+ return usb9pfs_dev;
+ }
+
+ return NULL;
+}
+
+static int usb9pfs_tag_instance(struct f_usb9pfs_dev *dev, const char *tag)
+{
+ struct f_usb9pfs_dev *existing;
+ int ret = 0;
+
+ mutex_lock(&usb9pfs_lock);
+ existing = _usb9pfs_do_find_dev(tag);
+ if (!existing)
+ strscpy(dev->tag, tag, ARRAY_SIZE(dev->tag));
+ else if (existing != dev)
+ ret = -EBUSY;
+ mutex_unlock(&usb9pfs_lock);
+
+ return ret;
+}
+
+static int usb9pfs_set_inst_tag(struct usb_function_instance *fi, const char *tag)
+{
+ if (strlen(tag) >= sizeof_field(struct f_usb9pfs_dev, tag))
+ return -ENAMETOOLONG;
+ return usb9pfs_tag_instance(fi_to_f_usb9pfs_opts(fi)->dev, tag);
+}
+
+static void usb9pfs_free_instance(struct usb_function_instance *fi)
+{
+ struct f_usb9pfs_opts *usb9pfs_opts =
+ container_of(fi, struct f_usb9pfs_opts, func_inst);
+ struct f_usb9pfs_dev *dev = usb9pfs_opts->dev;
+
+ list_del(&dev->usb9pfs_instance);
+ kfree(usb9pfs_opts);
+}
+
+static struct usb_function_instance *usb9pfs_alloc_instance(void)
+{
+ struct f_usb9pfs_opts *usb9pfs_opts;
+ struct f_usb9pfs_dev *dev;
+
+ usb9pfs_opts = kzalloc(sizeof(*usb9pfs_opts), GFP_KERNEL);
+ if (!usb9pfs_opts)
+ return ERR_PTR(-ENOMEM);
+
+ mutex_init(&usb9pfs_opts->lock);
+
+ usb9pfs_opts->func_inst.set_inst_name = usb9pfs_set_inst_tag;
+ usb9pfs_opts->func_inst.free_func_inst = usb9pfs_free_instance;
+
+ usb9pfs_opts->buflen = DEFAULT_BUFLEN;
+
+ mutex_lock(&usb9pfs_lock);
+ dev = kzalloc(sizeof(*dev), GFP_KERNEL);
+ if (IS_ERR(dev)) {
+ mutex_unlock(&usb9pfs_lock);
+ kfree(usb9pfs_opts);
+ return ERR_CAST(dev);
+ }
+ list_add_tail(&dev->usb9pfs_instance, &usbg_instance_list);
+ mutex_unlock(&usb9pfs_lock);
+
+ usb9pfs_opts->dev = dev;
+ dev->opts = usb9pfs_opts;
+
+ config_group_init_type_name(&usb9pfs_opts->func_inst.group, "",
+ &usb9pfs_func_type);
+
+ return &usb9pfs_opts->func_inst;
+}
+DECLARE_USB_FUNCTION(usb9pfs, usb9pfs_alloc_instance, usb9pfs_alloc);
+
+static int __init usb9pfs_modinit(void)
+{
+ int ret;
+
+ INIT_LIST_HEAD(&usbg_instance_list);
+
+ ret = usb_function_register(&usb9pfsusb_func);
+ if (!ret)
+ v9fs_register_trans(&p9_usbg_trans);
+
+ return ret;
+}
+
+static void __exit usb9pfs_modexit(void)
+{
+ usb_function_unregister(&usb9pfsusb_func);
+ v9fs_unregister_trans(&p9_usbg_trans);
+}
+
+module_init(usb9pfs_modinit);
+module_exit(usb9pfs_modexit);
+
+MODULE_ALIAS_9P("usbg");
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("USB gadget 9pfs transport");
+MODULE_AUTHOR("Michael Grzeschik");

--
2.39.2


2024-05-28 18:29:48

by Andrzej Pietrasiewicz

[permalink] [raw]
Subject: Re: [PATCH v5 2/3] net/9p/usbg: Add new usb gadget function transport

Hi,

W dniu 28.05.2024 o 00:08, Michael Grzeschik pisze:
> Add the new gadget function for 9pfs transport. This function is
> defining an simple 9pfs transport interface that consists of one in and
> one out endpoint. The endpoints transmit and receive the 9pfs protocol
> payload when mounting a 9p filesystem over usb.
>
> Signed-off-by: Michael Grzeschik <[email protected]>

Tested-by: Andrzej Pietrasiewicz <[email protected]>

>
> ---
> v4 -> v5:
> - remove instance from instance list on free
> - bail out if no req was queued in disable
> - free requests on disable
> - use irqsave/restore in other functions then the completion handlers
> - simplify the completion handlers
> - reset tx_req pointer to NULL after the status update
> - add unbind function
> - use the right lock when protecting 9p requets
> - refactor giveback of tx_request to extra function clear_tx
> - enable endpoints once on set_alt
> - only disable endpoints on unbind
> - reset request pointers on free
> - remove req_lock which is not necessary since only one transfer can
> be queued for now
> v3 -> v4:
> - implemented conn_cancel
> - moved from mount_tag to device name like ffs does
> - renamed function_list to usb9pfs_instance
> - renamed usbg_function_list to usb9pfs_instance_list
> - renamed rx/tx_fill to queue_rx/tx
> - added use of retvals
> - added locking comment over usb9pfs_transmit
> - using if_else instead of two pathed switch calls
> - fixed return values of completion handler
> - using void pointer as parameter in rx_header
> - added a missed req_put in rx_header
> - removed extra disable function but call disable_ep directly
> - repaired several return values
> - remove the suspicious disable / enable path in usbg_create
> v2 -> v3: -
> v1 -> v2:
> - move the driver to net/9p/trans_usbg.c
> - fixed the commit title
> - fix %zu usage in p9_debug for size_t type
> - define functions static that are only used localy
> - return 0 in alloc_requests
> - use id_alloc_max instead of ida_simple_get
> - call ida_free in usb9pfs_free_func
> - call kfree for usb9pfs->tag and usb9pfs in usb9pfs_free_func
> - add MODULE_ALIAS_9P to load this module automatic when mounting
> - improved the documentation
> - added depends on USB_GADGET in Kconfig
> ---
> Documentation/filesystems/9p.rst | 17 +-
> net/9p/Kconfig | 6 +
> net/9p/Makefile | 4 +
> net/9p/trans_usbg.c | 974 +++++++++++++++++++++++++++++++++++++++
> 4 files changed, 1000 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/filesystems/9p.rst b/Documentation/filesystems/9p.rst
> index 1e0e0bb6fdf91..10cf79dc287f8 100644
> --- a/Documentation/filesystems/9p.rst
> +++ b/Documentation/filesystems/9p.rst
> @@ -48,11 +48,25 @@ For server running on QEMU host with virtio transport::
>
> mount -t 9p -o trans=virtio <mount_tag> /mnt/9
>
> -where mount_tag is the tag associated by the server to each of the exported
> +where mount_tag is the tag generated by the server to each of the exported
> mount points. Each 9P export is seen by the client as a virtio device with an
> associated "mount_tag" property. Available mount tags can be
> seen by reading /sys/bus/virtio/drivers/9pnet_virtio/virtio<n>/mount_tag files.
>
> +USBG Usage
> +==========
> +
> +To mount a 9p FS on a USB Host accessible via the gadget at runtime::
> +
> + mount -t 9p -o trans=usbg,aname=/path/to/fs <device> /mnt/9
> +
> +To mount a 9p FS on a USB Host accessible via the gadget as root filesystem::
> +
> + root=<device> rootfstype=9p rootflags=trans=usbg,cache=loose,uname=root,access=0,dfltuid=0,dfltgid=0,aname=/path/to/rootfs
> +
> +where <device> is the tag associated by the usb gadget transport.
> +It is defined by the configfs instance name.
> +
> Options
> =======
>
> @@ -68,6 +82,7 @@ Options
> virtio connect to the next virtio channel available
> (from QEMU with trans_virtio module)
> rdma connect to a specified RDMA channel
> + usbg connect to a specified usb gadget channel
> ======== ============================================
>
> uname=name user name to attempt mount as on the remote server. The
> diff --git a/net/9p/Kconfig b/net/9p/Kconfig
> index bcdab9c23b402..2d7e596e22c3f 100644
> --- a/net/9p/Kconfig
> +++ b/net/9p/Kconfig
> @@ -40,6 +40,12 @@ config NET_9P_XEN
> This builds support for a transport for 9pfs between
> two Xen domains.
>
> +config NET_9P_USBG
> + bool "9P USB Gadget Transport"
> + depends on USB_GADGET
> + help
> + This builds support for a transport for 9pfs over
> + usb gadget.
>
> config NET_9P_RDMA
> depends on INET && INFINIBAND && INFINIBAND_ADDR_TRANS
> diff --git a/net/9p/Makefile b/net/9p/Makefile
> index 1df9b344c30bd..22794a451c3f7 100644
> --- a/net/9p/Makefile
> +++ b/net/9p/Makefile
> @@ -4,6 +4,7 @@ obj-$(CONFIG_NET_9P_FD) += 9pnet_fd.o
> obj-$(CONFIG_NET_9P_XEN) += 9pnet_xen.o
> obj-$(CONFIG_NET_9P_VIRTIO) += 9pnet_virtio.o
> obj-$(CONFIG_NET_9P_RDMA) += 9pnet_rdma.o
> +obj-$(CONFIG_NET_9P_USBG) += 9pnet_usbg.o
>
> 9pnet-objs := \
> mod.o \
> @@ -23,3 +24,6 @@ obj-$(CONFIG_NET_9P_RDMA) += 9pnet_rdma.o
>
> 9pnet_rdma-objs := \
> trans_rdma.o \
> +
> +9pnet_usbg-objs := \
> + trans_usbg.o \
> diff --git a/net/9p/trans_usbg.c b/net/9p/trans_usbg.c
> new file mode 100644
> index 0000000000000..3a84097256837
> --- /dev/null
> +++ b/net/9p/trans_usbg.c
> @@ -0,0 +1,974 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * trans_usbg.c - USB peripheral usb9pfs configuration driver and transport.
> + *
> + * Copyright (C) 2024 Michael Grzeschik <[email protected]>
> + */
> +
> +/* Gadget usb9pfs only needs two bulk endpoints, and will use the usb9pfs
> + * transport to mount host exported filesystem via usb gadget.
> + */
> +
> +/* +--------------------------+ | +--------------------------+
> + * | 9PFS mounting client | | | 9PFS exporting server |
> + * SW | | | | |
> + * | (this:trans_usbg) | | |(e.g. diod or nfs-ganesha)|
> + * +-------------^------------+ | +-------------^------------+
> + * | | |
> + * ------------------|------------------------------------|-------------
> + * | | |
> + * +-------------v------------+ | +-------------v------------+
> + * | | | | |
> + * HW | USB Device Controller <---------> USB Host Controller |
> + * | | | | |
> + * +--------------------------+ | +--------------------------+
> + */
> +
> +#include <linux/slab.h>
> +#include <linux/kernel.h>
> +#include <linux/device.h>
> +#include <linux/module.h>
> +#include <linux/err.h>
> +#include <linux/usb/composite.h>
> +#include <linux/usb/func_utils.h>
> +
> +#include <net/9p/9p.h>
> +#include <net/9p/client.h>
> +#include <net/9p/transport.h>
> +
> +#define DEFAULT_BUFLEN 16384
> +
> +struct f_usb9pfs {
> + struct p9_client *client;
> +
> + struct p9_req_t *p9_tx_req;
> +
> + struct list_head tx_req_list;
> +
> + /* 9p request lock for en/dequeue */
> + spinlock_t lock;
> +
> + struct usb_request *in_req;
> + struct usb_request *out_req;
> +
> + struct usb_ep *in_ep;
> + struct usb_ep *out_ep;
> +
> + unsigned int buflen;
> +
> + struct usb_function function;
> +};
> +
> +static inline struct f_usb9pfs *func_to_usb9pfs(struct usb_function *f)
> +{
> + return container_of(f, struct f_usb9pfs, function);
> +}
> +
> +struct f_usb9pfs_opts {
> + struct usb_function_instance func_inst;
> + unsigned int buflen;
> +
> + struct f_usb9pfs_dev *dev;
> +
> + /* Read/write access to configfs attributes is handled by configfs.
> + *
> + * This is to protect the data from concurrent access by read/write
> + * and create symlink/remove symlink.
> + */
> + struct mutex lock;
> + int refcnt;
> +};
> +
> +struct f_usb9pfs_dev {
> + struct f_usb9pfs *usb9pfs;
> + struct f_usb9pfs_opts *opts;
> + char tag[41];
> + bool inuse;
> +
> + struct list_head usb9pfs_instance;
> +};
> +
> +static DEFINE_MUTEX(usb9pfs_lock);
> +static struct list_head usbg_instance_list;
> +
> +static int usb9pfs_queue_tx(struct f_usb9pfs *usb9pfs, struct usb_request *req,
> + gfp_t gfp_flags)
> +{
> + struct usb_composite_dev *cdev = usb9pfs->function.config->cdev;
> + int ret = -ENOMEM;
> +
> + if (!(usb9pfs->p9_tx_req->tc.size % usb9pfs->in_ep->maxpacket))
> + req->zero = 1;
> +
> + req->buf = usb9pfs->p9_tx_req->tc.sdata;
> + req->length = usb9pfs->p9_tx_req->tc.size;
> +
> + dev_dbg(&cdev->gadget->dev, "%s usb9pfs send --> %d/%d, zero: %d\n",
> + usb9pfs->in_ep->name, req->actual, req->length, req->zero);
> +
> + ret = usb_ep_queue(usb9pfs->in_ep, req, gfp_flags);
> +
> + dev_dbg(&cdev->gadget->dev, "tx submit --> %d\n", ret);
> +
> + return ret;
> +}
> +
> +static int usb9pfs_queue_rx(struct f_usb9pfs *usb9pfs, struct usb_request *req,
> + gfp_t gfp_flags)
> +{
> + struct usb_composite_dev *cdev = usb9pfs->function.config->cdev;
> + int ret = -ENOMEM;
> +
> + ret = usb_ep_queue(usb9pfs->out_ep, req, gfp_flags);
> +
> + dev_dbg(&cdev->gadget->dev, "rx submit --> %d\n", ret);
> +
> + return ret;
> +}
> +
> +static int usb9pfs_transmit(struct f_usb9pfs *usb9pfs)
> +{
> + struct p9_req_t *p9_req = NULL;
> + unsigned long flags;
> + int ret = 0;
> +
> + spin_lock_irqsave(&usb9pfs->lock, flags);
> +
> + if (usb9pfs->p9_tx_req) {
> + spin_unlock_irqrestore(&usb9pfs->lock, flags);
> + return -EBUSY;
> + }
> +
> + p9_req = list_first_entry_or_null(&usb9pfs->tx_req_list,
> + struct p9_req_t, req_list);
> + if (!p9_req) {
> + spin_unlock_irqrestore(&usb9pfs->lock, flags);
> + return -ENOENT;
> + }
> +
> + list_del(&p9_req->req_list);
> +
> + usb9pfs->p9_tx_req = p9_req;
> +
> + p9_req_get(usb9pfs->p9_tx_req);
> +
> + ret = usb9pfs_queue_tx(usb9pfs, usb9pfs->in_req, GFP_ATOMIC);
> +
> + spin_unlock_irqrestore(&usb9pfs->lock, flags);
> +
> + return ret;
> +}
> +
> +static void usb9pfs_tx_complete(struct usb_ep *ep, struct usb_request *req)
> +{
> + struct f_usb9pfs *usb9pfs = ep->driver_data;
> + struct usb_composite_dev *cdev = usb9pfs->function.config->cdev;
> + int ret = 0;
> +
> + if (req->status) {
> + dev_err(&cdev->gadget->dev, "%s usb9pfs complete --> %d, %d/%d\n",
> + ep->name, req->status, req->actual, req->length);
> + return;
> + }
> +
> + /* reset zero packages */
> + req->zero = 0;
> +
> + dev_dbg(&cdev->gadget->dev, "%s usb9pfs complete --> %d, %d/%d\n",
> + ep->name, req->status, req->actual, req->length);
> +
> + WRITE_ONCE(usb9pfs->p9_tx_req->status, REQ_STATUS_SENT);
> +
> + p9_req_put(usb9pfs->client, usb9pfs->p9_tx_req);
> +
> + ret = usb9pfs_queue_rx(usb9pfs, usb9pfs->out_req, GFP_ATOMIC);
> + if (ret)
> + return;
> +
> + return;
> +}
> +
> +static struct p9_req_t *usb9pfs_rx_header(struct f_usb9pfs *usb9pfs, void *buf)
> +{
> + struct p9_req_t *p9_rx_req;
> + struct p9_fcall rc;
> + int ret;
> +
> + /* start by reading header */
> + rc.sdata = buf;
> + rc.offset = 0;
> + rc.capacity = P9_HDRSZ;
> + rc.size = P9_HDRSZ;
> +
> + p9_debug(P9_DEBUG_TRANS, "mux %p got %zu bytes\n", usb9pfs,
> + rc.capacity - rc.offset);
> +
> + ret = p9_parse_header(&rc, &rc.size, NULL, NULL, 0);
> + if (ret) {
> + p9_debug(P9_DEBUG_ERROR,
> + "error parsing header: %d\n", ret);
> + return NULL;
> + }
> +
> + p9_debug(P9_DEBUG_TRANS,
> + "mux %p pkt: size: %d bytes tag: %d\n",
> + usb9pfs, rc.size, rc.tag);
> +
> + p9_rx_req = p9_tag_lookup(usb9pfs->client, rc.tag);
> + if (!p9_rx_req || p9_rx_req->status != REQ_STATUS_SENT) {
> + p9_debug(P9_DEBUG_ERROR, "Unexpected packet tag %d\n", rc.tag);
> + return NULL;
> + }
> +
> + if (rc.size > p9_rx_req->rc.capacity) {
> + p9_debug(P9_DEBUG_ERROR,
> + "requested packet size too big: %d for tag %d with capacity %zd\n",
> + rc.size, rc.tag, p9_rx_req->rc.capacity);
> + p9_req_put(usb9pfs->client, p9_rx_req);
> + return NULL;
> + }
> +
> + if (!p9_rx_req->rc.sdata) {
> + p9_debug(P9_DEBUG_ERROR,
> + "No recv fcall for tag %d (req %p), disconnecting!\n",
> + rc.tag, p9_rx_req);
> + p9_req_put(usb9pfs->client, p9_rx_req);
> + return NULL;
> + }
> +
> + return p9_rx_req;
> +}
> +
> +static void usb9pfs_rx_complete(struct usb_ep *ep, struct usb_request *req)
> +{
> + struct f_usb9pfs *usb9pfs = ep->driver_data;
> + struct usb_composite_dev *cdev = usb9pfs->function.config->cdev;
> + struct p9_req_t *p9_rx_req;
> + unsigned long flags;
> +
> + if (req->status) {
> + dev_err(&cdev->gadget->dev, "%s usb9pfs complete --> %d, %d/%d\n",
> + ep->name, req->status, req->actual, req->length);
> + return;
> + }
> +
> + p9_rx_req = usb9pfs_rx_header(usb9pfs, req->buf);
> + if (!p9_rx_req)
> + return;
> +
> + memcpy(p9_rx_req->rc.sdata, req->buf, req->actual);
> +
> + p9_rx_req->rc.size = req->actual;
> +
> + p9_client_cb(usb9pfs->client, p9_rx_req, REQ_STATUS_RCVD);
> + p9_req_put(usb9pfs->client, p9_rx_req);
> +
> + spin_lock_irqsave(&usb9pfs->lock, flags);
> + usb9pfs->p9_tx_req = NULL;
> +
> + spin_unlock_irqrestore(&usb9pfs->lock, flags);
> +
> + usb9pfs_transmit(usb9pfs);
> +
> + return;
> +}
> +
> +static void disable_ep(struct usb_composite_dev *cdev, struct usb_ep *ep)
> +{
> + int value;
> +
> + value = usb_ep_disable(ep);
> + if (value < 0)
> + dev_info(&cdev->gadget->dev,
> + "disable %s --> %d\n", ep->name, value);
> +}
> +
> +static void disable_usb9pfs(struct f_usb9pfs *usb9pfs)
> +{
> + struct usb_composite_dev *cdev =
> + usb9pfs->function.config->cdev;
> +
> + if (usb9pfs->in_req) {
> + usb_ep_free_request(usb9pfs->in_ep, usb9pfs->in_req);
> + usb9pfs->in_req = NULL;
> + }
> +
> + if (usb9pfs->out_req) {
> + usb_ep_free_request(usb9pfs->out_ep, usb9pfs->out_req);
> + usb9pfs->out_req = NULL;
> + }
> +
> + disable_ep(cdev, usb9pfs->in_ep);
> + disable_ep(cdev, usb9pfs->out_ep);
> + dev_dbg(&cdev->gadget->dev, "%s disabled\n",
> + usb9pfs->function.name);
> +}
> +
> +static int alloc_requests(struct usb_composite_dev *cdev,
> + struct f_usb9pfs *usb9pfs)
> +{
> + int ret = 0;
> +
> + usb9pfs->in_req = usb_ep_alloc_request(usb9pfs->in_ep, GFP_ATOMIC);
> + if (!usb9pfs->in_req) {
> + ret = -ENOENT;
> + goto fail;
> + }
> +
> + usb9pfs->out_req = alloc_ep_req(usb9pfs->out_ep, usb9pfs->buflen);
> + if (!usb9pfs->out_req) {
> + ret = -ENOENT;
> + goto fail_in;
> + }
> +
> + usb9pfs->in_req->complete = usb9pfs_tx_complete;
> + usb9pfs->out_req->complete = usb9pfs_rx_complete;
> +
> + /* length will be set in complete routine */
> + usb9pfs->in_req->context = usb9pfs;
> + usb9pfs->out_req->context = usb9pfs;
> +
> + return ret;
> +
> +fail_in:
> + usb_ep_free_request(usb9pfs->in_ep, usb9pfs->in_req);
> +fail:
> + return ret;
> +}
> +
> +static int enable_endpoint(struct usb_composite_dev *cdev,
> + struct f_usb9pfs *usb9pfs, struct usb_ep *ep)
> +{
> + int ret;
> +
> + ret = config_ep_by_speed(cdev->gadget, &usb9pfs->function, ep);
> + if (ret)
> + return ret;
> +
> + ret = usb_ep_enable(ep);
> + if (ret < 0)
> + return ret;
> +
> + ep->driver_data = usb9pfs;
> +
> + return ret;
> +}
> +
> +static int
> +enable_usb9pfs(struct usb_composite_dev *cdev, struct f_usb9pfs *usb9pfs)
> +{
> + int result = 0;
> +
> + result = enable_endpoint(cdev, usb9pfs, usb9pfs->in_ep);
> + if (result)
> + goto out;
> +
> + result = enable_endpoint(cdev, usb9pfs, usb9pfs->out_ep);
> + if (result)
> + goto disable_in;
> +
> + result = alloc_requests(cdev, usb9pfs);
> + if (result)
> + goto disable_out;
> +
> + dev_dbg(&cdev->gadget->dev, "%s enabled\n", usb9pfs->function.name);
> + return 0;
> +
> +disable_out:
> + usb_ep_disable(usb9pfs->out_ep);
> +disable_in:
> + usb_ep_disable(usb9pfs->in_ep);
> +out:
> + return result;
> +}
> +
> +static int p9_usbg_create(struct p9_client *client, const char *devname, char *args)
> +{
> + struct usb_composite_dev *cdev;
> + struct f_usb9pfs_dev *dev;
> + struct f_usb9pfs_dev *tmp;
> + struct f_usb9pfs *usb9pfs;
> + struct usb_function *f;
> + int ret = -ENOENT;
> + int found = 0;
> +
> + if (!devname)
> + return -EINVAL;
> +
> + mutex_lock(&usb9pfs_lock);
> + list_for_each_entry_safe(dev, tmp, &usbg_instance_list, usb9pfs_instance) {
> + if (!strncmp(devname, dev->tag, strlen(devname))) {
> + if (!dev->inuse) {
> + dev->inuse = true;
> + found = 1;
> + break;
> + }
> + ret = -EBUSY;
> + break;
> + }
> + }
> + mutex_unlock(&usb9pfs_lock);
> +
> + if (!found) {
> + pr_err("no channels available for device %s\n", devname);
> + return ret;
> + }
> +
> + usb9pfs = dev->usb9pfs;
> + if (!usb9pfs)
> + return -EINVAL;
> +
> + INIT_LIST_HEAD(&usb9pfs->tx_req_list);
> +
> + spin_lock_init(&usb9pfs->lock);
> +
> + f = &usb9pfs->function;
> + cdev = f->config->cdev;
> +
> + client->trans = (void *)usb9pfs;
> + client->status = Connected;
> + usb9pfs->client = client;
> +
> + client->trans_mod->maxsize = usb9pfs->buflen;
> +
> + return 0;
> +}
> +
> +static void usb9pfs_clear_tx(struct f_usb9pfs *usb9pfs)
> +{
> + struct p9_req_t *req;
> + unsigned long flags;
> +
> + spin_lock_irqsave(&usb9pfs->lock, flags);
> +
> + req = usb9pfs->p9_tx_req;
> + if (!req) {
> + spin_unlock_irqrestore(&usb9pfs->lock, flags);
> + return;
> + }
> +
> + if (!req->t_err)
> + req->t_err = -ECONNRESET;
> +
> + p9_client_cb(usb9pfs->client, req, REQ_STATUS_ERROR);
> +
> + usb9pfs->p9_tx_req = NULL;
> +
> + spin_unlock_irqrestore(&usb9pfs->lock, flags);
> +}
> +
> +static void p9_usbg_close(struct p9_client *client)
> +{
> + struct f_usb9pfs *usb9pfs;
> + struct f_usb9pfs_dev *dev;
> + struct f_usb9pfs_opts *opts;
> +
> + if (!client)
> + return;
> +
> + usb9pfs = client->trans;
> + if (!usb9pfs)
> + return;
> +
> + client->status = Disconnected;
> +
> + usb9pfs_clear_tx(usb9pfs);
> +
> + opts = container_of(usb9pfs->function.fi,
> + struct f_usb9pfs_opts, func_inst);
> +
> + dev = opts->dev;
> +
> + mutex_lock(&usb9pfs_lock);
> + dev->inuse = false;
> + mutex_unlock(&usb9pfs_lock);
> +}
> +
> +static int p9_usbg_request(struct p9_client *client, struct p9_req_t *p9_req)
> +{
> + struct f_usb9pfs *usb9pfs = client->trans;
> + unsigned long flags;
> +
> + if (client->status != Connected)
> + return -EBUSY;
> +
> + spin_lock_irqsave(&usb9pfs->lock, flags);
> + list_add_tail(&p9_req->req_list, &usb9pfs->tx_req_list);
> + spin_unlock_irqrestore(&usb9pfs->lock, flags);
> +
> + return usb9pfs_transmit(usb9pfs);
> +}
> +
> +static int p9_usbg_cancel(struct p9_client *client, struct p9_req_t *req)
> +{
> + struct f_usb9pfs *usb9pfs = client->trans;
> + unsigned long flags;
> + int ret = 1;
> +
> + p9_debug(P9_DEBUG_TRANS, "client %p req %p\n", client, req);
> +
> + spin_lock_irqsave(&usb9pfs->lock, flags);
> +
> + if (req->status == REQ_STATUS_UNSENT) {
> + list_del(&req->req_list);
> + WRITE_ONCE(req->status, REQ_STATUS_FLSHD);
> + p9_req_put(client, req);
> + ret = 0;
> + }
> + spin_unlock_irqrestore(&usb9pfs->lock, flags);
> +
> + return ret;
> +}
> +
> +static struct p9_trans_module p9_usbg_trans = {
> + .name = "usbg",
> + .create = p9_usbg_create,
> + .close = p9_usbg_close,
> + .request = p9_usbg_request,
> + .cancel = p9_usbg_cancel,
> + .owner = THIS_MODULE,
> +};
> +
> +/*-------------------------------------------------------------------------*/
> +
> +#define USB_PROTOCOL_9PFS 0x09
> +
> +static struct usb_interface_descriptor usb9pfs_intf = {
> + .bLength = sizeof(usb9pfs_intf),
> + .bDescriptorType = USB_DT_INTERFACE,
> +
> + .bNumEndpoints = 2,
> + .bInterfaceClass = USB_CLASS_VENDOR_SPEC,
> + .bInterfaceSubClass = USB_SUBCLASS_VENDOR_SPEC,
> + .bInterfaceProtocol = USB_PROTOCOL_9PFS,
> +
> + /* .iInterface = DYNAMIC */
> +};
> +
> +/* full speed support: */
> +
> +static struct usb_endpoint_descriptor fs_usb9pfs_source_desc = {
> + .bLength = USB_DT_ENDPOINT_SIZE,
> + .bDescriptorType = USB_DT_ENDPOINT,
> +
> + .bEndpointAddress = USB_DIR_IN,
> + .bmAttributes = USB_ENDPOINT_XFER_BULK,
> +};
> +
> +static struct usb_endpoint_descriptor fs_usb9pfs_sink_desc = {
> + .bLength = USB_DT_ENDPOINT_SIZE,
> + .bDescriptorType = USB_DT_ENDPOINT,
> +
> + .bEndpointAddress = USB_DIR_OUT,
> + .bmAttributes = USB_ENDPOINT_XFER_BULK,
> +};
> +
> +static struct usb_descriptor_header *fs_usb9pfs_descs[] = {
> + (struct usb_descriptor_header *)&usb9pfs_intf,
> + (struct usb_descriptor_header *)&fs_usb9pfs_sink_desc,
> + (struct usb_descriptor_header *)&fs_usb9pfs_source_desc,
> + NULL,
> +};
> +
> +/* high speed support: */
> +
> +static struct usb_endpoint_descriptor hs_usb9pfs_source_desc = {
> + .bLength = USB_DT_ENDPOINT_SIZE,
> + .bDescriptorType = USB_DT_ENDPOINT,
> +
> + .bmAttributes = USB_ENDPOINT_XFER_BULK,
> + .wMaxPacketSize = cpu_to_le16(512),
> +};
> +
> +static struct usb_endpoint_descriptor hs_usb9pfs_sink_desc = {
> + .bLength = USB_DT_ENDPOINT_SIZE,
> + .bDescriptorType = USB_DT_ENDPOINT,
> +
> + .bmAttributes = USB_ENDPOINT_XFER_BULK,
> + .wMaxPacketSize = cpu_to_le16(512),
> +};
> +
> +static struct usb_descriptor_header *hs_usb9pfs_descs[] = {
> + (struct usb_descriptor_header *)&usb9pfs_intf,
> + (struct usb_descriptor_header *)&hs_usb9pfs_source_desc,
> + (struct usb_descriptor_header *)&hs_usb9pfs_sink_desc,
> + NULL,
> +};
> +
> +/* super speed support: */
> +
> +static struct usb_endpoint_descriptor ss_usb9pfs_source_desc = {
> + .bLength = USB_DT_ENDPOINT_SIZE,
> + .bDescriptorType = USB_DT_ENDPOINT,
> +
> + .bmAttributes = USB_ENDPOINT_XFER_BULK,
> + .wMaxPacketSize = cpu_to_le16(1024),
> +};
> +
> +static struct usb_ss_ep_comp_descriptor ss_usb9pfs_source_comp_desc = {
> + .bLength = USB_DT_SS_EP_COMP_SIZE,
> + .bDescriptorType = USB_DT_SS_ENDPOINT_COMP,
> + .bMaxBurst = 0,
> + .bmAttributes = 0,
> + .wBytesPerInterval = 0,
> +};
> +
> +static struct usb_endpoint_descriptor ss_usb9pfs_sink_desc = {
> + .bLength = USB_DT_ENDPOINT_SIZE,
> + .bDescriptorType = USB_DT_ENDPOINT,
> +
> + .bmAttributes = USB_ENDPOINT_XFER_BULK,
> + .wMaxPacketSize = cpu_to_le16(1024),
> +};
> +
> +static struct usb_ss_ep_comp_descriptor ss_usb9pfs_sink_comp_desc = {
> + .bLength = USB_DT_SS_EP_COMP_SIZE,
> + .bDescriptorType = USB_DT_SS_ENDPOINT_COMP,
> + .bMaxBurst = 0,
> + .bmAttributes = 0,
> + .wBytesPerInterval = 0,
> +};
> +
> +static struct usb_descriptor_header *ss_usb9pfs_descs[] = {
> + (struct usb_descriptor_header *)&usb9pfs_intf,
> + (struct usb_descriptor_header *)&ss_usb9pfs_source_desc,
> + (struct usb_descriptor_header *)&ss_usb9pfs_source_comp_desc,
> + (struct usb_descriptor_header *)&ss_usb9pfs_sink_desc,
> + (struct usb_descriptor_header *)&ss_usb9pfs_sink_comp_desc,
> + NULL,
> +};
> +
> +/* function-specific strings: */
> +static struct usb_string strings_usb9pfs[] = {
> + [0].s = "usb9pfs input to output",
> + { } /* end of list */
> +};
> +
> +static struct usb_gadget_strings stringtab_usb9pfs = {
> + .language = 0x0409, /* en-us */
> + .strings = strings_usb9pfs,
> +};
> +
> +static struct usb_gadget_strings *usb9pfs_strings[] = {
> + &stringtab_usb9pfs,
> + NULL,
> +};
> +
> +/*-------------------------------------------------------------------------*/
> +
> +static int usb9pfs_func_bind(struct usb_configuration *c,
> + struct usb_function *f)
> +{
> + struct f_usb9pfs *usb9pfs = func_to_usb9pfs(f);
> + struct f_usb9pfs_opts *opts;
> + struct usb_composite_dev *cdev = c->cdev;
> + int ret;
> + int id;
> +
> + /* allocate interface ID(s) */
> + id = usb_interface_id(c, f);
> + if (id < 0)
> + return id;
> + usb9pfs_intf.bInterfaceNumber = id;
> +
> + id = usb_string_id(cdev);
> + if (id < 0)
> + return id;
> + strings_usb9pfs[0].id = id;
> + usb9pfs_intf.iInterface = id;
> +
> + /* allocate endpoints */
> + usb9pfs->in_ep = usb_ep_autoconfig(cdev->gadget,
> + &fs_usb9pfs_source_desc);
> + if (!usb9pfs->in_ep)
> + goto autoconf_fail;
> +
> + usb9pfs->out_ep = usb_ep_autoconfig(cdev->gadget,
> + &fs_usb9pfs_sink_desc);
> + if (!usb9pfs->out_ep)
> + goto autoconf_fail;
> +
> + /* support high speed hardware */
> + hs_usb9pfs_source_desc.bEndpointAddress =
> + fs_usb9pfs_source_desc.bEndpointAddress;
> + hs_usb9pfs_sink_desc.bEndpointAddress =
> + fs_usb9pfs_sink_desc.bEndpointAddress;
> +
> + /* support super speed hardware */
> + ss_usb9pfs_source_desc.bEndpointAddress =
> + fs_usb9pfs_source_desc.bEndpointAddress;
> + ss_usb9pfs_sink_desc.bEndpointAddress =
> + fs_usb9pfs_sink_desc.bEndpointAddress;
> +
> + ret = usb_assign_descriptors(f, fs_usb9pfs_descs, hs_usb9pfs_descs,
> + ss_usb9pfs_descs, ss_usb9pfs_descs);
> + if (ret)
> + return ret;
> +
> + opts = container_of(f->fi, struct f_usb9pfs_opts, func_inst);
> + opts->dev->usb9pfs = usb9pfs;
> +
> + dev_dbg(&cdev->gadget->dev, "%s speed %s: IN/%s, OUT/%s\n",
> + (gadget_is_superspeed(c->cdev->gadget) ? "super" :
> + (gadget_is_dualspeed(c->cdev->gadget) ? "dual" : "full")),
> + f->name, usb9pfs->in_ep->name, usb9pfs->out_ep->name);
> +
> + return 0;
> +
> +autoconf_fail:
> + ERROR(cdev, "%s: can't autoconfigure on %s\n",
> + f->name, cdev->gadget->name);
> + return -ENODEV;
> +}
> +
> +static void usb9pfs_func_unbind(struct usb_configuration *c,
> + struct usb_function *f)
> +{
> + struct f_usb9pfs *usb9pfs = func_to_usb9pfs(f);
> +
> + disable_usb9pfs(usb9pfs);
> +}
> +
> +static void usb9pfs_free_func(struct usb_function *f)
> +{
> + struct f_usb9pfs *usb9pfs = func_to_usb9pfs(f);
> + struct f_usb9pfs_opts *opts;
> +
> + kfree(usb9pfs);
> +
> + opts = container_of(f->fi, struct f_usb9pfs_opts, func_inst);
> +
> + mutex_lock(&opts->lock);
> + opts->refcnt--;
> + mutex_unlock(&opts->lock);
> +
> + usb_free_all_descriptors(f);
> +}
> +
> +static int usb9pfs_set_alt(struct usb_function *f,
> + unsigned int intf, unsigned int alt)
> +{
> + struct f_usb9pfs *usb9pfs = func_to_usb9pfs(f);
> + struct usb_composite_dev *cdev = f->config->cdev;
> +
> + return enable_usb9pfs(cdev, usb9pfs);
> +}
> +
> +static void usb9pfs_disable(struct usb_function *f)
> +{
> + struct f_usb9pfs *usb9pfs = func_to_usb9pfs(f);
> +
> + usb9pfs_clear_tx(usb9pfs);
> +}
> +
> +static struct usb_function *usb9pfs_alloc(struct usb_function_instance *fi)
> +{
> + struct f_usb9pfs_opts *usb9pfs_opts;
> + struct f_usb9pfs *usb9pfs;
> +
> + usb9pfs = kzalloc(sizeof(*usb9pfs), GFP_KERNEL);
> + if (!usb9pfs)
> + return ERR_PTR(-ENOMEM);
> +
> + usb9pfs_opts = container_of(fi, struct f_usb9pfs_opts, func_inst);
> +
> + mutex_lock(&usb9pfs_opts->lock);
> + usb9pfs_opts->refcnt++;
> + mutex_unlock(&usb9pfs_opts->lock);
> +
> + usb9pfs->buflen = usb9pfs_opts->buflen;
> +
> + usb9pfs->function.name = "usb9pfs";
> + usb9pfs->function.bind = usb9pfs_func_bind;
> + usb9pfs->function.unbind = usb9pfs_func_unbind;
> + usb9pfs->function.set_alt = usb9pfs_set_alt;
> + usb9pfs->function.disable = usb9pfs_disable;
> + usb9pfs->function.strings = usb9pfs_strings;
> +
> + usb9pfs->function.free_func = usb9pfs_free_func;
> +
> + return &usb9pfs->function;
> +}
> +
> +static inline struct f_usb9pfs_opts *to_f_usb9pfs_opts(struct config_item *item)
> +{
> + return container_of(to_config_group(item), struct f_usb9pfs_opts,
> + func_inst.group);
> +}
> +
> +static inline struct f_usb9pfs_opts *fi_to_f_usb9pfs_opts(struct usb_function_instance *fi)
> +{
> + return container_of(fi, struct f_usb9pfs_opts, func_inst);
> +}
> +
> +static void usb9pfs_attr_release(struct config_item *item)
> +{
> + struct f_usb9pfs_opts *usb9pfs_opts = to_f_usb9pfs_opts(item);
> +
> + usb_put_function_instance(&usb9pfs_opts->func_inst);
> +}
> +
> +static struct configfs_item_operations usb9pfs_item_ops = {
> + .release = usb9pfs_attr_release,
> +};
> +
> +static ssize_t f_usb9pfs_opts_buflen_show(struct config_item *item, char *page)
> +{
> + struct f_usb9pfs_opts *opts = to_f_usb9pfs_opts(item);
> + int result;
> +
> + mutex_lock(&opts->lock);
> + result = sprintf(page, "%d\n", opts->buflen);
> + mutex_unlock(&opts->lock);
> +
> + return result;
> +}
> +
> +static ssize_t f_usb9pfs_opts_buflen_store(struct config_item *item,
> + const char *page, size_t len)
> +{
> + struct f_usb9pfs_opts *opts = to_f_usb9pfs_opts(item);
> + int ret;
> + u32 num;
> +
> + mutex_lock(&opts->lock);
> + if (opts->refcnt) {
> + ret = -EBUSY;
> + goto end;
> + }
> +
> + ret = kstrtou32(page, 0, &num);
> + if (ret)
> + goto end;
> +
> + opts->buflen = num;
> + ret = len;
> +end:
> + mutex_unlock(&opts->lock);
> + return ret;
> +}
> +
> +CONFIGFS_ATTR(f_usb9pfs_opts_, buflen);
> +
> +static struct configfs_attribute *usb9pfs_attrs[] = {
> + &f_usb9pfs_opts_attr_buflen,
> + NULL,
> +};
> +
> +static const struct config_item_type usb9pfs_func_type = {
> + .ct_item_ops = &usb9pfs_item_ops,
> + .ct_attrs = usb9pfs_attrs,
> + .ct_owner = THIS_MODULE,
> +};
> +
> +static struct f_usb9pfs_dev *_usb9pfs_do_find_dev(const char *tag)
> +{
> + struct f_usb9pfs_dev *usb9pfs_dev;
> + struct f_usb9pfs_dev *tmp;
> +
> + if (!tag)
> + return NULL;
> +
> + list_for_each_entry_safe(usb9pfs_dev, tmp, &usbg_instance_list, usb9pfs_instance) {
> + if (strcmp(usb9pfs_dev->tag, tag) == 0)
> + return usb9pfs_dev;
> + }
> +
> + return NULL;
> +}
> +
> +static int usb9pfs_tag_instance(struct f_usb9pfs_dev *dev, const char *tag)
> +{
> + struct f_usb9pfs_dev *existing;
> + int ret = 0;
> +
> + mutex_lock(&usb9pfs_lock);
> + existing = _usb9pfs_do_find_dev(tag);
> + if (!existing)
> + strscpy(dev->tag, tag, ARRAY_SIZE(dev->tag));
> + else if (existing != dev)
> + ret = -EBUSY;
> + mutex_unlock(&usb9pfs_lock);
> +
> + return ret;
> +}
> +
> +static int usb9pfs_set_inst_tag(struct usb_function_instance *fi, const char *tag)
> +{
> + if (strlen(tag) >= sizeof_field(struct f_usb9pfs_dev, tag))
> + return -ENAMETOOLONG;
> + return usb9pfs_tag_instance(fi_to_f_usb9pfs_opts(fi)->dev, tag);
> +}
> +
> +static void usb9pfs_free_instance(struct usb_function_instance *fi)
> +{
> + struct f_usb9pfs_opts *usb9pfs_opts =
> + container_of(fi, struct f_usb9pfs_opts, func_inst);
> + struct f_usb9pfs_dev *dev = usb9pfs_opts->dev;
> +
> + list_del(&dev->usb9pfs_instance);
> + kfree(usb9pfs_opts);
> +}
> +
> +static struct usb_function_instance *usb9pfs_alloc_instance(void)
> +{
> + struct f_usb9pfs_opts *usb9pfs_opts;
> + struct f_usb9pfs_dev *dev;
> +
> + usb9pfs_opts = kzalloc(sizeof(*usb9pfs_opts), GFP_KERNEL);
> + if (!usb9pfs_opts)
> + return ERR_PTR(-ENOMEM);
> +
> + mutex_init(&usb9pfs_opts->lock);
> +
> + usb9pfs_opts->func_inst.set_inst_name = usb9pfs_set_inst_tag;
> + usb9pfs_opts->func_inst.free_func_inst = usb9pfs_free_instance;
> +
> + usb9pfs_opts->buflen = DEFAULT_BUFLEN;
> +
> + mutex_lock(&usb9pfs_lock);
> + dev = kzalloc(sizeof(*dev), GFP_KERNEL);
> + if (IS_ERR(dev)) {
> + mutex_unlock(&usb9pfs_lock);
> + kfree(usb9pfs_opts);
> + return ERR_CAST(dev);
> + }
> + list_add_tail(&dev->usb9pfs_instance, &usbg_instance_list);
> + mutex_unlock(&usb9pfs_lock);
> +
> + usb9pfs_opts->dev = dev;
> + dev->opts = usb9pfs_opts;
> +
> + config_group_init_type_name(&usb9pfs_opts->func_inst.group, "",
> + &usb9pfs_func_type);
> +
> + return &usb9pfs_opts->func_inst;
> +}
> +DECLARE_USB_FUNCTION(usb9pfs, usb9pfs_alloc_instance, usb9pfs_alloc);
> +
> +static int __init usb9pfs_modinit(void)
> +{
> + int ret;
> +
> + INIT_LIST_HEAD(&usbg_instance_list);
> +
> + ret = usb_function_register(&usb9pfsusb_func);
> + if (!ret)
> + v9fs_register_trans(&p9_usbg_trans);
> +
> + return ret;
> +}
> +
> +static void __exit usb9pfs_modexit(void)
> +{
> + usb_function_unregister(&usb9pfsusb_func);
> + v9fs_unregister_trans(&p9_usbg_trans);
> +}
> +
> +module_init(usb9pfs_modinit);
> +module_exit(usb9pfs_modexit);
> +
> +MODULE_ALIAS_9P("usbg");
> +MODULE_LICENSE("GPL");
> +MODULE_DESCRIPTION("USB gadget 9pfs transport");
> +MODULE_AUTHOR("Michael Grzeschik");
>


2024-05-28 18:41:31

by Andrzej Pietrasiewicz

[permalink] [raw]
Subject: Re: [PATCH v5 3/3] tools: usb: p9_fwd: add usb gadget packet forwarder script

Hi,

W dniu 28.05.2024 o 00:08, Michael Grzeschik pisze:
> This patch is adding an small python tool to forward 9pfs requests
> from the USB gadget to an existing 9pfs TCP server. Since currently all
> 9pfs servers lack support for the usb transport this tool is an useful
> helper to get started.
>
> Refer the Documentation section "USBG Example" in
> Documentation/filesystems/9p.rst on how to use it.
>
> Signed-off-by: Jan Luebbe <[email protected]>
> Signed-off-by: Michael Grzeschik <[email protected]>
>
> ---
> v4 -> v5:
> - updated documentation for new subcommands list/connect
> - run ruff format
> - make vid and pid parameterized

Thanks for adding that.

> - add list as subcommand to scan for devices
> - move connect to extra subcommand
> v3 -> v4: -
> v2 -> v3: -
> v1 -> v2:
> - added usbg 9pfs detailed instructions to 9p.rst doc
> ---
> Documentation/filesystems/9p.rst | 41 +++++++
> tools/usb/p9_fwd.py | 243 +++++++++++++++++++++++++++++++++++++++
> 2 files changed, 284 insertions(+)
>
> diff --git a/Documentation/filesystems/9p.rst b/Documentation/filesystems/9p.rst
> index 10cf79dc287f8..2cc85f3e8659f 100644
> --- a/Documentation/filesystems/9p.rst
> +++ b/Documentation/filesystems/9p.rst
> @@ -67,6 +67,47 @@ To mount a 9p FS on a USB Host accessible via the gadget as root filesystem::
> where <device> is the tag associated by the usb gadget transport.
> It is defined by the configfs instance name.
>
> +USBG Example
> +============
> +
> +The USB host exports a filesystem, while the gadget on the USB device
> +side makes it mountable.
> +
> +Diod (9pfs server) and the forwarder are on the development host, where
> +the root filesystem is actually stored. The gadget is initialized during
> +boot (or later) on the embedded board. Then the forwarder will find it
> +on the USB bus and start forwarding requests.
> +
> +In this case the 9p requests come from the device and are handled by the
> +host. The reason is that USB device ports are normally not available on
> +PCs, so a connection in the other direction would not work.
> +
> +When using the usbg transport, for now there is no native usb host
> +service capable to handle the requests from the gadget driver. For
> +this we have to use the extra python tool p9_fwd.py from tools/usb.
> +
> +Just start the 9pfs capable network server like diod/nfs-ganesha e.g.:
> +
> + $ diod -f -n -d 0 -S -l 0.0.0.0:9999 -e $PWD
> +
> +Optionaly scan your bus if there are more then one usbg gadgets to find their path:
> +
> + $ python $kernel_dir/tools/usb/p9_fwd.py list
> +
> + Bus | Addr | Manufacturer | Product | ID | Path
> + --- | ---- | ---------------- | ---------------- | --------- | ----
> + 2 | 67 | unknown | unknown | 1d6b:0109 | 2-1.1.2
> + 2 | 68 | unknown | unknown | 1d6b:0109 | 2-1.1.3
> +
> +Then start the python transport:
> +
> + $ python $kernel_dir/tools/usb/p9_fwd.py --path 2-1.1.2 connect -p 9999
> +
> +After that the gadget driver can be used as described above.
> +
> +One use-case is to use it as an alternative to NFS root booting during
> +the development of embedded Linux devices.
> +
> Options
> =======
>
> diff --git a/tools/usb/p9_fwd.py b/tools/usb/p9_fwd.py
> new file mode 100755
> index 0000000000000..7bedefce75c7b
> --- /dev/null
> +++ b/tools/usb/p9_fwd.py
> @@ -0,0 +1,243 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: GPL-2.0
> +
> +import argparse
> +import errno
> +import logging
> +import socket
> +import struct
> +import time
> +
> +import usb.core
> +import usb.util
> +
> +
> +def path_from_usb_dev(dev):
> + """Takes a pyUSB device as argument and returns a string.
> + The string is a Path representation of the position of the USB device on the USB bus tree.
> +
> + This path is used to find a USB device on the bus or all devices connected to a HUB.
> + The path is made up of the number of the USB controller followed be the ports of the HUB tree."""
> + if dev.port_numbers:
> + dev_path = ".".join(str(i) for i in dev.port_numbers)
> + return f"{dev.bus}-{dev_path}"
> + return ""
> +
> +
> +HEXDUMP_FILTER = "".join(chr(x).isprintable() and chr(x) or "." for x in range(128)) + "." * 128
> +
> +
> +class Forwarder:
> + @staticmethod
> + def _log_hexdump(data):
> + if not logging.root.isEnabledFor(logging.TRACE):
> + return
> + L = 16
> + for c in range(0, len(data), L):
> + chars = data[c : c + L]
> + dump = " ".join(f"{x:02x}" for x in chars)
> + printable = "".join(HEXDUMP_FILTER[x] for x in chars)
> + line = f"{c:08x} {dump:{L*3}s} |{printable:{L}s}|"
> + logging.root.log(logging.TRACE, "%s", line)
> +
> + def __init__(self, server, vid, pid, path):
> + self.stats = {
> + "c2s packets": 0,
> + "c2s bytes": 0,
> + "s2c packets": 0,
> + "s2c bytes": 0,
> + }
> + self.stats_logged = time.monotonic()
> +
> + def find_filter(dev):
> + dev_path = path_from_usb_dev(dev)
> + if path is not None:
> + return dev_path == path
> + return True
> +
> + dev = usb.core.find(idVendor=vid, idProduct=pid, custom_match=find_filter)
> + if dev is None:
> + raise ValueError("Device not found")
> +
> + logging.info(f"found device: {dev.bus}/{dev.address} located at {path_from_usb_dev(dev)}")
> +
> + # dev.set_configuration() is not necessary since g_multi has only one
> + usb9pfs = None
> + # g_multi adds 9pfs as last interface
> + cfg = dev.get_active_configuration()
> + for intf in cfg:
> + # we have to detach the usb-storage driver from multi gadget since
> + # stall option could be set, which will lead to spontaneous port
> + # resets and our transfers will run dead
> + if intf.bInterfaceClass == 0x08:
> + if dev.is_kernel_driver_active(intf.bInterfaceNumber):
> + dev.detach_kernel_driver(intf.bInterfaceNumber)
> +
> + if intf.bInterfaceClass == 0xFF and intf.bInterfaceSubClass == 0xFF and intf.bInterfaceProtocol == 0x09:
> + usb9pfs = intf
> + if usb9pfs is None:
> + raise ValueError("Interface not found")
> +
> + logging.info(f"claiming interface:\n{usb9pfs}")
> + usb.util.claim_interface(dev, usb9pfs.bInterfaceNumber)
> + ep_out = usb.util.find_descriptor(
> + usb9pfs,
> + custom_match=lambda e: usb.util.endpoint_direction(e.bEndpointAddress) == usb.util.ENDPOINT_OUT,
> + )
> + assert ep_out is not None
> + ep_in = usb.util.find_descriptor(
> + usb9pfs,
> + custom_match=lambda e: usb.util.endpoint_direction(e.bEndpointAddress) == usb.util.ENDPOINT_IN,
> + )
> + assert ep_in is not None
> + logging.info("interface claimed")
> +
> + self.ep_out = ep_out
> + self.ep_in = ep_in
> + self.dev = dev
> +
> + # create and connect socket
> + self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> + self.s.connect(server)
> +
> + logging.info("connected to server")
> +
> + def c2s(self):
> + """forward a request from the USB client to the TCP server"""
> + data = None
> + while data is None:
> + try:
> + logging.log(logging.TRACE, "c2s: reading")
> + data = self.ep_in.read(self.ep_in.wMaxPacketSize)
> + except usb.core.USBTimeoutError:
> + logging.log(logging.TRACE, "c2s: reading timed out")
> + continue
> + except usb.core.USBError as e:
> + if e.errno == errno.EIO:
> + logging.debug("c2s: reading failed with %s, retrying", repr(e))
> + time.sleep(0.5)
> + continue
> + logging.error("c2s: reading failed with %s, aborting", repr(e))
> + raise
> + size = struct.unpack("<I", data[:4])[0]
> + while len(data) < size:
> + data += self.ep_in.read(size - len(data))
> + logging.log(logging.TRACE, "c2s: writing")
> + self._log_hexdump(data)
> + self.s.send(data)
> + logging.debug("c2s: forwarded %i bytes", size)
> + self.stats["c2s packets"] += 1
> + self.stats["c2s bytes"] += size
> +
> + def s2c(self):
> + """forward a response from the TCP server to the USB client"""
> + logging.log(logging.TRACE, "s2c: reading")
> + data = self.s.recv(4)
> + size = struct.unpack("<I", data[:4])[0]
> + while len(data) < size:
> + data += self.s.recv(size - len(data))
> + logging.log(logging.TRACE, "s2c: writing")
> + self._log_hexdump(data)
> + while data:
> + written = self.ep_out.write(data)
> + assert written > 0
> + data = data[written:]
> + if size % self.ep_out.wMaxPacketSize == 0:
> + logging.log(logging.TRACE, "sending zero length packet")
> + self.ep_out.write(b"")
> + logging.debug("s2c: forwarded %i bytes", size)
> + self.stats["s2c packets"] += 1
> + self.stats["s2c bytes"] += size
> +
> + def log_stats(self):
> + logging.info("statistics:")
> + for k, v in self.stats.items():
> + logging.info(f" {k+':':14s} {v}")
> +
> + def log_stats_interval(self, interval=5):
> + if (time.monotonic() - self.stats_logged) < interval:
> + return
> +
> + self.log_stats()
> + self.stats_logged = time.monotonic()
> +
> +
> +def try_get_usb_str(dev, name):
> + try:
> + with open(f"/sys/bus/usb/devices/{dev.bus}-{dev.address}/{name}") as f:
> + return f.read().strip()
> + except FileNotFoundError:
> + return None
> +
> +
> +def list_usb(args):
> + vid, pid = [int(x, 16) for x in args.id.split(":", 1)]
> +
> + print("Bus | Addr | Manufacturer | Product | ID | Path")
> + print("--- | ---- | ---------------- | ---------------- | --------- | ----")
> + for dev in usb.core.find(find_all=True, idVendor=vid, idProduct=pid):
> + path = path_from_usb_dev(dev) or ""
> + manufacturer = try_get_usb_str(dev, "manufacturer") or "unknown"
> + product = try_get_usb_str(dev, "product") or "unknown"
> + print(
> + f"{dev.bus:3} | {dev.address:4} | {manufacturer:16} | {product:16} | {dev.idVendor:04x}:{dev.idProduct:04x} | {path:18}"
> + )
> +
> +
> +def connect(args):
> + vid, pid = [int(x, 16) for x in args.id.split(":", 1)]
> +
> + f = Forwarder(server=(args.server, args.port), vid=vid, pid=pid, path=args.path)
> +
> + try:
> + while True:
> + f.c2s()
> + f.s2c()
> + f.log_stats_interval()
> + finally:
> + f.log_stats()
> +
> +
> +def main():
> + parser = argparse.ArgumentParser(
> + description="Forward 9PFS requests from USB to TCP",
> + )
> +
> + parser.add_argument("--id", type=str, default="1d6b:0109", help="vid:pid of target device")
> + parser.add_argument("--path", type=str, default="", help="path of target device")

I had to specify both --id and --path, otherwise I was getting
"device not found".

Regards,

Andrzej

> + parser.add_argument("-v", "--verbose", action="count", default=0)
> +
> + subparsers = parser.add_subparsers()
> + subparsers.required = True
> + subparsers.dest = "command"
> +
> + parser_list = subparsers.add_parser("list", help="List all connected 9p gadgets")
> + parser_list.set_defaults(func=list_usb)
> +
> + parser_connect = subparsers.add_parser(
> + "connect", help="Forward messages between the usb9pfs gadget and the 9p server"
> + )
> + parser_connect.set_defaults(func=connect)
> + connect_group = parser_connect.add_argument_group()
> + connect_group.required = True
> + parser_connect.add_argument("-s", "--server", type=str, default="127.0.0.1", help="server hostname")
> + parser_connect.add_argument("-p", "--port", type=int, default=564, help="server port")> +
> + args = parser.parse_args()
> +
> + logging.TRACE = logging.DEBUG - 5
> + logging.addLevelName(logging.TRACE, "TRACE")
> +
> + if args.verbose >= 2:
> + level = logging.TRACE
> + elif args.verbose:
> + level = logging.DEBUG
> + else:
> + level = logging.INFO
> + logging.basicConfig(level=level, format="%(asctime)-15s %(levelname)-8s %(message)s")
> +
> + args.func(args)
> +
> +
> +if __name__ == "__main__":
> + main()
>


2024-05-29 21:44:44

by Michael Grzeschik

[permalink] [raw]
Subject: Re: [PATCH v5 3/3] tools: usb: p9_fwd: add usb gadget packet forwarder script

On Tue, May 28, 2024 at 08:41:12PM +0200, Andrzej Pietrasiewicz wrote:
>Hi,
>
>W dniu 28.05.2024 o?00:08, Michael Grzeschik pisze:
>>This patch is adding an small python tool to forward 9pfs requests
>>from the USB gadget to an existing 9pfs TCP server. Since currently all
>>9pfs servers lack support for the usb transport this tool is an useful
>>helper to get started.
>>
>>Refer the Documentation section "USBG Example" in
>>Documentation/filesystems/9p.rst on how to use it.
>>
>>Signed-off-by: Jan Luebbe <[email protected]>
>>Signed-off-by: Michael Grzeschik <[email protected]>
>>
>>---
>>v4 -> v5:
>> - updated documentation for new subcommands list/connect
>> - run ruff format
>> - make vid and pid parameterized
>
>Thanks for adding that.
>
>> - add list as subcommand to scan for devices
>> - move connect to extra subcommand
>>v3 -> v4: -
>>v2 -> v3: -
>>v1 -> v2:
>> - added usbg 9pfs detailed instructions to 9p.rst doc
>>---
>> Documentation/filesystems/9p.rst | 41 +++++++
>> tools/usb/p9_fwd.py | 243 +++++++++++++++++++++++++++++++++++++++
>> 2 files changed, 284 insertions(+)
>>
>>diff --git a/Documentation/filesystems/9p.rst b/Documentation/filesystems/9p.rst
>>index 10cf79dc287f8..2cc85f3e8659f 100644
>>--- a/Documentation/filesystems/9p.rst
>>+++ b/Documentation/filesystems/9p.rst
>>@@ -67,6 +67,47 @@ To mount a 9p FS on a USB Host accessible via the gadget as root filesystem::
>> where <device> is the tag associated by the usb gadget transport.
>> It is defined by the configfs instance name.
>>+USBG Example
>>+============
>>+
>>+The USB host exports a filesystem, while the gadget on the USB device
>>+side makes it mountable.
>>+
>>+Diod (9pfs server) and the forwarder are on the development host, where
>>+the root filesystem is actually stored. The gadget is initialized during
>>+boot (or later) on the embedded board. Then the forwarder will find it
>>+on the USB bus and start forwarding requests.
>>+
>>+In this case the 9p requests come from the device and are handled by the
>>+host. The reason is that USB device ports are normally not available on
>>+PCs, so a connection in the other direction would not work.
>>+
>>+When using the usbg transport, for now there is no native usb host
>>+service capable to handle the requests from the gadget driver. For
>>+this we have to use the extra python tool p9_fwd.py from tools/usb.
>>+
>>+Just start the 9pfs capable network server like diod/nfs-ganesha e.g.:
>>+
>>+ $ diod -f -n -d 0 -S -l 0.0.0.0:9999 -e $PWD
>>+
>>+Optionaly scan your bus if there are more then one usbg gadgets to find their path:
>>+
>>+ $ python $kernel_dir/tools/usb/p9_fwd.py list
>>+
>>+ Bus | Addr | Manufacturer | Product | ID | Path
>>+ --- | ---- | ---------------- | ---------------- | --------- | ----
>>+ 2 | 67 | unknown | unknown | 1d6b:0109 | 2-1.1.2
>>+ 2 | 68 | unknown | unknown | 1d6b:0109 | 2-1.1.3
>>+
>>+Then start the python transport:
>>+
>>+ $ python $kernel_dir/tools/usb/p9_fwd.py --path 2-1.1.2 connect -p 9999
>>+
>>+After that the gadget driver can be used as described above.
>>+
>>+One use-case is to use it as an alternative to NFS root booting during
>>+the development of embedded Linux devices.
>>+
>> Options
>> =======
>>diff --git a/tools/usb/p9_fwd.py b/tools/usb/p9_fwd.py
>>new file mode 100755
>>index 0000000000000..7bedefce75c7b
>>--- /dev/null
>>+++ b/tools/usb/p9_fwd.py
>>@@ -0,0 +1,243 @@
>>+#!/usr/bin/env python3
>>+# SPDX-License-Identifier: GPL-2.0
>>+
>>+import argparse
>>+import errno
>>+import logging
>>+import socket
>>+import struct
>>+import time
>>+
>>+import usb.core
>>+import usb.util
>>+
>>+
>>+def path_from_usb_dev(dev):
>>+ """Takes a pyUSB device as argument and returns a string.
>>+ The string is a Path representation of the position of the USB device on the USB bus tree.
>>+
>>+ This path is used to find a USB device on the bus or all devices connected to a HUB.
>>+ The path is made up of the number of the USB controller followed be the ports of the HUB tree."""
>>+ if dev.port_numbers:
>>+ dev_path = ".".join(str(i) for i in dev.port_numbers)
>>+ return f"{dev.bus}-{dev_path}"
>>+ return ""
>>+
>>+
>>+HEXDUMP_FILTER = "".join(chr(x).isprintable() and chr(x) or "." for x in range(128)) + "." * 128
>>+
>>+
>>+class Forwarder:
>>+ @staticmethod
>>+ def _log_hexdump(data):
>>+ if not logging.root.isEnabledFor(logging.TRACE):
>>+ return
>>+ L = 16
>>+ for c in range(0, len(data), L):
>>+ chars = data[c : c + L]
>>+ dump = " ".join(f"{x:02x}" for x in chars)
>>+ printable = "".join(HEXDUMP_FILTER[x] for x in chars)
>>+ line = f"{c:08x} {dump:{L*3}s} |{printable:{L}s}|"
>>+ logging.root.log(logging.TRACE, "%s", line)
>>+
>>+ def __init__(self, server, vid, pid, path):
>>+ self.stats = {
>>+ "c2s packets": 0,
>>+ "c2s bytes": 0,
>>+ "s2c packets": 0,
>>+ "s2c bytes": 0,
>>+ }
>>+ self.stats_logged = time.monotonic()
>>+
>>+ def find_filter(dev):
>>+ dev_path = path_from_usb_dev(dev)
>>+ if path is not None:
>>+ return dev_path == path
>>+ return True
>>+
>>+ dev = usb.core.find(idVendor=vid, idProduct=pid, custom_match=find_filter)
>>+ if dev is None:
>>+ raise ValueError("Device not found")
>>+
>>+ logging.info(f"found device: {dev.bus}/{dev.address} located at {path_from_usb_dev(dev)}")
>>+
>>+ # dev.set_configuration() is not necessary since g_multi has only one
>>+ usb9pfs = None
>>+ # g_multi adds 9pfs as last interface
>>+ cfg = dev.get_active_configuration()
>>+ for intf in cfg:
>>+ # we have to detach the usb-storage driver from multi gadget since
>>+ # stall option could be set, which will lead to spontaneous port
>>+ # resets and our transfers will run dead
>>+ if intf.bInterfaceClass == 0x08:
>>+ if dev.is_kernel_driver_active(intf.bInterfaceNumber):
>>+ dev.detach_kernel_driver(intf.bInterfaceNumber)
>>+
>>+ if intf.bInterfaceClass == 0xFF and intf.bInterfaceSubClass == 0xFF and intf.bInterfaceProtocol == 0x09:
>>+ usb9pfs = intf
>>+ if usb9pfs is None:
>>+ raise ValueError("Interface not found")
>>+
>>+ logging.info(f"claiming interface:\n{usb9pfs}")
>>+ usb.util.claim_interface(dev, usb9pfs.bInterfaceNumber)
>>+ ep_out = usb.util.find_descriptor(
>>+ usb9pfs,
>>+ custom_match=lambda e: usb.util.endpoint_direction(e.bEndpointAddress) == usb.util.ENDPOINT_OUT,
>>+ )
>>+ assert ep_out is not None
>>+ ep_in = usb.util.find_descriptor(
>>+ usb9pfs,
>>+ custom_match=lambda e: usb.util.endpoint_direction(e.bEndpointAddress) == usb.util.ENDPOINT_IN,
>>+ )
>>+ assert ep_in is not None
>>+ logging.info("interface claimed")
>>+
>>+ self.ep_out = ep_out
>>+ self.ep_in = ep_in
>>+ self.dev = dev
>>+
>>+ # create and connect socket
>>+ self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>+ self.s.connect(server)
>>+
>>+ logging.info("connected to server")
>>+
>>+ def c2s(self):
>>+ """forward a request from the USB client to the TCP server"""
>>+ data = None
>>+ while data is None:
>>+ try:
>>+ logging.log(logging.TRACE, "c2s: reading")
>>+ data = self.ep_in.read(self.ep_in.wMaxPacketSize)
>>+ except usb.core.USBTimeoutError:
>>+ logging.log(logging.TRACE, "c2s: reading timed out")
>>+ continue
>>+ except usb.core.USBError as e:
>>+ if e.errno == errno.EIO:
>>+ logging.debug("c2s: reading failed with %s, retrying", repr(e))
>>+ time.sleep(0.5)
>>+ continue
>>+ logging.error("c2s: reading failed with %s, aborting", repr(e))
>>+ raise
>>+ size = struct.unpack("<I", data[:4])[0]
>>+ while len(data) < size:
>>+ data += self.ep_in.read(size - len(data))
>>+ logging.log(logging.TRACE, "c2s: writing")
>>+ self._log_hexdump(data)
>>+ self.s.send(data)
>>+ logging.debug("c2s: forwarded %i bytes", size)
>>+ self.stats["c2s packets"] += 1
>>+ self.stats["c2s bytes"] += size
>>+
>>+ def s2c(self):
>>+ """forward a response from the TCP server to the USB client"""
>>+ logging.log(logging.TRACE, "s2c: reading")
>>+ data = self.s.recv(4)
>>+ size = struct.unpack("<I", data[:4])[0]
>>+ while len(data) < size:
>>+ data += self.s.recv(size - len(data))
>>+ logging.log(logging.TRACE, "s2c: writing")
>>+ self._log_hexdump(data)
>>+ while data:
>>+ written = self.ep_out.write(data)
>>+ assert written > 0
>>+ data = data[written:]
>>+ if size % self.ep_out.wMaxPacketSize == 0:
>>+ logging.log(logging.TRACE, "sending zero length packet")
>>+ self.ep_out.write(b"")
>>+ logging.debug("s2c: forwarded %i bytes", size)
>>+ self.stats["s2c packets"] += 1
>>+ self.stats["s2c bytes"] += size
>>+
>>+ def log_stats(self):
>>+ logging.info("statistics:")
>>+ for k, v in self.stats.items():
>>+ logging.info(f" {k+':':14s} {v}")
>>+
>>+ def log_stats_interval(self, interval=5):
>>+ if (time.monotonic() - self.stats_logged) < interval:
>>+ return
>>+
>>+ self.log_stats()
>>+ self.stats_logged = time.monotonic()
>>+
>>+
>>+def try_get_usb_str(dev, name):
>>+ try:
>>+ with open(f"/sys/bus/usb/devices/{dev.bus}-{dev.address}/{name}") as f:
>>+ return f.read().strip()
>>+ except FileNotFoundError:
>>+ return None
>>+
>>+
>>+def list_usb(args):
>>+ vid, pid = [int(x, 16) for x in args.id.split(":", 1)]
>>+
>>+ print("Bus | Addr | Manufacturer | Product | ID | Path")
>>+ print("--- | ---- | ---------------- | ---------------- | --------- | ----")
>>+ for dev in usb.core.find(find_all=True, idVendor=vid, idProduct=pid):
>>+ path = path_from_usb_dev(dev) or ""
>>+ manufacturer = try_get_usb_str(dev, "manufacturer") or "unknown"
>>+ product = try_get_usb_str(dev, "product") or "unknown"
>>+ print(
>>+ f"{dev.bus:3} | {dev.address:4} | {manufacturer:16} | {product:16} | {dev.idVendor:04x}:{dev.idProduct:04x} | {path:18}"
>>+ )
>>+
>>+
>>+def connect(args):
>>+ vid, pid = [int(x, 16) for x in args.id.split(":", 1)]
>>+
>>+ f = Forwarder(server=(args.server, args.port), vid=vid, pid=pid, path=args.path)
>>+
>>+ try:
>>+ while True:
>>+ f.c2s()
>>+ f.s2c()
>>+ f.log_stats_interval()
>>+ finally:
>>+ f.log_stats()
>>+
>>+
>>+def main():
>>+ parser = argparse.ArgumentParser(
>>+ description="Forward 9PFS requests from USB to TCP",
>>+ )
>>+
>>+ parser.add_argument("--id", type=str, default="1d6b:0109", help="vid:pid of target device")
>>+ parser.add_argument("--path", type=str, default="", help="path of target device")
>
>I had to specify both --id and --path, otherwise I was getting
>"device not found".

This is odd. What was your list command saying about the available
devices?

I tested this with the default vid:pid as described above (1d6b:0109)
which worked fine with path alone.

Michael

>>+ parser.add_argument("-v", "--verbose", action="count", default=0)
>>+
>>+ subparsers = parser.add_subparsers()
>>+ subparsers.required = True
>>+ subparsers.dest = "command"
>>+
>>+ parser_list = subparsers.add_parser("list", help="List all connected 9p gadgets")
>>+ parser_list.set_defaults(func=list_usb)
>>+
>>+ parser_connect = subparsers.add_parser(
>>+ "connect", help="Forward messages between the usb9pfs gadget and the 9p server"
>>+ )
>>+ parser_connect.set_defaults(func=connect)
>>+ connect_group = parser_connect.add_argument_group()
>>+ connect_group.required = True
>>+ parser_connect.add_argument("-s", "--server", type=str, default="127.0.0.1", help="server hostname")
>>+ parser_connect.add_argument("-p", "--port", type=int, default=564, help="server port")> +
>>+ args = parser.parse_args()
>>+
>>+ logging.TRACE = logging.DEBUG - 5
>>+ logging.addLevelName(logging.TRACE, "TRACE")
>>+
>>+ if args.verbose >= 2:
>>+ level = logging.TRACE
>>+ elif args.verbose:
>>+ level = logging.DEBUG
>>+ else:
>>+ level = logging.INFO
>>+ logging.basicConfig(level=level, format="%(asctime)-15s %(levelname)-8s %(message)s")
>>+
>>+ args.func(args)
>>+
>>+
>>+if __name__ == "__main__":
>>+ main()
>>
>
>

--
Pengutronix e.K. | |
Steuerwalder Str. 21 | http://www.pengutronix.de/ |
31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |


Attachments:
(No filename) (13.50 kB)
signature.asc (849.00 B)
Download all attachments

2024-05-31 07:37:41

by Andrzej Pietrasiewicz

[permalink] [raw]
Subject: Re: [PATCH v5 3/3] tools: usb: p9_fwd: add usb gadget packet forwarder script

Hi,

W dniu 29.05.2024 o 23:44, Michael Grzeschik pisze:
> On Tue, May 28, 2024 at 08:41:12PM +0200, Andrzej Pietrasiewicz wrote:
>> Hi,
>>
>> W dniu 28.05.2024 o 00:08, Michael Grzeschik pisze:
>>> This patch is adding an small python tool to forward 9pfs requests
>>> from the USB gadget to an existing 9pfs TCP server. Since currently all
>>> 9pfs servers lack support for the usb transport this tool is an useful
>>> helper to get started.
>>>
>>> Refer the Documentation section "USBG Example" in
>>> Documentation/filesystems/9p.rst on how to use it.
>>>
>>> Signed-off-by: Jan Luebbe <[email protected]>
>>> Signed-off-by: Michael Grzeschik <[email protected]>
>>>
>>> ---
>>> v4 -> v5:
>>>   - updated documentation for new subcommands list/connect
>>>   - run ruff format
>>>   - make vid and pid parameterized
>>
>> Thanks for adding that.
>>
>>>   - add list as subcommand to scan for devices
>>>   - move connect to extra subcommand
>>> v3 -> v4: -
>>> v2 -> v3: -
>>> v1 -> v2:
>>>   - added usbg 9pfs detailed instructions to 9p.rst doc
>>> ---
>>>  Documentation/filesystems/9p.rst |  41 +++++++
>>>  tools/usb/p9_fwd.py              | 243 +++++++++++++++++++++++++++++++++++++++
>>>  2 files changed, 284 insertions(+)
>>>
>>> diff --git a/Documentation/filesystems/9p.rst b/Documentation/filesystems/9p.rst
>>> index 10cf79dc287f8..2cc85f3e8659f 100644
>>> --- a/Documentation/filesystems/9p.rst
>>> +++ b/Documentation/filesystems/9p.rst
>>> @@ -67,6 +67,47 @@ To mount a 9p FS on a USB Host accessible via the gadget as root filesystem::
>>>  where <device> is the tag associated by the usb gadget transport.
>>>  It is defined by the configfs instance name.
>>> +USBG Example
>>> +============
>>> +
>>> +The USB host exports a filesystem, while the gadget on the USB device
>>> +side makes it mountable.
>>> +
>>> +Diod (9pfs server) and the forwarder are on the development host, where
>>> +the root filesystem is actually stored. The gadget is initialized during
>>> +boot (or later) on the embedded board. Then the forwarder will find it
>>> +on the USB bus and start forwarding requests.
>>> +
>>> +In this case the 9p requests come from the device and are handled by the
>>> +host. The reason is that USB device ports are normally not available on
>>> +PCs, so a connection in the other direction would not work.
>>> +
>>> +When using the usbg transport, for now there is no native usb host
>>> +service capable to handle the requests from the gadget driver. For
>>> +this we have to use the extra python tool p9_fwd.py from tools/usb.
>>> +
>>> +Just start the 9pfs capable network server like diod/nfs-ganesha e.g.:
>>> +
>>> +        $ diod -f -n -d 0 -S -l 0.0.0.0:9999 -e $PWD
>>> +
>>> +Optionaly scan your bus if there are more then one usbg gadgets to find their path:
>>> +
>>> +        $ python $kernel_dir/tools/usb/p9_fwd.py list
>>> +
>>> +        Bus | Addr | Manufacturer     | Product          | ID        | Path
>>> +        --- | ---- | ---------------- | ---------------- | --------- | ----
>>> +          2 |   67 | unknown          | unknown          | 1d6b:0109 | 2-1.1.2
>>> +          2 |   68 | unknown          | unknown          | 1d6b:0109 | 2-1.1.3
>>> +
>>> +Then start the python transport:
>>> +
>>> +        $ python $kernel_dir/tools/usb/p9_fwd.py --path 2-1.1.2 connect -p 9999
>>> +
>>> +After that the gadget driver can be used as described above.
>>> +
>>> +One use-case is to use it as an alternative to NFS root booting during
>>> +the development of embedded Linux devices.
>>> +
>>>  Options
>>>  =======
>>> diff --git a/tools/usb/p9_fwd.py b/tools/usb/p9_fwd.py
>>> new file mode 100755
>>> index 0000000000000..7bedefce75c7b
>>> --- /dev/null
>>> +++ b/tools/usb/p9_fwd.py
>>> @@ -0,0 +1,243 @@
>>> +#!/usr/bin/env python3
>>> +# SPDX-License-Identifier: GPL-2.0
>>> +
>>> +import argparse
>>> +import errno
>>> +import logging
>>> +import socket
>>> +import struct
>>> +import time
>>> +
>>> +import usb.core
>>> +import usb.util
>>> +
>>> +
>>> +def path_from_usb_dev(dev):
>>> +    """Takes a pyUSB device as argument and returns a string.
>>> +    The string is a Path representation of the position of the USB device on the USB bus tree.
>>> +
>>> +    This path is used to find a USB device on the bus or all devices connected to a HUB.
>>> +    The path is made up of the number of the USB controller followed be the ports of the HUB tree."""
>>> +    if dev.port_numbers:
>>> +        dev_path = ".".join(str(i) for i in dev.port_numbers)
>>> +        return f"{dev.bus}-{dev_path}"
>>> +    return ""
>>> +
>>> +
>>> +HEXDUMP_FILTER = "".join(chr(x).isprintable() and chr(x) or "." for x in range(128)) + "." * 128
>>> +
>>> +
>>> +class Forwarder:
>>> +    @staticmethod
>>> +    def _log_hexdump(data):
>>> +        if not logging.root.isEnabledFor(logging.TRACE):
>>> +            return
>>> +        L = 16
>>> +        for c in range(0, len(data), L):
>>> +            chars = data[c : c + L]
>>> +            dump = " ".join(f"{x:02x}" for x in chars)
>>> +            printable = "".join(HEXDUMP_FILTER[x] for x in chars)
>>> +            line = f"{c:08x}  {dump:{L*3}s} |{printable:{L}s}|"
>>> +            logging.root.log(logging.TRACE, "%s", line)
>>> +
>>> +    def __init__(self, server, vid, pid, path):
>>> +        self.stats = {
>>> +            "c2s packets": 0,
>>> +            "c2s bytes": 0,
>>> +            "s2c packets": 0,
>>> +            "s2c bytes": 0,
>>> +        }
>>> +        self.stats_logged = time.monotonic()
>>> +
>>> +        def find_filter(dev):
>>> +            dev_path = path_from_usb_dev(dev)
>>> +            if path is not None:
>>> +                return dev_path == path
>>> +            return True
>>> +
>>> +        dev = usb.core.find(idVendor=vid, idProduct=pid, custom_match=find_filter)
>>> +        if dev is None:
>>> +            raise ValueError("Device not found")
>>> +
>>> +        logging.info(f"found device: {dev.bus}/{dev.address} located at {path_from_usb_dev(dev)}")
>>> +
>>> +        # dev.set_configuration() is not necessary since g_multi has only one
>>> +        usb9pfs = None
>>> +        # g_multi adds 9pfs as last interface
>>> +        cfg = dev.get_active_configuration()
>>> +        for intf in cfg:
>>> +            # we have to detach the usb-storage driver from multi gadget since
>>> +            # stall option could be set, which will lead to spontaneous port
>>> +            # resets and our transfers will run dead
>>> +            if intf.bInterfaceClass == 0x08:
>>> +                if dev.is_kernel_driver_active(intf.bInterfaceNumber):
>>> +                    dev.detach_kernel_driver(intf.bInterfaceNumber)
>>> +
>>> +            if intf.bInterfaceClass == 0xFF and intf.bInterfaceSubClass == 0xFF and intf.bInterfaceProtocol == 0x09:
>>> +                usb9pfs = intf
>>> +        if usb9pfs is None:
>>> +            raise ValueError("Interface not found")
>>> +
>>> +        logging.info(f"claiming interface:\n{usb9pfs}")
>>> +        usb.util.claim_interface(dev, usb9pfs.bInterfaceNumber)
>>> +        ep_out = usb.util.find_descriptor(
>>> +            usb9pfs,
>>> +            custom_match=lambda e: usb.util.endpoint_direction(e.bEndpointAddress) == usb.util.ENDPOINT_OUT,
>>> +        )
>>> +        assert ep_out is not None
>>> +        ep_in = usb.util.find_descriptor(
>>> +            usb9pfs,
>>> +            custom_match=lambda e: usb.util.endpoint_direction(e.bEndpointAddress) == usb.util.ENDPOINT_IN,
>>> +        )
>>> +        assert ep_in is not None
>>> +        logging.info("interface claimed")
>>> +
>>> +        self.ep_out = ep_out
>>> +        self.ep_in = ep_in
>>> +        self.dev = dev
>>> +
>>> +        # create and connect socket
>>> +        self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> +        self.s.connect(server)
>>> +
>>> +        logging.info("connected to server")
>>> +
>>> +    def c2s(self):
>>> +        """forward a request from the USB client to the TCP server"""
>>> +        data = None
>>> +        while data is None:
>>> +            try:
>>> +                logging.log(logging.TRACE, "c2s: reading")
>>> +                data = self.ep_in.read(self.ep_in.wMaxPacketSize)
>>> +            except usb.core.USBTimeoutError:
>>> +                logging.log(logging.TRACE, "c2s: reading timed out")
>>> +                continue
>>> +            except usb.core.USBError as e:
>>> +                if e.errno == errno.EIO:
>>> +                    logging.debug("c2s: reading failed with %s, retrying", repr(e))
>>> +                    time.sleep(0.5)
>>> +                    continue
>>> +                logging.error("c2s: reading failed with %s, aborting", repr(e))
>>> +                raise
>>> +        size = struct.unpack("<I", data[:4])[0]
>>> +        while len(data) < size:
>>> +            data += self.ep_in.read(size - len(data))
>>> +        logging.log(logging.TRACE, "c2s: writing")
>>> +        self._log_hexdump(data)
>>> +        self.s.send(data)
>>> +        logging.debug("c2s: forwarded %i bytes", size)
>>> +        self.stats["c2s packets"] += 1
>>> +        self.stats["c2s bytes"] += size
>>> +
>>> +    def s2c(self):
>>> +        """forward a response from the TCP server to the USB client"""
>>> +        logging.log(logging.TRACE, "s2c: reading")
>>> +        data = self.s.recv(4)
>>> +        size = struct.unpack("<I", data[:4])[0]
>>> +        while len(data) < size:
>>> +            data += self.s.recv(size - len(data))
>>> +        logging.log(logging.TRACE, "s2c: writing")
>>> +        self._log_hexdump(data)
>>> +        while data:
>>> +            written = self.ep_out.write(data)
>>> +            assert written > 0
>>> +            data = data[written:]
>>> +        if size % self.ep_out.wMaxPacketSize == 0:
>>> +            logging.log(logging.TRACE, "sending zero length packet")
>>> +            self.ep_out.write(b"")
>>> +        logging.debug("s2c: forwarded %i bytes", size)
>>> +        self.stats["s2c packets"] += 1
>>> +        self.stats["s2c bytes"] += size
>>> +
>>> +    def log_stats(self):
>>> +        logging.info("statistics:")
>>> +        for k, v in self.stats.items():
>>> +            logging.info(f"  {k+':':14s} {v}")
>>> +
>>> +    def log_stats_interval(self, interval=5):
>>> +        if (time.monotonic() - self.stats_logged) < interval:
>>> +            return
>>> +
>>> +        self.log_stats()
>>> +        self.stats_logged = time.monotonic()
>>> +
>>> +
>>> +def try_get_usb_str(dev, name):
>>> +    try:
>>> +        with open(f"/sys/bus/usb/devices/{dev.bus}-{dev.address}/{name}") as f:
>>> +            return f.read().strip()
>>> +    except FileNotFoundError:
>>> +        return None
>>> +
>>> +
>>> +def list_usb(args):
>>> +    vid, pid = [int(x, 16) for x in args.id.split(":", 1)]
>>> +
>>> +    print("Bus | Addr | Manufacturer     | Product          | ID        | Path")
>>> +    print("--- | ---- | ---------------- | ---------------- | --------- | ----")
>>> +    for dev in usb.core.find(find_all=True, idVendor=vid, idProduct=pid):
>>> +        path = path_from_usb_dev(dev) or ""
>>> +        manufacturer = try_get_usb_str(dev, "manufacturer") or "unknown"
>>> +        product = try_get_usb_str(dev, "product") or "unknown"
>>> +        print(
>>> +            f"{dev.bus:3} | {dev.address:4} | {manufacturer:16} | {product:16} | {dev.idVendor:04x}:{dev.idProduct:04x} | {path:18}"
>>> +        )
>>> +
>>> +
>>> +def connect(args):
>>> +    vid, pid = [int(x, 16) for x in args.id.split(":", 1)]
>>> +
>>> +    f = Forwarder(server=(args.server, args.port), vid=vid, pid=pid, path=args.path)
>>> +
>>> +    try:
>>> +        while True:
>>> +            f.c2s()
>>> +            f.s2c()
>>> +            f.log_stats_interval()
>>> +    finally:
>>> +        f.log_stats()
>>> +
>>> +
>>> +def main():
>>> +    parser = argparse.ArgumentParser(
>>> +        description="Forward 9PFS requests from USB to TCP",
>>> +    )
>>> +
>>> +    parser.add_argument("--id", type=str, default="1d6b:0109", help="vid:pid of target device")
>>> +    parser.add_argument("--path", type=str, default="", help="path of target device")
>>
>> I had to specify both --id and --path, otherwise I was getting
>> "device not found".
>
> This is odd. What was your list command saying about the available
> devices?
>

<snip>

[ 4260.577064] usb 3-1.3.3.5.4: new high-speed USB device number 25 using xhci_hcd
[ 4260.677769] usb 3-1.3.3.5.4: New USB device found, idVendor=abcd, idProduct=ef01, bcdDevice= 6.09
[ 4260.677788] usb 3-1.3.3.5.4: New USB device strings: Mfr=0, Product=0, SerialNumber=0

<snip>

$ sudo python3 tools/usb/p9_fwd.py --id 0xabcd:0xef01 list
Bus | Addr | Manufacturer | Product | ID | Path
--- | ---- | ---------------- | ---------------- | --------- | ----
3 | 25 | unknown | unknown | abcd:ef01 | 3-1.3.3.5.4


$ sudo python3 tools/usb/p9_fwd.py --id 0xabcd:0xef01 connect -p 9999
Traceback (most recent call last):
File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 243, in <module>
main()
File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 239, in main
args.func(args)
File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 190, in connect
f = Forwarder(server=(args.server, args.port), vid=vid, pid=pid, path=args.path)
File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 60, in __init__
raise ValueError("Device not found")
ValueError: Device not found

$ sudo python3 tools/usb/p9_fwd.py --path 3-1.3.3.5.4 connect -p 9999
Traceback (most recent call last):
File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 243, in <module>
main()
File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 239, in main
args.func(args)
File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 190, in connect
f = Forwarder(server=(args.server, args.port), vid=vid, pid=pid, path=args.path)
File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 60, in __init__
raise ValueError("Device not found")
ValueError: Device not found

$ sudo python3 tools/usb/p9_fwd.py --id 0xabcd:0xef01 --path 3-1.3.3.5.4 connect -p 9999
2024-05-31 09:34:44,047 INFO found device: 3/25 located at 3-1.3.3.5.4
2024-05-31 09:34:44,053 INFO claiming interface:
INTERFACE 0: Vendor Specific ===========================
bLength : 0x9 (9 bytes)
bDescriptorType : 0x4 Interface
bInterfaceNumber : 0x0
bAlternateSetting : 0x0
bNumEndpoints : 0x2
bInterfaceClass : 0xff Vendor Specific
bInterfaceSubClass : 0xff
bInterfaceProtocol : 0x9
iInterface : 0x1 usb9pfs input to output
ENDPOINT 0x81: Bulk IN ===============================
bLength : 0x7 (7 bytes)
bDescriptorType : 0x5 Endpoint
bEndpointAddress : 0x81 IN
bmAttributes : 0x2 Bulk
wMaxPacketSize : 0x200 (512 bytes)
bInterval : 0x0
ENDPOINT 0x1: Bulk OUT ===============================
bLength : 0x7 (7 bytes)
bDescriptorType : 0x5 Endpoint
bEndpointAddress : 0x1 OUT
bmAttributes : 0x2 Bulk
wMaxPacketSize : 0x200 (512 bytes)
bInterval : 0x0
2024-05-31 09:34:44,054 INFO interface claimed
2024-05-31 09:34:44,054 INFO connected to server

Regards,

Andrzej

> I tested this with the default vid:pid as described above (1d6b:0109)
> which worked fine with path alone.
>
> Michael
>
>>> +    parser.add_argument("-v", "--verbose", action="count", default=0)
>>> +
>>> +    subparsers = parser.add_subparsers()
>>> +    subparsers.required = True
>>> +    subparsers.dest = "command"
>>> +
>>> +    parser_list = subparsers.add_parser("list", help="List all connected 9p gadgets")
>>> +    parser_list.set_defaults(func=list_usb)
>>> +
>>> +    parser_connect = subparsers.add_parser(
>>> +        "connect", help="Forward messages between the usb9pfs gadget and the 9p server"
>>> +    )
>>> +    parser_connect.set_defaults(func=connect)
>>> +    connect_group = parser_connect.add_argument_group()
>>> +    connect_group.required = True
>>> +    parser_connect.add_argument("-s", "--server", type=str, default="127.0.0.1", help="server hostname")
>>> +    parser_connect.add_argument("-p", "--port", type=int, default=564, help="server port")> +
>>> +    args = parser.parse_args()
>>> +
>>> +    logging.TRACE = logging.DEBUG - 5
>>> +    logging.addLevelName(logging.TRACE, "TRACE")
>>> +
>>> +    if args.verbose >= 2:
>>> +        level = logging.TRACE
>>> +    elif args.verbose:
>>> +        level = logging.DEBUG
>>> +    else:
>>> +        level = logging.INFO
>>> +    logging.basicConfig(level=level, format="%(asctime)-15s %(levelname)-8s %(message)s")
>>> +
>>> +    args.func(args)
>>> +
>>> +
>>> +if __name__ == "__main__":
>>> +    main()
>>>
>>
>>
>


2024-06-05 14:17:24

by Michael Grzeschik

[permalink] [raw]
Subject: Re: [PATCH v5 3/3] tools: usb: p9_fwd: add usb gadget packet forwarder script

On Fri, May 31, 2024 at 09:37:23AM +0200, Andrzej Pietrasiewicz wrote:
>Hi,
>
>W dniu 29.05.2024 o?23:44, Michael Grzeschik pisze:
>>On Tue, May 28, 2024 at 08:41:12PM +0200, Andrzej Pietrasiewicz wrote:
>>>Hi,
>>>
>>>W dniu 28.05.2024 o?00:08, Michael Grzeschik pisze:
>>>>This patch is adding an small python tool to forward 9pfs requests
>>>>from the USB gadget to an existing 9pfs TCP server. Since currently all
>>>>9pfs servers lack support for the usb transport this tool is an useful
>>>>helper to get started.
>>>>
>>>>Refer the Documentation section "USBG Example" in
>>>>Documentation/filesystems/9p.rst on how to use it.
>>>>
>>>>Signed-off-by: Jan Luebbe <[email protected]>
>>>>Signed-off-by: Michael Grzeschik <[email protected]>
>>>>
>>>>---
>>>>v4 -> v5:
>>>>? - updated documentation for new subcommands list/connect
>>>>? - run ruff format
>>>>? - make vid and pid parameterized
>>>
>>>Thanks for adding that.
>>>
>>>>? - add list as subcommand to scan for devices
>>>>? - move connect to extra subcommand
>>>>v3 -> v4: -
>>>>v2 -> v3: -
>>>>v1 -> v2:
>>>>? - added usbg 9pfs detailed instructions to 9p.rst doc
>>>>---
>>>>?Documentation/filesystems/9p.rst |? 41 +++++++
>>>>?tools/usb/p9_fwd.py????????????? | 243 +++++++++++++++++++++++++++++++++++++++
>>>>?2 files changed, 284 insertions(+)
>>>>
>>>>diff --git a/Documentation/filesystems/9p.rst b/Documentation/filesystems/9p.rst
>>>>index 10cf79dc287f8..2cc85f3e8659f 100644
>>>>--- a/Documentation/filesystems/9p.rst
>>>>+++ b/Documentation/filesystems/9p.rst
>>>>@@ -67,6 +67,47 @@ To mount a 9p FS on a USB Host accessible via the gadget as root filesystem::
>>>>?where <device> is the tag associated by the usb gadget transport.
>>>>?It is defined by the configfs instance name.
>>>>+USBG Example
>>>>+============
>>>>+
>>>>+The USB host exports a filesystem, while the gadget on the USB device
>>>>+side makes it mountable.
>>>>+
>>>>+Diod (9pfs server) and the forwarder are on the development host, where
>>>>+the root filesystem is actually stored. The gadget is initialized during
>>>>+boot (or later) on the embedded board. Then the forwarder will find it
>>>>+on the USB bus and start forwarding requests.
>>>>+
>>>>+In this case the 9p requests come from the device and are handled by the
>>>>+host. The reason is that USB device ports are normally not available on
>>>>+PCs, so a connection in the other direction would not work.
>>>>+
>>>>+When using the usbg transport, for now there is no native usb host
>>>>+service capable to handle the requests from the gadget driver. For
>>>>+this we have to use the extra python tool p9_fwd.py from tools/usb.
>>>>+
>>>>+Just start the 9pfs capable network server like diod/nfs-ganesha e.g.:
>>>>+
>>>>+??????? $ diod -f -n -d 0 -S -l 0.0.0.0:9999 -e $PWD
>>>>+
>>>>+Optionaly scan your bus if there are more then one usbg gadgets to find their path:
>>>>+
>>>>+??????? $ python $kernel_dir/tools/usb/p9_fwd.py list
>>>>+
>>>>+??????? Bus | Addr | Manufacturer???? | Product????????? | ID??????? | Path
>>>>+??????? --- | ---- | ---------------- | ---------------- | --------- | ----
>>>>+????????? 2 |?? 67 | unknown????????? | unknown????????? | 1d6b:0109 | 2-1.1.2
>>>>+????????? 2 |?? 68 | unknown????????? | unknown????????? | 1d6b:0109 | 2-1.1.3
>>>>+
>>>>+Then start the python transport:
>>>>+
>>>>+??????? $ python $kernel_dir/tools/usb/p9_fwd.py --path 2-1.1.2 connect -p 9999
>>>>+
>>>>+After that the gadget driver can be used as described above.
>>>>+
>>>>+One use-case is to use it as an alternative to NFS root booting during
>>>>+the development of embedded Linux devices.
>>>>+
>>>>?Options
>>>>?=======
>>>>diff --git a/tools/usb/p9_fwd.py b/tools/usb/p9_fwd.py
>>>>new file mode 100755
>>>>index 0000000000000..7bedefce75c7b
>>>>--- /dev/null
>>>>+++ b/tools/usb/p9_fwd.py
>>>>@@ -0,0 +1,243 @@
>>>>+#!/usr/bin/env python3
>>>>+# SPDX-License-Identifier: GPL-2.0
>>>>+
>>>>+import argparse
>>>>+import errno
>>>>+import logging
>>>>+import socket
>>>>+import struct
>>>>+import time
>>>>+
>>>>+import usb.core
>>>>+import usb.util
>>>>+
>>>>+
>>>>+def path_from_usb_dev(dev):
>>>>+??? """Takes a pyUSB device as argument and returns a string.
>>>>+??? The string is a Path representation of the position of the USB device on the USB bus tree.
>>>>+
>>>>+??? This path is used to find a USB device on the bus or all devices connected to a HUB.
>>>>+??? The path is made up of the number of the USB controller followed be the ports of the HUB tree."""
>>>>+??? if dev.port_numbers:
>>>>+??????? dev_path = ".".join(str(i) for i in dev.port_numbers)
>>>>+??????? return f"{dev.bus}-{dev_path}"
>>>>+??? return ""
>>>>+
>>>>+
>>>>+HEXDUMP_FILTER = "".join(chr(x).isprintable() and chr(x) or "." for x in range(128)) + "." * 128
>>>>+
>>>>+
>>>>+class Forwarder:
>>>>+??? @staticmethod
>>>>+??? def _log_hexdump(data):
>>>>+??????? if not logging.root.isEnabledFor(logging.TRACE):
>>>>+??????????? return
>>>>+??????? L = 16
>>>>+??????? for c in range(0, len(data), L):
>>>>+??????????? chars = data[c : c + L]
>>>>+??????????? dump = " ".join(f"{x:02x}" for x in chars)
>>>>+??????????? printable = "".join(HEXDUMP_FILTER[x] for x in chars)
>>>>+??????????? line = f"{c:08x}? {dump:{L*3}s} |{printable:{L}s}|"
>>>>+??????????? logging.root.log(logging.TRACE, "%s", line)
>>>>+
>>>>+??? def __init__(self, server, vid, pid, path):
>>>>+??????? self.stats = {
>>>>+??????????? "c2s packets": 0,
>>>>+??????????? "c2s bytes": 0,
>>>>+??????????? "s2c packets": 0,
>>>>+??????????? "s2c bytes": 0,
>>>>+??????? }
>>>>+??????? self.stats_logged = time.monotonic()
>>>>+
>>>>+??????? def find_filter(dev):
>>>>+??????????? dev_path = path_from_usb_dev(dev)
>>>>+??????????? if path is not None:
>>>>+??????????????? return dev_path == path
>>>>+??????????? return True
>>>>+
>>>>+??????? dev = usb.core.find(idVendor=vid, idProduct=pid, custom_match=find_filter)
>>>>+??????? if dev is None:
>>>>+??????????? raise ValueError("Device not found")
>>>>+
>>>>+??????? logging.info(f"found device: {dev.bus}/{dev.address} located at {path_from_usb_dev(dev)}")
>>>>+
>>>>+??????? # dev.set_configuration() is not necessary since g_multi has only one
>>>>+??????? usb9pfs = None
>>>>+??????? # g_multi adds 9pfs as last interface
>>>>+??????? cfg = dev.get_active_configuration()
>>>>+??????? for intf in cfg:
>>>>+??????????? # we have to detach the usb-storage driver from multi gadget since
>>>>+??????????? # stall option could be set, which will lead to spontaneous port
>>>>+??????????? # resets and our transfers will run dead
>>>>+??????????? if intf.bInterfaceClass == 0x08:
>>>>+??????????????? if dev.is_kernel_driver_active(intf.bInterfaceNumber):
>>>>+??????????????????? dev.detach_kernel_driver(intf.bInterfaceNumber)
>>>>+
>>>>+??????????? if intf.bInterfaceClass == 0xFF and intf.bInterfaceSubClass == 0xFF and intf.bInterfaceProtocol == 0x09:
>>>>+??????????????? usb9pfs = intf
>>>>+??????? if usb9pfs is None:
>>>>+??????????? raise ValueError("Interface not found")
>>>>+
>>>>+??????? logging.info(f"claiming interface:\n{usb9pfs}")
>>>>+??????? usb.util.claim_interface(dev, usb9pfs.bInterfaceNumber)
>>>>+??????? ep_out = usb.util.find_descriptor(
>>>>+??????????? usb9pfs,
>>>>+??????????? custom_match=lambda e: usb.util.endpoint_direction(e.bEndpointAddress) == usb.util.ENDPOINT_OUT,
>>>>+??????? )
>>>>+??????? assert ep_out is not None
>>>>+??????? ep_in = usb.util.find_descriptor(
>>>>+??????????? usb9pfs,
>>>>+??????????? custom_match=lambda e: usb.util.endpoint_direction(e.bEndpointAddress) == usb.util.ENDPOINT_IN,
>>>>+??????? )
>>>>+??????? assert ep_in is not None
>>>>+??????? logging.info("interface claimed")
>>>>+
>>>>+??????? self.ep_out = ep_out
>>>>+??????? self.ep_in = ep_in
>>>>+??????? self.dev = dev
>>>>+
>>>>+??????? # create and connect socket
>>>>+??????? self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>>>+??????? self.s.connect(server)
>>>>+
>>>>+??????? logging.info("connected to server")
>>>>+
>>>>+??? def c2s(self):
>>>>+??????? """forward a request from the USB client to the TCP server"""
>>>>+??????? data = None
>>>>+??????? while data is None:
>>>>+??????????? try:
>>>>+??????????????? logging.log(logging.TRACE, "c2s: reading")
>>>>+??????????????? data = self.ep_in.read(self.ep_in.wMaxPacketSize)
>>>>+??????????? except usb.core.USBTimeoutError:
>>>>+??????????????? logging.log(logging.TRACE, "c2s: reading timed out")
>>>>+??????????????? continue
>>>>+??????????? except usb.core.USBError as e:
>>>>+??????????????? if e.errno == errno.EIO:
>>>>+??????????????????? logging.debug("c2s: reading failed with %s, retrying", repr(e))
>>>>+??????????????????? time.sleep(0.5)
>>>>+??????????????????? continue
>>>>+??????????????? logging.error("c2s: reading failed with %s, aborting", repr(e))
>>>>+??????????????? raise
>>>>+??????? size = struct.unpack("<I", data[:4])[0]
>>>>+??????? while len(data) < size:
>>>>+??????????? data += self.ep_in.read(size - len(data))
>>>>+??????? logging.log(logging.TRACE, "c2s: writing")
>>>>+??????? self._log_hexdump(data)
>>>>+??????? self.s.send(data)
>>>>+??????? logging.debug("c2s: forwarded %i bytes", size)
>>>>+??????? self.stats["c2s packets"] += 1
>>>>+??????? self.stats["c2s bytes"] += size
>>>>+
>>>>+??? def s2c(self):
>>>>+??????? """forward a response from the TCP server to the USB client"""
>>>>+??????? logging.log(logging.TRACE, "s2c: reading")
>>>>+??????? data = self.s.recv(4)
>>>>+??????? size = struct.unpack("<I", data[:4])[0]
>>>>+??????? while len(data) < size:
>>>>+??????????? data += self.s.recv(size - len(data))
>>>>+??????? logging.log(logging.TRACE, "s2c: writing")
>>>>+??????? self._log_hexdump(data)
>>>>+??????? while data:
>>>>+??????????? written = self.ep_out.write(data)
>>>>+??????????? assert written > 0
>>>>+??????????? data = data[written:]
>>>>+??????? if size % self.ep_out.wMaxPacketSize == 0:
>>>>+??????????? logging.log(logging.TRACE, "sending zero length packet")
>>>>+??????????? self.ep_out.write(b"")
>>>>+??????? logging.debug("s2c: forwarded %i bytes", size)
>>>>+??????? self.stats["s2c packets"] += 1
>>>>+??????? self.stats["s2c bytes"] += size
>>>>+
>>>>+??? def log_stats(self):
>>>>+??????? logging.info("statistics:")
>>>>+??????? for k, v in self.stats.items():
>>>>+??????????? logging.info(f"? {k+':':14s} {v}")
>>>>+
>>>>+??? def log_stats_interval(self, interval=5):
>>>>+??????? if (time.monotonic() - self.stats_logged) < interval:
>>>>+??????????? return
>>>>+
>>>>+??????? self.log_stats()
>>>>+??????? self.stats_logged = time.monotonic()
>>>>+
>>>>+
>>>>+def try_get_usb_str(dev, name):
>>>>+??? try:
>>>>+??????? with open(f"/sys/bus/usb/devices/{dev.bus}-{dev.address}/{name}") as f:
>>>>+??????????? return f.read().strip()
>>>>+??? except FileNotFoundError:
>>>>+??????? return None
>>>>+
>>>>+
>>>>+def list_usb(args):
>>>>+??? vid, pid = [int(x, 16) for x in args.id.split(":", 1)]
>>>>+
>>>>+??? print("Bus | Addr | Manufacturer???? | Product????????? | ID??????? | Path")
>>>>+??? print("--- | ---- | ---------------- | ---------------- | --------- | ----")
>>>>+??? for dev in usb.core.find(find_all=True, idVendor=vid, idProduct=pid):
>>>>+??????? path = path_from_usb_dev(dev) or ""
>>>>+??????? manufacturer = try_get_usb_str(dev, "manufacturer") or "unknown"
>>>>+??????? product = try_get_usb_str(dev, "product") or "unknown"
>>>>+??????? print(
>>>>+??????????? f"{dev.bus:3} | {dev.address:4} | {manufacturer:16} | {product:16} | {dev.idVendor:04x}:{dev.idProduct:04x} | {path:18}"
>>>>+??????? )
>>>>+
>>>>+
>>>>+def connect(args):
>>>>+??? vid, pid = [int(x, 16) for x in args.id.split(":", 1)]
>>>>+
>>>>+??? f = Forwarder(server=(args.server, args.port), vid=vid, pid=pid, path=args.path)
>>>>+
>>>>+??? try:
>>>>+??????? while True:
>>>>+??????????? f.c2s()
>>>>+??????????? f.s2c()
>>>>+??????????? f.log_stats_interval()
>>>>+??? finally:
>>>>+??????? f.log_stats()
>>>>+
>>>>+
>>>>+def main():
>>>>+??? parser = argparse.ArgumentParser(
>>>>+??????? description="Forward 9PFS requests from USB to TCP",
>>>>+??? )
>>>>+
>>>>+??? parser.add_argument("--id", type=str, default="1d6b:0109", help="vid:pid of target device")
>>>>+??? parser.add_argument("--path", type=str, default="", help="path of target device")
>>>
>>>I had to specify both --id and --path, otherwise I was getting
>>>"device not found".
>>
>>This is odd. What was your list command saying about the available
>>devices?
>>
>
><snip>
>
>[ 4260.577064] usb 3-1.3.3.5.4: new high-speed USB device number 25 using xhci_hcd
>[ 4260.677769] usb 3-1.3.3.5.4: New USB device found, idVendor=abcd, idProduct=ef01, bcdDevice= 6.09
>[ 4260.677788] usb 3-1.3.3.5.4: New USB device strings: Mfr=0, Product=0, SerialNumber=0
>
><snip>
>
>$ sudo python3 tools/usb/p9_fwd.py --id 0xabcd:0xef01 list
>Bus | Addr | Manufacturer | Product | ID | Path
>--- | ---- | ---------------- | ---------------- | --------- | ----
> 3 | 25 | unknown | unknown | abcd:ef01 | 3-1.3.3.5.4
>
>
>$ sudo python3 tools/usb/p9_fwd.py --id 0xabcd:0xef01 connect -p 9999
>Traceback (most recent call last):
> File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 243, in <module>
> main()
> File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 239, in main
> args.func(args)
> File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 190, in connect
> f = Forwarder(server=(args.server, args.port), vid=vid, pid=pid, path=args.path)
> File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 60, in __init__
> raise ValueError("Device not found")
>ValueError: Device not found
>
>$ sudo python3 tools/usb/p9_fwd.py --path 3-1.3.3.5.4 connect -p 9999
>Traceback (most recent call last):
> File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 243, in <module>
> main()
> File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 239, in main
> args.func(args)
> File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 190, in connect
> f = Forwarder(server=(args.server, args.port), vid=vid, pid=pid, path=args.path)
> File "/home/ap/Collabora/kernel-rk/tools/usb/p9_fwd.py", line 60, in __init__
> raise ValueError("Device not found")
>ValueError: Device not found
>
>$ sudo python3 tools/usb/p9_fwd.py --id 0xabcd:0xef01 --path 3-1.3.3.5.4 connect -p 9999
>2024-05-31 09:34:44,047 INFO found device: 3/25 located at 3-1.3.3.5.4
>2024-05-31 09:34:44,053 INFO claiming interface:
> INTERFACE 0: Vendor Specific ===========================
> bLength : 0x9 (9 bytes)
> bDescriptorType : 0x4 Interface
> bInterfaceNumber : 0x0
> bAlternateSetting : 0x0
> bNumEndpoints : 0x2
> bInterfaceClass : 0xff Vendor Specific
> bInterfaceSubClass : 0xff
> bInterfaceProtocol : 0x9
> iInterface : 0x1 usb9pfs input to output
> ENDPOINT 0x81: Bulk IN ===============================
> bLength : 0x7 (7 bytes)
> bDescriptorType : 0x5 Endpoint
> bEndpointAddress : 0x81 IN
> bmAttributes : 0x2 Bulk
> wMaxPacketSize : 0x200 (512 bytes)
> bInterval : 0x0
> ENDPOINT 0x1: Bulk OUT ===============================
> bLength : 0x7 (7 bytes)
> bDescriptorType : 0x5 Endpoint
> bEndpointAddress : 0x1 OUT
> bmAttributes : 0x2 Bulk
> wMaxPacketSize : 0x200 (512 bytes)
> bInterval : 0x0
>2024-05-31 09:34:44,054 INFO interface claimed
>2024-05-31 09:34:44,054 INFO connected to server

Ok, there was a bug that args.path wath never None. Therefor it was
always necessary. While there I also fixed the short window
between gadget load and mount, where the user could actually
use unprepared endpoints which led to some NULL pointer derefs.

Please try my last series v6:

https://lore.kernel.org/all/[email protected]/

Regards,
Michael

>>I tested this with the default vid:pid as described above (1d6b:0109)
>>which worked fine with path alone.
>>
>>Michael
>>
>>>>+??? parser.add_argument("-v", "--verbose", action="count", default=0)
>>>>+
>>>>+??? subparsers = parser.add_subparsers()
>>>>+??? subparsers.required = True
>>>>+??? subparsers.dest = "command"
>>>>+
>>>>+??? parser_list = subparsers.add_parser("list", help="List all connected 9p gadgets")
>>>>+??? parser_list.set_defaults(func=list_usb)
>>>>+
>>>>+??? parser_connect = subparsers.add_parser(
>>>>+??????? "connect", help="Forward messages between the usb9pfs gadget and the 9p server"
>>>>+??? )
>>>>+??? parser_connect.set_defaults(func=connect)
>>>>+??? connect_group = parser_connect.add_argument_group()
>>>>+??? connect_group.required = True
>>>>+??? parser_connect.add_argument("-s", "--server", type=str, default="127.0.0.1", help="server hostname")
>>>>+??? parser_connect.add_argument("-p", "--port", type=int, default=564, help="server port")> +
>>>>+??? args = parser.parse_args()
>>>>+
>>>>+??? logging.TRACE = logging.DEBUG - 5
>>>>+??? logging.addLevelName(logging.TRACE, "TRACE")
>>>>+
>>>>+??? if args.verbose >= 2:
>>>>+??????? level = logging.TRACE
>>>>+??? elif args.verbose:
>>>>+??????? level = logging.DEBUG
>>>>+??? else:
>>>>+??????? level = logging.INFO
>>>>+??? logging.basicConfig(level=level, format="%(asctime)-15s %(levelname)-8s %(message)s")
>>>>+
>>>>+??? args.func(args)
>>>>+
>>>>+
>>>>+if __name__ == "__main__":
>>>>+??? main()
>>>>
>>>
>>>
>>
>
>

--
Pengutronix e.K. | |
Steuerwalder Str. 21 | http://www.pengutronix.de/ |
31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |


Attachments:
(No filename) (17.88 kB)
signature.asc (849.00 B)
Download all attachments