Subject: [PATCH V3] usb: dwc3: gadget: Fix suspend/resume warning when no-gadget is connected

This patch avoid to disconnect an already gadget in not connected state

[ 45.597274] dwc3 31000000.usb: wait for SETUP phase timed out
[ 45.599140] dwc3 31000000.usb: failed to set STALL on ep0out
[ 45.601069] ------------[ cut here ]------------
[ 45.601073] WARNING: CPU: 0 PID: 150 at drivers/usb/dwc3/ep0.c:289 dwc3_ep0_out_start+0xcc/0xd4
[ 45.601102] Modules linked in: cfg80211 rfkill ipv6 rpmsg_ctrl rpmsg_char crct10dif_ce rti_wdt k3_j72xx_bandgap rtc_ti_k3 omap_mailbox sa2ul authenc [last unloaded: ti_k3_r5_remoteproc]
[ 45.601151] CPU: 0 PID: 150 Comm: sh Not tainted 6.8.0-rc5 #1
[ 45.601159] Hardware name: BSH - CCM-M3 (DT)
[ 45.601164] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 45.601172] pc : dwc3_ep0_out_start+0xcc/0xd4
[ 45.601179] lr : dwc3_ep0_out_start+0x50/0xd4
[ 45.601186] sp : ffff8000832739e0
[ 45.601189] x29: ffff8000832739e0 x28: ffff800082a21000 x27: ffff8000808dc630
[ 45.601200] x26: 0000000000000002 x25: ffff800082530a44 x24: 0000000000000000
[ 45.601210] x23: ffff000000e079a0 x22: ffff000000e07a68 x21: 0000000000000001
[ 45.601219] x20: ffff000000e07880 x19: ffff000000e07880 x18: 0000000000000040
[ 45.601229] x17: ffff7fff8e1ce000 x16: ffff800080000000 x15: fffffffffffe5260
[ 45.601239] x14: 0000000000000000 x13: 206e6f204c4c4154 x12: 5320746573206f74
[ 45.601249] x11: 0000000000000001 x10: 000000000000000a x9 : ffff800083273930
[ 45.601259] x8 : 000000000000000a x7 : ffffffffffff3f0c x6 : ffffffffffff3f00
[ 45.601268] x5 : ffffffffffff3f0c x4 : 0000000000000000 x3 : 0000000000000000
[ 45.601278] x2 : 0000000000000000 x1 : ffff000004e7e600 x0 : 00000000ffffff92
[ 45.601289] Call trace:
[ 45.601293] dwc3_ep0_out_start+0xcc/0xd4
[ 45.601301] dwc3_ep0_stall_and_restart+0x98/0xbc
[ 45.601309] dwc3_ep0_reset_state+0x5c/0x88
[ 45.601315] dwc3_gadget_soft_disconnect+0x144/0x160
[ 45.601323] dwc3_gadget_suspend+0x18/0xb0
[ 45.601329] dwc3_suspend_common+0x5c/0x18c
[ 45.601341] dwc3_suspend+0x20/0x44
[ 45.601350] platform_pm_suspend+0x2c/0x6c
[ 45.601360] __device_suspend+0x10c/0x34c
[ 45.601372] dpm_suspend+0x1a8/0x240
[ 45.601382] dpm_suspend_start+0x80/0x9c
[ 45.601391] suspend_devices_and_enter+0x1c4/0x584
[ 45.601402] pm_suspend+0x1b0/0x264
[ 45.601408] state_store+0x80/0xec
[ 45.601415] kobj_attr_store+0x18/0x2c
[ 45.601426] sysfs_kf_write+0x44/0x54
[ 45.601434] kernfs_fop_write_iter+0x120/0x1ec
[ 45.601445] vfs_write+0x23c/0x358
[ 45.601458] ksys_write+0x70/0x104
[ 45.601467] __arm64_sys_write+0x1c/0x28
[ 45.601477] invoke_syscall+0x48/0x114
[ 45.601488] el0_svc_common.constprop.0+0x40/0xe0
[ 45.601498] do_el0_svc+0x1c/0x28
[ 45.601506] el0_svc+0x34/0xb8
[ 45.601516] el0t_64_sync_handler+0x100/0x12c
[ 45.601522] el0t_64_sync+0x190/0x194
[ 45.601531] ---[ end trace 0000000000000000 ]---
[ 45.608794] Disabling non-boot CPUs ...
[ 45.611029] psci: CPU1 killed (polled 0 ms)
[ 45.611837] Enabling non-boot CPUs ...
[ 45.612247] Detected VIPT I-cache on CPU1

Tested on a am62x board with a usbnet gadget

Fixes: 61a348857e86 ("usb: dwc3: gadget: Fix NULL pointer dereference in dwc3_gadget_suspend)
Cc: [email protected]
Signed-off-by: Michael Trimarchi <[email protected]>
---
V2->V3:
- Change the logic of the patch using the gadget connected state
- Change of the commit message
V1->V2:
- Add stable in CC
---
drivers/usb/dwc3/gadget.c | 9 +++++++++
1 file changed, 9 insertions(+)

diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
index 4c8dd6724678..a7316a1703ad 100644
--- a/drivers/usb/dwc3/gadget.c
+++ b/drivers/usb/dwc3/gadget.c
@@ -2650,6 +2650,15 @@ static int dwc3_gadget_soft_disconnect(struct dwc3 *dwc)
int ret;

spin_lock_irqsave(&dwc->lock, flags);
+ /*
+ * Attempt to disconnect a no connected gadget
+ */
+ if (!dwc->connected) {
+ dev_warn(dwc->dev, "No connected device\n");
+ spin_unlock_irqrestore(&dwc->lock, flags);
+ return 0;
+ }
+
dwc->connected = false;

/*
--
2.40.1



Subject: Re: [PATCH V3] usb: dwc3: gadget: Fix suspend/resume warning when no-gadget is connected

Hi Greg

On Mon, Feb 26, 2024 at 11:14 AM Greg Kroah-Hartman
<[email protected]> wrote:
>
> On Mon, Feb 26, 2024 at 11:05:02AM +0100, Michael Trimarchi wrote:
> > This patch avoid to disconnect an already gadget in not connected state
> >
> > [ 45.597274] dwc3 31000000.usb: wait for SETUP phase timed out
> > [ 45.599140] dwc3 31000000.usb: failed to set STALL on ep0out
> > [ 45.601069] ------------[ cut here ]------------
> > [ 45.601073] WARNING: CPU: 0 PID: 150 at drivers/usb/dwc3/ep0.c:289 dwc3_ep0_out_start+0xcc/0xd4
> > [ 45.601102] Modules linked in: cfg80211 rfkill ipv6 rpmsg_ctrl rpmsg_char crct10dif_ce rti_wdt k3_j72xx_bandgap rtc_ti_k3 omap_mailbox sa2ul authenc [last unloaded: ti_k3_r5_remoteproc]
> > [ 45.601151] CPU: 0 PID: 150 Comm: sh Not tainted 6.8.0-rc5 #1
> > [ 45.601159] Hardware name: BSH - CCM-M3 (DT)
> > [ 45.601164] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> > [ 45.601172] pc : dwc3_ep0_out_start+0xcc/0xd4
> > [ 45.601179] lr : dwc3_ep0_out_start+0x50/0xd4
> > [ 45.601186] sp : ffff8000832739e0
> > [ 45.601189] x29: ffff8000832739e0 x28: ffff800082a21000 x27: ffff8000808dc630
> > [ 45.601200] x26: 0000000000000002 x25: ffff800082530a44 x24: 0000000000000000
> > [ 45.601210] x23: ffff000000e079a0 x22: ffff000000e07a68 x21: 0000000000000001
> > [ 45.601219] x20: ffff000000e07880 x19: ffff000000e07880 x18: 0000000000000040
> > [ 45.601229] x17: ffff7fff8e1ce000 x16: ffff800080000000 x15: fffffffffffe5260
> > [ 45.601239] x14: 0000000000000000 x13: 206e6f204c4c4154 x12: 5320746573206f74
> > [ 45.601249] x11: 0000000000000001 x10: 000000000000000a x9 : ffff800083273930
> > [ 45.601259] x8 : 000000000000000a x7 : ffffffffffff3f0c x6 : ffffffffffff3f00
> > [ 45.601268] x5 : ffffffffffff3f0c x4 : 0000000000000000 x3 : 0000000000000000
> > [ 45.601278] x2 : 0000000000000000 x1 : ffff000004e7e600 x0 : 00000000ffffff92
> > [ 45.601289] Call trace:
> > [ 45.601293] dwc3_ep0_out_start+0xcc/0xd4
> > [ 45.601301] dwc3_ep0_stall_and_restart+0x98/0xbc
> > [ 45.601309] dwc3_ep0_reset_state+0x5c/0x88
> > [ 45.601315] dwc3_gadget_soft_disconnect+0x144/0x160
> > [ 45.601323] dwc3_gadget_suspend+0x18/0xb0
> > [ 45.601329] dwc3_suspend_common+0x5c/0x18c
> > [ 45.601341] dwc3_suspend+0x20/0x44
> > [ 45.601350] platform_pm_suspend+0x2c/0x6c
> > [ 45.601360] __device_suspend+0x10c/0x34c
> > [ 45.601372] dpm_suspend+0x1a8/0x240
> > [ 45.601382] dpm_suspend_start+0x80/0x9c
> > [ 45.601391] suspend_devices_and_enter+0x1c4/0x584
> > [ 45.601402] pm_suspend+0x1b0/0x264
> > [ 45.601408] state_store+0x80/0xec
> > [ 45.601415] kobj_attr_store+0x18/0x2c
> > [ 45.601426] sysfs_kf_write+0x44/0x54
> > [ 45.601434] kernfs_fop_write_iter+0x120/0x1ec
> > [ 45.601445] vfs_write+0x23c/0x358
> > [ 45.601458] ksys_write+0x70/0x104
> > [ 45.601467] __arm64_sys_write+0x1c/0x28
> > [ 45.601477] invoke_syscall+0x48/0x114
> > [ 45.601488] el0_svc_common.constprop.0+0x40/0xe0
> > [ 45.601498] do_el0_svc+0x1c/0x28
> > [ 45.601506] el0_svc+0x34/0xb8
> > [ 45.601516] el0t_64_sync_handler+0x100/0x12c
> > [ 45.601522] el0t_64_sync+0x190/0x194
> > [ 45.601531] ---[ end trace 0000000000000000 ]---
> > [ 45.608794] Disabling non-boot CPUs ...
> > [ 45.611029] psci: CPU1 killed (polled 0 ms)
> > [ 45.611837] Enabling non-boot CPUs ...
> > [ 45.612247] Detected VIPT I-cache on CPU1
> >
> > Tested on a am62x board with a usbnet gadget
> >
> > Fixes: 61a348857e86 ("usb: dwc3: gadget: Fix NULL pointer dereference in dwc3_gadget_suspend)
> > Cc: [email protected]
> > Signed-off-by: Michael Trimarchi <[email protected]>
> > ---
> > V2->V3:
> > - Change the logic of the patch using the gadget connected state
> > - Change of the commit message
> > V1->V2:
> > - Add stable in CC
> > ---
> > drivers/usb/dwc3/gadget.c | 9 +++++++++
> > 1 file changed, 9 insertions(+)
> >
> > diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
> > index 4c8dd6724678..a7316a1703ad 100644
> > --- a/drivers/usb/dwc3/gadget.c
> > +++ b/drivers/usb/dwc3/gadget.c
> > @@ -2650,6 +2650,15 @@ static int dwc3_gadget_soft_disconnect(struct dwc3 *dwc)
> > int ret;
> >
> > spin_lock_irqsave(&dwc->lock, flags);
> > + /*
> > + * Attempt to disconnect a no connected gadget
>
> What does this mean? And why a 3 line comment?
>
> > + */
> > + if (!dwc->connected) {
> > + dev_warn(dwc->dev, "No connected device\n");
>
> You are printing while a spinlock is held? What can userspace do with
> this message?

I will drop it.

>
> > + spin_unlock_irqrestore(&dwc->lock, flags);
> > + return 0;
>
> No error handling? Why not?

The function tries to disconnect an already disconnected gadget, so in
this case is a nop. If we want to handle
an error I can check the impact on the dwc3 gadget.

Michael

>
> thanks,
>
> greg k-h



--
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
[email protected]
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
[email protected]
http://www.amarulasolutions.com

2024-02-26 10:37:04

by Greg KH

[permalink] [raw]
Subject: Re: [PATCH V3] usb: dwc3: gadget: Fix suspend/resume warning when no-gadget is connected

On Mon, Feb 26, 2024 at 11:05:02AM +0100, Michael Trimarchi wrote:
> This patch avoid to disconnect an already gadget in not connected state
>
> [ 45.597274] dwc3 31000000.usb: wait for SETUP phase timed out
> [ 45.599140] dwc3 31000000.usb: failed to set STALL on ep0out
> [ 45.601069] ------------[ cut here ]------------
> [ 45.601073] WARNING: CPU: 0 PID: 150 at drivers/usb/dwc3/ep0.c:289 dwc3_ep0_out_start+0xcc/0xd4
> [ 45.601102] Modules linked in: cfg80211 rfkill ipv6 rpmsg_ctrl rpmsg_char crct10dif_ce rti_wdt k3_j72xx_bandgap rtc_ti_k3 omap_mailbox sa2ul authenc [last unloaded: ti_k3_r5_remoteproc]
> [ 45.601151] CPU: 0 PID: 150 Comm: sh Not tainted 6.8.0-rc5 #1
> [ 45.601159] Hardware name: BSH - CCM-M3 (DT)
> [ 45.601164] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [ 45.601172] pc : dwc3_ep0_out_start+0xcc/0xd4
> [ 45.601179] lr : dwc3_ep0_out_start+0x50/0xd4
> [ 45.601186] sp : ffff8000832739e0
> [ 45.601189] x29: ffff8000832739e0 x28: ffff800082a21000 x27: ffff8000808dc630
> [ 45.601200] x26: 0000000000000002 x25: ffff800082530a44 x24: 0000000000000000
> [ 45.601210] x23: ffff000000e079a0 x22: ffff000000e07a68 x21: 0000000000000001
> [ 45.601219] x20: ffff000000e07880 x19: ffff000000e07880 x18: 0000000000000040
> [ 45.601229] x17: ffff7fff8e1ce000 x16: ffff800080000000 x15: fffffffffffe5260
> [ 45.601239] x14: 0000000000000000 x13: 206e6f204c4c4154 x12: 5320746573206f74
> [ 45.601249] x11: 0000000000000001 x10: 000000000000000a x9 : ffff800083273930
> [ 45.601259] x8 : 000000000000000a x7 : ffffffffffff3f0c x6 : ffffffffffff3f00
> [ 45.601268] x5 : ffffffffffff3f0c x4 : 0000000000000000 x3 : 0000000000000000
> [ 45.601278] x2 : 0000000000000000 x1 : ffff000004e7e600 x0 : 00000000ffffff92
> [ 45.601289] Call trace:
> [ 45.601293] dwc3_ep0_out_start+0xcc/0xd4
> [ 45.601301] dwc3_ep0_stall_and_restart+0x98/0xbc
> [ 45.601309] dwc3_ep0_reset_state+0x5c/0x88
> [ 45.601315] dwc3_gadget_soft_disconnect+0x144/0x160
> [ 45.601323] dwc3_gadget_suspend+0x18/0xb0
> [ 45.601329] dwc3_suspend_common+0x5c/0x18c
> [ 45.601341] dwc3_suspend+0x20/0x44
> [ 45.601350] platform_pm_suspend+0x2c/0x6c
> [ 45.601360] __device_suspend+0x10c/0x34c
> [ 45.601372] dpm_suspend+0x1a8/0x240
> [ 45.601382] dpm_suspend_start+0x80/0x9c
> [ 45.601391] suspend_devices_and_enter+0x1c4/0x584
> [ 45.601402] pm_suspend+0x1b0/0x264
> [ 45.601408] state_store+0x80/0xec
> [ 45.601415] kobj_attr_store+0x18/0x2c
> [ 45.601426] sysfs_kf_write+0x44/0x54
> [ 45.601434] kernfs_fop_write_iter+0x120/0x1ec
> [ 45.601445] vfs_write+0x23c/0x358
> [ 45.601458] ksys_write+0x70/0x104
> [ 45.601467] __arm64_sys_write+0x1c/0x28
> [ 45.601477] invoke_syscall+0x48/0x114
> [ 45.601488] el0_svc_common.constprop.0+0x40/0xe0
> [ 45.601498] do_el0_svc+0x1c/0x28
> [ 45.601506] el0_svc+0x34/0xb8
> [ 45.601516] el0t_64_sync_handler+0x100/0x12c
> [ 45.601522] el0t_64_sync+0x190/0x194
> [ 45.601531] ---[ end trace 0000000000000000 ]---
> [ 45.608794] Disabling non-boot CPUs ...
> [ 45.611029] psci: CPU1 killed (polled 0 ms)
> [ 45.611837] Enabling non-boot CPUs ...
> [ 45.612247] Detected VIPT I-cache on CPU1
>
> Tested on a am62x board with a usbnet gadget
>
> Fixes: 61a348857e86 ("usb: dwc3: gadget: Fix NULL pointer dereference in dwc3_gadget_suspend)
> Cc: [email protected]
> Signed-off-by: Michael Trimarchi <[email protected]>
> ---
> V2->V3:
> - Change the logic of the patch using the gadget connected state
> - Change of the commit message
> V1->V2:
> - Add stable in CC
> ---
> drivers/usb/dwc3/gadget.c | 9 +++++++++
> 1 file changed, 9 insertions(+)
>
> diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
> index 4c8dd6724678..a7316a1703ad 100644
> --- a/drivers/usb/dwc3/gadget.c
> +++ b/drivers/usb/dwc3/gadget.c
> @@ -2650,6 +2650,15 @@ static int dwc3_gadget_soft_disconnect(struct dwc3 *dwc)
> int ret;
>
> spin_lock_irqsave(&dwc->lock, flags);
> + /*
> + * Attempt to disconnect a no connected gadget

What does this mean? And why a 3 line comment?

> + */
> + if (!dwc->connected) {
> + dev_warn(dwc->dev, "No connected device\n");

You are printing while a spinlock is held? What can userspace do with
this message?

> + spin_unlock_irqrestore(&dwc->lock, flags);
> + return 0;

No error handling? Why not?

thanks,

greg k-h

2024-02-26 22:37:10

by Thinh Nguyen

[permalink] [raw]
Subject: Re: [PATCH V3] usb: dwc3: gadget: Fix suspend/resume warning when no-gadget is connected

Hi,

On Mon, Feb 26, 2024, Michael Trimarchi wrote:
> This patch avoid to disconnect an already gadget in not connected state
>
> [ 45.597274] dwc3 31000000.usb: wait for SETUP phase timed out
> [ 45.599140] dwc3 31000000.usb: failed to set STALL on ep0out
> [ 45.601069] ------------[ cut here ]------------
> [ 45.601073] WARNING: CPU: 0 PID: 150 at drivers/usb/dwc3/ep0.c:289 dwc3_ep0_out_start+0xcc/0xd4
> [ 45.601102] Modules linked in: cfg80211 rfkill ipv6 rpmsg_ctrl rpmsg_char crct10dif_ce rti_wdt k3_j72xx_bandgap rtc_ti_k3 omap_mailbox sa2ul authenc [last unloaded: ti_k3_r5_remoteproc]
> [ 45.601151] CPU: 0 PID: 150 Comm: sh Not tainted 6.8.0-rc5 #1
> [ 45.601159] Hardware name: BSH - CCM-M3 (DT)
> [ 45.601164] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [ 45.601172] pc : dwc3_ep0_out_start+0xcc/0xd4
> [ 45.601179] lr : dwc3_ep0_out_start+0x50/0xd4
> [ 45.601186] sp : ffff8000832739e0
> [ 45.601189] x29: ffff8000832739e0 x28: ffff800082a21000 x27: ffff8000808dc630
> [ 45.601200] x26: 0000000000000002 x25: ffff800082530a44 x24: 0000000000000000
> [ 45.601210] x23: ffff000000e079a0 x22: ffff000000e07a68 x21: 0000000000000001
> [ 45.601219] x20: ffff000000e07880 x19: ffff000000e07880 x18: 0000000000000040
> [ 45.601229] x17: ffff7fff8e1ce000 x16: ffff800080000000 x15: fffffffffffe5260
> [ 45.601239] x14: 0000000000000000 x13: 206e6f204c4c4154 x12: 5320746573206f74
> [ 45.601249] x11: 0000000000000001 x10: 000000000000000a x9 : ffff800083273930
> [ 45.601259] x8 : 000000000000000a x7 : ffffffffffff3f0c x6 : ffffffffffff3f00
> [ 45.601268] x5 : ffffffffffff3f0c x4 : 0000000000000000 x3 : 0000000000000000
> [ 45.601278] x2 : 0000000000000000 x1 : ffff000004e7e600 x0 : 00000000ffffff92
> [ 45.601289] Call trace:
> [ 45.601293] dwc3_ep0_out_start+0xcc/0xd4
> [ 45.601301] dwc3_ep0_stall_and_restart+0x98/0xbc
> [ 45.601309] dwc3_ep0_reset_state+0x5c/0x88
> [ 45.601315] dwc3_gadget_soft_disconnect+0x144/0x160
> [ 45.601323] dwc3_gadget_suspend+0x18/0xb0
> [ 45.601329] dwc3_suspend_common+0x5c/0x18c
> [ 45.601341] dwc3_suspend+0x20/0x44
> [ 45.601350] platform_pm_suspend+0x2c/0x6c
> [ 45.601360] __device_suspend+0x10c/0x34c
> [ 45.601372] dpm_suspend+0x1a8/0x240
> [ 45.601382] dpm_suspend_start+0x80/0x9c
> [ 45.601391] suspend_devices_and_enter+0x1c4/0x584
> [ 45.601402] pm_suspend+0x1b0/0x264
> [ 45.601408] state_store+0x80/0xec
> [ 45.601415] kobj_attr_store+0x18/0x2c
> [ 45.601426] sysfs_kf_write+0x44/0x54
> [ 45.601434] kernfs_fop_write_iter+0x120/0x1ec
> [ 45.601445] vfs_write+0x23c/0x358
> [ 45.601458] ksys_write+0x70/0x104
> [ 45.601467] __arm64_sys_write+0x1c/0x28
> [ 45.601477] invoke_syscall+0x48/0x114
> [ 45.601488] el0_svc_common.constprop.0+0x40/0xe0
> [ 45.601498] do_el0_svc+0x1c/0x28
> [ 45.601506] el0_svc+0x34/0xb8
> [ 45.601516] el0t_64_sync_handler+0x100/0x12c
> [ 45.601522] el0t_64_sync+0x190/0x194
> [ 45.601531] ---[ end trace 0000000000000000 ]---
> [ 45.608794] Disabling non-boot CPUs ...
> [ 45.611029] psci: CPU1 killed (polled 0 ms)
> [ 45.611837] Enabling non-boot CPUs ...
> [ 45.612247] Detected VIPT I-cache on CPU1
>
> Tested on a am62x board with a usbnet gadget
>
> Fixes: 61a348857e86 ("usb: dwc3: gadget: Fix NULL pointer dereference in dwc3_gadget_suspend)
> Cc: [email protected]
> Signed-off-by: Michael Trimarchi <[email protected]>
> ---
> V2->V3:
> - Change the logic of the patch using the gadget connected state
> - Change of the commit message
> V1->V2:
> - Add stable in CC
> ---
> drivers/usb/dwc3/gadget.c | 9 +++++++++
> 1 file changed, 9 insertions(+)
>
> diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
> index 4c8dd6724678..a7316a1703ad 100644
> --- a/drivers/usb/dwc3/gadget.c
> +++ b/drivers/usb/dwc3/gadget.c
> @@ -2650,6 +2650,15 @@ static int dwc3_gadget_soft_disconnect(struct dwc3 *dwc)
> int ret;
>
> spin_lock_irqsave(&dwc->lock, flags);
> + /*
> + * Attempt to disconnect a no connected gadget
> + */
> + if (!dwc->connected) {
> + dev_warn(dwc->dev, "No connected device\n");
> + spin_unlock_irqrestore(&dwc->lock, flags);
> + return 0;
> + }
> +
> dwc->connected = false;
>
> /*
> --
> 2.40.1
>

There's already a fix for this, and it's already in mainline. Let me
know if this works for you:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b191a18cb5c47109ca696370a74a5062a70adfd0

Thanks,
Thinh

Subject: Re: [PATCH V3] usb: dwc3: gadget: Fix suspend/resume warning when no-gadget is connected

Hi

On Mon, Feb 26, 2024 at 11:36 PM Thinh Nguyen <[email protected]> wrote:
>
> Hi,
>
> On Mon, Feb 26, 2024, Michael Trimarchi wrote:
> > This patch avoid to disconnect an already gadget in not connected state
> >
> > [ 45.597274] dwc3 31000000.usb: wait for SETUP phase timed out
> > [ 45.599140] dwc3 31000000.usb: failed to set STALL on ep0out
> > [ 45.601069] ------------[ cut here ]------------
> > [ 45.601073] WARNING: CPU: 0 PID: 150 at drivers/usb/dwc3/ep0.c:289 dwc3_ep0_out_start+0xcc/0xd4
> > [ 45.601102] Modules linked in: cfg80211 rfkill ipv6 rpmsg_ctrl rpmsg_char crct10dif_ce rti_wdt k3_j72xx_bandgap rtc_ti_k3 omap_mailbox sa2ul authenc [last unloaded: ti_k3_r5_remoteproc]
> > [ 45.601151] CPU: 0 PID: 150 Comm: sh Not tainted 6.8.0-rc5 #1
> > [ 45.601159] Hardware name: BSH - CCM-M3 (DT)
> > [ 45.601164] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> > [ 45.601172] pc : dwc3_ep0_out_start+0xcc/0xd4
> > [ 45.601179] lr : dwc3_ep0_out_start+0x50/0xd4
> > [ 45.601186] sp : ffff8000832739e0
> > [ 45.601189] x29: ffff8000832739e0 x28: ffff800082a21000 x27: ffff8000808dc630
> > [ 45.601200] x26: 0000000000000002 x25: ffff800082530a44 x24: 0000000000000000
> > [ 45.601210] x23: ffff000000e079a0 x22: ffff000000e07a68 x21: 0000000000000001
> > [ 45.601219] x20: ffff000000e07880 x19: ffff000000e07880 x18: 0000000000000040
> > [ 45.601229] x17: ffff7fff8e1ce000 x16: ffff800080000000 x15: fffffffffffe5260
> > [ 45.601239] x14: 0000000000000000 x13: 206e6f204c4c4154 x12: 5320746573206f74
> > [ 45.601249] x11: 0000000000000001 x10: 000000000000000a x9 : ffff800083273930
> > [ 45.601259] x8 : 000000000000000a x7 : ffffffffffff3f0c x6 : ffffffffffff3f00
> > [ 45.601268] x5 : ffffffffffff3f0c x4 : 0000000000000000 x3 : 0000000000000000
> > [ 45.601278] x2 : 0000000000000000 x1 : ffff000004e7e600 x0 : 00000000ffffff92
> > [ 45.601289] Call trace:
> > [ 45.601293] dwc3_ep0_out_start+0xcc/0xd4
> > [ 45.601301] dwc3_ep0_stall_and_restart+0x98/0xbc
> > [ 45.601309] dwc3_ep0_reset_state+0x5c/0x88
> > [ 45.601315] dwc3_gadget_soft_disconnect+0x144/0x160
> > [ 45.601323] dwc3_gadget_suspend+0x18/0xb0
> > [ 45.601329] dwc3_suspend_common+0x5c/0x18c
> > [ 45.601341] dwc3_suspend+0x20/0x44
> > [ 45.601350] platform_pm_suspend+0x2c/0x6c
> > [ 45.601360] __device_suspend+0x10c/0x34c
> > [ 45.601372] dpm_suspend+0x1a8/0x240
> > [ 45.601382] dpm_suspend_start+0x80/0x9c
> > [ 45.601391] suspend_devices_and_enter+0x1c4/0x584
> > [ 45.601402] pm_suspend+0x1b0/0x264
> > [ 45.601408] state_store+0x80/0xec
> > [ 45.601415] kobj_attr_store+0x18/0x2c
> > [ 45.601426] sysfs_kf_write+0x44/0x54
> > [ 45.601434] kernfs_fop_write_iter+0x120/0x1ec
> > [ 45.601445] vfs_write+0x23c/0x358
> > [ 45.601458] ksys_write+0x70/0x104
> > [ 45.601467] __arm64_sys_write+0x1c/0x28
> > [ 45.601477] invoke_syscall+0x48/0x114
> > [ 45.601488] el0_svc_common.constprop.0+0x40/0xe0
> > [ 45.601498] do_el0_svc+0x1c/0x28
> > [ 45.601506] el0_svc+0x34/0xb8
> > [ 45.601516] el0t_64_sync_handler+0x100/0x12c
> > [ 45.601522] el0t_64_sync+0x190/0x194
> > [ 45.601531] ---[ end trace 0000000000000000 ]---
> > [ 45.608794] Disabling non-boot CPUs ...
> > [ 45.611029] psci: CPU1 killed (polled 0 ms)
> > [ 45.611837] Enabling non-boot CPUs ...
> > [ 45.612247] Detected VIPT I-cache on CPU1
> >
> > Tested on a am62x board with a usbnet gadget
> >
> > Fixes: 61a348857e86 ("usb: dwc3: gadget: Fix NULL pointer dereference in dwc3_gadget_suspend)
> > Cc: [email protected]
> > Signed-off-by: Michael Trimarchi <[email protected]>
> > ---
> > V2->V3:
> > - Change the logic of the patch using the gadget connected state
> > - Change of the commit message
> > V1->V2:
> > - Add stable in CC
> > ---
> > drivers/usb/dwc3/gadget.c | 9 +++++++++
> > 1 file changed, 9 insertions(+)
> >
> > diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
> > index 4c8dd6724678..a7316a1703ad 100644
> > --- a/drivers/usb/dwc3/gadget.c
> > +++ b/drivers/usb/dwc3/gadget.c
> > @@ -2650,6 +2650,15 @@ static int dwc3_gadget_soft_disconnect(struct dwc3 *dwc)
> > int ret;
> >
> > spin_lock_irqsave(&dwc->lock, flags);
> > + /*
> > + * Attempt to disconnect a no connected gadget
> > + */
> > + if (!dwc->connected) {
> > + dev_warn(dwc->dev, "No connected device\n");
> > + spin_unlock_irqrestore(&dwc->lock, flags);
> > + return 0;
> > + }
> > +
> > dwc->connected = false;
> >
> > /*
> > --
> > 2.40.1
> >
>
> There's already a fix for this, and it's already in mainline. Let me
> know if this works for you:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b191a18cb5c47109ca696370a74a5062a70adfd0
>

Can you explain to me the logic here? I mean pullsup_connected seems
never protected by spin lock and so I can not figure
out it easily and the commit message does not explain so much

Michael

> Thanks,
> Thinh

2024-03-07 01:30:55

by Thinh Nguyen

[permalink] [raw]
Subject: Re: [PATCH V3] usb: dwc3: gadget: Fix suspend/resume warning when no-gadget is connected

Hi,

On Mon, Feb 26, 2024, Michael Nazzareno Trimarchi wrote:
> Hi
>
> On Mon, Feb 26, 2024 at 11:36 PM Thinh Nguyen <[email protected]> wrote:
> >
> > Hi,
> >
> > On Mon, Feb 26, 2024, Michael Trimarchi wrote:
> > > This patch avoid to disconnect an already gadget in not connected state
> > >
> > > [ 45.597274] dwc3 31000000.usb: wait for SETUP phase timed out
> > > [ 45.599140] dwc3 31000000.usb: failed to set STALL on ep0out
> > > [ 45.601069] ------------[ cut here ]------------
> > > [ 45.601073] WARNING: CPU: 0 PID: 150 at drivers/usb/dwc3/ep0.c:289 dwc3_ep0_out_start+0xcc/0xd4
> > > [ 45.601102] Modules linked in: cfg80211 rfkill ipv6 rpmsg_ctrl rpmsg_char crct10dif_ce rti_wdt k3_j72xx_bandgap rtc_ti_k3 omap_mailbox sa2ul authenc [last unloaded: ti_k3_r5_remoteproc]
> > > [ 45.601151] CPU: 0 PID: 150 Comm: sh Not tainted 6.8.0-rc5 #1
> > > [ 45.601159] Hardware name: BSH - CCM-M3 (DT)
> > > [ 45.601164] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> > > [ 45.601172] pc : dwc3_ep0_out_start+0xcc/0xd4
> > > [ 45.601179] lr : dwc3_ep0_out_start+0x50/0xd4
> > > [ 45.601186] sp : ffff8000832739e0
> > > [ 45.601189] x29: ffff8000832739e0 x28: ffff800082a21000 x27: ffff8000808dc630
> > > [ 45.601200] x26: 0000000000000002 x25: ffff800082530a44 x24: 0000000000000000
> > > [ 45.601210] x23: ffff000000e079a0 x22: ffff000000e07a68 x21: 0000000000000001
> > > [ 45.601219] x20: ffff000000e07880 x19: ffff000000e07880 x18: 0000000000000040
> > > [ 45.601229] x17: ffff7fff8e1ce000 x16: ffff800080000000 x15: fffffffffffe5260
> > > [ 45.601239] x14: 0000000000000000 x13: 206e6f204c4c4154 x12: 5320746573206f74
> > > [ 45.601249] x11: 0000000000000001 x10: 000000000000000a x9 : ffff800083273930
> > > [ 45.601259] x8 : 000000000000000a x7 : ffffffffffff3f0c x6 : ffffffffffff3f00
> > > [ 45.601268] x5 : ffffffffffff3f0c x4 : 0000000000000000 x3 : 0000000000000000
> > > [ 45.601278] x2 : 0000000000000000 x1 : ffff000004e7e600 x0 : 00000000ffffff92
> > > [ 45.601289] Call trace:
> > > [ 45.601293] dwc3_ep0_out_start+0xcc/0xd4
> > > [ 45.601301] dwc3_ep0_stall_and_restart+0x98/0xbc
> > > [ 45.601309] dwc3_ep0_reset_state+0x5c/0x88
> > > [ 45.601315] dwc3_gadget_soft_disconnect+0x144/0x160
> > > [ 45.601323] dwc3_gadget_suspend+0x18/0xb0
> > > [ 45.601329] dwc3_suspend_common+0x5c/0x18c
> > > [ 45.601341] dwc3_suspend+0x20/0x44
> > > [ 45.601350] platform_pm_suspend+0x2c/0x6c
> > > [ 45.601360] __device_suspend+0x10c/0x34c
> > > [ 45.601372] dpm_suspend+0x1a8/0x240
> > > [ 45.601382] dpm_suspend_start+0x80/0x9c
> > > [ 45.601391] suspend_devices_and_enter+0x1c4/0x584
> > > [ 45.601402] pm_suspend+0x1b0/0x264
> > > [ 45.601408] state_store+0x80/0xec
> > > [ 45.601415] kobj_attr_store+0x18/0x2c
> > > [ 45.601426] sysfs_kf_write+0x44/0x54
> > > [ 45.601434] kernfs_fop_write_iter+0x120/0x1ec
> > > [ 45.601445] vfs_write+0x23c/0x358
> > > [ 45.601458] ksys_write+0x70/0x104
> > > [ 45.601467] __arm64_sys_write+0x1c/0x28
> > > [ 45.601477] invoke_syscall+0x48/0x114
> > > [ 45.601488] el0_svc_common.constprop.0+0x40/0xe0
> > > [ 45.601498] do_el0_svc+0x1c/0x28
> > > [ 45.601506] el0_svc+0x34/0xb8
> > > [ 45.601516] el0t_64_sync_handler+0x100/0x12c
> > > [ 45.601522] el0t_64_sync+0x190/0x194
> > > [ 45.601531] ---[ end trace 0000000000000000 ]---
> > > [ 45.608794] Disabling non-boot CPUs ...
> > > [ 45.611029] psci: CPU1 killed (polled 0 ms)
> > > [ 45.611837] Enabling non-boot CPUs ...
> > > [ 45.612247] Detected VIPT I-cache on CPU1
> > >
> > > Tested on a am62x board with a usbnet gadget
> > >
> > > Fixes: 61a348857e86 ("usb: dwc3: gadget: Fix NULL pointer dereference in dwc3_gadget_suspend)
> > > Cc: [email protected]
> > > Signed-off-by: Michael Trimarchi <[email protected]>
> > > ---
> > > V2->V3:
> > > - Change the logic of the patch using the gadget connected state
> > > - Change of the commit message
> > > V1->V2:
> > > - Add stable in CC
> > > ---
> > > drivers/usb/dwc3/gadget.c | 9 +++++++++
> > > 1 file changed, 9 insertions(+)
> > >
> > > diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
> > > index 4c8dd6724678..a7316a1703ad 100644
> > > --- a/drivers/usb/dwc3/gadget.c
> > > +++ b/drivers/usb/dwc3/gadget.c
> > > @@ -2650,6 +2650,15 @@ static int dwc3_gadget_soft_disconnect(struct dwc3 *dwc)
> > > int ret;
> > >
> > > spin_lock_irqsave(&dwc->lock, flags);
> > > + /*
> > > + * Attempt to disconnect a no connected gadget
> > > + */
> > > + if (!dwc->connected) {
> > > + dev_warn(dwc->dev, "No connected device\n");
> > > + spin_unlock_irqrestore(&dwc->lock, flags);
> > > + return 0;
> > > + }
> > > +
> > > dwc->connected = false;
> > >
> > > /*
> > > --
> > > 2.40.1
> > >
> >
> > There's already a fix for this, and it's already in mainline. Let me
> > know if this works for you:
> >
> > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b191a18cb5c47109ca696370a74a5062a70adfd0__;!!A4F2R9G_pg!dqF-dGSHGR41Bep_7ZG2mWfPNYWfC4T-_FpOf_pFIvqa9L1n1e6l6D3oP9bOlRRFobW4Uvh0VAP8qBkLMfNkUlfaeTc$
> >
>
> Can you explain to me the logic here? I mean pullsup_connected seems
> never protected by spin lock and so I can not figure
> out it easily and the commit message does not explain so much
>

Sorry for the delay response.

Ah.. you're right, the spin_lock isn't useful there. My intention was to
avoid soft disconnect flow to occur during system suspend along with
pullup(off). I somehow forgot that it isn't protected with spin_lock.

It's a very unlikely scenario. Even if it did happen, it should be
harmless. But to solve that perhaps we can:
1) Keep the code simple and enforce no sleep and hold spin_lock during
the entirety of run_stop() sequence
2) Enforce a must-lock check to run_stop() and unlock during sleep. This
requires more code changes and reviews.

If you have some ideas, please share. (Note that using dwc->connected
isn't sufficient)

Thanks,
Thinh

Subject: Re: [PATCH V3] usb: dwc3: gadget: Fix suspend/resume warning when no-gadget is connected

Hi

On Thu, Mar 7, 2024 at 2:30 AM Thinh Nguyen <[email protected]> wrote:
>
> Hi,
>
> On Mon, Feb 26, 2024, Michael Nazzareno Trimarchi wrote:
> > Hi
> >
> > On Mon, Feb 26, 2024 at 11:36 PM Thinh Nguyen <[email protected]> wrote:
> > >
> > > Hi,
> > >
> > > On Mon, Feb 26, 2024, Michael Trimarchi wrote:
> > > > This patch avoid to disconnect an already gadget in not connected state
> > > >
> > > > [ 45.597274] dwc3 31000000.usb: wait for SETUP phase timed out
> > > > [ 45.599140] dwc3 31000000.usb: failed to set STALL on ep0out
> > > > [ 45.601069] ------------[ cut here ]------------
> > > > [ 45.601073] WARNING: CPU: 0 PID: 150 at drivers/usb/dwc3/ep0.c:289 dwc3_ep0_out_start+0xcc/0xd4
> > > > [ 45.601102] Modules linked in: cfg80211 rfkill ipv6 rpmsg_ctrl rpmsg_char crct10dif_ce rti_wdt k3_j72xx_bandgap rtc_ti_k3 omap_mailbox sa2ul authenc [last unloaded: ti_k3_r5_remoteproc]
> > > > [ 45.601151] CPU: 0 PID: 150 Comm: sh Not tainted 6.8.0-rc5 #1
> > > > [ 45.601159] Hardware name: BSH - CCM-M3 (DT)
> > > > [ 45.601164] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> > > > [ 45.601172] pc : dwc3_ep0_out_start+0xcc/0xd4
> > > > [ 45.601179] lr : dwc3_ep0_out_start+0x50/0xd4
> > > > [ 45.601186] sp : ffff8000832739e0
> > > > [ 45.601189] x29: ffff8000832739e0 x28: ffff800082a21000 x27: ffff8000808dc630
> > > > [ 45.601200] x26: 0000000000000002 x25: ffff800082530a44 x24: 0000000000000000
> > > > [ 45.601210] x23: ffff000000e079a0 x22: ffff000000e07a68 x21: 0000000000000001
> > > > [ 45.601219] x20: ffff000000e07880 x19: ffff000000e07880 x18: 0000000000000040
> > > > [ 45.601229] x17: ffff7fff8e1ce000 x16: ffff800080000000 x15: fffffffffffe5260
> > > > [ 45.601239] x14: 0000000000000000 x13: 206e6f204c4c4154 x12: 5320746573206f74
> > > > [ 45.601249] x11: 0000000000000001 x10: 000000000000000a x9 : ffff800083273930
> > > > [ 45.601259] x8 : 000000000000000a x7 : ffffffffffff3f0c x6 : ffffffffffff3f00
> > > > [ 45.601268] x5 : ffffffffffff3f0c x4 : 0000000000000000 x3 : 0000000000000000
> > > > [ 45.601278] x2 : 0000000000000000 x1 : ffff000004e7e600 x0 : 00000000ffffff92
> > > > [ 45.601289] Call trace:
> > > > [ 45.601293] dwc3_ep0_out_start+0xcc/0xd4
> > > > [ 45.601301] dwc3_ep0_stall_and_restart+0x98/0xbc
> > > > [ 45.601309] dwc3_ep0_reset_state+0x5c/0x88
> > > > [ 45.601315] dwc3_gadget_soft_disconnect+0x144/0x160
> > > > [ 45.601323] dwc3_gadget_suspend+0x18/0xb0
> > > > [ 45.601329] dwc3_suspend_common+0x5c/0x18c
> > > > [ 45.601341] dwc3_suspend+0x20/0x44
> > > > [ 45.601350] platform_pm_suspend+0x2c/0x6c
> > > > [ 45.601360] __device_suspend+0x10c/0x34c
> > > > [ 45.601372] dpm_suspend+0x1a8/0x240
> > > > [ 45.601382] dpm_suspend_start+0x80/0x9c
> > > > [ 45.601391] suspend_devices_and_enter+0x1c4/0x584
> > > > [ 45.601402] pm_suspend+0x1b0/0x264
> > > > [ 45.601408] state_store+0x80/0xec
> > > > [ 45.601415] kobj_attr_store+0x18/0x2c
> > > > [ 45.601426] sysfs_kf_write+0x44/0x54
> > > > [ 45.601434] kernfs_fop_write_iter+0x120/0x1ec
> > > > [ 45.601445] vfs_write+0x23c/0x358
> > > > [ 45.601458] ksys_write+0x70/0x104
> > > > [ 45.601467] __arm64_sys_write+0x1c/0x28
> > > > [ 45.601477] invoke_syscall+0x48/0x114
> > > > [ 45.601488] el0_svc_common.constprop.0+0x40/0xe0
> > > > [ 45.601498] do_el0_svc+0x1c/0x28
> > > > [ 45.601506] el0_svc+0x34/0xb8
> > > > [ 45.601516] el0t_64_sync_handler+0x100/0x12c
> > > > [ 45.601522] el0t_64_sync+0x190/0x194
> > > > [ 45.601531] ---[ end trace 0000000000000000 ]---
> > > > [ 45.608794] Disabling non-boot CPUs ...
> > > > [ 45.611029] psci: CPU1 killed (polled 0 ms)
> > > > [ 45.611837] Enabling non-boot CPUs ...
> > > > [ 45.612247] Detected VIPT I-cache on CPU1
> > > >
> > > > Tested on a am62x board with a usbnet gadget
> > > >
> > > > Fixes: 61a348857e86 ("usb: dwc3: gadget: Fix NULL pointer dereference in dwc3_gadget_suspend)
> > > > Cc: [email protected]
> > > > Signed-off-by: Michael Trimarchi <[email protected]>
> > > > ---
> > > > V2->V3:
> > > > - Change the logic of the patch using the gadget connected state
> > > > - Change of the commit message
> > > > V1->V2:
> > > > - Add stable in CC
> > > > ---
> > > > drivers/usb/dwc3/gadget.c | 9 +++++++++
> > > > 1 file changed, 9 insertions(+)
> > > >
> > > > diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
> > > > index 4c8dd6724678..a7316a1703ad 100644
> > > > --- a/drivers/usb/dwc3/gadget.c
> > > > +++ b/drivers/usb/dwc3/gadget.c
> > > > @@ -2650,6 +2650,15 @@ static int dwc3_gadget_soft_disconnect(struct dwc3 *dwc)
> > > > int ret;
> > > >
> > > > spin_lock_irqsave(&dwc->lock, flags);
> > > > + /*
> > > > + * Attempt to disconnect a no connected gadget
> > > > + */
> > > > + if (!dwc->connected) {
> > > > + dev_warn(dwc->dev, "No connected device\n");
> > > > + spin_unlock_irqrestore(&dwc->lock, flags);
> > > > + return 0;
> > > > + }
> > > > +
> > > > dwc->connected = false;
> > > >
> > > > /*
> > > > --
> > > > 2.40.1
> > > >
> > >
> > > There's already a fix for this, and it's already in mainline. Let me
> > > know if this works for you:
> > >
> > > https://urldefense.com/v3/__https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b191a18cb5c47109ca696370a74a5062a70adfd0__;!!A4F2R9G_pg!dqF-dGSHGR41Bep_7ZG2mWfPNYWfC4T-_FpOf_pFIvqa9L1n1e6l6D3oP9bOlRRFobW4Uvh0VAP8qBkLMfNkUlfaeTc$
> > >
> >
> > Can you explain to me the logic here? I mean pullsup_connected seems
> > never protected by spin lock and so I can not figure
> > out it easily and the commit message does not explain so much
> >
>
> Sorry for the delay response.
>
> Ah.. you're right, the spin_lock isn't useful there. My intention was to
> avoid soft disconnect flow to occur during system suspend along with
> pullup(off). I somehow forgot that it isn't protected with spin_lock.
>

You reduce the window of failure but we need to rework anyway

> It's a very unlikely scenario. Even if it did happen, it should be
> harmless. But to solve that perhaps we can:
> 1) Keep the code simple and enforce no sleep and hold spin_lock during
> the entirety of run_stop() sequence


If the run_stop is executing I can imagine that you only need to mark it as
ongoing and protect with a spin_lock, returning -EINVAL on the suspend
hook. I will keep
it simple

Michael

> 2) Enforce a must-lock check to run_stop() and unlock during sleep. This
> requires more code changes and reviews.
>
> If you have some ideas, please share. (Note that using dwc->connected
> isn't sufficient)
>
> Thanks,
> Thinh



--
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
[email protected]
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
[email protected]
http://www.amarulasolutions.com