For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
("macvtap: Add support of packet capture on macvtap
device."). Multiqueue macvtap suffers from single qdisc lock
contention. This is because macvtap claims a non zero tx_queue_len and
it reuses this value as it socket receive queue size.Thanks to
IFF_NO_QUEUE, we can remove the lock contention without breaking
existing socket receive queue length logic.
Cc: Patrick McHardy <[email protected]>
Cc: Vladislav Yasevich <[email protected]>
Cc: Michael S. Tsirkin <[email protected]>
Signed-off-by: Jason Wang <[email protected]>
---
drivers/net/macvlan.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
index 47da435..09d8718 100644
--- a/drivers/net/macvlan.c
+++ b/drivers/net/macvlan.c
@@ -1056,7 +1056,7 @@ void macvlan_common_setup(struct net_device *dev)
dev->priv_flags &= ~IFF_TX_SKB_SHARING;
netif_keep_dst(dev);
- dev->priv_flags |= IFF_UNICAST_FLT;
+ dev->priv_flags |= IFF_UNICAST_FLT | IFF_NO_QUEUE;
dev->netdev_ops = &macvlan_netdev_ops;
dev->destructor = free_netdev;
dev->header_ops = &macvlan_hard_header_ops;
@@ -1067,7 +1067,6 @@ EXPORT_SYMBOL_GPL(macvlan_common_setup);
static void macvlan_setup(struct net_device *dev)
{
macvlan_common_setup(dev);
- dev->tx_queue_len = 0;
}
static int macvlan_port_create(struct net_device *dev)
--
2.1.4
On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
> For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
>
> For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
> ("macvtap: Add support of packet capture on macvtap
> device."). Multiqueue macvtap suffers from single qdisc lock
> contention. This is because macvtap claims a non zero tx_queue_len and
> it reuses this value as it socket receive queue size.Thanks to
> IFF_NO_QUEUE, we can remove the lock contention without breaking
> existing socket receive queue length logic.
>
> Cc: Patrick McHardy <[email protected]>
> Cc: Vladislav Yasevich <[email protected]>
> Cc: Michael S. Tsirkin <[email protected]>
> Signed-off-by: Jason Wang <[email protected]>
Seems to make sense. Give me a day or two to get over the jet lag
(and get out from under the pile of mail accumulated while I was traveling),
I'll review properly and ack.
> ---
> drivers/net/macvlan.c | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
> index 47da435..09d8718 100644
> --- a/drivers/net/macvlan.c
> +++ b/drivers/net/macvlan.c
> @@ -1056,7 +1056,7 @@ void macvlan_common_setup(struct net_device *dev)
>
> dev->priv_flags &= ~IFF_TX_SKB_SHARING;
> netif_keep_dst(dev);
> - dev->priv_flags |= IFF_UNICAST_FLT;
> + dev->priv_flags |= IFF_UNICAST_FLT | IFF_NO_QUEUE;
> dev->netdev_ops = &macvlan_netdev_ops;
> dev->destructor = free_netdev;
> dev->header_ops = &macvlan_hard_header_ops;
> @@ -1067,7 +1067,6 @@ EXPORT_SYMBOL_GPL(macvlan_common_setup);
> static void macvlan_setup(struct net_device *dev)
> {
> macvlan_common_setup(dev);
> - dev->tx_queue_len = 0;
> }
>
> static int macvlan_port_create(struct net_device *dev)
> --
> 2.1.4
On 08/25/2015 06:17 PM, Michael S. Tsirkin wrote:
> On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
>> > For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
>> >
>> > For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>> > ("macvtap: Add support of packet capture on macvtap
>> > device."). Multiqueue macvtap suffers from single qdisc lock
>> > contention. This is because macvtap claims a non zero tx_queue_len and
>> > it reuses this value as it socket receive queue size.Thanks to
>> > IFF_NO_QUEUE, we can remove the lock contention without breaking
>> > existing socket receive queue length logic.
>> >
>> > Cc: Patrick McHardy <[email protected]>
>> > Cc: Vladislav Yasevich <[email protected]>
>> > Cc: Michael S. Tsirkin <[email protected]>
>> > Signed-off-by: Jason Wang <[email protected]>
> Seems to make sense. Give me a day or two to get over the jet lag
> (and get out from under the pile of mail accumulated while I was traveling),
> I'll review properly and ack.
>
A note on this patch: only default qdisc were removed but we don't lose
the ability to attach a qdisc to macvtap (though it may cause lock
contention on multiqueue case).
On 08/25/2015 07:30 AM, Jason Wang wrote:
>
>
> On 08/25/2015 06:17 PM, Michael S. Tsirkin wrote:
>> On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
>>>> For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
>>>>
>>>> For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>> ("macvtap: Add support of packet capture on macvtap
>>>> device."). Multiqueue macvtap suffers from single qdisc lock
>>>> contention. This is because macvtap claims a non zero tx_queue_len and
>>>> it reuses this value as it socket receive queue size.Thanks to
>>>> IFF_NO_QUEUE, we can remove the lock contention without breaking
>>>> existing socket receive queue length logic.
>>>>
>>>> Cc: Patrick McHardy <[email protected]>
>>>> Cc: Vladislav Yasevich <[email protected]>
>>>> Cc: Michael S. Tsirkin <[email protected]>
>>>> Signed-off-by: Jason Wang <[email protected]>
>> Seems to make sense. Give me a day or two to get over the jet lag
>> (and get out from under the pile of mail accumulated while I was traveling),
>> I'll review properly and ack.
>>
>
> A note on this patch: only default qdisc were removed but we don't lose
> the ability to attach a qdisc to macvtap (though it may cause lock
> contention on multiqueue case).
>
Wouldn't that lock contention be solved if we really had multiple queues
for multi-queue macvtaps?
-vlad
On 08/26/2015 12:32 AM, Vlad Yasevich wrote:
> On 08/25/2015 07:30 AM, Jason Wang wrote:
>>
>> On 08/25/2015 06:17 PM, Michael S. Tsirkin wrote:
>>> On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
>>>>> For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
>>>>>
>>>>> For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>>> ("macvtap: Add support of packet capture on macvtap
>>>>> device."). Multiqueue macvtap suffers from single qdisc lock
>>>>> contention. This is because macvtap claims a non zero tx_queue_len and
>>>>> it reuses this value as it socket receive queue size.Thanks to
>>>>> IFF_NO_QUEUE, we can remove the lock contention without breaking
>>>>> existing socket receive queue length logic.
>>>>>
>>>>> Cc: Patrick McHardy <[email protected]>
>>>>> Cc: Vladislav Yasevich <[email protected]>
>>>>> Cc: Michael S. Tsirkin <[email protected]>
>>>>> Signed-off-by: Jason Wang <[email protected]>
>>> Seems to make sense. Give me a day or two to get over the jet lag
>>> (and get out from under the pile of mail accumulated while I was traveling),
>>> I'll review properly and ack.
>>>
>> A note on this patch: only default qdisc were removed but we don't lose
>> the ability to attach a qdisc to macvtap (though it may cause lock
>> contention on multiqueue case).
>>
> Wouldn't that lock contention be solved if we really had multiple queues
> for multi-queue macvtaps?
>
> -vlad
Yes, but this introduce another layer of txq locks contention? And it
also needs macvlan multiqueue support. We used to do something like this
but switch to NETIF_F_LLTX finally. You may refer:
2c11455321f37da6fe6cc36353149f9ac9183334 macvlan: add multiqueue capability
8ffab51b3dfc54876f145f15b351c41f3f703195 macvlan: lockless tx path
On Wed, Aug 26, 2015 at 01:45:30PM +0800, Jason Wang wrote:
>
>
> On 08/26/2015 12:32 AM, Vlad Yasevich wrote:
> > On 08/25/2015 07:30 AM, Jason Wang wrote:
> >>
> >> On 08/25/2015 06:17 PM, Michael S. Tsirkin wrote:
> >>> On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
> >>>>> For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
> >>>>>
> >>>>> For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
> >>>>> ("macvtap: Add support of packet capture on macvtap
> >>>>> device."). Multiqueue macvtap suffers from single qdisc lock
> >>>>> contention. This is because macvtap claims a non zero tx_queue_len and
> >>>>> it reuses this value as it socket receive queue size.Thanks to
> >>>>> IFF_NO_QUEUE, we can remove the lock contention without breaking
> >>>>> existing socket receive queue length logic.
> >>>>>
> >>>>> Cc: Patrick McHardy <[email protected]>
> >>>>> Cc: Vladislav Yasevich <[email protected]>
> >>>>> Cc: Michael S. Tsirkin <[email protected]>
> >>>>> Signed-off-by: Jason Wang <[email protected]>
> >>> Seems to make sense. Give me a day or two to get over the jet lag
> >>> (and get out from under the pile of mail accumulated while I was traveling),
> >>> I'll review properly and ack.
> >>>
> >> A note on this patch: only default qdisc were removed but we don't lose
> >> the ability to attach a qdisc to macvtap (though it may cause lock
> >> contention on multiqueue case).
> >>
> > Wouldn't that lock contention be solved if we really had multiple queues
> > for multi-queue macvtaps?
> >
> > -vlad
>
> Yes, but this introduce another layer of txq locks contention?
I don't follow - why does it? Could you clarify please?
> And it
> also needs macvlan multiqueue support. We used to do something like this
> but switch to NETIF_F_LLTX finally. You may refer:
>
> 2c11455321f37da6fe6cc36353149f9ac9183334 macvlan: add multiqueue capability
> 8ffab51b3dfc54876f145f15b351c41f3f703195 macvlan: lockless tx path
My concern is that the moment someone configures a non-standard qdisc
scalability suddenly disappears. That would also be tricky to debug in the
field as not a lot of developers use non-standard qdiscs.
What do you think?
--
MST
On 08/27/2015 06:43 PM, Michael S. Tsirkin wrote:
> On Wed, Aug 26, 2015 at 01:45:30PM +0800, Jason Wang wrote:
>>
>> On 08/26/2015 12:32 AM, Vlad Yasevich wrote:
>>> On 08/25/2015 07:30 AM, Jason Wang wrote:
>>>> On 08/25/2015 06:17 PM, Michael S. Tsirkin wrote:
>>>>> On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
>>>>>>> For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
>>>>>>>
>>>>>>> For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>>>>> ("macvtap: Add support of packet capture on macvtap
>>>>>>> device."). Multiqueue macvtap suffers from single qdisc lock
>>>>>>> contention. This is because macvtap claims a non zero tx_queue_len and
>>>>>>> it reuses this value as it socket receive queue size.Thanks to
>>>>>>> IFF_NO_QUEUE, we can remove the lock contention without breaking
>>>>>>> existing socket receive queue length logic.
>>>>>>>
>>>>>>> Cc: Patrick McHardy <[email protected]>
>>>>>>> Cc: Vladislav Yasevich <[email protected]>
>>>>>>> Cc: Michael S. Tsirkin <[email protected]>
>>>>>>> Signed-off-by: Jason Wang <[email protected]>
>>>>> Seems to make sense. Give me a day or two to get over the jet lag
>>>>> (and get out from under the pile of mail accumulated while I was traveling),
>>>>> I'll review properly and ack.
>>>>>
>>>> A note on this patch: only default qdisc were removed but we don't lose
>>>> the ability to attach a qdisc to macvtap (though it may cause lock
>>>> contention on multiqueue case).
>>>>
>>> Wouldn't that lock contention be solved if we really had multiple queues
>>> for multi-queue macvtaps?
>>>
>>> -vlad
>> Yes, but this introduce another layer of txq locks contention?
> I don't follow - why does it? Could you clarify please?
I believe Vlad wants to remove NETIF_F_LLTX. If yes, core will do an
extra tx lock at macvlan layer.
>
>> And it
>> also needs macvlan multiqueue support. We used to do something like this
>> but switch to NETIF_F_LLTX finally. You may refer:
>>
>> 2c11455321f37da6fe6cc36353149f9ac9183334 macvlan: add multiqueue capability
>> 8ffab51b3dfc54876f145f15b351c41f3f703195 macvlan: lockless tx path
> My concern is that the moment someone configures a non-standard qdisc
> scalability suddenly disappears. That would also be tricky to debug in the
> field as not a lot of developers use non-standard qdiscs.
> What do you think?
Probably not an issue. Non-standard qdisc may need be attached manually
after device creation, and we don't lose this ability with this patch.
(Unless somebody changes default_qdisc). Actually, user before
6acf54f1cf0a6747bac9fea26f34cfc5a9029523 does not expect any qdisc work
for macvtap like other stacked devices. This patch also restore this.
On 08/27/2015 10:42 PM, Jason Wang wrote:
>
>
> On 08/27/2015 06:43 PM, Michael S. Tsirkin wrote:
>> On Wed, Aug 26, 2015 at 01:45:30PM +0800, Jason Wang wrote:
>>>
>>> On 08/26/2015 12:32 AM, Vlad Yasevich wrote:
>>>> On 08/25/2015 07:30 AM, Jason Wang wrote:
>>>>> On 08/25/2015 06:17 PM, Michael S. Tsirkin wrote:
>>>>>> On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
>>>>>>>> For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
>>>>>>>>
>>>>>>>> For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>>>>>> ("macvtap: Add support of packet capture on macvtap
>>>>>>>> device."). Multiqueue macvtap suffers from single qdisc lock
>>>>>>>> contention. This is because macvtap claims a non zero tx_queue_len and
>>>>>>>> it reuses this value as it socket receive queue size.Thanks to
>>>>>>>> IFF_NO_QUEUE, we can remove the lock contention without breaking
>>>>>>>> existing socket receive queue length logic.
>>>>>>>>
>>>>>>>> Cc: Patrick McHardy <[email protected]>
>>>>>>>> Cc: Vladislav Yasevich <[email protected]>
>>>>>>>> Cc: Michael S. Tsirkin <[email protected]>
>>>>>>>> Signed-off-by: Jason Wang <[email protected]>
>>>>>> Seems to make sense. Give me a day or two to get over the jet lag
>>>>>> (and get out from under the pile of mail accumulated while I was traveling),
>>>>>> I'll review properly and ack.
>>>>>>
>>>>> A note on this patch: only default qdisc were removed but we don't lose
>>>>> the ability to attach a qdisc to macvtap (though it may cause lock
>>>>> contention on multiqueue case).
>>>>>
>>>> Wouldn't that lock contention be solved if we really had multiple queues
>>>> for multi-queue macvtaps?
>>>>
>>>> -vlad
>>> Yes, but this introduce another layer of txq locks contention?
>> I don't follow - why does it? Could you clarify please?
>
> I believe Vlad wants to remove NETIF_F_LLTX. If yes, core will do an
> extra tx lock at macvlan layer.
No, I don't want to remove it. In a sense, it would function similar to
how it works when fwd_priv is populated. I am still testing the code
as it's showing some strange artifacts... could be due to keeping LLTX.
-vlad
>
>>
>>> And it
>>> also needs macvlan multiqueue support. We used to do something like this
>>> but switch to NETIF_F_LLTX finally. You may refer:
>>>
>>> 2c11455321f37da6fe6cc36353149f9ac9183334 macvlan: add multiqueue capability
>>> 8ffab51b3dfc54876f145f15b351c41f3f703195 macvlan: lockless tx path
>> My concern is that the moment someone configures a non-standard qdisc
>> scalability suddenly disappears. That would also be tricky to debug in the
>> field as not a lot of developers use non-standard qdiscs.
>> What do you think?
>
> Probably not an issue. Non-standard qdisc may need be attached manually
> after device creation, and we don't lose this ability with this patch.
> (Unless somebody changes default_qdisc). Actually, user before
> 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 does not expect any qdisc work
> for macvtap like other stacked devices. This patch also restore this.
>
>
On 08/28/2015 08:25 PM, Vlad Yasevich wrote:
> On 08/27/2015 10:42 PM, Jason Wang wrote:
>> >
>> >
>> > On 08/27/2015 06:43 PM, Michael S. Tsirkin wrote:
>>> >> On Wed, Aug 26, 2015 at 01:45:30PM +0800, Jason Wang wrote:
>>>> >>>
>>>> >>> On 08/26/2015 12:32 AM, Vlad Yasevich wrote:
>>>>> >>>> On 08/25/2015 07:30 AM, Jason Wang wrote:
>>>>>> >>>>> On 08/25/2015 06:17 PM, Michael S. Tsirkin wrote:
>>>>>>> >>>>>> On Mon, Aug 24, 2015 at 04:33:12PM +0800, Jason Wang wrote:
>>>>>>>>> >>>>>>>> For macvlan, switch to use IFF_NO_QUEUE instead of tx_queue_len = 0.
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> For macvtap, after commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
>>>>>>>>> >>>>>>>> ("macvtap: Add support of packet capture on macvtap
>>>>>>>>> >>>>>>>> device."). Multiqueue macvtap suffers from single qdisc lock
>>>>>>>>> >>>>>>>> contention. This is because macvtap claims a non zero tx_queue_len and
>>>>>>>>> >>>>>>>> it reuses this value as it socket receive queue size.Thanks to
>>>>>>>>> >>>>>>>> IFF_NO_QUEUE, we can remove the lock contention without breaking
>>>>>>>>> >>>>>>>> existing socket receive queue length logic.
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> Cc: Patrick McHardy <[email protected]>
>>>>>>>>> >>>>>>>> Cc: Vladislav Yasevich <[email protected]>
>>>>>>>>> >>>>>>>> Cc: Michael S. Tsirkin <[email protected]>
>>>>>>>>> >>>>>>>> Signed-off-by: Jason Wang <[email protected]>
>>>>>>> >>>>>> Seems to make sense. Give me a day or two to get over the jet lag
>>>>>>> >>>>>> (and get out from under the pile of mail accumulated while I was traveling),
>>>>>>> >>>>>> I'll review properly and ack.
>>>>>>> >>>>>>
>>>>>> >>>>> A note on this patch: only default qdisc were removed but we don't lose
>>>>>> >>>>> the ability to attach a qdisc to macvtap (though it may cause lock
>>>>>> >>>>> contention on multiqueue case).
>>>>>> >>>>>
>>>>> >>>> Wouldn't that lock contention be solved if we really had multiple queues
>>>>> >>>> for multi-queue macvtaps?
>>>>> >>>>
>>>>> >>>> -vlad
>>>> >>> Yes, but this introduce another layer of txq locks contention?
>>> >> I don't follow - why does it? Could you clarify please?
>> >
>> > I believe Vlad wants to remove NETIF_F_LLTX. If yes, core will do an
>> > extra tx lock at macvlan layer.
> No, I don't want to remove it. In a sense, it would function similar to
> how it works when fwd_priv is populated. I am still testing the code
> as it's showing some strange artifacts... could be due to keeping LLTX.
>
> -vlad
>
I see. I'm ok to wait for your code. But if a patch of just two lines
works, probably no need to try complex method.
Thanks