When processing upcall commands, two groups of data are available to
userspace for processing: the actual packet data and the kernel
sw flow key data. The inclusion of the flow key allows the userspace
avoid running through the dissection again.
However, the userspace can choose to ignore the flow key data, as is
the case in some ovs-vswitchd upcall processing. For these messages,
having the flow key data merely adds additional data to the upcall
pipeline without any actual gain. Userspace simply throws the data
away anyway.
Introduce a new feature OVS_DP_F_EXCLUDE_UPCALL_FLOW_KEY which signals
that the userspace doesn't want upcalls included with specific class
of message (for example MISS messages). The associated attribute
OVS_DP_ATTR_EXCLUDE_CMDS tells which specific commands to omit via a
bitmask.
A test will be added to showcase using the feature.
Signed-off-by: Aaron Conole <[email protected]>
---
include/uapi/linux/openvswitch.h | 6 ++++++
net/openvswitch/datapath.c | 26 ++++++++++++++++++++++----
net/openvswitch/datapath.h | 2 ++
3 files changed, 30 insertions(+), 4 deletions(-)
diff --git a/include/uapi/linux/openvswitch.h b/include/uapi/linux/openvswitch.h
index 94066f87e9ee..238e62ecba46 100644
--- a/include/uapi/linux/openvswitch.h
+++ b/include/uapi/linux/openvswitch.h
@@ -95,6 +95,9 @@ enum ovs_datapath_attr {
* per-cpu dispatch mode
*/
OVS_DP_ATTR_IFINDEX,
+ OVS_DP_ATTR_EXCLUDE_CMDS, /* u32 mask of OVS_PACKET_CMDs for
+ * omitting FLOW_KEY attribute
+ */
__OVS_DP_ATTR_MAX
};
@@ -138,6 +141,9 @@ struct ovs_vport_stats {
/* Allow per-cpu dispatch of upcalls */
#define OVS_DP_F_DISPATCH_UPCALL_PER_CPU (1 << 3)
+/* Drop Flow key data from upcall packet cmds */
+#define OVS_DP_F_EXCLUDE_UPCALL_FLOW_KEY (1 << 4)
+
/* Fixed logical ports. */
#define OVSP_LOCAL ((__u32)0)
diff --git a/net/openvswitch/datapath.c b/net/openvswitch/datapath.c
index 861dfb8daf4a..6afde7de492c 100644
--- a/net/openvswitch/datapath.c
+++ b/net/openvswitch/datapath.c
@@ -470,9 +470,13 @@ static int queue_userspace_packet(struct datapath *dp, struct sk_buff *skb,
}
upcall->dp_ifindex = dp_ifindex;
- err = ovs_nla_put_key(key, key, OVS_PACKET_ATTR_KEY, false, user_skb);
- if (err)
- goto out;
+ if (!(dp->user_features & OVS_DP_F_EXCLUDE_UPCALL_FLOW_KEY) ||
+ !(dp->upcall_exclude_cmds & (1U << upcall_info->cmd))) {
+ err = ovs_nla_put_key(key, key, OVS_PACKET_ATTR_KEY, false,
+ user_skb);
+ if (err)
+ goto out;
+ }
if (upcall_info->userdata)
__nla_put(user_skb, OVS_PACKET_ATTR_USERDATA,
@@ -1526,6 +1530,7 @@ static size_t ovs_dp_cmd_msg_size(void)
msgsize += nla_total_size(sizeof(u32)); /* OVS_DP_ATTR_USER_FEATURES */
msgsize += nla_total_size(sizeof(u32)); /* OVS_DP_ATTR_MASKS_CACHE_SIZE */
msgsize += nla_total_size(sizeof(u32) * nr_cpu_ids); /* OVS_DP_ATTR_PER_CPU_PIDS */
+ msgsize += nla_total_size(sizeof(u32)); /* OVS_DP_ATTR_EXCLUDE_CMDS */
return msgsize;
}
@@ -1574,6 +1579,10 @@ static int ovs_dp_cmd_fill_info(struct datapath *dp, struct sk_buff *skb,
goto nla_put_failure;
}
+ if (nla_put_u32(skb, OVS_DP_ATTR_EXCLUDE_CMDS,
+ dp->upcall_exclude_cmds))
+ goto nla_put_failure;
+
genlmsg_end(skb, ovs_header);
return 0;
@@ -1684,7 +1693,8 @@ static int ovs_dp_change(struct datapath *dp, struct nlattr *a[])
if (user_features & ~(OVS_DP_F_VPORT_PIDS |
OVS_DP_F_UNALIGNED |
OVS_DP_F_TC_RECIRC_SHARING |
- OVS_DP_F_DISPATCH_UPCALL_PER_CPU))
+ OVS_DP_F_DISPATCH_UPCALL_PER_CPU |
+ OVS_DP_F_EXCLUDE_UPCALL_FLOW_KEY))
return -EOPNOTSUPP;
#if !IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
@@ -1705,6 +1715,14 @@ static int ovs_dp_change(struct datapath *dp, struct nlattr *a[])
dp->user_features = user_features;
+ if (dp->user_features & OVS_DP_F_EXCLUDE_UPCALL_FLOW_KEY) {
+ if (!a[OVS_DP_ATTR_EXCLUDE_CMDS])
+ return -EINVAL;
+
+ dp->upcall_exclude_cmds =
+ nla_get_u32(a[OVS_DP_ATTR_EXCLUDE_CMDS]);
+ }
+
if (dp->user_features & OVS_DP_F_DISPATCH_UPCALL_PER_CPU &&
a[OVS_DP_ATTR_PER_CPU_PIDS]) {
/* Upcall Netlink Port IDs have been updated */
diff --git a/net/openvswitch/datapath.h b/net/openvswitch/datapath.h
index 0cd29971a907..3c951e25509e 100644
--- a/net/openvswitch/datapath.h
+++ b/net/openvswitch/datapath.h
@@ -101,6 +101,8 @@ struct datapath {
u32 max_headroom;
+ u32 upcall_exclude_cmds;
+
/* Switch meters. */
struct dp_meter_table meter_tbl;
--
2.34.3
On 11/22/22 15:03, Aaron Conole wrote:
> When processing upcall commands, two groups of data are available to
> userspace for processing: the actual packet data and the kernel
> sw flow key data. The inclusion of the flow key allows the userspace
> avoid running through the dissection again.
>
> However, the userspace can choose to ignore the flow key data, as is
> the case in some ovs-vswitchd upcall processing. For these messages,
> having the flow key data merely adds additional data to the upcall
> pipeline without any actual gain. Userspace simply throws the data
> away anyway.
Hi, Aaron. While it's true that OVS in userpsace is re-parsing the
packet from scratch and using the newly parsed key for the OpenFlow
translation, the kernel-porvided key is still used in a few important
places. Mainly for the compatibility checking. The use is described
here in more details:
https://docs.kernel.org/networking/openvswitch.html#flow-key-compatibility
We need to compare the key generated in userspace with the key
generated by the kernel to know if it's safe to install the new flow
to the kernel, i.e. if the kernel and OVS userpsace are parsing the
packet in the same way.
On the other hand, OVS today doesn't check the data, it only checks
which fields are present. So, if we can generate and pass the bitmap
of fields present in the key or something similar without sending the
full key, that might still save some CPU cycles and memory in the
socket buffer while preserving the ability to check for forward and
backward compatibility. What do you think?
The rest of the patch set seems useful even without patch #1 though.
Nit: This patch #1 should probably be merged with the patch #6 and be
at the end of a patch set, so the selftest and the main code are updated
at the same time.
Best regards, Ilya Maximets.
On 11/23/22 22:22, Ilya Maximets wrote:
> On 11/22/22 15:03, Aaron Conole wrote:
>> When processing upcall commands, two groups of data are available to
>> userspace for processing: the actual packet data and the kernel
>> sw flow key data. The inclusion of the flow key allows the userspace
>> avoid running through the dissection again.
>>
>> However, the userspace can choose to ignore the flow key data, as is
>> the case in some ovs-vswitchd upcall processing. For these messages,
>> having the flow key data merely adds additional data to the upcall
>> pipeline without any actual gain. Userspace simply throws the data
>> away anyway.
>
> Hi, Aaron. While it's true that OVS in userpsace is re-parsing the
> packet from scratch and using the newly parsed key for the OpenFlow
> translation, the kernel-porvided key is still used in a few important
> places. Mainly for the compatibility checking. The use is described
> here in more details:
> https://docs.kernel.org/networking/openvswitch.html#flow-key-compatibility
>
> We need to compare the key generated in userspace with the key
> generated by the kernel to know if it's safe to install the new flow
> to the kernel, i.e. if the kernel and OVS userpsace are parsing the
> packet in the same way.
>
Hi Ilya,
Do we need to do that for every packet?
Could we send a bitmask of supported fields to userspace at feature negotiation
and let OVS slowpath flows that it knows the kernel won't be able to handle
properly?
> On the other hand, OVS today doesn't check the data, it only checks
> which fields are present. So, if we can generate and pass the bitmap
> of fields present in the key or something similar without sending the
> full key, that might still save some CPU cycles and memory in the
> socket buffer while preserving the ability to check for forward and
> backward compatibility. What do you think?
>
>
> The rest of the patch set seems useful even without patch #1 though.
>
> Nit: This patch #1 should probably be merged with the patch #6 and be
> at the end of a patch set, so the selftest and the main code are updated
> at the same time.
>
> Best regards, Ilya Maximets.
> _______________________________________________
> dev mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
Thanks
--
Adrián Moreno
On 11/25/22 16:29, Adrian Moreno wrote:
>
>
> On 11/23/22 22:22, Ilya Maximets wrote:
>> On 11/22/22 15:03, Aaron Conole wrote:
>>> When processing upcall commands, two groups of data are available to
>>> userspace for processing: the actual packet data and the kernel
>>> sw flow key data. The inclusion of the flow key allows the userspace
>>> avoid running through the dissection again.
>>>
>>> However, the userspace can choose to ignore the flow key data, as is
>>> the case in some ovs-vswitchd upcall processing. For these messages,
>>> having the flow key data merely adds additional data to the upcall
>>> pipeline without any actual gain. Userspace simply throws the data
>>> away anyway.
>>
>> Hi, Aaron. While it's true that OVS in userpsace is re-parsing the
>> packet from scratch and using the newly parsed key for the OpenFlow
>> translation, the kernel-porvided key is still used in a few important
>> places. Mainly for the compatibility checking. The use is described
>> here in more details:
>> https://docs.kernel.org/networking/openvswitch.html#flow-key-compatibility
>>
>> We need to compare the key generated in userspace with the key
>> generated by the kernel to know if it's safe to install the new flow
>> to the kernel, i.e. if the kernel and OVS userpsace are parsing the
>> packet in the same way.
>>
>
> Hi Ilya,
>
> Do we need to do that for every packet?
> Could we send a bitmask of supported fields to userspace at feature
> negotiation and let OVS slowpath flows that it knows the kernel won't
> be able to handle properly?
It's not that simple, because supported fields in a packet depend
on previous fields in that same packet. For example, parsing TCP
header is generally supported, but it won't be parsed for IPv6
fragments (even the first one), number of vlan headers will affect
the parsing as we do not parse deeper than 2 vlan headers, etc.
So, I'm afraid we have to have a per-packet information, unless we
can somehow probe all the possible valid combinations of packet
headers.
>
>
>> On the other hand, OVS today doesn't check the data, it only checks
>> which fields are present. So, if we can generate and pass the bitmap
>> of fields present in the key or something similar without sending the
>> full key, that might still save some CPU cycles and memory in the
>> socket buffer while preserving the ability to check for forward and
>> backward compatibility. What do you think?
>>
>>
>> The rest of the patch set seems useful even without patch #1 though.
>>
>> Nit: This patch #1 should probably be merged with the patch #6 and be
>> at the end of a patch set, so the selftest and the main code are updated
>> at the same time.
>>
>> Best regards, Ilya Maximets.
>> _______________________________________________
>> dev mailing list
>> [email protected]
>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>>
>
> Thanks
On 11/25/22 16:51, Ilya Maximets wrote:
> On 11/25/22 16:29, Adrian Moreno wrote:
>>
>>
>> On 11/23/22 22:22, Ilya Maximets wrote:
>>> On 11/22/22 15:03, Aaron Conole wrote:
>>>> When processing upcall commands, two groups of data are available to
>>>> userspace for processing: the actual packet data and the kernel
>>>> sw flow key data. The inclusion of the flow key allows the userspace
>>>> avoid running through the dissection again.
>>>>
>>>> However, the userspace can choose to ignore the flow key data, as is
>>>> the case in some ovs-vswitchd upcall processing. For these messages,
>>>> having the flow key data merely adds additional data to the upcall
>>>> pipeline without any actual gain. Userspace simply throws the data
>>>> away anyway.
>>>
>>> Hi, Aaron. While it's true that OVS in userpsace is re-parsing the
>>> packet from scratch and using the newly parsed key for the OpenFlow
>>> translation, the kernel-porvided key is still used in a few important
>>> places. Mainly for the compatibility checking. The use is described
>>> here in more details:
>>> https://docs.kernel.org/networking/openvswitch.html#flow-key-compatibility
>>>
>>> We need to compare the key generated in userspace with the key
>>> generated by the kernel to know if it's safe to install the new flow
>>> to the kernel, i.e. if the kernel and OVS userpsace are parsing the
>>> packet in the same way.
>>>
>>
>> Hi Ilya,
>>
>> Do we need to do that for every packet?
>> Could we send a bitmask of supported fields to userspace at feature
>> negotiation and let OVS slowpath flows that it knows the kernel won't
>> be able to handle properly?
>
> It's not that simple, because supported fields in a packet depend
> on previous fields in that same packet. For example, parsing TCP
> header is generally supported, but it won't be parsed for IPv6
> fragments (even the first one), number of vlan headers will affect
> the parsing as we do not parse deeper than 2 vlan headers, etc.
> So, I'm afraid we have to have a per-packet information, unless we
> can somehow probe all the possible valid combinations of packet
> headers.
>
Surely. I understand that we'd need more than just a bit per field. Things like
L4 on IPv6 frags would need another bit and the number of VLAN headers would
need some more. But, are these a handful of exceptions or do we really need all
the possible combinations of headers? If it's a matter of naming a handful of
corner cases I think we could consider expressing them at initialization time
and safe some buffer space plus computation time both in kernel and userspace.
--
Adrián Moreno
>>
>>
>>> On the other hand, OVS today doesn't check the data, it only checks
>>> which fields are present. So, if we can generate and pass the bitmap
>>> of fields present in the key or something similar without sending the
>>> full key, that might still save some CPU cycles and memory in the
>>> socket buffer while preserving the ability to check for forward and
>>> backward compatibility. What do you think?
>>>
>>>
>>> The rest of the patch set seems useful even without patch #1 though.
>>>
>>> Nit: This patch #1 should probably be merged with the patch #6 and be
>>> at the end of a patch set, so the selftest and the main code are updated
>>> at the same time.
>>>
>>> Best regards, Ilya Maximets.
>>> _______________________________________________
>>> dev mailing list
>>> [email protected]
>>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>>>
>>
>> Thanks
>
Adrian Moreno <[email protected]> writes:
> On 11/25/22 16:51, Ilya Maximets wrote:
>> On 11/25/22 16:29, Adrian Moreno wrote:
>>>
>>>
>>> On 11/23/22 22:22, Ilya Maximets wrote:
>>>> On 11/22/22 15:03, Aaron Conole wrote:
>>>>> When processing upcall commands, two groups of data are available to
>>>>> userspace for processing: the actual packet data and the kernel
>>>>> sw flow key data. The inclusion of the flow key allows the userspace
>>>>> avoid running through the dissection again.
>>>>>
>>>>> However, the userspace can choose to ignore the flow key data, as is
>>>>> the case in some ovs-vswitchd upcall processing. For these messages,
>>>>> having the flow key data merely adds additional data to the upcall
>>>>> pipeline without any actual gain. Userspace simply throws the data
>>>>> away anyway.
>>>>
>>>> Hi, Aaron. While it's true that OVS in userpsace is re-parsing the
>>>> packet from scratch and using the newly parsed key for the OpenFlow
>>>> translation, the kernel-porvided key is still used in a few important
>>>> places. Mainly for the compatibility checking. The use is described
>>>> here in more details:
>>>> https://docs.kernel.org/networking/openvswitch.html#flow-key-compatibility
>>>>
>>>> We need to compare the key generated in userspace with the key
>>>> generated by the kernel to know if it's safe to install the new flow
>>>> to the kernel, i.e. if the kernel and OVS userpsace are parsing the
>>>> packet in the same way.
>>>>
>>>
>>> Hi Ilya,
>>>
>>> Do we need to do that for every packet?
>>> Could we send a bitmask of supported fields to userspace at feature
>>> negotiation and let OVS slowpath flows that it knows the kernel won't
>>> be able to handle properly?
>> It's not that simple, because supported fields in a packet depend
>> on previous fields in that same packet. For example, parsing TCP
>> header is generally supported, but it won't be parsed for IPv6
>> fragments (even the first one), number of vlan headers will affect
>> the parsing as we do not parse deeper than 2 vlan headers, etc.
>> So, I'm afraid we have to have a per-packet information, unless we
>> can somehow probe all the possible valid combinations of packet
>> headers.
>>
>
> Surely. I understand that we'd need more than just a bit per
> field. Things like L4 on IPv6 frags would need another bit and the
> number of VLAN headers would need some more. But, are these a handful
> of exceptions or do we really need all the possible combinations of
> headers? If it's a matter of naming a handful of corner cases I think
> we could consider expressing them at initialization time and safe some
> buffer space plus computation time both in kernel and userspace.
I will take a bit more of a look here - there must surely be a way to
express this when pulling information via DP_GET command so that we
don't need to wait for a packet to come in to figure out whether we can
parse it.
Ilya Maximets <[email protected]> writes:
> On 11/22/22 15:03, Aaron Conole wrote:
>> When processing upcall commands, two groups of data are available to
>> userspace for processing: the actual packet data and the kernel
>> sw flow key data. The inclusion of the flow key allows the userspace
>> avoid running through the dissection again.
>>
>> However, the userspace can choose to ignore the flow key data, as is
>> the case in some ovs-vswitchd upcall processing. For these messages,
>> having the flow key data merely adds additional data to the upcall
>> pipeline without any actual gain. Userspace simply throws the data
>> away anyway.
>
> Hi, Aaron. While it's true that OVS in userpsace is re-parsing the
> packet from scratch and using the newly parsed key for the OpenFlow
> translation, the kernel-porvided key is still used in a few important
> places. Mainly for the compatibility checking. The use is described
> here in more details:
> https://docs.kernel.org/networking/openvswitch.html#flow-key-compatibility
>
> We need to compare the key generated in userspace with the key
> generated by the kernel to know if it's safe to install the new flow
> to the kernel, i.e. if the kernel and OVS userpsace are parsing the
> packet in the same way.
>
> On the other hand, OVS today doesn't check the data, it only checks
> which fields are present. So, if we can generate and pass the bitmap
> of fields present in the key or something similar without sending the
> full key, that might still save some CPU cycles and memory in the
> socket buffer while preserving the ability to check for forward and
> backward compatibility. What do you think?
Maybe that can work. I will try testing. If so, then I would change
this semantic to send just the bitmap rather than omitting everything.
> The rest of the patch set seems useful even without patch #1 though.
I agree - but I didn't know if it made sense to submit the series
without adding something impactful (like a test). I will work a bit
more on the flow area - maybe I can add enough actions and matches to
implement basic flow tests to submit while we think more about the feature.
> Nit: This patch #1 should probably be merged with the patch #6 and be
> at the end of a patch set, so the selftest and the main code are updated
> at the same time.
Okay - I can restructure them this way.
> Best regards, Ilya Maximets.