Hi Herbert,
Sorry for the late reply
On 15/01/2020 14:02:34 +0800, Herbert Xu wrote:
> On Tue, Jan 14, 2020 at 03:23:30PM +0530, Ayush Sawal wrote
>> Hi all,
>>
>> The hardware crypto drivers have a limit on max number of sgs they can
>> handle per crypto request.If data size in one crypto request is
>> huge,hardware crypto driver may not be able to send the request in single
>> shot to hardware and end up using fallback to software.
>>
>> Does it make sense to have a new API for crypto drivers using that
>> drivers
>> can advertise the max number of sg it can handle in one crypto request?
>>
>> and then? crypto framework may also have to include the similar API which
>> crypto framework user can use while forming the crypto request .
>>
>> Does this implementation make sense?
>
> What is the actual limit? Are you running into this limit with
> real-life requests?
The max data limit is 15 sgs where each sg contains data of mtu size .
we are running a netperf udp stream test over ipsec tunnel .The ipsec
tunnel is established between two hosts which are directly connected
Thanks,
Ayush
On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
>
> The max data limit is 15 sgs where each sg contains data of mtu size .
> we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
> is established between two hosts which are directly connected
Are you actually getting 15-element SG lists from IPsec? What is
generating an skb with 15-element SG lists?
Cheers,
--
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Hi Herbert,
On 1/17/2020 11:53 AM, Herbert Xu wrote:
> On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
>> The max data limit is 15 sgs where each sg contains data of mtu size .
>> we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
>> is established between two hosts which are directly connected
> Are you actually getting 15-element SG lists from IPsec? What is
> generating an skb with 15-element SG lists?
we have established the ipsec tunnel in transport mode using ip xfrm.
and running traffic using netserver and netperf.
In server side we are running
netserver -4
In client side we are running
"netperf -H <serverip> -p <port> -t UDP_STREAM -Cc -- -m 21k"
where the packet size is 21k ,which is then fragmented into 15 ip
fragments each of mtu size.
The mtu size currently is 1500bytes.
On Fri, Jan 17, 2020 at 12:13:07PM +0530, Ayush Sawal wrote:
> Hi Herbert,
>
> On 1/17/2020 11:53 AM, Herbert Xu wrote:
> > On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
> > > The max data limit is 15 sgs where each sg contains data of mtu size .
> > > we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
> > > is established between two hosts which are directly connected
> > Are you actually getting 15-element SG lists from IPsec? What is
> > generating an skb with 15-element SG lists?
> we have established the ipsec tunnel in transport mode using ip xfrm.
> and running traffic using netserver and netperf.
>
> In server side we are running
> netserver -4
> In client side we are running
> "netperf -H <serverip> -p <port> -t UDP_STREAM? -Cc -- -m 21k"
> where the packet size is 21k ,which is then fragmented into 15 ip fragments
> each of mtu size.
I'm lacking a bit of context here, but this should generate 15 IP
packets that are encrypted one by one.
Hi steffen,
On 1/17/2020 12:34 PM, Steffen Klassert wrote:
> On Fri, Jan 17, 2020 at 12:13:07PM +0530, Ayush Sawal wrote:
>> Hi Herbert,
>>
>> On 1/17/2020 11:53 AM, Herbert Xu wrote:
>>> On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
>>>> The max data limit is 15 sgs where each sg contains data of mtu size .
>>>> we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
>>>> is established between two hosts which are directly connected
>>> Are you actually getting 15-element SG lists from IPsec? What is
>>> generating an skb with 15-element SG lists?
>> we have established the ipsec tunnel in transport mode using ip xfrm.
>> and running traffic using netserver and netperf.
>>
>> In server side we are running
>> netserver -4
>> In client side we are running
>> "netperf -H <serverip> -p <port> -t UDP_STREAM -Cc -- -m 21k"
>> where the packet size is 21k ,which is then fragmented into 15 ip fragments
>> each of mtu size.
> I'm lacking a bit of context here, but this should generate 15 IP
> packets that are encrypted one by one.
This is what i observed ,please correct me if i am wrong.
The packet when reaches esp_output(),is in socket buffer and based on
the number of frags ,sg is initialized using
sg_init_table(sg,frags),where frags are 15 in our case.
The socket buffer data is then copied to this sg and then struct
aead_request members are filled.
After this crypto aead request which contains all data in its sg list
goes to hw crypto driver for encryption in a single request.
In the crypto driver we are receiving a single aead-request with all 15
sgs in that request.
Thanks,
Ayush
On Fri, Jan 17, 2020 at 04:28:54PM +0530, Ayush Sawal wrote:
> Hi steffen,
>
> On 1/17/2020 12:34 PM, Steffen Klassert wrote:
> > On Fri, Jan 17, 2020 at 12:13:07PM +0530, Ayush Sawal wrote:
> > > Hi Herbert,
> > >
> > > On 1/17/2020 11:53 AM, Herbert Xu wrote:
> > > > On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
> > > > > The max data limit is 15 sgs where each sg contains data of mtu size .
> > > > > we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
> > > > > is established between two hosts which are directly connected
> > > > Are you actually getting 15-element SG lists from IPsec? What is
> > > > generating an skb with 15-element SG lists?
> > > we have established the ipsec tunnel in transport mode using ip xfrm.
> > > and running traffic using netserver and netperf.
> > >
> > > In server side we are running
> > > netserver -4
> > > In client side we are running
> > > "netperf -H <serverip> -p <port> -t UDP_STREAM? -Cc -- -m 21k"
> > > where the packet size is 21k ,which is then fragmented into 15 ip fragments
> > > each of mtu size.
> > I'm lacking a bit of context here, but this should generate 15 IP
> > packets that are encrypted one by one.
> This is what i observed ,please correct me if i am wrong.
> The packet when reaches esp_output(),is in socket buffer and based on the
> number of frags ,sg is initialized? using
> sg_init_table(sg,frags),where frags are 15 in our case.
The packet should be IP fragmented before it enters esp_output()
unless this is a UDP GSO packet. What kind of device do you use
here? Is it a crypto accelerator or a NIC that can do ESP offloads?
Hi steffen,
On 1/17/2020 5:47 PM, Steffen Klassert wrote:
> On Fri, Jan 17, 2020 at 04:28:54PM +0530, Ayush Sawal wrote:
>> Hi steffen,
>>
>> On 1/17/2020 12:34 PM, Steffen Klassert wrote:
>>> On Fri, Jan 17, 2020 at 12:13:07PM +0530, Ayush Sawal wrote:
>>>> Hi Herbert,
>>>>
>>>> On 1/17/2020 11:53 AM, Herbert Xu wrote:
>>>>> On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
>>>>>> The max data limit is 15 sgs where each sg contains data of mtu size .
>>>>>> we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
>>>>>> is established between two hosts which are directly connected
>>>>> Are you actually getting 15-element SG lists from IPsec? What is
>>>>> generating an skb with 15-element SG lists?
>>>> we have established the ipsec tunnel in transport mode using ip xfrm.
>>>> and running traffic using netserver and netperf.
>>>>
>>>> In server side we are running
>>>> netserver -4
>>>> In client side we are running
>>>> "netperf -H <serverip> -p <port> -t UDP_STREAM -Cc -- -m 21k"
>>>> where the packet size is 21k ,which is then fragmented into 15 ip fragments
>>>> each of mtu size.
>>> I'm lacking a bit of context here, but this should generate 15 IP
>>> packets that are encrypted one by one.
>> This is what i observed ,please correct me if i am wrong.
>> The packet when reaches esp_output(),is in socket buffer and based on the
>> number of frags ,sg is initialized using
>> sg_init_table(sg,frags),where frags are 15 in our case.
> The packet should be IP fragmented before it enters esp_output()
> unless this is a UDP GSO packet. What kind of device do you use
> here? Is it a crypto accelerator or a NIC that can do ESP offloads?
We have device which works as a crypto accelerator . It just encrypts
the packets and send it back to kernel.
On Fri, Jan 17, 2020 at 07:08:05PM +0530, Ayush Sawal wrote:
> Hi steffen,
>
> On 1/17/2020 5:47 PM, Steffen Klassert wrote:
> > On Fri, Jan 17, 2020 at 04:28:54PM +0530, Ayush Sawal wrote:
> > > Hi steffen,
> > >
> > > On 1/17/2020 12:34 PM, Steffen Klassert wrote:
> > > > On Fri, Jan 17, 2020 at 12:13:07PM +0530, Ayush Sawal wrote:
> > > > > Hi Herbert,
> > > > >
> > > > > On 1/17/2020 11:53 AM, Herbert Xu wrote:
> > > > > > On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
> > > > > > > The max data limit is 15 sgs where each sg contains data of mtu size .
> > > > > > > we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
> > > > > > > is established between two hosts which are directly connected
> > > > > > Are you actually getting 15-element SG lists from IPsec? What is
> > > > > > generating an skb with 15-element SG lists?
> > > > > we have established the ipsec tunnel in transport mode using ip xfrm.
> > > > > and running traffic using netserver and netperf.
> > > > >
> > > > > In server side we are running
> > > > > netserver -4
> > > > > In client side we are running
> > > > > "netperf -H <serverip> -p <port> -t UDP_STREAM? -Cc -- -m 21k"
> > > > > where the packet size is 21k ,which is then fragmented into 15 ip fragments
> > > > > each of mtu size.
> > > > I'm lacking a bit of context here, but this should generate 15 IP
> > > > packets that are encrypted one by one.
> > > This is what i observed ,please correct me if i am wrong.
> > > The packet when reaches esp_output(),is in socket buffer and based on the
> > > number of frags ,sg is initialized? using
> > > sg_init_table(sg,frags),where frags are 15 in our case.
> > The packet should be IP fragmented before it enters esp_output()
> > unless this is a UDP GSO packet. What kind of device do you use
> > here? Is it a crypto accelerator or a NIC that can do ESP offloads?
>
> We have device which works as a crypto accelerator . It just encrypts the
> packets and send it back to kernel.
I just did a test and I see the same behaviour. Seems like I was
mistaken, we actually fragment the ESP packets. The only case
where we do pre-encap fragmentation is IPv6 tunnel mode. But I
wonder if it would make sense to avoid to have ESP fragments on
the wire.
Hi Steffen,
On 1/20/2020 3:07 PM, Steffen Klassert wrote:
> On Fri, Jan 17, 2020 at 07:08:05PM +0530, Ayush Sawal wrote:
>> Hi steffen,
>>
>> On 1/17/2020 5:47 PM, Steffen Klassert wrote:
>>> On Fri, Jan 17, 2020 at 04:28:54PM +0530, Ayush Sawal wrote:
>>>> Hi steffen,
>>>>
>>>> On 1/17/2020 12:34 PM, Steffen Klassert wrote:
>>>>> On Fri, Jan 17, 2020 at 12:13:07PM +0530, Ayush Sawal wrote:
>>>>>> Hi Herbert,
>>>>>>
>>>>>> On 1/17/2020 11:53 AM, Herbert Xu wrote:
>>>>>>> On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
>>>>>>>> The max data limit is 15 sgs where each sg contains data of mtu size .
>>>>>>>> we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
>>>>>>>> is established between two hosts which are directly connected
>>>>>>> Are you actually getting 15-element SG lists from IPsec? What is
>>>>>>> generating an skb with 15-element SG lists?
>>>>>> we have established the ipsec tunnel in transport mode using ip xfrm.
>>>>>> and running traffic using netserver and netperf.
>>>>>>
>>>>>> In server side we are running
>>>>>> netserver -4
>>>>>> In client side we are running
>>>>>> "netperf -H <serverip> -p <port> -t UDP_STREAM -Cc -- -m 21k"
>>>>>> where the packet size is 21k ,which is then fragmented into 15 ip fragments
>>>>>> each of mtu size.
>>>>> I'm lacking a bit of context here, but this should generate 15 IP
>>>>> packets that are encrypted one by one.
>>>> This is what i observed ,please correct me if i am wrong.
>>>> The packet when reaches esp_output(),is in socket buffer and based on the
>>>> number of frags ,sg is initialized using
>>>> sg_init_table(sg,frags),where frags are 15 in our case.
>>> The packet should be IP fragmented before it enters esp_output()
>>> unless this is a UDP GSO packet. What kind of device do you use
>>> here? Is it a crypto accelerator or a NIC that can do ESP offloads?
>> We have device which works as a crypto accelerator . It just encrypts the
>> packets and send it back to kernel.
> I just did a test and I see the same behaviour. Seems like I was
> mistaken, we actually fragment the ESP packets. The only case
> where we do pre-encap fragmentation is IPv6 tunnel mode. But I
> wonder if it would make sense to avoid to have ESP fragments on
> the wire.
As we have a crypto accelarator as device when the request is send to
the crypto driver from esp_output ,
the aead_request has all the fragments in its src sg and the problem
which we are facing is when this
src sg nents becomes greater than 15 ,15 is our crypto driver's max sg
limit to handle the request in one shot.
Does it make sense for a crypto driver to advertise the maximum amount
of sg it can handle for a single
request and then handling this in crypto framework while forming the
crypto request?
Thanks,
Ayush
On Mon, Jan 20, 2020 at 2:35 PM Ayush Sawal
<[email protected]> wrote:
> As we have a crypto accelarator as device when the request is send to
> the crypto driver from esp_output ,
> the aead_request has all the fragments in its src sg and the problem
> which we are facing is when this
> src sg nents becomes greater than 15 ,15 is our crypto driver's max sg
> limit to handle the request in one shot.
>
> Does it make sense for a crypto driver to advertise the maximum amount
> of sg it can handle for a single
> request and then handling this in crypto framework while forming the
> crypto request?
>
As I maintain the driver of another crypto accelerator I sympathize
with the need but I question the proposed solution.
Consider: your specific driver is limited by the number of
scattergather entries. Another implementation might be limited
by something else such as the total overall size of the request buffer
and probably half a dozen other considerations.
Should we now be passing all this capability information to the crypto
API core? and what happens if a new driver
has a limitation in a different quality?
So no, the solution to advertise the specific capability limitation of
each implementation does not seem to be a good one.
We already have a solution to the problem - initiate a fallback TFM
request and use it if you cannot fulfill the request on your own.
I do agree however that having each implementation registering and
keeping their own fallback tfm request just for these cases has some
overhead and a redundancy.
Maybe a better solution would be to allow implementation to return to
the Crypto API core a special return value (maybe -EAGAIN?) that tells
it that although the request is a valid one, this specific
implementation cannot fulfil it and let the crypto API core do the
fallback?
It sounds like it can be simpler to the implementation providers AND
save some redundant code...
--
Gilad Ben-Yossef
Chief Coffee Drinker
values of β will give rise to dom!