Hi! Herbert:
In a system of multi cpu core, with multiple crypto hardware, different irq for each, is it possible the udp receive get out of order for the same tfm?
Let us say, of the same tfm, driver distributes requests to any available crypto hardware. For the same tfm, driver re-sequences response before complete callout to the same arrival sequence of the requests. The complete callout is running at tasklet level of the system.
Each irq is assigned to a different irq affinity cpu core to improve the performance. Therefore the iprecv and ipsec processing can be done by multiple cpu cores at the same time for the same tfm/udp.
I am seeing the following - if I assign irq affinity to different cpu cores, iperf server of udp session displays receive out of order message. If I assign them to the same core, or take default, then message stops. The assignment is done by hand while the system is running, by updating /proc/irq/xxx/smp_affinity, where xxx is the irq number. This scenario is very repeatable.
We are on linux-3.10.
Does it make sense?
Regards,
Chemin
On Fri, May 30, 2014 at 02:22:17PM +0000, Hsieh, Che-Min wrote:
> Hi! Herbert:
>
> In a system of multi cpu core, with multiple crypto hardware, different irq for each, is it possible the udp receive get out of order for the same tfm?
>
> Let us say, of the same tfm, driver distributes requests to any available crypto hardware. For the same tfm, driver re-sequences response before complete callout to the same arrival sequence of the requests. The complete callout is running at tasklet level of the system.
>
> Each irq is assigned to a different irq affinity cpu core to improve the performance. Therefore the iprecv and ipsec processing can be done by multiple cpu cores at the same time for the same tfm/udp.
>
> I am seeing the following - if I assign irq affinity to different cpu cores, iperf server of udp session displays receive out of order message. If I assign them to the same core, or take default, then message stops. The assignment is done by hand while the system is running, by updating /proc/irq/xxx/smp_affinity, where xxx is the irq number. This scenario is very repeatable.
>
> We are on linux-3.10.
>
> Does it make sense?
Are we talking about an in-tree driver? If so which one?
It's not certainly not meant to change packet ordering.
Cheers,
--
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Herbert:
Sorry, what is "in-tree" driver? This is Qcom crypto driver. It is part of android linux 3-10. I think Stanimir Varbanov is working on upstreaming the crypto driver. The driver version he cuts off, does static crypto engine assignment, i.e. crypto HW is assigned to a TFM on cra_init. That scheme is simple. But, under the same TFM, it does not take advantage of multiple crypto HW working in parallel. Later, we put in dynamic engine assignment per request, as described in the previous mail thread.
I am trying to understand why we are seeing iperf server out of order message once we have different cores assigned to crypto irq affinity.
Thanks.
Chemin
-----Original Message-----
From: Herbert Xu [mailto:[email protected]]
Sent: Monday, June 02, 2014 8:47 AM
To: Hsieh, Che-Min
Cc: [email protected]; [email protected]; Steffen Klassert
Subject: Re: [QUESTION] multi-core, multi-crypto, ipsec/udp out of order.
On Fri, May 30, 2014 at 02:22:17PM +0000, Hsieh, Che-Min wrote:
> Hi! Herbert:
>
> In a system of multi cpu core, with multiple crypto hardware, different irq for each, is it possible the udp receive get out of order for the same tfm?
>
> Let us say, of the same tfm, driver distributes requests to any available crypto hardware. For the same tfm, driver re-sequences response before complete callout to the same arrival sequence of the requests. The complete callout is running at tasklet level of the system.
>
> Each irq is assigned to a different irq affinity cpu core to improve the performance. Therefore the iprecv and ipsec processing can be done by multiple cpu cores at the same time for the same tfm/udp.
>
> I am seeing the following - if I assign irq affinity to different cpu cores, iperf server of udp session displays receive out of order message. If I assign them to the same core, or take default, then message stops. The assignment is done by hand while the system is running, by updating /proc/irq/xxx/smp_affinity, where xxx is the irq number. This scenario is very repeatable.
>
> We are on linux-3.10.
>
> Does it make sense?
Are we talking about an in-tree driver? If so which one?
It's not certainly not meant to change packet ordering.
Cheers,
--
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Tue, Jun 03, 2014 at 01:48:56PM +0000, Hsieh, Che-Min wrote:
> Herbert:
> Sorry, what is "in-tree" driver? This is Qcom crypto driver. It is part of android linux 3-10. I think Stanimir Varbanov is working on upstreaming the crypto driver. The driver version he cuts off, does static crypto engine assignment, i.e. crypto HW is assigned to a TFM on cra_init. That scheme is simple. But, under the same TFM, it does not take advantage of multiple crypto HW working in parallel. Later, we put in dynamic engine assignment per request, as described in the previous mail thread.
An in-tree driver is one that's in the mainline kernel.
Cheers,
--
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt