I'm not sure how to describe this.
4.11rc2 worked, after that, no.
My ipsec tunnel comes up ok. ICMP works. UDP works. But TCP, the
sender [which is the ipsec client] does not reach the destination.
Its not a routing rule issue (since ICMP/UDP work).
Its not a traffic selector just selecting TCP (I think) since ipsec
status shows just a subnet, no protocol.
Using tcpdump:
# iptables -t mangle -I PREROUTING -m policy --pol ipsec --dir in -j
NFLOG --nflog-group 5
# iptables -t mangle -I POSTROUTING -m policy --pol ipsec --dir out -j
NFLOG --nflog-group 5
# tcpdump -s 0 -n -i nflog:5
I see that it thinks it is sending the TCP packet, but the server end
does not receive.
Does anyone have any suggestion to try?
strongswan is 5.5.1 [on ubuntu 17.04]
kernel is 4.11.0-041100rc8-generic
My rightsubnet is
rightsubnet = 192.168.128.0/17,10.0.0.0/8
so no specific protocol selected, the result is:
CHILD_SA sv{1} established with SPIs c05f1b6c_i 0d58815a_o and TS
192.168.130.4/32 === 10.0.0.0/8 192.168.128.0/17
I tried changing charondebug net=3, but i'm not sure how to interpret
the output:
Apr 25 21:06:34 office charon: 04[NET] received packet: from
64.7.137.180[4500] to 172.16.0.8[4500] (80 bytes)
Apr 25 21:06:34 office charon: 04[ENC] parsed INFORMATIONAL request 4 [ ]
Apr 25 21:06:34 office charon: 04[ENC] generating INFORMATIONAL response 4 [ ]
Apr 25 21:06:34 office charon: 04[NET] sending packet: from
172.16.0.8[4500] to 64.7.137.180[4500] (80 bytes)
Hi Don,
Can you see if this bug started happening in v4.11-rc3? Since your
running Ubuntu, a kernel is already available here:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.11-rc3/
If v4.11-rc2 is good and -rc3 is bad, I can help you bisect between the two.
Thanks,
Joe
On 04/25/2017 09:08 PM, Don Bowman wrote:
> I'm not sure how to describe this.
>
> 4.11rc2 worked, after that, no.
>
> My ipsec tunnel comes up ok. ICMP works. UDP works. But TCP, the
> sender [which is the ipsec client] does not reach the destination.
>
> Its not a routing rule issue (since ICMP/UDP work).
> Its not a traffic selector just selecting TCP (I think) since ipsec
> status shows just a subnet, no protocol.
>
> Using tcpdump:
> # iptables -t mangle -I PREROUTING -m policy --pol ipsec --dir in -j
> NFLOG --nflog-group 5
> # iptables -t mangle -I POSTROUTING -m policy --pol ipsec --dir out -j
> NFLOG --nflog-group 5
> # tcpdump -s 0 -n -i nflog:5
>
> I see that it thinks it is sending the TCP packet, but the server end
> does not receive.
>
> Does anyone have any suggestion to try?
>
> strongswan is 5.5.1 [on ubuntu 17.04]
> kernel is 4.11.0-041100rc8-generic
>
> My rightsubnet is
> rightsubnet = 192.168.128.0/17,10.0.0.0/8
>
> so no specific protocol selected, the result is:
> CHILD_SA sv{1} established with SPIs c05f1b6c_i 0d58815a_o and TS
> 192.168.130.4/32 === 10.0.0.0/8 192.168.128.0/17
>
> I tried changing charondebug net=3, but i'm not sure how to interpret
> the output:
>
> Apr 25 21:06:34 office charon: 04[NET] received packet: from
> 64.7.137.180[4500] to 172.16.0.8[4500] (80 bytes)
> Apr 25 21:06:34 office charon: 04[ENC] parsed INFORMATIONAL request 4 [ ]
> Apr 25 21:06:34 office charon: 04[ENC] generating INFORMATIONAL response 4 [ ]
> Apr 25 21:06:34 office charon: 04[NET] sending packet: from
> 172.16.0.8[4500] to 64.7.137.180[4500] (80 bytes)
(Cc'ing netdev and IPSec maintainers)
On Tue, Apr 25, 2017 at 6:08 PM, Don Bowman <[email protected]> wrote:
> I'm not sure how to describe this.
>
> 4.11rc2 worked, after that, no.
>
> My ipsec tunnel comes up ok. ICMP works. UDP works. But TCP, the
> sender [which is the ipsec client] does not reach the destination.
>
> Its not a routing rule issue (since ICMP/UDP work).
> Its not a traffic selector just selecting TCP (I think) since ipsec
> status shows just a subnet, no protocol.
>
> Using tcpdump:
> # iptables -t mangle -I PREROUTING -m policy --pol ipsec --dir in -j
> NFLOG --nflog-group 5
> # iptables -t mangle -I POSTROUTING -m policy --pol ipsec --dir out -j
> NFLOG --nflog-group 5
> # tcpdump -s 0 -n -i nflog:5
>
> I see that it thinks it is sending the TCP packet, but the server end
> does not receive.
>
> Does anyone have any suggestion to try?
>
> strongswan is 5.5.1 [on ubuntu 17.04]
> kernel is 4.11.0-041100rc8-generic
>
> My rightsubnet is
> rightsubnet = 192.168.128.0/17,10.0.0.0/8
>
> so no specific protocol selected, the result is:
> CHILD_SA sv{1} established with SPIs c05f1b6c_i 0d58815a_o and TS
> 192.168.130.4/32 === 10.0.0.0/8 192.168.128.0/17
>
> I tried changing charondebug net=3, but i'm not sure how to interpret
> the output:
>
> Apr 25 21:06:34 office charon: 04[NET] received packet: from
> 64.7.137.180[4500] to 172.16.0.8[4500] (80 bytes)
> Apr 25 21:06:34 office charon: 04[ENC] parsed INFORMATIONAL request 4 [ ]
> Apr 25 21:06:34 office charon: 04[ENC] generating INFORMATIONAL response 4 [ ]
> Apr 25 21:06:34 office charon: 04[NET] sending packet: from
> 172.16.0.8[4500] to 64.7.137.180[4500] (80 bytes)
On Wed, Apr 26, 2017 at 10:01:34PM -0700, Cong Wang wrote:
> (Cc'ing netdev and IPSec maintainers)
>
> On Tue, Apr 25, 2017 at 6:08 PM, Don Bowman <[email protected]> wrote:
> > I'm not sure how to describe this.
> >
> > 4.11rc2 worked, after that, no.
We had some recent IPsec GRO changes, this could influence TCP.
But these changes were introduced before rc2. If I read this correct,
the regression was introduced between rc2 and rc3, right?
> >
> > My ipsec tunnel comes up ok.
When talking about IPsec, I guess you use ESP, right?
> > ICMP works. UDP works. But TCP, the
> > sender [which is the ipsec client] does not reach the destination.
> >
> > Its not a routing rule issue (since ICMP/UDP work).
> > Its not a traffic selector just selecting TCP (I think) since ipsec
> > status shows just a subnet, no protocol.
> >
> > Using tcpdump:
> > # iptables -t mangle -I PREROUTING -m policy --pol ipsec --dir in -j
> > NFLOG --nflog-group 5
> > # iptables -t mangle -I POSTROUTING -m policy --pol ipsec --dir out -j
> > NFLOG --nflog-group 5
> > # tcpdump -s 0 -n -i nflog:5
> >
> > I see that it thinks it is sending the TCP packet, but the server end
> > does not receive.
> >
> > Does anyone have any suggestion to try?
If it is a GRO issue, then it is on the receive side, could you do
tcpdump on the receiving interface to see what you get there?
What shows /proc/net/xfrm_stat?
Can you do 'ip -s x s' to see if the SAs are used?
Do you have INET_ESP_OFFLOAD enabled?
On 27 April 2017 at 04:42, Steffen Klassert
<[email protected]> wrote:
> On Wed, Apr 26, 2017 at 10:01:34PM -0700, Cong Wang wrote:
>> (Cc'ing netdev and IPSec maintainers)
>>
>> On Tue, Apr 25, 2017 at 6:08 PM, Don Bowman <[email protected]> wrote:
for 'esp' question, i have ' esp = aes256-sha256-modp1536!' is that
what you mean?
its nat-aware tunnel [from my desktop pc to my office]
root@office:~# ip -s x s
src 172.16.0.8 dst 64.7.137.180
proto esp spi 0x0d588366(223904614) reqid 1(0x00000001) mode tunnel
replay-window 0 seq 0x00000000 flag af-unspec (0x00100000)
auth-trunc hmac(sha256)
0x046cafdf19c5d78d1c29165d96a0b9fce1c500029d77be0fe956dce1bf80a86a
(256 bits) 128
enc cbc(aes)
0x79ff2fbc2178eb468de6ff16612f0603b514a1d1d5f375c67222294463ec7c62
(256 bits)
encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
anti-replay context: seq 0x0, oseq 0x28, bitmap 0x00000000
lifetime config:
limit: soft (INF)(bytes), hard (INF)(bytes)
limit: soft (INF)(packets), hard (INF)(packets)
expire add: soft 42843(sec), hard 43200(sec)
expire use: soft 0(sec), hard 0(sec)
lifetime current:
2986(bytes), 40(packets)
add 2017-04-27 18:08:12 use 2017-04-27 18:08:12
stats:
replay-window 0 replay 0 failed 0
src 64.7.137.180 dst 172.16.0.8
proto esp spi 0xcd366c03(3442895875) reqid 1(0x00000001) mode tunnel
replay-window 32 seq 0x00000000 flag af-unspec (0x00100000)
auth-trunc hmac(sha256)
0x4158741cc971c49417d60165f19ed02249385c7bba808927d4a9e7b45fb30c5b
(256 bits) 128
enc cbc(aes)
0x77592c79c964787bca5012214b85172b06deb7b3f06aac02e3934dd9ead67c15
(256 bits)
encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
anti-replay context: seq 0x27, oseq 0x0, bitmap 0xffffffff
lifetime config:
limit: soft (INF)(bytes), hard (INF)(bytes)
limit: soft (INF)(packets), hard (INF)(packets)
expire add: soft 42873(sec), hard 43200(sec)
expire use: soft 0(sec), hard 0(sec)
lifetime current:
4501(bytes), 38(packets)
add 2017-04-27 18:08:12 use 2017-04-27 18:08:12
stats:
replay-window 0 replay 0 failed 0
>> >
>> > My ipsec tunnel comes up ok.
>
> When talking about IPsec, I guess you use ESP, right?
...
>
> If it is a GRO issue, then it is on the receive side, could you do
> tcpdump on the receiving interface to see what you get there?
I'm not sure what you mean the receiving interface, you mean the
outer, the native interface?
listening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes
18:11:32.061501 IP 172.16.0.8.3416 > 64.7.137.180.33638: truncated-udplength 0
18:11:32.788091 IP 64.7.137.180.4500 > 172.16.0.8.4500: NONESP-encap:
isakmp: child_sa inf2
18:11:32.788354 IP 172.16.0.8.4500 > 64.7.137.180.4500: NONESP-encap:
isakmp: child_sa inf2[IR]
18:11:33.066830 IP 172.16.0.8.3416 > 64.7.137.180.33638: truncated-udplength 0
18:11:35.082839 IP 172.16.0.8.3416 > 64.7.137.180.33638: truncated-udplength 0
18:11:37.807945 IP 64.7.137.180.4500 > 172.16.0.8.4500: NONESP-encap:
isakmp: child_sa inf2
18:11:37.808300 IP 172.16.0.8.4500 > 64.7.137.180.4500: NONESP-encap:
isakmp: child_sa inf2[IR]
is what i see there for the 'curl' command that doesn't complete.
>
> What shows /proc/net/xfrm_stat?
root@office:~# cat /proc/net/xfrm_stat
XfrmInError 0
XfrmInBufferError 0
XfrmInHdrError 0
XfrmInNoStates 0
XfrmInStateProtoError 0
XfrmInStateModeError 0
XfrmInStateSeqError 0
XfrmInStateExpired 0
XfrmInStateMismatch 0
XfrmInStateInvalid 0
XfrmInTmplMismatch 0
XfrmInNoPols 0
XfrmInPolBlock 0
XfrmInPolError 0
XfrmOutError 0
XfrmOutBundleGenError 0
XfrmOutBundleCheckError 0
XfrmOutNoStates 0
XfrmOutStateProtoError 0
XfrmOutStateModeError 0
XfrmOutStateSeqError 0
XfrmOutStateExpired 0
XfrmOutPolBlock 0
XfrmOutPolDead 0
XfrmOutPolError 0
XfrmFwdHdrError 0
XfrmOutStateInvalid 0
XfrmAcquireError 0
>
> Can you do 'ip -s x s' to see if the SAs are used?
>
> Do you have INET_ESP_OFFLOAD enabled?
>
CONFIG_INET_ESP=m
CONFIG_INET_ESP_OFFLOAD=m
CONFIG_INET6_ESP=m
CONFIG_INET6_ESP_OFFLOAD=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
# lsmod |grep esp
esp4 20480 2
xfrm_algo 16384 5 xfrm_user,esp4,ah4,af_key,xfrm_ipcomp
On 26 April 2017 at 15:06, Joseph Salisbury
<[email protected]> wrote:
> Hi Don,
>
> Can you see if this bug started happening in v4.11-rc3? Since your
> running Ubuntu, a kernel is already available here:
> http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.11-rc3/
It appears I was incorrect about where this was introduced.
4.10.1-041001-generic works correctly.
4.11.0-041100rc1-generic does not
THese are both from the kernel-mainline ubuntu.
>
> If v4.11-rc2 is good and -rc3 is bad, I can help you bisect between the two.
>
> Thanks,
>
> Joe
>
>
>
> On 04/25/2017 09:08 PM, Don Bowman wrote:
>> I'm not sure how to describe this.
>>
>> 4.11rc2 worked, after that, no.
>>
>> My ipsec tunnel comes up ok. ICMP works. UDP works. But TCP, the
>> sender [which is the ipsec client] does not reach the destination.
>>
>> Its not a routing rule issue (since ICMP/UDP work).
>> Its not a traffic selector just selecting TCP (I think) since ipsec
>> status shows just a subnet, no protocol.
>>
>> Using tcpdump:
>> # iptables -t mangle -I PREROUTING -m policy --pol ipsec --dir in -j
>> NFLOG --nflog-group 5
>> # iptables -t mangle -I POSTROUTING -m policy --pol ipsec --dir out -j
>> NFLOG --nflog-group 5
>> # tcpdump -s 0 -n -i nflog:5
>>
>> I see that it thinks it is sending the TCP packet, but the server end
>> does not receive.
>>
>> Does anyone have any suggestion to try?
>>
>> strongswan is 5.5.1 [on ubuntu 17.04]
>> kernel is 4.11.0-041100rc8-generic
>>
>h> My rightsubnet is
>> rightsubnet = 192.168.128.0/17,10.0.0.0/8
>>
>> so no specific protocol selected, the result is:
>> CHILD_SA sv{1} established with SPIs c05f1b6c_i 0d58815a_o and TS
>> 192.168.130.4/32 === 10.0.0.0/8 192.168.128.0/17
>>
>> I tried changing charondebug net=3, but i'm not sure how to interpret
>> the output:
>>
>> Apr 25 21:06:34 office charon: 04[NET] received packet: from
>> 64.7.137.180[4500] to 172.16.0.8[4500] (80 bytes)
>> Apr 25 21:06:34 office charon: 04[ENC] parsed INFORMATIONAL request 4 [ ]
>> Apr 25 21:06:34 office charon: 04[ENC] generating INFORMATIONAL response 4 [ ]
>> Apr 25 21:06:34 office charon: 04[NET] sending packet: from
>> 172.16.0.8[4500] to 64.7.137.180[4500] (80 bytes)
>
>
On Thu, Apr 27, 2017 at 06:13:38PM -0400, Don Bowman wrote:
> On 27 April 2017 at 04:42, Steffen Klassert <[email protected]>
> wrote:
> > On Wed, Apr 26, 2017 at 10:01:34PM -0700, Cong Wang wrote:
> >> (Cc'ing netdev and IPSec maintainers)
> >>
> >> On Tue, Apr 25, 2017 at 6:08 PM, Don Bowman <[email protected]> wrote:
>
> for 'esp' question, i have ' esp = aes256-sha256-modp1536!' is that what
> you mean?
> its nat-aware tunnel [from my desktop pc to my office]
>
> root@office:~# ip -s x s
> src 172.16.0.8 dst 64.7.137.180
> proto esp spi 0x0d588366(223904614) reqid 1(0x00000001) mode tunnel
> replay-window 0 seq 0x00000000 flag af-unspec (0x00100000)
> auth-trunc hmac(sha256)
> 0x046cafdf19c5d78d1c29165d96a0b9fce1c500029d77be0fe956dce1bf80a86a (256
> bits) 128
> enc cbc(aes)
> 0x79ff2fbc2178eb468de6ff16612f0603b514a1d1d5f375c67222294463ec7c62 (256
> bits)
> encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
Ok, this is espinudp. This information was important.
>
> I'm not sure what you mean the receiving interface, you mean the outer, the
> native interface?
> listening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes
> 18:11:32.061501 IP 172.16.0.8.3416 > 64.7.137.180.33638:
> truncated-udplength 0
> 18:11:32.788091 IP 64.7.137.180.4500 > 172.16.0.8.4500: NONESP-encap:
> isakmp: child_sa inf2
> 18:11:32.788354 IP 172.16.0.8.4500 > 64.7.137.180.4500: NONESP-encap:
> isakmp: child_sa inf2[IR]
> 18:11:33.066830 IP 172.16.0.8.3416 > 64.7.137.180.33638:
> truncated-udplength 0
> 18:11:35.082839 IP 172.16.0.8.3416 > 64.7.137.180.33638:
> truncated-udplength 0
This is not a GRO issue as I thought, the TX side is already broken.
Could you please try the patch below?
Subject: [PATCH] esp4: Fix udpencap for local TCP packets.
Locally generated TCP packets are usually cloned, so we
do skb_cow_data() on this packets. After that we need to
reload the pointer to the esp header. On udpencap this
header has an offset to skb_transport_header, so take this
offset into account.
Fixes: commit cac2661c53f ("esp4: Avoid skb_cow_data whenever possible")
Signed-off-by: Steffen Klassert <[email protected]>
---
net/ipv4/esp4.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c
index b1e2444..ab71fbb 100644
--- a/net/ipv4/esp4.c
+++ b/net/ipv4/esp4.c
@@ -223,6 +223,7 @@ static int esp_output(struct xfrm_state *x, struct sk_buff *skb)
int extralen;
int tailen;
__be64 seqno;
+ int esp_offset = 0;
__u8 proto = *skb_mac_header(skb);
/* skb is pure payload to encrypt */
@@ -288,6 +289,8 @@ static int esp_output(struct xfrm_state *x, struct sk_buff *skb)
break;
}
+ esp_offset = (unsigned char *)esph - (unsigned char *)uh;
+
*skb_mac_header(skb) = IPPROTO_UDP;
}
@@ -397,7 +400,7 @@ static int esp_output(struct xfrm_state *x, struct sk_buff *skb)
goto error;
nfrags = err;
tail = skb_tail_pointer(trailer);
- esph = ip_esp_hdr(skb);
+ esph = (struct ip_esp_hdr *)(skb_transport_header(skb) + esp_offset);
skip_cow:
esp_output_fill_trailer(tail, tfclen, plen, proto);
--
2.7.4
On Fri, 2017-04-28 at 09:13 +0200, Steffen Klassert wrote:
> encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
>
> Ok, this is espinudp. This information was important.
> This is not a GRO issue as I thought, the TX side is already broken.
>
> Could you please try the patch below?
>
> Subject: [PATCH] esp4: Fix udpencap for local TCP packets.
>
> Locally generated TCP packets are usually cloned, so we
> do skb_cow_data() on this packets. After that we need to
> reload the pointer to the esp header. On udpencap this
> header has an offset to skb_transport_header, so take this
> offset into account.
It looks like locally generated TCP packets could avoid the
skb_cow_data(), if you were using skb_header_cloned() instead of
skb_cloned() ?
On 28 April 2017 at 03:13, Steffen Klassert
<[email protected]> wrote:
> On Thu, Apr 27, 2017 at 06:13:38PM -0400, Don Bowman wrote:
>> On 27 April 2017 at 04:42, Steffen Klassert <[email protected]>
>> wrote:
>> > On Wed, Apr 26, 2017 at 10:01:34PM -0700, Cong Wang wrote:
>> >> (Cc'ing netdev and IPSec maintainers)
>> >>
>> >> On Tue, Apr 25, 2017 at 6:08 PM, Don Bowman <[email protected]> wrote:
>>
<snip>
confirmed, with this patch in place that the tcp functions properly.