2015-07-17 09:00:32

by Toralf Förster

[permalink] [raw]
Subject: nf_conntrack: falling back to vmalloc.

I do run a server with a 64 bit hardened Gentoo Linux (kernel currently 4.0.8).
Around 12th of July it started to spew those messages into kern.log :

/var/log/kern.log:Jul 12 15:26:07 tor-relay kernel: [538360.650490] nf_conntrack: falling back to vmalloc.
/var/log/kern.log:Jul 12 15:26:07 tor-relay kernel: [538360.650615] nf_conntrack: falling back to vmalloc.
/var/log/kern.log:Jul 12 15:26:08 tor-relay kernel: [538361.673649] nf_conntrack: falling back to vmalloc.
/var/log/kern.log:Jul 12 15:26:08 tor-relay kernel: [538361.673786] nf_conntrack: falling back to vmalloc.

I read https://bugzilla.openvz.org/show_bug.cgi?id=3092 but a reboot did not help.
I got 1-2 thousends of those lines per day.

Any hints ?

--
Toralf, pgp key: 872AE508 0076E94E


2015-07-17 10:10:14

by Florian Westphal

[permalink] [raw]
Subject: Re: nf_conntrack: falling back to vmalloc.

Toralf F?rster <[email protected]> wrote:
> I do run a server with a 64 bit hardened Gentoo Linux (kernel currently 4.0.8).
> Around 12th of July it started to spew those messages into kern.log :
>
> /var/log/kern.log:Jul 12 15:26:07 tor-relay kernel: [538360.650490] nf_conntrack: falling back to vmalloc.
> /var/log/kern.log:Jul 12 15:26:07 tor-relay kernel: [538360.650615] nf_conntrack: falling back to vmalloc.
> /var/log/kern.log:Jul 12 15:26:08 tor-relay kernel: [538361.673649] nf_conntrack: falling back to vmalloc.
> /var/log/kern.log:Jul 12 15:26:08 tor-relay kernel: [538361.673786] nf_conntrack: falling back to vmalloc.
>
> I read https://bugzilla.openvz.org/show_bug.cgi?id=3092 but a reboot did not help.
> I got 1-2 thousends of those lines per day.

Most likely result of 88eab472ec21f01d3e36ff ("netfilter: conntrack:
adjust nf_conntrack_buckets default value".

Do you run containers?

This message can only be printed when a new network namespace is created
(or something is rmmod/modprobing nf_conntrack module all the time).

I wonder if this is caused by some program creating netns for
sandboxing?

Pablo, Patrick -- any idea on how to stop conntrack from becoming active
in a newly created netns automatically without breaking anything?

With upcoming per netns hooks, we might be able to delay registering
conntrack, defrag etc. until after a -m conntrack rule has been added.
Dou you think that could work?

For nft we could create an expression to configure conntrack explicitly
(inverse NOTRACK).

[ obviously we can also add that for xtables but that would break
setups if we suddently move to "you must ask for conntrack via
ruleset" model. ]

2015-07-17 14:34:24

by Toralf Förster

[permalink] [raw]
Subject: Re: nf_conntrack: falling back to vmalloc.

On 07/17/2015 12:10 PM, Florian Westphal wrote:
> Do you run containers?

No, I just run 4 Gentoo chroot images in parallel - but I do that since autumn last year.

Unfortunately I cannot see any correlation to an os/sw upgrade or a config change which could cause these messages nowadays to be appear.
Therefore I do wonder if some network related "traffic" causes this, b/c that box acts as a Tor exit relay too.


--
Toralf, pgp key: 872AE508 0076E94E

2015-07-17 15:02:13

by Florian Westphal

[permalink] [raw]
Subject: Re: nf_conntrack: falling back to vmalloc.

Toralf F?rster <[email protected]> wrote:
> On 07/17/2015 12:10 PM, Florian Westphal wrote:
> > Do you run containers?
>
> No, I just run 4 Gentoo chroot images in parallel - but I do that since autumn last year.
>
> Unfortunately I cannot see any correlation to an os/sw upgrade or a config change which could cause these messages nowadays to be appear.
> Therefore I do wonder if some network related "traffic" causes this, b/c that box acts as a Tor exit relay too.

No, I see no way how this would happen in response to network traffic.

A conntrack hash table is only allocated on network namespace creation, or when
a resize happens (someone changing /sys/module/nf_conntrack/parameters/hashsize).

2015-07-23 10:39:52

by Pablo Neira Ayuso

[permalink] [raw]
Subject: Re: nf_conntrack: falling back to vmalloc.

On Fri, Jul 17, 2015 at 12:10:09PM +0200, Florian Westphal wrote:
> Toralf F?rster <[email protected]> wrote:
> > I do run a server with a 64 bit hardened Gentoo Linux (kernel currently 4.0.8).
> > Around 12th of July it started to spew those messages into kern.log :
> >
> > /var/log/kern.log:Jul 12 15:26:07 tor-relay kernel: [538360.650490] nf_conntrack: falling back to vmalloc.
> > /var/log/kern.log:Jul 12 15:26:07 tor-relay kernel: [538360.650615] nf_conntrack: falling back to vmalloc.
> > /var/log/kern.log:Jul 12 15:26:08 tor-relay kernel: [538361.673649] nf_conntrack: falling back to vmalloc.
> > /var/log/kern.log:Jul 12 15:26:08 tor-relay kernel: [538361.673786] nf_conntrack: falling back to vmalloc.
> >
> > I read https://bugzilla.openvz.org/show_bug.cgi?id=3092 but a reboot did not help.
> > I got 1-2 thousends of those lines per day.
>
> Most likely result of 88eab472ec21f01d3e36ff ("netfilter: conntrack:
> adjust nf_conntrack_buckets default value".

I think we can get rid of that warning, it will most likely shown
every time after that patch.

[...]
> Pablo, Patrick -- any idea on how to stop conntrack from becoming active
> in a newly created netns automatically without breaking anything?
>
> With upcoming per netns hooks, we might be able to delay registering
> conntrack, defrag etc. until after a -m conntrack rule has been added.
> Dou you think that could work?
>
> For nft we could create an expression to configure conntrack explicitly
> (inverse NOTRACK).
>
> [ obviously we can also add that for xtables but that would break
> setups if we suddently move to "you must ask for conntrack via
> ruleset" model. ]

With netns in place, I think it's now clear that we need that this is
configurable at ruleset level that we've been discussing for a while.
But users will have to request this new behaviour to avoid breaking
existing setups...


Attachments:
(No filename) (1.85 kB)
x.patch (603.00 B)
Download all attachments