Hi,
We encounter a performance problem in a large scale computer
cluster, which needs to handle a lot of incoming concurrent TCP
connection requests.
The top shows the kernel is most cpu hog, the testing is simple,
just a accept() -> epoll_ctl(ADD) loop, the ratio of cpu util sys% to
si% is about 2:5.
I also asked some experienced webserver/proxy developers in my team
for suggestions, it seem that behavior of many userland programs already
called accept() multiple times after it is waked up by
epoll_wait(). And the common action is adding the fd that accept()
return into epoll interface by epoll_ctl() syscall then.
Therefore, I think that we'd better to introduce to batch variants of
accept() and epoll_ctl() syscall, just like sendmmsg() or recvmmsg().
For accept(), we may need a new syscall, it may like this,
struct accept_result {
int fd;
struct sockaddr addr;
socklen_t addr_len;
};
int maccept4(int fd, int flags, int nr_accept_result, struct
accept_result *results);
For epoll_ctl(), there are two means to extend it, I prefer to extend
current interface instead of introduce to new syscall. We may introduce
to a new flag EPOLL_CTL_BATCH. If userland call epoll_ctl() with this
flag set, the meaning of last two arguments of epoll_ctl() change, .e.g:
struct batch_epoll_event batch_event[] = {
{
.fd = a_newsock_fd;
.epoll_event = { ... };
},
...
};
ret = epoll_ctl(fd, EPOLL_CTL_ADD|EPOLL_CTL_BATCH, nr_batch_events,
batch_events);
Thanks.
Yu
On Fri, Jun 15, 2012 at 12:13 PM, Li Yu <[email protected]> wrote:
> Hi,
>
> ?We encounter a performance problem in a large scale computer
> cluster, which needs to handle a lot of incoming concurrent TCP
> connection requests.
>
> ?The top shows the kernel is most cpu hog, the testing is simple,
> just a accept() -> epoll_ctl(ADD) loop, the ratio of cpu util sys% to
> si% is about 2:5.
>
> ?I also asked some experienced webserver/proxy developers in my team
> for suggestions, it seem that behavior of many userland programs already
> called accept() multiple times after it is waked up by
> epoll_wait(). And the common action is adding the fd that accept()
> return into epoll interface by epoll_ctl() syscall then.
>
> ?Therefore, I think that we'd better to introduce to batch variants of
> accept() and epoll_ctl() syscall, just like sendmmsg() or recvmmsg().
>
> ?For accept(), we may need a new syscall, it may like this,
>
> ?struct accept_result {
> ? ? ?int fd;
> ? ? ?struct sockaddr addr;
> ? ? ?socklen_t addr_len;
> ?};
>
> ?int maccept4(int fd, int flags, int nr_accept_result, struct
> accept_result *results);
>
> ?For epoll_ctl(), there are two means to extend it, I prefer to extend
> current interface instead of introduce to new syscall. We may introduce
> to a new flag EPOLL_CTL_BATCH. If userland call epoll_ctl() with this
> flag set, the meaning of last two arguments of epoll_ctl() change, .e.g:
>
> ?struct batch_epoll_event batch_event[] = {
> ? ? ? ? {
> ? ? ? ? ? ? ?.fd = a_newsock_fd;
> ? ? ? ? ? ? ?.epoll_event = { ... };
> ? ? ? ? },
> ? ? ? ? ...
> ?};
>
> ?ret = epoll_ctl(fd, EPOLL_CTL_ADD|EPOLL_CTL_BATCH, nr_batch_events,
> batch_events);
>
I think it is good idea. Would you please implement a prototype and
give some numbers? This kind of data may help selling this idea.
Thanks.
--
Regards,
Changli Gao([email protected])
于 2012年06月15日 12:29, Changli Gao 写道:
> On Fri, Jun 15, 2012 at 12:13 PM, Li Yu<[email protected]> wrote:
>> Hi,
>>
>> We encounter a performance problem in a large scale computer
>> cluster, which needs to handle a lot of incoming concurrent TCP
>> connection requests.
>>
>> The top shows the kernel is most cpu hog, the testing is simple,
>> just a accept() -> epoll_ctl(ADD) loop, the ratio of cpu util sys% to
>> si% is about 2:5.
>>
>> I also asked some experienced webserver/proxy developers in my team
>> for suggestions, it seem that behavior of many userland programs already
>> called accept() multiple times after it is waked up by
>> epoll_wait(). And the common action is adding the fd that accept()
>> return into epoll interface by epoll_ctl() syscall then.
>>
>> Therefore, I think that we'd better to introduce to batch variants of
>> accept() and epoll_ctl() syscall, just like sendmmsg() or recvmmsg().
>>
>> For accept(), we may need a new syscall, it may like this,
>>
>> struct accept_result {
>> int fd;
>> struct sockaddr addr;
>> socklen_t addr_len;
>> };
>>
>> int maccept4(int fd, int flags, int nr_accept_result, struct
>> accept_result *results);
>>
>> For epoll_ctl(), there are two means to extend it, I prefer to extend
>> current interface instead of introduce to new syscall. We may introduce
>> to a new flag EPOLL_CTL_BATCH. If userland call epoll_ctl() with this
>> flag set, the meaning of last two arguments of epoll_ctl() change, .e.g:
>>
>> struct batch_epoll_event batch_event[] = {
>> {
>> .fd = a_newsock_fd;
>> .epoll_event = { ... };
>> },
>> ...
>> };
>>
>> ret = epoll_ctl(fd, EPOLL_CTL_ADD|EPOLL_CTL_BATCH, nr_batch_events,
>> batch_events);
>>
>
> I think it is good idea. Would you please implement a prototype and
> give some numbers? This kind of data may help selling this idea.
> Thanks.
>
Of course, I think that implementing them should not be a hard work :)
Em. I really do not know whether it is necessary to introduce to a new
syscall here. An alternative solution to add new socket option to handle
such batch requirement, so applications also can detect if kernel has
this extended ability with a easy getsockopt() call.
Any way, I am going to try to write a prototype first.
Thanks
Yu
> We encounter a performance problem in a large scale computer
> cluster, which needs to handle a lot of incoming concurrent TCP
> connection requests.
>
> The top shows the kernel is most cpu hog, the testing is simple,
> just a accept() -> epoll_ctl(ADD) loop, the ratio of cpu util sys% to
> si% is about 2:5.
>
> I also asked some experienced webserver/proxy developers in my team
> for suggestions, it seem that behavior of many userland
> programs already
> called accept() multiple times after it is waked up by
> epoll_wait(). And the common action is adding the fd that accept()
> return into epoll interface by epoll_ctl() syscall then.
>
> Therefore, I think that we'd better to introduce to batch
> variants of
> accept() and epoll_ctl() syscall, just like sendmmsg() or recvmmsg().
...
Having seen the support added to NetBSD for sendmmsg() and
recvmmsg() (and I'm told the linux code is much the same),
I'm surprised that just cutting out a system call entry/exit
and fd lookup is significant above the rest of the costs
involved in sending a message (which I presume is UDP here).
I'd be even more surprised if it is significant for an
incoming connection.
David
On Fri, 2012-06-15 at 13:37 +0800, Li Yu wrote:
> Of course, I think that implementing them should not be a hard work :)
>
> Em. I really do not know whether it is necessary to introduce to a new
> syscall here. An alternative solution to add new socket option to handle
> such batch requirement, so applications also can detect if kernel has
> this extended ability with a easy getsockopt() call.
>
> Any way, I am going to try to write a prototype first.
Before that, could you post the result of "perf top", or "perf
record ...;perf report"
> The top shows the kernel is most cpu hog, the testing is simple,
> just a accept() -> epoll_ctl(ADD) loop, the ratio of cpu util sys% to
> si% is about 2:5.
This ratio is not meaningful, if we dont know where time is spent.
I doubt epoll_ctl(ADD) is a problem here...
If it is, batching the fds wont speed the thing anyway...
I believe accept() is the problem here, because it contends with the
softirq processing the tcp session handshake.
Eric Dumazet <[email protected]> writes:
>
> I believe accept() is the problem here, because it contends with the
> softirq processing the tcp session handshake.
The MOSBENCH people some time ago did a per CPU accept queue. This is
probably overkill, but there are clearly some scaling problems here
with enough cores.
-Andi
--
[email protected] -- Speaking for myself only