Let multishot support multishot mode, currently only add accept as its
first comsumer.
theoretical analysis:
1) when connections come in fast
- singleshot:
add accept sqe(userpsace) --> accept inline
^ |
|-----------------|
- multishot:
add accept sqe(userspace) --> accept inline
^ |
|--*--|
we do accept repeatedly in * place until get EAGAIN
2) when connections come in at a low pressure
similar thing like 1), we reduce a lot of userspace-kernel context
switch and useless vfs_poll()
tests:
Did some tests, which goes in this way:
server client(multiple)
accept connect
read write
write read
close close
Basically, raise up a number of clients(on same machine with server) to
connect to the server, and then write some data to it, the server will
write those data back to the client after it receives them, and then
close the connection after write return. Then the client will read the
data and then close the connection. Here I test 10000 clients connect
one server, data size 128 bytes. And each client has a go routine for
it, so they come to the server in short time.
test 20 times before/after this patchset, time spent:(unit cycle, which
is the return value of clock())
before:
1930136+1940725+1907981+1947601+1923812+1928226+1911087+1905897+1941075
+1934374+1906614+1912504+1949110+1908790+1909951+1941672+1969525+1934984
+1934226+1914385)/20.0 = 1927633.75
after:
1858905+1917104+1895455+1963963+1892706+1889208+1874175+1904753+1874112
+1874985+1882706+1884642+1864694+1906508+1916150+1924250+1869060+1889506
+1871324+1940803)/20.0 = 1894750.45
(1927633.75 - 1894750.45) / 1927633.75 = 1.65%
A liburing test is here:
https://github.com/HowHsu/liburing/blob/multishot_accept/test/accept.c
v1->v2:
- re-implement it against the reworked poll code
v2->v3:
- fold in code tweak and clean from Jens
- use io_issue_sqe rather than io_queue_sqe, since the former one
return the internal error back which makes more sense
- remove io_poll_clean() and its friends since they are not needed
Hao Xu (4):
io_uring: add IORING_ACCEPT_MULTISHOT for accept
io_uring: add REQ_F_APOLL_MULTISHOT for requests
io_uring: let fast poll support multishot
io_uring: implement multishot mode for accept
fs/io_uring.c | 94 +++++++++++++++++++++++++++--------
include/uapi/linux/io_uring.h | 5 ++
2 files changed, 79 insertions(+), 20 deletions(-)
base-commit: 0a194603ba7ee67b4e39ec0ee5cda70a356ea618
--
2.36.0
On 5/7/22 8:06 AM, Hao Xu wrote:
> Let multishot support multishot mode, currently only add accept as its
> first comsumer.
consumer
> theoretical analysis:
> 1) when connections come in fast
> - singleshot:
> add accept sqe(userpsace) --> accept inline
userspace
> ^ |
> |-----------------|
> - multishot:
> add accept sqe(userspace) --> accept inline
> ^ |
> |--*--|
>
> we do accept repeatedly in * place until get EAGAIN
>
> 2) when connections come in at a low pressure
> similar thing like 1), we reduce a lot of userspace-kernel context
> switch and useless vfs_poll()
Overall this looks better than v2 for sure, just some minor tweaks
needed I believe.
But we still need to consider direct accept with multishot... Should
probably be an add-on patch as I think it'd get a bit more complicated
if we need to be able to cheaply find an available free fixed fd slot.
I'll try and play with that.
--
Jens Axboe
On 5/7/22 11:21 AM, Hao Xu wrote:
> 在 2022/5/8 上午12:11, Jens Axboe 写道:
>> On 5/7/22 10:05 AM, Hao Xu wrote:
>>>> But we still need to consider direct accept with multishot... Should
>>>> probably be an add-on patch as I think it'd get a bit more complicated
>>>> if we need to be able to cheaply find an available free fixed fd slot.
>>>> I'll try and play with that.
>>>
>>> I'm tending to use a new mail account to send v4 rather than the gmail
>>> account since the git issue seems to be network related.
>>> I'll also think about the fixed fd problem.
>>
>> Two basic attached patches that attempt do just alloc a fixed file
>> descriptor for this case. Not tested at all... We return the fixed file
>> slot in this case since we have to, to let the application know what was
>> picked. I kind of wish we'd done that with direct open/accept to begin
>> with anyway, a bit annoying that fixed vs normal open/accept behave
>> differently.
>>
>> Anyway, something to play with, and I'm sure it can be made better.
>>
> Thanks. I tried to fix the mail account issue, still unclear what is
> wrong, and too late at my timezone now, I'll try to send v4 tomorrow
No worries. IN the meantime, I played with allocated direct descriptors
yesterday and implemented them for openat/openat2/accept:
https://git.kernel.dk/cgit/linux-block/log/?h=fastpoll-mshot
It's independent of multishot accept in the sense that you can use it
without that, but multishot accept requires it with fixed files.
--
Jens Axboe
在 2022/5/7 下午11:28, Jens Axboe 写道:
> On 5/7/22 8:06 AM, Hao Xu wrote:
>> Let multishot support multishot mode, currently only add accept as its
>> first comsumer.
>
> consumer
>
>> theoretical analysis:
>> 1) when connections come in fast
>> - singleshot:
>> add accept sqe(userpsace) --> accept inline
>
> userspace
>
>> ^ |
>> |-----------------|
>> - multishot:
>> add accept sqe(userspace) --> accept inline
>> ^ |
>> |--*--|
>>
>> we do accept repeatedly in * place until get EAGAIN
>>
>> 2) when connections come in at a low pressure
>> similar thing like 1), we reduce a lot of userspace-kernel context
>> switch and useless vfs_poll()
>
> Overall this looks better than v2 for sure, just some minor tweaks
> needed I believe.
>
> But we still need to consider direct accept with multishot... Should
> probably be an add-on patch as I think it'd get a bit more complicated
> if we need to be able to cheaply find an available free fixed fd slot.
> I'll try and play with that.
I'm tending to use a new mail account to send v4 rather than the gmail
account since the git issue seems to be network related.
I'll also think about the fixed fd problem.
Thanks,
Hao
>
Something wrong with my git sendemail, trying to fix it. Please ignore
this...
在 2022/5/7 下午8:38, Hao Xu 写道:
> Let multishot support multishot mode, currently only add accept as its
> first comsumer.
> theoretical analysis:
> 1) when connections come in fast
> - singleshot:
> add accept sqe(userpsace) --> accept inline
> ^ |
> |-----------------|
> - multishot:
> add accept sqe(userspace) --> accept inline
> ^ |
> |--*--|
>
> we do accept repeatedly in * place until get EAGAIN
>
> 2) when connections come in at a low pressure
> similar thing like 1), we reduce a lot of userspace-kernel context
> switch and useless vfs_poll()
>
>
> tests:
> Did some tests, which goes in this way:
>
> server client(multiple)
> accept connect
> read write
> write read
> close close
>
> Basically, raise up a number of clients(on same machine with server) to
> connect to the server, and then write some data to it, the server will
> write those data back to the client after it receives them, and then
> close the connection after write return. Then the client will read the
> data and then close the connection. Here I test 10000 clients connect
> one server, data size 128 bytes. And each client has a go routine for
> it, so they come to the server in short time.
> test 20 times before/after this patchset, time spent:(unit cycle, which
> is the return value of clock())
> before:
> 1930136+1940725+1907981+1947601+1923812+1928226+1911087+1905897+1941075
> +1934374+1906614+1912504+1949110+1908790+1909951+1941672+1969525+1934984
> +1934226+1914385)/20.0 = 1927633.75
> after:
> 1858905+1917104+1895455+1963963+1892706+1889208+1874175+1904753+1874112
> +1874985+1882706+1884642+1864694+1906508+1916150+1924250+1869060+1889506
> +1871324+1940803)/20.0 = 1894750.45
>
> (1927633.75 - 1894750.45) / 1927633.75 = 1.65%
>
>
> A liburing test is here:
> https://github.com/HowHsu/liburing/blob/multishot_accept/test/accept.c
>
> v1->v2:
> - re-implement it against the reworked poll code
>
> v2->v3:
> - fold in code tweak and clean from Jens
> - use io_issue_sqe rather than io_queue_sqe, since the former one
> return the internal error back which makes more sense
> - remove io_poll_clean() and its friends since they are not needed
>
>
> Hao Xu (4):
> io_uring: add IORING_ACCEPT_MULTISHOT for accept
> io_uring: add REQ_F_APOLL_MULTISHOT for requests
> io_uring: let fast poll support multishot
> io_uring: implement multishot mode for accept
>
> fs/io_uring.c | 94 +++++++++++++++++++++++++++--------
> include/uapi/linux/io_uring.h | 5 ++
> 2 files changed, 79 insertions(+), 20 deletions(-)
>
>
> base-commit: 0a194603ba7ee67b4e39ec0ee5cda70a356ea618
On 5/7/22 10:05 AM, Hao Xu wrote:
>> But we still need to consider direct accept with multishot... Should
>> probably be an add-on patch as I think it'd get a bit more complicated
>> if we need to be able to cheaply find an available free fixed fd slot.
>> I'll try and play with that.
>
> I'm tending to use a new mail account to send v4 rather than the gmail
> account since the git issue seems to be network related.
> I'll also think about the fixed fd problem.
Two basic attached patches that attempt do just alloc a fixed file
descriptor for this case. Not tested at all... We return the fixed file
slot in this case since we have to, to let the application know what was
picked. I kind of wish we'd done that with direct open/accept to begin
with anyway, a bit annoying that fixed vs normal open/accept behave
differently.
Anyway, something to play with, and I'm sure it can be made better.
--
Jens Axboe
在 2022/5/8 上午12:11, Jens Axboe 写道:
> On 5/7/22 10:05 AM, Hao Xu wrote:
>>> But we still need to consider direct accept with multishot... Should
>>> probably be an add-on patch as I think it'd get a bit more complicated
>>> if we need to be able to cheaply find an available free fixed fd slot.
>>> I'll try and play with that.
>>
>> I'm tending to use a new mail account to send v4 rather than the gmail
>> account since the git issue seems to be network related.
>> I'll also think about the fixed fd problem.
>
> Two basic attached patches that attempt do just alloc a fixed file
> descriptor for this case. Not tested at all... We return the fixed file
> slot in this case since we have to, to let the application know what was
> picked. I kind of wish we'd done that with direct open/accept to begin
> with anyway, a bit annoying that fixed vs normal open/accept behave
> differently.
>
> Anyway, something to play with, and I'm sure it can be made better.
>
Thanks. I tried to fix the mail account issue, still unclear what is
wrong, and too late at my timezone now, I'll try to send v4 tomorrow