When the child reaper of a pid namespace exits, it invokes
zap_pid_ns_processes() to send SIGKILL to all processes in the
namespace and wait them exit. But one of the child processes get
stuck and its call trace like this:
[<0>] request_wait_answer+0x132/0x210 [fuse]
[<0>] fuse_simple_request+0x1a8/0x2e0 [fuse]
[<0>] fuse_flush+0x193/0x1d0 [fuse]
[<0>] filp_close+0x34/0x70
[<0>] close_fd+0x38/0x50
[<0>] __x64_sys_close+0x12/0x40
[<0>] do_syscall_64+0x59/0xc0
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xae
The flags of fuse request is (FR_ISREPLY | FR_FORCE | FR_WAITING
| FR_INTERRUPTED | FR_SENT). For interrupt requests, fuse_dev_do_write()
doesn't invoke fuse_request_end() to wake the client thread, so it will
get stuck forever and the child reaper can't exit.
In order to write reply to the client thread and make it exit the
namespace, so do not generate interrupt requests for fatal signals.
Signed-off-by: Haifeng Xu <[email protected]>
---
fs/fuse/dev.c | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index 9eb191b5c4de..5fb830ad860d 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -374,11 +374,14 @@ static void request_wait_answer(struct fuse_req *req)
if (!err)
return;
- set_bit(FR_INTERRUPTED, &req->flags);
- /* matches barrier in fuse_dev_do_read() */
- smp_mb__after_atomic();
- if (test_bit(FR_SENT, &req->flags))
- queue_interrupt(req);
+ /* Any signal except fatal can generate an interrupt request */
+ if (!__fatal_signal_pending(current)) {
+ set_bit(FR_INTERRUPTED, &req->flags);
+ /* matches barrier in fuse_dev_do_read() */
+ smp_mb__after_atomic();
+ if (test_bit(FR_SENT, &req->flags))
+ queue_interrupt(req);
+ }
}
if (!test_bit(FR_FORCE, &req->flags)) {
--
2.25.1
On Thu, 13 Jun 2024 at 06:02, Haifeng Xu <[email protected]> wrote:
>
> When the child reaper of a pid namespace exits, it invokes
> zap_pid_ns_processes() to send SIGKILL to all processes in the
> namespace and wait them exit. But one of the child processes get
> stuck and its call trace like this:
>
> [<0>] request_wait_answer+0x132/0x210 [fuse]
> [<0>] fuse_simple_request+0x1a8/0x2e0 [fuse]
> [<0>] fuse_flush+0x193/0x1d0 [fuse]
> [<0>] filp_close+0x34/0x70
> [<0>] close_fd+0x38/0x50
> [<0>] __x64_sys_close+0x12/0x40
> [<0>] do_syscall_64+0x59/0xc0
> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xae
Which process is this?
In my experience such lockups are caused by badly written fuse servers.
> The flags of fuse request is (FR_ISREPLY | FR_FORCE | FR_WAITING
> | FR_INTERRUPTED | FR_SENT). For interrupt requests, fuse_dev_do_write()
> doesn't invoke fuse_request_end() to wake the client thread, so it will
> get stuck forever and the child reaper can't exit.
>
> In order to write reply to the client thread and make it exit the
> namespace, so do not generate interrupt requests for fatal signals.
Interrupt request must be generated for all signals. Not generating
them for SIGKILL would break existing filesystems.
Thanks,
Miklos
On 2024/6/13 15:55, Miklos Szeredi wrote:
> On Thu, 13 Jun 2024 at 06:02, Haifeng Xu <[email protected]> wrote:
>>
>> When the child reaper of a pid namespace exits, it invokes
>> zap_pid_ns_processes() to send SIGKILL to all processes in the
>> namespace and wait them exit. But one of the child processes get
>> stuck and its call trace like this:
>>
>> [<0>] request_wait_answer+0x132/0x210 [fuse]
>> [<0>] fuse_simple_request+0x1a8/0x2e0 [fuse]
>> [<0>] fuse_flush+0x193/0x1d0 [fuse]
>> [<0>] filp_close+0x34/0x70
>> [<0>] close_fd+0x38/0x50
>> [<0>] __x64_sys_close+0x12/0x40
>> [<0>] do_syscall_64+0x59/0xc0
>> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xae
>
> Which process is this?
The client process is one of the processes in container. And the server process is lxcfs
which belongs to global namespace.
>
> In my experience such lockups are caused by badly written fuse servers.
In this case, if the interrupt request is processed before the original request is processed,
for the arriving original request, fuse_session_process_buf_int（）which used in libfuse
invokes check_interrupt() can find the interrupt request and mark the req as interrupted,
so the server thread which invokes fuse_lib_flush() will sleep for some time and eventually
send reply to client without setting FUSE_INT_REQ_BIT in unique.
So why the client doesn't get woken up?
>
>> The flags of fuse request is (FR_ISREPLY | FR_FORCE | FR_WAITING
>> | FR_INTERRUPTED | FR_SENT). For interrupt requests, fuse_dev_do_write()
>> doesn't invoke fuse_request_end() to wake the client thread, so it will
>> get stuck forever and the child reaper can't exit.
>>
>> In order to write reply to the client thread and make it exit the
>> namespace, so do not generate interrupt requests for fatal signals.
>
> Interrupt request must be generated for all signals. Not generating
> them for SIGKILL would break existing filesystems.
>
> Thanks,
> Miklos
On Thu, 13 Jun 2024 at 12:44, Haifeng Xu <[email protected]> wrote:
> So why the client doesn't get woken up?
Need to find out what the server (lxcfs) is doing. Can you do a
strace of lxcfs to see the communication on the fuse device?
Thanks,
Miklos
On 6/13/24 09:55, Miklos Szeredi wrote:
> On Thu, 13 Jun 2024 at 06:02, Haifeng Xu <[email protected]> wrote:
>>
>> When the child reaper of a pid namespace exits, it invokes
>> zap_pid_ns_processes() to send SIGKILL to all processes in the
>> namespace and wait them exit. But one of the child processes get
>> stuck and its call trace like this:
>>
>> [<0>] request_wait_answer+0x132/0x210 [fuse]
>> [<0>] fuse_simple_request+0x1a8/0x2e0 [fuse]
>> [<0>] fuse_flush+0x193/0x1d0 [fuse]
>> [<0>] filp_close+0x34/0x70
>> [<0>] close_fd+0x38/0x50
>> [<0>] __x64_sys_close+0x12/0x40
>> [<0>] do_syscall_64+0x59/0xc0
>> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xae
>
> Which process is this?
>
> In my experience such lockups are caused by badly written fuse servers.
Btw, if libfuse should be used, it now supports disabling interrupts
https://github.com/libfuse/libfuse/commit/cef8c8b249023fb8129ae791e0998cbca771f96a
Cheers,
Bernd
On 2024/6/14 18:01, Miklos Szeredi wrote:
> On Thu, 13 Jun 2024 at 12:44, Haifeng Xu <[email protected]> wrote:
>
>> So why the client doesn't get woken up?
>
> Need to find out what the server (lxcfs) is doing. Can you do a
> strace of lxcfs to see the communication on the fuse device?
ok, I use strace to track one of the server threads. The output
can be seen in attachment.
FD: 6 DIR /run/lxcfs/controllers/sys/fs/cgroup/
FD: 7 CHR /dev/fuse
>
> Thanks,
> Miklos
On 2024/6/14 18:31, Bernd Schubert wrote:
>
>
> On 6/13/24 09:55, Miklos Szeredi wrote:
>> On Thu, 13 Jun 2024 at 06:02, Haifeng Xu <[email protected]> wrote:
>>>
>>> When the child reaper of a pid namespace exits, it invokes
>>> zap_pid_ns_processes() to send SIGKILL to all processes in the
>>> namespace and wait them exit. But one of the child processes get
>>> stuck and its call trace like this:
>>>
>>> [<0>] request_wait_answer+0x132/0x210 [fuse]
>>> [<0>] fuse_simple_request+0x1a8/0x2e0 [fuse]
>>> [<0>] fuse_flush+0x193/0x1d0 [fuse]
>>> [<0>] filp_close+0x34/0x70
>>> [<0>] close_fd+0x38/0x50
>>> [<0>] __x64_sys_close+0x12/0x40
>>> [<0>] do_syscall_64+0x59/0xc0
>>> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xae
>>
>> Which process is this?
>>
>> In my experience such lockups are caused by badly written fuse servers.
>
>
> Btw, if libfuse should be used, it now supports disabling interrupts
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_libfuse_libfuse_commit_cef8c8b249023fb8129ae791e0998cbca771f96a&d=DwICaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=3uoFsejk1jN2oga47MZfph01lLGODc93n4Zqe7b0NRk&m=tF8m9nGSWX4QZ_jfhLnEAE5bia1XekX0a_EojRtTFs2ZqfhKCrhY4cwO6K9UrW8x&s=X5dxXdmPhGVwknoinaLMbPYdHeOnrfVdOXs8HPCLT0A&e=
>
>
OK, Thank you for your reminding.
>
> Cheers,
> Bernd