A livepatch transition may stall indefinitely when a kvm vCPU is heavily
loaded. To the host, the vCPU task is a user thread which is spending a
very long time in the ioctl(KVM_RUN) syscall. During livepatch
transition, set_notify_signal() will be called on such tasks to
interrupt the syscall so that the task can be transitioned. This
interrupts guest execution, but when xfer_to_guest_mode_work() sees that
TIF_NOTIFY_SIGNAL is set but not TIF_SIGPENDING it concludes that an
exit to user mode is unnecessary, and guest execution is resumed without
transitioning the task for the livepatch.
This handling of TIF_NOTIFY_SIGNAL is incorrect, as set_notify_signal()
is expected to break tasks out of interruptible kernel loops and cause
them to return to userspace. Change xfer_to_guest_mode_work() to handle
TIF_NOTIFY_SIGNAL the same as TIF_SIGPENDING, signaling to the vCPU run
loop that an exit to userpsace is needed. Any pending task_work will be
run when get_signal() is called from exit_to_user_mode_loop(), so there
is no longer any need to run task work from xfer_to_guest_mode_work().
Suggested-by: "Eric W. Biederman" <[email protected]>
Cc: Petr Mladek <[email protected]>
Signed-off-by: Seth Forshee <[email protected]>
---
kernel/entry/kvm.c | 6 ------
1 file changed, 6 deletions(-)
diff --git a/kernel/entry/kvm.c b/kernel/entry/kvm.c
index 9d09f489b60e..2e0f75bcb7fd 100644
--- a/kernel/entry/kvm.c
+++ b/kernel/entry/kvm.c
@@ -9,12 +9,6 @@ static int xfer_to_guest_mode_work(struct kvm_vcpu *vcpu, unsigned long ti_work)
int ret;
if (ti_work & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL)) {
- clear_notify_signal();
- if (task_work_pending(current))
- task_work_run();
- }
-
- if (ti_work & _TIF_SIGPENDING) {
kvm_handle_signal_exit(vcpu);
return -EINTR;
}
--
2.32.0
On Wed 2022-05-04 13:08:40, Seth Forshee wrote:
> A livepatch transition may stall indefinitely when a kvm vCPU is heavily
> loaded. To the host, the vCPU task is a user thread which is spending a
> very long time in the ioctl(KVM_RUN) syscall. During livepatch
> transition, set_notify_signal() will be called on such tasks to
> interrupt the syscall so that the task can be transitioned. This
> interrupts guest execution, but when xfer_to_guest_mode_work() sees that
> TIF_NOTIFY_SIGNAL is set but not TIF_SIGPENDING it concludes that an
> exit to user mode is unnecessary, and guest execution is resumed without
> transitioning the task for the livepatch.
>
> This handling of TIF_NOTIFY_SIGNAL is incorrect, as set_notify_signal()
> is expected to break tasks out of interruptible kernel loops and cause
> them to return to userspace. Change xfer_to_guest_mode_work() to handle
> TIF_NOTIFY_SIGNAL the same as TIF_SIGPENDING, signaling to the vCPU run
> loop that an exit to userpsace is needed. Any pending task_work will be
> run when get_signal() is called from exit_to_user_mode_loop(), so there
> is no longer any need to run task work from xfer_to_guest_mode_work().
>
> Suggested-by: "Eric W. Biederman" <[email protected]>
> Cc: Petr Mladek <[email protected]>
> Signed-off-by: Seth Forshee <[email protected]>
Acked-by: Petr Mladek <[email protected]>
Thanks Seth for discovering the problem.
Thanks everyone who helped to find the right solution.
Best Regards.
Petr
On 5/4/22 12:08 PM, Seth Forshee wrote:
> A livepatch transition may stall indefinitely when a kvm vCPU is heavily
> loaded. To the host, the vCPU task is a user thread which is spending a
> very long time in the ioctl(KVM_RUN) syscall. During livepatch
> transition, set_notify_signal() will be called on such tasks to
> interrupt the syscall so that the task can be transitioned. This
> interrupts guest execution, but when xfer_to_guest_mode_work() sees that
> TIF_NOTIFY_SIGNAL is set but not TIF_SIGPENDING it concludes that an
> exit to user mode is unnecessary, and guest execution is resumed without
> transitioning the task for the livepatch.
>
> This handling of TIF_NOTIFY_SIGNAL is incorrect, as set_notify_signal()
> is expected to break tasks out of interruptible kernel loops and cause
> them to return to userspace. Change xfer_to_guest_mode_work() to handle
> TIF_NOTIFY_SIGNAL the same as TIF_SIGPENDING, signaling to the vCPU run
> loop that an exit to userpsace is needed. Any pending task_work will be
> run when get_signal() is called from exit_to_user_mode_loop(), so there
> is no longer any need to run task work from xfer_to_guest_mode_work().
Reviewed-by: Jens Axboe <[email protected]>
--
Jens Axboe
Seth Forshee <[email protected]> writes:
> A livepatch transition may stall indefinitely when a kvm vCPU is heavily
> loaded. To the host, the vCPU task is a user thread which is spending a
> very long time in the ioctl(KVM_RUN) syscall. During livepatch
> transition, set_notify_signal() will be called on such tasks to
> interrupt the syscall so that the task can be transitioned. This
> interrupts guest execution, but when xfer_to_guest_mode_work() sees that
> TIF_NOTIFY_SIGNAL is set but not TIF_SIGPENDING it concludes that an
> exit to user mode is unnecessary, and guest execution is resumed without
> transitioning the task for the livepatch.
>
> This handling of TIF_NOTIFY_SIGNAL is incorrect, as set_notify_signal()
> is expected to break tasks out of interruptible kernel loops and cause
> them to return to userspace. Change xfer_to_guest_mode_work() to handle
> TIF_NOTIFY_SIGNAL the same as TIF_SIGPENDING, signaling to the vCPU run
> loop that an exit to userpsace is needed. Any pending task_work will be
> run when get_signal() is called from exit_to_user_mode_loop(), so there
> is no longer any need to run task work from xfer_to_guest_mode_work().
>
> Suggested-by: "Eric W. Biederman" <[email protected]>
> Cc: Petr Mladek <[email protected]>
> Signed-off-by: Seth Forshee <[email protected]>
Acked-by: "Eric W. Biederman" <[email protected]>
> ---
> kernel/entry/kvm.c | 6 ------
> 1 file changed, 6 deletions(-)
>
> diff --git a/kernel/entry/kvm.c b/kernel/entry/kvm.c
> index 9d09f489b60e..2e0f75bcb7fd 100644
> --- a/kernel/entry/kvm.c
> +++ b/kernel/entry/kvm.c
> @@ -9,12 +9,6 @@ static int xfer_to_guest_mode_work(struct kvm_vcpu *vcpu, unsigned long ti_work)
> int ret;
>
> if (ti_work & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL)) {
> - clear_notify_signal();
> - if (task_work_pending(current))
> - task_work_run();
> - }
> -
> - if (ti_work & _TIF_SIGPENDING) {
> kvm_handle_signal_exit(vcpu);
> return -EINTR;
> }
On Wed, May 04, 2022 at 01:08:40PM -0500, Seth Forshee wrote:
> A livepatch transition may stall indefinitely when a kvm vCPU is heavily
> loaded. To the host, the vCPU task is a user thread which is spending a
> very long time in the ioctl(KVM_RUN) syscall. During livepatch
> transition, set_notify_signal() will be called on such tasks to
> interrupt the syscall so that the task can be transitioned. This
> interrupts guest execution, but when xfer_to_guest_mode_work() sees that
> TIF_NOTIFY_SIGNAL is set but not TIF_SIGPENDING it concludes that an
> exit to user mode is unnecessary, and guest execution is resumed without
> transitioning the task for the livepatch.
>
> This handling of TIF_NOTIFY_SIGNAL is incorrect, as set_notify_signal()
> is expected to break tasks out of interruptible kernel loops and cause
> them to return to userspace. Change xfer_to_guest_mode_work() to handle
> TIF_NOTIFY_SIGNAL the same as TIF_SIGPENDING, signaling to the vCPU run
> loop that an exit to userpsace is needed. Any pending task_work will be
> run when get_signal() is called from exit_to_user_mode_loop(), so there
> is no longer any need to run task work from xfer_to_guest_mode_work().
>
> Suggested-by: "Eric W. Biederman" <[email protected]>
> Cc: Petr Mladek <[email protected]>
> Signed-off-by: Seth Forshee <[email protected]>
Friendly reminder as it seems like this patch may have been forgotten.
Thanks,
Seth
> ---
> kernel/entry/kvm.c | 6 ------
> 1 file changed, 6 deletions(-)
>
> diff --git a/kernel/entry/kvm.c b/kernel/entry/kvm.c
> index 9d09f489b60e..2e0f75bcb7fd 100644
> --- a/kernel/entry/kvm.c
> +++ b/kernel/entry/kvm.c
> @@ -9,12 +9,6 @@ static int xfer_to_guest_mode_work(struct kvm_vcpu *vcpu, unsigned long ti_work)
> int ret;
>
> if (ti_work & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL)) {
> - clear_notify_signal();
> - if (task_work_pending(current))
> - task_work_run();
> - }
> -
> - if (ti_work & _TIF_SIGPENDING) {
> kvm_handle_signal_exit(vcpu);
> return -EINTR;
> }
> --
> 2.32.0
>
On 6/6/22 16:13, Seth Forshee wrote:
> On Wed, May 04, 2022 at 01:08:40PM -0500, Seth Forshee wrote:
>> A livepatch transition may stall indefinitely when a kvm vCPU is heavily
>> loaded. To the host, the vCPU task is a user thread which is spending a
>> very long time in the ioctl(KVM_RUN) syscall. During livepatch
>> transition, set_notify_signal() will be called on such tasks to
>> interrupt the syscall so that the task can be transitioned. This
>> interrupts guest execution, but when xfer_to_guest_mode_work() sees that
>> TIF_NOTIFY_SIGNAL is set but not TIF_SIGPENDING it concludes that an
>> exit to user mode is unnecessary, and guest execution is resumed without
>> transitioning the task for the livepatch.
>>
>> This handling of TIF_NOTIFY_SIGNAL is incorrect, as set_notify_signal()
>> is expected to break tasks out of interruptible kernel loops and cause
>> them to return to userspace. Change xfer_to_guest_mode_work() to handle
>> TIF_NOTIFY_SIGNAL the same as TIF_SIGPENDING, signaling to the vCPU run
>> loop that an exit to userpsace is needed. Any pending task_work will be
>> run when get_signal() is called from exit_to_user_mode_loop(), so there
>> is no longer any need to run task work from xfer_to_guest_mode_work().
>>
>> Suggested-by: "Eric W. Biederman" <[email protected]>
>> Cc: Petr Mladek <[email protected]>
>> Signed-off-by: Seth Forshee <[email protected]>
>
> Friendly reminder as it seems like this patch may have been forgotten.
Probably AB-BA maintainer deadlock. I have queued it now.
Paolo
> Thanks,
> Seth
>
>> ---
>> kernel/entry/kvm.c | 6 ------
>> 1 file changed, 6 deletions(-)
>>
>> diff --git a/kernel/entry/kvm.c b/kernel/entry/kvm.c
>> index 9d09f489b60e..2e0f75bcb7fd 100644
>> --- a/kernel/entry/kvm.c
>> +++ b/kernel/entry/kvm.c
>> @@ -9,12 +9,6 @@ static int xfer_to_guest_mode_work(struct kvm_vcpu *vcpu, unsigned long ti_work)
>> int ret;
>>
>> if (ti_work & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL)) {
>> - clear_notify_signal();
>> - if (task_work_pending(current))
>> - task_work_run();
>> - }
>> -
>> - if (ti_work & _TIF_SIGPENDING) {
>> kvm_handle_signal_exit(vcpu);
>> return -EINTR;
>> }
>> --
>> 2.32.0
>>
>