Message-ID: <a3bd6fe7-1775-6fdd-3a02-e779c4d4e498@oracle.com>
Date:   Wed, 22 Dec 2021 11:32:25 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.4.0
Subject: Re: [PATCH V6 01/10] Use copy_process in vhost layer
Content-Language: en-US
To:     "Eric W. Biederman" <ebiederm@xmission.com>
Cc:     geert@linux-m68k.org, vverma@digitalocean.com, hdanton@sina.com,
        hch@infradead.org, stefanha@redhat.com, jasowang@redhat.com,
        mst@redhat.com, sgarzare@redhat.com,
        virtualization@lists.linux-foundation.org,
        christian.brauner@ubuntu.com, axboe@kernel.dk,
        linux-kernel@vger.kernel.org
References: <20211129194707.5863-1-michael.christie@oracle.com>
 <87tuf79gni.fsf@email.froward.int.ebiederm.org>
 <a171238e-d731-1c22-af72-0f7faf7f4bea@oracle.com>
 <87tuf11oe6.fsf@email.froward.int.ebiederm.org>
From:   Mike Christie <michael.christie@oracle.com>
In-Reply-To: <87tuf11oe6.fsf@email.froward.int.ebiederm.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NGVtcWJPbTMxVWRjNXlSaFFIQlo1cjFjUG9hTURYR3VGSk9PTFIvaG5TczYr?=
 =?utf-8?B?b0pTNzhqWDdYSzNySW8wN1RZNkZVcHZBaG5xblJtUWVOOWJOR2NqRmZXR0d6?=
 =?utf-8?B?d3gwUWl1NGFCZUNERklaSy9oZVBVZVZyNjBwMjAybGJTL2lKNndZSTRPNWM5?=
 =?utf-8?B?d1pBUFRmdkRWZmVjK2t6YUlyVWgrZzVYeW4wWGFkaVRSZWZHZFU3YkFPeGtZ?=
 =?utf-8?B?WTRFVjRtTWdkVnhJcGMxcDRvcENacG1jUUdaYmdoaytydExiQlVkb2RHREE4?=
 =?utf-8?B?SWpMdGMvN3FBeWtNeDdlK2VwdVNqMU8rR0YvaFUxWW5iRGtVY3RxTnV3Qk9T?=
 =?utf-8?B?NmJuK2FKQkVaNU9GdmEzeXNkb0NBYU9EcDBSWEhaeUh3cVRhN0Fkdi9hb29U?=
 =?utf-8?B?YU5Fcmk2bWZpMy9qbjZTMG5EbjM4MHkvVFFJelV5L1ZWK0xqTE5QRklhRWll?=
 =?utf-8?B?UE9VQzNCWmxvYXJuSkEvbXJzbm8xUzQ2cTJPVjh6b014UU5hemZScHJ5ZXpm?=
 =?utf-8?B?cHhka0c1bklQdWdwNjhoZ25mWUNOMmZFdWpoUlJvK1U2NFVIVFBlaVYwNWFs?=
 =?utf-8?B?SnBIcktGc01lNjh3KzVaSkgyQjlvRkliRDI1NS9pRURBME5QTllKSjFhU3Fu?=
 =?utf-8?B?c1I1bE9kQzg3TEd4UkJlemkwV1FkenNudm1YQ0pSUmJad2FzWFlVL2ZtQVZ6?=
 =?utf-8?B?cXh4QW83YUs3U3dJdHVFNHY2R05uZURsOWNXMWFSM1lMZS9zYlN6dWhsRWJ2?=
 =?utf-8?B?MXdGYXBtR25QSFozaTc5bVd5SWg0dVZFa3FBdEpDaG4wWnRVZmM3aEpSdy8w?=
 =?utf-8?B?amRKUTFWa1dOMUxGc3dnU3EwT0UxS1ozMTJqNHI3VC9Id1R0enpWVDJZaTNS?=
 =?utf-8?B?ck1OOEJHdkUrL2NZYzhXdmo5VjYva3IwbUgzcWVDanl5L2pxeTdRZWQ5QzhV?=
 =?utf-8?B?ajczcVFRN0lyZVZYYzN4eTNhd3lwUjRPWVE3R1kxL21mV01yZWJUc3VkUVBy?=
 =?utf-8?B?TmJ5bW9RK2NvM0pTYm0vM2c5dHJDbHF5c3cyUlRQWElVUjVDWmk1ak82VU8y?=
 =?utf-8?B?WTI1RitwdldZaFFucjhDMkdacE5tTmJLbUxVZkR2L1ZMZVkxUTBPWUExbUVl?=
 =?utf-8?B?MFVTREIyRjc3VnhSYkVieUZmUzNmTDBzbWZtL1JXZHhIa0JySzJac081cThE?=
 =?utf-8?B?dVkyenRlb21vK0pHZW9acG5iQjFHYSs4aTNpUlZOQXhoaXBvajcxdk5CbVdX?=
 =?utf-8?B?eHBzNWo5b2RsQy9HaG9OUFhxQVMrMTFldmFXZTF3cHRQKy9RV2NkK1QzSkFM?=
 =?utf-8?B?TWR2L1NWTG9CS1YzUDlmQzN0aDlpSmtiZFN1dGdlMXYzeTUzRm00WVZONXg4?=
 =?utf-8?B?dUpzVnc1R0RoLzUzNDU2Ukd3eHFFb09MWHBoTWpudGJ5d09MRys2bVAvNURF?=
 =?utf-8?B?OVZKVjVEZUJMeFIzSUl4VmFiclNRTVluaUVCMGNxellEWWxPWGRxUlVwalpT?=
 =?utf-8?B?N0ZLZ0hHZ1FERTJUdy82bERHNktWWnV1V2RvUVZjMDRRN2VuOEQ2SFd3dlRT?=
 =?utf-8?B?RnozTDJIMnovajRaajNsdWdZUExnWjk4cCtBclhMTi96Y0FRRTR5Nlp5dUFv?=
 =?utf-8?B?ZzBHU1VEWWFEb1B5NmJyemt5TzRDWVRIZkVqczV6VU5ZajVQei9UYWhyRHY5?=
 =?utf-8?B?d0w1aThiWnN5a3VoVy9ESGh4VS9YcTMwT25OaU5mZi81Zjd6TXRpc01wcjNC?=
 =?utf-8?B?bFpINVp6bFNNK3k2V3AzbUNEMEdxdDF6SnBGZGtxRlh1WFh6Q1pCemM2Rlow?=
 =?utf-8?B?UjN0cnBrZmVERDlFU1JnME1iQkFJQzNsM2wrSXJENk5FVUxzNWE5d3g4N3hF?=
 =?utf-8?B?YlNHZFQvUi9RNkJYOTRiMm1WRWlDTmVmaTJobTRnVHRSN1pvMDlMZWFqWGlT?=
 =?utf-8?B?REdRWUlDWFpHTFZRcVFLT1NkRnJPRDRlaVFBRGJ5dVliR0xRbis0NjFaTHEw?=
 =?utf-8?B?MEhYclo4VEtPcTA1NXB1RitQS3VJWmRLRFJzR2NQN1F4eG5qQktObmdBUkQr?=
 =?utf-8?B?Z1dDSTN6RjZSbEMzcmlhMXo4M0YzNkN5STFQUGZtdXppdEcxN1BoSmlJRE9U?=
 =?utf-8?B?R0dvOTN3UVY2dzE5VFpKQjdVdE4xUXNzdFRUUkxoblI5eUc4NEp6UUo5OUNL?=
 =?utf-8?B?clBubE5USGUyMks2WUZOTitIK05WY21FZTB0VkNEcFVXWms3TTl0azlaQnlM?=
 =?utf-8?B?L1dqR01LZ1RQTnNXUjcvNTdNcXBRPT0=?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 62a631aa-10f3-4962-e295-08d9c57105e1
X-MS-Exchange-CrossTenant-AuthSource: DM5PR10MB1466.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Dec 2021 17:32:27.2830
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YfnTXgW5KJP4U3EPYoBfpC7t5zxq50jAtkvpod8NBTfjHwfHbMrpD/UCOPQ50kIvkiUlIBwleK2GsH7g3SJo/5bY3avb8rDGcHVA2CbKmpM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5262
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10206 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=706 malwarescore=0
 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 phishscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2110150000 definitions=main-2112220097
X-Proofpoint-ORIG-GUID: O1uDCKRqVt4bfH8n1kPepy8YNrRJbIdU
X-Proofpoint-GUID: O1uDCKRqVt4bfH8n1kPepy8YNrRJbIdU
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 12/21/21 6:20 PM, Eric W. Biederman wrote:
> michael.christie@oracle.com writes:
> 
>> On 12/17/21 1:26 PM, Eric W. Biederman wrote:
>>> Mike Christie <michael.christie@oracle.com> writes:
>>>
>>>> The following patches made over Linus's tree, allow the vhost layer to do
>>>> a copy_process on the thread that does the VHOST_SET_OWNER ioctl like how
>>>> io_uring does a copy_process against its userspace app. This allows the
>>>> vhost layer's worker threads to inherit cgroups, namespaces, address
>>>> space, etc and this worker thread will also be accounted for against that
>>>> owner/parent process's RLIMIT_NPROC limit.
>>>>
>>>> If you are not familiar with qemu and vhost here is more detailed
>>>> problem description:
>>>>
>>>> Qemu will create vhost devices in the kernel which perform network, SCSI,
>>>> etc IO and management operations from worker threads created by the
>>>> kthread API. Because the kthread API does a copy_process on the kthreadd
>>>> thread, the vhost layer has to use kthread_use_mm to access the Qemu
>>>> thread's memory and cgroup_attach_task_all to add itself to the Qemu
>>>> thread's cgroups.
>>>>
>>>> The problem with this approach is that we then have to add new functions/
>>>> args/functionality for every thing we want to inherit. I started doing
>>>> that here:
>>>>
>>>> https://urldefense.com/v3/__https://lkml.org/lkml/2021/6/23/1233__;!!ACWV5N9M2RV99hQ!eIaEe9V8mCgGU6vyvaWTKGi3Zlnz0rgk5Y-0nsBXRbsuVZsM8lYfHr8I8GRuoLYPYrOB$ 
>>>>
>>>> for the RLIMIT_NPROC check, but it seems it might be easier to just
>>>> inherit everything from the beginning, becuase I'd need to do something
>>>> like that patch several times.
>>>
>>> I read through the code and I don't see why you want to make these
>>> almost threads of a process not actually threads of that process
>>> (like the io_uring threads are).
>>>
>>> As a separate process there are many things that will continue to be
>>> disjoint.  If the thread changes cgroups for example your new process
>>> won't follow.
>>>
>>> If you want them to be actually processes with an lifetime independent
>>> of the creating process I expect you want to reparent them to the local
>>> init process.  Just so they don't confuse the process tree.  Plus init
>>> processes know how to handle unexpected children.
>>>
>>> What are the semantics you are aiming for?
>>>
>>
>> Hi Eric,
>>
>> Right now, for vhost we need the thread being created:
>>
>> 1. added to the caller's cgroup.
>> 2. to share the mm struct with the caller.
>> 3. to be accounted for under the caller's nproc rlimit value.
>>
>> For 1 and 2, we have cgroup_attach_task_all and get_task_mm
>> already.
>>
>> This patchset started with me just trying to handle #3 by modifying kthreads
>> like here:
>>
>> https://urldefense.com/v3/__https://lkml.org/lkml/2021/6/23/1234__;!!ACWV5N9M2RV99hQ!bvqZOWy7TxQyq18L4I_a5MxP2OX0V2imOYEJrWsc-LkyVTI_zpFzxyV2pM_dgYywwH2y$ 
>>
>> So we can use kthreads and the existing helpers and add:
>>
>> A. a ucounts version of the above patches in the link
>>
>> or
>>
>> B. a helper that does something like copy_process's use of
>> is_ucounts_overlimit and vhost can call that.
>>
>> instead of this patchset.
> 
> I don't fundamentally hate the patchset.  I do have concerns about
> the completely broken patch.
> 
> With respect this patchset my gut says decide.  Are you a thread of the
> process (just use create_io_worker) are you a separate process forked
> from the caller (use a cousin of create_io_worker but don't touch
> create_io_worker).  I think being a process vs being a thread is such a
> fundamental difference we don't want to mix the helpers.
> 
>> Before we even get to the next section below, do you consider items 1 - 3
>> something we need an API based on copy_process for?
> 
> I think 3 staying in the callers nproc strongly suggests you want to
> reuse copy_process.  Which gets back to my question do you want
> a thread or do you want a process.
> 
> 
> For me a key detail is what is the lifetime of the vhost device?
> 
> Does the vhost go away when the caller goes away?

Yes. When the caller, normally qemu in our case, that created the worker
thread exits, then we free the vhost devices and stop and free the worker
threads we are creating in this patchset.

However, I'm not sure if it makes a difference to you, but we also have second
way to free a vhost device and its worker thread. The user can run a command
that instructs the the qemu process to free the vhost device and its worker
thread.

> 
>   If so you can create a thread in the caller's process that only performs
>   work in kernel space.  At which point you are essentially
>   create_io_thread.
> 
> If the vhost device can live after the caller goes away how is that managed?

When the caller goes away we free the devices and their worker threads.

Either before the caller exists it does an explicit close to release the device
which frees the device and its worker thread, or when the process exits and the
kernel does a put on its open devices that will trigger the vhost device's release
function and we free device and its thread at that time.


>   Especially when you are tied to the caller's mm.
> 
>   If your device lives on after the caller then you should be a separate
>   process forked from the caller.
> 
> 
>> Do you think I should just do A or B above, or do you have another idea? If
>> so can we get agreement on that from everyone?
> 
> Like I said.  I don't hate this patchset.  I think you just tried to be
> a little too general.
> > 
> With kthreads you are fighting very hard to get something that is not
> part of the process tree to act like it is part of the process tree.
> Which strongly suggests you should be part of the process tree.
> 
> As long as you don't have important state you will need to fight to get
> separate from the process.
> 
> If you do have important state that needs to not come from the process
> figuring out how to make a kthread work may be safer.
> 
> I don't currently know vhost devices well enough to answer that question.
> 
>> I thought my patches in that link were a little hacky in how they passed
>> around the user/creds info. I thought maybe it shouldn't be passed around like
>> that, so switched to the copy_process based approach which did everything for
>> me. And I thought io_uring needed something similar as us so I made it generic.
>>
>> I don't have a preference. You and Christian are the experts, so I'll leave it
>> to you guys.
> 
> I hope this gives you some useful direction.
> 
> Eric
> 
> 
>>> I can see sense in generalizing some of the pieces of create_io_thread
>>> but I think generalizing create_io_thread itself is premature.  The code
>>> lives in kernel/fork.c because it is a very special thing that we want
>>> to keep our eyes on.
>>>
>>> Some of your generalization makes it much more difficult to tell what
>>> your code is going to use because you remove hard codes that are there
>>> to simplify the analysis of the situation.
>>>
>>> My gut says adding a new create_vhost_worker and putting that in
>>> kernel/fork.c is a lot safer and will allow much better code analysis.
>>>
>>> If there a really are commonalities between creating a userspace process
>>> that runs completely in the kernel and creating an additional userspace
>>> thread we refactor the code and simplify things.
>>>
>>> I am especially nervous about generalizing the io_uring code as it's
>>> signal handling just barely works, and any generalization will cause it
>>> to break.  So you are in the process of generalizing code that simply
>>> can not handle the general case.  That scares me