Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <c4ff442d-706d-bb00-84af-c991454a37de@intel.com>
Date:   Thu, 30 Mar 2023 17:44:27 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Firefox/91.0 Thunderbird/91.11.0
Subject: Re: [PATCH v2 08/16] iommu: define and export
 iommu_access_remote_vm()
Content-Language: en-US
To:     Christoph Hellwig <hch@infradead.org>
CC:     Baolu Lu <baolu.lu@linux.intel.com>, Vinod Koul <vkoul@kernel.org>,
        "Dave Jiang" <dave.jiang@intel.com>, <dmaengine@vger.kernel.org>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        Alistair Popple <apopple@nvidia.com>,
        "Joerg Roedel" <joro@8bytes.org>, Will Deacon <will@kernel.org>,
        Robin Murphy <robin.murphy@arm.com>,
        Lorenzo Stoakes <lstoakes@gmail.com>, <iommu@lists.linux.dev>
References: <20230306163138.587484-1-fenghua.yu@intel.com>
 <20230306163138.587484-9-fenghua.yu@intel.com>
 <f7b0cbef-3d7d-e5aa-1971-fe230d0c80e1@linux.intel.com>
 <932aebd3-c655-3266-1acb-e41e8cbfb771@intel.com>
 <ZBhhEK1V+u2oV2Ll@infradead.org>
From:   Fenghua Yu <fenghua.yu@intel.com>
In-Reply-To: <ZBhhEK1V+u2oV2Ll@infradead.org>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cVFCSHY1eElDdGpnUExGdkpQeXRuYm4yL2xVM29xZjQrTTM2cWRmY3l3MFI3?=
 =?utf-8?B?N1ZKaDFVS1M4a0xPSjNKREErK0czYVlidVUyeEZNQlZzMENBdDVzcFc0dWkz?=
 =?utf-8?B?WXVlWFdEd2JDZWxyanU0bU9oRXUyb1l2RHlOblNaRzZHTVQ2VTJlRUNqNzRZ?=
 =?utf-8?B?V2t0Q25HblcyV21PcGdPYm9Za1JUSlJ6WjJJS0R5OTNoRThIUGE4MGpEM1d2?=
 =?utf-8?B?TUxWdm5LREQ3MkRac3ViazFHbm12dFFwME51VjNlN3RjQWd5dlFKZzMzdndh?=
 =?utf-8?B?TFd0VG9aQk92SmJTOWVTT1I5RzFnRnJKb1pCTkZrUTVQMk5rL1BRTVRwY0FD?=
 =?utf-8?B?eU5nR2dWWUFYVWtqNmFDbEtvQUFKMDBpcEVRZ2FZVWlQL2RTR3poMXM1eWxa?=
 =?utf-8?B?SEFrS0ZFeVVLZlVOMjA4SS9rdDBaNEN4c3FCY1htOW8zVEVOSjJZaEUwb0pP?=
 =?utf-8?B?UTJrcTRKY2JZdWZRZTM0SlMzTHZ0S3VZWUZjQStrcFFXcm5yeXVVNnYzR01U?=
 =?utf-8?B?QUhSejlFZGtsS0FWM3VhYmlpdFdabmM1ZnkwdkZrbm9Bbit2YXQybktSZUFL?=
 =?utf-8?B?WXFlRzVWQ1BTSzFHY2VRN1ZHS2JXS3NNdm5GdUtDbjZhaSsrSTBnd1o1aUd2?=
 =?utf-8?B?bHdxTnRCQjkvcGZlQkVYckQrOCtuaEFvcjhpNlNoSzl5ejQzcVp5VEhvcW9a?=
 =?utf-8?B?aXQyNDJKUFdJaUxseUNaMjNPRG1kOWhjUnNPUXhMcGRIYVE2Ujh6N1RVa0tv?=
 =?utf-8?B?STNGSnp1L2ZDS1BxMFgxWUorUXlPTFlIUFpjOWpJREtmOFVDQmdiNU1PSU4x?=
 =?utf-8?B?Yk4rQ0IyU0gwbFQvNFBpOUM1UXhzaklpS2lHeG9KZkxpUmVacGFFNU92K2hu?=
 =?utf-8?B?bHIvRHB4dFNoSXJrUnc1d2hhOVorSVJNSERadkNNZlJOeEl2NlpDcm5oYmdw?=
 =?utf-8?B?RC9XcVhiSU84dEhZcWhmaHV0OFlPRzRmWXRUaWdlRU5oODc3cnhqTHpOeTNV?=
 =?utf-8?B?YVpnYnpLWVo1R3IxdmNzVW0wRUc2d2lTZFJoYWZLZXpHcDNPZkN4Z1VjN3M3?=
 =?utf-8?B?MDdUbnZmZUtXeTVUbERGRTVnL3o1bTE4aHQ1OGdUMTQ1OGExa1REaUVVcGRn?=
 =?utf-8?B?MHcxeUxLdHdOd1hsZ2tobnpydi9nb1RGcGk2OEFTZWtJdUcxb2x3aUxJaWlv?=
 =?utf-8?B?REpJbWQ5RzRJWHNvNGJhRStFeTF4SmEzcm9SdENSR3FyQ09xWHI5dUpMUEdC?=
 =?utf-8?B?dmNGQmR5a3VSUFhFNVNsa1dUT0xSbVFHdzhTMzNweWplWTBRRDdlanhFOExt?=
 =?utf-8?B?OUVYK3RYditoNENZVVZFd2dTdzFqNEtpd3B5b09HYVZ6Vnc4VkJEMzJxS0Nl?=
 =?utf-8?B?bkVQY1dvdFpxTzkrQXZUbTcyWDE2OXdyT2VtcDJJMkJlU2QxWnBiQUR0R3g5?=
 =?utf-8?B?RjlkRUlydkxubW01cTJiQmVyU3hncGtWemhQS1VPK1BSSUYydGtNTE4yRzJw?=
 =?utf-8?B?WlVYY1F0U0ZvdUhrcnhhU21YQmRQYzl5UlRGdFRDQ1VCckVUbFhRc29yZFhI?=
 =?utf-8?B?Q1JpWVRzK1Y4MFRvUFpibU5qTUdra3A5NUNGTDdDejErN0FBM3k1UWlsV2Vr?=
 =?utf-8?B?VXk5aHV4Qys4VnRLVzNteXZHaFhWSVJyVEE2YmxsOHdiZXJxamczQ3NKT1Rw?=
 =?utf-8?B?dVNoS2Y2OUx5enNVSVlNajRPZjNSdzRvZkR4eFZqR2dzRlExdmxJMy9WOHF6?=
 =?utf-8?B?Q1h0SE9HTjZBN0ZuNGpHdjc0U3pBcGpiZnEybDM2V3RwSzJmRy9UWENxd0xF?=
 =?utf-8?B?TW85Nm91VFFzK1djaWJXMUVrcXc5Z2gvRXpLSkt5NHphQ1hwdVYwYStSR1R2?=
 =?utf-8?B?VTl1UVU5REI4RjhYM2gwYzJLeTFXYTgxTUYvWVd3eVFsTlpGMjlES052MHZo?=
 =?utf-8?B?bFYvVGVDR3BUQXBQVS84RWlLcXQvQTFtd2dWSUVJY2dxR2V1YkJSUWNzejJY?=
 =?utf-8?B?OWJIUDdnRnE3YXJ1TE5kUnFZWVBveS9aeEppTzBTZHl3dmhNcTZUT1NKZE41?=
 =?utf-8?B?MFhrWkNEelBCOGVIZm5KclRIODd4SU1rbGhjeEJRYVo0bzM0NkFiM0NmekYz?=
 =?utf-8?Q?hf3iS/S9Gtfo9cpXh7CjX9MUh?=
X-MS-Exchange-CrossTenant-Network-Message-Id: cec9f87a-44ef-4542-d2fb-08db31810901
X-MS-Exchange-CrossTenant-AuthSource: IA1PR11MB6097.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Mar 2023 00:44:07.6757
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: wAhp+eW+KQPsymVFZyv2j/e0V3HP7v72qPzG/N9jtbzazGVlhAJExWPdxQm2I2Zql8f5P/3lum0jjGbf829W/w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB6540
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi, Christoph,

On 3/20/23 06:35, Christoph Hellwig wrote:
> On Tue, Mar 07, 2023 at 09:55:28AM -0800, Fenghua Yu wrote:
>>>
>>> I don't quite follow here. Isn't I/O page fault already supported?
>>
>> The following patch 9 in this series explains in details why IDXD device
>> cannot use page fault to write to user memory: https://lore.kernel.org/dmaengine/20230306163138.587484-10-fenghua.yu@intel.com/
>>
>> "DSA supports page fault handling through PRS. However, the DMA engine
>> that's processing the descriptor is blocked until the PRS response is
>> received. Other workqueues sharing the engine are also blocked.
>> Page fault handing by the driver with PRS disabled can be used to
>> mitigate the stalling.
>>
>> With PRS disabled while ATS remain enabled, DSA handles page faults on
>> a completion record by reporting an event in the event log. In this
>> instance, the descriptor is completed and the event log contains the
>> completion record address and the contents of the completion record."
> 
> This seems like a completely broken I/O model, and I don't think Linux
> should support this model when it requires operations like
> access_remote_vm.

This patch set have two parts:
1. Basic event log support and PRS disabling knob.
2. Completion record page fault fixup part. The current patch is the 
major patch in this part. It tries to fix up completion record page 
fault from user space.

Since the fix up in part 2 is debatable and part 1 can be sent out 
independently, how about we send out the parts separately?

In the new part 1, simply warn on completion record page fault and don't 
try to fix it up. Usually a process executes ENQCMD instruction and then 
keeps polling completion record periodically. So the completion record 
will be likely backed by page and won't generate page fault. If page 
fault is really an issue, sysadmin can enable PRS (which is enabled by 
default anyway) and let PRS to handle page fault.

Then in the next step, we will send out a new part 2 to eliminate 
completion record page fault. One proposal is to mmap() kernel allocated 
completion record area. So there won't be completion record page fault 
and fix up(no access_remote_vm() of course).

Is this OK for you?

Thank you very much for your comment!

-Fenghua