Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 20 Jul 2023 21:00:22 +0000
From:   Matthew Brost <matthew.brost@intel.com>
To:     Thomas =?iso-8859-1?Q?Hellstr=F6m?= 
        <thomas.hellstrom@linux.intel.com>
CC:     <intel-xe@lists.freedesktop.org>,
        Nirmoy Das <nirmoy.das@intel.com>,
        Danilo Krummrich <dakr@redhat.com>,
        Joonas Lahtinen <joonas.lahtinen@linux.intel.com>,
        Oak Zeng <oak.zeng@intel.com>,
        "Daniel Vetter" <daniel@ffwll.ch>,
        Maarten Lankhorst <maarten.lankhorst@linux.intel.com>,
        Francois Dugast <francois.dugast@intel.com>,
        <dri-devel@lists.freedesktop.org>, <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v5] Documentation/gpu: Add a VM_BIND async draft document
Message-ID: <ZLmgZl8fc59kPe4/@DUT025-TGLU.fm.intel.com>
References: <20230715154543.13183-1-thomas.hellstrom@linux.intel.com>
Content-Type: text/plain; charset="iso-8859-1"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20230715154543.13183-1-thomas.hellstrom@linux.intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?IY6nOJ45LxA/RULYf++L1X7D6+odt6RdAxGZA2Zq+zuweR6RdgpXiW96Ry?=
 =?iso-8859-1?Q?IFPnBiWvoNbGK2P5UGRglbPMCuG1NT+J0onMfYMYbiWnPJ9ykwB71rsYFD?=
 =?iso-8859-1?Q?peOxsbLS9V3RG6DVqnuc2elRSX23ZbOjsL1o89ixonFdyUinbfHrT9BkFF?=
 =?iso-8859-1?Q?UxP0wfzraGss2APOnJI9f0x6gGD32AUsAg9rdPbEdm47Q7h1AgmgyUrpLL?=
 =?iso-8859-1?Q?7OIFMSH6zb0F64sV6+G5P27SBHveH0qhiBUveK9QM7nzVTtJNW+8qebLj2?=
 =?iso-8859-1?Q?4a3ZZoZgDKeFcZpIUf2ri6yDzrzsYNfz/DuqY06lnQe7Pcxdjs60elSDTc?=
 =?iso-8859-1?Q?bHEf1m+cP2Mxt2crklS5F+uwd6sqipf4XZHeeQu7ZKWsjhII4C4ieaW1PQ?=
 =?iso-8859-1?Q?PcIcT7qFXEfTiOiWsr2vFmCW5fs8ZTwc9/8U3YlDbUM+l3vaLyxt9MR4v3?=
 =?iso-8859-1?Q?7k7nshmrZSMZchiFjJyIKBZ7tMbh7ws6OdBmlH/Li1FR72IgBKwv/jmeSK?=
 =?iso-8859-1?Q?ncZeNTj7Fyyo8GmZRYHSeQywGeSbYaez+h0ZI7h71gl60QQOV41yXaUwLz?=
 =?iso-8859-1?Q?Wxi7uxpmOrLzEYYeVx4y0rFhlJvOnQDYCQk4os45m1X9PR+6k32v6bW+IG?=
 =?iso-8859-1?Q?lLS/3YZuU+v8rMx57PrRNX9KlMCSU6mL2X4CPgllVTe7uU+6EOYUkQfFoq?=
 =?iso-8859-1?Q?+twXRP0fVbyqWUgCOjqpff8yKyjKYF0RRm/8jiTQgo2872PKSDcsM4oNLi?=
 =?iso-8859-1?Q?NYpYgtG2ybpi4zBxgijU8gKDcwMyz8i8xKY19t4HCag5TgNicW6KTGNuPG?=
 =?iso-8859-1?Q?K/01I5LPSLwzosvScLpQZ8eQdxhjeHfV/eAu8I/r4s5XPCA9nNSk6CYUOF?=
 =?iso-8859-1?Q?uE0WtS1ePhijijMX6FREyOBaCs73rHq0X9aZFSQJ04K9YvHyWS2WYpBRhm?=
 =?iso-8859-1?Q?E/tOHr/ALgNFXkL7vohuO8eecDfWIpbcsX/uFFVV+UGp6wjE0/w8d3cuU7?=
 =?iso-8859-1?Q?MMjpcX1UIM10SzKoXxZKLHBegKLpGB6yTg2LMEjrrrxn/tgDwYaWGlIjR5?=
 =?iso-8859-1?Q?63TW1xiG66ykCPKqx5pEQ/upBmrNEFy06ut3dvEhunHHX7BlfxfcmmOigg?=
 =?iso-8859-1?Q?q2frJF/XrJDtAmapk/+EnjObPNTYuYq1bKnT4U5fnV9PHAXyAEfTUn9NSv?=
 =?iso-8859-1?Q?yj+7YJGkH0XZO38kuNWuC0WCC1RK8jF1C7rYqvaT49pDOpxv957Sw6TeOY?=
 =?iso-8859-1?Q?mJFZI6T/Vi4/lb51h9fuI8E1INEqNLW4lipXeDWGogjzLeb6bv28JBkExO?=
 =?iso-8859-1?Q?/EE0vTMTQhdoRfCInyyyL7QXnW0qkYDJiML4kmSiLIiaXVcEuJa01YoJpE?=
 =?iso-8859-1?Q?+UJWiiidJ3ZBUaZGK8GvCO5+VD4RhHQQ3PSS7neasjfcvJpPKo6MWC0b19?=
 =?iso-8859-1?Q?/ALHln0QrtTVQ4J9GvylxscYKBBQ9ZT8kzPb5LOzGonio9xjRoh3Zobp9q?=
 =?iso-8859-1?Q?OQHVVVc6GsF8V3AvcC0nk3TOXkboMbNEerFefJ+HCvObDHCIlNF9SZBeqz?=
 =?iso-8859-1?Q?+TN2Pmv6XKfYyuKTvKGC5lrImG7Du6O+3Addf8zj2Wt5/Jgm/9laX2tviD?=
 =?iso-8859-1?Q?eiSDWtsW2YjiUYJg/JLO9Txc5GPH5x+7kI0RJcIQf5alNQbY/Szq8mnA?=
 =?iso-8859-1?Q?=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 236d014d-fb5e-4282-e325-08db896470b1
X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB6522.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jul 2023 21:01:08.7174
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ghzkBOmpLH0pvdHdLQ+W4pNn4tEPw5M2tIY+ApJKfkVvS3me443+Bx8O2B4cWNAImV9qeXSsDGqzqCUaNYnscQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR11MB8243
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
        RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Sat, Jul 15, 2023 at 05:45:43PM +0200, Thomas Hellstr?m wrote:
> Add a motivation for and description of asynchronous VM_BIND operation
> 
> v2:
> - Fix typos (Nirmoy Das)
> - Improve the description of a memory fence (Oak Zeng)
> - Add a reference to the document in the Xe RFC.
> - Add pointers to sample uAPI suggestions
> v3:
> - Address review comments (Danilo Krummrich)
> - Formatting fixes
> v4:
> - Address typos (Francois Dugast)
> - Explain why in-fences are not allowed for VM_BIND operations for long-
>   running workloads (Matthew Brost)
> v5:
> - More typo- and style fixing
> - Further clarify the implications of disallowing in-fences for VM_BIND
>   operations for long-running workloads (Matthew Brost)
> 
> Signed-off-by: Thomas Hellstr?m <thomas.hellstrom@linux.intel.com>

Reviewed-by: Matthew Brost <matthew.brost@intel.com>

> Acked-by: Nirmoy Das <nirmoy.das@intel.com>
> ---
>  Documentation/gpu/drm-vm-bind-async.rst | 171 ++++++++++++++++++++++++
>  Documentation/gpu/rfc/xe.rst            |   4 +-
>  2 files changed, 173 insertions(+), 2 deletions(-)
>  create mode 100644 Documentation/gpu/drm-vm-bind-async.rst
> 
> diff --git a/Documentation/gpu/drm-vm-bind-async.rst b/Documentation/gpu/drm-vm-bind-async.rst
> new file mode 100644
> index 000000000000..d2b02a38198a
> --- /dev/null
> +++ b/Documentation/gpu/drm-vm-bind-async.rst
> @@ -0,0 +1,171 @@
> +====================
> +Asynchronous VM_BIND
> +====================
> +
> +Nomenclature:
> +=============
> +
> +* ``VRAM``: On-device memory. Sometimes referred to as device local memory.
> +
> +* ``gpu_vm``: A GPU address space. Typically per process, but can be shared by
> +  multiple processes.
> +
> +* ``VM_BIND``: An operation or a list of operations to modify a gpu_vm using
> +  an IOCTL. The operations include mapping and unmapping system- or
> +  VRAM memory.
> +
> +* ``syncobj``: A container that abstracts synchronization objects. The
> +  synchronization objects can be either generic, like dma-fences or
> +  driver specific. A syncobj typically indicates the type of the
> +  underlying synchronization object.
> +
> +* ``in-syncobj``: Argument to a VM_BIND IOCTL, the VM_BIND operation waits
> +  for these before starting.
> +
> +* ``out-syncobj``: Argument to a VM_BIND_IOCTL, the VM_BIND operation
> +  signals these when the bind operation is complete.
> +
> +* ``memory fence``: A synchronization object, different from a dma-fence.
> +  A memory fence uses the value of a specified memory location to determine
> +  signaled status. A memory fence can be awaited and signaled by both
> +  the GPU and CPU. Memory fences are sometimes referred to as
> +  user-fences, userspace-fences or gpu futexes and do not necessarily obey
> +  the dma-fence rule of signaling within a "reasonable amount of time".
> +  The kernel should thus avoid waiting for memory fences with locks held.
> +
> +* ``long-running workload``: A workload that may take more than the
> +  current stipulated dma-fence maximum signal delay to complete and
> +  which therefore needs to set the gpu_vm or the GPU execution context in
> +  a certain mode that disallows completion dma-fences.
> +
> +* ``exec function``: An exec function is a function that revalidates all
> +  affected gpu_vmas, submits a GPU command batch and registers the
> +  dma_fence representing the GPU command's activity with all affected
> +  dma_resvs. For completeness, although not covered by this document,
> +  it's worth mentioning that an exec function may also be the
> +  revalidation worker that is used by some drivers in compute /
> +  long-running mode.
> +
> +* ``bind context``: A context identifier used for the VM_BIND
> +  operation. VM_BIND operations that use the same bind context can be
> +  assumed, where it matters, to complete in order of submission. No such
> +  assumptions can be made for VM_BIND operations using separate bind contexts.
> +
> +* ``UMD``: User-mode driver.
> +
> +* ``KMD``: Kernel-mode driver.
> +
> +
> +Synchronous / Asynchronous VM_BIND operation
> +============================================
> +
> +Synchronous VM_BIND
> +___________________
> +With Synchronous VM_BIND, the VM_BIND operations all complete before the
> +IOCTL returns. A synchronous VM_BIND takes neither in-fences nor
> +out-fences. Synchronous VM_BIND may block and wait for GPU operations;
> +for example swap-in or clearing, or even previous binds.
> +
> +Asynchronous VM_BIND
> +____________________
> +Asynchronous VM_BIND accepts both in-syncobjs and out-syncobjs. While the
> +IOCTL may return immediately, the VM_BIND operations wait for the in-syncobjs
> +before modifying the GPU page-tables, and signal the out-syncobjs when
> +the modification is done in the sense that the next exec function that
> +awaits for the out-syncobjs will see the change. Errors are reported
> +synchronously assuming that the asynchronous part of the job never errors.
> +In low-memory situations the implementation may block, performing the
> +VM_BIND synchronously, because there might not be enough memory
> +immediately available for preparing the asynchronous operation.
> +
> +If the VM_BIND IOCTL takes a list or an array of operations as an argument,
> +the in-syncobjs needs to signal before the first operation starts to
> +execute, and the out-syncobjs signal after the last operation
> +completes. Operations in the operation list can be assumed, where it
> +matters, to complete in order.
> +
> +Since asynchronous VM_BIND operations may use dma-fences embedded in
> +out-syncobjs and internally in KMD to signal bind completion,  any
> +memory fences given as VM_BIND in-fences need to be awaited
> +synchronously before the VM_BIND ioctl returns, since dma-fences,
> +required to signal in a reasonable amount of time, can never be made
> +to depend on memory fences that don't have such a restriction.
> +
> +To aid in supporting user-space queues, the VM_BIND may take a bind context.
> +
> +The purpose of an Asynchronous VM_BIND operation is for user-mode
> +drivers to be able to pipeline interleaved gpu_vm modifications and
> +exec functions. For long-running workloads, such pipelining of a bind
> +operation is not allowed and any in-fences need to be awaited
> +synchronously. The reason for this is twofold. First, any memory
> +fences gated by a long-running workload and used as in-syncobjs for the
> +VM_BIND operation will need to be awaited synchronously anyway (see
> +above). Second, any dma-fences used as in-syncobjs for VM_BIND
> +operations for long-running workloads will not allow for pipelining
> +anyway since long-running workloads don't allow for dma-fences as
> +out-syncobjs, so while theoretically possible the use of them is
> +questionable and should be rejected until there is a valuable use-case.
> +Note that this is not a limitation imposed by dma-fence rules, but
> +rather a limitation imposed to keep KMD implementation simple. It does
> +not affect using dma-fences as dependencies for the long-running
> +workload itself, which is allowed by dma-fence rules, but rather for
> +the VM_BIND operation only.
> +
> +Also for VM_BINDS for long-running gpu_vms the user-mode driver should typically
> +select memory fences as out-fences since that gives greater flexibility for
> +the kernel mode driver to inject other operations into the bind /
> +unbind operations. Like for example inserting breakpoints into batch
> +buffers. The workload execution can then easily be pipelined behind
> +the bind completion using the memory out-fence as the signal condition
> +for a GPU semaphore embedded by UMD in the workload.
> +
> +Multi-operation VM_BIND IOCTL error handling and interrupts
> +===========================================================
> +
> +The VM_BIND operations of the IOCTL may error due to lack of resources
> +to complete and also due to interrupted waits. In both situations UMD
> +should preferably restart the IOCTL after taking suitable action. If
> +UMD has over-committed a memory resource, an -ENOSPC error will be
> +returned, and UMD may then unbind resources that are not used at the
> +moment and restart the IOCTL. On -EINTR, UMD should simply restart the
> +IOCTL and on -ENOMEM user-space may either attempt to free known
> +system memory resources or abort the operation. If aborting as a
> +result of a failed operation in a list of operations, some operations
> +may still have completed, and to get back to a known state, user-space
> +should therefore attempt to unbind all virtual memory regions touched
> +by the failing IOCTL.
> +Unbind operations are guaranteed not to cause any errors due to
> +resource constraints.
> +In between a failed VM_BIND IOCTL and a successful restart there may
> +be implementation defined restrictions on the use of the gpu_vm. For a
> +description why, please see KMD implementation details under `error
> +state saving`_.
> +
> +Sample uAPI implementations
> +===========================
> +Suggested uAPI implementations at the moment of writing can be found for
> +the Nouveau driver `here
> +<https://patchwork.freedesktop.org/patch/543260/?series=112994&rev=6>`_.
> +and for the Xe driver `here
> +<https://cgit.freedesktop.org/drm/drm-xe/diff/include/uapi/drm/xe_drm.h?h=drm-xe-next&id=9cb016ebbb6a275f57b1cb512b95d5a842391ad7>`_.
> +
> +KMD implementation details
> +==========================
> +
> +Error state saving
> +__________________
> +Open: When the VM_BIND IOCTL returns an error, some or even parts of
> +an operation may have been completed. If the IOCTL is restarted, in
> +order to know where to restart, the KMD can either put the gpu_vm in
> +an error state and save one instance of the needed restart state
> +internally. In this case, KMD needs to block further modifications of
> +the gpu_vm state that may cause additional failures requiring a
> +restart state save, until the error has been fully resolved. If the
> +uAPI instead defines a pointer to a UMD allocated cookie in the IOCTL
> +struct, it could also choose to store the restart state in that cookie.
> +
> +The restart state may, for example, be the number of successfully
> +completed operations.
> +
> +Easiest for UMD would of course be if KMD did a full unwind on error
> +so that no error state needs to be saved.
> diff --git a/Documentation/gpu/rfc/xe.rst b/Documentation/gpu/rfc/xe.rst
> index 2516fe141db6..0f062e1346d2 100644
> --- a/Documentation/gpu/rfc/xe.rst
> +++ b/Documentation/gpu/rfc/xe.rst
> @@ -138,8 +138,8 @@ memory fences. Ideally with helper support so people don't get it wrong in all
>  possible ways.
>  
>  As a key measurable result, the benefits of ASYNC VM_BIND and a discussion of
> -various flavors, error handling and a sample API should be documented here or in
> -a separate document pointed to by this document.
> +various flavors, error handling and sample API suggestions are documented in
> +Documentation/gpu/drm-vm-bind-async.rst
>  
>  Userptr integration and vm_bind
>  -------------------------------
> -- 
> 2.40.1
>