Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 12 Jun 2023 13:22:43 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Daisuke Matsuda <matsuda-daisuke@fujitsu.com>
Cc:     linux-rdma@vger.kernel.org, leonro@nvidia.com,
        zyjzyj2000@gmail.com, linux-kernel@vger.kernel.org,
        rpearsonhpe@gmail.com, yangx.jy@fujitsu.com, lizhijian@fujitsu.com,
        y-goto@fujitsu.com
Subject: Re: [PATCH for-next v5 6/7] RDMA/rxe: Add support for
 Send/Recv/Write/Read with ODP
Message-ID: <ZIdGU709e1h5h4JJ@nvidia.com>
References: <cover.1684397037.git.matsuda-daisuke@fujitsu.com>
 <25d903e0136ea1e65c612d8f6b8c18c1f010add7.1684397037.git.matsuda-daisuke@fujitsu.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <25d903e0136ea1e65c612d8f6b8c18c1f010add7.1684397037.git.matsuda-daisuke@fujitsu.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?GrLmOc3LZ3lALydO5VTnVrk98EoF7U7lj/FPMa3YBvTctdAzG50fPUEW84zB?=
 =?us-ascii?Q?3w6t3WUGbKFXXwPraq9oNKIAN6H1NmMnz7gGhIOiBotOOa2ALV63dLe6qiFy?=
 =?us-ascii?Q?GG215lQvZFG0Y4FRixVMW5EJd4Vb+Iu/7HPKvZuUOdp9QJwWulrmCEXzeISU?=
 =?us-ascii?Q?DTrBnNbyM5gP2VvxCx73rhio07ueHyQq28dHLD0EhFEEA7LPYa12dLp2ymEB?=
 =?us-ascii?Q?VXWOMCZ3h9/xK61KcmOP7fI3cO3RQ5EuAxMNeLQMPV2z2NjZSvRv/aztZQ3/?=
 =?us-ascii?Q?aprORsHRwSQtUmz5xpoz6bJeAzr8d3GzOsXSd8m+X3vy65/OcEyrjEqgUELD?=
 =?us-ascii?Q?VYzOYEsn278HBun77wDsbQ0wBSI3SPtFdoMvYH9+tn9Dl+sEnBBJaGRMsc6v?=
 =?us-ascii?Q?1FTHZmiBZYdmPALLsseqOIaEmRJUlJuunz2MS047KIeXgn/zDUFjCiUzcI88?=
 =?us-ascii?Q?nYwe9CSgOYiw/bX5TMTuS2pCLzlF6DiKsLrO+ZMgKOcNEJ5CFRuw+GCSi1gf?=
 =?us-ascii?Q?8qe3GHfFZZvbMuBde4MaOwIm2VSlgCWH4zW6xJtz5Et9n1IO7HhLkkw98LKn?=
 =?us-ascii?Q?jiORodSbEWalIlOQHm99Pfh1hpnmZiAyoYbu68xt73bxKgD7iq2JunnF/i3f?=
 =?us-ascii?Q?g/Bexph46Q/jpIqgIXYZGSP/eBqLmuEp2xqHk89xWlHAD5/g2RaZXPM7dr5T?=
 =?us-ascii?Q?wkPcWozCHqO+0+U5dFD97YB4CEjmtZ7Xz99Wx7nSJl2k0HRZJgq0+3jJsoFh?=
 =?us-ascii?Q?haxwIV2roC9JJjkXbnJ9lKu62ONmTgZPllxa802ekp3Tz6HTAGc1FeEvpqaT?=
 =?us-ascii?Q?blUMltLsuFDgcgLrkqWpFYTEEAalqBvLGjxng8nuhRFI6zwINfqxUETpqycz?=
 =?us-ascii?Q?DgbaFzswPHK8cU4pMzmMvr8LivmOLmLN4B1aekRnJMSwKp1pG+9Oq46j8V1Q?=
 =?us-ascii?Q?40IdCGRmCwMmd8QShbfiOfTvE3jSHUVxJRmHqPaWY9FjHhINu5eJETLOV3It?=
 =?us-ascii?Q?vecdQ1gPUVI0+xWHn7MYscSn9JmT75f6odcylRsLwYFGckTfhk5+jTEB5pIw?=
 =?us-ascii?Q?ZfFeSx/3joNe9ciaqjby7p7XKSO0n9bIIovzkYx3iriXSdPL5z5dVgqelMUv?=
 =?us-ascii?Q?126g7VhImX0VLlRE4Eb9B+dnrponQxu0gYrItsXS4lX212Wi8/zKJjbCklO3?=
 =?us-ascii?Q?Hok4VhtnwoOySlBEZRTektKCND0agqRoPh0Auf5tTF7RgatCZO0IC6MhLf9C?=
 =?us-ascii?Q?x0mc1X/EOKQwfHCxaV1Wqzp6ZDwsHW1u4Pufk+ZgWNLzeURRkL75D2b1dEzQ?=
 =?us-ascii?Q?8KvLYvFyuKNcGu/TDFZleShEoOuhoyf2fIpoU4XjPjybsGc1HWQ8+co8tD9i?=
 =?us-ascii?Q?lVuLUG2gQ+4rX/WKJwc5SkuaQ5Aj7xl/SLBY0HZtdfSCxo8kQOQ8IQrCPzX3?=
 =?us-ascii?Q?aMuHxwwGdf2f68ngzhyG/k5x/7TUBhY1gxZxihZ/qynjQuyn3Hkyz7lUm9BQ?=
 =?us-ascii?Q?YP6rrePomvR/veJOSUaCU0aroMx5nAwyr0CWLhaZbRO+KVY26MHbbDDMTwRq?=
 =?us-ascii?Q?3J9jsNHedNDG7x+2yDw/OKgwWVH5w5cMZVLXLXUK?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 25064ba2-94d7-4061-ac5b-08db6b6140bf
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jun 2023 16:22:44.6332
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: eoMbosVB61pJkG4EskozNn/C3sSTP0UOGCSJfINmlS6H0rUjHAkXUEu6wW0f5CIj
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5096
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, May 18, 2023 at 05:21:51PM +0900, Daisuke Matsuda wrote:

> +/* umem mutex must be locked before entering this function. */
> +static int rxe_odp_map_range(struct rxe_mr *mr, u64 iova, int length, u32 flags)
> +{
> +	struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem);
> +	const int max_tries = 3;
> +	int cnt = 0;
> +
> +	int err;
> +	u64 perm;
> +	bool need_fault;
> +
> +	if (unlikely(length < 1)) {
> +		mutex_unlock(&umem_odp->umem_mutex);
> +		return -EINVAL;
> +	}
> +
> +	perm = ODP_READ_ALLOWED_BIT;
> +	if (!(flags & RXE_PAGEFAULT_RDONLY))
> +		perm |= ODP_WRITE_ALLOWED_BIT;
> +
> +	/*
> +	 * A successful return from rxe_odp_do_pagefault() does not guarantee
> +	 * that all pages in the range became present. Recheck the DMA address
> +	 * array, allowing max 3 tries for pagefault.
> +	 */
> +	while ((need_fault = rxe_is_pagefault_neccesary(umem_odp,
> +							iova, length, perm))) {
> +		if (cnt >= max_tries)
> +			break;

I don't think this makes sense..

You need to make this work more like mlx5 does, you take the spinlock
on the xarray, you search it for your index and whatever is there
tells what to do. Hold the spinlock while doing the copy. This is
enough locking for the fast path.

If there is no index present, or it is not writable and you need
write, then you unlock the spinlock and prefetch the missing entry and
try again, this time also holding the mutex so there isn't a race.

You shouldn't probe into parts of the umem to discover information
already stored in the xarray then do the same lookup into the xarray.

IIRC this also needs to keep track in the xarray on a per page basis
if the page is writable.

Jason