Received: by 2002:a05:6a10:206:0:0:0:0 with SMTP id 6csp97639pxj; Thu, 3 Jun 2021 01:40:58 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwwDGFYGvgkrLjOw4AQJkNOAfDrcQ0R4AIeAcLJ8KxM592CSGOsDtfYk4aYFDfBBhkJVkwt X-Received: by 2002:a50:bec1:: with SMTP id e1mr42970066edk.116.1622709658104; Thu, 03 Jun 2021 01:40:58 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1622709658; cv=pass; d=google.com; s=arc-20160816; b=vARpQDOD1U0fm06C2a7FZEst3Uh1r1ddQj0D7w+FeaYRKLoq7/Z/s6jh5mhHezeCw5 CYLoa8ZxOs2mYrKaKZog1rt7x0CGf0evaIi19oMamuA5fO0U1EIjNXwlDOO/UCeP5UDJ u0InEOzFLLZUHKC8xJ0BVkdZd4lrH454XINeb7rEVlKMnX2npHLiAnEL0Tz3TaH2NZhh 3kmqqz/PbIyXwxrrWiKqn5eM02arj8dKKZSuvTHiNdptIgY9nECUCjMDK9ditkc5DhCs BxrVeyUIwndL8mx7ARi3WNIWR9aotCA/ujT1RQAGBbgV9vPF2EuJ74axsDxClCFzubaa 4vsQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:content-language :in-reply-to:mime-version:user-agent:date:message-id:from:references :cc:to:subject:dkim-signature; bh=FL2uzsbXBWrjP+SIFPUn7wbHLp3iwHEEMohKzHLy/z8=; b=rBwjQ89JgHq3q4Gx1sMygf/vO5XUUEw0AGm026UMEl6v5zh4g9qjwKllXZ9eycygG1 iU46N83mNCwO8FKoQBle3bmOYrjgotAOcLVGngGKvNIGrFNpLao0i2mTsP2aSh05Cr5Q yZDmJAXX1THBWMeYMFyRqnSpq8V6JQHn0NzaWOILHoCKfOaO1OoxcOg5BAC/TN9BvVHP FhEtZLh1wgnkoJO9YaLot4dpZ5SbDBLbsZnLzXyKqr1VlHQWChD2wz9IQmsVra9GaIij 9P53LcyUiCnz41xHJwJcVql1OyVjS9elqrNDGTazm0Pc9uJtRvkul/NM0IgiBLwYhknm xgsw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=KxoO2vng; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id f19si1702576ejl.205.2021.06.03.01.40.35; Thu, 03 Jun 2021 01:40:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=KxoO2vng; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229794AbhFCIjD (ORCPT + 99 others); Thu, 3 Jun 2021 04:39:03 -0400 Received: from mail-dm6nam10on2084.outbound.protection.outlook.com ([40.107.93.84]:40417 "EHLO NAM10-DM6-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S229656AbhFCIjD (ORCPT ); Thu, 3 Jun 2021 04:39:03 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hSH+qaX1SnOlyEmcoVSaA9G3P7nYNQtYbYojObgkB9lRIdTKplfV5MkLgWelTkAVMxmczou9mDPwZQMG2ZPtFSqUC23resEf4Ee68Kkvaz490UNjxkXa4e4g4YLHeovGCOZAOsaQTV9jE9j0tjRAAkQxqQrXYq4Izp3gA/5rn7i6sgBCCh7PEF0tWTs5qTfSEw3xeRb8m8+seFmopC9OKYEL1rZNLafQWcbrMWov+QdvIEbxqMFYJSlxFfNYh9S2GM3oSY5wiNTWKvgV3WATGqe8XmWDpnqqBflUSguCHPe0/FXv5yafzF04AQfrdE1RFmqdx+DsxwmtpiNZ3PABQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FL2uzsbXBWrjP+SIFPUn7wbHLp3iwHEEMohKzHLy/z8=; b=f3557TstwwD2E68SctVrzDnjAvJqaTBbeq0f2KoTeorMcdGJzbvTIOz7clvUTgpUXkqAD/SK1qOb2TfOwDtUHiMsi8a+vUV3fCH0gg8dGDE/vHpno3Q7R84t7QciZ16p91dHbohSvLkCd19ZzdBE+VI0vwCFH+nYBh/LCo3uS/MB8RHyKvVtGKNu3HeYmAdMkWrlTb4qaU9tlbm+ve2AYpHCPIoB4sEH7PAKWKleGHFu6m77Y9/QplTWFc1yJJG5X65z+g8xva9fuuPy75LypmiT3MeYPhgYeMg+Hz4HCsCiBnHt4x/84+poUfJ2PpCWFsRtHdsQJaWPf7e3Z6NFGg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FL2uzsbXBWrjP+SIFPUn7wbHLp3iwHEEMohKzHLy/z8=; b=KxoO2vng76C7u+1AdtI03U8ULyIuuM4r2D/EmAl1t9mH4lZGz44ei3VlDcoHvz5XY639QN4eV4I5KQgm6u93Kz8t7GqwyemMVC8eCT4ZWD4kruYhdw5zPRpc+Dy0Dd2uCzhQWcGCzJ+/h2Cl2jYpJ6ZtivRO7ckVaoQ97qQixa30xqy//qeTt4Ax+Zh0+5kXYRAiYIYRNqOFlrP/a6B6h7Rt9bNktVURRuRQ9fUNaASGDZFZlUP2j9yWLH1ZUZBgD7+SWSnrVU5bGD3Khc2d/eO5yhwKQyoojRj974s7x0Ed541a3XsczZ9gAhIcTZX2AOxJekrkugjb6UEnZ4kgcA== Received: from BN9PR03CA0410.namprd03.prod.outlook.com (2603:10b6:408:111::25) by CY4PR12MB1622.namprd12.prod.outlook.com (2603:10b6:910:f::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4173.22; Thu, 3 Jun 2021 08:37:17 +0000 Received: from BN8NAM11FT054.eop-nam11.prod.protection.outlook.com (2603:10b6:408:111:cafe::a2) by BN9PR03CA0410.outlook.office365.com (2603:10b6:408:111::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4195.15 via Frontend Transport; Thu, 3 Jun 2021 08:37:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT054.mail.protection.outlook.com (10.13.177.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4195.22 via Frontend Transport; Thu, 3 Jun 2021 08:37:16 +0000 Received: from [10.2.95.151] (172.20.187.5) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 3 Jun 2021 08:37:16 +0000 Subject: Re: [PATCH v9 07/10] mm: Device exclusive memory access To: Balbir Singh CC: Andrew Morton , Alistair Popple , , , , , , , , , , , , , , Christoph Hellwig References: <20210524132725.12697-1-apopple@nvidia.com> <20210524132725.12697-8-apopple@nvidia.com> <20210524151157.2dc5d2bb510ff86dc449bf0c@linux-foundation.org> <8844f8c1-d78c-e0f9-c046-592bd75d4c07@nvidia.com> From: John Hubbard Message-ID: Date: Thu, 3 Jun 2021 01:37:16 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: cca83f86-ae35-4faa-f833-08d9266acb74 X-MS-TrafficTypeDiagnostic: CY4PR12MB1622: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5BMoTZa8CIFjTNk7JYTPWu7j/MbG0hR/H09hEWXuCsTOU06b0rKYmw22+dKm+a9/tW2pH89C5Mj/WcSvR6lwO2nfBlmt2COYdMywnkOowk699Icqdkmg7CfHvzeuAAf24WDZinnlE5FNMmo2zHSYGGSkmW6pY4Yl1dGifqg+sYXIwairkBhmymtr4nGwZRVD9lOYQAkGsykKsJe8QcWHCiAvN4j7qU3OWUZAZVdc4fU8yZClhGBvqr5RDjKeNPFPSKHQSSe1JRlGT3Uy2zMIurUBaZ/iu23KpkAPnYgIO3chW7KTOi6OI+0DEoKSYHmCGcstnKmYWUwBCteH9F7dWS0zz+RbaVIr2976zriT2o8l3vuMQg/7y4S4ypSwRTNES+hkIu0KL0odZUF2FC0R+yMPN1NK+VQKLrxj9+ZkYUGEMC+B4QNOmbt+Z1PX/ONQpTn0oTDDG/V1KkJtQFmckbOtzCYXuDWhcEilG8lYdeaDjim+73J+4UtC/Dnkw8hV/yILkyty456NdJIH2kWOzEQVhxjjoAhB75N3evjKyNp4QXQRulunWADWokr8WaIYpg+7ofWNS+6PXgbjFQiCrXqhhwIQFJ5BLEK0AZppzrkiX+81BJLRwkjc0WJL0EidE3X7ABf/1Ay/+hVNl7tnk6+Whq2YUtRQR2/al/EahCk2qgd1W7+Zacw4c6kAQqTt X-Forefront-Antispam-Report: CIP:216.228.112.34;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid03.nvidia.com;CAT:NONE;SFS:(4636009)(396003)(346002)(376002)(136003)(39860400002)(46966006)(36840700001)(36906005)(316002)(16576012)(36860700001)(26005)(82740400003)(31686004)(7416002)(70206006)(478600001)(5660300002)(356005)(6916009)(70586007)(8676002)(83380400001)(47076005)(86362001)(16526019)(186003)(2616005)(336012)(31696002)(426003)(8936002)(82310400003)(4326008)(7636003)(54906003)(2906002)(53546011)(36756003)(43740500002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2021 08:37:16.9716 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cca83f86-ae35-4faa-f833-08d9266acb74 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.34];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT054.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1622 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 6/2/21 1:50 AM, Balbir Singh wrote: ... >>> only impact the address space of programs using the GPU. Should the exclusively >>> marked range live in the unreclaimable list and recycled back to active/in-active >>> to account for the fact that >>> >>> 1. It is not reclaimable and reclaim will only hurt via page faults? >>> 2. It ages the page correctly or at-least allows for that possibility when the >>> page is used by the GPU. >> >> I'm not sure that that is *necessarily* something we can conclude. It depends upon >> access patterns of each program. For example, a "reduction" parallel program sends >> over lots of data to the GPU, and only a tiny bit of (reduced!) data comes back >> to the CPU. In that case, freeing the physical page on the CPU is actually the >> best decision for the OS to make (if the OS is sufficiently prescient). >> > > With a shared device or a device exclusive range, it would be good to get the device > usage pattern and update the mm with that knowledge, so that the LRU can be better Integrating a GPU (or "device") processor and it's mm behavior with the Linux kernel is always an interesting concept. Certainly worth exploring, although it's probably not a small project by any means. > maintained. With your comment you seem to suggest that a page used by the GPU might > be a good candidate for reclaim based on the CPU's understanding of the age of > the page should not account for use by the device > (are GPU workloads - access once and discard?) > Well, that's a little too narrow of an interpretation. The GPU is a fairly general purpose processor, and so it has all kinds of workloads. I'm trying to discourage any hopes that one can know, in advance, precisely how the GPU's pages need to be managed. It's similar to the the CPU, in that regard. My example was just one, out of a vast pool of possible behaviors. thanks, -- John Hubbard NVIDIA