Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8;
Message-ID: <1228d85a-011b-8b48-61c7-fc760edaf7a1@collabora.com>
Date:   Mon, 6 Nov 2023 05:02:56 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.1
Subject: Re: [PATCH v18 25/26] drm/virtio: Support shmem shrinking
To:     Gurchetan Singh <gurchetansingh@chromium.org>
Cc:     David Airlie <airlied@gmail.com>,
        Gerd Hoffmann <kraxel@redhat.com>,
        Chia-I Wu <olvaffe@gmail.com>, Daniel Vetter <daniel@ffwll.ch>,
        Maarten Lankhorst <maarten.lankhorst@linux.intel.com>,
        Maxime Ripard <mripard@kernel.org>,
        Thomas Zimmermann <tzimmermann@suse.de>,
        =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>,
        Qiang Yu <yuq825@gmail.com>,
        Steven Price <steven.price@arm.com>,
        Boris Brezillon <boris.brezillon@collabora.com>,
        Emma Anholt <emma@anholt.net>, Melissa Wen <mwen@igalia.com>,
        dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org,
        kernel@collabora.com, virtualization@lists.linux-foundation.org
References: <20231029230205.93277-1-dmitry.osipenko@collabora.com>
 <20231029230205.93277-26-dmitry.osipenko@collabora.com>
 <CAAfnVBk3gbxVL+WwY9pi27ts8JGPM=0VuOkTo8KOZcuO634UeQ@mail.gmail.com>
Content-Language: en-US
From:   Dmitry Osipenko <dmitry.osipenko@collabora.com>
In-Reply-To: <CAAfnVBk3gbxVL+WwY9pi27ts8JGPM=0VuOkTo8KOZcuO634UeQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 11/4/23 01:55, Gurchetan Singh wrote:
> On Sun, Oct 29, 2023 at 4:03 PM Dmitry Osipenko <
> dmitry.osipenko@collabora.com> wrote:
> 
>> Support generic drm-shmem memory shrinker and add new madvise IOCTL to
>> the VirtIO-GPU driver. BO cache manager of Mesa driver will mark BOs as
>> "don't need" using the new IOCTL to let shrinker purge the marked BOs on
>> OOM, the shrinker will also evict unpurgeable shmem BOs from memory if
>> guest supports SWAP file or partition.
>>
>> Acked-by: Gerd Hoffmann <kraxel@redhat.com>
>> Signed-off-by: Daniel Almeida <daniel.almeida@collabora.com>
>> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
>> ---
>>  drivers/gpu/drm/virtio/virtgpu_drv.h    | 13 +++++-
>>  drivers/gpu/drm/virtio/virtgpu_gem.c    | 35 ++++++++++++++
>>  drivers/gpu/drm/virtio/virtgpu_ioctl.c  | 25 ++++++++++
>>  drivers/gpu/drm/virtio/virtgpu_kms.c    |  8 ++++
>>  drivers/gpu/drm/virtio/virtgpu_object.c | 61 +++++++++++++++++++++++++
>>  drivers/gpu/drm/virtio/virtgpu_vq.c     | 40 ++++++++++++++++
>>  include/uapi/drm/virtgpu_drm.h          | 14 ++++++
>>  7 files changed, 195 insertions(+), 1 deletion(-)
...
> 
> Link to open-source userspace?

https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15278

I'll add it to the commit message

> Also, don't you need a VIRTGPU_PARAM_MADVISE_SUPPORTED or is the plan to
> use a DRM version?

The ioctl() returns error if DRM_VIRTGPU_MADVISE unsupported, extra
flags not needed by userspace

> Overall, the series for a generic shrinker seems useful for a wide variety
> of DRM drivers, such as Panfrost.
> 
> For virtio-gpu, it could be a tad VIRGIL specific: since other context
> types (gfxstream gles, DRM, vk contexts) decrease memory consumption by
> either not allocating shadow buffers for textures/buffers, or using blob
> memory.
> 
> Maybe we need to design with blob in mind, since we expect virgl to be
> deprecated.  On Android, it basically is with ANGLE and native contexts.
> On Linux, Zink looks good too.  Even with memory issues fixed, virgl is
> unattractive compared to those solutions.

We should finish shmem first since started with it, then move to blobs.

My rough idea for the blobs is to use a timer-based approach to avoid
frequent virtio syncing with host that will be bad for performance
otherwise. Virtio-gpu kernel driver will maintain a list of purgeable
blobs and will send the list of idling blobs down to host after a period
of inactivity.

Virgl may be not interesting for CrOS, but Qemu will continue supporting
it. I also expect that today's ARM Chromebooks which use Virgl and only
support GL will continue to use Virgl for the next 4 years.

> Android specific idea: I wonder if we could tie GEM blob buffers usage to
> the lmkd and kill based on that ... ?
> 
> https://source.android.com/docs/core/perf/lmkd
> 
> Is there GEM buffer accounting infrastructure already?

Are you talking about killing a guest APP that uses host blobs when host
is under pressure? I'm not familiar with how arcvm works, but may assume
that it runs a VM per APP. In that case VM is just another process from
the lmkd perspective that will be taken down on OOM, i.e. blob buffers
already should be seen as a part of a VM's memory by lmkd when they
reside in sysmem.

-- 
Best regards,
Dmitry