Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
To:     Cornelia Huck <cohuck@redhat.com>, <alex.williamson@redhat.com>
CC:     <kvm@vger.kernel.org>, Ohad Ben-Cohen <ohad@wizery.com>,
        Bjorn Andersson <bjorn.andersson@linaro.org>,
        Mathieu Poirier <mathieu.poirier@linaro.org>,
        linux-remoteproc <linux-remoteproc@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "Vutla, Lokesh" <lokeshvutla@ti.com>,
        Vignesh Raghavendra <vigneshr@ti.com>,
        "Strashko, Grygorii" <grygorii.strashko@ti.com>
From:   Kishon Vijay Abraham I <kishon@ti.com>
Subject: [QUERY] Flushing cache from userspace using VFIO
Message-ID: <d338414f-ed88-20d4-7da0-6742dedb8579@ti.com>
Date:   Mon, 6 Sep 2021 21:22:15 +0530
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Precedence: bulk

Hi Alex, Cornelia,

I'm trying to see if I can use VFIO (Versatile Framework for userspace I/O
[1]) for communication between two cores within the same SoC. I've tried to put
down a picture like below which tries to communicate between ARM64 (running
Linux) and CORTEX R5 (running firmware). It uses rpmsg/remoteproc for the
control messages and the actual data buffers are directly accessed from the
userspace. The location of the data buffers can be informed to the userspace via
rpmsg_vfio (which has to be built as a rpmsg endpoint).

My question is after the userspace application in ARM64 writes to a buffer in
the SYSTEM MEMORY, can it flush it (through a VFIO IOCTL) before handing the
buffer to the CORTEX R5.

If it's implemented within kernel either we use dma_alloc_coherent() for
allocating coherent memory or streaming DMA APIs like
dma_map_single()/dma_unmap_single() for flushing/invalidate the cache.

Trying to see if that is already supported in VFIO or if not, would it be
acceptable to implement it.

Please let me know your thoughts.

┌───────────────────────────────────────────────────────────────────────────┐
│                                                                           │
│ ┌────────────────────┐                                                    │
│ │                    │                                                    │
│ │ ┌──────────────┐   │                                                    │
│ │ │ userspace    │   │  Data Buffers                                      │
│ │ │ Application  ├───┼──────────────┐                                     │
│ │ │              │   │              │                                     │
│ │ └──────▲──┬────┘   │              │                                     │
│ │        │  │        │              │                                     │
│ │        │  │ user   │              │                 ┌─────────────────┐ │
│ │  ──────┼──┼─────── │              │                 │                 │ │
│ │        │  │ kernel │              │                 │                 │ │
│ │  ┌─────┴──▼────┐   │    ┌─────────┼────────────┐    │                 │ │
│ │  │             │   │    │         │            │    │  Data           │ │
│ │  │  rpmsg_vfio │   │    │  ┌──────▼─────────┐  │    │  Buffers        │ │
│ │  │             │   │    │  │ Reserved Region◄──┼────┼────────┐        │ │
│ │  └─────▲──┬────┘   │    │  │                │  │    │        │        │ │
│ │        │  │        │    │  └────────────────┘  │    │        │        │ │
│ │  ┌─────┴──▼────┐   │    │                      │    │        │        │ │
│ │  │             │   │    │                      │    │ ┌──────┴──────┐ │ │
│ │  │  rpmsg      │   │    │     SYSTEM MEMORY    │    │ │ Application │ │ │
│ │  │             │   │    │       (DDR)          │    │ │   Logic     │ │ │
│ │  └─────▲──┬────┘   │    └──────────────────────┘    │ └───▲────┬────┘ │ │
│ │        │  │        │                                │     │    │      │ │
│ │  ┌─────┴──▼────┐   │Notify Firmware/Control Message │ ┌───┴────▼────┐ │ │
│ │  │             ├───┼────────────────────────────────┼─►             │ │ │
│ │  │  remoteproc │   │Interrupt ARM/Control Message   │ │   Firmware  │ │ │
│ │  │             ◄───┼────────────────────────────────┼─┤             │ │ │
│ │  └─────────────┘   │                                │ └─────────────┘ │ │
│ │      ARM64(Linux)  │                                │   ARM CORTEX R5 │ │
│ └────────────────────┘                                └─────────────────┘ │
│                                                                           │
│                                                                       SoC │
└───────────────────────────────────────────────────────────────────────────┘

Thank You,
Kishon

[1] -> https://youtu.be/WFkdTFTOTpA