Received: by 2002:a05:6a10:8c0a:0:0:0:0 with SMTP id go10csp3962534pxb;
        Mon, 8 Feb 2021 04:47:43 -0800 (PST)
X-Google-Smtp-Source: ABdhPJzxHbksd68XUUQ4ONubGPAtpr0Zb5AzdIPVPUSLXR1b0gi+4LA7INiza6ZU++B5gzBBoWW+
X-Received: by 2002:a17:906:2299:: with SMTP id p25mr16916986eja.279.1612788463369;
        Mon, 08 Feb 2021 04:47:43 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; t=1612788463; cv=none;
        d=google.com; s=arc-20160816;
        b=Ak/gLr/bMiLiDdec4D3gthKDdK4ODb6ePG0z5K8xprsReEVL5ZH/a9P8uLhaVoO9lB
         rD/ULJlNo6HpaqN6jq6EHvwuca1sXaVFR1UVNOsEfWF1jSXrlvf14Nvpqfk+2mg3T2Zy
         8vOqAAf5odQy6sRyr225TSQ0ZWwV0MUQZxfzrItVI8WqNBXvEiTYxXDqjPidicvph+C6
         6eGj2XB9YGq8T3SRDMRG0D+DEd4Buk3J4geHCiTYWBIfmiYuP8o3/jkHhHzILrfOZP+w
         bjDUQ8E4qCg5JRVtRCODyWe/U1Or41c4KEn+qCT9biP8fF9xSzaRXswiuIOkTQeZCMyS
         VJWg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=list-id:precedence:dkim-signature:content-language
         :content-transfer-encoding:in-reply-to:mime-version:user-agent:date
         :message-id:from:references:cc:to:subject;
        bh=+JrPIiWybx2rgSx+ma5GOztR5YPtENuEAC7IyP6QYiQ=;
        b=aMDUz2+5QIW2IFUuxDPpNTs9u6pm8rTCPyQpbS18EPSjjXojbyfX6923Ts0JqTQzY0
         izrdrziC8hOSskKMXjoDLI+/Iz8StWgON1LaflJlyxkKXFp97Vt810kOMgFPhQMpwvDX
         uDabxL5zcoJz4ECN9oKX70uKol4EXXpbbddxSc65JJpuHCZAGKWKiyufFTFOndrGo0wE
         iks+XYsnEEvq1QEiBv8b7usFu8t+j0sZ7T8s5wXl6Ha2P0/cA5pIYCUzrmFIdZTOVLPp
         Yv2mtuhEjcN8eFVOyRm6gPNtfPKnQaRUFdraFnlA2QCMzrP/UqBP+0xMqd1xgoELyWW2
         +loQ==
ARC-Authentication-Results: i=1; mx.google.com;
       dkim=pass header.i=@nvidia.com header.s=n1 header.b=mOxLdd2F;
       spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org;
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18])
        by mx.google.com with ESMTP id b4si11382564edk.452.2021.02.08.04.47.17;
        Mon, 08 Feb 2021 04:47:43 -0800 (PST)
Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@nvidia.com header.s=n1 header.b=mOxLdd2F;
       spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org;
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S232348AbhBHMpz (ORCPT <rfc822;lkml.email@gmail.com> + 99 others);
        Mon, 8 Feb 2021 07:45:55 -0500
Received: from hqnvemgate25.nvidia.com ([216.228.121.64]:3485 "EHLO
        hqnvemgate25.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S232339AbhBHMpg (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Mon, 8 Feb 2021 07:45:36 -0500
Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, AES256-SHA)
        id <B602132440001>; Mon, 08 Feb 2021 04:44:52 -0800
Received: from [172.27.1.173] (172.20.145.6) by HQMAIL107.nvidia.com
 (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Mon, 8 Feb
 2021 12:44:46 +0000
Subject: Re: [PATCH 8/9] vfio/pci: use x86 naming instead of igd
To:     Alexey Kardashevskiy <aik@ozlabs.ru>,
        Jason Gunthorpe <jgg@nvidia.com>
CC:     Cornelia Huck <cohuck@redhat.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        Matthew Rosato <mjrosato@linux.ibm.com>, <kvm@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <liranl@nvidia.com>,
        <oren@nvidia.com>, <tzahio@nvidia.com>, <leonro@nvidia.com>,
        <yarong@nvidia.com>, <aviadye@nvidia.com>, <shahafs@nvidia.com>,
        <artemp@nvidia.com>, <kwankhede@nvidia.com>, <ACurrid@nvidia.com>,
        <gmataev@nvidia.com>, <cjia@nvidia.com>, <yishaih@nvidia.com>
References: <20210201162828.5938-1-mgurtovoy@nvidia.com>
 <20210201162828.5938-9-mgurtovoy@nvidia.com>
 <20210201181454.22112b57.cohuck@redhat.com>
 <599c6452-8ba6-a00a-65e7-0167f21eac35@linux.ibm.com>
 <20210201114230.37c18abd@omen.home.shazbot.org>
 <20210202170659.1c62a9e8.cohuck@redhat.com>
 <a413334c-3319-c6a3-3d8a-0bb68a10b9c1@nvidia.com>
 <806c138e-685c-0955-7c15-93cb1d4fe0d9@ozlabs.ru>
 <34be24e6-7f62-9908-c56d-9e469c3b6965@nvidia.com>
 <83ef0164-6291-c3d1-0ce5-2c9d6c97469e@ozlabs.ru>
 <20210204125123.GI4247@nvidia.com>
 <d69a7f3c-f552-cd25-4e15-3e894f4eb15a@ozlabs.ru>
From:   Max Gurtovoy <mgurtovoy@nvidia.com>
Message-ID: <5bf0b8f7-38fb-bfdf-d093-d8d9b28b9679@nvidia.com>
Date:   Mon, 8 Feb 2021 14:44:14 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.7.1
MIME-Version: 1.0
In-Reply-To: <d69a7f3c-f552-cd25-4e15-3e894f4eb15a@ozlabs.ru>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
X-Originating-IP: [172.20.145.6]
X-ClientProxiedBy: HQMAIL107.nvidia.com (172.20.187.13) To
 HQMAIL107.nvidia.com (172.20.187.13)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1;
        t=1612788292; bh=+JrPIiWybx2rgSx+ma5GOztR5YPtENuEAC7IyP6QYiQ=;
        h=Subject:To:CC:References:From:Message-ID:Date:User-Agent:
         MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:
         Content-Language:X-Originating-IP:X-ClientProxiedBy;
        b=mOxLdd2FAPZCQMEGx7pVfF0ZFPK0Id0n7fGaNKe8iR8/YbtntxI985Q6lr7hn/uZf
         uXaoZJOlRhlFLl7/c4sW5YpPj3+9dGx5P1IRLDY+bkA+YF3lzI4+7o7DZBp5JwRE1k
         +FTilFUDsMMnjaTbWTX2bo4cwD6AXODBmrNnCY53P/1kZu+1cIM/NUTLqX9q2/sh5G
         LIdtNUIkBjQSuef/8GE5f7Si4NEdTU8x6JACeg120x6luYRA/0/9t3/L1GUI2rtR47
         QQyHxK/TbS5pnB5lSxTGi6WNwpeWVuxtlqahhCmbvuL+bpsrJhCr6EyNl/RgOlxM4c
         ugSyl9hoRmuMA==
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 2/5/2021 2:42 AM, Alexey Kardashevskiy wrote:
>
>
> On 04/02/2021 23:51, Jason Gunthorpe wrote:
>> On Thu, Feb 04, 2021 at 12:05:22PM +1100, Alexey Kardashevskiy wrote:
>>
>>> It is system firmware (==bios) which puts stuff in the device tree. The
>>> stuff is:
>>> 1. emulated pci devices (custom pci bridges), one per nvlink, 
>>> emulated by
>>> the firmware, the driver is "ibmnpu" and it is a part on the nvidia 
>>> driver;
>>> these are basically config space proxies to the cpu's side of nvlink.
>>> 2. interconnect information - which of 6 gpus nvlinks connected to 
>>> which
>>> nvlink on the cpu side, and memory ranges.
>>
>> So what is this vfio_nvlink driver supposed to be bound to?
>>
>> The "emulated pci devices"?
>
> Yes.
>
>> A real GPU function?
>
> Yes.
>
>> A real nvswitch function?
>
> What do you mean by this exactly? The cpu side of nvlink is "emulated 
> pci devices", the gpu side is not in pci space at all, the nvidia 
> driver manages it via the gpu's mmio or/and cfg space.
>
>> Something else?
>
> Nope :)
> In this new scheme which you are proposing it should be 2 drivers, I 
> guess.

I see.

So should it be nvidia_vfio_pci.ko ? and it will do the NVLINK stuff in 
case the class code matches and otherwise just work as simple vfio_pci GPU ?

What about the second driver ? should it be called ibmnpu_vfio_pci.ko ?

>
>>
>> Jason
>>
>