Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH 8/9] vfio/pci: use x86 naming instead of igd
To:     Jason Gunthorpe <jgg@nvidia.com>
Cc:     Max Gurtovoy <mgurtovoy@nvidia.com>,
        Cornelia Huck <cohuck@redhat.com>,
        Alex Williamson <alex.williamson@redhat.com>,
        Matthew Rosato <mjrosato@linux.ibm.com>, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, liranl@nvidia.com, oren@nvidia.com,
        tzahio@nvidia.com, leonro@nvidia.com, yarong@nvidia.com,
        aviadye@nvidia.com, shahafs@nvidia.com, artemp@nvidia.com,
        kwankhede@nvidia.com, ACurrid@nvidia.com, gmataev@nvidia.com,
        cjia@nvidia.com, yishaih@nvidia.com
References: <20210201181454.22112b57.cohuck@redhat.com>
 <599c6452-8ba6-a00a-65e7-0167f21eac35@linux.ibm.com>
 <20210201114230.37c18abd@omen.home.shazbot.org>
 <20210202170659.1c62a9e8.cohuck@redhat.com>
 <a413334c-3319-c6a3-3d8a-0bb68a10b9c1@nvidia.com>
 <806c138e-685c-0955-7c15-93cb1d4fe0d9@ozlabs.ru>
 <34be24e6-7f62-9908-c56d-9e469c3b6965@nvidia.com>
 <83ef0164-6291-c3d1-0ce5-2c9d6c97469e@ozlabs.ru>
 <20210204125123.GI4247@nvidia.com>
 <d69a7f3c-f552-cd25-4e15-3e894f4eb15a@ozlabs.ru>
 <20210208181313.GH4247@nvidia.com>
From:   Alexey Kardashevskiy <aik@ozlabs.ru>
Message-ID: <7c5bf697-86a6-5147-061c-7a7c38b6897c@ozlabs.ru>
Date:   Tue, 9 Feb 2021 12:51:59 +1100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:85.0) Gecko/20100101
 Thunderbird/85.0
MIME-Version: 1.0
In-Reply-To: <20210208181313.GH4247@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 09/02/2021 05:13, Jason Gunthorpe wrote:
> On Fri, Feb 05, 2021 at 11:42:11AM +1100, Alexey Kardashevskiy wrote:
>>> A real nvswitch function?
>>
>> What do you mean by this exactly? The cpu side of nvlink is "emulated pci
>> devices", the gpu side is not in pci space at all, the nvidia driver manages
>> it via the gpu's mmio or/and cfg space.
> 
> Some versions of the nvswitch chip have a PCI-E link too, that is what
> I though this was all about when I first saw it.

> So, it is really a special set of functions for NVIDIA GPU device
> assignment only applicable to P9 systems, much like IGD is for Intel
> on x86.


These GPUs are not P9 specific and they all have both PCIe and NVLink2 
links. The special part is that some nvlinks are between P9 and GPU and 
the rest are between GPUs, everywhere else (x86, may be ARM) the nvlinks 
are used between GPUs but even there I do not think the nvlink logic is 
presented to the host in the PCI space.


-- 
Alexey