Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752604AbdHBOOk (ORCPT ); Wed, 2 Aug 2017 10:14:40 -0400 Received: from mga14.intel.com ([192.55.52.115]:28411 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752284AbdHBOOj (ORCPT ); Wed, 2 Aug 2017 10:14:39 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.41,311,1498546800"; d="scan'208";a="135100875" Date: Wed, 2 Aug 2017 22:07:28 +0800 From: Wu Hao To: Alan Tull Cc: Rob Herring , Moritz Fischer , linux-fpga@vger.kernel.org, linux-kernel , "linux-api@vger.kernel.org" , "Kang, Luwei" , "Zhang, Yi Z" , Tim Whisonant , Enno Luebbers , Shiva Rao , Christopher Rauer , "devicetree@vger.kernel.org" Subject: Re: [PATCH v2 02/22] fpga: add FPGA device framework Message-ID: <20170802140728.GB31045@hao-dev> References: <1498441938-14046-1-git-send-email-hao.wu@intel.com> <1498441938-14046-3-git-send-email-hao.wu@intel.com> <20170801084326.GB26227@hao-dev> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 6301 Lines: 155 On Tue, Aug 01, 2017 at 04:04:44PM -0500, Alan Tull wrote: > On Tue, Aug 1, 2017 at 3:43 AM, Wu Hao wrote: > > On Mon, Jul 31, 2017 at 04:40:16PM -0500, Alan Tull wrote: > >> On Thu, Jul 27, 2017 at 2:10 PM, Rob Herring wrote: > >> > On Thu, Jul 27, 2017 at 11:35 AM, Alan Tull wrote: > >> >> On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao wrote: > >> >> > >> >> Hi Rob, > >> >> > >> >> I was hoping to pick your brain a bit on a DT question. > >> >> > >> >>> During FPGA device (e.g PCI-based) discovery, platform devices are > >> >>> registered for different FPGA function units. But the device node path > >> >>> isn't quite friendly to applications. > >> >>> > >> >>> Consider this case, applications want to access child device's sysfs file > >> >>> for some information. > >> >>> > >> >>> 1) Access using bus-based path (e.g PCI) > >> >>> > >> >>> /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file > >> >>> > >> >>> From the path, it's clear which PCI device is the parent, but not perfect > >> >>> solution for applications. PCI device BDF is not fixed, application may > >> >>> need to search all PCI device to find the actual FPGA Device. > >> >>> > >> >>> 2) Or access using platform device path > >> >>> > >> >>> /sys/bus/platform/devices/fpga_func_a.0/sysfs_file > >> >>> > >> >>> Applications find the actual function by name easily, but no information > >> >>> about which fpga device it belongs to. It's quite confusing if multiple > >> >>> FPGA devices are in one system. > >> >> > >> >> There's a proposal for adding sysfs nodes that correspond to each FPGA > >> >> device., with the devices located on each FPGA under them. It makes > >> >> it easier to see which device is on which FPGA. > >> > > >> > Makes sense. > >> > > >> >>> 'FPGA Device' class is introduced to resolve this problem. Each node under > >> >>> this class represents a fpga device, which may have one or more child > >> >>> devices. Applications only need to search under this FPGA Device class > >> >>> folder to find the child device node it needs. > >> >>> > >> >>> For example, for the platform has 2 fpga devices, each fpga device has > >> >>> 3 child devices, the hierarchy looks like this. > >> >>> > >> >>> Two nodes are under /sys/class/fpga/: > >> >>> /sys/class/fpga/fpga.0 > >> >>> /sys/class/fpga/fpga.1 > >> >>> > >> >>> Each node has 1 function A device and 2 function B devices: > >> >>> /sys/class/fpga/fpga.0/func_a.0 > >> >>> /sys/class/fpga/fpga.0/func_b.0 > >> >>> /sys/class/fpga/fpga.0/func_b.1 > >> >>> > >> >>> /sys/class/fpga/fpga.1/func_a.1 > >> >>> /sys/class/fpga/fpga.1/func_b.2 > >> >>> /sys/class/fpga/fpga.1/func_b.3 > >> > > >> > A class is generally what is the function of the device, not how it is > >> > attached. Seems like what you want here is a new bus type if the > >> > existing PCI and platform bus types don't work. > >> > > >> >> > >> >> I can see the value of having sysfs nodes that correspond to fpga > >> >> devices and being able to find devices under them. I'm thinking what > >> >> that would mean for Device Tree when fpga-dev is used on DT enabled > >> >> systems. In Device Tree, what is a fpga-dev? > >> > > >> > Just properly setting the parent struct device on the functions should > >> > be enough to figure out which function is in which fpga. I don't see > >> > why a new class is needed. > >> > > >> >> Currently the DT would have a FPGA bridge corresponding to each FPGA's > >> >> hardware bridge and a heirarchy of bridges, regions and devices under > >> >> it. On systems that don't support partial reconfiguration under the > >> >> OS (so not main bridge that was controlled by the OS), there would be > >> >> a FPGA region, then its child regions, bridges, and devices. > >> > > >> > The FPGA bridges could instantiate fpga bus type devices instead of > >> > platform devices. > >> > >> Yes > >> > >> Some FPGA use cases already have a base bridge per FPGA that could > >> serve as this bus. But this use case has a static FPGA image + > >> reprogrammable child fpga regions. There's no base bridge under Linux > >> since the FPGA was programmed and the bridge enabled before Linux > >> boots. An added base bridge that doesn't touch hardware will be > >> required for this type of use. > > > > Hi Alan > > > > Does 'base bridge' mentioned above mean a hardware bridge just like > > PCIe or USB? > > Whatever connects each FPGA to the CPU. One base bridge per FPGA > device to create the fpga bus type devices. Each PR region's bridge > would also be a bus. > > > > > I tried to use fpga bus type device instead of fpga-dev class today, > > it works for me, e.g Intel FPGA device PCIe driver could create a > > fpga bus type dev as a child of PCIe device and its sysfs path will be > > changed to /sys/bus/fpga/devices/fpga.x/ from /sys/class/fpga/fpga.x/. > > For now, this fpga bus type device is only used as container device, > > so no driver needed for it. > > That's great! I'd like to see the code to try it out with device > tree. Is it part of fpga-bridge or something separate for now? > Hi Alan I just sent the patch I did as a RFC Patch[1] to the mailing list. Please take a look. I only replaced the original fpga-dev class with new 'fpga' bus type, and keep the original interface not changed. [1] http://marc.info/?l=linux-fpga&m=150167682312708&w=2 > > > > Do you have any concern on this? I see fpga bus type works fine, but > > I didn't see other advantages for this case, as we only use it as a > > container device to represent a FPGA device in sysfs hierarchy. :) > > I could not see a way to make the fpga-dev class compatible with the > FPGA Device Tree bindings. This was a red flag. That's why I asked > Rob's opinion. Sysfs classes collect devices of a specific type > together; busses describe topology. I think the goal of fpga-dev was > to describe topology. It's more correct to define this as a bus, not > a class. If it's done right, it can work for device tree also. Got it. Thanks. :) Hao > > Alan > > > > > Thanks > > Hao > > > >> > >> > That's really up to Linux and outside the scope of > >> > the bindings. > >> > >> Thanks for the feedback. > >> > >> Alan Tull > >> > >> > > >> > Rob