=======================================================================
| Introduction |
=======================================================================
This document provides a detailed overview of the proposed Kernel feature for FPGA Manager subsystem user interface.
It describes the problem statement behind the proposal, the problem to be solved, a top-level solution design.
Table of Contents:
------------------
A. Problem Statement and Background
B. Scope and Out of scope of the proposal
B.1 Scope
B.2 Out of scope
C. Proposed Solution
D. Proposed User Interface Details
=======================================================================
| A. Problem Statement and Background |
=======================================================================
The existing FPGA manager subsystem didn't have any user space interface (other than the status/state in sysfs) in Kernel.
Basically, FPGAs are semiconductor devices that can be reprogrammed for desired hardware functionality.
FPGAs can be reprogrammed at runtime with different types of logic and IPs as per user need and hence there is a need to use device tree overlays for removing/updating/adding the devices at runtime for the IPs/controllers that are present in FPGA.
But we don't have any user interface in kernel for updating the device tree at runtime.
Sometime back there was a series sent by Pantelis Antoniou (https://lore.kernel.org/lkml/[email protected]/).
This patch introduced a user interface configfs for Device Tree overlays, a method of dynamically altering the kernel's live Device Tree. However, this patch series was not accepted in mainline due to various concerns.
For more details refer to this link: https://elinux.org/Frank%27s_Evolving_Overlay_Thoughts#issues_and_what_needs_to_be_completed_--_Not_an_exhaustive_list
One of the major valid concerns that were raised with this configfs interface was security as it opens up the interface to users for modifying the live device tree.
So, in order to configure/program the FPGA devices, All the major vendors of FPGA are using this configfs series as out-of-tree patch for configuring the FPGAs
and there was never an attempt to introduce a generic interface to configure/program the FPGA in upstream and hence upstream kernel ended up in not having proper support for FPGAs.
The proposal below tries to address this gap of FPGA programmability by providing an interface to the user.
=======================================================================
| B. Proposed Solution |
=======================================================================
The proposed interface adds a new sysfs interface (of-fpga-region.c) as part of the fpga subsystem and it is responsible for supporting the below functionalities.
--> Provide the user interface for the FPGA subsystem to handle the below FPGA relevant stuff.
- Bridges.
- FPGA Configuration.
- Driver - Probe/Remove
--> The new sysfs interface uses Device Tree overlay (DTO) files to configure/ reprogram an FPGA while an operating system is running.
- Restrict the overlay's subsystem usage only to FPGA regions in order to mitigate the major security concern with configfs.
- Do validation checks on the user provided DTO files.
- If the user provided DTO doesn't target an FPGA Region which is already part of the running kernel, then return -INVALID error.
- If the DTO file contains multiple targets, then return -INVALID error.
- It will allow only Child nodes which are part of targeted FPGA Region.
- It avoids Overlay notification calls . So that it will not interrupt the other subsystem's(Like; GPIO, I2C.....etc) exists in the kernel.
-->This proposed solution will not change the existing sequence When a
-->DT overlay that targets an FPGA Region is applied
- The FPGA Region will do the following:
- 1. Disable appropriate FPGA bridges.
- 2. Program the FPGA using the FPGA manager.
- 3. Enable the FPGA bridges.
- 4. The Device Tree overlay is accepted into the live tree.
- 5. Child devices are populated.
- When the overlay is removed, the child nodes will be removed, and the FPGA Region will disable the bridges.
. --------------------------------------. .-----------------------------------------.
| | | |
| .------------------| |---------------------. |
| | sysfs_load() |<=======> |Overaly_apply()| |
.---------------------------------. | '------------------| |---------------------' |
| | | | | |
| New Sysfs interface | ====> | of-fpga-region .c | | DT Overlay.c |
| load/unload | | | | |
'--------------------------------' | .---------------------| |-------------------------. |
| | sysfs_unload() |<=======> | Overlay_remove() | |
| '-------------------- | |-------------------------' |
| | | |
'----------------------------------- --' '------------------------------------------'
=======================================================================
| D. Proposed User Interface Details |
=======================================================================
How to use the fpga sysfs interface.
To load Image:
- echo "DTBO file" > /sys/class/of-fpga_region/<region>/load
To unload Image:
- /sys/class/of-fpga_region/<region>/unload
To get the image status (Load/Unload):
- cat /sys/class/of-fpga_region/<region>/status
Base Image
- Also called the "static image"
- An FPGA image that is designed to do full reconfiguration of the FPGA.
- A base image may set up a set of partial reconfiguration regions that may later be reprogrammed.
.-----------------------. .--------------------------------------------.
| Host CPU | | FPGA |
| | | |
| -- -| | ----------- --------- |
| | H | | |==>| Bridge0 |<==>| PRR0 | |
| | W| | | ----------- -------- |
| | | | | |
| | B |<=====> |<== | ----------- -------- |
| | R | | |==>| Bridge1 |<==>| PRR1| |
| | I | | | ----------- -------- |
| | D | | | |
| | G | | | ----------- ------- |
| | E | | |==>| Bridge2 |<==>| PRR2 ||
| ----| | ----------- -------- |
| | | |
'-----------------------' '---------------------------------------------'
In the above diagram a typical FPGA is setup with a base image that created three regions.
Each region (PRR0 - 2) gets its own split of the busses that is independently gated by a soft logic bridge (Bridge0 - 2) in the FPGA.
The contents of each PRR can be reprogrammed independently while the rest of the system continues to function.
Form the above tropology the sysfs interface looks like as follows.
For Base/static region:
To load Image:
- echo "DTBO file" > /sys/class/of-fpga_region/FPGA/load
To unload Image:
- /sys/class/of-fpga_region/FPGA/unload
To get the image status (Load/Unload):
- cat /sys/class/of-fpga_region/FPGA/status
For PRR0:
To load Image:
- echo "DTBO file" > /sys/class/of-fpga_region/PRR0/load
To unload Image:
- /sys/class/of-fpga_region/PRR0/unload
To get the image status (Load/Unload):
- cat /sys/class/of-fpga_region/PRR0/status
For PRR1:
To load Image:
- echo "DTBO file" > /sys/class/of-fpga_region/PRR1/load
To unload Image:
- /sys/class/of-fpga_region/PRR1/unload
To get the image status (Load/Unload):
- cat /sys/class/of-fpga_region/PRR1/status
For PRR1:
To load Image:
- echo "DTBO file" > /sys/class/of-fpga_region/PRR1/load
To unload Image:
- /sys/class/of-fpga_region/PRR1/unload
To get the image status (Load/Unload):
- cat /sys/class/of-fpga_region/PRR1/status
On Thu, Jan 04, 2024 at 04:52:15AM +0000, Manne, Nava kishore wrote:
> =======================================================================
> | Introduction |
> =======================================================================
> This document provides a detailed overview of the proposed Kernel feature for FPGA Manager subsystem user interface.
> It describes the problem statement behind the proposal, the problem to be solved, a top-level solution design.
>
> Table of Contents:
> ------------------
> A. Problem Statement and Background
> B. Scope and Out of scope of the proposal
> B.1 Scope
> B.2 Out of scope
> C. Proposed Solution
> D. Proposed User Interface Details
> =======================================================================
> | A. Problem Statement and Background |
> =======================================================================
> The existing FPGA manager subsystem didn't have any user space interface (other than the status/state in sysfs) in Kernel.
> Basically, FPGAs are semiconductor devices that can be reprogrammed for desired hardware functionality.
> FPGAs can be reprogrammed at runtime with different types of logic and IPs as per user need and hence there is a need to use device tree overlays for removing/updating/adding the devices at runtime for the IPs/controllers that are present in FPGA.
> But we don't have any user interface in kernel for updating the device tree at runtime.
Can you line-wrap your emails to make them easier to read?
Also, something went wrong here:
> . --------------------------------------. .-----------------------------------------.
> | | | |
> | .------------------| |---------------------. |
> | | sysfs_load() |<=======> |Overaly_apply()| |
> .---------------------------------. | '------------------| |---------------------' |
> | | | | | |
> | New Sysfs interface | ====> | of-fpga-region .c | | DT Overlay.c |
> | load/unload | | | | |
> '--------------------------------' | .---------------------| |-------------------------. |
> | | sysfs_unload() |<=======> | Overlay_remove() | |
> | '-------------------- | |-------------------------' |
> | | | |
> '----------------------------------- --' '------------------------------------------'
And here:
> .-----------------------. .--------------------------------------------.
> | Host CPU | | FPGA |
> | | | |
> | -- -| | ----------- --------- |
> | | H | | |==>| Bridge0 |<==>| PRR0 | |
> | | W| | | ----------- -------- |
> | | | | | |
> | | B |<=====> |<== | ----------- -------- |
> | | R | | |==>| Bridge1 |<==>| PRR1| |
> | | I | | | ----------- -------- |
> | | D | | | |
> | | G | | | ----------- ------- |
> | | E | | |==>| Bridge2 |<==>| PRR2 ||
> | ----| | ----------- -------- |
> | | | |
> '-----------------------' '---------------------------------------------'
Your spaces all seem too big, making this really hard to read.
But really, we need to see some code before we can go further.
"proposals" are great, but that's not how kernel development works, we
need actual patches to be able to evaluate anything, as that shows you
at least have a working implementation to start with, and that you have
resolved any of the obvious issues ahead of time.
thanks,
greg k-h
On Thu, Jan 04, 2024 at 04:52:15AM +0000, Manne, Nava kishore wrote:
> =======================================================================
> | Introduction |
> =======================================================================
> This document provides a detailed overview of the proposed Kernel feature for FPGA Manager subsystem user interface.
> It describes the problem statement behind the proposal, the problem to be solved, a top-level solution design.
>
> Table of Contents:
> ------------------
> A. Problem Statement and Background
> B. Scope and Out of scope of the proposal
> B.1 Scope
> B.2 Out of scope
> C. Proposed Solution
> D. Proposed User Interface Details
> =======================================================================
> | A. Problem Statement and Background |
> =======================================================================
> The existing FPGA manager subsystem didn't have any user space interface (other than the status/state in sysfs) in Kernel.
> Basically, FPGAs are semiconductor devices that can be reprogrammed for desired hardware functionality.
> FPGAs can be reprogrammed at runtime with different types of logic and IPs as per user need and hence there is a need to use device tree overlays for removing/updating/adding the devices at runtime for the IPs/controllers that are present in FPGA.
> But we don't have any user interface in kernel for updating the device tree at runtime.
>
> Sometime back there was a series sent by Pantelis Antoniou (https://lore.kernel.org/lkml/[email protected]/).
> This patch introduced a user interface configfs for Device Tree overlays, a method of dynamically altering the kernel's live Device Tree. However, this patch series was not accepted in mainline due to various concerns.
> For more details refer to this link: https://elinux.org/Frank%27s_Evolving_Overlay_Thoughts#issues_and_what_needs_to_be_completed_--_Not_an_exhaustive_list
>
> One of the major valid concerns that were raised with this configfs interface was security as it opens up the interface to users for modifying the live device tree.
>
> So, in order to configure/program the FPGA devices, All the major vendors of FPGA are using this configfs series as out-of-tree patch for configuring the FPGAs
> and there was never an attempt to introduce a generic interface to configure/program the FPGA in upstream and hence upstream kernel ended up in not having proper support for FPGAs.
>
> The proposal below tries to address this gap of FPGA programmability by providing an interface to the user.
>
> =======================================================================
> | B. Proposed Solution |
> =======================================================================
> The proposed interface adds a new sysfs interface (of-fpga-region.c) as part of the fpga subsystem and it is responsible for supporting the below functionalities.
Why only for of-fpga-region? There are also FPGA regions that don't rely
on OF. My quick idea is that an interface for /sys/class/fpga-region/
and OF could be one of the implementation.
Thanks,
Yilun
> --> Provide the user interface for the FPGA subsystem to handle the below FPGA relevant stuff.
> - Bridges.
> - FPGA Configuration.
> - Driver - Probe/Remove
>
> --> The new sysfs interface uses Device Tree overlay (DTO) files to configure/ reprogram an FPGA while an operating system is running.
> - Restrict the overlay's subsystem usage only to FPGA regions in order to mitigate the major security concern with configfs.
> - Do validation checks on the user provided DTO files.
> - If the user provided DTO doesn't target an FPGA Region which is already part of the running kernel, then return -INVALID error.
> - If the DTO file contains multiple targets, then return -INVALID error.
> - It will allow only Child nodes which are part of targeted FPGA Region.
> - It avoids Overlay notification calls . So that it will not interrupt the other subsystem's(Like; GPIO, I2C.....etc) exists in the kernel.
>
> -->This proposed solution will not change the existing sequence When a
> -->DT overlay that targets an FPGA Region is applied
> - The FPGA Region will do the following:
> - 1. Disable appropriate FPGA bridges.
> - 2. Program the FPGA using the FPGA manager.
> - 3. Enable the FPGA bridges.
> - 4. The Device Tree overlay is accepted into the live tree.
> - 5. Child devices are populated.
> - When the overlay is removed, the child nodes will be removed, and the FPGA Region will disable the bridges.
>
> . --------------------------------------. .-----------------------------------------.
> | | | |
> | .------------------| |---------------------. |
> | | sysfs_load() |<=======> |Overaly_apply()| |
> .---------------------------------. | '------------------| |---------------------' |
> | | | | | |
> | New Sysfs interface | ====> | of-fpga-region .c | | DT Overlay.c |
> | load/unload | | | | |
> '--------------------------------' | .---------------------| |-------------------------. |
> | | sysfs_unload() |<=======> | Overlay_remove() | |
> | '-------------------- | |-------------------------' |
> | | | |
> '----------------------------------- --' '------------------------------------------'
> =======================================================================
> | D. Proposed User Interface Details |
> =======================================================================
> How to use the fpga sysfs interface.
>
> To load Image:
> - echo "DTBO file" > /sys/class/of-fpga_region/<region>/load
>
> To unload Image:
> - /sys/class/of-fpga_region/<region>/unload
>
> To get the image status (Load/Unload):
> - cat /sys/class/of-fpga_region/<region>/status
>
> Base Image
> - Also called the "static image"
> - An FPGA image that is designed to do full reconfiguration of the FPGA.
> - A base image may set up a set of partial reconfiguration regions that may later be reprogrammed.
>
> .-----------------------. .--------------------------------------------.
> | Host CPU | | FPGA |
> | | | |
> | -- -| | ----------- --------- |
> | | H | | |==>| Bridge0 |<==>| PRR0 | |
> | | W| | | ----------- -------- |
> | | | | | |
> | | B |<=====> |<== | ----------- -------- |
> | | R | | |==>| Bridge1 |<==>| PRR1| |
> | | I | | | ----------- -------- |
> | | D | | | |
> | | G | | | ----------- ------- |
> | | E | | |==>| Bridge2 |<==>| PRR2 ||
> | ----| | ----------- -------- |
> | | | |
> '-----------------------' '---------------------------------------------'
>
> In the above diagram a typical FPGA is setup with a base image that created three regions.
> Each region (PRR0 - 2) gets its own split of the busses that is independently gated by a soft logic bridge (Bridge0 - 2) in the FPGA.
> The contents of each PRR can be reprogrammed independently while the rest of the system continues to function.
>
> Form the above tropology the sysfs interface looks like as follows.
>
> For Base/static region:
> To load Image:
> - echo "DTBO file" > /sys/class/of-fpga_region/FPGA/load
>
> To unload Image:
> - /sys/class/of-fpga_region/FPGA/unload
>
> To get the image status (Load/Unload):
> - cat /sys/class/of-fpga_region/FPGA/status
>
> For PRR0:
> To load Image:
> - echo "DTBO file" > /sys/class/of-fpga_region/PRR0/load
>
> To unload Image:
> - /sys/class/of-fpga_region/PRR0/unload
>
> To get the image status (Load/Unload):
> - cat /sys/class/of-fpga_region/PRR0/status
>
> For PRR1:
> To load Image:
> - echo "DTBO file" > /sys/class/of-fpga_region/PRR1/load
>
> To unload Image:
> - /sys/class/of-fpga_region/PRR1/unload
>
> To get the image status (Load/Unload):
> - cat /sys/class/of-fpga_region/PRR1/status
>
> For PRR1:
> To load Image:
> - echo "DTBO file" > /sys/class/of-fpga_region/PRR1/load
>
> To unload Image:
> - /sys/class/of-fpga_region/PRR1/unload
>
> To get the image status (Load/Unload):
> - cat /sys/class/of-fpga_region/PRR1/status
>
Hi Yilun,
Please find my response inline.
> -----Original Message-----
> From: Xu Yilun <[email protected]>
> Sent: Monday, January 8, 2024 12:02 PM
> To: Manne, Nava kishore <[email protected]>
> Cc: [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; Simek, Michal <[email protected]>; Marco
> Pagani <[email protected]>; [email protected];
> [email protected]; [email protected]; linux-
> [email protected]; [email protected]; git (AMD-Xilinx)
> <[email protected]>
> Subject: Re: [RFC] FPGA Subsystem User Space Interface Proposal
>
> On Thu, Jan 04, 2024 at 04:52:15AM +0000, Manne, Nava kishore wrote:
> >
> ================================================================
> ======
> > =
> > | Introduction |
> >
> ================================================================
> ======
> > = This document provides a detailed overview of the proposed Kernel
> > feature for FPGA Manager subsystem user interface.
> > It describes the problem statement behind the proposal, the problem to be
> solved, a top-level solution design.
> >
> > Table of Contents:
> > ------------------
> > A. Problem Statement and Background
> > B. Scope and Out of scope of the proposal
> > B.1 Scope
> > B.2 Out of scope
> > C. Proposed Solution
> > D. Proposed User Interface Details
> >
> ================================================================
> ======
> > =
> > | A. Problem Statement and Background |
> >
> ================================================================
> ======
> > = The existing FPGA manager subsystem didn't have any user space
> > interface (other than the status/state in sysfs) in Kernel.
> > Basically, FPGAs are semiconductor devices that can be reprogrammed for
> desired hardware functionality.
> > FPGAs can be reprogrammed at runtime with different types of logic and IPs
> as per user need and hence there is a need to use device tree overlays for
> removing/updating/adding the devices at runtime for the IPs/controllers that
> are present in FPGA.
> > But we don't have any user interface in kernel for updating the device tree
> at runtime.
> >
> > Sometime back there was a series sent by Pantelis Antoniou
> (https://lore.kernel.org/lkml/1414528565-10907-4-git-send-email-
> [email protected]/).
> > This patch introduced a user interface configfs for Device Tree overlays, a
> method of dynamically altering the kernel's live Device Tree. However, this
> patch series was not accepted in mainline due to various concerns.
> > For more details refer to this link:
> >
> https://elinux.org/Frank%27s_Evolving_Overlay_Thoughts#issues_and_what
> > _needs_to_be_completed_--_Not_an_exhaustive_list
> >
> > One of the major valid concerns that were raised with this configfs interface
> was security as it opens up the interface to users for modifying the live device
> tree.
> >
> > So, in order to configure/program the FPGA devices, All the major
> > vendors of FPGA are using this configfs series as out-of-tree patch for
> configuring the FPGAs and there was never an attempt to introduce a generic
> interface to configure/program the FPGA in upstream and hence upstream
> kernel ended up in not having proper support for FPGAs.
> >
> > The proposal below tries to address this gap of FPGA programmability by
> providing an interface to the user.
> >
> >
> ================================================================
> ======
> > =
> > | B. Proposed Solution |
> >
> ================================================================
> ======
> > = The proposed interface adds a new sysfs interface (of-fpga-region.c)
> > as part of the fpga subsystem and it is responsible for supporting the below
> functionalities.
>
> Why only for of-fpga-region? There are also FPGA regions that don't rely on
> OF. My quick idea is that an interface for /sys/class/fpga-region/ and OF could
> be one of the implementation.
>
I agree, few devices(Like - dfl-fme-region.c) are rely only on FPGA region and they are not using OF. Thanks for pointing out this case.
I will look at the possibility of having a generic interface for both Fpga region and OF and I will get back to you soon.
Regards,
Navakishore.