=======================
The fmc-bus Eradication
=======================
The *fmc-bus* Linux sub-system was developed under CERN supervision
around 2012. During these years, the fmc-bus has evolved and drivers
for particular FMC applications (FMC DEL, FMC ADC, FMC TDC, etc.)
have been based on it. The sub-system works fine and it is running in
the biggest particle accelerator on Earth without major problems.
Despite the fact that the fmc-bus works, the faults in its design
reveal, with the experience acquired, that its use is unduly limiting
and, actually, counter productive.
The purpose of this document is to explain the rationale behind
fmc-bus eradication from the Linux kernel mainline and to propose a
better alternative.
The fmc-bus
===========
The FMC Standard
----------------
The FMC standard (`VITA-57.1`_) describes FMC modules and carriers
from the mechanical and the electrical point of view. `VITA-57.1`_
also specifies the information stored in the (mandatory) EEPROM on the
FMC module.
On the other hand, `VITA-57.1`_ does not specify at all how FMC carriers and
its FMC modules communicate.
From the software engineering point of view, the most interesting parts of
this standard are the following chapters:
5.5 I2C bus signal
This chapter specifies that an I2C bus connecting an FMC carrier
and its FMC modules is mandatory. Consequently, it specifies the FMC
signals that are reserved for that purpose.
By this chapter, an FMC module must contain an EEPROM connected to that
I2C bus. The purpose of this EEPROM is to store
*hardware definition information* using the
`Platform Management FRU Information Storage Definition V1.0`_
Optionally, it can support the base IPMI commands defined in the
*PICMG 2.6* specification.
So, from a software point of view, we are interested in reading the FRU
information that provides useful data to identify the FMC modules, and
to do so at a standardized I2C address and EEPROM programming interface.
5.6 Geographical address
This chapter describes how to address the different I2C devices hosted
in the FMC mezzanines that live in a single FMC carrier. As the
standard specifies up to four mezzanines per I2C bus, it enforces the
convention that the two least significant bits of the I2C address
identify the slot.
For example, the mandatory I2C EEPROM must be addressable at the 7-bit
address b10100XX, the suffix XX=00,01,10,11 determining the geographical
addressing of the module in the carrier. Other I2C devices follow the same
schema.
The fmc-bus Design
------------------
The Linux fmc-bus was designed in 2012 with the purpose of adding
generic support for the FMC modules that were under development at CERN
at that time.
Everything was targeted to work with Linux kernels 2.6.24 and later.
The idea behind the Linux fmc-bus subsystem is to enumerate the FMC modules
and to provide a driver for them using the Linux bus mechanism. With this idea
in mind, the Linux fmc-bus uses the information contained in the FMC module
EEPROM to do the match against the available fmc-bus drivers. Then, the fmc-
bus
drivers will use a set of functions from the FMC bus to make use of
the resources available on the FMC carrier (which is plugged on another Linux
sub-system, for example: PCI or VME).
Flawed conceptual assumptions in the fmc-bus
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The main wrong assumption in the Linux fmc-bus is the idea that drivers
are bound to FMC modules through the carrier FPGA: this is not true.
Behind this lies another wrong assumption: that FMC modules are in a
one-to-one relationship with the FPGA code: this is wrong. The actual
relationship is one-to-many: one FMC module can be driven by many FPGA
code designs, which might be wildly varying. The real behavior of a card
(FMC carrier + FMC module[s]) is determined by the FPGA code that has been
loaded on the FPGA.
A simple example will clarify the concept.
We can have an FMC module with a simple 5 digital I/O channel. We all know
that
such module is versatile, so we can have FPGA code that implements: two I2C
masters, an SPI master, a GPIO interface, a TDC, custom logic. In all these
cases the FMC module is the same but the FPGA code that drives it is
different,
as well as the Linux driver that should drive it.
So, the fmc-bus design is wrong in trying to match devices and drivers using
the FRU information from the FMC module EEPROM. Instead, the real match must
be done between the devices instantiated internally in the FPGA code and their
drivers.
Eradication Proposal
====================
The fmc-bus *today* and in its *current status* is not just inadequate: it has
too many overlaps with other Linux components (it was not the case back
in 2012). It seems appropriate to completely replace it by an FMC device
class.
Here is how to do it and how existing drivers (not in Linux mainline) should
be migrated to it.
FMC Device Class
----------------
I propose to retain all the functionality of the fmc-bus that should
remain (see below) in a kernel module implementing an FMC device class
that registers the presence of individual FMC modules.
The new FMC device class must be limited to the real FMC business logic. This
means that existing Linux kernel interfaces will be reused, instead of
implementing dedicated interfaces as the fmc-bus does.
There are four overlaps between the Linux fmc-bus and other Linux
sub-systems/drivers: fpga-manager, i2c sub-system, eeprom drivers and
irq management.
FPGA Manager
The Linux fmc-bus asks the FMC carrier driver to export the necessary
operations to program the FMC carrier FPGA. Currently, this is handled
by the Linux kernel FPGA manager subsystem.
So, the FMC carrier driver should create a device instance for the FPGA
manager.
I2C Master Driver
The Linux fmc-bus current implementation asks the FMC carrier drivers to
implement the I2C operations in order to access the EEPROM. In theory this
is not really necessary; Linux provides a nice I2C sub-system for this
purpose. Only the particular I2C master instance needs to be registered by
the FMC carrier driver. A driver for that master has to be provided only if
missing in the stock Linux kernel.
EEPROM Driver
The Linux fmc-bus uses the I2C programming interface of an FMC module EEPROM
to extract its contents, in particular the module identification for driver
Again, this is rendered superfluous by the Linux kernel I2C EEPROM
drivers. There is, however, a difficulty with the particular EEPROM
to use: which one is it? `VITA-57.1`_ is unclear about this.
rule 5.69 about what kind of EEPROM is mandatory:
Rule 5.69: The IO mezzanine module shall provide an onboard EEPROM,
which shall interface with the I2C bus signals.
Rule 5.74: Data fields provided by the IPMI resource on the mezzanine
module shall include the minimum records defined in the Platform
Management FRU Information Storage Definition V1.0.
FRU Information Storage Definition V1.0:
are SEEPROMs that use a '24C02'-compatible interface, and SEEPROMs that
are compatible with the Dallas Semiconductor DS1624 Temperature
Sensor/SEEPROM interface.
to support 24C02 EEPROMs. Support of other EEPROMs should not be
excluded, though, as `VITA-57.1`_ is admittedly open to misinterpretaion
on this point.
IRQ Controller
The Linux fmc-bus asks the FMC carrier to implement the operations to
request, free and acknowledge interrupts. This is necessary because
FMC modules plugged into an FMC carrier share the unique interrupt
line or vector of the latter. However, this function does not belong
in the fmc-bus. As the obvious need for such logic suggests,
the FMC carrier contains interrupt routing logic; thus, an IRQ
controller is the proper way to handle this.
This approach will significantly reduce the FMC scope to its true business.
Most of the existent code will be re-used and the future FMC carrier drivers
need only to glue toghether the device instances required for them to run.
Another duty left to the FMC class is to export an interface to extract the
information from the FMC module EEPROM.
Platform Device For FPGA Internal Devices
-----------------------------------------
As stated before, there is no such thing as an "FMC module driver", nor
a "device driver for the FPGA code".
The FPGA code is typically made of N ip-cores; groups of ip-cores may need
different drivers.
The simplest case to imagine is an FMC carrier with one FPGA and 4 FMC slots.
Within the same FPGA code we have different ip-cores to drive potentially 4
different FMC modules.
So, the only way I see it possible is by using platform drivers for the
*FPGA internal devices*. The description of the internal devices is not within
the scope of the FMC software layer.
.. _`Platform Management FRU Information Storage Definition V1.0`: https://
http://www.intel.com/content/www/us/en/servers/ipmi/information-storage-definition-