This series converts the mlx4 drivers to use auxiliary bus, similarly to
how mlx5 was converted [1]. The first 6 patches are preparatory changes,
the remaining 4 are the final conversion.
Initial motivation for this change was to address a problem related to
loading mlx4_en/mlx4_ib by mlx4_core using request_module_nowait(). When
doing such a load in initrd, the operation is asynchronous to any init
control and can get unexpectedly affected/interrupted by an eventual
root switch. Using an auxiliary bus leaves these module loads to udevd
which better integrates with systemd processing. [2]
General benefit is to get rid of custom interface logic and instead use
a common facility available for this task. An obvious risk is that some
new bug is introduced by the conversion.
Leon Romanovsky was kind enough to check for me that the series passes
their verification tests.
[1] https://lore.kernel.org/netdev/[email protected]/
[2] https://lore.kernel.org/netdev/[email protected]/
Petr Pavlu (10):
mlx4: Get rid of the mlx4_interface.get_dev callback
mlx4: Rename member mlx4_en_dev.nb to netdev_nb
mlx4: Replace the mlx4_interface.event callback with a notifier
mlx4: Get rid of the mlx4_interface.activate callback
mlx4: Move the bond work to the core driver
mlx4: Avoid resetting MLX4_INTFF_BONDING per driver
mlx4: Register mlx4 devices to an auxiliary virtual bus
mlx4: Connect the ethernet part to the auxiliary bus
mlx4: Connect the infiniband part to the auxiliary bus
mlx4: Delete custom device management logic
drivers/infiniband/hw/mlx4/main.c | 207 ++++++----
drivers/infiniband/hw/mlx4/mlx4_ib.h | 2 +
drivers/net/ethernet/mellanox/mlx4/Kconfig | 1 +
drivers/net/ethernet/mellanox/mlx4/en_main.c | 141 ++++---
.../net/ethernet/mellanox/mlx4/en_netdev.c | 64 +---
drivers/net/ethernet/mellanox/mlx4/intf.c | 361 ++++++++++++------
drivers/net/ethernet/mellanox/mlx4/main.c | 110 ++++--
drivers/net/ethernet/mellanox/mlx4/mlx4.h | 16 +-
drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 4 +-
include/linux/mlx4/device.h | 20 +
include/linux/mlx4/driver.h | 42 +-
11 files changed, 572 insertions(+), 396 deletions(-)
base-commit: 86b7e033d684a9d4ca20ad8e6f8b9300cf99668f
--
2.35.3
Simplify the mlx4 driver interface by removing mlx4_get_protocol_dev()
and the associated mlx4_interface.get_dev callbacks. This is done in
preparation to use an auxiliary bus to model the mlx4 driver structure.
The change is motivated by the following situation:
* The mlx4_en interface is being initialized by mlx4_en_add() and
mlx4_en_activate().
* The latter activate function calls mlx4_en_init_netdev() ->
register_netdev() to register a new net_device.
* A netdev event NETDEV_REGISTER is raised for the device.
* The netdev notififier mlx4_ib_netdev_event() is called and it invokes
mlx4_ib_scan_netdevs() -> mlx4_get_protocol_dev() ->
mlx4_en_get_netdev() [via mlx4_interface.get_dev].
This chain creates a problem when mlx4_en gets switched to be an
auxiliary driver. It contains two device calls which would both need to
take a respective device lock.
Avoid this situation by updating mlx4_ib_scan_netdevs() to no longer
call mlx4_get_protocol_dev() but instead to utilize the information
passed in net_device.parent and net_device.dev_port. This data is
sufficient to determine that an updated port is one that the mlx4_ib
driver should take care of and to keep mlx4_ib_dev.iboe.netdevs up to
date.
Following that, update mlx4_ib_get_netdev() to also not call
mlx4_get_protocol_dev() and instead scan all current netdevs to find
find a matching one. Note that mlx4_ib_get_netdev() is called early from
ib_register_device() and cannot use data tracked in
mlx4_ib_dev.iboe.netdevs which is not at that point yet set.
Finally, remove function mlx4_get_protocol_dev() and the
mlx4_interface.get_dev callbacks (only mlx4_en_get_netdev()) as they
became unused.
Signed-off-by: Petr Pavlu <[email protected]>
Tested-by: Leon Romanovsky <[email protected]>
---
drivers/infiniband/hw/mlx4/main.c | 89 ++++++++++----------
drivers/net/ethernet/mellanox/mlx4/en_main.c | 8 --
drivers/net/ethernet/mellanox/mlx4/intf.c | 21 -----
include/linux/mlx4/driver.h | 3 -
4 files changed, 43 insertions(+), 78 deletions(-)
diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index b18e9f2adc82..7dd70d778b6b 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -125,12 +125,14 @@ static struct net_device *mlx4_ib_get_netdev(struct ib_device *device,
u32 port_num)
{
struct mlx4_ib_dev *ibdev = to_mdev(device);
- struct net_device *dev;
+ struct net_device *dev, *ret = NULL;
rcu_read_lock();
- dev = mlx4_get_protocol_dev(ibdev->dev, MLX4_PROT_ETH, port_num);
+ for_each_netdev_rcu(&init_net, dev) {
+ if (dev->dev.parent != ibdev->ib_dev.dev.parent ||
+ dev->dev_port + 1 != port_num)
+ continue;
- if (dev) {
if (mlx4_is_bonded(ibdev->dev)) {
struct net_device *upper = NULL;
@@ -143,11 +145,14 @@ static struct net_device *mlx4_ib_get_netdev(struct ib_device *device,
dev = active;
}
}
+
+ dev_hold(dev);
+ ret = dev;
+ break;
}
- dev_hold(dev);
rcu_read_unlock();
- return dev;
+ return ret;
}
static int mlx4_ib_update_gids_v1(struct gid_entry *gids,
@@ -2319,61 +2324,53 @@ static void mlx4_ib_update_qps(struct mlx4_ib_dev *ibdev,
mutex_unlock(&ibdev->qp1_proxy_lock[port - 1]);
}
-static void mlx4_ib_scan_netdevs(struct mlx4_ib_dev *ibdev,
- struct net_device *dev,
- unsigned long event)
+static void mlx4_ib_scan_netdev(struct mlx4_ib_dev *ibdev,
+ struct net_device *dev,
+ unsigned long event)
{
- struct mlx4_ib_iboe *iboe;
- int update_qps_port = -1;
- int port;
+ struct mlx4_ib_iboe *iboe = &ibdev->iboe;
ASSERT_RTNL();
- iboe = &ibdev->iboe;
+ if (dev->dev.parent != ibdev->ib_dev.dev.parent)
+ return;
spin_lock_bh(&iboe->lock);
- mlx4_foreach_ib_transport_port(port, ibdev->dev) {
-
- iboe->netdevs[port - 1] =
- mlx4_get_protocol_dev(ibdev->dev, MLX4_PROT_ETH, port);
- if (dev == iboe->netdevs[port - 1] &&
- (event == NETDEV_CHANGEADDR || event == NETDEV_REGISTER ||
- event == NETDEV_UP || event == NETDEV_CHANGE))
- update_qps_port = port;
+ iboe->netdevs[dev->dev_port] = event != NETDEV_UNREGISTER ? dev : NULL;
- if (dev == iboe->netdevs[port - 1] &&
- (event == NETDEV_UP || event == NETDEV_DOWN)) {
- enum ib_port_state port_state;
- struct ib_event ibev = { };
-
- if (ib_get_cached_port_state(&ibdev->ib_dev, port,
- &port_state))
- continue;
+ if (event == NETDEV_UP || event == NETDEV_DOWN) {
+ enum ib_port_state port_state;
+ struct ib_event ibev = { };
- if (event == NETDEV_UP &&
- (port_state != IB_PORT_ACTIVE ||
- iboe->last_port_state[port - 1] != IB_PORT_DOWN))
- continue;
- if (event == NETDEV_DOWN &&
- (port_state != IB_PORT_DOWN ||
- iboe->last_port_state[port - 1] != IB_PORT_ACTIVE))
- continue;
- iboe->last_port_state[port - 1] = port_state;
+ if (ib_get_cached_port_state(&ibdev->ib_dev, dev->dev_port + 1,
+ &port_state))
+ goto iboe_out;
- ibev.device = &ibdev->ib_dev;
- ibev.element.port_num = port;
- ibev.event = event == NETDEV_UP ? IB_EVENT_PORT_ACTIVE :
- IB_EVENT_PORT_ERR;
- ib_dispatch_event(&ibev);
- }
+ if (event == NETDEV_UP &&
+ (port_state != IB_PORT_ACTIVE ||
+ iboe->last_port_state[dev->dev_port] != IB_PORT_DOWN))
+ goto iboe_out;
+ if (event == NETDEV_DOWN &&
+ (port_state != IB_PORT_DOWN ||
+ iboe->last_port_state[dev->dev_port] != IB_PORT_ACTIVE))
+ goto iboe_out;
+ iboe->last_port_state[dev->dev_port] = port_state;
+ ibev.device = &ibdev->ib_dev;
+ ibev.element.port_num = dev->dev_port + 1;
+ ibev.event = event == NETDEV_UP ? IB_EVENT_PORT_ACTIVE :
+ IB_EVENT_PORT_ERR;
+ ib_dispatch_event(&ibev);
}
+
+iboe_out:
spin_unlock_bh(&iboe->lock);
- if (update_qps_port > 0)
- mlx4_ib_update_qps(ibdev, dev, update_qps_port);
+ if (event == NETDEV_CHANGEADDR || event == NETDEV_REGISTER ||
+ event == NETDEV_UP || event == NETDEV_CHANGE)
+ mlx4_ib_update_qps(ibdev, dev, dev->dev_port + 1);
}
static int mlx4_ib_netdev_event(struct notifier_block *this,
@@ -2386,7 +2383,7 @@ static int mlx4_ib_netdev_event(struct notifier_block *this,
return NOTIFY_DONE;
ibdev = container_of(this, struct mlx4_ib_dev, iboe.nb);
- mlx4_ib_scan_netdevs(ibdev, dev, event);
+ mlx4_ib_scan_netdev(ibdev, dev, event);
return NOTIFY_DONE;
}
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c
index f1259bdb1a29..6a42bec6bd85 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c
@@ -183,13 +183,6 @@ static void mlx4_en_get_profile(struct mlx4_en_dev *mdev)
}
}
-static void *mlx4_en_get_netdev(struct mlx4_dev *dev, void *ctx, u8 port)
-{
- struct mlx4_en_dev *endev = ctx;
-
- return endev->pndev[port];
-}
-
static void mlx4_en_event(struct mlx4_dev *dev, void *endev_ptr,
enum mlx4_dev_event event, unsigned long port)
{
@@ -354,7 +347,6 @@ static struct mlx4_interface mlx4_en_interface = {
.add = mlx4_en_add,
.remove = mlx4_en_remove,
.event = mlx4_en_event,
- .get_dev = mlx4_en_get_netdev,
.protocol = MLX4_PROT_ETH,
.activate = mlx4_en_activate,
};
diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c
index 65482f004e50..28d7da925d36 100644
--- a/drivers/net/ethernet/mellanox/mlx4/intf.c
+++ b/drivers/net/ethernet/mellanox/mlx4/intf.c
@@ -245,27 +245,6 @@ void mlx4_unregister_device(struct mlx4_dev *dev)
mutex_unlock(&intf_mutex);
}
-void *mlx4_get_protocol_dev(struct mlx4_dev *dev, enum mlx4_protocol proto, int port)
-{
- struct mlx4_priv *priv = mlx4_priv(dev);
- struct mlx4_device_context *dev_ctx;
- unsigned long flags;
- void *result = NULL;
-
- spin_lock_irqsave(&priv->ctx_lock, flags);
-
- list_for_each_entry(dev_ctx, &priv->ctx_list, list)
- if (dev_ctx->intf->protocol == proto && dev_ctx->intf->get_dev) {
- result = dev_ctx->intf->get_dev(dev, dev_ctx->context, port);
- break;
- }
-
- spin_unlock_irqrestore(&priv->ctx_lock, flags);
-
- return result;
-}
-EXPORT_SYMBOL_GPL(mlx4_get_protocol_dev);
-
struct devlink_port *mlx4_get_devlink_port(struct mlx4_dev *dev, int port)
{
struct mlx4_port_info *info = &mlx4_priv(dev)->port[port];
diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h
index 1834c8fad12e..923951e19300 100644
--- a/include/linux/mlx4/driver.h
+++ b/include/linux/mlx4/driver.h
@@ -59,7 +59,6 @@ struct mlx4_interface {
void (*remove)(struct mlx4_dev *dev, void *context);
void (*event) (struct mlx4_dev *dev, void *context,
enum mlx4_dev_event event, unsigned long param);
- void * (*get_dev)(struct mlx4_dev *dev, void *context, u8 port);
void (*activate)(struct mlx4_dev *dev, void *context);
struct list_head list;
enum mlx4_protocol protocol;
@@ -88,8 +87,6 @@ struct mlx4_port_map {
int mlx4_port_map_set(struct mlx4_dev *dev, struct mlx4_port_map *v2p);
-void *mlx4_get_protocol_dev(struct mlx4_dev *dev, enum mlx4_protocol proto, int port);
-
struct devlink_port *mlx4_get_devlink_port(struct mlx4_dev *dev, int port);
#endif /* MLX4_DRIVER_H */
--
2.35.3
The mlx4_core driver has a logic that allows a sub-driver to set the
MLX4_INTFF_BONDING flag which then causes that function mlx4_do_bond()
asks the sub-driver to fully re-probe a device when its bonding
configuration changes.
Performing this operation is disallowed in mlx4_register_interface()
when it is detected that any mlx4 device is multifunction (SRIOV). The
code then resets MLX4_INTFF_BONDING in the driver flags.
Move this check directly into mlx4_do_bond(). It provides a better
separation as mlx4_core no longer directly modifies the sub-driver flags
and it will allow to get rid of explicitly keeping track of all mlx4
devices by the intf.c code when it is switched to an auxiliary bus.
Signed-off-by: Petr Pavlu <[email protected]>
Tested-by: Leon Romanovsky <[email protected]>
---
drivers/net/ethernet/mellanox/mlx4/intf.c | 19 +++++++++++--------
1 file changed, 11 insertions(+), 8 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c
index 8b2c1404cb66..30aead34ce08 100644
--- a/drivers/net/ethernet/mellanox/mlx4/intf.c
+++ b/drivers/net/ethernet/mellanox/mlx4/intf.c
@@ -96,11 +96,6 @@ int mlx4_register_interface(struct mlx4_interface *intf)
list_add_tail(&intf->list, &intf_list);
list_for_each_entry(priv, &dev_list, dev_list) {
- if (mlx4_is_mfunc(&priv->dev) && (intf->flags & MLX4_INTFF_BONDING)) {
- mlx4_dbg(&priv->dev,
- "SRIOV, disabling HA mode for intf proto %d\n", intf->protocol);
- intf->flags &= ~MLX4_INTFF_BONDING;
- }
mlx4_add_device(intf, priv);
}
@@ -155,10 +150,18 @@ int mlx4_do_bond(struct mlx4_dev *dev, bool enable)
spin_lock_irqsave(&priv->ctx_lock, flags);
list_for_each_entry_safe(dev_ctx, temp_dev_ctx, &priv->ctx_list, list) {
- if (dev_ctx->intf->flags & MLX4_INTFF_BONDING) {
- list_add_tail(&dev_ctx->bond_list, &bond_list);
- list_del(&dev_ctx->list);
+ if (!(dev_ctx->intf->flags & MLX4_INTFF_BONDING))
+ continue;
+
+ if (mlx4_is_mfunc(dev)) {
+ mlx4_dbg(dev,
+ "SRIOV, disabled HA mode for intf proto %d\n",
+ dev_ctx->intf->protocol);
+ continue;
}
+
+ list_add_tail(&dev_ctx->bond_list, &bond_list);
+ list_del(&dev_ctx->list);
}
spin_unlock_irqrestore(&priv->ctx_lock, flags);
--
2.35.3
Use the auxiliary bus to perform device management of the ethernet part
of the mlx4 driver.
Signed-off-by: Petr Pavlu <[email protected]>
Tested-by: Leon Romanovsky <[email protected]>
---
drivers/net/ethernet/mellanox/mlx4/en_main.c | 67 ++++++++++++++------
drivers/net/ethernet/mellanox/mlx4/intf.c | 13 +++-
2 files changed, 59 insertions(+), 21 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c
index 3824884ab515..2827d5373d9f 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c
@@ -222,9 +222,11 @@ static int mlx4_en_event(struct notifier_block *this,
return NOTIFY_DONE;
}
-static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr)
+static void mlx4_en_remove(struct auxiliary_device *adev)
{
- struct mlx4_en_dev *mdev = endev_ptr;
+ struct mlx4_adev *madev = container_of(adev, struct mlx4_adev, adev);
+ struct mlx4_dev *dev = madev->mdev;
+ struct mlx4_en_dev *mdev = auxiliary_get_drvdata(adev);
int i;
mlx4_unregister_event_notifier(dev, &mdev->mlx_nb);
@@ -247,27 +249,36 @@ static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr)
kfree(mdev);
}
-static void *mlx4_en_add(struct mlx4_dev *dev)
+static int mlx4_en_probe(struct auxiliary_device *adev,
+ const struct auxiliary_device_id *id)
{
+ struct mlx4_adev *madev = container_of(adev, struct mlx4_adev, adev);
+ struct mlx4_dev *dev = madev->mdev;
struct mlx4_en_dev *mdev;
- int i;
+ int err, i;
printk_once(KERN_INFO "%s", mlx4_en_version);
mdev = kzalloc(sizeof(*mdev), GFP_KERNEL);
- if (!mdev)
+ if (!mdev) {
+ err = -ENOMEM;
goto err_free_res;
+ }
- if (mlx4_pd_alloc(dev, &mdev->priv_pdn))
+ err = mlx4_pd_alloc(dev, &mdev->priv_pdn);
+ if (err)
goto err_free_dev;
- if (mlx4_uar_alloc(dev, &mdev->priv_uar))
+ err = mlx4_uar_alloc(dev, &mdev->priv_uar);
+ if (err)
goto err_pd;
mdev->uar_map = ioremap((phys_addr_t) mdev->priv_uar.pfn << PAGE_SHIFT,
PAGE_SIZE);
- if (!mdev->uar_map)
+ if (!mdev->uar_map) {
+ err = -ENOMEM;
goto err_uar;
+ }
spin_lock_init(&mdev->uar_lock);
mdev->dev = dev;
@@ -279,13 +290,15 @@ static void *mlx4_en_add(struct mlx4_dev *dev)
if (!mdev->LSO_support)
mlx4_warn(mdev, "LSO not supported, please upgrade to later FW version to enable LSO\n");
- if (mlx4_mr_alloc(mdev->dev, mdev->priv_pdn, 0, ~0ull,
- MLX4_PERM_LOCAL_WRITE | MLX4_PERM_LOCAL_READ,
- 0, 0, &mdev->mr)) {
+ err = mlx4_mr_alloc(mdev->dev, mdev->priv_pdn, 0, ~0ull,
+ MLX4_PERM_LOCAL_WRITE | MLX4_PERM_LOCAL_READ, 0, 0,
+ &mdev->mr);
+ if (err) {
mlx4_err(mdev, "Failed allocating memory region\n");
goto err_map;
}
- if (mlx4_mr_enable(mdev->dev, &mdev->mr)) {
+ err = mlx4_mr_enable(mdev->dev, &mdev->mr);
+ if (err) {
mlx4_err(mdev, "Failed enabling memory region\n");
goto err_mr;
}
@@ -305,8 +318,10 @@ static void *mlx4_en_add(struct mlx4_dev *dev)
* Note: we cannot use the shared workqueue because of deadlocks caused
* by the rtnl lock */
mdev->workqueue = create_singlethread_workqueue("mlx4_en");
- if (!mdev->workqueue)
+ if (!mdev->workqueue) {
+ err = -ENOMEM;
goto err_mr;
+ }
/* At this stage all non-port specific tasks are complete:
* mark the card state as up */
@@ -334,7 +349,8 @@ static void *mlx4_en_add(struct mlx4_dev *dev)
mlx4_err(mdev, "Failed to create netdev notifier\n");
}
- return mdev;
+ auxiliary_set_drvdata(adev, mdev);
+ return 0;
err_mr:
(void) mlx4_mr_free(dev, &mdev->mr);
@@ -348,12 +364,23 @@ static void *mlx4_en_add(struct mlx4_dev *dev)
err_free_dev:
kfree(mdev);
err_free_res:
- return NULL;
+ return err;
}
-static struct mlx4_interface mlx4_en_interface = {
- .add = mlx4_en_add,
- .remove = mlx4_en_remove,
+static const struct auxiliary_device_id mlx4_en_id_table[] = {
+ { .name = MLX4_ADEV_NAME ".eth" },
+ {},
+};
+
+MODULE_DEVICE_TABLE(auxiliary, mlx4_en_id_table);
+
+static struct mlx4_adrv mlx4_en_adrv = {
+ .adrv = {
+ .name = "eth",
+ .probe = mlx4_en_probe,
+ .remove = mlx4_en_remove,
+ .id_table = mlx4_en_id_table,
+ },
.protocol = MLX4_PROT_ETH,
};
@@ -383,12 +410,12 @@ static int __init mlx4_en_init(void)
mlx4_en_verify_params();
mlx4_en_init_ptys2ethtool_map();
- return mlx4_register_interface(&mlx4_en_interface);
+ return mlx4_register_auxiliary_driver(&mlx4_en_adrv);
}
static void __exit mlx4_en_cleanup(void)
{
- mlx4_unregister_interface(&mlx4_en_interface);
+ mlx4_unregister_auxiliary_driver(&mlx4_en_adrv);
}
module_init(mlx4_en_init);
diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c
index 4b1e18e4a682..0a27820ece2e 100644
--- a/drivers/net/ethernet/mellanox/mlx4/intf.c
+++ b/drivers/net/ethernet/mellanox/mlx4/intf.c
@@ -50,10 +50,21 @@ static LIST_HEAD(dev_list);
static DEFINE_MUTEX(intf_mutex);
static DEFINE_IDA(mlx4_adev_ida);
+static bool is_eth_supported(struct mlx4_dev *dev)
+{
+ for (int port = 1; port <= dev->caps.num_ports; port++)
+ if (dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH)
+ return true;
+
+ return false;
+}
+
static const struct mlx4_adev_device {
const char *suffix;
bool (*is_supported)(struct mlx4_dev *dev);
-} mlx4_adev_devices[1] = {};
+} mlx4_adev_devices[] = {
+ { "eth", is_eth_supported },
+};
int mlx4_adev_init(struct mlx4_dev *dev)
{
--
2.35.3
Rename the mlx4_en_dev.nb notifier_block member to netdev_nb in
preparation to add a mlx4 core notifier_block.
Signed-off-by: Petr Pavlu <[email protected]>
Tested-by: Leon Romanovsky <[email protected]>
---
drivers/net/ethernet/mellanox/mlx4/en_main.c | 14 +++++++-------
drivers/net/ethernet/mellanox/mlx4/en_netdev.c | 2 +-
drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 2 +-
3 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c
index 6a42bec6bd85..be8ba34c9025 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c
@@ -235,8 +235,8 @@ static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr)
iounmap(mdev->uar_map);
mlx4_uar_free(dev, &mdev->priv_uar);
mlx4_pd_free(dev, mdev->priv_pdn);
- if (mdev->nb.notifier_call)
- unregister_netdevice_notifier(&mdev->nb);
+ if (mdev->netdev_nb.notifier_call)
+ unregister_netdevice_notifier(&mdev->netdev_nb);
kfree(mdev);
}
@@ -252,11 +252,11 @@ static void mlx4_en_activate(struct mlx4_dev *dev, void *ctx)
mdev->pndev[i] = NULL;
}
- /* register notifier */
- mdev->nb.notifier_call = mlx4_en_netdev_event;
- if (register_netdevice_notifier(&mdev->nb)) {
- mdev->nb.notifier_call = NULL;
- mlx4_err(mdev, "Failed to create notifier\n");
+ /* register netdev notifier */
+ mdev->netdev_nb.notifier_call = mlx4_en_netdev_event;
+ if (register_netdevice_notifier(&mdev->netdev_nb)) {
+ mdev->netdev_nb.notifier_call = NULL;
+ mlx4_err(mdev, "Failed to create netdev notifier\n");
}
}
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
index 403604ceebc8..7066c426b95c 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
@@ -2967,7 +2967,7 @@ int mlx4_en_netdev_event(struct notifier_block *this,
if (!net_eq(dev_net(ndev), &init_net))
return NOTIFY_DONE;
- mdev = container_of(this, struct mlx4_en_dev, nb);
+ mdev = container_of(this, struct mlx4_en_dev, netdev_nb);
dev = mdev->dev;
/* Go into this mode only when two network devices set on two ports
diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
index 321f801c1d7c..72a3fea36702 100644
--- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
+++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
@@ -432,7 +432,7 @@ struct mlx4_en_dev {
unsigned long last_overflow_check;
struct ptp_clock *ptp_clock;
struct ptp_clock_info ptp_clock_info;
- struct notifier_block nb;
+ struct notifier_block netdev_nb;
};
--
2.35.3
On Fri, Aug 04, 2023 at 05:05:17PM +0200, Petr Pavlu wrote:
> This series converts the mlx4 drivers to use auxiliary bus, similarly to
> how mlx5 was converted [1]. The first 6 patches are preparatory changes,
> the remaining 4 are the final conversion.
>
> Initial motivation for this change was to address a problem related to
> loading mlx4_en/mlx4_ib by mlx4_core using request_module_nowait(). When
> doing such a load in initrd, the operation is asynchronous to any init
> control and can get unexpectedly affected/interrupted by an eventual
> root switch. Using an auxiliary bus leaves these module loads to udevd
> which better integrates with systemd processing. [2]
Neat, I didn't realize that was a pain point for distros.
Jason
On Fri, Aug 04, 2023 at 05:05:23PM +0200, Petr Pavlu wrote:
> The mlx4_core driver has a logic that allows a sub-driver to set the
> MLX4_INTFF_BONDING flag which then causes that function mlx4_do_bond()
> asks the sub-driver to fully re-probe a device when its bonding
> configuration changes.
>
> Performing this operation is disallowed in mlx4_register_interface()
> when it is detected that any mlx4 device is multifunction (SRIOV). The
> code then resets MLX4_INTFF_BONDING in the driver flags.
>
> Move this check directly into mlx4_do_bond(). It provides a better
> separation as mlx4_core no longer directly modifies the sub-driver flags
> and it will allow to get rid of explicitly keeping track of all mlx4
> devices by the intf.c code when it is switched to an auxiliary bus.
>
> Signed-off-by: Petr Pavlu <[email protected]>
> Tested-by: Leon Romanovsky <[email protected]>
> ---
> drivers/net/ethernet/mellanox/mlx4/intf.c | 19 +++++++++++--------
> 1 file changed, 11 insertions(+), 8 deletions(-)
>
Thanks,
Reviewed-by: Leon Romanovsky <[email protected]>
On Fri, Aug 04, 2023 at 05:05:25PM +0200, Petr Pavlu wrote:
> Use the auxiliary bus to perform device management of the ethernet part
> of the mlx4 driver.
>
> Signed-off-by: Petr Pavlu <[email protected]>
> Tested-by: Leon Romanovsky <[email protected]>
> ---
> drivers/net/ethernet/mellanox/mlx4/en_main.c | 67 ++++++++++++++------
> drivers/net/ethernet/mellanox/mlx4/intf.c | 13 +++-
> 2 files changed, 59 insertions(+), 21 deletions(-)
>
Thanks,
Reviewed-by: Leon Romanovsky <[email protected]>
On Fri, Aug 04, 2023 at 05:05:19PM +0200, Petr Pavlu wrote:
> Rename the mlx4_en_dev.nb notifier_block member to netdev_nb in
> preparation to add a mlx4 core notifier_block.
>
> Signed-off-by: Petr Pavlu <[email protected]>
> Tested-by: Leon Romanovsky <[email protected]>
> ---
> drivers/net/ethernet/mellanox/mlx4/en_main.c | 14 +++++++-------
> drivers/net/ethernet/mellanox/mlx4/en_netdev.c | 2 +-
> drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 2 +-
> 3 files changed, 9 insertions(+), 9 deletions(-)
>
Thanks,
Reviewed-by: Leon Romanovsky <[email protected]>
On Fri, Aug 04, 2023 at 05:05:18PM +0200, Petr Pavlu wrote:
> Simplify the mlx4 driver interface by removing mlx4_get_protocol_dev()
> and the associated mlx4_interface.get_dev callbacks. This is done in
> preparation to use an auxiliary bus to model the mlx4 driver structure.
>
> The change is motivated by the following situation:
> * The mlx4_en interface is being initialized by mlx4_en_add() and
> mlx4_en_activate().
> * The latter activate function calls mlx4_en_init_netdev() ->
> register_netdev() to register a new net_device.
> * A netdev event NETDEV_REGISTER is raised for the device.
> * The netdev notififier mlx4_ib_netdev_event() is called and it invokes
> mlx4_ib_scan_netdevs() -> mlx4_get_protocol_dev() ->
> mlx4_en_get_netdev() [via mlx4_interface.get_dev].
>
> This chain creates a problem when mlx4_en gets switched to be an
> auxiliary driver. It contains two device calls which would both need to
> take a respective device lock.
>
> Avoid this situation by updating mlx4_ib_scan_netdevs() to no longer
> call mlx4_get_protocol_dev() but instead to utilize the information
> passed in net_device.parent and net_device.dev_port. This data is
> sufficient to determine that an updated port is one that the mlx4_ib
> driver should take care of and to keep mlx4_ib_dev.iboe.netdevs up to
> date.
>
> Following that, update mlx4_ib_get_netdev() to also not call
> mlx4_get_protocol_dev() and instead scan all current netdevs to find
> find a matching one. Note that mlx4_ib_get_netdev() is called early from
> ib_register_device() and cannot use data tracked in
> mlx4_ib_dev.iboe.netdevs which is not at that point yet set.
>
> Finally, remove function mlx4_get_protocol_dev() and the
> mlx4_interface.get_dev callbacks (only mlx4_en_get_netdev()) as they
> became unused.
>
> Signed-off-by: Petr Pavlu <[email protected]>
> Tested-by: Leon Romanovsky <[email protected]>
> ---
> drivers/infiniband/hw/mlx4/main.c | 89 ++++++++++----------
> drivers/net/ethernet/mellanox/mlx4/en_main.c | 8 --
> drivers/net/ethernet/mellanox/mlx4/intf.c | 21 -----
> include/linux/mlx4/driver.h | 3 -
> 4 files changed, 43 insertions(+), 78 deletions(-)
>
Thanks,
Reviewed-by: Leon Romanovsky <[email protected]>
On 04/08/2023 18:05, Petr Pavlu wrote:
> This series converts the mlx4 drivers to use auxiliary bus, similarly to
> how mlx5 was converted [1]. The first 6 patches are preparatory changes,
> the remaining 4 are the final conversion.
>
> Initial motivation for this change was to address a problem related to
> loading mlx4_en/mlx4_ib by mlx4_core using request_module_nowait(). When
> doing such a load in initrd, the operation is asynchronous to any init
> control and can get unexpectedly affected/interrupted by an eventual
> root switch. Using an auxiliary bus leaves these module loads to udevd
> which better integrates with systemd processing. [2]
>
> General benefit is to get rid of custom interface logic and instead use
> a common facility available for this task. An obvious risk is that some
> new bug is introduced by the conversion.
>
> Leon Romanovsky was kind enough to check for me that the series passes
> their verification tests.
>
> [1] https://lore.kernel.org/netdev/[email protected]/
> [2] https://lore.kernel.org/netdev/[email protected]/
>
> Petr Pavlu (10):
> mlx4: Get rid of the mlx4_interface.get_dev callback
> mlx4: Rename member mlx4_en_dev.nb to netdev_nb
> mlx4: Replace the mlx4_interface.event callback with a notifier
> mlx4: Get rid of the mlx4_interface.activate callback
> mlx4: Move the bond work to the core driver
> mlx4: Avoid resetting MLX4_INTFF_BONDING per driver
> mlx4: Register mlx4 devices to an auxiliary virtual bus
> mlx4: Connect the ethernet part to the auxiliary bus
> mlx4: Connect the infiniband part to the auxiliary bus
> mlx4: Delete custom device management logic
>
> drivers/infiniband/hw/mlx4/main.c | 207 ++++++----
> drivers/infiniband/hw/mlx4/mlx4_ib.h | 2 +
> drivers/net/ethernet/mellanox/mlx4/Kconfig | 1 +
> drivers/net/ethernet/mellanox/mlx4/en_main.c | 141 ++++---
> .../net/ethernet/mellanox/mlx4/en_netdev.c | 64 +---
> drivers/net/ethernet/mellanox/mlx4/intf.c | 361 ++++++++++++------
> drivers/net/ethernet/mellanox/mlx4/main.c | 110 ++++--
> drivers/net/ethernet/mellanox/mlx4/mlx4.h | 16 +-
> drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 4 +-
> include/linux/mlx4/device.h | 20 +
> include/linux/mlx4/driver.h | 42 +-
> 11 files changed, 572 insertions(+), 396 deletions(-)
>
>
> base-commit: 86b7e033d684a9d4ca20ad8e6f8b9300cf99668f
For the series:
Acked-by: Tariq Toukan <[email protected]>