I have an issue where I have an onboard NIC with effectively three
ports, no other NIC ports/chips are in the system
Nic0: xx:xx:xx:00:00:02
Nic1: xx:xx:xx:00:00:01
IPMI: xx:xx:xx:00:00:03
When I boot RedHat Linux, Nic1 becomes Eth0 and Nic0 becomes Eth1.
I understand there is a disconnect between BIOS and linux on which
device should get which Eth, as well as, what ensues when you have PCI
cards along side onboard ports.
What I'm curious about is how/why Linux actually decides Nic1 should be Eth0?
My theory is it starts on the lowest MAC address and works up,
depending partly on driver load order.
Given the above scenario, swapping the MAC addresses between Nic1 and
Nic0 would clear this issue.
Can anyone confirm or deny this? Or explain/point me to, how it actually works.
Michael Di Domenico <[email protected]> 2010-07-07 18:25:
> I have an issue where I have an onboard NIC with effectively three
> ports, no other NIC ports/chips are in the system
>
> Nic0: xx:xx:xx:00:00:02
> Nic1: xx:xx:xx:00:00:01
> IPMI: xx:xx:xx:00:00:03
>
> When I boot RedHat Linux, Nic1 becomes Eth0 and Nic0 becomes Eth1.
>
> I understand there is a disconnect between BIOS and linux on which
> device should get which Eth, as well as, what ensues when you have PCI
> cards along side onboard ports.
>
> What I'm curious about is how/why Linux actually decides Nic1 should be Eth0?
>
> My theory is it starts on the lowest MAC address and works up,
> depending partly on driver load order.
>
> Given the above scenario, swapping the MAC addresses between Nic1 and
> Nic0 would clear this issue.
>
> Can anyone confirm or deny this? Or explain/point me to, how it actually works.
Look into udev. It should maintain some rules that dictate that
mapping. If not you can potentially write/alter some to make it do what
you want/expect.
On my systems the place to look is generally here:
/etc/udev/rules.d (user rules)
/lib/udev/rules.d (system rules)
There may be others.
Cheers,
Brian
On Wed, Jul 7, 2010 at 3:25 PM, Michael Di Domenico
<[email protected]> wrote:
> What I'm curious about is how/why Linux actually decides Nic1 should be Eth0?
>
> My theory is it starts on the lowest MAC address and works up,
> depending partly on driver load order.
I may be spewing misinformation here, but my understanding of how this
works is that the specific ordering is pretty much random. It's rather
unlikely to be based on MAC address value, except accidentally; it's
probably just whatever comes up first when it enumerates the
interfaces, which could easily be timing-dependent. (From personal
experience, two identical but physically separate NICs will come up in
a different order on every boot. Your case may be more consistent.)
The first time it does this, though, it records the MAC address into
the appropriate ifcfg-* script. From then on, at boot time it checks
the MAC address of each interface as it comes up. If it mismatches
with what is recorded in the ifcfg-* script, then it skips that name
and tries using the next.
In other words, all you should have to do is set HWADDR appropriately
in the the ifcfg-* scripts and it should do the right thing.
On Wed, Jul 07, 2010 at 06:25:53PM -0400, Michael Di Domenico wrote:
> I have an issue where I have an onboard NIC with effectively three
> ports, no other NIC ports/chips are in the system
>
> Nic0: xx:xx:xx:00:00:02
> Nic1: xx:xx:xx:00:00:01
> IPMI: xx:xx:xx:00:00:03
>
> When I boot RedHat Linux, Nic1 becomes Eth0 and Nic0 becomes Eth1.
>
> I understand there is a disconnect between BIOS and linux on which
> device should get which Eth, as well as, what ensues when you have PCI
> cards along side onboard ports.
>
> What I'm curious about is how/why Linux actually decides Nic1 should be Eth0?
>
> My theory is it starts on the lowest MAC address and works up,
> depending partly on driver load order.
>
> Given the above scenario, swapping the MAC addresses between Nic1 and
> Nic0 would clear this issue.
>
> Can anyone confirm or deny this? Or explain/point me to, how it actually works.
http://lwn.net/Articles/356900/
And on a system where you can use udev rules, the
70-persistent-net.rules file can be used to make them constant, by
using the MAC address to force the names to be what you want.
--
Matt Domsch
Technology Strategist
Dell | Office of the CTO
> -----Original Message-----
> From: [email protected] [mailto:linux-kernel-
> [email protected]] On Behalf Of Matt Domsch
> Sent: July 07, 2010 9:21 PM
>
> http://lwn.net/Articles/356900/
>
> And on a system where you can use udev rules, the
> 70-persistent-net.rules file can be used to make them constant, by
> using the MAC address to force the names to be what you want.
>
Ok, but can we achieve the renaming w/o a reboot?
> --
> Matt Domsch
Chetan Loke
On Thu, Jul 8, 2010 at 1:36 PM, Loke, Chetan <[email protected]> wrote:
>> -----Original Message-----
>> From: [email protected] [mailto:linux-kernel-
>> [email protected]] On Behalf Of Matt Domsch
>> Sent: July 07, 2010 9:21 PM
>>
>> http://lwn.net/Articles/356900/
>>
>> And on a system where you can use udev rules, the
>> 70-persistent-net.rules file can be used to make them constant, by
>> using the MAC address to force the names to be what you want.
>>
>
> Ok, but can we achieve the renaming w/o a reboot?
You probably can restart the udev subsystem.
>
>
>> --
>> Matt Domsch
>
> Chetan Loke
> --
> To unsubscribe from this list: send the line "unsubscribe linux-net" in
> the body of a message to [email protected]
> More majordomo info at ?http://vger.kernel.org/majordomo-info.html
>
On Thu, Jul 8, 2010 at 22:47, Andrew Fong <[email protected]> wrote:
> On Thu, Jul 8, 2010 at 1:36 PM, Loke, Chetan <[email protected]> wrote:
>>> -----Original Message-----
>>> From: [email protected] [mailto:linux-kernel-
>>> [email protected]] On Behalf Of Matt Domsch
>>> Sent: July 07, 2010 9:21 PM
>>>
>>> http://lwn.net/Articles/356900/
>>>
>>> And on a system where you can use udev rules, the
>>> 70-persistent-net.rules file can be used to make them constant, by
>>> using the MAC address to force the names to be what you want.
>>
>> Ok, but can we achieve the renaming w/o a reboot?
>
> You probably can restart the udev subsystem.
No, udev has not really a concept of being restarted, it will not do
anything. You can only synthesize events, so it looks like the device
is just plugged in. For that, you need to make sure the interface is
not busy, otherwise renaming will fail in interesting ways, especially
if names need to be swapped between several devices.
Live renaming of interfaces is something you really need to know what
you are doing, especially on remote machines.
Kay
On Thu, 8 Jul 2010, Loke, Chetan wrote:
> > -----Original Message-----
> > From: [email protected] [mailto:linux-kernel-
> > [email protected]] On Behalf Of Matt Domsch
> > Sent: July 07, 2010 9:21 PM
> >
> > http://lwn.net/Articles/356900/
> >
> > And on a system where you can use udev rules, the
> > 70-persistent-net.rules file can be used to make them constant, by
> > using the MAC address to force the names to be what you want.
>
> Ok, but can we achieve the renaming w/o a reboot?
Yes, you can rename interfaces using the 'ip' command.
ip link set oldname name newname
> -----Original Message-----
> From: Kay Sievers [mailto:[email protected]]
> Sent: July 08, 2010 5:28 PM
> To: Andrew Fong
> Cc: Loke, Chetan; Matt Domsch; Michael Di Domenico; linux-
> [email protected]; [email protected]
> Subject: Re: nic enumeration
>
> On Thu, Jul 8, 2010 at 22:47, Andrew Fong <[email protected]> wrote:
> > On Thu, Jul 8, 2010 at 1:36 PM, Loke, Chetan
> <[email protected]> wrote:
> >>> -----Original Message-----
> >>> From: [email protected] [mailto:linux-kernel-
> >>> [email protected]] On Behalf Of Matt Domsch
> >>> Sent: July 07, 2010 9:21 PM
> >>>
> >>> http://lwn.net/Articles/356900/
> >>>
> >>> And on a system where you can use udev rules, the
> >>> 70-persistent-net.rules file can be used to make them constant, by
> >>> using the MAC address to force the names to be what you want.
> >>
> >> Ok, but can we achieve the renaming w/o a reboot?
> >
> > You probably can restart the udev subsystem.
>
> No, udev has not really a concept of being restarted, it will not do
> anything. You can only synthesize events, so it looks like the device
> is just plugged in. For that, you need to make sure the interface is
> not busy, otherwise renaming will fail in interesting ways, especially
> if names need to be swapped between several devices.
>
I'm not a udev guru so I don't know how will renaming a simple net_device->name work?
What about the msi-x queues created by the driver? May be I missed it but I don't see a single driver tearing apart everything and restarting unless a NETDEV_RENAME somehow automagically reloads the driver. I know it's silly for the driver to tear apart the whole world. But then why allow renaming in the first place, correct?
> Live renaming of interfaces is something you really need to know what
> you are doing, especially on remote machines.
>
I agree, I wanted someone to say 'try the symlink attrib'. I really need to create a 'symlink' or a reference to the newly added 'ethX' interface. I have to support 75+ VMs(each w/ 5+ vNICs) and I can't [re]name the newly added vNIC and then reboot. Rebooting 75+ VMs is non-practical. Adding a new vNIC is as simple as point-and-click and the guest sees the new vNIC. The VM doesn't need to reboot.
How do I achieve this 'symlinking' effect minus the reboot?
> Kay
Regards
Chetan Loke
????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay? ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m???? ????????I?
* Chetan Loke:
>> And on a system where you can use udev rules, the
>> 70-persistent-net.rules file can be used to make them constant, by
>> using the MAC address to force the names to be what you want.
>>
>
> Ok, but can we achieve the renaming w/o a reboot?
"udev-trigger" or "udevadm trigger" works for renaming network
interfaces. The kernel has some restrictions on renaming interfaces
which are in use, so this might not work in your case.
--
Florian Weimer <[email protected]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstra?e 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
On Fri, Jul 9, 2010 at 09:18, Florian Weimer <[email protected]> wrote:
>> Ok, but can we achieve the renaming w/o a reboot?
>
> "udev-trigger" or "udevadm trigger" works for renaming network
> interfaces. The kernel has some restrictions on renaming interfaces
> which are in use, so this might not work in your case.
On older systems it would be:
udevtrigger
on newer systems it would be:
udevadm trigger
On recent systems you need to add:
udevadm trigger --action=add
otherwise 'change' events will be triggered, which don't rename any
netif. The default 'add' events caused too many problems with tools
with not well integrated tools.
And you should always limit the trigger to the subsystem you want to work with:
udevadm trigger --action=add --subsystem-match=net
otherwise you run events for all devices on the system, which may
cause completely unrelated trouble.
Anyway, it's not recommended to do anything like this. On many distros
it will cause trouble with the network scripts/services, which are run
again for already active interfaces. Using the udevadm trigger
'hammer' on the running system may lead to all sorts of problems. This
tool is usually only used at coldplug time during bootup.
Also, busy interfaces which might be included in the rename attempt
will cause ethX_rename failures if names need to be swapped.
In general, there is no safe way, or a tool to do this. You have to
know very well what you are doing. It's pretty common, that you
disconnect yourself from a remote box if you try anything like this.
Kay
* Kay Sievers:
> Anyway, it's not recommended to do anything like this. On many distros
> it will cause trouble with the network scripts/services, which are run
> again for already active interfaces. Using the udevadm trigger
> 'hammer' on the running system may lead to all sorts of problems. This
> tool is usually only used at coldplug time during bootup.
There was a time when the Debian scripts were pretty robust. But
maybe we've just been lucky.
> In general, there is no safe way, or a tool to do this. You have to
> know very well what you are doing. It's pretty common, that you
> disconnect yourself from a remote box if you try anything like this.
Yeah, but there should be a way to do this over OOB management,
without requiring a reboot. 8-(
--
Florian Weimer <[email protected]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstra?e 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
> -----Original Message-----
> From: Florian Weimer [mailto:[email protected]]
> Sent: July 09, 2010 3:18 AM
> To: Loke, Chetan
> Cc: Matt Domsch; Michael Di Domenico; [email protected];
linux-
> [email protected]
> Subject: Re: nic enumeration
>
> * Chetan Loke:
>
> >> And on a system where you can use udev rules, the
> >> 70-persistent-net.rules file can be used to make them constant, by
> >> using the MAC address to force the names to be what you want.
> >>
> >
> > Ok, but can we achieve the renaming w/o a reboot?
>
> "udev-trigger" or "udevadm trigger" works for renaming network
> interfaces. The kernel has some restrictions on renaming interfaces
> which are in use, so this might not work in your case.
>
> --
Ok, no renaming, I would like a reference. And symlink just doesn't work
w/ the udevadm trigger business. We've tried that already.
What needs to be changed in udev etc to create a soft link? Any place
where I should start digging and gotchas to lookout for?
> Florian Weimer <[email protected]>
Chetan Loke
On Fri, Jul 9, 2010 at 9:27 AM, Loke, Chetan <[email protected]> wrote:
> Ok, no renaming, I would like a reference. And symlink just doesn't work
> w/ the udevadm trigger business. We've tried that already.
>
> What needs to be changed in udev etc to create a soft link? Any place
> where I should start digging and gotchas to lookout for?
By "soft link", I assume you're talking figuratively? Soft (symbolic)
links are a filesystem concept, implemented by filesystem-specific
logic that knows how to read a filename out of an inode and restart
lookup. In order for something similar to work for network devices,
somebody would have had to explicitly implement similar functionality.
(Symlinks are a big headache and source of security holes -- access
control, loops, pointing to nonexistent files, etc. -- so there's a
good reason to NOT have an equivalent for network devices.)
But perhaps there *is* an equivalent: what about (ab)using bridges?
Create a set of bridges and bind each one to a single NIC. You can
name the bridges whatever you want, and set up the mapping through
arbitrary logic.
On Fri, Jul 09, 2010 at 12:27:53PM -0400, Loke, Chetan wrote:
> Ok, no renaming, I would like a reference. And symlink just doesn't work
> w/ the udevadm trigger business. We've tried that already.
>
> What needs to be changed in udev etc to create a soft link? Any place
> where I should start digging and gotchas to lookout for?
Unlike disks which have /dev/disk/by-{id,path,uuid}/* symlinks back to
/dev/sd*, which udev manages, network devices have no such on-disk
representation for udev to manage. My proposal to create such on-disk
mappings was shot down, as was my suggestion to create (multiple)
network device symlinks/references/aliases inside the kernel.
The approach I'm pursuing now is called libnetdevname[1], which can
take as input a device name from various name spaces (BIOS-provided
from SMBIOS or in future, ACPI, or perhaps a userspace mapping similar
to /etc/mactab in concept), and which as output is the single kernel
name for a device. Once plumbed into the tools that manipulate
network devices, it would allow you to do:
# ip addr add xx.xx.xx.xx dev Embedded_NIC_1
and have "Embedded_NIC_1" map to the appropriate "ethN", for use by
the calls into the kernel.
In this case, Embedded_NIC_1 would come from BIOS; conceivably other
namespace sources could hook into this same framework then.
FWIW, I have a talk at LinuxCon Boston coming up on this, and have a
proposal in for Linux Plumbers Conference as well.
Thanks,
Matt
[1] http://linux.dell.com/wiki/index.php/Oss/libnetdevname
--
Matt Domsch
Technology Strategist
Dell | Office of the CTO
> -----Original Message-----
> From: Steve Fink [mailto:[email protected]]
> Sent: July 09, 2010 1:11 PM
> To: Loke, Chetan
> Cc: Florian Weimer; [email protected]; Matt Domsch; Michael Di
> Domenico; [email protected]; [email protected]
> Subject: Re: nic enumeration
>
> On Fri, Jul 9, 2010 at 9:27 AM, Loke, Chetan
<[email protected]>
> wrote:
> > Ok, no renaming, I would like a reference. And symlink just doesn't
> work
> > w/ the udevadm trigger business. We've tried that already.
> >
> > What needs to be changed in udev etc to create a soft link? Any
place
> > where I should start digging and gotchas to lookout for?
>
> By "soft link", I assume you're talking figuratively?
Yes, I was using it loosely. symlink == I would like 'udev' to do the
linkage w/ the network-devices and create a reference to it.
> Soft (symbolic)
> links are a filesystem concept, implemented by filesystem-specific
> logic that knows how to read a filename out of an inode and restart
> lookup. In order for something similar to work for network devices,
> somebody would have had to explicitly implement similar functionality.
> (Symlinks are a big headache and source of security holes -- access
> control, loops, pointing to nonexistent files, etc. -- so there's a
> good reason to NOT have an equivalent for network devices.)
>
Huh? If the 'ethX' interface doesn't exist just don't create the 'link'.
So are you saying there are no security issues with udev-symlinks for
other subsystems? Why is renaming allowed on the network-interface then?
Isn't that a problem? I don't see the driver re-registering their RX/TX
queues with the new-if-name.
May be I'm overlooking at something really obvious. But how are people
managing [pv]drivers with multiple vNICs in their VMs if they need a
consistent naming scheme?
Regards
Chetan Loke
> -----Original Message-----
> From: Matt Domsch [mailto:[email protected]]
> Sent: July 09, 2010 1:15 PM
> To: Loke, Chetan
> Cc: Florian Weimer; [email protected]; Michael Di Domenico;
> [email protected]; [email protected]
> Subject: Re: nic enumeration
>
> On Fri, Jul 09, 2010 at 12:27:53PM -0400, Loke, Chetan wrote:
> > Ok, no renaming, I would like a reference. And symlink just doesn't
> work
> > w/ the udevadm trigger business. We've tried that already.
> >
> > What needs to be changed in udev etc to create a soft link? Any
place
> > where I should start digging and gotchas to lookout for?
>
> Unlike disks which have /dev/disk/by-{id,path,uuid}/* symlinks back to
> /dev/sd*, which udev manages, network devices have no such on-disk
> representation for udev to manage. My proposal to create such on-disk
> mappings was shot down, as was my suggestion to create (multiple)
> network device symlinks/references/aliases inside the kernel.
>
Ok, I'm reading your link below/libnetdevname and original patch too.
I'm willing to bend over to get something done. It's so much better than
getting repeatedly tortured by customers.
> FWIW, I have a talk at LinuxCon Boston coming up on this, and have a
> proposal in for Linux Plumbers Conference as well.
>
> Thanks,
> Matt
>
Regards
Chetan Loke
On Fri, Jul 9, 2010 at 11:23 AM, Loke, Chetan <[email protected]> wrote:
>> -----Original Message-----
>> From: Steve Fink [mailto:[email protected]]
>>
>> Soft (symbolic)
>> links are a filesystem concept, implemented by filesystem-specific
>> logic that knows how to read a filename out of an inode and restart
>> lookup. In order for something similar to work for network devices,
>> somebody would have had to explicitly implement similar functionality.
>> (Symlinks are a big headache and source of security holes -- access
>> control, loops, pointing to nonexistent files, etc. -- so there's a
>> good reason to NOT have an equivalent for network devices.)
>>
>
> Huh? If the 'ethX' interface doesn't exist just don't create the 'link'.
> So are you saying there are no security issues with udev-symlinks for
> other subsystems? Why is renaming allowed on the network-interface then?
> Isn't that a problem? I don't see the driver re-registering their RX/TX
> queues with the new-if-name.
The issues are different between network device and filesystem names,
yes. I was just saying that symbolic linking / aliasing are tricky.
udev symlinks introduce no *new* security issues because they're
filesystem links. Renaming the network interface certainly does cause
problems. And I don't know how the queues are connected to the devices
internally, but I doubt it's by name.
> May be I'm overlooking at something really obvious. But how are people
> managing [pv]drivers with multiple vNICs in their VMs if they need a
> consistent naming scheme?
Your problem is different from the one that began this thread. Can you
describe your situation more fully?
I only partially understand your problem, but I am still wondering if
my bridge idea would work for you. (Preferably combined with udev
rules or whatever so the bridges are unnecessary after the next
reboot.)
> -----Original Message-----
> From: Steve Fink [mailto:[email protected]]
> Sent: July 09, 2010 5:27 PM
> To: Loke, Chetan
> Cc: Florian Weimer; [email protected]; Matt Domsch; Michael Di
> Domenico; [email protected]; [email protected]
> Subject: Re: nic enumeration
>
>
> Your problem is different from the one that began this thread. Can you
> describe your situation more fully?
>
Sorry guys, I thought I mentioned my requirement. And I apologize for
pulling in my discussion in this thread.
Requirement:
I've a box with 'M' NICs. During install time, all the h/w config is
static, customers will configure the box. They would say use 'ethX' for
task-foo and 'ethY' for task-bar.
All this while our-apps where happy because we would never modify the
h/w config(single NIC vendor). So the configuration was pretty static.
However, we now let customers add/replace NICs from different vendors(so
different drivers will get loaded).The older 'ethX' might now map to a
newly-inserted-NICs-mac. So I would now like an 'ethX' agnostic way of
addressing the NICs and BTW we can't reboot the box[look below why].
Possible solution:
S1)I thought symlink[or a reference etc] would solve this problem.
S2)I have another idea in mind(using mac-ids,enhance dev_ioctl[introduce
SIOCGHWADDR_TO_IFNAMEINDEXMAP] and traverse
for_each_netdev::for_each_dev_addrs) but I will wait for everyone's
suggestions because there might be an easy way.
Why can't I reboot?
Imagine the above configuration within a VM. Some customers have as many
as 75+ VMs deployed and we cannot afford to reboot the VMs after adding
a new (virtual)vNIC. The newly added vNIC is seen by the guest as if it
were a hot-insertion.
> I only partially understand your problem, but I am still wondering if
> my bridge idea would work for you. (Preferably combined with udev
> rules or whatever so the bridges are unnecessary after the next
> reboot.)
Why a bridge? Just to get around naming issues?
Regards
Chetan Loke
>> I only partially understand your problem, but I am still wondering if
>> my bridge idea would work for you. (Preferably combined with udev
>> rules or whatever so the bridges are unnecessary after the next
>> reboot.)
>
> Why a bridge? Just to get around naming issues?
Yes. I wasn't claiming it was a *good* way to fix it. Just wondering
if it would work.
I have no idea whether it makes sense or not, not really knowing much
about this stuff myself. I tried it on my workstation. It's a good way
to kick yourself off the network. But it does work in the simplest
possible case, if you fix up routing afterwards. I don't know whether
bridges can be usefully used by VMs.