I'm not sure if this has surfaced before for discussion (searching
hasn't shown me any relevant threads), but the 2.6 kernel's new
implementation of xprt_bindresvport can conflict with port 664 on
IPMI-enabled hosts with a BMC. IPMI uses both ports 623 and 664 for
communications, and several implementations appear to intercept all
traffic for these ports and not permit them to pass to the standard
packet driver (tcpdump in promiscuous mode shows no packets coming in).
I know that in the discussions back in 2005/02 and 2005/07 of the patch
that implements the new privileged port binding for NFS mount (I can't
tell whose patch it is, perhaps Olaf Kirch's?), Charles Lever noted that
the new sunrpc.min_resvport of 650 avoids conflicts with port 623. Is it
worth increasing this default to 665 to avoid this port as well?
If not, I just wanted to get this information onto the list, since I
searched pretty heavily while researching the intermittent hangs I've
been getting with my NFS mounts before finally nailing this to my IPMI
BMCs. Manually setting sunrpc.min_resvport to 700 has stabilized NFS on
my boxes. Thanks.
Michael Han
[email protected]
-----------------------------------------------------------
This message may contain confidential and/or privileged
information. This information is intended to be read only
by the individual or entity to whom it is addressed. If
you are not the intended recipient, you are on notice that
any review, disclosure, copying, distribution or use of
the contents of this message is strictly prohibited. If
you have received this message in error, please notify the
sender immediately and delete or destroy any copy of this
message.
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On 7/28/06, Roger Heflin <[email protected]> wrote:
> Michael Han wrote:
> >>From Chuck Lever:
> >> "For the record," some sites have a requirement for a larger
> >> port space.
> >
> > Naturally they do. auto-home systems with thousands of users could
> > easily cause this. I'm just pointing out that I'm satisfied with my own
> > workaround.
> >
> >> The daemon actually wouldn't show up on the security scan. The
> >> hardware IPMI listener would, however. The daemon is not visible on
> >> the network because the IPMI listener diverts packets to that port.
> >
> > Of course, you are correct. That's the crux of the problem I
> > encountered. Silly me.
> >
> >> Other workarounds worth mentioning: disable IPMI in the hardware, or
> >> don't use the built-in NIC for NFS traffic.
> >
> > Yes. Another possible alternative is to divert IPMI traffic to an
> > IPMI-only address. I'm not certain this works, but I know the SuperMicro
> > BMCs support use of alternate MAC & IP. I just don't know if the port
> > 623/664 intercepts are promiscuous. I tried changing this on a hot
> > system to no avail, but not after rebooting a system and all that good
> > stuff.
> >
> >> Indeed. I'm not familiar enough with IPMI to know if it listens on
> >> both the UDP and the TCP port.
> >
> > I believe that in all implementations, IPMI only uses UDP
> > conventionally, however the port allocation from IANA is for both
> > transports and it appears that more than one implementation intercepts
> > both transports (I've seen this issue referenced on systems using Intel
> > NICs with IPMI support and on Sun x86 hardware). I'm pretty uneducated
> > as far as IPMI goes, myself.
> >
>
>
> Some versions (maybe all) of broadcom chips will intercept *ANY* udp frame
> with the proper port number in the proper byte, even if that byte is not
> a port number, ie they will intercept the additional frames that make up a
> 32k udp packet that has the proper data in the port space, even though it
> is not a port number as the extra frames don't have a port designation.
>
> The broadcom chip is doing this in firmware, and it results in the nth
> frame of a 32k udp packet always being lost even on retransmit, and it
> very much depends on the data being sent to have the wrong number in
> the wrong location, and it has been reported to Broadcom.
>
> It means that it is difficult to run the Broadcom ethernet ports with ipmi
> the same ip/mac address and have things reliable.
>
> The last time I worked with the intel e1000's bmc they were troublesome and
> unreliable in many other different way,s though that may be fixed now.
>
> None of the IPMI variants I have dealt with have been "reliable" they all
> seem to have major issues of various sorts, ie they don't always work
> exactly right, and if you are doing Serial over lan, things were even
> worse.
Chris-
Can you craft a diplomatically worded FAQ entry that encapsulates this
issue (without pointing fingers at specific NIC vendors ;-) ?
--
"We who cut mere stones must always be envisioning cathedrals"
-- Quarry worker's creed
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On 7/26/06, Michael Han <[email protected]> wrote:
> I'm not sure if this has surfaced before for discussion (searching
> hasn't shown me any relevant threads), but the 2.6 kernel's new
> implementation of xprt_bindresvport can conflict with port 664 on
> IPMI-enabled hosts with a BMC. IPMI uses both ports 623 and 664 for
> communications, and several implementations appear to intercept all
> traffic for these ports and not permit them to pass to the standard
> packet driver (tcpdump in promiscuous mode shows no packets coming in).
>
> I know that in the discussions back in 2005/02 and 2005/07 of the patch
> that implements the new privileged port binding for NFS mount (I can't
> tell whose patch it is, perhaps Olaf Kirch's?), Charles Lever noted that
> the new sunrpc.min_resvport of 650 avoids conflicts with port 623. Is it
> worth increasing this default to 665 to avoid this port as well?
>
> If not, I just wanted to get this information onto the list, since I
> searched pretty heavily while researching the intermittent hangs I've
> been getting with my NFS mounts before finally nailing this to my IPMI
> BMCs. Manually setting sunrpc.min_resvport to 700 has stabilized NFS on
> my boxes. Thanks.
Michael-
IPMI port number collisions are particularly hard to diagnose.
Congratulations on tracking this down.
Raising the minimum port number is one way to avoid port number
collisions. However, you are chopping away a lot of usable ports when
you do this, something that some sites can't afford. Especially if
you use automounter, or expect your clients to support more than a
couple hundred NFS mounts at once, you will probably need the expanded
port space.
A better way to eliminate the problem is to start a dummy daemon on
port 664 (or any other ports that the hardware might use
"transparently") so that the RPC client entirely avoids choosing those
port numbers. I believe that some enterprise distributions (RHEL? I
can't remember) have chosen to provide such daemons for just this
purpose.
Ask your distributor for more detail on this.
In the meantime, we should consider raising the default minimum port
number by 15 (or if we wanted to be particularly evil, we should chose
666 and be done with it).
--
"We who cut mere stones must always be envisioning cathedrals"
-- Quarry worker's creed
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On 7/28/06, Michael Han <[email protected]> wrote:
> > From Chuck Lever:
> > Raising the minimum port number is one way to avoid port number
> > collisions. However, you are chopping away a lot of usable ports when
> > you do this, something that some sites can't afford. Especially if
> > you use automounter, or expect your clients to support more than a
> > couple hundred NFS mounts at once, you will probably need the expanded
> > port space.
>
> For my own peculiar requirements, a diminished number of privileged
> source ports is perfectly fine. I don't have hosts that mount a large
> number of shares, even though I do have an automount home in some cases
> (I just don't have enough users on my production systems to need this).
"For the record," some sites have a requirement for a larger port space.
> > A better way to eliminate the problem is to start a dummy daemon on
> > port 664 (or any other ports that the hardware might use
> > "transparently") so that the RPC client entirely avoids choosing those
> > port numbers. I believe that some enterprise distributions (RHEL? I
> > can't remember) have chosen to provide such daemons for just this
> > purpose.
>
> I saw that this was the suggested workaround from mailing list postings,
> but saw a reference to some kind of automount or mount bug that still
> ran afoul port 664 somehow. Anyway, a "fake" daemon would show up on
> security port scans and be one more thing to explain at audit time
> (paperwork avoidance is always a worthy goal in my view ;).
The daemon actually wouldn't show up on the security scan. The
hardware IPMI listener would, however. The daemon is not visible on
the network because the IPMI listener diverts packets to that port.
Other workarounds worth mentioning: disable IPMI in the hardware, or
don't use the built-in NIC for NFS traffic.
> I notice that some NFS implementations can accept client mounts from
> non-privileged source ports (>= 1024). I'm not sure what the security
> ramifications are (off the top of my head, that doesn't seem appropriate
> for anyone relying on host-based security), but for truly large NFS
> setups, perhaps this might be something to consider? Or perhaps a scheme
> for re-using source ports may be feasible (or even using a single
> connection to plex multiple mounts from the same host as you'd see with
> an auto-home setup?).
For AUTH_SYS, the privileged port is expected, as a guarantee that the
uid's passed in the requests are more or less trusted. For AUTH_GSS,
non-privileged ports are just fine to use, as long as the server
supports it, and as long as dropping back to AUTH_SYS changes back to
a privileged port.
That's been considered, but is a low priority.
> Finally, fundamentally this seems to me a defect in the implmentation of
> IPMI BMCs. Shouldn't packets that cannot be processed by the BMC be
> passed to the standard kernel driver? Particularly for TCP, this seems
> like a good idea...
Indeed. I'm not familiar enough with IPMI to know if it listens on
both the UDP and the TCP port.
--
"We who cut mere stones must always be envisioning cathedrals"
-- Quarry worker's creed
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> From Chuck Lever:
> Raising the minimum port number is one way to avoid port number
> collisions. However, you are chopping away a lot of usable ports when
> you do this, something that some sites can't afford. Especially if
> you use automounter, or expect your clients to support more than a
> couple hundred NFS mounts at once, you will probably need the expanded
> port space.
For my own peculiar requirements, a diminished number of privileged
source ports is perfectly fine. I don't have hosts that mount a large
number of shares, even though I do have an automount home in some cases
(I just don't have enough users on my production systems to need this).
> A better way to eliminate the problem is to start a dummy daemon on
> port 664 (or any other ports that the hardware might use
> "transparently") so that the RPC client entirely avoids choosing those
> port numbers. I believe that some enterprise distributions (RHEL? I
> can't remember) have chosen to provide such daemons for just this
> purpose.
I saw that this was the suggested workaround from mailing list postings,
but saw a reference to some kind of automount or mount bug that still
ran afoul port 664 somehow. Anyway, a "fake" daemon would show up on
security port scans and be one more thing to explain at audit time
(paperwork avoidance is always a worthy goal in my view ;).
> In the meantime, we should consider raising the default minimum port
> number by 15 (or if we wanted to be particularly evil, we should chose
> 666 and be done with it).
That's why I suggested it. I'm not really sure adjusting port ranges is
a good generalized-case solution, but the loss of 15 (or 16 for the more
entertaining minimum value) usable ports doesn't seem likely to be
meaningful.
I notice that some NFS implementations can accept client mounts from
non-privileged source ports (>= 1024). I'm not sure what the security
ramifications are (off the top of my head, that doesn't seem appropriate
for anyone relying on host-based security), but for truly large NFS
setups, perhaps this might be something to consider? Or perhaps a scheme
for re-using source ports may be feasible (or even using a single
connection to plex multiple mounts from the same host as you'd see with
an auto-home setup?).
Finally, fundamentally this seems to me a defect in the implmentation of
IPMI BMCs. Shouldn't packets that cannot be processed by the BMC be
passed to the standard kernel driver? Particularly for TCP, this seems
like a good idea...
--
Michael Han
-----------------------------------------------------------
This message may contain confidential and/or privileged
information. This information is intended to be read only
by the individual or entity to whom it is addressed. If
you are not the intended recipient, you are on notice that
any review, disclosure, copying, distribution or use of
the contents of this message is strictly prohibited. If
you have received this message in error, please notify the
sender immediately and delete or destroy any copy of this
message.
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
>From Chuck Lever:
> "For the record," some sites have a requirement for a larger
> port space.
Naturally they do. auto-home systems with thousands of users could
easily cause this. I'm just pointing out that I'm satisfied with my own
workaround.
> The daemon actually wouldn't show up on the security scan. The
> hardware IPMI listener would, however. The daemon is not visible on
> the network because the IPMI listener diverts packets to that port.
Of course, you are correct. That's the crux of the problem I
encountered. Silly me.
> Other workarounds worth mentioning: disable IPMI in the hardware, or
> don't use the built-in NIC for NFS traffic.
Yes. Another possible alternative is to divert IPMI traffic to an
IPMI-only address. I'm not certain this works, but I know the SuperMicro
BMCs support use of alternate MAC & IP. I just don't know if the port
623/664 intercepts are promiscuous. I tried changing this on a hot
system to no avail, but not after rebooting a system and all that good
stuff.
> Indeed. I'm not familiar enough with IPMI to know if it listens on
> both the UDP and the TCP port.
I believe that in all implementations, IPMI only uses UDP
conventionally, however the port allocation from IANA is for both
transports and it appears that more than one implementation intercepts
both transports (I've seen this issue referenced on systems using Intel
NICs with IPMI support and on Sun x86 hardware). I'm pretty uneducated
as far as IPMI goes, myself.
--
Michael Han
-----------------------------------------------------------
This message may contain confidential and/or privileged
information. This information is intended to be read only
by the individual or entity to whom it is addressed. If
you are not the intended recipient, you are on notice that
any review, disclosure, copying, distribution or use of
the contents of this message is strictly prohibited. If
you have received this message in error, please notify the
sender immediately and delete or destroy any copy of this
message.
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Michael Han wrote:
>>From Chuck Lever:
>> "For the record," some sites have a requirement for a larger
>> port space.
>
> Naturally they do. auto-home systems with thousands of users could
> easily cause this. I'm just pointing out that I'm satisfied with my own
> workaround.
>
>> The daemon actually wouldn't show up on the security scan. The
>> hardware IPMI listener would, however. The daemon is not visible on
>> the network because the IPMI listener diverts packets to that port.
>
> Of course, you are correct. That's the crux of the problem I
> encountered. Silly me.
>
>> Other workarounds worth mentioning: disable IPMI in the hardware, or
>> don't use the built-in NIC for NFS traffic.
>
> Yes. Another possible alternative is to divert IPMI traffic to an
> IPMI-only address. I'm not certain this works, but I know the SuperMicro
> BMCs support use of alternate MAC & IP. I just don't know if the port
> 623/664 intercepts are promiscuous. I tried changing this on a hot
> system to no avail, but not after rebooting a system and all that good
> stuff.
>
>> Indeed. I'm not familiar enough with IPMI to know if it listens on
>> both the UDP and the TCP port.
>
> I believe that in all implementations, IPMI only uses UDP
> conventionally, however the port allocation from IANA is for both
> transports and it appears that more than one implementation intercepts
> both transports (I've seen this issue referenced on systems using Intel
> NICs with IPMI support and on Sun x86 hardware). I'm pretty uneducated
> as far as IPMI goes, myself.
>
Some versions (maybe all) of broadcom chips will intercept *ANY* udp frame
with the proper port number in the proper byte, even if that byte is not
a port number, ie they will intercept the additional frames that make up a
32k udp packet that has the proper data in the port space, even though it
is not a port number as the extra frames don't have a port designation.
The broadcom chip is doing this in firmware, and it results in the nth
frame of a 32k udp packet always being lost even on retransmit, and it
very much depends on the data being sent to have the wrong number in
the wrong location, and it has been reported to Broadcom.
It means that it is difficult to run the Broadcom ethernet ports with ipmi
the same ip/mac address and have things reliable.
The last time I worked with the intel e1000's bmc they were troublesome and
unreliable in many other different way,s though that may be fixed now.
None of the IPMI variants I have dealt with have been "reliable" they all
seem to have major issues of various sorts, ie they don't always work
exactly right, and if you are doing Serial over lan, things were even
worse.
Roger
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs