On Sun, Dec 22, 2013 at 5:14 PM, Yinghai Lu <[email protected]> wrote:
> On Sun, Dec 22, 2013 at 4:00 PM, Bjorn Helgaas <[email protected]> wrote:
>> On Thu, Dec 19, 2013 at 1:44 PM, Yinghai Lu <[email protected]> wrote:
>>
>> Let me see if I can figure out what you're trying to do here. Please
>> correct me if I'm wrong:
>>
>>> When one of children resources does not support MEM_64, MEM_64 for
>>> bridge get reset, so pull down whole pref resource on the bridge under 4G.
>>
>> When we allocate space for a bridge's prefetchable window, we
>> currently look at the devices behind the bridge and put the window
>> below 4GB if any of those children has a 32-bit prefetchable BAR.
>>
>> This maximizes the use of prefetch, at the cost of using more 32-bit
>> address space.
>
> yes. and we have problem when we have 8 sockets or 32 sockets system,
> will have limit 32bit space.
> but we have enough above 4G 64bit mmio for prefetchable.
>
>>
>>> If the bridge support pref mem 64, will only allocate that with pref mem64 to
>>> children that support it.
>>> For children resources if they only support pref mem 32, will allocate them
>>> from non pref mem instead.
>>
>> You are changing this so that we will always try to put a bridge's
>> 64-bit prefetchable window above 4GB, regardless of what devices are
>> behind the bridge. If a device behind the bridge has a 32-bit
>> prefetchable BAR, we will place that BAR in the bridge's 32-bit
>> non-prefetchable window.
>
> Yes. so we can keep IORESOURCE_MEM64 in the flags for PREF.
>
>>
>> This minimizes the use of the 32-bit address space, at the cost of not
>> being able to use prefetch as much.
>>
>>> If the bridge only support 32bit pref mmio, will still have all children pref
>>> mmio under that.
>>
>> Obviously, if a bridge has a prefetchable window that's only 32 bits,
>> 64-bit prefetchable BARs behind the bridge will have to be in that
>> 32-bit prefetchable window or the 32-bit non-prefetchable window. And
>> if the bridge has no prefetchable window at all, every memory BAR
>> behind the bridge will have to be in the 32-bit non-prefetchable
>> window.
>
> Yes.
>
>>
>> I'll look at the actual patch later; I just want to make sure I
>> understand your intent first.
Hi, Bjorn,
Can you check and add this one to your pci/resource branch?
With that we can close the loop for 64bit mmio resource allocation.
Thanks
Yinghai
On Wed, Jan 08, 2014 at 03:34:54PM -0800, Yinghai Lu wrote:
> On Sun, Dec 22, 2013 at 5:14 PM, Yinghai Lu <[email protected]> wrote:
> > On Sun, Dec 22, 2013 at 4:00 PM, Bjorn Helgaas <[email protected]> wrote:
> >> On Thu, Dec 19, 2013 at 1:44 PM, Yinghai Lu <[email protected]> wrote:
> >>
> >> Let me see if I can figure out what you're trying to do here. Please
> >> correct me if I'm wrong:
> >>
> >>> When one of children resources does not support MEM_64, MEM_64 for
> >>> bridge get reset, so pull down whole pref resource on the bridge under 4G.
> >>
> >> When we allocate space for a bridge's prefetchable window, we
> >> currently look at the devices behind the bridge and put the window
> >> below 4GB if any of those children has a 32-bit prefetchable BAR.
> >>
> >> This maximizes the use of prefetch, at the cost of using more 32-bit
> >> address space.
> >
> > yes. and we have problem when we have 8 sockets or 32 sockets system,
> > will have limit 32bit space.
> > but we have enough above 4G 64bit mmio for prefetchable.
> >
> >>
> >>> If the bridge support pref mem 64, will only allocate that with pref mem64 to
> >>> children that support it.
> >>> For children resources if they only support pref mem 32, will allocate them
> >>> from non pref mem instead.
> >>
> >> You are changing this so that we will always try to put a bridge's
> >> 64-bit prefetchable window above 4GB, regardless of what devices are
> >> behind the bridge. If a device behind the bridge has a 32-bit
> >> prefetchable BAR, we will place that BAR in the bridge's 32-bit
> >> non-prefetchable window.
> >
> > Yes. so we can keep IORESOURCE_MEM64 in the flags for PREF.
> >
> >>
> >> This minimizes the use of the 32-bit address space, at the cost of not
> >> being able to use prefetch as much.
> >>
> >>> If the bridge only support 32bit pref mmio, will still have all children pref
> >>> mmio under that.
> >>
> >> Obviously, if a bridge has a prefetchable window that's only 32 bits,
> >> 64-bit prefetchable BARs behind the bridge will have to be in that
> >> 32-bit prefetchable window or the 32-bit non-prefetchable window. And
> >> if the bridge has no prefetchable window at all, every memory BAR
> >> behind the bridge will have to be in the 32-bit non-prefetchable
> >> window.
> >
> > Yes.
> >
> >>
> >> I'll look at the actual patch later; I just want to make sure I
> >> understand your intent first.
>
> Hi, Bjorn,
>
> Can you check and add this one to your pci/resource branch?
> With that we can close the loop for 64bit mmio resource allocation.
>
Just FYI, a Mellanox net card failed after exactly this patch.
3.13-rc7 + bjorn's series is OK. After this patch applied, Mellanox
driver complains:
|mlx4_core 0003:05:00.0: Multiple PFs not yet supported. Skipping PF.
|mlx4_core: probe of 0003:05:00.0 failed with error -22
This is caused by MMIO read from BAR 0 (64-bit non-prefetchable) returns
non-zore value.
Resource assignment, as far as we can see, works fine. The noticable
effect of this patch is putting ROM BAR under non-prefetachable. I try
to revert this effect by adding MEM_64 to its ROM resource and it works
again (system does not expose 4G above aperture yet). Not sure what's
the root cause, looks like a driver/firmware/hardware defect.
Thanks
Guo Chao
> Thanks
>
> Yinghai
>
On Fri, Jan 10, 2014 at 1:41 AM, Guo Chao <[email protected]> wrote:
> On Wed, Jan 08, 2014 at 03:34:54PM -0800, Yinghai Lu wrote:
> Just FYI, a Mellanox net card failed after exactly this patch.
>
> 3.13-rc7 + bjorn's series is OK. After this patch applied, Mellanox
> driver complains:
>
> |mlx4_core 0003:05:00.0: Multiple PFs not yet supported. Skipping PF.
> |mlx4_core: probe of 0003:05:00.0 failed with error -22
>
> This is caused by MMIO read from BAR 0 (64-bit non-prefetchable) returns
> non-zore value.
>
> Resource assignment, as far as we can see, works fine. The noticable
> effect of this patch is putting ROM BAR under non-prefetachable. I try
> to revert this effect by adding MEM_64 to its ROM resource and it works
> again (system does not expose 4G above aperture yet). Not sure what's
> the root cause, looks like a driver/firmware/hardware defect.
Interesting. Can you post boot log with "debug ignore_loglevel initcall_debug"
and with/without this patch?
Thanks
Yinghai