2015-04-01 21:48:44

by Alex Williamson

Subject: Re: [PATCH kernel v7 04/31] vfio: powerpc/spapr: Use it_page_size

On Sat, 2015-03-28 at 01:54 +1100, Alexey Kardashevskiy wrote:
> This makes use of the it_page_size from the iommu_table struct
> as page size can differ.
>
> This replaces missing IOMMU_PAGE_SHIFT macro in commented debug code
> as recently introduced IOMMU_PAGE_XXX macros do not include
> IOMMU_PAGE_SHIFT.
>
> Signed-off-by: Alexey Kardashevskiy <[email protected]>
> Reviewed-by: David Gibson <[email protected]>
> ---
> drivers/vfio/vfio_iommu_spapr_tce.c | 26 +++++++++++++-------------
> 1 file changed, 13 insertions(+), 13 deletions(-)
>
> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> index f835e63..8bbee22 100644
> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> @@ -91,7 +91,7 @@ static int tce_iommu_enable(struct tce_container *container)
> * enforcing the limit based on the max that the guest can map.
> */
> down_write(&current->mm->mmap_sem);
> - npages = (tbl->it_size << IOMMU_PAGE_SHIFT_4K) >> PAGE_SHIFT;
> + npages = (tbl->it_size << tbl->it_page_shift) >> PAGE_SHIFT;
> locked = current->mm->locked_vm + npages;
> lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> if (locked > lock_limit && !capable(CAP_IPC_LOCK)) {
> @@ -120,7 +120,7 @@ static void tce_iommu_disable(struct tce_container *container)
>
> down_write(&current->mm->mmap_sem);
> current->mm->locked_vm -= (container->tbl->it_size <<
> - IOMMU_PAGE_SHIFT_4K) >> PAGE_SHIFT;
> + container->tbl->it_page_shift) >> PAGE_SHIFT;
> up_write(&current->mm->mmap_sem);
> }
>
> @@ -222,7 +222,7 @@ static long tce_iommu_build(struct tce_container *container,
> tce, ret);
> break;
> }
> - tce += IOMMU_PAGE_SIZE_4K;
> + tce += IOMMU_PAGE_SIZE(tbl);


Is PAGE_SIZE ever smaller than IOMMU_PAGE_SIZE(tbl)? IOW, can the page
we got from get_user_pages_fast() ever not completely fill the tce
entry?

(Have I asked this before? Sorry if so)

2015-04-02 02:30:56

by Alexey Kardashevskiy

Subject: Re: [PATCH kernel v7 04/31] vfio: powerpc/spapr: Use it_page_size

On 04/02/2015 08:48 AM, Alex Williamson wrote:
> On Sat, 2015-03-28 at 01:54 +1100, Alexey Kardashevskiy wrote:
>> This makes use of the it_page_size from the iommu_table struct
>> as page size can differ.
>>
>> This replaces missing IOMMU_PAGE_SHIFT macro in commented debug code
>> as recently introduced IOMMU_PAGE_XXX macros do not include
>> IOMMU_PAGE_SHIFT.
>>
>> Signed-off-by: Alexey Kardashevskiy <[email protected]>
>> Reviewed-by: David Gibson <[email protected]>
>> ---
>> drivers/vfio/vfio_iommu_spapr_tce.c | 26 +++++++++++++-------------
>> 1 file changed, 13 insertions(+), 13 deletions(-)
>>
>> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
>> index f835e63..8bbee22 100644
>> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
>> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
>> @@ -91,7 +91,7 @@ static int tce_iommu_enable(struct tce_container *container)
>> * enforcing the limit based on the max that the guest can map.
>> */
>> down_write(&current->mm->mmap_sem);
>> - npages = (tbl->it_size << IOMMU_PAGE_SHIFT_4K) >> PAGE_SHIFT;
>> + npages = (tbl->it_size << tbl->it_page_shift) >> PAGE_SHIFT;
>> locked = current->mm->locked_vm + npages;
>> lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
>> if (locked > lock_limit && !capable(CAP_IPC_LOCK)) {
>> @@ -120,7 +120,7 @@ static void tce_iommu_disable(struct tce_container *container)
>>
>> down_write(&current->mm->mmap_sem);
>> current->mm->locked_vm -= (container->tbl->it_size <<
>> - IOMMU_PAGE_SHIFT_4K) >> PAGE_SHIFT;
>> + container->tbl->it_page_shift) >> PAGE_SHIFT;
>> up_write(&current->mm->mmap_sem);
>> }
>>
>> @@ -222,7 +222,7 @@ static long tce_iommu_build(struct tce_container *container,
>> tce, ret);
>> break;
>> }
>> - tce += IOMMU_PAGE_SIZE_4K;
>> + tce += IOMMU_PAGE_SIZE(tbl);
>
>
> Is PAGE_SIZE ever smaller than IOMMU_PAGE_SIZE(tbl)? IOW, can the page
> we got from get_user_pages_fast() ever not completely fill the tce
> entry?


Yes. IOMMU_PAGE_SIZE is 4K/64K/16M (16M is with huge pages enabled in QEMU
with -mempath), PAGE_SIZE is 4K/64K (normally 64K).


> (Have I asked this before? Sorry if so)


:)



--
Alexey

2015-04-02 02:51:42

by Alex Williamson

Subject: Re: [PATCH kernel v7 04/31] vfio: powerpc/spapr: Use it_page_size

On Thu, 2015-04-02 at 13:30 +1100, Alexey Kardashevskiy wrote:
> On 04/02/2015 08:48 AM, Alex Williamson wrote:
> > On Sat, 2015-03-28 at 01:54 +1100, Alexey Kardashevskiy wrote:
> >> This makes use of the it_page_size from the iommu_table struct
> >> as page size can differ.
> >>
> >> This replaces missing IOMMU_PAGE_SHIFT macro in commented debug code
> >> as recently introduced IOMMU_PAGE_XXX macros do not include
> >> IOMMU_PAGE_SHIFT.
> >>
> >> Signed-off-by: Alexey Kardashevskiy <[email protected]>
> >> Reviewed-by: David Gibson <[email protected]>
> >> ---
> >> drivers/vfio/vfio_iommu_spapr_tce.c | 26 +++++++++++++-------------
> >> 1 file changed, 13 insertions(+), 13 deletions(-)
> >>
> >> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> >> index f835e63..8bbee22 100644
> >> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> >> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> >> @@ -91,7 +91,7 @@ static int tce_iommu_enable(struct tce_container *container)
> >> * enforcing the limit based on the max that the guest can map.
> >> */
> >> down_write(&current->mm->mmap_sem);
> >> - npages = (tbl->it_size << IOMMU_PAGE_SHIFT_4K) >> PAGE_SHIFT;
> >> + npages = (tbl->it_size << tbl->it_page_shift) >> PAGE_SHIFT;
> >> locked = current->mm->locked_vm + npages;
> >> lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> >> if (locked > lock_limit && !capable(CAP_IPC_LOCK)) {
> >> @@ -120,7 +120,7 @@ static void tce_iommu_disable(struct tce_container *container)
> >>
> >> down_write(&current->mm->mmap_sem);
> >> current->mm->locked_vm -= (container->tbl->it_size <<
> >> - IOMMU_PAGE_SHIFT_4K) >> PAGE_SHIFT;
> >> + container->tbl->it_page_shift) >> PAGE_SHIFT;
> >> up_write(&current->mm->mmap_sem);
> >> }
> >>
> >> @@ -222,7 +222,7 @@ static long tce_iommu_build(struct tce_container *container,
> >> tce, ret);
> >> break;
> >> }
> >> - tce += IOMMU_PAGE_SIZE_4K;
> >> + tce += IOMMU_PAGE_SIZE(tbl);
> >
> >
> > Is PAGE_SIZE ever smaller than IOMMU_PAGE_SIZE(tbl)? IOW, can the page
> > we got from get_user_pages_fast() ever not completely fill the tce
> > entry?
>
>
> Yes. IOMMU_PAGE_SIZE is 4K/64K/16M (16M is with huge pages enabled in QEMU
> with -mempath), PAGE_SIZE is 4K/64K (normally 64K).

Isn't that a problem then that you're filling the tce with processor
page sizes via get_user_pages_fast(), but incrementing the tce by by
IOMMU page size? For example, if PAGE_SIZE = 4K and IOMMU_PAGE_SIZE !=
4K have we really pinned all of the memory backed by the tce? Where do
you make sure the 4K page is really contiguous for the IOMMU page?

2015-04-02 03:40:17

by Alexey Kardashevskiy

Subject: Re: [PATCH kernel v7 04/31] vfio: powerpc/spapr: Use it_page_size

On 04/02/2015 01:50 PM, Alex Williamson wrote:
> On Thu, 2015-04-02 at 13:30 +1100, Alexey Kardashevskiy wrote:
>> On 04/02/2015 08:48 AM, Alex Williamson wrote:
>>> On Sat, 2015-03-28 at 01:54 +1100, Alexey Kardashevskiy wrote:
>>>> This makes use of the it_page_size from the iommu_table struct
>>>> as page size can differ.
>>>>
>>>> This replaces missing IOMMU_PAGE_SHIFT macro in commented debug code
>>>> as recently introduced IOMMU_PAGE_XXX macros do not include
>>>> IOMMU_PAGE_SHIFT.
>>>>
>>>> Signed-off-by: Alexey Kardashevskiy <[email protected]>
>>>> Reviewed-by: David Gibson <[email protected]>
>>>> ---
>>>> drivers/vfio/vfio_iommu_spapr_tce.c | 26 +++++++++++++-------------
>>>> 1 file changed, 13 insertions(+), 13 deletions(-)
>>>>
>>>> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
>>>> index f835e63..8bbee22 100644
>>>> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
>>>> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
>>>> @@ -91,7 +91,7 @@ static int tce_iommu_enable(struct tce_container *container)
>>>> * enforcing the limit based on the max that the guest can map.
>>>> */
>>>> down_write(&current->mm->mmap_sem);
>>>> - npages = (tbl->it_size << IOMMU_PAGE_SHIFT_4K) >> PAGE_SHIFT;
>>>> + npages = (tbl->it_size << tbl->it_page_shift) >> PAGE_SHIFT;
>>>> locked = current->mm->locked_vm + npages;
>>>> lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
>>>> if (locked > lock_limit && !capable(CAP_IPC_LOCK)) {
>>>> @@ -120,7 +120,7 @@ static void tce_iommu_disable(struct tce_container *container)
>>>>
>>>> down_write(&current->mm->mmap_sem);
>>>> current->mm->locked_vm -= (container->tbl->it_size <<
>>>> - IOMMU_PAGE_SHIFT_4K) >> PAGE_SHIFT;
>>>> + container->tbl->it_page_shift) >> PAGE_SHIFT;
>>>> up_write(&current->mm->mmap_sem);
>>>> }
>>>>
>>>> @@ -222,7 +222,7 @@ static long tce_iommu_build(struct tce_container *container,
>>>> tce, ret);
>>>> break;
>>>> }
>>>> - tce += IOMMU_PAGE_SIZE_4K;
>>>> + tce += IOMMU_PAGE_SIZE(tbl);
>>>
>>>
>>> Is PAGE_SIZE ever smaller than IOMMU_PAGE_SIZE(tbl)? IOW, can the page
>>> we got from get_user_pages_fast() ever not completely fill the tce
>>> entry?
>>
>>
>> Yes. IOMMU_PAGE_SIZE is 4K/64K/16M (16M is with huge pages enabled in QEMU
>> with -mempath), PAGE_SIZE is 4K/64K (normally 64K).
>
> Isn't that a problem then that you're filling the tce with processor
> page sizes via get_user_pages_fast(), but incrementing the tce by by
> IOMMU page size? For example, if PAGE_SIZE = 4K and IOMMU_PAGE_SIZE !=
> 4K have we really pinned all of the memory backed by the tce?Where do
> you make sure the 4K page is really contiguous for the IOMMU page?


Aaaah. This is just not supported. Instead, after the previous patch
("vfio: powerpc/spapr: Check that TCE page size is equal to it_page_size",
which need fixed subject), tce_page_is_contained(page4K, 64K) will return
false and the caller - tce_iommu_build() - will return -EPERM.


--
Alexey

2015-04-02 03:46:21

by Alex Williamson

Subject: Re: [PATCH kernel v7 04/31] vfio: powerpc/spapr: Use it_page_size

On Thu, 2015-04-02 at 14:40 +1100, Alexey Kardashevskiy wrote:
> On 04/02/2015 01:50 PM, Alex Williamson wrote:
> > On Thu, 2015-04-02 at 13:30 +1100, Alexey Kardashevskiy wrote:
> >> On 04/02/2015 08:48 AM, Alex Williamson wrote:
> >>> On Sat, 2015-03-28 at 01:54 +1100, Alexey Kardashevskiy wrote:
> >>>> This makes use of the it_page_size from the iommu_table struct
> >>>> as page size can differ.
> >>>>
> >>>> This replaces missing IOMMU_PAGE_SHIFT macro in commented debug code
> >>>> as recently introduced IOMMU_PAGE_XXX macros do not include
> >>>> IOMMU_PAGE_SHIFT.
> >>>>
> >>>> Signed-off-by: Alexey Kardashevskiy <[email protected]>
> >>>> Reviewed-by: David Gibson <[email protected]>
> >>>> ---
> >>>> drivers/vfio/vfio_iommu_spapr_tce.c | 26 +++++++++++++-------------
> >>>> 1 file changed, 13 insertions(+), 13 deletions(-)
> >>>>
> >>>> diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
> >>>> index f835e63..8bbee22 100644
> >>>> --- a/drivers/vfio/vfio_iommu_spapr_tce.c
> >>>> +++ b/drivers/vfio/vfio_iommu_spapr_tce.c
> >>>> @@ -91,7 +91,7 @@ static int tce_iommu_enable(struct tce_container *container)
> >>>> * enforcing the limit based on the max that the guest can map.
> >>>> */
> >>>> down_write(&current->mm->mmap_sem);
> >>>> - npages = (tbl->it_size << IOMMU_PAGE_SHIFT_4K) >> PAGE_SHIFT;
> >>>> + npages = (tbl->it_size << tbl->it_page_shift) >> PAGE_SHIFT;
> >>>> locked = current->mm->locked_vm + npages;
> >>>> lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> >>>> if (locked > lock_limit && !capable(CAP_IPC_LOCK)) {
> >>>> @@ -120,7 +120,7 @@ static void tce_iommu_disable(struct tce_container *container)
> >>>>
> >>>> down_write(&current->mm->mmap_sem);
> >>>> current->mm->locked_vm -= (container->tbl->it_size <<
> >>>> - IOMMU_PAGE_SHIFT_4K) >> PAGE_SHIFT;
> >>>> + container->tbl->it_page_shift) >> PAGE_SHIFT;
> >>>> up_write(&current->mm->mmap_sem);
> >>>> }
> >>>>
> >>>> @@ -222,7 +222,7 @@ static long tce_iommu_build(struct tce_container *container,
> >>>> tce, ret);
> >>>> break;
> >>>> }
> >>>> - tce += IOMMU_PAGE_SIZE_4K;
> >>>> + tce += IOMMU_PAGE_SIZE(tbl);
> >>>
> >>>
> >>> Is PAGE_SIZE ever smaller than IOMMU_PAGE_SIZE(tbl)? IOW, can the page
> >>> we got from get_user_pages_fast() ever not completely fill the tce
> >>> entry?
> >>
> >>
> >> Yes. IOMMU_PAGE_SIZE is 4K/64K/16M (16M is with huge pages enabled in QEMU
> >> with -mempath), PAGE_SIZE is 4K/64K (normally 64K).
> >
> > Isn't that a problem then that you're filling the tce with processor
> > page sizes via get_user_pages_fast(), but incrementing the tce by by
> > IOMMU page size? For example, if PAGE_SIZE = 4K and IOMMU_PAGE_SIZE !=
> > 4K have we really pinned all of the memory backed by the tce?Where do
> > you make sure the 4K page is really contiguous for the IOMMU page?
>
>
> Aaaah. This is just not supported. Instead, after the previous patch
> ("vfio: powerpc/spapr: Check that TCE page size is equal to it_page_size",
> which need fixed subject), tce_page_is_contained(page4K, 64K) will return
> false and the caller - tce_iommu_build() - will return -EPERM.

Ok, makes sense. Thanks