Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Krishna Reddy <vdumpa@nvidia.com>
To:     Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org>,
        Robin Murphy <robin.murphy@arm.com>
CC:     "linux-arm-msm@vger.kernel.org" <linux-arm-msm@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        Will Deacon <will@kernel.org>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        Thierry Reding <treding@nvidia.com>
Subject: RE: [PATCH] iommu/io-pgtable-arm: Optimize partial walk flush for
 large scatter-gather list
Thread-Topic: [PATCH] iommu/io-pgtable-arm: Optimize partial walk flush for
 large scatter-gather list
Thread-Index: AQHXXT84btg0466hEE2rkvq94oyuZasMBHAAgACy+QCAAD6WAIAAB6mAgAAg2ACAAAWcgIAAPESAgACQdxCAAA05AIABAqfQgACvFQCABA+AcA==
Date:   Mon, 14 Jun 2021 17:48:36 +0000
Message-ID: <BY5PR12MB376433B3FD0A59EF57C4522DB3319@BY5PR12MB3764.namprd12.prod.outlook.com>
References: <20210609145315.25750-1-saiprakash.ranjan@codeaurora.org>
 <dbcd394a-4d85-316c-5dd0-033546a66132@arm.com>
 <c600e9b2534d54082a5272b508a7985f@codeaurora.org>
 <35bfd245-45e2-8083-b620-330d6dbd7bd7@arm.com>
 <12067ffb8243b220cf03e83aaac3e823@codeaurora.org>
 <266f190e-99ae-9175-cf13-7a77730af389@arm.com>
 <dfdabcdec99a4c6e3bf2b3c5eebe067f@codeaurora.org>
 <61c69d23-324a-85d7-2458-dfff8df9280b@arm.com>
 <BY5PR12MB37646698F37C00381EFF7C77B3349@BY5PR12MB3764.namprd12.prod.outlook.com>
 <07001b4ed6c0a491eacce6e4dc13ab5e@codeaurora.org>
 <BY5PR12MB376480219C42E5FCE0FE0FFBB3349@BY5PR12MB3764.namprd12.prod.outlook.com>
 <f749ba0957b516ab5f0ea57033d308c7@codeaurora.org>
In-Reply-To: <f749ba0957b516ab5f0ea57033d308c7@codeaurora.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?fsK7q6nWPaFK6tMDkv18bFSpUPNSltGdZrBDLBmtd+VaWy4mlIAgJf6XNhsd?=
 =?us-ascii?Q?p6ywU9lzGHnMLp/ny6x509F5AdJLg//HImuR7GG0pj+QITwBRshEyns2cVN/?=
 =?us-ascii?Q?aHNc7hOGUAJJLg0/QGeTQwX0zKNshslQktzYsE9bxVunEyvokLtGfimhNiPT?=
 =?us-ascii?Q?m9Of6hQjI6h/Cq/nVzZ57ThiER7q3TuS5biOHHNCSH3t/oa957eCZXcx/EUB?=
 =?us-ascii?Q?XXvIBtzn6v257CrPyWDraGuoeoNy7ecp+2uDRBgMpcrr8fB/mN4Pt/UiqXDV?=
 =?us-ascii?Q?+FdHK78jkhhj5JCfKyF6LJgok0h75G4xLFVDckxeTPHmq2J87iVVQe5fk/sD?=
 =?us-ascii?Q?vodsFoY1ku1wVg8aQKQOc/jJJ66IFoKamP06CtIhkpYBw2BEkd9pdBSghUhT?=
 =?us-ascii?Q?gcw8YClt4l2XbAWYSppycMndH/Z/yu9U91bRjX/CIqDADo132bgD5ApqfJYb?=
 =?us-ascii?Q?fJDZh+eDD3V2P+w5Wyic4YLdOxIFT7jxA3b/UEQTC7qOXHEPNxqQNEPRdCfS?=
 =?us-ascii?Q?skia5c0VRJ1ZWL+3n8XEGTIhL5LPY3RzMEt10w5cGHt4RlMBk/pIAosvkEnN?=
 =?us-ascii?Q?OjdES098PKM1IpgorJLXI9hBROKwCGl7Tmz5HK+U6heVMelJMhEaQumv0Q96?=
 =?us-ascii?Q?KqLCw9rGGYF9YUW7vT62nW6nfWSkIl/FRtqAY/n95WhIAiuyk8kBWTKf0WbO?=
 =?us-ascii?Q?COzsC8D0nfd322mMIrToH6CZVzInyBrmzbTabOWVR6H8caqZVO6Hlqw3MSPZ?=
 =?us-ascii?Q?30SVAlQlfkozb3tN9e0du2wnfZVfrO8ZtG/v25LMErx0Qq5q316bK9kb9mvP?=
 =?us-ascii?Q?gTyqm6PIQDE/sJCE+UAnLgVE0fhJ1USpUIe8MRCbdLqd8P/5zoGQVXLWhJW7?=
 =?us-ascii?Q?NQHPQz5HpsE5l0vm4l4+PNeN2QS74U+iFDYoSgmUckRf3YXVWeY1eHpZME8z?=
 =?us-ascii?Q?eA+kQY4K/Xk7A+qyBkTQi7oKoSQQdLO2EEBDopUAW/p7vjFAW/wSN+kkndE5?=
 =?us-ascii?Q?bmJRa29WNfMK1R7AMhFleebp+zQIANSyDXxfUs9zmdy+Rw70+wtb7V78BVJX?=
 =?us-ascii?Q?vXxldBEBI3g3bG9VtNMoWtXlmM0PxjhbCLt9DEQWZ3zH0jd3mwLhrKmOs3b+?=
 =?us-ascii?Q?+K/H1zXINC0/QHtpi7adPVQD2VUbyoBkhjbgP4ZAo2UkuC0f/RYDu1o3vGUK?=
 =?us-ascii?Q?jDe9cgy0SyUholiVqCbgY3/aWHIjSfUsRikTvfPVqKmwS7efMZsuxAH/7rq8?=
 =?us-ascii?Q?T8iu2lfkVcQZGdTG6wE/JzvPcUslNd1qAQDc9cQp3HVSkvYDSCb5dasPM6B9?=
 =?us-ascii?Q?8HQ9rLAD3+WOAKXMX9656/nv?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3764.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d243624d-1bd2-47de-3da1-08d92f5ca2ee
X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Jun 2021 17:48:36.5055
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: SeXKMeWw3I0Cszd3zBKVhSjtAvsjiYvd0C/nrRcsS9VrQjeEEcdRz02iJb5198l9NdxjvsEXeX2e5Dy+oYVL2g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB3077
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> Right but we won't know until we profile the specific usecases or try the=
m in
> generic workload to see if they affect the performance. Sure, over invali=
dation is
> a concern where multiple buffers can be mapped to same context and the ca=
che
> is not usable at the time for lookup and such but we don't do it for smal=
l buffers
> and only for large buffers which means thousands of TLB entry mappings in
> which case TLBIASID is preferred (note: I mentioned the HW team
> recommendation to use it for anything greater than 128 TLB entries) in my
> earlier reply. And also note that we do this only for partial walk flush,=
 we are not
> arbitrarily changing all the TLBIs to ASID based.

Most of the heavy bw use cases does involve processing larger buffers.
When the physical memory is allocated dis-contiguously at page_size (let's =
use 4KB here)
granularity, each aligned 2MB chunks IOVA unmap would involve performing a =
TLBIASID
as 2MB is not a leaf. Essentially, It happens all the time during large buf=
fer unmaps and
potentially impact active traffic on other large buffers. Depending on how =
much
latency HW engines can absorb, the overflow/underflow issues for ISO engine=
s can be
sporadic and vendor specific.=20
Performing TLBIASID as default for all SoCs is not a safe operation.


> I am no camera expert but from what the camera team mentioned is that the=
re
> is a thread which frees memory(large unused memory buffers) periodically =
which
> ends up taking around 100+ms and causing some camera test failures with
> frame drops. Parallel efforts are already being made to optimize this usa=
ge of
> thread but as I mentioned previously, this is *not a camera specific*, le=
ts say
> someone else invokes such large unmaps, it's going to face the same issue=
.

From the above, It doesn't look like the root cause of frame drops is fully=
 understood.
Why is 100+ms delay causing camera frame drop?  Is the same thread submitti=
ng the buffers
to camera after unmap is complete? If not, how is the unmap latency causing=
 issue here?
=20

> > If unmap is queued and performed on a back ground thread, would it
> > resolve the frame drops?
>=20
> Not sure I understand what you mean by queuing on background thread but w=
ith
> that or not, we still do the same number of TLBIs and hop through
> iommu->io-pgtable->arm-smmu to perform the the unmap, so how will that
> help?

I mean adding the unmap requests into a queue and processing them from a di=
fferent thread.
It is not to reduce the TLBIs. But, not to block subsequent buffer allocati=
on, IOVA map requests, if they
are being requested from same thread that is performing unmap. If unmap is =
already performed from
a different thread, then the issue still need to be root caused to understa=
nd it fully. Check for any
serialization issues.=20


-KR