Received-SPF: pass (google.com: domain of linux-kernel+bounces-55460-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <39ae9ceb-e872-4dda-9dd0-b9e97babe39f@amd.com>
Date: Tue, 6 Feb 2024 12:32:50 -0600
User-Agent: Mozilla Thunderbird Beta
Subject: Re: [PATCH] remoteproc: zynqmp: fix lockstep mode memory region
Content-Language: en-US
To: Mathieu Poirier <mathieu.poirier@linaro.org>
Cc: andersson@kernel.org, linux-remoteproc@vger.kernel.org,
 linux-kernel@vger.kernel.org
References: <20240201004812.1471407-1-tanmay.shah@amd.com>
 <ZcJu10CIVYF9iodF@p14s>
From: Tanmay Shah <tanmay.shah@amd.com>
In-Reply-To: <ZcJu10CIVYF9iodF@p14s>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?c1FnOEh3QStYNGphL1VHSnlaZFV4c0czVUErdk4xaW9Ma2JlNVNzYzIxTnhD?=
 =?utf-8?B?eW9RRWFPUERlZmZnQ1BvcWkzL3ZRcUFScmlzRmtWeVpBaUFNMmFEMDFFdDhT?=
 =?utf-8?B?WVphT3QvYU5oNzBzTU9yck9JTnFBSHgzRG9ldzZUZmk0dThRYllNNGdXQzM2?=
 =?utf-8?B?b1Jrd2dSQzNxbEVKT2thbUQwcVhFTHNvNk95emRIU3JMUGJFZVBOV0dtY2F0?=
 =?utf-8?B?NW5YR0FadGx6RDkrM3R4UWpJMVprb3dUb2d6a3FYSHFZbllPQmZnd1VJNjA5?=
 =?utf-8?B?L25JVW4rUnEyczB3bHRxZzVseGhZZnRjaEM3OTh2OTkwdFplSGpWTzdCMnBR?=
 =?utf-8?B?ODlsejFiVmNMaFdtTmRNcG9UUDZNalBwV1htQ285a0Y0QmgwNWNPWnExcTlG?=
 =?utf-8?B?bzMvcUNpQW00WWZ3R3pyNGkzeEw3b0dyVllDV1lLU1JJcW05NTlTWUJSZklT?=
 =?utf-8?B?UTNGZlVkb1dMWUF6UXJNOE1aZzloMU1LWFFwRlZWcnJwSzRMZUUrS283cnBm?=
 =?utf-8?B?VkFCK0g0OGswZHZFakl6TEl1S1VGRlltTjdXdUdzOGhBVGZpQVQwSU5NZ2Fr?=
 =?utf-8?B?UHpMOWxGUzkyVmxHc2ptT0FJYjhTV242UWxaNm1qSGN1NFZQVHV4SkdkeGUv?=
 =?utf-8?B?V094bkFzR2M2eXpad1l5bDBJM0YrcFMvRU13SG01MmF4NzNQeFQ3M1MvbU5R?=
 =?utf-8?B?VytDUys2NzVYN0pJWkRKejhkU09Sazdrb3ZQWUY2ek5QS2ZkU251a1VBZjcr?=
 =?utf-8?B?NjVpWGNDeWNZamNVTEk2c2MyQjR6Vk94L1lYMEdZS2lBZVVtVGtpeUxOa1NR?=
 =?utf-8?B?VUlSTWE3ZSszU253bzdsWmx3TTNpcG1RSUtXdlNVS0puYVFneE55UDk2RU5P?=
 =?utf-8?B?TWxoUk81d3c4bFRKU0xhdHZXK2dSK1NPYVIwaVJYRXNmUVlLdDdyaVhmSHYz?=
 =?utf-8?B?RC9ybm93N01UTjQ2Rld3SnVRQkszQzhBTi9EVnRMUmp6ZlIrUTByaU9WdlFp?=
 =?utf-8?B?KzN0aU5ZcTdJVFZzTXJFWS94RzZ0ZFdkcTVmdC9qT3RjRERNVFZNUXIrSjZo?=
 =?utf-8?B?bk9XM21uMWVVWkNsbS85OHh0VDFJb0R1aWpkbGs1U0VIb1cxcDR2cHpHU0FX?=
 =?utf-8?B?TFZnR2Q1TlArZkdtQm1ZRUxYSTFzeVA3SUljeWhwODJTdmkxZzZWb2hGeG11?=
 =?utf-8?B?NTAwM25IYm5EM3B5T2tGby95NWZhckhkMmVQS0s1T095ek1IOGhTMktESUdN?=
 =?utf-8?B?dkk1bDBNc0RRK0g2NnE3NGlaTUpmREFLZ0x6cUlkaTZaM1d2VnpuNGZMcWU3?=
 =?utf-8?B?WS94OHpYZFVXdGhER0ZNWEVGbzltTEthdU1qYk12R29WenAxV0dLMFUrT0Vv?=
 =?utf-8?B?cXpabm0xcUlyRFFQRmJoQjFFOUVDSndxWEZmTkNaSkM3TWtNdHVBakdYS0Vo?=
 =?utf-8?B?dDV1RlE4aEQvbjRIRStDZFdDcklxMUN2OUUyRkIrdzl5dUNFTFcyemZRazdF?=
 =?utf-8?B?OXpGcjlaeGtRREwrVnlOZGFDWUUrMkpncURQZWtieUlmU0R5N3YySXliSDNU?=
 =?utf-8?B?Y0RBa2lBRlFUNStCZGFCbEZyVUdBNkRwdXdSdWZUV01UU2ZEZTRCc3dnSEVB?=
 =?utf-8?B?cyt6UFJhVWNSaStGQzR4OEliVFpYcGtwYkFmS1lCZFZ3d1pwWjh1cWcrR2sy?=
 =?utf-8?B?Rmt6MG9kZDNDcWtrS21QVXE1N3ArL1dHb1ZkMklaUG11anpFK0VXUHNHbFlI?=
 =?utf-8?B?dXhCUmR3U1NmUkJQWEVPWHAwMEs2VjBOL0M5ajZJSEZOQ2pjSzNuS0ZIMVly?=
 =?utf-8?B?V1hBSmlzV283TlhPbWZkdWdjUzUzSm5BN3hCVm15SkdSSUhoN1MrNG1nT2lJ?=
 =?utf-8?B?ZDF0dyt5M252N1NNLzBSZHcrdkxJeDQ5NExGS1pZdS9LamhBUE9PQzhHZm1P?=
 =?utf-8?B?Nk5MaCtjUDBXazlwc1ZINnlyZnhjWklNSkpGd1NKNzVRK21zYjgybkx3VVhC?=
 =?utf-8?B?b3g1RUh3dFhZc0FMOERweTlmWWc4NEt2UDc3S2preXVReUZQdHhrQzJSdWYx?=
 =?utf-8?B?akg3NkYrbFFiVzRpTWo5WWlteEFTUmNSOCtTQ2diU3BZSjVWRFRjSHNPa05B?=
 =?utf-8?Q?G/V4tJC8x+hkugvDZ9/803kwa?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b479abd2-b987-4b02-ebc3-08dc274207da
X-MS-Exchange-CrossTenant-AuthSource: BL1PR12MB5874.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2024 18:32:53.4093
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Vk+CL7rEz1rDHyr5NFQDOC2yjbkb4cLPXSQsqZxPYU56WPxcmajZH5zhcKXXV/sg
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6809


On 2/6/24 11:39 AM, Mathieu Poirier wrote:
> On Wed, Jan 31, 2024 at 04:48:12PM -0800, Tanmay Shah wrote:
> > In lockstep mode, r5 core0 uses TCM of R5 core1. Following is lockstep
> > mode memory region as per hardware reference manual.
> > 
> >     |      *TCM*         |   *R5 View* | *Linux view* |
> >     | R5_0 ATCM (128 KB) | 0x0000_0000 | 0xFFE0_0000  |
> >     | R5_0 BTCM (128 KB) | 0x0002_0000 | 0xFFE2_0000  |
> > 
> > However, driver shouldn't model it as above because R5 core0 TCM and core1
> > TCM has different power-domains mapped to it.
> > Hence, TCM address space in lockstep mode should be modeled as 64KB
> > regions only where each region has its own power-domain as following:
> > 
> >     |      *TCM*         |   *R5 View* | *Linux view* |
> >     | R5_0 ATCM0 (64 KB) | 0x0000_0000 | 0xFFE0_0000  |
> >     | R5_0 BTCM0 (64 KB) | 0x0002_0000 | 0xFFE2_0000  |
> >     | R5_0 ATCM1 (64 KB) | 0x0001_0000 | 0xFFE1_0000  |
> >     | R5_0 BTCM1 (64 KB) | 0x0003_0000 | 0xFFE3_0000  |
> > 
> > This fix makes driver maintanance easy and makes design robust for future
> > platorms as well.
> > 
> > Fixes: 9af45bbdcbbb ("remoteproc: zynqmp: fix TCM carveouts in lockstep mode")
>
> This patch doesn't fix a bug, i.e users won't see any changes once this patch
> gets applied.  As such there is no need for a "Fixes" tag.  When you resend this
> patch (see below), please remove the line.

Ack.


> > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > ---
> >  drivers/remoteproc/xlnx_r5_remoteproc.c | 145 ++----------------------
> >  1 file changed, 12 insertions(+), 133 deletions(-)
> >
>
> I am happy with this patch but won't apply it because I want to see what comes
> next.  Please include it in your next patchet.

Thanks. Sure, I will include this with zynqmp tcm bindings patch series.


> Reviewed-by: Mathieu Poirier <mathieu.poirier@linaro.org>
>
> > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > index 4395edea9a64..42b0384d34f2 100644
> > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > @@ -84,12 +84,12 @@ static const struct mem_bank_data zynqmp_tcm_banks_split[] = {
> >  	{0xffeb0000UL, 0x20000, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
> >  };
> >  
> > -/* In lockstep mode cluster combines each 64KB TCM and makes 128KB TCM */
> > +/* In lockstep mode cluster uses each 64KB TCM from second core as well */
> >  static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > -	{0xffe00000UL, 0x0, 0x20000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 128KB each */
> > -	{0xffe20000UL, 0x20000, 0x20000UL, PD_R5_0_BTCM, "btcm0"},
> > -	{0, 0, 0, PD_R5_1_ATCM, ""},
> > -	{0, 0, 0, PD_R5_1_BTCM, ""},
> > +	{0xffe00000UL, 0x0, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> > +	{0xffe20000UL, 0x20000, 0x10000UL, PD_R5_0_BTCM, "btcm0"},
> > +	{0xffe10000UL, 0x10000, 0x10000UL, PD_R5_1_ATCM, "atcm1"},
> > +	{0xffe30000UL, 0x30000, 0x10000UL, PD_R5_1_BTCM, "btcm1"},
> >  };
> >  
> >  /**
> > @@ -540,14 +540,14 @@ static int tcm_mem_map(struct rproc *rproc,
> >  }
> >  
> >  /*
> > - * add_tcm_carveout_split_mode()
> > + * add_tcm_banks()
> >   * @rproc: single R5 core's corresponding rproc instance
> >   *
> > - * allocate and add remoteproc carveout for TCM memory in split mode
> > + * allocate and add remoteproc carveout for TCM memory
> >   *
> >   * return 0 on success, otherwise non-zero value on failure
> >   */
> > -static int add_tcm_carveout_split_mode(struct rproc *rproc)
> > +static int add_tcm_banks(struct rproc *rproc)
> >  {
> >  	struct rproc_mem_entry *rproc_mem;
> >  	struct zynqmp_r5_core *r5_core;
> > @@ -580,10 +580,10 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> >  					     ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> >  		if (ret < 0) {
> >  			dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > -			goto release_tcm_split;
> > +			goto release_tcm;
> >  		}
> >  
> > -		dev_dbg(dev, "TCM carveout split mode %s addr=%llx, da=0x%x, size=0x%lx",
> > +		dev_dbg(dev, "TCM carveout %s addr=%llx, da=0x%x, size=0x%lx",
> >  			bank_name, bank_addr, da, bank_size);
> >  
> >  		rproc_mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> > @@ -593,7 +593,7 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> >  		if (!rproc_mem) {
> >  			ret = -ENOMEM;
> >  			zynqmp_pm_release_node(pm_domain_id);
> > -			goto release_tcm_split;
> > +			goto release_tcm;
> >  		}
> >  
> >  		rproc_add_carveout(rproc, rproc_mem);
> > @@ -601,7 +601,7 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> >  
> >  	return 0;
> >  
> > -release_tcm_split:
> > +release_tcm:
> >  	/* If failed, Turn off all TCM banks turned on before */
> >  	for (i--; i >= 0; i--) {
> >  		pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > @@ -610,127 +610,6 @@ static int add_tcm_carveout_split_mode(struct rproc *rproc)
> >  	return ret;
> >  }
> >  
> > -/*
> > - * add_tcm_carveout_lockstep_mode()
> > - * @rproc: single R5 core's corresponding rproc instance
> > - *
> > - * allocate and add remoteproc carveout for TCM memory in lockstep mode
> > - *
> > - * return 0 on success, otherwise non-zero value on failure
> > - */
> > -static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > -{
> > -	struct rproc_mem_entry *rproc_mem;
> > -	struct zynqmp_r5_core *r5_core;
> > -	int i, num_banks, ret;
> > -	phys_addr_t bank_addr;
> > -	size_t bank_size = 0;
> > -	struct device *dev;
> > -	u32 pm_domain_id;
> > -	char *bank_name;
> > -	u32 da;
> > -
> > -	r5_core = rproc->priv;
> > -	dev = r5_core->dev;
> > -
> > -	/* Go through zynqmp banks for r5 node */
> > -	num_banks = r5_core->tcm_bank_count;
> > -
> > -	/*
> > -	 * In lockstep mode, TCM is contiguous memory block
> > -	 * However, each TCM block still needs to be enabled individually.
> > -	 * So, Enable each TCM block individually.
> > -	 * Although ATCM and BTCM is contiguous memory block, add two separate
> > -	 * carveouts for both.
> > -	 */
> > -	for (i = 0; i < num_banks; i++) {
> > -		pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > -
> > -		/* Turn on each TCM bank individually */
> > -		ret = zynqmp_pm_request_node(pm_domain_id,
> > -					     ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > -					     ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > -		if (ret < 0) {
> > -			dev_err(dev, "failed to turn on TCM 0x%x", pm_domain_id);
> > -			goto release_tcm_lockstep;
> > -		}
> > -
> > -		bank_size = r5_core->tcm_banks[i]->size;
> > -		if (bank_size == 0)
> > -			continue;
> > -
> > -		bank_addr = r5_core->tcm_banks[i]->addr;
> > -		da = r5_core->tcm_banks[i]->da;
> > -		bank_name = r5_core->tcm_banks[i]->bank_name;
> > -
> > -		/* Register TCM address range, TCM map and unmap functions */
> > -		rproc_mem = rproc_mem_entry_init(dev, NULL, bank_addr,
> > -						 bank_size, da,
> > -						 tcm_mem_map, tcm_mem_unmap,
> > -						 bank_name);
> > -		if (!rproc_mem) {
> > -			ret = -ENOMEM;
> > -			zynqmp_pm_release_node(pm_domain_id);
> > -			goto release_tcm_lockstep;
> > -		}
> > -
> > -		/* If registration is success, add carveouts */
> > -		rproc_add_carveout(rproc, rproc_mem);
> > -
> > -		dev_dbg(dev, "TCM carveout lockstep mode %s addr=0x%llx, da=0x%x, size=0x%lx",
> > -			bank_name, bank_addr, da, bank_size);
> > -	}
> > -
> > -	return 0;
> > -
> > -release_tcm_lockstep:
> > -	/* If failed, Turn off all TCM banks turned on before */
> > -	for (i--; i >= 0; i--) {
> > -		pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > -		zynqmp_pm_release_node(pm_domain_id);
> > -	}
> > -	return ret;
> > -}
> > -
> > -/*
> > - * add_tcm_banks()
> > - * @rproc: single R5 core's corresponding rproc instance
> > - *
> > - * allocate and add remoteproc carveouts for TCM memory based on cluster mode
> > - *
> > - * return 0 on success, otherwise non-zero value on failure
> > - */
> > -static int add_tcm_banks(struct rproc *rproc)
> > -{
> > -	struct zynqmp_r5_cluster *cluster;
> > -	struct zynqmp_r5_core *r5_core;
> > -	struct device *dev;
> > -
> > -	r5_core = rproc->priv;
> > -	if (!r5_core)
> > -		return -EINVAL;
> > -
> > -	dev = r5_core->dev;
> > -
> > -	cluster = dev_get_drvdata(dev->parent);
> > -	if (!cluster) {
> > -		dev_err(dev->parent, "Invalid driver data\n");
> > -		return -EINVAL;
> > -	}
> > -
> > -	/*
> > -	 * In lockstep mode TCM banks are one contiguous memory region of 256Kb
> > -	 * In split mode, each TCM bank is 64Kb and not contiguous.
> > -	 * We add memory carveouts accordingly.
> > -	 */
> > -	if (cluster->mode == SPLIT_MODE)
> > -		return add_tcm_carveout_split_mode(rproc);
> > -	else if (cluster->mode == LOCKSTEP_MODE)
> > -		return add_tcm_carveout_lockstep_mode(rproc);
> > -
> > -	return -EINVAL;
> > -}
> > -
> >  /*
> >   * zynqmp_r5_parse_fw()
> >   * @rproc: single R5 core's corresponding rproc instance
> > 
> > base-commit: 99f59b148871dadb9104366e3d25b120a97f897b
> > -- 
> > 2.25.1
> >