Received-SPF: pass (google.com: domain of linux-kernel+bounces-195970-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <462c0b21-c800-4b9b-bd79-72ecd135ad0f@amd.com>
Date: Thu, 30 May 2024 15:34:02 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH net-next] ionic: advertise 52-bit addressing limitation
 for MSI-X
To: David Christensen <drc@linux.ibm.com>,
 Brett Creeley <brett.creeley@amd.com>,
 "supporter:PENSANDO ETHERNET DRIVERS" <drivers@pensando.io>,
 "David S. Miller" <davem@davemloft.net>, Eric Dumazet <edumazet@google.com>,
 Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
 "open list:PENSANDO ETHERNET DRIVERS" <netdev@vger.kernel.org>,
 open list <linux-kernel@vger.kernel.org>
References: <20240530214026.774256-1-drc@linux.ibm.com>
Content-Language: en-US
From: "Nelson, Shannon" <shannon.nelson@amd.com>
In-Reply-To: <20240530214026.774256-1-drc@linux.ibm.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?VUdTdklYMldRNXRmckJwdUVpOGQ2OEFORENXdEt6MXpqSVh4cXNWckwwZ3FG?=
 =?utf-8?B?Q3U0L08va2hYWVFoQlRJZVljaWRXcDVUeU9rVFJYZzJWamZONWk0K1o2Q3li?=
 =?utf-8?B?SW9PTzBmZDBtMlcvYlhzSTlGcmxQeEhxdTdHa0NxN1RHeVAxY2VvM3pQU2Yz?=
 =?utf-8?B?ZGk4Y1phTjNsNU9VcSszcjBHekRtUnl2Q21qTEF2N1J6aW0vM3N1Sy9uVlN6?=
 =?utf-8?B?cm1tZGRSQzhpRXRzOUFPRWpwQU5uOWNjejlLVlIyUkRGcXgwa1FOc1NodURj?=
 =?utf-8?B?THkrYlExdHVjbVFqOTJPY3hlV2haYTQ0aWo4MndZU3R1NUVrSDNQMm9wQ1N1?=
 =?utf-8?B?c2Q4MDl4dlJkbzNWU0V0TlZ5WG9kMVh0am92SXlDOHdYdUxXUXFZYno3Vm0v?=
 =?utf-8?B?bzRxdlQyeFMrUUFzaEg2OVdZcWpRYnhzODdBN0UrbnlLTXBqeTBhdDN2UDlD?=
 =?utf-8?B?M0MrZ1gvRkRFTEpWVnRrWXNLUzNrNmFhYWQrQlhxUWpTekhESVgrRzJObGRl?=
 =?utf-8?B?VmpLdVNiWXVlREZWVHVDR3haMTJvZGtKSWhscmJIMGVuZmdKdzYzSkRlV1Jo?=
 =?utf-8?B?blRXcUFXWlJMeDdlVmVETUg1MDVpVVVuWGFQV2J6dDUxUk9qL3hlRndDT1ZR?=
 =?utf-8?B?Mk5RK01Ed1VZTVRtNHlxdDhvdkovbFo2MzR3UTBleTVsenpBcFBDUm1jSEVJ?=
 =?utf-8?B?cW54dnRUVEQydy82Wk5zTmRlRTlwSjNML2diajFrZm1vYS93MCthL1UxTXdI?=
 =?utf-8?B?RnZhS2FmRGVQS1ROZjFUYWQxdytBVkhvZ3FpTDJPNHNBalAzNmZpcU1YWm5y?=
 =?utf-8?B?UktkYzFpcHAwTkRGVS9QMlltTWxFdXFiU0wzbVFsK256T2k0cGUyZlRTbDZn?=
 =?utf-8?B?Wit0ZFZPTklVSzBReVJNaEw4Ymg3ekpMdGIxYkNWUVFOL2NMc0ZPZ0toMEY4?=
 =?utf-8?B?K2VmbkJxQmkwQ0hJZ3RDUVpyNmNLellIUzVmOGtOT2k2OURqSDF2Nkp3UDRZ?=
 =?utf-8?B?aUhlb2IrcGlpZzJ0NDI1Wmc3amNQUnBud2FCajBpcGNOenE1RUEyZ3BlTEp3?=
 =?utf-8?B?ampBRUZ1dlN4dmxGaVJRRzRmSllrTlBnOWhoYjFUUXpQOUZpODczMHBpRkFi?=
 =?utf-8?B?TE5FQkg3V3ZsUGxNa2FCUHNVVFBPSkZMaTd6QVZNY0NZTHZYMk4vdm5RM0Rq?=
 =?utf-8?B?a21QcndydXRYTEl3eEpFamN3Mk1sSnpWckJKV2xGc05HNWRqdFArNERycFc2?=
 =?utf-8?B?ZmVwUlVoZkxxN0xLM1pXVXJzMmZST1lYUXBldDY1VSsvaFJscnBUTDhLTjV2?=
 =?utf-8?B?QkhEZ3V1WmR3YnlYRnhFRVJzVm1pQ2ZUWnF0akVaUlFaL1F3MTQ1aHVsYVpn?=
 =?utf-8?B?a0VUSURNRHFaK1loRTgwOExvRWpmczBWSEhZcVoxOTlxY2duY29peTZjRytv?=
 =?utf-8?B?TWdqZnY4alZJSjV4SFQzRi92Z2FiN0diZmFIRmtqWjZQVEc4UldnV0JxOVcy?=
 =?utf-8?B?U1Z6MnVnRk9TZkJHWXV0M3J2Z3NQVlNhQ0UyeUFsMEVOSzJjQmRnTXBQSVBJ?=
 =?utf-8?B?cktINk9UTGRXdGR4R0lOV2JDUXBYdkdLTitkZkZ1WHhpNU5oVDJRdHJaUFRT?=
 =?utf-8?B?RmRucURNSkR5NDhmemFqRnRETDJJVWdBZzlXZDViVmxaSEhmelR6bEZrcFZH?=
 =?utf-8?B?NDI4cHhxbmJiaDVuT0V5SG13WGZaSC8xUVBvbHNGTHVJRUJ5eVdDNWhUUkt6?=
 =?utf-8?B?RTBWbzh5Wlc0RWFVMEtGTXVEcDBDbkluZFBLWi9EdXF6blUrUzk4OXpqR2FK?=
 =?utf-8?B?SmdKYWVtWmNXWWE5bXYzSE15cUhFR1VMNjFoR2lSSDVQNWZDcGpnd3RpdW1x?=
 =?utf-8?B?Ulp3RTZwRGl6NlBaVTFEdnplbVJUVFc4YWNnK0g3N1JqWmdPUHhGQStvZU42?=
 =?utf-8?B?b2ZBc3ZhckNqdUVxd3lqd2hqTjdmb0dUKzNVZHVwU2dmb3lzOWN4UjhVZ1lI?=
 =?utf-8?B?eERvRWlOcjdBR2MrbXAwR0VIdGVNMmw3MG00VDlMRWlpSHZ6YkRFUXpGOUJq?=
 =?utf-8?B?Qk44VHdLWlFFNHBKTFdMYjBkUWVOa1dNUW12bzBtRWtmL3o0NUFncHByOVNn?=
 =?utf-8?Q?LblxI8c6Uch0QceUkThl4ovhL?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8eb43aec-d9c8-4316-67bf-08dc80f89c1e
X-MS-Exchange-CrossTenant-AuthSource: DS0PR12MB6583.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2024 22:34:04.1265
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Q+4kRvlnPaUus+v6LTQDXi8DbsF2IVTTKWV0d+qVTZwRZgor9+WgcEOcuTGkLybLGp1/nhSihXAv82tiN4Sj1g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8405

On 5/30/2024 2:40 PM, David Christensen wrote:
> 
> Current ionic devices only support 52 internal physical address
> lines. This is sufficient for x86_64 systems which have similar
> limitations but does not apply to all other architectures,
> notably IBM POWER (ppc64). To ensure that MSI/MSI-X vectors are
> not set outside the physical address limits of the NIC, set the
> recently added no_64bit_msi value of the pci_dev structure
> during device probe.
> 
> Signed-off-by: David Christensen <drc@linux.ibm.com>
> ---
>   drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c b/drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c
> index 6ba8d4aca0a0..1e7f507f461f 100644
> --- a/drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c
> +++ b/drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c
> @@ -326,6 +326,10 @@ static int ionic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
>                  goto err_out;
>          }
> 
> +       /* Ensure MSI/MSI-X interrupts lie within addressable physical memory */
> +       if (IONIC_ADDR_LEN < 64)
> +               pdev->no_64bit_msi = 1;
> +

Thanks, David, for the reminder that we have something like this in our 
out-of-tree driver.  I'm far from being a DMA expert, but this seems 
limiting for those architectures that don't need it, and I would have 
thought that setting the dma mask would already be the way of telling 
the system what our limitation was so there wouldn't be this problem. 
(I'm sure someone will point out the error in my thinking...)

Perhaps to solve your problem with less limitation on others we could do 
something like:

#ifdef CONFIG_PPC64
        pdev->no_64bit_msi = 1;
#endif

Thanks,
sln

>          err = ionic_setup_one(ionic);
>          if (err)
>                  goto err_out;
> --
> 2.43.0
>