Involve an new and common mathod to send pme_turn_off() message. Previously
pme_turn_off() implement by platform related special register to trigge
it.
But Yoshihiro give good idea by using iATU to send out message. Previously
Yoshihiro provide patches to raise INTx message by dummy write to outbound
iATU.
Use similar mathod to send out pme_turn_off message.
Previous two patches is picked from Yoshihiro' big patch serialise.
PCI: dwc: Change arguments of dw_pcie_prog_outbound_atu()
PCI: Add INTx Mechanism Messages macros
PCI: Add PME_TURN_OFF message macro
dt-bindings: PCI: dwc: Add 'msg" register region, Add "msg" region to use
to map PCI msg.
PCI: dwc: Add common pme_turn_off message method
Using common pme_turn_off() message if platform have not define their.
Signed-off-by: Frank Li <[email protected]>
---
Changes in v6:
- Move PCIE_MSG_CODE_PME_TURN_OFF ahead before INTx to keep the codes
sorted.
- Added reason why not alloc_resesource in commit messag.
- Change title PCI: dwc: Add generic MSG TLP support for sending
PME_Turn_Off when system suspend.
- Rework commit message
"Instead of relying on the vendor specific implementations to send the
PME_Turn_Off message, let's introduce a generic way of sending the message using
the MSG TLP.
This is achieved by reserving a region for MSG TLP of size 'pci->region_align',
at the end of the first IORESOURCE_MEM window of the host bridge. And then
sending the PME_Turn_Off message during system suspend with the help of iATU.
It should be noted that this generic implementation is optional for the glue
drivers and can be overridden by a custom 'pme_turn_off' callback.
- using devm_request_resource()
- using mani's suggest comments
/*
* Allocate the resource for MSG TLP before programming the iATU
* outbound window in dw_pcie_setup_rc(). Since the allocation depends
* on the value of 'region_align', this has to be done after
* dw_pcie_iatu_detect().
*/
- Add comments, about use_atu_msg, need glue driver set it.
- m -> mem in dw_pcie_pme_turn_off()
- remove redundant check() if (!atu.size) in dw_pcie_pme_turn_off().
- The reason can't adjust windows->end
This windows already reported to pci system before this function. It is
not good to modify window-res-end. It just add child resource like below.
windows is root resource, which will create may child when call
request_resource.
bridge -> windows
child1 -> msg
child2 -> pci ep1
child3 -> pci_ep2.
...
Although you see whole bridge window, 'msg' already used and put under root
resource, new pci devices will never use 'msg' resource.
If change windows->res->end here, I worry about it may broken resource
tree.
Detail below thread:
- Link to v5: https://lore.kernel.org/r/[email protected]
Changes in v5:
- Default disable allocate TLP message memory windows. If driver need use
this feature, need set use_atu_msg = true before call dw_host_init().
- Link to v4: https://lore.kernel.org/r/[email protected]
Changes in v4:
- Remove dt-binding patch. Needn't change any dts file and binding doc.
Reserve a region at end of first IORESOURCE_MEM window by call
request_resource(). So PCIe stack will not use this reserve region to any
PCIe devices.
I tested it by reserve at begin of IORESOURCE_MEM window. PCIe stack
will skip it as expection.
Fixed a issue, forget set iATU index when sent PME_turn_off.
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- fix 'MSG"
- Add pcie spec ref in head file
- using function name dw_pci_pme_turn_off()
- Using PCIE_ prefix macro
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Add my sign off at PCI: dwc: Add outbound MSG TLPs support
- Add Bjorn review tag at Add INTx Mechanism Messages macros
- using PME_Turn_Off match PCIe spec
- ref to pcie spec v6.1
- using section number.
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Frank Li (2):
PCI: Add PCIE_MSG_CODE_PME_TURN_OFF message macro
PCI: dwc: Add generic MSG TLP support for sending PME_Turn_Off when system suspend
Yoshihiro Shimoda (3):
PCI: Add INTx Mechanism Messages macros
PCI: dwc: Consolidate args of dw_pcie_prog_outbound_atu() into a structure
PCI: dwc: Add outbound MSG TLPs support
drivers/pci/controller/dwc/pcie-designware-ep.c | 21 ++--
drivers/pci/controller/dwc/pcie-designware-host.c | 146 +++++++++++++++++++---
drivers/pci/controller/dwc/pcie-designware.c | 54 ++++----
drivers/pci/controller/dwc/pcie-designware.h | 22 +++-
drivers/pci/pci.h | 20 +++
5 files changed, 199 insertions(+), 64 deletions(-)
---
base-commit: ef6a734484e939cf67bc41c45bc936a78ea210f0
change-id: 20240130-pme_msg-dd2d81ee9886
Best regards,
---
Frank Li <[email protected]>
From: Yoshihiro Shimoda <[email protected]>
Add "Message Routing" and "INTx Mechanism Messages" macros to enable
a PCIe driver to send messages for INTx Interrupt Signaling.
The "Message Routing" is in the section 2.2.8, and the "INTx Mechanism
Messages" is in the section 2.2.8.1 on the PCI Express Base Specification,
Rev 6.1.
Signed-off-by: Yoshihiro Shimoda <[email protected]>
Reviewed-by: Serge Semin <[email protected]>
Reviewed-by: Manivannan Sadhasivam <[email protected]>
Acked-by: Bjorn Helgaas <[email protected]>
Signed-off-by: Frank Li <[email protected]>
---
drivers/pci/pci.h | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 17fed18468474..19b4227a8a7e8 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -22,6 +22,24 @@
*/
#define PCIE_PME_TO_L2_TIMEOUT_US 10000
+/* Message Routing (r[2:0]) See: PCIe r6.0, sec 2.2.8 */
+#define PCIE_MSG_TYPE_R_RC 0
+#define PCIE_MSG_TYPE_R_ADDR 1
+#define PCIE_MSG_TYPE_R_ID 2
+#define PCIE_MSG_TYPE_R_BC 3
+#define PCIE_MSG_TYPE_R_LOCAL 4
+#define PCIE_MSG_TYPE_R_GATHER 5
+
+/* INTx Mechanism Messages See: PCIe r6.0, sec 2.2.8.1 */
+#define PCIE_MSG_CODE_ASSERT_INTA 0x20
+#define PCIE_MSG_CODE_ASSERT_INTB 0x21
+#define PCIE_MSG_CODE_ASSERT_INTC 0x22
+#define PCIE_MSG_CODE_ASSERT_INTD 0x23
+#define PCIE_MSG_CODE_DEASSERT_INTA 0x24
+#define PCIE_MSG_CODE_DEASSERT_INTB 0x25
+#define PCIE_MSG_CODE_DEASSERT_INTC 0x26
+#define PCIE_MSG_CODE_DEASSERT_INTD 0x27
+
extern const unsigned char pcie_link_speed[];
extern bool pci_early_dump;
--
2.34.1
From: Yoshihiro Shimoda <[email protected]>
This is a preparation before adding the Msg-type outbound iATU
mapping. The respective update will require two more arguments added
to __dw_pcie_prog_outbound_atu(). That will make the already
complicated function prototype even more hard to comprehend accepting
_eight_ arguments. In order to prevent that and keep the code
more-or-less readable all the outbound iATU-related arguments are
moved to the new config-structure: struct dw_pcie_ob_atu_cfg pointer
to which shall be passed to dw_pcie_prog_outbound_atu(). The structure
is supposed to be locally defined and populated with the outbound iATU
settings implied by the caller context.
As a result of the denoted change there is no longer need in having
the two distinctive methods for the Host and End-point outbound iATU
setups since the corresponding code can directly call the
dw_pcie_prog_outbound_atu() method with the config-structure
populated. Thus dw_pcie_prog_ep_outbound_atu() is dropped.
Signed-off-by: Yoshihiro Shimoda <[email protected]>
Reviewed-by: Serge Semin <[email protected]>
Reviewed-by: Manivannan Sadhasivam <[email protected]>
Signed-off-by: Frank Li <[email protected]>
---
drivers/pci/controller/dwc/pcie-designware-ep.c | 21 +++++----
drivers/pci/controller/dwc/pcie-designware-host.c | 52 ++++++++++++++++-------
drivers/pci/controller/dwc/pcie-designware.c | 49 ++++++++-------------
drivers/pci/controller/dwc/pcie-designware.h | 15 +++++--
4 files changed, 77 insertions(+), 60 deletions(-)
diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index 746a11dcb67f1..9dd6a2edc7ae5 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -160,9 +160,8 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, int type,
return 0;
}
-static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, u8 func_no,
- phys_addr_t phys_addr,
- u64 pci_addr, size_t size)
+static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep,
+ struct dw_pcie_ob_atu_cfg *atu)
{
struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
u32 free_win;
@@ -174,13 +173,13 @@ static int dw_pcie_ep_outbound_atu(struct dw_pcie_ep *ep, u8 func_no,
return -EINVAL;
}
- ret = dw_pcie_prog_ep_outbound_atu(pci, func_no, free_win, PCIE_ATU_TYPE_MEM,
- phys_addr, pci_addr, size);
+ atu->index = free_win;
+ ret = dw_pcie_prog_outbound_atu(pci, atu);
if (ret)
return ret;
set_bit(free_win, ep->ob_window_map);
- ep->outbound_addr[free_win] = phys_addr;
+ ep->outbound_addr[free_win] = atu->cpu_addr;
return 0;
}
@@ -280,8 +279,14 @@ static int dw_pcie_ep_map_addr(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
int ret;
struct dw_pcie_ep *ep = epc_get_drvdata(epc);
struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
-
- ret = dw_pcie_ep_outbound_atu(ep, func_no, addr, pci_addr, size);
+ struct dw_pcie_ob_atu_cfg atu = { 0 };
+
+ atu.func_no = func_no;
+ atu.type = PCIE_ATU_TYPE_MEM;
+ atu.cpu_addr = addr;
+ atu.pci_addr = pci_addr;
+ atu.size = size;
+ ret = dw_pcie_ep_outbound_atu(ep, &atu);
if (ret) {
dev_err(pci->dev, "Failed to enable address\n");
return ret;
diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index d15a5c2d5b480..3a9cb4be22ab2 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -554,6 +554,7 @@ static void __iomem *dw_pcie_other_conf_map_bus(struct pci_bus *bus,
{
struct dw_pcie_rp *pp = bus->sysdata;
struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+ struct dw_pcie_ob_atu_cfg atu = { 0 };
int type, ret;
u32 busdev;
@@ -576,8 +577,12 @@ static void __iomem *dw_pcie_other_conf_map_bus(struct pci_bus *bus,
else
type = PCIE_ATU_TYPE_CFG1;
- ret = dw_pcie_prog_outbound_atu(pci, 0, type, pp->cfg0_base, busdev,
- pp->cfg0_size);
+ atu.type = type;
+ atu.cpu_addr = pp->cfg0_base;
+ atu.pci_addr = busdev;
+ atu.size = pp->cfg0_size;
+
+ ret = dw_pcie_prog_outbound_atu(pci, &atu);
if (ret)
return NULL;
@@ -589,6 +594,7 @@ static int dw_pcie_rd_other_conf(struct pci_bus *bus, unsigned int devfn,
{
struct dw_pcie_rp *pp = bus->sysdata;
struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+ struct dw_pcie_ob_atu_cfg atu = { 0 };
int ret;
ret = pci_generic_config_read(bus, devfn, where, size, val);
@@ -596,9 +602,12 @@ static int dw_pcie_rd_other_conf(struct pci_bus *bus, unsigned int devfn,
return ret;
if (pp->cfg0_io_shared) {
- ret = dw_pcie_prog_outbound_atu(pci, 0, PCIE_ATU_TYPE_IO,
- pp->io_base, pp->io_bus_addr,
- pp->io_size);
+ atu.type = PCIE_ATU_TYPE_IO;
+ atu.cpu_addr = pp->io_base;
+ atu.pci_addr = pp->io_bus_addr;
+ atu.size = pp->io_size;
+
+ ret = dw_pcie_prog_outbound_atu(pci, &atu);
if (ret)
return PCIBIOS_SET_FAILED;
}
@@ -611,6 +620,7 @@ static int dw_pcie_wr_other_conf(struct pci_bus *bus, unsigned int devfn,
{
struct dw_pcie_rp *pp = bus->sysdata;
struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+ struct dw_pcie_ob_atu_cfg atu = { 0 };
int ret;
ret = pci_generic_config_write(bus, devfn, where, size, val);
@@ -618,9 +628,12 @@ static int dw_pcie_wr_other_conf(struct pci_bus *bus, unsigned int devfn,
return ret;
if (pp->cfg0_io_shared) {
- ret = dw_pcie_prog_outbound_atu(pci, 0, PCIE_ATU_TYPE_IO,
- pp->io_base, pp->io_bus_addr,
- pp->io_size);
+ atu.type = PCIE_ATU_TYPE_IO;
+ atu.cpu_addr = pp->io_base;
+ atu.pci_addr = pp->io_bus_addr;
+ atu.size = pp->io_size;
+
+ ret = dw_pcie_prog_outbound_atu(pci, &atu);
if (ret)
return PCIBIOS_SET_FAILED;
}
@@ -655,6 +668,7 @@ static struct pci_ops dw_pcie_ops = {
static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
{
struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+ struct dw_pcie_ob_atu_cfg atu = { 0 };
struct resource_entry *entry;
int i, ret;
@@ -682,10 +696,13 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
if (pci->num_ob_windows <= ++i)
break;
- ret = dw_pcie_prog_outbound_atu(pci, i, PCIE_ATU_TYPE_MEM,
- entry->res->start,
- entry->res->start - entry->offset,
- resource_size(entry->res));
+ atu.index = i;
+ atu.type = PCIE_ATU_TYPE_MEM;
+ atu.cpu_addr = entry->res->start;
+ atu.pci_addr = entry->res->start - entry->offset;
+ atu.size = resource_size(entry->res);
+
+ ret = dw_pcie_prog_outbound_atu(pci, &atu);
if (ret) {
dev_err(pci->dev, "Failed to set MEM range %pr\n",
entry->res);
@@ -695,10 +712,13 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
if (pp->io_size) {
if (pci->num_ob_windows > ++i) {
- ret = dw_pcie_prog_outbound_atu(pci, i, PCIE_ATU_TYPE_IO,
- pp->io_base,
- pp->io_bus_addr,
- pp->io_size);
+ atu.index = i;
+ atu.type = PCIE_ATU_TYPE_IO;
+ atu.cpu_addr = pp->io_base;
+ atu.pci_addr = pp->io_bus_addr;
+ atu.size = pp->io_size;
+
+ ret = dw_pcie_prog_outbound_atu(pci, &atu);
if (ret) {
dev_err(pci->dev, "Failed to set IO range %pr\n",
entry->res);
diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
index 250cf7f40b858..df2575ec5f44c 100644
--- a/drivers/pci/controller/dwc/pcie-designware.c
+++ b/drivers/pci/controller/dwc/pcie-designware.c
@@ -465,56 +465,56 @@ static inline u32 dw_pcie_enable_ecrc(u32 val)
return val | PCIE_ATU_TD;
}
-static int __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8 func_no,
- int index, int type, u64 cpu_addr,
- u64 pci_addr, u64 size)
+int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
+ const struct dw_pcie_ob_atu_cfg *atu)
{
+ u64 cpu_addr = atu->cpu_addr;
u32 retries, val;
u64 limit_addr;
if (pci->ops && pci->ops->cpu_addr_fixup)
cpu_addr = pci->ops->cpu_addr_fixup(pci, cpu_addr);
- limit_addr = cpu_addr + size - 1;
+ limit_addr = cpu_addr + atu->size - 1;
if ((limit_addr & ~pci->region_limit) != (cpu_addr & ~pci->region_limit) ||
!IS_ALIGNED(cpu_addr, pci->region_align) ||
- !IS_ALIGNED(pci_addr, pci->region_align) || !size) {
+ !IS_ALIGNED(atu->pci_addr, pci->region_align) || !atu->size) {
return -EINVAL;
}
- dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_LOWER_BASE,
+ dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_LOWER_BASE,
lower_32_bits(cpu_addr));
- dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_UPPER_BASE,
+ dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_BASE,
upper_32_bits(cpu_addr));
- dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_LIMIT,
+ dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_LIMIT,
lower_32_bits(limit_addr));
if (dw_pcie_ver_is_ge(pci, 460A))
- dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_UPPER_LIMIT,
+ dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_LIMIT,
upper_32_bits(limit_addr));
- dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_LOWER_TARGET,
- lower_32_bits(pci_addr));
- dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_UPPER_TARGET,
- upper_32_bits(pci_addr));
+ dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_LOWER_TARGET,
+ lower_32_bits(atu->pci_addr));
+ dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_TARGET,
+ upper_32_bits(atu->pci_addr));
- val = type | PCIE_ATU_FUNC_NUM(func_no);
+ val = atu->type | PCIE_ATU_FUNC_NUM(atu->func_no);
if (upper_32_bits(limit_addr) > upper_32_bits(cpu_addr) &&
dw_pcie_ver_is_ge(pci, 460A))
val |= PCIE_ATU_INCREASE_REGION_SIZE;
if (dw_pcie_ver_is(pci, 490A))
val = dw_pcie_enable_ecrc(val);
- dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_REGION_CTRL1, val);
+ dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL1, val);
- dw_pcie_writel_atu_ob(pci, index, PCIE_ATU_REGION_CTRL2, PCIE_ATU_ENABLE);
+ dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2, PCIE_ATU_ENABLE);
/*
* Make sure ATU enable takes effect before any subsequent config
* and I/O accesses.
*/
for (retries = 0; retries < LINK_WAIT_MAX_IATU_RETRIES; retries++) {
- val = dw_pcie_readl_atu_ob(pci, index, PCIE_ATU_REGION_CTRL2);
+ val = dw_pcie_readl_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2);
if (val & PCIE_ATU_ENABLE)
return 0;
@@ -526,21 +526,6 @@ static int __dw_pcie_prog_outbound_atu(struct dw_pcie *pci, u8 func_no,
return -ETIMEDOUT;
}
-int dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
- u64 cpu_addr, u64 pci_addr, u64 size)
-{
- return __dw_pcie_prog_outbound_atu(pci, 0, index, type,
- cpu_addr, pci_addr, size);
-}
-
-int dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int index,
- int type, u64 cpu_addr, u64 pci_addr,
- u64 size)
-{
- return __dw_pcie_prog_outbound_atu(pci, func_no, index, type,
- cpu_addr, pci_addr, size);
-}
-
static inline u32 dw_pcie_readl_atu_ib(struct dw_pcie *pci, u32 index, u32 reg)
{
return dw_pcie_readl_atu(pci, PCIE_ATU_REGION_DIR_IB, index, reg);
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index 26dae48374627..d21db82e586d5 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -299,6 +299,15 @@ enum dw_pcie_ltssm {
DW_PCIE_LTSSM_UNKNOWN = 0xFFFFFFFF,
};
+struct dw_pcie_ob_atu_cfg {
+ int index;
+ int type;
+ u8 func_no;
+ u64 cpu_addr;
+ u64 pci_addr;
+ u64 size;
+};
+
struct dw_pcie_host_ops {
int (*init)(struct dw_pcie_rp *pp);
void (*deinit)(struct dw_pcie_rp *pp);
@@ -434,10 +443,8 @@ void dw_pcie_write_dbi2(struct dw_pcie *pci, u32 reg, size_t size, u32 val);
int dw_pcie_link_up(struct dw_pcie *pci);
void dw_pcie_upconfig_setup(struct dw_pcie *pci);
int dw_pcie_wait_for_link(struct dw_pcie *pci);
-int dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type,
- u64 cpu_addr, u64 pci_addr, u64 size);
-int dw_pcie_prog_ep_outbound_atu(struct dw_pcie *pci, u8 func_no, int index,
- int type, u64 cpu_addr, u64 pci_addr, u64 size);
+int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
+ const struct dw_pcie_ob_atu_cfg *atu);
int dw_pcie_prog_inbound_atu(struct dw_pcie *pci, int index, int type,
u64 cpu_addr, u64 pci_addr, u64 size);
int dw_pcie_prog_ep_inbound_atu(struct dw_pcie *pci, u8 func_no, int index,
--
2.34.1
From: Yoshihiro Shimoda <[email protected]>
Add "code" and "routing" into struct dw_pcie_ob_atu_cfg for triggering
INTx IRQs by iATU in the PCIe endpoint mode in near the future.
PCIE_ATU_INHIBIT_PAYLOAD is set to issue TLP type of Msg instead of
MsgD. So, this implementation supports the data-less messages only
for now.
Signed-off-by: Yoshihiro Shimoda <[email protected]>
Reviewed-by: Serge Semin <[email protected]>
Reviewed-by: Manivannan Sadhasivam <[email protected]>
Signed-off-by: Frank Li <[email protected]>
---
drivers/pci/controller/dwc/pcie-designware.c | 9 +++++++--
drivers/pci/controller/dwc/pcie-designware.h | 4 ++++
2 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
index df2575ec5f44c..ba909fade9db1 100644
--- a/drivers/pci/controller/dwc/pcie-designware.c
+++ b/drivers/pci/controller/dwc/pcie-designware.c
@@ -499,7 +499,7 @@ int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_UPPER_TARGET,
upper_32_bits(atu->pci_addr));
- val = atu->type | PCIE_ATU_FUNC_NUM(atu->func_no);
+ val = atu->type | atu->routing | PCIE_ATU_FUNC_NUM(atu->func_no);
if (upper_32_bits(limit_addr) > upper_32_bits(cpu_addr) &&
dw_pcie_ver_is_ge(pci, 460A))
val |= PCIE_ATU_INCREASE_REGION_SIZE;
@@ -507,7 +507,12 @@ int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
val = dw_pcie_enable_ecrc(val);
dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL1, val);
- dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2, PCIE_ATU_ENABLE);
+ val = PCIE_ATU_ENABLE;
+ if (atu->type == PCIE_ATU_TYPE_MSG) {
+ /* The data-less messages only for now */
+ val |= PCIE_ATU_INHIBIT_PAYLOAD | atu->code;
+ }
+ dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL2, val);
/*
* Make sure ATU enable takes effect before any subsequent config
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index d21db82e586d5..703b50bc5e0f1 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -148,11 +148,13 @@
#define PCIE_ATU_TYPE_IO 0x2
#define PCIE_ATU_TYPE_CFG0 0x4
#define PCIE_ATU_TYPE_CFG1 0x5
+#define PCIE_ATU_TYPE_MSG 0x10
#define PCIE_ATU_TD BIT(8)
#define PCIE_ATU_FUNC_NUM(pf) ((pf) << 20)
#define PCIE_ATU_REGION_CTRL2 0x004
#define PCIE_ATU_ENABLE BIT(31)
#define PCIE_ATU_BAR_MODE_ENABLE BIT(30)
+#define PCIE_ATU_INHIBIT_PAYLOAD BIT(22)
#define PCIE_ATU_FUNC_NUM_MATCH_EN BIT(19)
#define PCIE_ATU_LOWER_BASE 0x008
#define PCIE_ATU_UPPER_BASE 0x00C
@@ -303,6 +305,8 @@ struct dw_pcie_ob_atu_cfg {
int index;
int type;
u8 func_no;
+ u8 code;
+ u8 routing;
u64 cpu_addr;
u64 pci_addr;
u64 size;
--
2.34.1
Add PCIE_MSG_CODE_PME_TURN_OFF macros to enable a PCIe host driver to send
PME_Turn_Off messages.
Signed-off-by: Frank Li <[email protected]>
---
drivers/pci/pci.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 19b4227a8a7e8..1f6d54a5a7cfc 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -30,6 +30,8 @@
#define PCIE_MSG_TYPE_R_LOCAL 4
#define PCIE_MSG_TYPE_R_GATHER 5
+#define PCIE_MSG_CODE_PME_TURN_OFF 0x19
+
/* INTx Mechanism Messages See: PCIe r6.0, sec 2.2.8.1 */
#define PCIE_MSG_CODE_ASSERT_INTA 0x20
#define PCIE_MSG_CODE_ASSERT_INTB 0x21
--
2.34.1
Instead of relying on the vendor specific implementations to send the
PME_Turn_Off message, let's introduce a generic way of sending the message
using the MSG TLP.
This is achieved by reserving a region for MSG TLP of size
'pci->region_align', at the end of the first IORESOURCE_MEM window of the
host bridge. And then sending the PME_Turn_Off message during system
suspend with the help of iATU.
The reserve space at end is a little bit better than alloc_resource()
because the below reasons.
- alloc_resource() will allocate space at begin of IORESOURCE_MEM window.
There will be a hole when first Endpoint Device (EP) try to alloc a bigger
space then 'region_align' size.
- Keep EP device's IORESOURCE_MEM space unchange with/without this patch.
It should be noted that this generic implementation is optional for the
glue drivers and can be overridden by a custom 'pme_turn_off' callback.
Signed-off-by: Frank Li <[email protected]>
---
drivers/pci/controller/dwc/pcie-designware-host.c | 94 ++++++++++++++++++++++-
drivers/pci/controller/dwc/pcie-designware.h | 3 +
2 files changed, 93 insertions(+), 4 deletions(-)
diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index 3a9cb4be22ab2..5001cdf220123 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -398,6 +398,31 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)
return 0;
}
+static void dw_pcie_host_request_msg_tlp_res(struct dw_pcie_rp *pp)
+{
+ struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+ struct resource_entry *win;
+ struct resource *res;
+
+ win = resource_list_first_type(&pp->bridge->windows, IORESOURCE_MEM);
+ if (win) {
+ res = devm_kzalloc(pci->dev, sizeof(*res), GFP_KERNEL);
+ if (!res)
+ return;
+
+ /* Reserve last region_align block as message TLP space */
+ res->start = win->res->end - pci->region_align + 1;
+ res->end = win->res->end;
+ res->name = "msg";
+ res->flags = win->res->flags | IORESOURCE_BUSY;
+
+ if (!devm_request_resource(pci->dev, win->res, res))
+ pp->msg_res = res;
+ else
+ devm_kfree(pci->dev, res);
+ }
+}
+
int dw_pcie_host_init(struct dw_pcie_rp *pp)
{
struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
@@ -484,6 +509,16 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
dw_pcie_iatu_detect(pci);
+ /*
+ * Allocate the resource for MSG TLP before programming the iATU outbound window in
+ * dw_pcie_setup_rc(). Since the allocation depends on the value of 'region_align', this has
+ * to be done after dw_pcie_iatu_detect().
+ *
+ * Glue driver need set use_atu_msg before dw_pcie_host_init() if want MSG TLP feature.
+ */
+ if (pp->use_atu_msg)
+ dw_pcie_host_request_msg_tlp_res(pp);
+
ret = dw_pcie_edma_detect(pci);
if (ret)
goto err_free_msi;
@@ -541,6 +576,11 @@ void dw_pcie_host_deinit(struct dw_pcie_rp *pp)
dw_pcie_edma_remove(pci);
+ if (pp->msg_res) {
+ release_resource(pp->msg_res);
+ devm_kfree(pci->dev, pp->msg_res);
+ }
+
if (pp->has_msi_ctrl)
dw_pcie_free_msi(pp);
@@ -702,6 +742,10 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
atu.pci_addr = entry->res->start - entry->offset;
atu.size = resource_size(entry->res);
+ /* MSG TLB window resource reserve at one of end of IORESOURCE_MEM resource */
+ if (pp->msg_res && pp->msg_res->parent == entry->res)
+ atu.size -= resource_size(pp->msg_res);
+
ret = dw_pcie_prog_outbound_atu(pci, &atu);
if (ret) {
dev_err(pci->dev, "Failed to set MEM range %pr\n",
@@ -733,6 +777,8 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
dev_warn(pci->dev, "Ranges exceed outbound iATU size (%d)\n",
pci->num_ob_windows);
+ pp->msg_atu_index = i;
+
i = 0;
resource_list_for_each_entry(entry, &pp->bridge->dma_ranges) {
if (resource_type(entry->res) != IORESOURCE_MEM)
@@ -838,11 +884,48 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
}
EXPORT_SYMBOL_GPL(dw_pcie_setup_rc);
+/* Using message outbound ATU to send out PME_Turn_Off message for dwc PCIe controller */
+static int dw_pcie_pme_turn_off(struct dw_pcie *pci)
+{
+ struct dw_pcie_ob_atu_cfg atu = { 0 };
+ void __iomem *mem;
+ int ret;
+
+ if (pci->num_ob_windows <= pci->pp.msg_atu_index)
+ return -EINVAL;
+
+ if (!pci->pp.msg_res)
+ return -EINVAL;
+
+ atu.code = PCIE_MSG_CODE_PME_TURN_OFF;
+ atu.routing = PCIE_MSG_TYPE_R_BC;
+ atu.type = PCIE_ATU_TYPE_MSG;
+ atu.size = resource_size(pci->pp.msg_res);
+ atu.index = pci->pp.msg_atu_index;
+
+ atu.cpu_addr = pci->pp.msg_res->start;
+
+ ret = dw_pcie_prog_outbound_atu(pci, &atu);
+ if (ret)
+ return ret;
+
+ mem = ioremap(atu.cpu_addr, pci->region_align);
+ if (!mem)
+ return -ENOMEM;
+
+ /* A dummy write is converted to a Msg TLP */
+ writel(0, mem);
+
+ iounmap(mem);
+
+ return 0;
+}
+
int dw_pcie_suspend_noirq(struct dw_pcie *pci)
{
u8 offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP);
u32 val;
- int ret;
+ int ret = 0;
/*
* If L1SS is supported, then do not put the link into L2 as some
@@ -854,10 +937,13 @@ int dw_pcie_suspend_noirq(struct dw_pcie *pci)
if (dw_pcie_get_ltssm(pci) <= DW_PCIE_LTSSM_DETECT_ACT)
return 0;
- if (!pci->pp.ops->pme_turn_off)
- return 0;
+ if (pci->pp.ops->pme_turn_off)
+ pci->pp.ops->pme_turn_off(&pci->pp);
+ else
+ ret = dw_pcie_pme_turn_off(pci);
- pci->pp.ops->pme_turn_off(&pci->pp);
+ if (ret)
+ return ret;
ret = read_poll_timeout(dw_pcie_get_ltssm, val, val == DW_PCIE_LTSSM_L2_IDLE,
PCIE_PME_TO_L2_TIMEOUT_US/10,
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index 703b50bc5e0f1..dca5de4c6e877 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -341,6 +341,9 @@ struct dw_pcie_rp {
struct pci_host_bridge *bridge;
raw_spinlock_t lock;
DECLARE_BITMAP(msi_irq_in_use, MAX_MSI_IRQS);
+ bool use_atu_msg;
+ int msg_atu_index;
+ struct resource *msg_res;
};
struct dw_pcie_ep_ops {
--
2.34.1
On Mon, Apr 15, 2024 at 03:33:28PM -0400, Frank Li wrote:
> Add PCIE_MSG_CODE_PME_TURN_OFF macros to enable a PCIe host driver to send
> PME_Turn_Off messages.
>
> Signed-off-by: Frank Li <[email protected]>
> ---
> drivers/pci/pci.h | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> index 19b4227a8a7e8..1f6d54a5a7cfc 100644
> --- a/drivers/pci/pci.h
> +++ b/drivers/pci/pci.h
> @@ -30,6 +30,8 @@
> #define PCIE_MSG_TYPE_R_LOCAL 4
> #define PCIE_MSG_TYPE_R_GATHER 5
>
> +#define PCIE_MSG_CODE_PME_TURN_OFF 0x19
This is defined in PCIe r6.0, sec 2.2.8.2, so move this below the INTx
#defines so they're all in the order of the spec sections and add the
spec citation to follow the same style as the surrounding #defines,
i.e.,
/* Power Management Messages; PCIe r6.0, sec 2.2.8.2 */
#define PCIE_MSG_CODE_PME_TURN_OFF 0x19
While you're at it, tweak the [1/5] comments to be:
/* Message Routing (r[2:0]); PCIe r6.0, sec 2.2.8 */
/* INTx Mechanism Messages; PCIe r6.0, sec 2.2.8.1 */
since the "Message Routing See: PCIe ..." comments are run together
and need some punctuation between the text and the citation.
With these,
Acked-by: Bjorn Helgaas <[email protected]>
> /* INTx Mechanism Messages See: PCIe r6.0, sec 2.2.8.1 */
> #define PCIE_MSG_CODE_ASSERT_INTA 0x20
> #define PCIE_MSG_CODE_ASSERT_INTB 0x21
>
> --
> 2.34.1
>
On Tue, Apr 16, 2024 at 12:02:42PM -0500, Bjorn Helgaas wrote:
> On Mon, Apr 15, 2024 at 03:33:28PM -0400, Frank Li wrote:
> > Add PCIE_MSG_CODE_PME_TURN_OFF macros to enable a PCIe host driver to send
> > PME_Turn_Off messages.
> >
> > Signed-off-by: Frank Li <[email protected]>
> > ---
> > drivers/pci/pci.h | 2 ++
> > 1 file changed, 2 insertions(+)
> >
> > diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> > index 19b4227a8a7e8..1f6d54a5a7cfc 100644
> > --- a/drivers/pci/pci.h
> > +++ b/drivers/pci/pci.h
> > @@ -30,6 +30,8 @@
> > #define PCIE_MSG_TYPE_R_LOCAL 4
> > #define PCIE_MSG_TYPE_R_GATHER 5
> >
> > +#define PCIE_MSG_CODE_PME_TURN_OFF 0x19
>
> This is defined in PCIe r6.0, sec 2.2.8.2, so move this below the INTx
> #defines so they're all in the order of the spec sections and add the
I think it was me who suggested moving this up to keep it sorted in ascending
order :/
- Mani
> spec citation to follow the same style as the surrounding #defines,
> i.e.,
>
> /* Power Management Messages; PCIe r6.0, sec 2.2.8.2 */
> #define PCIE_MSG_CODE_PME_TURN_OFF 0x19
>
> While you're at it, tweak the [1/5] comments to be:
>
> /* Message Routing (r[2:0]); PCIe r6.0, sec 2.2.8 */
> /* INTx Mechanism Messages; PCIe r6.0, sec 2.2.8.1 */
>
> since the "Message Routing See: PCIe ..." comments are run together
> and need some punctuation between the text and the citation.
>
> With these,
>
> Acked-by: Bjorn Helgaas <[email protected]>
>
> > /* INTx Mechanism Messages See: PCIe r6.0, sec 2.2.8.1 */
> > #define PCIE_MSG_CODE_ASSERT_INTA 0x20
> > #define PCIE_MSG_CODE_ASSERT_INTB 0x21
> >
> > --
> > 2.34.1
> >
--
மணிவண்ணன் சதாசிவம்
On Mon, Apr 15, 2024 at 03:33:29PM -0400, Frank Li wrote:
> Instead of relying on the vendor specific implementations to send the
> PME_Turn_Off message, let's introduce a generic way of sending the message
> using the MSG TLP.
>
> This is achieved by reserving a region for MSG TLP of size
> 'pci->region_align', at the end of the first IORESOURCE_MEM window of the
> host bridge. And then sending the PME_Turn_Off message during system
> suspend with the help of iATU.
>
> The reserve space at end is a little bit better than alloc_resource()
> because the below reasons.
> - alloc_resource() will allocate space at begin of IORESOURCE_MEM window.
> There will be a hole when first Endpoint Device (EP) try to alloc a bigger
> space then 'region_align' size.
> - Keep EP device's IORESOURCE_MEM space unchange with/without this patch.
>
I'd rewrite the above sentence as:
'The reason for reserving the MSG TLP region at the end of the
IORESOURCE_MEM is to avoid generating holes in between. Because, when the region
is allocated using allocate_resource(), memory will be allocated from the start
of the window. Later, if memory gets allocated for an endpoint of size bigger
than 'region_align', there will be a hole between MSG TLP region and endpoint
memory.'
> It should be noted that this generic implementation is optional for the
> glue drivers and can be overridden by a custom 'pme_turn_off' callback.
>
> Signed-off-by: Frank Li <[email protected]>
> ---
> drivers/pci/controller/dwc/pcie-designware-host.c | 94 ++++++++++++++++++++++-
> drivers/pci/controller/dwc/pcie-designware.h | 3 +
> 2 files changed, 93 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
> index 3a9cb4be22ab2..5001cdf220123 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> @@ -398,6 +398,31 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)
> return 0;
> }
>
> +static void dw_pcie_host_request_msg_tlp_res(struct dw_pcie_rp *pp)
> +{
> + struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> + struct resource_entry *win;
> + struct resource *res;
> +
> + win = resource_list_first_type(&pp->bridge->windows, IORESOURCE_MEM);
> + if (win) {
> + res = devm_kzalloc(pci->dev, sizeof(*res), GFP_KERNEL);
> + if (!res)
> + return;
> +
> + /* Reserve last region_align block as message TLP space */
/*
* Allocate MSG TLP region of size 'region_align' at the end of
* the host bridge window.
*/
> + res->start = win->res->end - pci->region_align + 1;
> + res->end = win->res->end;
> + res->name = "msg";
> + res->flags = win->res->flags | IORESOURCE_BUSY;
> +
> + if (!devm_request_resource(pci->dev, win->res, res))
> + pp->msg_res = res;
> + else
> + devm_kfree(pci->dev, res);
You are explicitly freeing 'msg_res' everywhere. So either drop devm_ or rely
on devm to free the memory.
> + }
> +}
> +
> int dw_pcie_host_init(struct dw_pcie_rp *pp)
> {
> struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> @@ -484,6 +509,16 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
>
> dw_pcie_iatu_detect(pci);
>
> + /*
> + * Allocate the resource for MSG TLP before programming the iATU outbound window in
> + * dw_pcie_setup_rc(). Since the allocation depends on the value of 'region_align', this has
> + * to be done after dw_pcie_iatu_detect().
Please wrap the comments to 80 columns.
> + *
> + * Glue driver need set use_atu_msg before dw_pcie_host_init() if want MSG TLP feature.
How about,
* Glue drivers need to set 'use_atu_msg' before dw_pcie_host_init() to
* make use of the generic MSG TLP implementation.
> + */
> + if (pp->use_atu_msg)
> + dw_pcie_host_request_msg_tlp_res(pp);
> +
> ret = dw_pcie_edma_detect(pci);
> if (ret)
> goto err_free_msi;
> @@ -541,6 +576,11 @@ void dw_pcie_host_deinit(struct dw_pcie_rp *pp)
>
> dw_pcie_edma_remove(pci);
>
> + if (pp->msg_res) {
> + release_resource(pp->msg_res);
> + devm_kfree(pci->dev, pp->msg_res);
> + }
> +
> if (pp->has_msi_ctrl)
> dw_pcie_free_msi(pp);
>
> @@ -702,6 +742,10 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
> atu.pci_addr = entry->res->start - entry->offset;
> atu.size = resource_size(entry->res);
>
> + /* MSG TLB window resource reserve at one of end of IORESOURCE_MEM resource */
How about,
/* Adjust iATU size if MSG TLP region was allocated before */
if (pp->msg_res && pp->msg_res->parent == entry->res)
atu.size = resource_size(entry->res) -
resource_size(pp->msg_res);
else
atu.size = resource_size(entry->res);
> + if (pp->msg_res && pp->msg_res->parent == entry->res)
> + atu.size -= resource_size(pp->msg_res);
> +
> ret = dw_pcie_prog_outbound_atu(pci, &atu);
> if (ret) {
> dev_err(pci->dev, "Failed to set MEM range %pr\n",
> @@ -733,6 +777,8 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
> dev_warn(pci->dev, "Ranges exceed outbound iATU size (%d)\n",
> pci->num_ob_windows);
>
> + pp->msg_atu_index = i;
> +
> i = 0;
> resource_list_for_each_entry(entry, &pp->bridge->dma_ranges) {
> if (resource_type(entry->res) != IORESOURCE_MEM)
> @@ -838,11 +884,48 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
> }
> EXPORT_SYMBOL_GPL(dw_pcie_setup_rc);
>
> +/* Using message outbound ATU to send out PME_Turn_Off message for dwc PCIe controller */
No need of this comment.
> +static int dw_pcie_pme_turn_off(struct dw_pcie *pci)
> +{
> + struct dw_pcie_ob_atu_cfg atu = { 0 };
> + void __iomem *mem;
> + int ret;
> +
> + if (pci->num_ob_windows <= pci->pp.msg_atu_index)
> + return -EINVAL;
return -ENOSPC;
> +
> + if (!pci->pp.msg_res)
> + return -EINVAL;
return -ENOSPC;
- Mani
--
மணிவண்ணன் சதாசிவம்
On Wed, Apr 17, 2024 at 01:26:55PM +0530, Manivannan Sadhasivam wrote:
> On Tue, Apr 16, 2024 at 12:02:42PM -0500, Bjorn Helgaas wrote:
> > On Mon, Apr 15, 2024 at 03:33:28PM -0400, Frank Li wrote:
> > > Add PCIE_MSG_CODE_PME_TURN_OFF macros to enable a PCIe host driver to send
> > > PME_Turn_Off messages.
> > >
> > > Signed-off-by: Frank Li <[email protected]>
> > > ---
> > > drivers/pci/pci.h | 2 ++
> > > 1 file changed, 2 insertions(+)
> > >
> > > diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> > > index 19b4227a8a7e8..1f6d54a5a7cfc 100644
> > > --- a/drivers/pci/pci.h
> > > +++ b/drivers/pci/pci.h
> > > @@ -30,6 +30,8 @@
> > > #define PCIE_MSG_TYPE_R_LOCAL 4
> > > #define PCIE_MSG_TYPE_R_GATHER 5
> > >
> > > +#define PCIE_MSG_CODE_PME_TURN_OFF 0x19
> >
> > This is defined in PCIe r6.0, sec 2.2.8.2, so move this below the INTx
> > #defines so they're all in the order of the spec sections and add the
>
> I think it was me who suggested moving this up to keep it sorted in ascending
> order :/
OK, the order you suggested is fine.
> > spec citation to follow the same style as the surrounding #defines,
> > i.e.,
> >
> > /* Power Management Messages; PCIe r6.0, sec 2.2.8.2 */
> > #define PCIE_MSG_CODE_PME_TURN_OFF 0x19
> >
> > While you're at it, tweak the [1/5] comments to be:
> >
> > /* Message Routing (r[2:0]); PCIe r6.0, sec 2.2.8 */
> > /* INTx Mechanism Messages; PCIe r6.0, sec 2.2.8.1 */
> >
> > since the "Message Routing See: PCIe ..." comments are run together
> > and need some punctuation between the text and the citation.
But please do add and tweak the comments.
> > With these,
> >
> > Acked-by: Bjorn Helgaas <[email protected]>
Still fine to add this :)
> > > /* INTx Mechanism Messages See: PCIe r6.0, sec 2.2.8.1 */
> > > #define PCIE_MSG_CODE_ASSERT_INTA 0x20
> > > #define PCIE_MSG_CODE_ASSERT_INTB 0x21
> > >
> > > --
> > > 2.34.1
> > >
>
> --
> மணிவண்ணன் சதாசிவம்
On Wed, Apr 17, 2024 at 03:49:44PM +0530, Manivannan Sadhasivam wrote:
> On Mon, Apr 15, 2024 at 03:33:29PM -0400, Frank Li wrote:
> > Instead of relying on the vendor specific implementations to send the
> > PME_Turn_Off message, let's introduce a generic way of sending the message
> > using the MSG TLP.
> >
> > This is achieved by reserving a region for MSG TLP of size
> > 'pci->region_align', at the end of the first IORESOURCE_MEM window of the
> > host bridge. And then sending the PME_Turn_Off message during system
> > suspend with the help of iATU.
> >
> > The reserve space at end is a little bit better than alloc_resource()
> > because the below reasons.
> > - alloc_resource() will allocate space at begin of IORESOURCE_MEM window.
> > There will be a hole when first Endpoint Device (EP) try to alloc a bigger
> > space then 'region_align' size.
> > - Keep EP device's IORESOURCE_MEM space unchange with/without this patch.
> >
>
> I'd rewrite the above sentence as:
>
> 'The reason for reserving the MSG TLP region at the end of the
> IORESOURCE_MEM is to avoid generating holes in between. Because, when the region
> is allocated using allocate_resource(), memory will be allocated from the start
> of the window. Later, if memory gets allocated for an endpoint of size bigger
> than 'region_align', there will be a hole between MSG TLP region and endpoint
> memory.'
>
> > It should be noted that this generic implementation is optional for the
> > glue drivers and can be overridden by a custom 'pme_turn_off' callback.
> >
> > Signed-off-by: Frank Li <[email protected]>
> > ---
> > drivers/pci/controller/dwc/pcie-designware-host.c | 94 ++++++++++++++++++++++-
> > drivers/pci/controller/dwc/pcie-designware.h | 3 +
> > 2 files changed, 93 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
> > index 3a9cb4be22ab2..5001cdf220123 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> > @@ -398,6 +398,31 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)
> > return 0;
> > }
> >
> > +static void dw_pcie_host_request_msg_tlp_res(struct dw_pcie_rp *pp)
> > +{
> > + struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > + struct resource_entry *win;
> > + struct resource *res;
> > +
> > + win = resource_list_first_type(&pp->bridge->windows, IORESOURCE_MEM);
> > + if (win) {
> > + res = devm_kzalloc(pci->dev, sizeof(*res), GFP_KERNEL);
> > + if (!res)
> > + return;
> > +
> > + /* Reserve last region_align block as message TLP space */
>
> /*
> * Allocate MSG TLP region of size 'region_align' at the end of
> * the host bridge window.
> */
>
> > + res->start = win->res->end - pci->region_align + 1;
> > + res->end = win->res->end;
> > + res->name = "msg";
> > + res->flags = win->res->flags | IORESOURCE_BUSY;
> > +
> > + if (!devm_request_resource(pci->dev, win->res, res))
> > + pp->msg_res = res;
> > + else
> > + devm_kfree(pci->dev, res);
>
> You are explicitly freeing 'msg_res' everywhere. So either drop devm_ or rely
> on devm to free the memory.
>
> > + }
> > +}
> > +
> > int dw_pcie_host_init(struct dw_pcie_rp *pp)
> > {
> > struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > @@ -484,6 +509,16 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
> >
> > dw_pcie_iatu_detect(pci);
> >
> > + /*
> > + * Allocate the resource for MSG TLP before programming the iATU outbound window in
> > + * dw_pcie_setup_rc(). Since the allocation depends on the value of 'region_align', this has
> > + * to be done after dw_pcie_iatu_detect().
>
> Please wrap the comments to 80 columns.
New code style is 100 columns. does comments still stick to 80 columns?
>
> > + *
> > + * Glue driver need set use_atu_msg before dw_pcie_host_init() if want MSG TLP feature.
>
> How about,
>
> * Glue drivers need to set 'use_atu_msg' before dw_pcie_host_init() to
> * make use of the generic MSG TLP implementation.
>
> > + */
> > + if (pp->use_atu_msg)
> > + dw_pcie_host_request_msg_tlp_res(pp);
> > +
> > ret = dw_pcie_edma_detect(pci);
> > if (ret)
> > goto err_free_msi;
> > @@ -541,6 +576,11 @@ void dw_pcie_host_deinit(struct dw_pcie_rp *pp)
> >
> > dw_pcie_edma_remove(pci);
> >
> > + if (pp->msg_res) {
> > + release_resource(pp->msg_res);
> > + devm_kfree(pci->dev, pp->msg_res);
> > + }
> > +
> > if (pp->has_msi_ctrl)
> > dw_pcie_free_msi(pp);
> >
> > @@ -702,6 +742,10 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
> > atu.pci_addr = entry->res->start - entry->offset;
> > atu.size = resource_size(entry->res);
> >
> > + /* MSG TLB window resource reserve at one of end of IORESOURCE_MEM resource */
>
> How about,
>
> /* Adjust iATU size if MSG TLP region was allocated before */
> if (pp->msg_res && pp->msg_res->parent == entry->res)
> atu.size = resource_size(entry->res) -
> resource_size(pp->msg_res);
> else
> atu.size = resource_size(entry->res);
>
> > + if (pp->msg_res && pp->msg_res->parent == entry->res)
> > + atu.size -= resource_size(pp->msg_res);
> > +
> > ret = dw_pcie_prog_outbound_atu(pci, &atu);
> > if (ret) {
> > dev_err(pci->dev, "Failed to set MEM range %pr\n",
> > @@ -733,6 +777,8 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
> > dev_warn(pci->dev, "Ranges exceed outbound iATU size (%d)\n",
> > pci->num_ob_windows);
> >
> > + pp->msg_atu_index = i;
> > +
> > i = 0;
> > resource_list_for_each_entry(entry, &pp->bridge->dma_ranges) {
> > if (resource_type(entry->res) != IORESOURCE_MEM)
> > @@ -838,11 +884,48 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
> > }
> > EXPORT_SYMBOL_GPL(dw_pcie_setup_rc);
> >
> > +/* Using message outbound ATU to send out PME_Turn_Off message for dwc PCIe controller */
>
> No need of this comment.
>
> > +static int dw_pcie_pme_turn_off(struct dw_pcie *pci)
> > +{
> > + struct dw_pcie_ob_atu_cfg atu = { 0 };
> > + void __iomem *mem;
> > + int ret;
> > +
> > + if (pci->num_ob_windows <= pci->pp.msg_atu_index)
> > + return -EINVAL;
>
> return -ENOSPC;
>
> > +
> > + if (!pci->pp.msg_res)
> > + return -EINVAL;
>
> return -ENOSPC;
> - Mani
>
> --
> மணிவண்ணன் சதாசிவம்
On Wed, Apr 17, 2024 at 10:38:04AM -0400, Frank Li wrote:
> On Wed, Apr 17, 2024 at 03:49:44PM +0530, Manivannan Sadhasivam wrote:
> > On Mon, Apr 15, 2024 at 03:33:29PM -0400, Frank Li wrote:
> > > Instead of relying on the vendor specific implementations to send the
> > > PME_Turn_Off message, let's introduce a generic way of sending the message
> > > using the MSG TLP.
> > >
> > > This is achieved by reserving a region for MSG TLP of size
> > > 'pci->region_align', at the end of the first IORESOURCE_MEM window of the
> > > host bridge. And then sending the PME_Turn_Off message during system
> > > suspend with the help of iATU.
> > >
> > > The reserve space at end is a little bit better than alloc_resource()
> > > because the below reasons.
> > > - alloc_resource() will allocate space at begin of IORESOURCE_MEM window.
> > > There will be a hole when first Endpoint Device (EP) try to alloc a bigger
> > > space then 'region_align' size.
> > > - Keep EP device's IORESOURCE_MEM space unchange with/without this patch.
> > >
> >
> > I'd rewrite the above sentence as:
> >
> > 'The reason for reserving the MSG TLP region at the end of the
> > IORESOURCE_MEM is to avoid generating holes in between. Because, when the region
> > is allocated using allocate_resource(), memory will be allocated from the start
> > of the window. Later, if memory gets allocated for an endpoint of size bigger
> > than 'region_align', there will be a hole between MSG TLP region and endpoint
> > memory.'
> >
> > > It should be noted that this generic implementation is optional for the
> > > glue drivers and can be overridden by a custom 'pme_turn_off' callback.
> > >
> > > Signed-off-by: Frank Li <[email protected]>
> > > ---
> > > drivers/pci/controller/dwc/pcie-designware-host.c | 94 ++++++++++++++++++++++-
> > > drivers/pci/controller/dwc/pcie-designware.h | 3 +
> > > 2 files changed, 93 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
> > > index 3a9cb4be22ab2..5001cdf220123 100644
> > > --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> > > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> > > @@ -398,6 +398,31 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp)
> > > return 0;
> > > }
> > >
> > > +static void dw_pcie_host_request_msg_tlp_res(struct dw_pcie_rp *pp)
> > > +{
> > > + struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > > + struct resource_entry *win;
> > > + struct resource *res;
> > > +
> > > + win = resource_list_first_type(&pp->bridge->windows, IORESOURCE_MEM);
> > > + if (win) {
> > > + res = devm_kzalloc(pci->dev, sizeof(*res), GFP_KERNEL);
> > > + if (!res)
> > > + return;
> > > +
> > > + /* Reserve last region_align block as message TLP space */
> >
> > /*
> > * Allocate MSG TLP region of size 'region_align' at the end of
> > * the host bridge window.
> > */
> >
> > > + res->start = win->res->end - pci->region_align + 1;
> > > + res->end = win->res->end;
> > > + res->name = "msg";
> > > + res->flags = win->res->flags | IORESOURCE_BUSY;
> > > +
> > > + if (!devm_request_resource(pci->dev, win->res, res))
> > > + pp->msg_res = res;
> > > + else
> > > + devm_kfree(pci->dev, res);
> >
> > You are explicitly freeing 'msg_res' everywhere. So either drop devm_ or rely
> > on devm to free the memory.
> >
> > > + }
> > > +}
> > > +
> > > int dw_pcie_host_init(struct dw_pcie_rp *pp)
> > > {
> > > struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > > @@ -484,6 +509,16 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
> > >
> > > dw_pcie_iatu_detect(pci);
> > >
> > > + /*
> > > + * Allocate the resource for MSG TLP before programming the iATU outbound window in
> > > + * dw_pcie_setup_rc(). Since the allocation depends on the value of 'region_align', this has
> > > + * to be done after dw_pcie_iatu_detect().
> >
> > Please wrap the comments to 80 columns.
>
> New code style is 100 columns. does comments still stick to 80 columns?
>
That's only applicable for code and not comments. I also learned that some time
back.
- Mani
> >
> > > + *
> > > + * Glue driver need set use_atu_msg before dw_pcie_host_init() if want MSG TLP feature.
> >
> > How about,
> >
> > * Glue drivers need to set 'use_atu_msg' before dw_pcie_host_init() to
> > * make use of the generic MSG TLP implementation.
> >
> > > + */
> > > + if (pp->use_atu_msg)
> > > + dw_pcie_host_request_msg_tlp_res(pp);
> > > +
> > > ret = dw_pcie_edma_detect(pci);
> > > if (ret)
> > > goto err_free_msi;
> > > @@ -541,6 +576,11 @@ void dw_pcie_host_deinit(struct dw_pcie_rp *pp)
> > >
> > > dw_pcie_edma_remove(pci);
> > >
> > > + if (pp->msg_res) {
> > > + release_resource(pp->msg_res);
> > > + devm_kfree(pci->dev, pp->msg_res);
> > > + }
> > > +
> > > if (pp->has_msi_ctrl)
> > > dw_pcie_free_msi(pp);
> > >
> > > @@ -702,6 +742,10 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
> > > atu.pci_addr = entry->res->start - entry->offset;
> > > atu.size = resource_size(entry->res);
> > >
> > > + /* MSG TLB window resource reserve at one of end of IORESOURCE_MEM resource */
> >
> > How about,
> >
> > /* Adjust iATU size if MSG TLP region was allocated before */
> > if (pp->msg_res && pp->msg_res->parent == entry->res)
> > atu.size = resource_size(entry->res) -
> > resource_size(pp->msg_res);
> > else
> > atu.size = resource_size(entry->res);
> >
> > > + if (pp->msg_res && pp->msg_res->parent == entry->res)
> > > + atu.size -= resource_size(pp->msg_res);
> > > +
> > > ret = dw_pcie_prog_outbound_atu(pci, &atu);
> > > if (ret) {
> > > dev_err(pci->dev, "Failed to set MEM range %pr\n",
> > > @@ -733,6 +777,8 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp)
> > > dev_warn(pci->dev, "Ranges exceed outbound iATU size (%d)\n",
> > > pci->num_ob_windows);
> > >
> > > + pp->msg_atu_index = i;
> > > +
> > > i = 0;
> > > resource_list_for_each_entry(entry, &pp->bridge->dma_ranges) {
> > > if (resource_type(entry->res) != IORESOURCE_MEM)
> > > @@ -838,11 +884,48 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp)
> > > }
> > > EXPORT_SYMBOL_GPL(dw_pcie_setup_rc);
> > >
> > > +/* Using message outbound ATU to send out PME_Turn_Off message for dwc PCIe controller */
> >
> > No need of this comment.
> >
> > > +static int dw_pcie_pme_turn_off(struct dw_pcie *pci)
> > > +{
> > > + struct dw_pcie_ob_atu_cfg atu = { 0 };
> > > + void __iomem *mem;
> > > + int ret;
> > > +
> > > + if (pci->num_ob_windows <= pci->pp.msg_atu_index)
> > > + return -EINVAL;
> >
> > return -ENOSPC;
> >
> > > +
> > > + if (!pci->pp.msg_res)
> > > + return -EINVAL;
> >
> > return -ENOSPC;
> > - Mani
> >
> > --
> > மணிவண்ணன் சதாசிவம்
--
மணிவண்ணன் சதாசிவம்
On Wed, Apr 17, 2024 at 10:38:04AM -0400, Frank Li wrote:
> On Wed, Apr 17, 2024 at 03:49:44PM +0530, Manivannan Sadhasivam wrote:
> > On Mon, Apr 15, 2024 at 03:33:29PM -0400, Frank Li wrote:
> > > + /*
> > > + * Allocate the resource for MSG TLP before programming the iATU outbound window in
> > > + * dw_pcie_setup_rc(). Since the allocation depends on the value of 'region_align', this has
> > > + * to be done after dw_pcie_iatu_detect().
> >
> > Please wrap the comments to 80 columns.
>
> New code style is 100 columns. does comments still stick to 80 columns?
I care more about files being consistent internally than I do about
adhering to a new 100 column rule.
With very few exceptions, pcie-designware-host.c fits in 80 columns.
I think there's no point in adding a comment using 100 columns. That
leadds to either a window sized to accommodate 100 columns, where 20
columns are wasted for all the existing code and comments, or a window
sized for 80 columns, where the new comment looks ugly. Neither is
attractive.
Bjorn