2024-04-16 06:00:44

by Luca Ceresoli

[permalink] [raw]
Subject: [PATCH 0/3] ASoC: dapm: improve debugfs output and introduce dapm-graph

This patch series improves the tools available to understand and debug
DAPM.

Patches 1-2 add more information to the widget files exposed in debugfs.

Patch 3 introduces dapm-graph, a script to generate a picture of the DAPM
state inspired to vizdapm.

Patch 1 is required for dapm-graph to work, while patch 2 allows to enhance
its features but it is not required.

Signed-off-by: Luca Ceresoli <[email protected]>
---
Luca Ceresoli (3):
ASoC: dapm: debugfs: add component to route lines
ASoC: dapm: debugfs: show the widget type
tools/sound/dapm-graph: new tool to visualize DAPM state

MAINTAINERS | 6 +
sound/soc/soc-dapm.c | 54 ++++++++-
tools/sound/dapm-graph | 303 +++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 361 insertions(+), 2 deletions(-)
---
base-commit: 5568f556e601b02d673bc7edab8de0f11815eec7
change-id: 20240405-vizdapm-ng-c7569ffa8e58

Best regards,
--
Luca Ceresoli <[email protected]>



2024-04-16 06:01:14

by Luca Ceresoli

[permalink] [raw]
Subject: [PATCH 2/3] ASoC: dapm: debugfs: show the widget type

To make the widget debugfs files more informative, add a line showing the
widget type string.

Keeping backward compatibility is nice to have being debugfs, and ease of
parsing by both humans and software is also good. To maximize both with a
reasonable effort add a new line without thouching the already complex
format of the first line. The syntax is meant to be a key/value pair.

The existing vizdapm tool continues working after this change, ignoring the
new line.

The new format is:

Left ADC: Off in 1 out 0 - R2(0x2) mask 0x2
stream Left HiFi Capture inactive
widget-type adc
out "static" "Capture" "cs42l51.0-004a"
in "static" "Left PGA" "cs42l51.0-004a"

Signed-off-by: Luca Ceresoli <[email protected]>
---
sound/soc/soc-dapm.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)

diff --git a/sound/soc/soc-dapm.c b/sound/soc/soc-dapm.c
index eb9535764693..16dad4a45443 100644
--- a/sound/soc/soc-dapm.c
+++ b/sound/soc/soc-dapm.c
@@ -2094,6 +2094,48 @@ static int dapm_power_widgets(struct snd_soc_card *card, int event)
}

#ifdef CONFIG_DEBUG_FS
+
+static const char * const snd_soc_dapm_type_name[] = {
+ [snd_soc_dapm_input] = "input",
+ [snd_soc_dapm_output] = "output",
+ [snd_soc_dapm_mux] = "mux",
+ [snd_soc_dapm_demux] = "demux",
+ [snd_soc_dapm_mixer] = "mixer",
+ [snd_soc_dapm_mixer_named_ctl] = "mixer_named_ctl",
+ [snd_soc_dapm_pga] = "pga",
+ [snd_soc_dapm_out_drv] = "out_drv",
+ [snd_soc_dapm_adc] = "adc",
+ [snd_soc_dapm_dac] = "dac",
+ [snd_soc_dapm_micbias] = "micbias",
+ [snd_soc_dapm_mic] = "mic",
+ [snd_soc_dapm_hp] = "hp",
+ [snd_soc_dapm_spk] = "spk",
+ [snd_soc_dapm_line] = "line",
+ [snd_soc_dapm_switch] = "switch",
+ [snd_soc_dapm_vmid] = "vmid",
+ [snd_soc_dapm_pre] = "pre",
+ [snd_soc_dapm_post] = "post",
+ [snd_soc_dapm_supply] = "supply",
+ [snd_soc_dapm_pinctrl] = "pinctrl",
+ [snd_soc_dapm_regulator_supply] = "regulator_supply",
+ [snd_soc_dapm_clock_supply] = "clock_supply",
+ [snd_soc_dapm_aif_in] = "aif_in",
+ [snd_soc_dapm_aif_out] = "aif_out",
+ [snd_soc_dapm_siggen] = "siggen",
+ [snd_soc_dapm_sink] = "sink",
+ [snd_soc_dapm_dai_in] = "dai_in",
+ [snd_soc_dapm_dai_out] = "dai_out",
+ [snd_soc_dapm_dai_link] = "dai_link",
+ [snd_soc_dapm_kcontrol] = "kcontrol",
+ [snd_soc_dapm_buffer] = "buffer",
+ [snd_soc_dapm_scheduler] = "scheduler",
+ [snd_soc_dapm_effect] = "effect",
+ [snd_soc_dapm_src] = "src",
+ [snd_soc_dapm_asrc] = "asrc",
+ [snd_soc_dapm_encoder] = "encoder",
+ [snd_soc_dapm_decoder] = "decoder",
+};
+
static ssize_t dapm_widget_power_read_file(struct file *file,
char __user *user_buf,
size_t count, loff_t *ppos)
@@ -2106,6 +2148,8 @@ static ssize_t dapm_widget_power_read_file(struct file *file,
struct snd_soc_dapm_path *p = NULL;
const char *c_name;

+ BUILD_BUG_ON(ARRAY_SIZE(snd_soc_dapm_type_name) != SND_SOC_DAPM_TYPE_COUNT);
+
buf = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!buf)
return -ENOMEM;
@@ -2137,6 +2181,9 @@ static ssize_t dapm_widget_power_read_file(struct file *file,
w->sname,
w->active ? "active" : "inactive");

+ ret += scnprintf(buf + ret, PAGE_SIZE - ret, " widget-type %s\n",
+ snd_soc_dapm_type_name[w->id]);
+
snd_soc_dapm_for_each_direction(dir) {
rdir = SND_SOC_DAPM_DIR_REVERSE(dir);
snd_soc_dapm_widget_for_each_path(w, dir, p) {

--
2.34.1


2024-04-16 06:02:19

by Luca Ceresoli

[permalink] [raw]
Subject: [PATCH 1/3] ASoC: dapm: debugfs: add component to route lines

debugfs entries for DAPM widgets have the following form:

Left ADC: Off in 1 out 0 - R2(0x2) mask 0x2
stream Left HiFi Capture inactive
out "static" "Capture"
in "static" "Left PGA"

Lines with the "in" and "out" prefixes describe routes from/to other
widgets presenting the path name and the connected widget name.

This is ambiguous in case of cards having multiple widgets with the same
name in different components. For example the STM32MP157A-DK1 board
(arch/arm/boot/dts/st/stm32mp15xx-dkx.dtsi) has a "Capture" widget in both
the "cs42l51.0-004a" and the "hdmi-audio-codec.1.auto" components.

Avoid the ambiguity by adding the component name to the "in" and "out"
lines. Add the new field at the end to minimize backward compatibility
issues. The existing vizdapm tool continues working after this change.

The output becomes:

Left ADC: Off in 1 out 0 - R2(0x2) mask 0x2
stream Left HiFi Capture inactive
out "static" "Capture" "cs42l51.0-004a"
in "static" "Left PGA" "cs42l51.0-004a"

Signed-off-by: Luca Ceresoli <[email protected]>
---
sound/soc/soc-dapm.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/sound/soc/soc-dapm.c b/sound/soc/soc-dapm.c
index ad8ba8fbbaee..eb9535764693 100644
--- a/sound/soc/soc-dapm.c
+++ b/sound/soc/soc-dapm.c
@@ -2104,6 +2104,7 @@ static ssize_t dapm_widget_power_read_file(struct file *file,
int in, out;
ssize_t ret;
struct snd_soc_dapm_path *p = NULL;
+ const char *c_name;

buf = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!buf)
@@ -2145,11 +2146,13 @@ static ssize_t dapm_widget_power_read_file(struct file *file,
if (!p->connect)
continue;

+ c_name = p->node[rdir]->dapm->component ?
+ p->node[rdir]->dapm->component->name : NULL;
ret += scnprintf(buf + ret, PAGE_SIZE - ret,
- " %s \"%s\" \"%s\"\n",
+ " %s \"%s\" \"%s\" \"%s\"\n",
(rdir == SND_SOC_DAPM_DIR_IN) ? "in" : "out",
p->name ? p->name : "static",
- p->node[rdir]->name);
+ p->node[rdir]->name, c_name);
}
}


--
2.34.1


2024-04-16 06:06:57

by Luca Ceresoli

[permalink] [raw]
Subject: [PATCH 3/3] tools/sound/dapm-graph: new tool to visualize DAPM state

Add a tool to generate a picture of the current DAPM state for a sound
card.

dapm-graph is inspired by vizdapm which used to be published on a Wolfson
Micro git repository now disappeared, and has a few forks around:

https://github.com/mihais/asoc-tools
https://github.com/alexandrebelloni/asoc-tools

dapm-graph is a full reimplementation with several improvements while still
being a self-contained shell script:

Improvements to rendered output:
- shows the entire card, not one component hierarchy only
- each component is rendered in a separate box
- shows widget on/off status based on widget information alone (the
original vizdapm propagates the "on" green colour to the first input
widget)
- use bold line and gray background and not only green/red line to show
on/off status (for the color blind)

Improvements for embedded system developers:
- remote mode: get state of remote device (possibly with minimal rootfs)
via SSH, but parsing locally for faster operation
- compatible with BusyBox shell, not only bash

Usability improvements:
- flexible command line (uses getopts for parsing)
- detailed help text
- flag to enable detailed debug logging
- graphviz output format detected from file extension, not hard coded
- a self-contained shell script

Usage is designed to be simple:

dapm-grpah -c CARD - get state from debugfs for CARD
dapm-grpah -c CARD -r REMOTE_TARGET - same, but remotely via SSH
dapm-grpah -d STATE_DIR - from a local copy of the debugfs
tree for a card

Signed-off-by: Luca Ceresoli <[email protected]>
---
MAINTAINERS | 6 +
tools/sound/dapm-graph | 303 +++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 309 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 997ebd1ae474..8c1be129419e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -20773,6 +20773,12 @@ F: include/trace/events/sof*.h
F: include/uapi/sound/asoc.h
F: sound/soc/

+SOUND - SOC LAYER / dapm-graph
+M: Luca Ceresoli <[email protected]>
+L: [email protected]
+S: Maintained
+F: tools/sound/dapm-graph
+
SOUND - SOUND OPEN FIRMWARE (SOF) DRIVERS
M: Pierre-Louis Bossart <[email protected]>
M: Liam Girdwood <[email protected]>
diff --git a/tools/sound/dapm-graph b/tools/sound/dapm-graph
new file mode 100755
index 000000000000..57d78f6df041
--- /dev/null
+++ b/tools/sound/dapm-graph
@@ -0,0 +1,303 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+#
+# Generate a graph of the current DAPM state for an audio card
+#
+# Copyright 2024 Bootlin
+# Author: Luca Ceresoli <[email protected]>
+
+set -eu
+
+STYLE_NODE_ON="shape=box,style=bold,color=green4"
+STYLE_NODE_OFF="shape=box,style=filled,color=gray30,fillcolor=gray95"
+
+# Print usage and exit
+#
+# $1 = exit return value
+# $2 = error string (required if $1 != 0)
+usage()
+{
+ if [ "${1}" -ne 0 ]; then
+ echo "${2}" >&2
+ fi
+
+ echo "
+Generate a graph of the current DAPM state for an audio card.
+
+The DAPM state can be obtained via debugfs for a card on the local host or
+a remote target, or from a local copy of the debugfs tree for the card.
+
+Usage:
+ $(basename $0) [options] -c CARD - Local sound card
+ $(basename $0) [options] -c CARD -r REMOTE_TARGET - Card on remote system
+ $(basename $0) [options] -d STATE_DIR - Local directory
+
+Options:
+ -c CARD Sound card to get DAPM state of
+ -r REMOTE_TARGET Get DAPM state from REMOTE_TARGET via SSH and SCP
+ instead of using a local sound card
+ -d STATE_DIR Get DAPM state from a local copy of a debugfs tree
+ -o OUT_FILE Output file (default: dapm.dot)
+ -D Show verbose debugging info
+ -h Print this help and exit
+
+The output format is implied by the extension of OUT_FILE:
+
+ * Use the .dot extension to generate a text graph representation in
+ graphviz dot syntax.
+ * Any other extension is assumed to be a format supported by graphviz for
+ rendering, e.g. 'png', 'svg', and will produce both the .dot file and a
+ picture from it. This requires the 'dot' program from the graphviz
+ package.
+"
+
+ exit ${1}
+}
+
+# Connect to a remote target via SSH, collect all DAPM files from debufs
+# into a tarball and get the tarball via SCP into $3/dapm.tar
+#
+# $1 = target as used by ssh and scp, e.g. "[email protected]"
+# $2 = sound card name
+# $3 = temp dir path (present on the host, created on the target)
+# $4 = local directory to extract the tarball into
+#
+# Requires an ssh+scp server, find and tar+gz on the target
+#
+# Note: the tarball is needed because plain 'scp -r' from debugfs would
+# copy only empty files
+grab_remote_files()
+{
+ echo "Collecting DAPM state from ${1}"
+ dbg_echo "Collected DAPM state in ${3}"
+
+ ssh "${1}" "
+set -eu &&
+cd \"/sys/kernel/debug/asoc/${2}\" &&
+find * -type d -exec mkdir -p ${3}/dapm-tree/{} \; &&
+find * -type f -exec cp \"{}\" \"${3}/dapm-tree/{}\" \; &&
+cd ${3}/dapm-tree &&
+tar cf ${3}/dapm.tar ."
+ scp -q "${1}:${3}/dapm.tar" "${3}"
+
+ mkdir -p "${4}"
+ tar xf "${tmp_dir}/dapm.tar" -C "${4}"
+}
+
+# Parse a widget file and generate graph description in graphviz dot format
+#
+# Skips any file named "bias_level".
+#
+# $1 = temporary work dir
+# $2 = component name
+# $3 = widget filename
+process_dapm_widget()
+{
+ local tmp_dir="${1}"
+ local c_name="${2}"
+ local w_file="${3}"
+ local dot_file="${tmp_dir}/main.dot"
+ local links_file="${tmp_dir}/links.dot"
+
+ local w_name="$(basename "${w_file}")"
+ local w_tag="${c_name}_${w_name}"
+
+ if [ "${w_name}" = "bias_level" ]; then
+ return 0
+ fi
+
+ dbg_echo " + Widget: ${w_name}"
+
+ cat "${w_file}" | (
+ read line
+
+ if echo "${line}" | grep -q ': On '
+ then local node_style="${STYLE_NODE_ON}"
+ else local node_style="${STYLE_NODE_OFF}"
+ fi
+
+ local w_type=""
+ while read line; do
+ # Collect widget type if present
+ if echo "${line}" | grep -q '^widget-type '; then
+ local w_type_raw="$(echo "$line" | cut -d ' ' -f 2)"
+ dbg_echo " - Widget type: ${w_type_raw}"
+
+ # Note: escaping '\n' is tricky to get working with both
+ # bash and busybox ash, so use a '%' here and replace it
+ # later
+ local w_type="%n[${w_type_raw}]"
+ fi
+
+ # Collect any links. We could use "in" links or "out" links,
+ # let's use "in" links
+ if echo "${line}" | grep -q '^in '; then
+ local w_src=$(echo "$line" |
+ awk -F\" '{print $6 "_" $4}' |
+ sed 's/^(null)_/ROOT_/')
+ dbg_echo " - Input route from: ${w_src}"
+ echo " \"${w_src}\" -> \"$w_tag\"" >> "${links_file}"
+ fi
+ done
+
+ echo " \"${w_tag}\" [label=\"${w_name}${w_type}\",${node_style}]" |
+ tr '%' '\\' >> "${dot_file}"
+ )
+}
+
+# Parse the DAPM tree for a sound card component and generate graph
+# description in graphviz dot format
+#
+# $1 = temporary work dir
+# $2 = component directory
+# $3 = forced component name (extracted for path if empty)
+process_dapm_component()
+{
+ local tmp_dir="${1}"
+ local c_dir="${2}"
+ local c_name="${3}"
+ local dot_file="${tmp_dir}/main.dot"
+ local links_file="${tmp_dir}/links.dot"
+
+ if [ -z "${c_name}" ]; then
+ # Extract directory name into component name:
+ # "./cs42l51.0-004a/dapm" -> "cs42l51.0-004a"
+ c_name="$(basename $(dirname "${c_dir}"))"
+ fi
+
+ dbg_echo " * Component: ${c_name}"
+
+ echo "" >> "${dot_file}"
+ echo " subgraph \"${c_name}\" {" >> "${dot_file}"
+ echo " cluster = true" >> "${dot_file}"
+ echo " label = \"${c_name}\"" >> "${dot_file}"
+ echo " color=dodgerblue" >> "${dot_file}"
+
+ # Create empty file to ensure it will exist in all cases
+ >"${links_file}"
+
+ # Iterate over widgets in the component dir
+ for w_file in ${c_dir}/*; do
+ process_dapm_widget "${tmp_dir}" "${c_name}" "${w_file}"
+ done
+
+ echo " }" >> "${dot_file}"
+
+ cat "${links_file}" >> "${dot_file}"
+}
+
+# Parse the DAPM tree for a sound card and generate graph description in
+# graphviz dot format
+#
+# $1 = temporary work dir
+# $2 = directory tree with DAPM state (either in debugfs or a mirror)
+process_dapm_tree()
+{
+ local tmp_dir="${1}"
+ local dapm_dir="${2}"
+ local dot_file="${tmp_dir}/main.dot"
+
+ echo "digraph G {" > "${dot_file}"
+ echo " fontname=\"sans-serif\"" >> "${dot_file}"
+ echo " node [fontname=\"sans-serif\"]" >> "${dot_file}"
+
+
+ # Process root directory (no component)
+ process_dapm_component "${tmp_dir}" "${dapm_dir}/dapm" "ROOT"
+
+ # Iterate over components
+ for c_dir in "${dapm_dir}"/*/dapm
+ do
+ process_dapm_component "${tmp_dir}" "${c_dir}" ""
+ done
+
+ echo "}" >> "${dot_file}"
+}
+
+main()
+{
+ # Parse command line
+ local out_file="dapm.dot"
+ local card_name=""
+ local remote_target=""
+ local dapm_tree=""
+ local dbg_on=""
+ while getopts "c:r:d:o:Dh" arg; do
+ case $arg in
+ c) card_name="${OPTARG}" ;;
+ r) remote_target="${OPTARG}" ;;
+ d) dapm_tree="${OPTARG}" ;;
+ o) out_file="${OPTARG}" ;;
+ D) dbg_on="1" ;;
+ h) usage 0 ;;
+ *) usage 1 ;;
+ esac
+ done
+ shift $(($OPTIND - 1))
+
+ if [ -n "${dapm_tree}" ]; then
+ if [ -n "${card_name}${remote_target}" ]; then
+ usage 1 "Cannot use -c and -r with -d"
+ fi
+ echo "Using local tree: ${dapm_tree}"
+ elif [ -n "${remote_target}" ]; then
+ if [ -z "${card_name}" ]; then
+ usage 1 "-r requires -c"
+ fi
+ echo "Using card ${card_name} from remote target ${remote_target}"
+ elif [ -n "${card_name}" ]; then
+ echo "Using local card: ${card_name}"
+ else
+ usage 1 "Please choose mode using -c, -r or -d"
+ fi
+
+ # Define logging function
+ if [ "${dbg_on}" ]; then
+ dbg_echo() {
+ echo "$*" >&2
+ }
+ else
+ dbg_echo() {
+ :
+ }
+ fi
+
+ # Filename must have a dot in order the infer the format from the
+ # extension
+ if ! echo "${out_file}" | grep -qE '\.'; then
+ echo "Missing extension in output filename ${out_file}" >&2
+ usage
+ exit 1
+ fi
+
+ local out_fmt="${out_file##*.}"
+ local dot_file="${out_file%.*}.dot"
+
+ dbg_echo "dot file: $dot_file"
+ dbg_echo "Output file: $out_file"
+ dbg_echo "Output format: $out_fmt"
+
+ tmp_dir="$(mktemp -d /tmp/$(basename $0).XXXXXX)"
+ trap "{ rm -fr ${tmp_dir}; }" INT TERM EXIT
+
+ if [ -z "${dapm_tree}" ]
+ then
+ dapm_tree="/sys/kernel/debug/asoc/${card_name}"
+ fi
+ if [ -n "${remote_target}" ]; then
+ dapm_tree="${tmp_dir}/dapm-tree"
+ grab_remote_files "${remote_target}" "${card_name}" "${tmp_dir}" "${dapm_tree}"
+ fi
+ # In all cases now ${dapm_tree} contains the DAPM state
+
+ process_dapm_tree "${tmp_dir}" "${dapm_tree}"
+ cp "${tmp_dir}/main.dot" "${dot_file}"
+
+ if [ "${out_file}" != "${dot_file}" ]; then
+ dot -T"${out_fmt}" "${dot_file}" -o "${out_file}"
+ fi
+
+ echo "Generated file ${out_file}"
+}
+
+main "${@}"

--
2.34.1


2024-04-16 23:37:59

by Alexandre Belloni

[permalink] [raw]
Subject: Re: [PATCH 0/3] ASoC: dapm: improve debugfs output and introduce dapm-graph

On 16/04/2024 08:00:23+0200, Luca Ceresoli wrote:
> This patch series improves the tools available to understand and debug
> DAPM.
>
> Patches 1-2 add more information to the widget files exposed in debugfs.
>
> Patch 3 introduces dapm-graph, a script to generate a picture of the DAPM
> state inspired to vizdapm.
>
> Patch 1 is required for dapm-graph to work, while patch 2 allows to enhance
> its features but it is not required.
>
> Signed-off-by: Luca Ceresoli <[email protected]>

Reviewed-by: Alexandre Belloni <[email protected]>

> ---
> Luca Ceresoli (3):
> ASoC: dapm: debugfs: add component to route lines
> ASoC: dapm: debugfs: show the widget type
> tools/sound/dapm-graph: new tool to visualize DAPM state
>
> MAINTAINERS | 6 +
> sound/soc/soc-dapm.c | 54 ++++++++-
> tools/sound/dapm-graph | 303 +++++++++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 361 insertions(+), 2 deletions(-)
> ---
> base-commit: 5568f556e601b02d673bc7edab8de0f11815eec7
> change-id: 20240405-vizdapm-ng-c7569ffa8e58
>
> Best regards,
> --
> Luca Ceresoli <[email protected]>
>

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

2024-04-22 08:10:07

by Mark Brown

[permalink] [raw]
Subject: Re: [PATCH 0/3] ASoC: dapm: improve debugfs output and introduce dapm-graph

On Tue, 16 Apr 2024 08:00:23 +0200, Luca Ceresoli wrote:
> This patch series improves the tools available to understand and debug
> DAPM.
>
> Patches 1-2 add more information to the widget files exposed in debugfs.
>
> Patch 3 introduces dapm-graph, a script to generate a picture of the DAPM
> state inspired to vizdapm.
>
> [...]

Applied to

https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound.git for-next

Thanks!

[1/3] ASoC: dapm: debugfs: add component to route lines
commit: cbd9eed87107bb6d6e537e79aeea65399898ca41
[2/3] ASoC: dapm: debugfs: show the widget type
commit: 5b1047dcf80b35bafcacbd10b57342d1a96139c0
[3/3] tools/sound/dapm-graph: new tool to visualize DAPM state
commit: e7bb43898bcf54da7ffb4819a04c8428f7db24db

All being well this means that it will be integrated into the linux-next
tree (usually sometime in the next 24 hours) and sent to Linus during
the next merge window (or sooner if it is a bug fix), however if
problems are discovered then the patch may be dropped or reverted.

You may get further e-mails resulting from automated or manual testing
and review of the tree, please engage with people reporting problems and
send followup patches addressing any issues that are reported if needed.

If any updates are required or you are submitting further changes they
should be sent as incremental updates against current git, existing
patches will not be replaced.

Please add any relevant lists and maintainers to the CCs when replying
to this mail.

Thanks,
Mark