LinuxLists.cc - [PATCH v2 0/7] Fix Python string escapes

2023-09-12 10:04:17

Subject: [PATCH v2 0/7] Fix Python string escapes

Changes from v1:
* Dropped some changes that were independently fixed[1]
* No longer separate the f strings to their own patch
* Use r strings when the value is a regular expression
* Updated verification script

In retrospect a script to find the instances and apply fixes isn't that
useful for review, so the attached script this time just looks for
differences in the AST. Apply the series and run the script, with
the two references to compare as arguments.

There are some intentional changes to the AST now though, as the r strings
turn '\t' from a single character tab into a backslash and 't' character
pair (similar for '\n'). This does not affect the correctness of the
regular expression though.

v1: https://lore.kernel.org/all/[email protected]/

[1]: https://lore.kernel.org/all/[email protected]/

---
#!/usr/bin/env python3

"""
Verify Python syntax trees are equivalent between two references
"""

import argparse
import ast
from pathlib import Path
import subprocess as sp

def read_file(path: Path, ref: str) -> str:
return sp.run(f"git show {ref}:{path}", stdout=sp.PIPE, shell=True, encoding="utf-8", check=True).stdout

parser = argparse.ArgumentParser("Compare Python ASTs between revisions")
parser.add_argument("ref1", type=str, help="First revision to use")
parser.add_argument("ref2", type=str, help="Second revision to use")
args = parser.parse_args()

for pyfile in Path(".").glob("**/*.py"):
try:
ref1_content = read_file(pyfile, args.ref1)
ref2_content = read_file(pyfile, args.ref2)
except Exception as e:
print(f"ERROR:{pyfile}: Failed to read ({e})")
continue

try:
ref1_syntax = ast.parse(ref1_content, filename=pyfile)
ref2_syntax = ast.parse(ref2_content, filename=pyfile)
except SyntaxError as e:
print(f"ERROR:{pyfile}: Failed to parse, is it Python3? ({e})")
continue

if ast.dump(ref1_syntax) != ast.dump(ref2_syntax):
print(f"ERROR:{pyfile}: Revisions have different AST")
cmd = f"diff <(git show {args.ref1}:{pyfile} | python -m ast) <(git show {args.ref2}:{pyfile} | python -m ast)"
print(cmd)
sp.run(cmd, shell=True)
continue

Benjamin Gray (7):
ia64: fix Python string escapes
Documentation/sphinx: fix Python string escapes
drivers/comedi: fix Python string escapes
scripts: fix Python string escapes
tools/perf: fix Python string escapes
tools/power: fix Python string escapes
selftests/bpf: fix Python string escapes

Documentation/sphinx/cdomain.py | 2 +-
Documentation/sphinx/kernel_abi.py | 2 +-
Documentation/sphinx/kernel_feat.py | 2 +-
Documentation/sphinx/kerneldoc.py | 2 +-
Documentation/sphinx/maintainers_include.py | 8 +++---
arch/ia64/scripts/unwcheck.py | 2 +-
.../ni_routing/tools/convert_csv_to_c.py | 2 +-
scripts/clang-tools/gen_compile_commands.py | 2 +-
scripts/gdb/linux/symbols.py | 2 +-
tools/perf/pmu-events/jevents.py | 2 +-
.../scripts/python/arm-cs-trace-disasm.py | 4 +--
tools/perf/scripts/python/compaction-times.py | 2 +-
.../scripts/python/exported-sql-viewer.py | 4 +--
tools/power/pm-graph/bootgraph.py | 12 ++++-----
.../selftests/bpf/test_bpftool_synctypes.py | 26 +++++++++----------
tools/testing/selftests/bpf/test_offload.py | 2 +-
16 files changed, 38 insertions(+), 38 deletions(-)

--
2.41.0

2023-09-12 10:51:00

by Benjamin Gray

[permalink] [raw]

Subject: [PATCH v2 6/7] tools/power: fix Python string escapes

2023-09-12 12:24:28

by Benjamin Gray

[permalink] [raw]

Subject: [PATCH v2 1/7] ia64: fix Python string escapes

2023-09-12 15:19:30

by Benjamin Gray

[permalink] [raw]

Subject: [PATCH v2 5/7] tools/perf: fix Python string escapes

2023-09-12 19:42:02

by Nick Desaulniers

[permalink] [raw]

Subject: Re: [PATCH v2 1/7] ia64: fix Python string escapes

On Mon, Sep 11, 2023 at 11:08 PM Benjamin Gray <[email protected]> wrote:
>
> Python 3.6 introduced a DeprecationWarning for invalid escape sequences.
> This is upgraded to a SyntaxWarning in Python 3.12, and will eventually
> be a syntax error.
>
> Fix these now to get ahead of it before it's an error.
>
> Signed-off-by: Benjamin Gray <[email protected]>
> ---
> arch/ia64/scripts/unwcheck.py | 2 +-

Ard is proposing removing this script, along with the rest of the architecture:
https://lore.kernel.org/linux-arch/[email protected]/

So this change can be dropped.

> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/arch/ia64/scripts/unwcheck.py b/arch/ia64/scripts/unwcheck.py
> index 9581742f0db2..adc24152d3b9 100644
> --- a/arch/ia64/scripts/unwcheck.py
> +++ b/arch/ia64/scripts/unwcheck.py
> @@ -21,7 +21,7 @@ if len(sys.argv) != 2:
>
> readelf = os.getenv("READELF", "readelf")
>
> -start_pattern = re.compile("<([^>]*)>: \[0x([0-9a-f]+)-0x([0-9a-f]+)\]")
> +start_pattern = re.compile(r"<([^>]*)>: \[0x([0-9a-f]+)-0x([0-9a-f]+)\]")
> rlen_pattern = re.compile(".*rlen=([0-9]+)")
>
> def check_func (func, slots, rlen_sum):
> --
> 2.41.0
>

--
Thanks,
~Nick Desaulniers

2023-09-12 23:24:18

by Benjamin Gray

[permalink] [raw]

Subject: [PATCH v2 7/7] selftests/bpf: fix Python string escapes

Python 3.6 introduced a DeprecationWarning for invalid escape sequences.
This is upgraded to a SyntaxWarning in Python 3.12, and will eventually
be a syntax error.

Fix these now to get ahead of it before it's an error.

Signed-off-by: Benjamin Gray <[email protected]>
---
.../selftests/bpf/test_bpftool_synctypes.py | 26 +++++++++----------
tools/testing/selftests/bpf/test_offload.py | 2 +-
2 files changed, 14 insertions(+), 14 deletions(-)

diff --git a/tools/testing/selftests/bpf/test_bpftool_synctypes.py b/tools/testing/selftests/bpf/test_bpftool_synctypes.py
index 0cfece7ff4f8..46862af44f86 100755
--- a/tools/testing/selftests/bpf/test_bpftool_synctypes.py
+++ b/tools/testing/selftests/bpf/test_bpftool_synctypes.py
@@ -66,7 +66,7 @@ class ArrayParser(BlockParser):

def __init__(self, reader, array_name):
self.array_name = array_name
- self.start_marker = re.compile(f'(static )?const bool {self.array_name}\[.*\] = {{\n')
+ self.start_marker = re.compile(rf'(static )?const bool {self.array_name}\[.*\] = {{\n')
super().__init__(reader)

def search_block(self):
@@ -80,7 +80,7 @@ class ArrayParser(BlockParser):
Parse a block and return data as a dictionary. Items to extract must be
on separate lines in the file.
"""
- pattern = re.compile('\[(BPF_\w*)\]\s*= (true|false),?$')
+ pattern = re.compile(r'\[(BPF_\w*)\]\s*= (true|false),?$')
entries = set()
while True:
line = self.reader.readline()
@@ -178,7 +178,7 @@ class FileExtractor(object):
@enum_name: name of the enum to parse
"""
start_marker = re.compile(f'enum {enum_name} {{\n')
- pattern = re.compile('^\s*(BPF_\w+),?(\s+/\*.*\*/)?$')
+ pattern = re.compile(r'^\s*(BPF_\w+),?(\s+/\*.*\*/)?$')
end_marker = re.compile('^};')
parser = BlockParser(self.reader)
parser.search_block(start_marker)
@@ -226,8 +226,8 @@ class FileExtractor(object):

@block_name: name of the blog to parse, 'TYPE' in the example
"""
- start_marker = re.compile(f'\*{block_name}\* := {{')
- pattern = re.compile('\*\*([\w/-]+)\*\*')
+ start_marker = re.compile(rf'\*{block_name}\* := {{')
+ pattern = re.compile(r'\*\*([\w/-]+)\*\*')
end_marker = re.compile('}\n')
return self.__get_description_list(start_marker, pattern, end_marker)

@@ -245,8 +245,8 @@ class FileExtractor(object):

@block_name: name of the blog to parse, 'TYPE' in the example
"""
- start_marker = re.compile(f'"\s*{block_name} := {{')
- pattern = re.compile('([\w/]+) [|}]')
+ start_marker = re.compile(rf'"\s*{block_name} := {{')
+ pattern = re.compile(r'([\w/]+) [|}]')
end_marker = re.compile('}')
return self.__get_description_list(start_marker, pattern, end_marker)

@@ -264,8 +264,8 @@ class FileExtractor(object):

@macro: macro starting the block, 'HELP_SPEC_OPTIONS' in the example
"""
- start_marker = re.compile(f'"\s*{macro}\s*" [|}}]')
- pattern = re.compile('([\w-]+) ?(?:\||}[ }\]])')
+ start_marker = re.compile(rf'"\s*{macro}\s*" [|}}]')
+ pattern = re.compile(r'([\w-]+) ?(?:\||}[ }\]])')
end_marker = re.compile('}\\\\n')
return self.__get_description_list(start_marker, pattern, end_marker)

@@ -284,7 +284,7 @@ class FileExtractor(object):
@block_name: name of the blog to parse, 'TYPE' in the example
"""
start_marker = re.compile(f'local {block_name}=\'')
- pattern = re.compile('(?:.*=\')?([\w/]+)')
+ pattern = re.compile(r"(?:.*=')?([\w/]+)")
end_marker = re.compile('\'$')
return self.__get_description_list(start_marker, pattern, end_marker)

@@ -316,7 +316,7 @@ class MainHeaderFileExtractor(SourceFileExtractor):
{'-p', '-d', '--pretty', '--debug', '--json', '-j'}
"""
start_marker = re.compile(f'"OPTIONS :=')
- pattern = re.compile('([\w-]+) ?(?:\||}[ }\]"])')
+ pattern = re.compile(r'([\w-]+) ?(?:\||}[ }\]"])')
end_marker = re.compile('#define')

parser = InlineListParser(self.reader)
@@ -338,8 +338,8 @@ class ManSubstitutionsExtractor(SourceFileExtractor):

{'-p', '-d', '--pretty', '--debug', '--json', '-j'}
"""
- start_marker = re.compile('\|COMMON_OPTIONS\| replace:: {')
- pattern = re.compile('\*\*([\w/-]+)\*\*')
+ start_marker = re.compile(r'\|COMMON_OPTIONS\| replace:: {')
+ pattern = re.compile(r'\*\*([\w/-]+)\*\*')
end_marker = re.compile('}$')

parser = InlineListParser(self.reader)
diff --git a/tools/testing/selftests/bpf/test_offload.py b/tools/testing/selftests/bpf/test_offload.py
index 40cba8d368d9..fffbc375a7e7 100755
--- a/tools/testing/selftests/bpf/test_offload.py
+++ b/tools/testing/selftests/bpf/test_offload.py
@@ -429,7 +429,7 @@ class NetdevSim:
def __init__(self, nsimdev, port_index, ifname):
# In case udev renamed the netdev to according to new schema,
# check if the name matches the port_index.
- nsimnamere = re.compile("eni\d+np(\d+)")
+ nsimnamere = re.compile(r"eni\d+np(\d+)")
match = nsimnamere.match(ifname)
if match and int(match.groups()[0]) != port_index + 1:
raise Exception("netdevice name mismatches the expected one")
--
2.41.0

2023-09-13 00:04:59

by Benjamin Gray

[permalink] [raw]

Subject: [PATCH v2 3/7] drivers/comedi: fix Python string escapes

2023-09-13 05:45:19

by Benjamin Gray

[permalink] [raw]

Subject: Re: [PATCH v2 5/7] tools/perf: fix Python string escapes

On 12/9/23 8:56 pm, Adrian Hunter wrote:
> On 12/09/23 09:07, Benjamin Gray wrote:
>> diff --git a/tools/perf/pmu-events/jevents.py b/tools/perf/pmu-events/jevents.py
>> index a7e88332276d..980f080a5a2c 100755
>> --- a/tools/perf/pmu-events/jevents.py
>> +++ b/tools/perf/pmu-events/jevents.py
>> @@ -83,7 +83,7 @@ def c_len(s: str) -> int:
>> """Return the length of s a C string
>>
>> This doesn't handle all escape characters properly. It first assumes
>> - all \ are for escaping, it then adjusts as it will have over counted
>> + all \\ are for escaping, it then adjusts as it will have over counted
>
> It looks like the whole string should be a raw string
>
...
>> - s = value.replace("%", "\%")
>> - s = s.replace("_", "\_")
>> + s = value.replace("%", "\\%")
>> + s = s.replace("_", "\\_")
>
> Raw strings seem more readable, so could be
> used here too

Yeah, sounds good. I normally use r strings only for regex, but there
shouldn't be any ambiguity here (it might have been misleading if the
search argument to replace looked like a regex).

Having the docstring be an r string is a good catch. There's probably a
few like that in the kernel, but finding them is a little more
complicated because they might be 'valid' syntax (e.g., the '\000' just
becomes a null byte. This series is focused on the syntax errors though,
so I'll just leave it be.

How is the following?
---
Subject: [PATCH] tools/perf: fix Python string escapes

Python 3.6 introduced a DeprecationWarning for invalid escape sequences.
This is upgraded to a SyntaxWarning in Python 3.12, and will eventually
be a syntax error.

Fix these now to get ahead of it before it's an error.

Signed-off-by: Benjamin Gray <[email protected]>
---
tools/perf/pmu-events/jevents.py | 2 +-
tools/perf/scripts/python/arm-cs-trace-disasm.py | 4 ++--
tools/perf/scripts/python/compaction-times.py | 2 +-
tools/perf/scripts/python/exported-sql-viewer.py | 4 ++--
4 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/tools/perf/pmu-events/jevents.py
b/tools/perf/pmu-events/jevents.py
index a7e88332276d..1b4519333a28 100755
--- a/tools/perf/pmu-events/jevents.py
+++ b/tools/perf/pmu-events/jevents.py
@@ -80,7 +80,7 @@ def file_name_to_table_name(prefix: str, parents:
Sequence[str],

def c_len(s: str) -> int:
- """Return the length of s a C string
+ r"""Return the length of s a C string

This doesn't handle all escape characters properly. It first assumes
all \ are for escaping, it then adjusts as it will have over counted
diff --git a/tools/perf/scripts/python/arm-cs-trace-disasm.py
b/tools/perf/scripts/python/arm-cs-trace-disasm.py
index d59ff53f1d94..de58991c78bb 100755
--- a/tools/perf/scripts/python/arm-cs-trace-disasm.py
+++ b/tools/perf/scripts/python/arm-cs-trace-disasm.py
@@ -45,8 +45,8 @@ parser = OptionParser(option_list=option_list)
# Initialize global dicts and regular expression
disasm_cache = dict()
cpu_data = dict()
-disasm_re = re.compile("^\s*([0-9a-fA-F]+):")
-disasm_func_re = re.compile("^\s*([0-9a-fA-F]+)\s.*:")
+disasm_re = re.compile(r"^\s*([0-9a-fA-F]+):")
+disasm_func_re = re.compile(r"^\s*([0-9a-fA-F]+)\s.*:")
cache_size = 64*1024

glb_source_file_name = None
diff --git a/tools/perf/scripts/python/compaction-times.py
b/tools/perf/scripts/python/compaction-times.py
index 2560a042dc6f..9401f7c14747 100644
--- a/tools/perf/scripts/python/compaction-times.py
+++ b/tools/perf/scripts/python/compaction-times.py
@@ -260,7 +260,7 @@ def pr_help():

comm_re = None
pid_re = None
-pid_regex = "^(\d*)-(\d*)$|^(\d*)$"
+pid_regex = r"^(\d*)-(\d*)$|^(\d*)$"

opt_proc = popt.DISP_DFL
opt_disp = topt.DISP_ALL
diff --git a/tools/perf/scripts/python/exported-sql-viewer.py
b/tools/perf/scripts/python/exported-sql-viewer.py
index 13f2d8a81610..78763531fe5a 100755
--- a/tools/perf/scripts/python/exported-sql-viewer.py
+++ b/tools/perf/scripts/python/exported-sql-viewer.py
@@ -677,8 +677,8 @@ class CallGraphModelBase(TreeModel):
# sqlite supports GLOB (text only) which uses * and ? and is case
sensitive
if not self.glb.dbref.is_sqlite3:
# Escape % and _
- s = value.replace("%", "\%")
- s = s.replace("_", "\_")
+ s = value.replace("%", r"\%")
+ s = s.replace("_", r"\_")
# Translate * and ? into SQL LIKE pattern characters % and _
trans = string.maketrans("*?", "%_")
match = " LIKE '" + str(s).translate(trans) + "'"
--
2.41.0

2023-09-13 05:48:03

by Benjamin Gray

[permalink] [raw]

Subject: [PATCH v2 4/7] scripts: fix Python string escapes

2023-09-13 10:23:56

by Adrian Hunter

[permalink] [raw]

Subject: Re: [PATCH v2 5/7] tools/perf: fix Python string escapes

On 12/09/23 09:07, Benjamin Gray wrote:
> Python 3.6 introduced a DeprecationWarning for invalid escape sequences.
> This is upgraded to a SyntaxWarning in Python 3.12, and will eventually
> be a syntax error.
>
> Fix these now to get ahead of it before it's an error.
>
> Signed-off-by: Benjamin Gray <[email protected]>
> ---
> tools/perf/pmu-events/jevents.py | 2 +-
> tools/perf/scripts/python/arm-cs-trace-disasm.py | 4 ++--
> tools/perf/scripts/python/compaction-times.py | 2 +-
> tools/perf/scripts/python/exported-sql-viewer.py | 4 ++--
> 4 files changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/tools/perf/pmu-events/jevents.py b/tools/perf/pmu-events/jevents.py
> index a7e88332276d..980f080a5a2c 100755
> --- a/tools/perf/pmu-events/jevents.py
> +++ b/tools/perf/pmu-events/jevents.py
> @@ -83,7 +83,7 @@ def c_len(s: str) -> int:
> """Return the length of s a C string
>
> This doesn't handle all escape characters properly. It first assumes
> - all \ are for escaping, it then adjusts as it will have over counted
> + all \\ are for escaping, it then adjusts as it will have over counted

It looks like the whole string should be a raw string

> \\. The code uses \000 rather than \0 as a terminator as an adjacent
> number would be folded into a string of \0 (ie. "\0" + "5" doesn't
> equal a terminator followed by the number 5 but the escape of
> diff --git a/tools/perf/scripts/python/arm-cs-trace-disasm.py b/tools/perf/scripts/python/arm-cs-trace-disasm.py
> index d59ff53f1d94..de58991c78bb 100755
> --- a/tools/perf/scripts/python/arm-cs-trace-disasm.py
> +++ b/tools/perf/scripts/python/arm-cs-trace-disasm.py
> @@ -45,8 +45,8 @@ parser = OptionParser(option_list=option_list)
> # Initialize global dicts and regular expression
> disasm_cache = dict()
> cpu_data = dict()
> -disasm_re = re.compile("^\s*([0-9a-fA-F]+):")
> -disasm_func_re = re.compile("^\s*([0-9a-fA-F]+)\s.*:")
> +disasm_re = re.compile(r"^\s*([0-9a-fA-F]+):")
> +disasm_func_re = re.compile(r"^\s*([0-9a-fA-F]+)\s.*:")
> cache_size = 64*1024
>
> glb_source_file_name = None
> diff --git a/tools/perf/scripts/python/compaction-times.py b/tools/perf/scripts/python/compaction-times.py
> index 2560a042dc6f..9401f7c14747 100644
> --- a/tools/perf/scripts/python/compaction-times.py
> +++ b/tools/perf/scripts/python/compaction-times.py
> @@ -260,7 +260,7 @@ def pr_help():
>
> comm_re = None
> pid_re = None
> -pid_regex = "^(\d*)-(\d*)$|^(\d*)$"
> +pid_regex = r"^(\d*)-(\d*)$|^(\d*)$"
>
> opt_proc = popt.DISP_DFL
> opt_disp = topt.DISP_ALL
> diff --git a/tools/perf/scripts/python/exported-sql-viewer.py b/tools/perf/scripts/python/exported-sql-viewer.py
> index 13f2d8a81610..121cf61ba1b3 100755
> --- a/tools/perf/scripts/python/exported-sql-viewer.py
> +++ b/tools/perf/scripts/python/exported-sql-viewer.py
> @@ -677,8 +677,8 @@ class CallGraphModelBase(TreeModel):
> # sqlite supports GLOB (text only) which uses * and ? and is case sensitive
> if not self.glb.dbref.is_sqlite3:
> # Escape % and _
> - s = value.replace("%", "\%")
> - s = s.replace("_", "\_")
> + s = value.replace("%", "\\%")
> + s = s.replace("_", "\\_")

Raw strings seem more readable, so could be
used here too

> # Translate * and ? into SQL LIKE pattern characters % and _
> trans = string.maketrans("*?", "%_")
> match = " LIKE '" + str(s).translate(trans) + "'"

2023-09-13 22:22:17

by Adrian Hunter

[permalink] [raw]

Subject: Re: [PATCH v2 5/7] tools/perf: fix Python string escapes

On 13/09/23 03:26, Benjamin Gray wrote:
> On 12/9/23 8:56 pm, Adrian Hunter wrote:
>> On 12/09/23 09:07, Benjamin Gray wrote:
>>> diff --git a/tools/perf/pmu-events/jevents.py b/tools/perf/pmu-events/jevents.py
>>> index a7e88332276d..980f080a5a2c 100755
>>> --- a/tools/perf/pmu-events/jevents.py
>>> +++ b/tools/perf/pmu-events/jevents.py
>>> @@ -83,7 +83,7 @@ def c_len(s: str) -> int:
>>>     """Return the length of s a C string
>>>     This doesn't handle all escape characters properly. It first assumes
>>> - all \ are for escaping, it then adjusts as it will have over counted
>>> + all \\ are for escaping, it then adjusts as it will have over counted
>>
>> It looks like the whole string should be a raw string
>>
> ...
>>> -                s = value.replace("%", "\%")
>>> -                s = s.replace("_", "\_")
>>> +                s = value.replace("%", "\\%")
>>> +                s = s.replace("_", "\\_")
>>
>> Raw strings seem more readable, so could be
>> used here too
>
> Yeah, sounds good. I normally use r strings only for regex, but there shouldn't be any ambiguity here (it might have been misleading if the search argument to replace looked like a regex).
>
> Having the docstring be an r string is a good catch. There's probably a few like that in the kernel, but finding them is a little more complicated because they might be 'valid' syntax (e.g., the '\000' just becomes a null byte. This series is focused on the syntax errors though, so I'll just leave it be.
>
> How is the following?
> ---
> Subject: [PATCH] tools/perf: fix Python string escapes
>
> Python 3.6 introduced a DeprecationWarning for invalid escape sequences.
> This is upgraded to a SyntaxWarning in Python 3.12, and will eventually
> be a syntax error.
>
> Fix these now to get ahead of it before it's an error.
>
> Signed-off-by: Benjamin Gray <[email protected]>

Acked-by: Adrian Hunter <[email protected]>

> ---
> tools/perf/pmu-events/jevents.py                 | 2 +-
> tools/perf/scripts/python/arm-cs-trace-disasm.py | 4 ++--
> tools/perf/scripts/python/compaction-times.py    | 2 +-
> tools/perf/scripts/python/exported-sql-viewer.py | 4 ++--
> 4 files changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/tools/perf/pmu-events/jevents.py b/tools/perf/pmu-events/jevents.py
> index a7e88332276d..1b4519333a28 100755
> --- a/tools/perf/pmu-events/jevents.py
> +++ b/tools/perf/pmu-events/jevents.py
> @@ -80,7 +80,7 @@ def file_name_to_table_name(prefix: str, parents: Sequence[str],
>
>
> def c_len(s: str) -> int:
> - """Return the length of s a C string
> + r"""Return the length of s a C string
>
>    This doesn't handle all escape characters properly. It first assumes
>    all \ are for escaping, it then adjusts as it will have over counted
> diff --git a/tools/perf/scripts/python/arm-cs-trace-disasm.py b/tools/perf/scripts/python/arm-cs-trace-disasm.py
> index d59ff53f1d94..de58991c78bb 100755
> --- a/tools/perf/scripts/python/arm-cs-trace-disasm.py
> +++ b/tools/perf/scripts/python/arm-cs-trace-disasm.py
> @@ -45,8 +45,8 @@ parser = OptionParser(option_list=option_list)
> # Initialize global dicts and regular expression
> disasm_cache = dict()
> cpu_data = dict()
> -disasm_re = re.compile("^\s*([0-9a-fA-F]+):")
> -disasm_func_re = re.compile("^\s*([0-9a-fA-F]+)\s.*:")
> +disasm_re = re.compile(r"^\s*([0-9a-fA-F]+):")
> +disasm_func_re = re.compile(r"^\s*([0-9a-fA-F]+)\s.*:")
> cache_size = 64*1024
>
> glb_source_file_name    = None
> diff --git a/tools/perf/scripts/python/compaction-times.py b/tools/perf/scripts/python/compaction-times.py
> index 2560a042dc6f..9401f7c14747 100644
> --- a/tools/perf/scripts/python/compaction-times.py
> +++ b/tools/perf/scripts/python/compaction-times.py
> @@ -260,7 +260,7 @@ def pr_help():
>
> comm_re = None
> pid_re = None
> -pid_regex = "^(\d*)-(\d*)$|^(\d*)$"
> +pid_regex = r"^(\d*)-(\d*)$|^(\d*)$"
>
> opt_proc = popt.DISP_DFL
> opt_disp = topt.DISP_ALL
> diff --git a/tools/perf/scripts/python/exported-sql-viewer.py b/tools/perf/scripts/python/exported-sql-viewer.py
> index 13f2d8a81610..78763531fe5a 100755
> --- a/tools/perf/scripts/python/exported-sql-viewer.py
> +++ b/tools/perf/scripts/python/exported-sql-viewer.py
> @@ -677,8 +677,8 @@ class CallGraphModelBase(TreeModel):
>              #   sqlite supports GLOB (text only) which uses * and ? and is case sensitive
>              if not self.glb.dbref.is_sqlite3:
>                  # Escape % and _
> -                s = value.replace("%", "\%")
> -                s = s.replace("_", "\_")
> +                s = value.replace("%", r"\%")
> +                s = s.replace("_", r"\_")
>                  # Translate * and ? into SQL LIKE pattern characters % and _
>                  trans = string.maketrans("*?", "%_")
>                  match = " LIKE '" + str(s).translate(trans) + "'"

2023-09-17 04:35:54

by Benjamin Gray

[permalink] [raw]

Subject: [PATCH v2 2/7] Documentation/sphinx: fix Python string escapes

Python 3.6 introduced a DeprecationWarning for invalid escape sequences.
This is upgraded to a SyntaxWarning in Python 3.12, and will eventually
be a syntax error.

Fix these now to get ahead of it before it's an error.

Signed-off-by: Benjamin Gray <[email protected]>
---
Documentation/sphinx/cdomain.py | 2 +-
Documentation/sphinx/kernel_abi.py | 2 +-
Documentation/sphinx/kernel_feat.py | 2 +-
Documentation/sphinx/kerneldoc.py | 2 +-
Documentation/sphinx/maintainers_include.py | 8 ++++----
5 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/Documentation/sphinx/cdomain.py b/Documentation/sphinx/cdomain.py
index a99716bf44b5..de5d132d94c5 100644
--- a/Documentation/sphinx/cdomain.py
+++ b/Documentation/sphinx/cdomain.py
@@ -93,7 +93,7 @@ def markup_ctype_refs(match):
#
RE_expr = re.compile(r':c:(expr|texpr):`([^\`]+)`')
def markup_c_expr(match):
- return '\ ``' + match.group(2) + '``\ '
+ return '\\ ``' + match.group(2) + '``\\ '

#
# Parse Sphinx 3.x C markups, replacing them by backward-compatible ones
diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index b5feb5b1d905..49797c55479c 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -138,7 +138,7 @@ class KernelCmd(Directive):
code_block += "\n " + l
lines = code_block + "\n\n"

- line_regex = re.compile("^\.\. LINENO (\S+)\#([0-9]+)$")
+ line_regex = re.compile(r"^\.\. LINENO (\S+)\#([0-9]+)$")
ln = 0
n = 0
f = fname
diff --git a/Documentation/sphinx/kernel_feat.py b/Documentation/sphinx/kernel_feat.py
index 27b701ed3681..b5fa2f0542a5 100644
--- a/Documentation/sphinx/kernel_feat.py
+++ b/Documentation/sphinx/kernel_feat.py
@@ -104,7 +104,7 @@ class KernelFeat(Directive):

lines = self.runCmd(cmd, shell=True, cwd=cwd, env=shell_env)

- line_regex = re.compile("^\.\. FILE (\S+)$")
+ line_regex = re.compile(r"^\.\. FILE (\S+)$")

out_lines = ""

diff --git a/Documentation/sphinx/kerneldoc.py b/Documentation/sphinx/kerneldoc.py
index 9395892c7ba3..8dc134904b90 100644
--- a/Documentation/sphinx/kerneldoc.py
+++ b/Documentation/sphinx/kerneldoc.py
@@ -130,7 +130,7 @@ class KernelDocDirective(Directive):
result = ViewList()

lineoffset = 0;
- line_regex = re.compile("^\.\. LINENO ([0-9]+)$")
+ line_regex = re.compile(r"^\.\. LINENO ([0-9]+)$")
for line in lines:
match = line_regex.search(line)
if match:
diff --git a/Documentation/sphinx/maintainers_include.py b/Documentation/sphinx/maintainers_include.py
index 328b3631a585..dcad0fff4723 100755
--- a/Documentation/sphinx/maintainers_include.py
+++ b/Documentation/sphinx/maintainers_include.py
@@ -77,7 +77,7 @@ class MaintainersInclude(Include):
line = line.rstrip()

# Linkify all non-wildcard refs to ReST files in Documentation/.
- pat = '(Documentation/([^\s\?\*]*)\.rst)'
+ pat = r'(Documentation/([^\s\?\*]*)\.rst)'
m = re.search(pat, line)
if m:
# maintainers.rst is in a subdirectory, so include "../".
@@ -90,11 +90,11 @@ class MaintainersInclude(Include):
output = "| %s" % (line.replace("\\", "\\\\"))
# Look for and record field letter to field name mappings:
# R: Designated *reviewer*: FullName <address@domain>
- m = re.search("\s(\S):\s", line)
+ m = re.search(r"\s(\S):\s", line)
if m:
field_letter = m.group(1)
if field_letter and not field_letter in fields:
- m = re.search("\*([^\*]+)\*", line)
+ m = re.search(r"\*([^\*]+)\*", line)
if m:
fields[field_letter] = m.group(1)
elif subsystems:
@@ -112,7 +112,7 @@ class MaintainersInclude(Include):
field_content = ""

# Collapse whitespace in subsystem name.
- heading = re.sub("\s+", " ", line)
+ heading = re.sub(r"\s+", " ", line)
output = output + "%s\n%s" % (heading, "~" * len(heading))
field_prev = ""
else:
--
2.41.0

2023-11-21 13:14:54

by Arnaldo Carvalho de Melo

[permalink] [raw]

Subject: Re: [PATCH v2 5/7] tools/perf: fix Python string escapes

Em Wed, Sep 13, 2023 at 08:53:26AM +0300, Adrian Hunter escreveu:
> On 13/09/23 03:26, Benjamin Gray wrote:
> > On 12/9/23 8:56 pm, Adrian Hunter wrote:
> >> On 12/09/23 09:07, Benjamin Gray wrote:
> >>> diff --git a/tools/perf/pmu-events/jevents.py b/tools/perf/pmu-events/jevents.py
> >>> index a7e88332276d..980f080a5a2c 100755
> >>> --- a/tools/perf/pmu-events/jevents.py
> >>> +++ b/tools/perf/pmu-events/jevents.py
> >>> @@ -83,7 +83,7 @@ def c_len(s: str) -> int:
> >>> ??? """Return the length of s a C string
> >>> ? ??? This doesn't handle all escape characters properly. It first assumes
> >>> -? all \ are for escaping, it then adjusts as it will have over counted
> >>> +? all \\ are for escaping, it then adjusts as it will have over counted
> >>
> >> It looks like the whole string should be a raw string
> >>
> > ...
> >>> -??????????????? s = value.replace("%", "\%")
> >>> -??????????????? s = s.replace("_", "\_")
> >>> +??????????????? s = value.replace("%", "\\%")
> >>> +??????????????? s = s.replace("_", "\\_")
> >>
> >> Raw strings seem more readable, so could be
> >> used here too
> >
> > Yeah, sounds good. I normally use r strings only for regex, but there shouldn't be any ambiguity here (it might have been misleading if the search argument to replace looked like a regex).
> >
> > Having the docstring be an r string is a good catch. There's probably a few like that in the kernel, but finding them is a little more complicated because they might be 'valid' syntax (e.g., the '\000' just becomes a null byte. This series is focused on the syntax errors though, so I'll just leave it be.
> >
> > How is the following?
> > ---
> > Subject: [PATCH] tools/perf: fix Python string escapes
> >
> > Python 3.6 introduced a DeprecationWarning for invalid escape sequences.
> > This is upgraded to a SyntaxWarning in Python 3.12, and will eventually
> > be a syntax error.
> >
> > Fix these now to get ahead of it before it's an error.
> >
> > Signed-off-by: Benjamin Gray <[email protected]>
>
> Acked-by: Adrian Hunter <[email protected]>

applied the tools/perf one.

- Arnaldo

> > ---
> > ?tools/perf/pmu-events/jevents.py???????????????? | 2 +-
> > ?tools/perf/scripts/python/arm-cs-trace-disasm.py | 4 ++--
> > ?tools/perf/scripts/python/compaction-times.py??? | 2 +-
> > ?tools/perf/scripts/python/exported-sql-viewer.py | 4 ++--
> > ?4 files changed, 6 insertions(+), 6 deletions(-)
> >
> > diff --git a/tools/perf/pmu-events/jevents.py b/tools/perf/pmu-events/jevents.py
> > index a7e88332276d..1b4519333a28 100755
> > --- a/tools/perf/pmu-events/jevents.py
> > +++ b/tools/perf/pmu-events/jevents.py
> > @@ -80,7 +80,7 @@ def file_name_to_table_name(prefix: str, parents: Sequence[str],
> >
> >
> > ?def c_len(s: str) -> int:
> > -? """Return the length of s a C string
> > +? r"""Return the length of s a C string
> >
> > ?? This doesn't handle all escape characters properly. It first assumes
> > ?? all \ are for escaping, it then adjusts as it will have over counted
> > diff --git a/tools/perf/scripts/python/arm-cs-trace-disasm.py b/tools/perf/scripts/python/arm-cs-trace-disasm.py
> > index d59ff53f1d94..de58991c78bb 100755
> > --- a/tools/perf/scripts/python/arm-cs-trace-disasm.py
> > +++ b/tools/perf/scripts/python/arm-cs-trace-disasm.py
> > @@ -45,8 +45,8 @@ parser = OptionParser(option_list=option_list)
> > ?# Initialize global dicts and regular expression
> > ?disasm_cache = dict()
> > ?cpu_data = dict()
> > -disasm_re = re.compile("^\s*([0-9a-fA-F]+):")
> > -disasm_func_re = re.compile("^\s*([0-9a-fA-F]+)\s.*:")
> > +disasm_re = re.compile(r"^\s*([0-9a-fA-F]+):")
> > +disasm_func_re = re.compile(r"^\s*([0-9a-fA-F]+)\s.*:")
> > ?cache_size = 64*1024
> >
> > ?glb_source_file_name??? = None
> > diff --git a/tools/perf/scripts/python/compaction-times.py b/tools/perf/scripts/python/compaction-times.py
> > index 2560a042dc6f..9401f7c14747 100644
> > --- a/tools/perf/scripts/python/compaction-times.py
> > +++ b/tools/perf/scripts/python/compaction-times.py
> > @@ -260,7 +260,7 @@ def pr_help():
> >
> > ?comm_re = None
> > ?pid_re = None
> > -pid_regex = "^(\d*)-(\d*)$|^(\d*)$"
> > +pid_regex = r"^(\d*)-(\d*)$|^(\d*)$"
> >
> > ?opt_proc = popt.DISP_DFL
> > ?opt_disp = topt.DISP_ALL
> > diff --git a/tools/perf/scripts/python/exported-sql-viewer.py b/tools/perf/scripts/python/exported-sql-viewer.py
> > index 13f2d8a81610..78763531fe5a 100755
> > --- a/tools/perf/scripts/python/exported-sql-viewer.py
> > +++ b/tools/perf/scripts/python/exported-sql-viewer.py
> > @@ -677,8 +677,8 @@ class CallGraphModelBase(TreeModel):
> > ???????????? #?? sqlite supports GLOB (text only) which uses * and ? and is case sensitive
> > ???????????? if not self.glb.dbref.is_sqlite3:
> > ???????????????? # Escape % and _
> > -??????????????? s = value.replace("%", "\%")
> > -??????????????? s = s.replace("_", "\_")
> > +??????????????? s = value.replace("%", r"\%")
> > +??????????????? s = s.replace("_", r"\_")
> > ???????????????? # Translate * and ? into SQL LIKE pattern characters % and _
> > ???????????????? trans = string.maketrans("*?", "%_")
> > ???????????????? match = " LIKE '" + str(s).translate(trans) + "'"
>

--

- Arnaldo