Re: [BUG] perf: intel_pt won't display kernel function

From: Jiri Olsa
Date: Thu Apr 04 2019 - 05:24:10 EST

Next message: Greg Kroah-Hartman: "[PATCH 5.0 156/246] iw_cxgb4: fix srqidx leak during connection abort"
Previous message: Greg Kroah-Hartman: "[PATCH 5.0 167/246] e1000e: fix cyclic resets at link up with active tx"
In reply to: Song Liu: "Re: [BUG] perf: intel_pt won't display kernel function"
Next in thread: Jiri Olsa: "Re: [BUG] perf: intel_pt won't display kernel function"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Wed, Apr 03, 2019 at 09:48:23PM +0000, Song Liu wrote:

SNIP

> >>>>> commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503
> >>>>> Author: Arnaldo Carvalho de Melo <acme@xxxxxxxxxx>
> >>>>> Date: Wed Apr 3 12:05:15 2019 -0300
> >>>>>
> >>>>> perf record: Do not synthesize BPF records when --no-bpf-event is used
> >>>>>
> >>>>> By default we synthesize and ask the kernel for BPF events, having a
> >>>>> --no-bpf-event option to disable that, which can be useful, for
> >>>>> instance, if there are still bugs in that code, which seems to be the
> >>>>> case as reported by Jiri Olsa in:
> >>>>>
> >>>>> "[BUG] perf: intel_pt won't display kernel function"
> >>>>> 20190403143738.GB32001@krava">https://lore.kernel.org/lkml/20190403143738.GB32001@krava
> >>>>>
> >>>>> So add the check for record_opts.no_bpf_event when considering
> >>>>> synthesizing BPF events for pre-existing BPF programs in 'perf record'.
> >>>>>
> >>>>> The reported bug needs further analysis and is a separate matter.
> >>>>>
> >>>>> Cc: Adrian Hunter <adrian.hunter@xxxxxxxxx>
> >>>>> Cc: Alexei Starovoitov <ast@xxxxxxxxxx>
> >>>>> Cc: Andrii Nakryiko <andrii.nakryiko@xxxxxxxxx>
> >>>>> Cc: Daniel Borkmann <daniel@xxxxxxxxxxxxx>
> >>>>> Cc: Jiri Olsa <jolsa@xxxxxxxxxx>
> >>>>> Cc: Martin KaFai Lau <kafai@xxxxxx>
> >>>>> Cc: Namhyung Kim <namhyung@xxxxxxxxxx>
> >>>>> Cc: Peter Zijlstra <peterz@xxxxxxxxxxxxx>
> >>>>> Cc: Song Liu <songliubraving@xxxxxx>
> >>>>> Cc: Yonghong Song <yhs@xxxxxx>
> >>>>> Fixes: 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs")
> >>>>> Link: https://lore.kernel.org/lkml/20190403145353.GE32553@xxxxxxxxxx
> >>>>> Signed-off-by: Arnaldo Carvalho de Melo <acme@xxxxxxxxxx>
> >>>>>
> >>>>> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> >>>>> index 4e2d953d4bc5..17d772f192ad 100644
> >>>>> --- a/tools/perf/builtin-record.c
> >>>>> +++ b/tools/perf/builtin-record.c
> >>>>> @@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail)
> >>>>> return err;
> >>>>> }
> >>>>>
> >>>>> - err = perf_event__synthesize_bpf_events(session, process_synthesized_event,
> >>>>> - machine, opts);
> >>>>> - if (err < 0)
> >>>>> - pr_warning("Couldn't synthesize bpf events.\n");
> >>>>> + if (!opts->no_bpf_event) {
> >>>>> + err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts);
> >>>>> + if (err < 0)
> >>>>> + pr_warning("Couldn't synthesize bpf events.\n");
> >>>>> + }
> >>>>>
> >>>>> err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads,
> >>>>> process_synthesized_event, opts->sample_address,
> >>>
> >>> --
> >>>
> >>> - Arnaldo
>
> Yet another version, with commit message and a little more logic:
>
>
> From 8c842b52905e747487447a014cef8c98d0db94ef Mon Sep 17 00:00:00 2001
> From: Song Liu <songliubraving@xxxxxx>
> Date: Wed, 3 Apr 2019 14:24:02 -0700
> Subject: [PATCH] perf util: check maps for bpf programs
>
> As reported by Jiri Olsa in:
>
> "[BUG] perf: intel_pt won't display kernel function"
> 20190403143738.GB32001@krava">https://lore.kernel.org/lkml/20190403143738.GB32001@krava
>
> Recent changes to support PERF_RECORD_KSYMBOL and PERF_RECORD_BPF_EVENT
> broke --kallsyms option. This is because it broke test __map__is_kmodule.
>
> This patch fixes this by adding check for bpf program, so that these maps
> are not mistaken as kernel modules.
>
> Reported-by: Jiri Olsa <jolsa@xxxxxxxxxx>
> Cc: Adrian Hunter <adrian.hunter@xxxxxxxxx>
> Cc: Alexei Starovoitov <ast@xxxxxxxxxx>
> Cc: Andrii Nakryiko <andrii.nakryiko@xxxxxxxxx>
> Cc: Daniel Borkmann <daniel@xxxxxxxxxxxxx>
> Cc: Martin KaFai Lau <kafai@xxxxxx>
> Cc: Namhyung Kim <namhyung@xxxxxxxxxx>
> Cc: Peter Zijlstra <peterz@xxxxxxxxxxxxx>
> Cc: Song Liu <songliubraving@xxxxxx>
> Cc: Yonghong Song <yhs@xxxxxx>
> Cc: Arnaldo Carvalho de Melo <acme@xxxxxxxxxx>
> Fixes: 76193a94522f ("perf, bpf: Introduce PERF_RECORD_KSYMBOL")
> Link: https://lore.kernel.org/lkml/20190403145353.GE32553@xxxxxxxxxx
>
> Signed-off-by: Song Liu <songliubraving@xxxxxx>
> ---
> tools/perf/util/map.c | 16 ++++++++++++++++
> tools/perf/util/map.h | 4 +++-
> 2 files changed, 19 insertions(+), 1 deletion(-)
>
> diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c
> index e32628cd20a7..28d484ef74ae 100644
> --- a/tools/perf/util/map.c
> +++ b/tools/perf/util/map.c
> @@ -261,6 +261,22 @@ bool __map__is_extra_kernel_map(const struct map *map)
> return kmap && kmap->name[0];
> }
>
> +bool __map__is_bpf_prog(const struct map *map)
> +{
> + const char *name;
> +
> + if (map->dso->binary_type == DSO_BINARY_TYPE__BPF_PROG_INFO)
> + return true;
> +
> + /*
> + * If PERF_RECORD_BPF_EVENT is not included, the dso will not have
> + * type of DSO_BINARY_TYPE__BPF_PROG_INFO. In such cases, we can
> + * guess the type based on name.
> + */
> + name = map->dso->short_name;
> + return name && (strstr(name, "bpf_prog_") == name);
> +}
> +
> bool map__has_symbols(const struct map *map)
> {
> return dso__has_symbols(map->dso);
> diff --git a/tools/perf/util/map.h b/tools/perf/util/map.h
> index 0e20749f2c55..01079f1f4375 100644
> --- a/tools/perf/util/map.h
> +++ b/tools/perf/util/map.h
> @@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name,
>
> bool __map__is_kernel(const struct map *map);
> bool __map__is_extra_kernel_map(const struct map *map);
> +bool __map__is_bpf_prog(const struct map *map);
>
> static inline bool __map__is_kmodule(const struct map *map)
> {
> - return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map);
> + return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) &&
> + !__map__is_bpf_prog(map);
> }
>
> bool map__has_symbols(const struct map *map);

it shows the kernel portion now, but it segfaults on me
in ksymbol map processing

recorded data with:

# perf-with-kcore record pt -e intel_pt// -- sleep 1

gdb session:

(gdb) r script -i pt/perf.data --kallsyms=./pt/kcore_dir/kallsyms
Starting program: /home/jolsa/bin/perf script -i pt/perf.data --kallsyms=./pt/kcore_dir/kallsyms

...

Program received signal SIGABRT, Aborted.
0x00007ffff75e60f5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.6-29.fc30.x86_64 elfutils-libelf-0.176-1.fc30.x86_64 elfutils-libs-0.176-1.fc30.x86_64 libgcc-9.0.1-0.10.fc30.x86_64 libunwind-1.3.1-2.fc30.x86_64 libxcrypt-4.4.4-2.fc30.x86_64 numactl-libs-2.0.12-2.fc30.x86_64 perl-libs-5.28.1-434.fc30.x86_64 python2-libs-2.7.16-1.fc30.x86_64 slang-2.3.2-5.fc30.x86_64 xz-libs-5.2.4-5.fc30.x86_64 zlib-1.2.11-15.fc30.x86_64
(gdb) bt
#0 0x00007ffff75e60f5 in raise () from /lib64/libc.so.6
#1 0x00007ffff75d0895 in abort () from /lib64/libc.so.6
#2 0x00007ffff75d0769 in __assert_fail_base.cold () from /lib64/libc.so.6
#3 0x00007ffff75de596 in __assert_fail () from /lib64/libc.so.6
#4 0x00000000004fc006 in refcount_sub_and_test (i=1, r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:131
#5 refcount_dec_and_test (r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:148
#6 map__put (map=0x1224df0) at util/map.c:299
#7 0x00000000004fdb95 in __maps__remove (map=0x1224df0, maps=0xb17d80) at util/map.c:953
#8 maps__remove (maps=0xb17d80, map=0x1224df0) at util/map.c:959
#9 0x00000000004f7d8a in map_groups__remove (map=<optimized out>, mg=<optimized out>) at util/map_groups.h:65
#10 machine__process_ksymbol_unregister (sample=<optimized out>, event=0x7ffff7279670, machine=<optimized out>) at util/machine.c:728
#11 machine__process_ksymbol (machine=<optimized out>, event=0x7ffff7279670, sample=<optimized out>) at util/machine.c:741
#12 0x00000000004fffbb in perf_session__deliver_event (session=0xb11390, event=0x7ffff7279670, tool=0x7fffffffc7b0, file_offset=13936) at util/session.c:1362
#13 0x00000000005039bb in do_flush (show_progress=false, oe=0xb17e80) at util/ordered-events.c:243
#14 __ordered_events__flush (oe=0xb17e80, how=OE_FLUSH__ROUND, timestamp=<optimized out>) at util/ordered-events.c:322
#15 0x00000000005005e4 in perf_session__process_user_event (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8,
file_offset=file_offset@entry=191224) at util/session.c:1402
#16 0x000000000050091e in perf_session__process_event (file_offset=191224, event=0x7ffff72a4af8, session=0xb11390) at util/session.c:1529
#17 process_simple (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8, file_offset=file_offset@entry=191224) at util/session.c:1962
#18 0x0000000000501f44 in reader__process_events (prog=0x7fffffffc680, session=0xb11390, rd=<synthetic pointer>) at util/session.c:1931
#19 __perf_session__process_events (session=0xb11390) at util/session.c:1985
#20 perf_session__process_events (session=0xb11390) at util/session.c:2018
#21 0x000000000045bb42 in __cmd_script (script=0x7fffffffc7b0) at builtin-script.c:2429
#22 cmd_script (argc=<optimized out>, argv=<optimized out>) at builtin-script.c:3770
#23 0x00000000004a65cb in run_builtin (p=p@entry=0x994918 <commands+408>, argc=argc@entry=4, argv=argv@entry=0x7fffffffe160) at perf.c:303
#24 0x000000000042da1e in handle_internal_command (argv=0x7fffffffe160, argc=4) at perf.c:355
#25 run_argv (argcp=<synthetic pointer>, argv=<synthetic pointer>) at perf.c:399
#26 main (argc=4, argv=0x7fffffffe160) at perf.c:521

jirka

Next message: Greg Kroah-Hartman: "[PATCH 5.0 156/246] iw_cxgb4: fix srqidx leak during connection abort"
Previous message: Greg Kroah-Hartman: "[PATCH 5.0 167/246] e1000e: fix cyclic resets at link up with active tx"
In reply to: Song Liu: "Re: [BUG] perf: intel_pt won't display kernel function"
Next in thread: Jiri Olsa: "Re: [BUG] perf: intel_pt won't display kernel function"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]