profile (libbpf): tool enhacements #5181
Open
+1,041
−53
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There are systems in which more than one source of DSOs exist, so overlap can exist. Think of systems with container or systemd-portabled loads. Those binaries can dynamically link to DSOs having conflicting paths to ones living in the root namespace, but the same-path DSOs might be completely different binary wise. This is a preparation commit for functionality that we want to add later: dumping DSO list state from profile, for offline symbol resolution. When resolving symbols like that, let matches by PID take place too, so the code does not go for bad addresses, on the wrong binaries. Signed-off-by: Gustavo Lima Chaves <[email protected]>
One could end up with no DSO state for those and thus missed flamechart name resolutions. Signed-off-by: Gustavo Lima Chaves <[email protected]>
…tempted Instead, under that mode, more information is output: the DSO list/state is also dumped, besides the stack traces (and counts). The latter also only gets addresses, never symbol names resolved. This is intended for systems where having the debug symbols in their images is prohibitive, but where one still wants to leverage BPF-based profiling. Offline symbol resolution is still possible with this textual output, paired with the system's /proc/kallsyms contents and a serialized version of the known DSO tree. With the new tool mode, all that info is the output, not just the stack traces and counts. With a version of the rootfs where profile ran (with the added missing debug symbols), paired with our added symbol resolution tool (profile_symbol_resolve.py) and the output in the new mode, one will be able to produce flamecharts outside of the running context, in an offline fashion. For flamechart generation, we start with an opinionated take and choose https://github.com/jonhoo/inferno as the leveraged tool. One can easily make that an argument for other choices, after this. NB: no support for folded stack output for the (offline) symbol resolution schema, as that mode does not outputs PIDs. It would be prohibitive to make right DSO matches without that information. NB: Offline VDSO (Virtual Dynamically Shared Object) symbol resolution also not possible. They are generally hardly expected to be bottlenecks anyway, so we don't lose much here. Signed-off-by: Gustavo Lima Chaves <[email protected]>
This is a mode where it runs forever (until SIGINT), but not profiling the system right away. It will only kick in if a given PSI threshold is hit for either CPU/MEM/IO, in the system. That is passed in form of the size (percentage) of a 1s (rolling) window while some process might be stalled, system wide. When that happens, the duration argument is going to be honored (can't be left blank in this mode) and a profiling burst of that length will take place. After that, it will continue to honor new PSI watermark hits, with the same behavior. TLDR: this is a mode for continuous system profiling, but only when the system is under stress. Signed-off-by: Gustavo Lima Chaves <[email protected]>
…t mode Specially for the "PSI loop" mode, outputting to stdout would make it clumsy to organize the info for the different runs again. This makes the tool capable of writing output to files named in the pattern passed to that option (if in a loop mode, timestamp suffixes are added, with local time). Signed-off-by: Gustavo Lima Chaves <[email protected]>
glima
requested review from
drzaeus77,
goldshtn,
yonghong-song,
4ast,
brendangregg and
davemarchevsky
as code owners
ekyooo
reviewed
Jan 8, 2025
| @@ -106,6 +107,7 @@ const char argp_program_doc[] = | |||
| " profile -p 185 # only profile process with PID 185\n" | |||
| " profile -L 185 # only profile thread with TID 185\n" | |||
| " profile -U # only show user space stacks (no kernel)\n" | |||
| " profile -A # only output addresses\n" | |||
There was a problem hiding this comment.
Can this not be achieved using the BPF_F_USER_BUILD_ID flag of the bpf_get_stackid helper function?
There was a problem hiding this comment.
Hey, thanks @ekyooo, let me take a look at that before responding.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This enhances the profile tool with two main tracks: