Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bolt instrumentation missing -update-debug-sections #128437

Closed
zanieb opened this issue Jan 3, 2025 · 0 comments · Fixed by #128455
Closed

Bolt instrumentation missing -update-debug-sections #128437

zanieb opened this issue Jan 3, 2025 · 0 comments · Fixed by #128455
Labels
build The build process and cross-build type-bug An unexpected behavior, bug, or error

Comments

@zanieb
Copy link
Contributor

zanieb commented Jan 3, 2025

Bug report

Bug description:

The following warning is displayed during BOLT instrumentation

BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.

The flag is provided during "apply"

-update-debug-sections

but not during "instrumentation"

BOLT_INSTRUMENT_FLAGS=

Reproduced with

make clean

export CC=clang
export CXX=clang++

./configure py_cv_module__openssl=n/a py_cv_module__hashlib=n/a py_cv_module__gdbm=n/a py_cv_module__tkinter=n/a \
    --without-ensurepip \
    --enable-optimizations --enable-bolt

make -j8
Additional logs
...
make profile-bolt-stamp
make[1]: Entering directory '/big/workspace/cpython'
# Ensure a pristine, pre-BOLT copy of the binary and no profile data from last run.
for bin in python; do \
  prebolt="${bin}.prebolt"; \
  if [ -e "${prebolt}" ]; then \
    echo "Restoring pre-BOLT binary ${prebolt}"; \
    mv "${bin}.prebolt" "${bin}"; \
  fi; \
  cp "${bin}" "${prebolt}"; \
  rm -f ${bin}.bolt.*.fdata ${bin}.fdata; \
done
# Instrument each binary.
for bin in python; do \
  /usr/bin/llvm-bolt "${bin}" -instrument -instrumentation-file-append-pid -instrumentation-file=/big/workspace/cpython/${bin}.bolt -o ${bin}.bolt_inst ; \
  mv "${bin}.bolt_inst" "${bin}"; \
done
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 6a0964d75628b15bafd078342120888c0e6d126f
BOLT-INFO: first alloc address is 0x400000
BOLT-INFO: creating new program header table at address 0xa00000, offset 0x600000
BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling lite mode
BOLT-INFO: 0 out of 7748 functions in the binary (0.0%) have non-empty execution profile
BOLT-INFO: validate-mem-refs updated 2 object references
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 1995
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 7719
BOLT-INSTRUMENTER: Number of function descriptors: 7719
BOLT-INSTRUMENTER: Number of branch counters: 150867
BOLT-INSTRUMENTER: Number of ST leaf node counters: 47195
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 198062
BOLT-INSTRUMENTER: Total size of counters: 1584496 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 166904 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 10927480 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /big/workspace/cpython/python.bolt
BOLT-INFO: 66759 instructions were shortened
BOLT-INFO: removed 84 empty blocks
BOLT-INFO: UCE removed 838 blocks and 51008 bytes of code
BOLT-INFO: padding code to 0x1600000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0x18e0950
BOLT-INFO: clear procedure is 0x18dc390
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting _end to 0x190247c
BOLT-INFO: setting _end to 0x190247c
BOLT-INFO: setting __bolt_runtime_start to 0x18e0900
BOLT-INFO: setting __bolt_runtime_fini to 0x18e0950
BOLT-INFO: setting __hot_start to 0xc00000
BOLT-INFO: setting __hot_end to 0x14b3986
# Run instrumented binaries to collect data.
./python -m test --pgo --timeout=
...

This was patched downstream in python-build-standalone astral-sh/python-build-standalone#463 — I'm interesting in upstreaming that patch.

A simple patch like

diff --git a/configure.ac b/configure.ac
index ee034e5a962..d0245debd5b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2167,7 +2167,7 @@ AC_ARG_VAR(
 AC_MSG_CHECKING([BOLT_INSTRUMENT_FLAGS])
 if test -z "${BOLT_INSTRUMENT_FLAGS}"
 then
-  BOLT_INSTRUMENT_FLAGS=
+  BOLT_INSTRUMENT_FLAGS="-update-debug-sections"
 fi
 AC_MSG_RESULT([$BOLT_INSTRUMENT_FLAGS])

is sufficient — but downstream a BOLT_COMMON_FLAGS variable was added which was useful for subsequent fixes.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

@zanieb zanieb added the type-bug An unexpected behavior, bug, or error label Jan 3, 2025
@picnixz picnixz added the build The build process and cross-build label Jan 3, 2025
corona10 pushed a commit that referenced this issue Jan 5, 2025
…128455)

Add `BOLT_COMMON_FLAGS` with `-update-debug-sections`

Co-authored-by: Gregory Szorc <[email protected]>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jan 5, 2025
pythongh-128455)

Add `BOLT_COMMON_FLAGS` with `-update-debug-sections`

(cherry picked from commit b60044b)

Co-authored-by: Zanie Blue <[email protected]>
Co-authored-by: Gregory Szorc <[email protected]>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jan 5, 2025
pythongh-128455)

Add `BOLT_COMMON_FLAGS` with `-update-debug-sections`

(cherry picked from commit b60044b)

Co-authored-by: Zanie Blue <[email protected]>
Co-authored-by: Gregory Szorc <[email protected]>
corona10 pushed a commit that referenced this issue Jan 5, 2025
…s` (gh-128455) (gh-128512)

gh-128437: Add `BOLT_COMMON_FLAGS` with `-update-debug-sections` (gh-128455)

Add `BOLT_COMMON_FLAGS` with `-update-debug-sections`

(cherry picked from commit b60044b)

Co-authored-by: Zanie Blue <[email protected]>
Co-authored-by: Gregory Szorc <[email protected]>
corona10 pushed a commit that referenced this issue Jan 5, 2025
…s` (gh-128455) (gh-128513)

gh-128437: Add `BOLT_COMMON_FLAGS` with `-update-debug-sections` (gh-128455)

Add `BOLT_COMMON_FLAGS` with `-update-debug-sections`

(cherry picked from commit b60044b)

Co-authored-by: Zanie Blue <[email protected]>
Co-authored-by: Gregory Szorc <[email protected]>
srinivasreddy pushed a commit to srinivasreddy/cpython that referenced this issue Jan 6, 2025
pythongh-128455)

Add `BOLT_COMMON_FLAGS` with `-update-debug-sections`

Co-authored-by: Gregory Szorc <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build The build process and cross-build type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants