Memory leak with anonymous sub scoped outside another anonymous sub

[trizen]

Cc: builduser
To: [email protected]
Message-Id: 5.40.0_37134_1736241991@ARCH
Subject: Memory leak with anonymous sub scoped outside another anonymous sub
From: [email protected]
Reply-To: [email protected]

This is a bug report for perl from [email protected],
generated with the help of perlbug 1.43 running under perl 5.40.0.

---

Description

There is a memory leak (illustrated in the code below), which can be reproduced with perl-5.40.0 and perl-5.41.7, but CANNOT be reproduced with old enough versions of Perl.

Running under perl-5.40.0, the code below, ends up using more than 1GB of RAM:

Running under perl-5.22.3, RAM usage stays stable at ~100MB:

Steps to Reproduce

# Leaks memory with perl >= 5.40.0
# No memory leak with perl 5.22.3

foreach my $i (1 .. 1e6) {

    print "$i\n";

    my $t;
    my $block;

    sub {
       $block = sub { $i * $_[0] };
       $t     = [map { $block->($_) } 1 .. 1e6];
    }->();
}

Expected behavior

The $block and $t variables should be destroyed after each loop iteration.

---

Flags

• category=core
• severity=medium

---

Perl configuration

Site configuration information for perl 5.40.0:

Configured by builduser at Sun Sep  1 11:21:17 UTC 2024.

Summary of my perl5 (revision 5 version 40 subversion 0) configuration:

  Platform:
    osname=linux
    osvers=5.12.15-arch1-1
    archname=x86_64-linux-thread-multi
    uname='archlinux'
    config_args='-des -Dusethreads -Duseshrplib -Doptimize=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions     -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security     -fstack-clash-protection -fcf-protection     -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/perl/src=/usr/src/debug/perl -flto=auto -Dprefix=/usr -Dvendorprefix=/usr -Dprivlib=/usr/share/perl5/core_perl -Darchlib=/usr/lib/perl5/5.40/core_perl -Dsitelib=/usr/share/perl5/site_perl -Dsitearch=/usr/lib/perl5/5.40/site_perl -Dvendorlib=/usr/share/perl5/vendor_perl -Dvendorarch=/usr/lib/perl5/5.40/vendor_perl -Dscriptdir=/usr/bin/core_perl -Dsitescript=/usr/bin/site_perl -Dvendorscript=/usr/bin/vendor_perl -Dinc_version_list=none -Dman1ext=1perl -Dman3ext=3perl -Dlddlflags=-shared -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now
-Wl,-z,pack-relative-relocs -flto=auto -Dldflags=-Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now      -Wl,-z,pack-relative-relocs -flto=auto -Dloclibpth=/usr/lib/db5.3 -Dlocincpth=/usr/include/db5.3'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
  Compiler:
    cc='cc'
    ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/include/db5.3 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    optimize='-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/perl/src=/usr/src/debug/perl -flto=auto'
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/include/db5.3'
    ccversion=''
    gccversion='14.2.1 20240805'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags ='-Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-z,pack-relative-relocs -flto=auto -fstack-protector-strong -L/usr/lib/db5.3'
    libpth=/usr/local/lib /usr/lib /usr/lib/db5.3
    libs=-lpthread -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lpthread -ldl -lm -lcrypt -lutil -lc
    libc=/lib/../lib/libc.so.6
    so=so
    useshrplib=true
    libperl=libperl.so
    gnulibc_version='2.40'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.40/core_perl/CORE'
    cccdlflags='-fPIC'
    lddlflags='-shared -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-z,pack-relative-relocs -flto=auto -L/usr/lib/db5.3 -fstack-protector-strong'


---
@INC for perl 5.40.0:
    /usr/lib/perl5/5.40/site_perl
    /usr/share/perl5/site_perl
    /usr/lib/perl5/5.40/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib/perl5/5.40/core_perl
    /usr/share/perl5/core_perl

---
Environment for perl 5.40.0:
    HOME=/home/swampyx
    LANG=en_US.UTF-8
    LANGUAGE=
    LC_ADDRESS=
    LC_COLLATE=
    LC_CTYPE=
    LC_IDENTIFICATION=
    LC_MEASUREMENT=
    LC_MESSAGES=
    LC_MONETARY=
    LC_NAME=
    LC_NUMERIC=
    LC_PAPER=
    LC_TELEPHONE=
    LC_TIME=
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/lib/ccache/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/android-sdk/platform-tools:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/share/perl6/vendor/bin:/home/swampyx/.raku/bin:/opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin
    PERL_BADLANG (unset)
    PERL_MM_OPT=OPTIMIZE="-march=native -O3 -pipe -fno-plt -fstack-clash-protection -fcf-protection"
    SHELL=/usr/bin/zsh

[mauke]

Based on vibes, this may be a duplicate of #22547.

[jkeenan]

I modified the OP's program, mostly to add debugging statements:

$ cat xgh-22893-leak.pl 
# Leaks memory with perl >= 5.40.0
# No memory leak with perl 5.22.3

my $j = 1e5;
print (join '|' => ("$]", $j)), "\n";
print "\n";

my $k = int($j / 10);
foreach my $i (1 .. $j) {

    $i % $k ? '' : print "$i\n";

    my $t;
    my $block;

    sub {
       $block = sub { $i * $_[0] };
       $t     = [map { $block->($_) } 1 .. $j];
    }->();
}
print (join '|' => ("$]", $j)), "\n";

I then used this program to bisect with this invocation:

$ perl Porting/bisect.pl --start=v5.38.0 --end=v5.40.0 \
-- ./perl -Ilib /tmp/xgh-22893-leak.pl

More than 4 hours later, I got this result:

386907f061c1812ecaa5f3c88d9f729828408097 is the first bad commit
commit 386907f061c1812ecaa5f3c88d9f729828408097
Author: Tony Cook <[email protected]>
Date:   Thu Sep 14 10:37:42 2023 +1000
Commit:     Tony Cook <[email protected]>
CommitDate: Mon Sep 25 10:24:07 2023 +1000

    Revert "[perl #89544] Non-eval closures don’t need CvOUTSIDE"
    
    But they do need CvOUTSIDE() for eval-in-package-DB's scope
    magic to work correctly.
    
    This also incidentally fixes a TODO Deparse, since the CvOUTSIDE
    is now available
    
    Fixes #19370 aka rt89544
    
    This breaks #11286, but I don't think that's fixable without breaking
    eval-in-DB's special scope behaviour.

    This reverts (most of) commit a0d2bbd5c47035a4f7369e4fddd46b502764d86e.
    
 cv.h                    | 6 +-----
 dump.c                  | 1 -
 ext/Devel-Peek/t/Peek.t | 8 ++++----
 lib/B/Deparse.t         | 1 -
 pad.c                   | 4 +---
 t/op/closure.t          | 3 ++-
 t/op/eval.t             | 1 -
 7 files changed, 8 insertions(+), 16 deletions(-)
bisect found first bad commit
That took 15437 seconds.

@tonycoz, can you take a look? Thanks.

[tonycoz]

Should be fixed by 9059509

[jkeenan]

Should be fixed by 9059509

Confirmed.

$ ./bin/perl -Ilib -v | head -2 | tail -1
This is perl 5, version 41, subversion 8 (v5.41.8 (v5.41.7-22-g61bacb933d)) built for x86_64-linux

$ time ./bin/perl -Ilib $P5P_DIR/xgh-22893-leak.pl
5.041008|100000
10000
20000
30000
40000
50000
60000
70000
80000
90000
100000
5.041008|100000
real	15m59.892s
user	15m59.780s
sys	0m0.026s