Nightly feedstock build failed #230

github-actions · 2024-11-20T11:04:54Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

johnkerl · 2024-11-20T13:27:09Z

#231

jdblischak · 2024-11-20T15:24:20Z

Documenting the errors:

# linux-64
* installing *source* package ‘tiledbsoma’ ...
** using staged installation
** updated src/Makevars for system library via pkg-config
** libs
Error: C++20 standard requested but CXX20 is not defined

# osx-64
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/tiledbsoma_types.h:23:
In file included from /Users/runner/miniforge3/conda-bld/tiledbsoma_1732068244867/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include/tiledbsoma/tiledbsoma:51:
/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068244867/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include/tiledbsoma/soma/soma_collection.h:107:32: error: 'path' is unavailable: introduced in macOS 10.15 - see https://conda-forge.org/docs/maintainer/knowledge_base.html#newer-c-features-with-old-sdk
  107 |               std::filesystem::path(uri).filename().string(),  // group name
      |                                ^
/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068244867/_build_env/bin/../include/c++/v1/__filesystem/path.h:383:33: note: 'path' has been explicitly marked unavailable here
  383 | class _LIBCPP_EXPORTED_FROM_ABI path {
      |                                 ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068244867/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/R/etc/Makeconf:200: RcppExports.o] Error 1
ERROR: compilation failed for package ‘tiledbsoma’

# osx-arm64
/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068269621/_build_env/bin/../include/c++/v1/__format/format_functions.h:427:15: note: in instantiation of function template specialization 'std::__vformat_to<std::back_insert_iterator<std::string>, char, std::back_insert_iterator<std::__format::__output_buffer<char>>>' requested here
  427 |   return std::__vformat_to(std::move(__out_it), __fmt, __args);
      |               ^
/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068269621/_build_env/bin/../include/c++/v1/__format/format_functions.h:458:8: note: in instantiation of function template specialization 'std::vformat_to<std::back_insert_iterator<std::string>>' requested here
  458 |   std::vformat_to(std::back_inserter(__res), __fmt, __args);
      |        ^
/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068269621/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/R/library/RcppSpdlog/include/spdl.h:39:86: note: in instantiation of function template specialization 'std::vformat<void>' requested here
   39 |     inline void trace(const char* fmt, Args&&... args ) { RcppSpdlog::log_trace(std::vformat(std::string_view(fmt), std::make_format_args(args...)).c_str()); }
      |                                                                                      ^
./rutilities.h:43:19: note: in instantiation of function template specialization 'spdl::trace<std::string, std::string>' requested here
   43 |             spdl::trace("[config_vector_to_map] adding '{}' = '{}'", std::string(namesvec[i]), std::string(confvec[i]));
      |                   ^
/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068269621/_build_env/bin/../include/c++/v1/__charconv/to_chars_floating_point.h:42:1: note: 'to_chars' has been explicitly marked unavailable here
   42 | to_chars(char* __first, char* __last, long double __value, chars_format __fmt);
      | ^
9 errors generated.
make: *** [/Users/runner/miniforge3/conda-bld/tiledbsoma_1732068269621/_build_env/lib/R/etc/Makeconf:201: RcppExports.o] Error 1
ERROR: compilation failed for package ‘tiledbsoma’

jdblischak · 2024-11-20T21:05:20Z

Unfortunately I ran out of time today. I applied the update ab4c6f5 from #231 to the nightly build on my fork, but unfortunately it failed with the same errors.

github-actions · 2024-11-21T11:16:10Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

jdblischak · 2024-11-21T18:38:34Z

As expected, the R build continues to fail due to C++20 compilation issues.

Today I did another experiment. I purposefully skipped the R client build in order to query the status of the Python build with C++20. Note that this build used the commit single-cell-data/TileDB-SOMA@ea3cd1a that was merged earlier this morning.

First the good news:

C++20 is being used to compile the Python client. Current builds on main (example) use the flag -std=c++17, whereas the nightly build uses -std=c++20
The Python client could be built with C++20 for all architectures (linux-64, osx-64, osx-arm64) and Python versions (3.9, 3.10, 3.11, 3.12)

But then the bad news:

The simple import test fails on both linux-64 and osx-64 (tests aren't run when cross-compiling osx-arm64 on osx-64 architecture).

The backtrace is super long (600+ lines) and only ends because Python core dumps. Below is the beginning and end of the backtrace. Note that I've also replaced the long conda installation path with $PREFIX for readability.

# linux-64 py39

import: 'tiledbsoma'
*** Error in `$PREFIX/bin/python': double free or corruption (out): 0x000055b5a86d6b90 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7f8ced766329]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../../libtiledbsoma.so(+0x647dd)[0x7f8cecc647dd]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../../libtiledbsoma.so(+0x64b0e)[0x7f8cecc64b0e]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../../libtiledbsoma.so(+0x1b9275)[0x7f8cecdb9275]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../../././libspdlog.so.1.14(_ZN6spdlog6logger13set_formatterESt10unique_ptrINS_9formatterESt14default_deleteIS2_EE+0xc5)[0x7f8ceb2ecd95]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../../././libspdlog.so.1.14(_ZN6spdlog6logger11set_patternENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_17pattern_time_typeE+0x4d)[0x7f8ceb2f765d]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../.././libtiledb.so.2.26(+0xc3672a)[0x7f8cec63672a]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../.././libtiledb.so.2.26(+0xc36ae9)[0x7f8cec636ae9]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../.././libtiledb.so.2.26(+0xb27547)[0x7f8cec527547]
$PREFIX/lib/python3.9/site-packages/tiledbsoma/../../.././libtiledb.so.2.26(+0x1c5240)[0x7f8cebbc5240]
/lib64/ld-linux-x86-64.so.2(+0xf9c3)[0x7f8cee3e79c3]
/lib64/ld-linux-x86-64.so.2(+0x1459e)[0x7f8cee3ec59e]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f8cee3e77d4]
/lib64/ld-linux-x86-64.so.2(+0x13b8b)[0x7f8cee3ebb8b]
/lib64/libdl.so.2(+0xfab)[0x7f8cedfb8fab]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f8cee3e77d4]
/lib64/libdl.so.2(+0x15ad)[0x7f8cedfb95ad]
/lib64/libdl.so.2(dlopen+0x31)[0x7f8cedfb9041]
$PREFIX/bin/python(+0x1f220c)[0x55b5a719120c]

# omitted hundreds of lines of similar output

7f8cee5fa000-7f8cee5fb000 rw-p 00022000 08:01 5236909                    /usr/lib64/ld-2.17.so
7f8cee5fb000-7f8cee5fc000 rw-p 00000000 00:00 0 
7ffda295a000-7ffda297c000 rw-p 00000000 00:00 0                          [stack]
7ffda29b4000-7ffda29b8000 r--p 00000000 00:00 0                          [vvar]
7ffda29b8000-7ffda29ba000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]
/home/conda/feedstock_root/build_artifacts/tiledbsoma_1732208694265/test_tmp/conda_test_runner.sh: line 3:  4617 Aborted                 (core dumped) "$PREFIX/bin/python" -s "/home/conda/feedstock_root/build_artifacts/tiledbsoma_1732208694265/test_tmp/run_test.py"
WARNING: Tests failed for tiledbsoma-py-1.15.0rc3.dev20241119-py39h91c51d3_1.conda - moving package to /home/conda/feedstock_root/build_artifacts/broken

jdblischak · 2024-11-21T19:27:13Z

I searched for the error message from the failed R package build on linux-64: Error: C++20 standard requested but CXX20 is not defined

From what I could find, I think we may need to define CXX (and possibly a few other env vars, similar to what we are already doing for the osx-64 build) in ~/.R/Makevars (Shians/NanoMethViz#32 (comment), microsoft/microsoft-r-open#26 (comment))

johnkerl · 2024-11-21T19:35:00Z

@jdblischak I agree this is probably Makevars / Makevars.in ... I'll look soon

jdblischak · 2024-11-21T19:42:07Z

And for tinkering with the feedstock recipe and/or scripts, you can start from the branch nightly-build. It gets overwritten every night, so if you want your experiments to persist beyond a day, I recommend creating a new branch.

johnkerl · 2024-11-22T01:57:55Z

@jdblischak I believe this will be fixed by #231 which addresses precisely this problem.

I'm looking at #231 (comment) and trying to decide what to sequence to make this all work ... 🤔

johnkerl · 2024-11-22T01:58:40Z

Re #231 (comment) I may be confused but I think we just need to merge #231 to address that bit.

johnkerl · 2024-11-22T02:10:20Z

@jdblischak sorry, I need to sync with you tomorrow ...

github-actions · 2024-11-22T11:15:54Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

jdblischak · 2024-11-22T15:38:53Z

I believe this will be fixed by #231 which addresses precisely this problem.

I already tried applying your commit to update build-r-tiledbsoma.sh. That was what I reported above in #230 (comment). Unfortunately it was sufficient to fix the nightly builds.

johnkerl · 2024-11-22T15:42:26Z

OK I'm entirely confused. I'll need to sit down and pencil-and-paper out the repro steps I need to follow to make this happen interactively, debug, and fix. Sorry for the delay.

jdblischak · 2024-11-22T19:16:47Z

Today I setting MACOSX_SDK_VERSION to 13.3, since this is also being explored upstream in single-cell-data/TileDB-SOMA#3361

commit
build

The results:

osx-64: Fixed the R build! And the test section passed too. However, the Python import test still failed as before (see details above in Nightly feedstock build failed #230 (comment))
osx-arm64: The R build still failed with the same error messages: error: 'to_chars' is unavailable: introduced in macOS 13.3. I confirmed that version 13.3 of the SDK was downloaded. Next I'll try setting c_stdlib_version too

jdblischak · 2024-11-22T19:31:00Z

OK I'm entirely confused. I'll need to sit down and pencil-and-paper out the repro steps I need to follow to make this happen interactively, debug, and fix. Sorry for the delay.

I created a dedicated branch for troubleshooting the C++20 compilation errors

https://github.com/TileDB-Inc/tiledbsoma-feedstock/tree/troubleshoot-cpp20

It pulls from the main branch of TileDB-SOMA (to mimic the nightly builds). I applied your fix from #231 and also updated the macOS SDK to 13.3

main...troubleshoot-cpp20

To troubleshoot, you can either push commits directly to this branch (or you can spin off your own branch if you'd prefer). Once we have a working feedstock build, then I'll worry about fixing the setup scripts for the nightly build.

johnkerl · 2024-11-22T19:37:30Z

To troubleshoot

Once we have a working feedstock build

Thanks @jdblischak ! I'll check my notes on how to 'drive' a branch interactively rather than waiting for the nightlies (which is how I usually interact with this repo)

jdblischak · 2024-11-22T20:33:44Z

Next I'll try setting c_stdlib_version too

Quick update. This didn't fix the osx-arm64 build of r-tiledbsoma. And I can tell why now. Despite having conda install version 13.3 of the macOS SDK, the compiler is still passed -mmacosx-version-min=11.0. Thus I think we may need to use wrapper scripts just like we do for osx-64. But unfortunately we'll need to use different wrapper scripts (or use an if/else) because the default conda-forge pin is different for osx-64 (10.3) and osx-arm64 (11.0)

https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/c8f57f58b160967c107a174452b2803dd4505f1e/recipe/conda_build_config.yaml#L21-L22

jdblischak · 2024-11-22T20:35:48Z

Also, another interesting development. After updating the macOS SDK to 13.3, the error message from the failed import test for Python is more informative:

import: 'tiledbsoma'
Traceback (most recent call last):
  File "/Users/runner/miniforge3/conda-bld/tiledbsoma_1732303363846/test_tmp/run_test.py", line 2, in <module>
    import tiledbsoma
  File "$PREFIX/lib/python3.9/site-packages/tiledbsoma/__init__.py", line 166, in <module>
    from ._experiment import Experiment
  File "$PREFIX/lib/python3.9/site-packages/tiledbsoma/_experiment.py", line 18, in <module>
    from ._query import ExperimentAxisQuery
  File "$PREFIX/lib/python3.9/site-packages/tiledbsoma/_query.py", line 48, in <module>
    from somacore.query.query import (
ImportError: cannot import name 'Numpyable' from 'somacore.query.query' ($PREFIX/lib/python3.9/site-packages/somacore/query/query.py)

github-actions · 2024-11-23T11:15:16Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

johnkerl · 2024-11-23T16:38:28Z

@jdblischak re Numpyable:

kerl@arvad[prod][main][SOMA]$ pwd
/Users/kerl/git/single-cell-data/SOMA

kerl@arvad[prod][main][SOMA]$ git diff -r python-v1.0.23 -r python-v1.0.24|grep Numpyable
-            Callable[[_Numpyable], npt.NDArray[np.intp]],
-_Numpyable = Union[pa.Array, pa.ChunkedArray, npt.NDArray[np.int64]]
+Numpyable = Union[pa.Array, pa.ChunkedArray, npt.NDArray[np.int64]]
-    def by_obs(self, coords: _Numpyable) -> npt.NDArray[np.intp]:
+    def by_obs(self, coords: Numpyable) -> npt.NDArray[np.intp]:
-    def by_var(self, coords: _Numpyable) -> npt.NDArray[np.intp]:
+    def by_var(self, coords: Numpyable) -> npt.NDArray[np.intp]:
-def _to_numpy(it: _Numpyable) -> np.ndarray:

So we do need (at least) an update to somacore==1.0.24 ... which is mod I'll push to the troubleshoot-cpp20 branch ...

johnkerl · 2024-11-23T16:48:10Z

Thanks @jdblischak ! I'll check my notes on how to 'drive' a branch interactively rather than waiting for the nightlies (which is how I usually interact with this repo)

Closing the loop on 'how to drive': @jdblischak established in Slack:

Commit and push to the troubleshoot-cpp20 branch
Visit https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary which auto-runs

That's it! I'd failed to realize how much autorun/trigger logic the inestimable John B has set up for this ✨

johnkerl · 2024-11-23T21:15:28Z

Closing the loop on 'how to drive'

Addendum:

Edit locally
./build-locally.py

github-actions · 2024-11-24T11:14:38Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-11-25T11:15:59Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-15T11:04:33Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-16T11:04:55Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-17T11:04:54Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-18T11:04:53Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-19T11:04:57Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-20T11:04:43Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-21T11:04:22Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-22T11:04:26Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

XanthosXanthopoulos · 2024-12-22T22:13:08Z

Good news, locally I had a successful build. Resetting the CXXFLAGS to empty for both libtiledbsoma and libtiledbsoma-r did the trick.

github-actions · 2024-12-23T11:04:51Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

jdblischak · 2024-12-23T18:34:53Z

Good news, locally I had a successful build. Resetting the CXXFLAGS to empty for both libtiledbsoma and libtiledbsoma-r did the trick.

@XanthosXanthopoulos Awesome! Thanks for working on this. It's great to see progress being made.

Looking at your branch xan/cpp20-experiments, it looks like as of 762936e you have successfully compiled r-tiledbsoma for linux-64 and osx-64 (build log). So the remaining to-do items are:

Fix C++20 compilation errors for r-tiledbsoma on osx-arm64
Fix tiledbsoma-py import errors

XanthosXanthopoulos · 2024-12-23T18:56:20Z

Ι will work on tiledbsoma-py today. For macos the min version bump is necessary

github-actions · 2024-12-24T11:04:40Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-25T11:04:37Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-26T11:04:34Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-27T11:04:37Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-28T11:04:30Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-29T11:04:17Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-30T11:04:32Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2024-12-31T11:04:37Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2025-01-01T11:04:33Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2025-01-02T11:04:34Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2025-01-03T11:04:44Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2025-01-04T11:04:29Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2025-01-05T11:04:33Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2025-01-06T11:04:43Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2025-01-07T11:04:40Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2025-01-08T11:04:46Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions · 2025-01-09T11:04:35Z

Nightly feedstock build failure for tiledbsoma-feedstock at https://dev.azure.com/TileDB-Inc/CI/_build?definitionId=43&_a=summary

github-actions bot added the nightly-failure Nightly Azure build failed label Nov 20, 2024

github-actions bot assigned johnkerl and jdblischak Nov 20, 2024

jdblischak mentioned this issue Nov 20, 2024

Build r-tiledbsoma with C++ 20 #231

Open

jdblischak mentioned this issue Nov 21, 2024

[c++/python] Update minimum macOS version to 13.3 single-cell-data/TileDB-SOMA#3361

Closed

johnkerl mentioned this issue Jan 7, 2025

[c++] Use C++20 single-cell-data/TileDB-SOMA#3154

Open

Nightly feedstock build failed #230

Nightly feedstock build failed #230

Comments

github-actions bot commented Nov 20, 2024

johnkerl commented Nov 20, 2024

jdblischak commented Nov 20, 2024

jdblischak commented Nov 20, 2024

github-actions bot commented Nov 21, 2024

jdblischak commented Nov 21, 2024

jdblischak commented Nov 21, 2024

johnkerl commented Nov 21, 2024

jdblischak commented Nov 21, 2024

johnkerl commented Nov 22, 2024

johnkerl commented Nov 22, 2024 • edited Loading

johnkerl commented Nov 22, 2024

github-actions bot commented Nov 22, 2024

jdblischak commented Nov 22, 2024

johnkerl commented Nov 22, 2024

jdblischak commented Nov 22, 2024

jdblischak commented Nov 22, 2024

johnkerl commented Nov 22, 2024

jdblischak commented Nov 22, 2024

jdblischak commented Nov 22, 2024

github-actions bot commented Nov 23, 2024

johnkerl commented Nov 23, 2024 • edited Loading

johnkerl commented Nov 23, 2024

johnkerl commented Nov 23, 2024

github-actions bot commented Nov 24, 2024

github-actions bot commented Nov 25, 2024

github-actions bot commented Dec 15, 2024

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 17, 2024

github-actions bot commented Dec 18, 2024

github-actions bot commented Dec 19, 2024

github-actions bot commented Dec 20, 2024

github-actions bot commented Dec 21, 2024

github-actions bot commented Dec 22, 2024

XanthosXanthopoulos commented Dec 22, 2024

github-actions bot commented Dec 23, 2024

jdblischak commented Dec 23, 2024

XanthosXanthopoulos commented Dec 23, 2024

github-actions bot commented Dec 24, 2024

github-actions bot commented Dec 25, 2024

github-actions bot commented Dec 26, 2024

github-actions bot commented Dec 27, 2024

github-actions bot commented Dec 28, 2024

github-actions bot commented Dec 29, 2024

github-actions bot commented Dec 30, 2024

github-actions bot commented Dec 31, 2024

github-actions bot commented Jan 1, 2025

github-actions bot commented Jan 2, 2025

github-actions bot commented Jan 3, 2025

github-actions bot commented Jan 4, 2025

github-actions bot commented Jan 5, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 7, 2025

github-actions bot commented Jan 8, 2025

github-actions bot commented Jan 9, 2025

johnkerl commented Nov 22, 2024 •

edited

Loading

johnkerl commented Nov 23, 2024 •

edited

Loading