Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cancelling jobs could result in "unexpected error: cannot schedule new futures after shutdown" #9307

Open
eivindjahren opened this issue Nov 22, 2024 · 4 comments · May be fixed by #9664
Open

Cancelling jobs could result in "unexpected error: cannot schedule new futures after shutdown" #9307

eivindjahren opened this issue Nov 22, 2024 · 4 comments · May be fixed by #9664
Assignees
@jonathan-eq
Labels
bug

Comments

@eivindjahren
Copy link
Contributor

eivindjahren commented Nov 22, 2024 •
edited
Loading

No reproducing steps yet, so need to investigate. The exception has been observed raised from

ert/src/ert/run_models/base_run_model.py

Line 504 in 71ec78c

await asyncio.get_running_loop().run_in_executor(

@eivindjahren eivindjahren moved this to Todo in SCOUT Nov 22, 2024
@jonathan-eq
Copy link
Contributor

@eivindjahren Are the any links to where this failure occurred, or was it locally?

@jonathan-eq jonathan-eq moved this from Todo to In Progress in SCOUT Nov 25, 2024
@jonathan-eq jonathan-eq self-assigned this Nov 25, 2024
@xjules
Copy link
Contributor

xjules commented Dec 11, 2024 •
edited
Loading

Let's wait if it happens again as this might have been already solved.

@eivindjahren
Copy link
Contributor Author

We have had a couple of more failures of this now :https://equinor.slack.com/archives/C07TJ1KSARX/p1735874865670509

@jonathan-eq jonathan-eq moved this from Todo to In Progress in SCOUT Jan 6, 2025
@jonathan-eq
Copy link
Contributor

I have not been able to reproduce this, but the easiest fix would be to check if the event loop is running before running things in the event loop executor.
It does not fix the root cause of the issue, but it seems to be due to how asyncio schedules tasks internally.

@jonathan-eq jonathan-eq linked a pull request Jan 6, 2025 that will close this issue
Open
9 tasks
@jonathan-eq jonathan-eq moved this from In Progress to Ready for Review in SCOUT Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jonathan-eq jonathan-eq

Labels
bug
Projects
SCOUT
Status: Ready for Review
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make sure event loop is running before running coroutines in executor jonathan-eq/ert
3 participants
@xjules @eivindjahren @jonathan-eq