Healthcheck for websockets

[sevdog]

Problem definition

I was looking for a way to handle session expiration or invalidation (ie: the user changed password). I tried to figure out how this is handled in other frameworks.

Sprint (Java)

In Spring the HTTP session can be used also in websocket connections and the session engine handles by itself expiration or logoff via an event-driven mechanism. The websocket can be setup to intercept these events to close connections server-side.

Ref: https://docs.spring.io/spring-session/reference/guides/boot-websocket.html

Socket.io (JavaScript)

socket.io does not handle the session by itself, relying on other libraries which implements the HTTP application server to handle it. However they suggest to use a setInterval (aka: periodic task) to check for session expiry.

Ref: https://socket.io/how-to/use-with-express-session#handling-session-expiration

ASP.Net (C#)

Microsoft has defined an authentication layer which can be configured to "enable authentication expiration tracking which will close connections when a token expires".

Ref: https://learn.microsoft.com/en-us/aspnet/core/signalr/configuration?view=aspnetcore-7.0&tabs=dotnet#advanced-http-configuration-options

Possible solution

Based on those solutions I belive that it may be useful to have a kind of healthcheck which can be configured for websocket consumers.

The healthcheck should be a periodic task with the following specs:

• the interval/frequency should be configurable
• it should be launched only after websocket connection gets accepted (websocket.accept)
• it must be canceled when the websocket connection is stopped (websocket.close and websocket.disconnect)
• it should be able to close the websocket (websocket.close)
• it should be an optional method of the consumer (to have access to scope and other variables)

sequenceDiagram
    client ->>+ wsconsumer: websocket.connect
    wsconsumer ->> client: websocket.accept
    wsconsumer ->>+ healthcheck: 
    par 
        loop communication
            client --) wsconsumer: websocket.receive
            wsconsumer --) client: websocket.send
        end
    and 
        loop healtcheck
            healthcheck --) wsconsumer: websocket.healthcheck
        end
    end

    deactivate healthcheck
    wsconsumer ->>- client: websocket.close

Loading

Implementation

This package (which I belive is based on this SO suggestion) which defines a class which should handle a periodic task with asyncio, the problem is that it is not tested on any supported python version but 3.7. It may be forked/derived to have this kind of support in channels.

The AsyncWebsocketConsumer/WebsocketConsumer may then be updated to perform the above methioned checks in the accept, close and websocket_disconnect methods only if an healthcheck is defined.

[carltongibson]

Hi @sevdog — thanks for this.

My initial leaning would be to something like a the consumer setting up a task to periodically run get_user(), ending the connection if/when it's no longer valid. That might be something we could begin quickly by adding as an example to the authentication docs. 🤔

I'm not sure I'm seeing the Why for the Periodic like take on that.

[sevdog]

Hi @carltongibson.

I tried to figure out if there was in asyncio something to manage a non-blocking periodic-task. If there was not any constraint the following may work:

async def periodic(time):
    while True:
        # do stuff
        ...
        await asyncion.sleep(time)

But calling this in .accept would have the callee waiting forever. In my mind this may not be an optimal implementation, since it does not "detach" from the connect flow. I do not believe that something like this is going to work well.

In an other scenario I would have used something different (ie: celery), but in this case I see these constraints:

• to have access to consumer's scope (and to the consumer itself)
• to do not interfere with the dispatch mechanism of consumers by having an handler running indefinitely
• to be able to cancel the periodic check if the consumer has to stop (ie: on disconnect, close or if StopConsumer is raised)

So I belive that it would fit better if this kind of task is handled by something "close to" the consumer, also to have an implementation which does not requires an extra dependency which is not in the scope of the current library.

For what I was able to find there is nothing in asyncio which works like the JS function setInterval: triggering a "scheduled task" in the current thread while not bloking (neither with an await) the current flow, which also provides an easy API for canceling. To be more clear what I mean is that there is no basic support for a simple "launch and forget" mechanism in asyncio as of now. Thus the Periodic class was taken as an example setup for such a use case.

I also think that to have this work within channels the mechanism should not give any problem to the AsyncConsumer.__call__ method.
Because we cannot leave any useless task running in the background, maybe to help this kind of feature it could be useful to add also a "cleanup" hook for the consumer to be called before it stops to clean-up any side effect (ie: detached tasks launched inside an handler which cannot be handled in the await_many_dispatch method).

Maybe the same mechanism may be achieved by using a dedicated consumer to handle session related validation and which then will send specific messages to the Websocket consumer, but this seems too far complex for what I want to achieve (also it would require further logic to handle what is going to be an other application to handle session...).

If anyone has a better solution I will be glad to listen and discuss.

[sevdog]

At the moment the simpliest way which somes in my mind with create_task is this: sevdog@931ba60

Obviously its a work-in-progress, tests are still to be defined.

With this implementation the healthcheck async method would in charge to handle re-scheduling according to given interval.

[sevdog]

I have setup a basic implementation with a couple of tests sevdog@65e5d85

The problem now is that when I run tests they succeed but I get a couple of errors from asyncio:

tests/security/test_sessions.py::test_websocket_check_session_password_change PASSED                                                                                                                        [100%]

======================================================================================== 1 passed, 64 deselected in 2.25s =========================================================================================
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<CookieMiddleware.__call__() running at ./venv/channels/sessions.py:47> wait_for=<Task pending name='Task-5' coro=<Queue.get() running at /usr/lib/python3.10/asyncio/queues.py:159> wait_for=<Future cancelled> cb=[Task.task_wakeup()]>>
Exception ignored in: <coroutine object CookieMiddleware.__call__ at 0x7fbe3e8176f0>
Traceback (most recent call last):
  File "./venv/channels/sessions.py", line 47, in __call__
    return await self.inner(dict(scope, cookies=cookies), receive, send)
  File "./venv/channels/sessions.py", line 263, in __call__
    return await self.inner(wrapper.scope, receive, wrapper.send)
  File "./venv/channels/security/sessions.py", line 22, in __call__
    self.check_task.cancel()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Task was destroyed but it is pending!
task: <Task pending name='Task-5' coro=<Queue.get() running at /usr/lib/python3.10/asyncio/queues.py:159> wait_for=<Future cancelled> cb=[Task.task_wakeup()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-4' coro=<SessionCheckAsyncWebsocketConsumer._run_session_check() running at ./venv/channels/security/sessions.py:37> wait_for=<Future cancelled>>

I will investigate about this to understand what is bothering the loop.

@carltongibson do you ever had this kind of errors with channels or other async code?

[carltongibson]

These are shutdown cleanup errors. You need to clean up your task before exiting.

[sevdog]

Ok, I got where the error was laying:

• I forgot not calling consumer.disconnect() in unittests
• the __call__ method was being called after the event loop was closed

I need to finish a check about session expiry before putting the corresponding test.

[sevdog]

I just added tests to handle session expiry.

As of now it is a bit hacky since it implements a forced session reload, since there is no public-facing API in django.contrib.sessions nor in channels.sessions to reload an active session.

https://github.com/sevdog/channels/tree/websocket-heatlhcheck