Screener AI Rewrite - Technical Details

We originally created Screener AI using WebSockets. While Django’s WebSockets worked pretty well, they had a few issues:

• The sessions were linked to the browser connection.
  
• If the user’s browser disconnected, the answer generation also stopped in the backend.
  
• This was especially problematic on mobile, where browsers are stopped after 5 seconds if the user switches apps or turns off the screen.
  

The above caused multiple cases where users didn’t receive the answer when they returned to the query they had asked.

That’s when we realised that we needed to decouple the user’s browser connection and the agent fetching the answer in the backend.

Here are the few building building blocks and discoveries that gave us aha moments as we did this entire rewrite.

Django’s StreamingHttpResponse

OpenAI and others use Server-Sent Events (SSE) instead of websockets. SSE uses one-way communication and is therefore much lighter and simpler than WebSockets.

What’s amazing is that Django has an inbuilt StreamingHttpResponse object which can be used for server-side events when served using an ASGI setup. I love the simplicity and ease of using StreamingHttpResponse. We created a generator to stream the contents from a Redis Stream and used it with StreamingHttpResponse. On the frontend, we consumed this endpoint using EventSource. The magic and benefit of EventSource is that it handles the browser reconnects automatically ✌🏽.

Django-VTasks for Background Tasks

Django 6 comes with a Tasks Framework - yay! We decoupled the user request and answer generation by offloading answer generation as a separate background task.

• Simple integration and documentation
  
• Concurrent processing of multiple tasks
  
• Native asyncio for lightweight and high performance
  

Markdown-WASM

LLM output needs to be converted from Markdown to HTML. This conversion needs to be very fast since it needs to be done again and again for the streaming content. It needs to do this conversion around 100 times a minute.

markdown-wasm, a maintained fork of rsms’s version, does this at the WebAssembly level and provides a very high throughput 🫡.

AlpineJS

I love AlpineJS for creating reactive UIs. It’s a library I can read and master in a day. Super lightweight and super simple. I used it for the original version too, but I'm mentioning it again because I love it.

During the rewrite, I discovered the technique of custom directives. I used it to simplify the whole Markdown conversion by adding an x-markdown directive 🕺.

Dj-Evals for Evals

A bit of a self-plug here. We created a simple open-source library, dj-evals, for evaluating and observing the responses of our AI agent.

Dj-Evals allows us to call multiple instances of any function and see their outputs in parallel streams.

The UI allows us to change the inputs for the function and run it again. This makes it so much easier to see and evaluate which AI model does what and how they perform.