Django

Serving a project of Django with the servers of Pyjoyment.

This document describes how a project of Django, as it is, runs on the servers of Pyjoyment in the place of gunicorn or uvicorn: its WSGI or its ASGI application mounted on / of an application of Pyjoyment.Lite with Pyjoyment.Plugin.Mount, with the static files, the probes and the access log of Pyjoyment around it. From gunicorn translates the options of gunicorn into the ones of Pyjoyment.Command.prefork.

The project

The project of this guide has the layout of django-admin startproject, with a model of notes in SQLite and a /healthz of its own, and one file more, config/server.py next to config/wsgi.py, the script of the servers of Pyjoyment.

config/server.py
import Pyjo.IOLoop

from Pyjoyment.Lite import *

import os

from django.conf import settings
from django.core.signals import request_finished, request_started
from django.db import connection


os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings')

# The static files of collectstatic, served before the routes, with the hashed names of ManifestStaticFilesStorage
# cached for a year
app.static.paths = [str(settings.STATIC_ROOT.parent)]
app.static.immutable = r'^static/.+\.[0-9a-f]{12}\.[^/.]+$'
app.static.cache_control = 'public, max-age=60'


# The database, in a request of Django, whose handlers close a connection that the database has dropped
def database():
    request_started.send(sender=__name__)
    try:
        with connection.cursor() as cursor:
            cursor.execute('SELECT 1')
    finally:
        request_finished.send(sender=__name__)


# /livez and /readyz, with the check of the database as blocking code, and the /healthz of the project, which stays its
# own
plugin('Health', checks={'database': lambda c: Pyjo.IOLoop.blocking(database)}, healthz=None)

# The access log of gunicorn, without the probes
plugin('AccessLog')

# The project, which gets every request that the routes before do not match
plugin('Mount', {'/': 'config.wsgi:application'})

if __name__ == '__main__':
    app.start()
config/settings.py
import os

from pathlib import Path


BASE_DIR = Path(__file__).resolve().parent.parent

SECRET_KEY = os.environ.get('DJANGO_SECRET_KEY', 'insecure')
DEBUG = os.environ.get('DJANGO_DEBUG') == '1'
# The kubelet sends the address of the pod as Host, which the probes of Pyjoyment take before Django
ALLOWED_HOSTS = os.environ.get('DJANGO_ALLOWED_HOSTS', 'localhost,127.0.0.1').split(',')

INSTALLED_APPS = ['django.contrib.staticfiles', 'notes']
MIDDLEWARE = ['django.middleware.security.SecurityMiddleware', 'django.middleware.common.CommonMiddleware']
ROOT_URLCONF = 'config.urls'

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.environ.get('DJANGO_DATABASE', BASE_DIR / 'db.sqlite3'),
    },
}
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'

# The time zone of the process, which Django sets in TZ, so the logs of Pyjoyment have it too
TIME_ZONE = 'UTC'
USE_TZ = True

# The output of collectstatic, which Pyjoyment serves from the parent directory, with favicon.ico and robots.txt
STATIC_URL = '/static/'
STATIC_ROOT = BASE_DIR / 'public' / 'static'

The script is a module of the project, run from the directory of manage.py, as manage.py imports the packages of the project from there, and it starts the server under if __name__ == '__main__', so that a test can import it.

$ python manage.py migrate
$ python manage.py collectstatic
$ python -m config.server prefork -w 8 -c 1 -r 1

The event loop

A WSGI application blocks, and runs in the event loop of its worker, which waits for it. Pyjo.App.WSGI calls it with Pyjo.IOLoop.blocking(), so that it sees no running loop of asyncio while it runs, as on a WSGI server, where Django would refuse its database otherwise, as every query would raise SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async., and a blocking request of Pyjo.UserAgent would raise too. The code of Django that runs outside the application, as the check of the database of the probes, is blocking code too, which Pyjo.IOLoop.blocking() runs.

The project runs on Pyjo.Reactor.Asyncio, the default reactor, next to what needs asyncio, as an ASGI application mounted on another path, and on Pyjo.Reactor.Poll as well, which PYJO_REACTOR selects, except that an async def view, which Django runs in a thread with an event loop of its own, awaits the promises of Pyjo.UserAgent only with the reactor of asyncio.

What passes through

The request comes to Django as a server of WSGI gives it:

  • SCRIPT_NAME is empty on /, so the URLs that Django builds need no FORCE_SCRIPT_NAME;

  • the headers are the HTTP_ variables of the environment, as X-Forwarded-Proto is HTTP_X_FORWARDED_PROTO, which SECURE_PROXY_SSL_HEADER reads;

  • REMOTE_ADDR is the address of the connection, and with -p, the reverse proxy support of the servers, the address of X-Forwarded-For, with the scheme of X-Forwarded-Proto as wsgi.url_scheme, so that request.is_secure() is true without SECURE_PROXY_SSL_HEADER.

Django sets TZ of the process to its TIME_ZONE when it loads its settings, America/Chicago unless the project sets another one, so the time of the log and of the access log of Pyjoyment is in it too, as of the rest of the process; tz of Pyjoyment.Plugin.AccessLog gives the access log another one.

Static files

Pyjoyment.Static serves the files of collectstatic before the routes, in the place of whitenoise, whose middleware leaves MIDDLEWARE: the parent of STATIC_ROOT is the directory of the static files, which has the files of / too, as robots.txt, and the hashed names of ManifestStaticFilesStorage are cached for a year, as Mounting describes it. The directory of WHITENOISE_ROOT, of the files of / as favicon.ico, is another directory of app.static.paths, and app.static.extra = {} leaves out the favicon and the images of Pyjoyment for a project that has its own.

With DEBUG and before collectstatic, staticfiles_urlpatterns() of Django serves the static files of the apps, as config/urls.py does, after the files that Pyjoyment has, as python -m config.server daemon of a checkout has none.

Probes

Pyjoyment.Plugin.Health answers /livez and /readyz, which Kubernetes requests, before the routes, so healthz=None leaves the /healthz of the project to it, which its monitors may request with other expectations. Django does not see the probes, so ALLOWED_HOSTS does not reject the address of the pod, which the kubelet sends as Host, and which a view of Django answers with 400 Bad Request: /readyz is the readiness probe without a Host header of its own.

With -c 1, a probe of HTTP is a connection as any other, which waits in the backlog while every worker is busy, so a liveness probe of HTTP restarts the pod in a slow spell of a busy application. A liveness probe on tcpSocket, which the kernel accepts into the backlog, does not.

readinessProbe:
  httpGet:
    path: /readyz
    port: http
livenessProbe:
  tcpSocket:
    port: http

Outside a request of Django, nothing closes a connection of Django to the database that the database has dropped, as after its restart, so a check that keeps using it fails every probe until the worker restarts. The check of config/server.py is framed as a request of Django, whose handlers of request_started and request_finished close a connection that is broken or older than CONN_MAX_AGE, as for a view, so that the next probe connects again.

The timeout of a check of Pyjoyment.Plugin.Health is the one of a check that returns a promise or a coroutine. A check that blocks runs to its end first, so the timeouts of its driver bound it, as connect_timeout in the OPTIONS of the database.

Access log

Pyjoyment.Plugin.AccessLog writes a line of each response to the standard output, in the combined format of the access log of gunicorn, without the probes, as Cookbook describes it. Behind a gateway that gives the address of the client in a header of its own, client takes it, as plugin('AccessLog', client='X-Envoy-External-Address'), as the first address of X-Forwarded-For is whatever the client wrote there.

Requests to other services

Pyjo.UserAgent is the HTTP client of a view, in the place of requests or httpx. In a view of WSGI, a blocking request runs an event loop of its own, while the worker waits for it as for a query of the database, so a request to the application itself needs another worker that is free.

django_wsgi.py
import Pyjo.UserAgent

from Pyjoyment.Lite import *

import os

from django.conf import settings
from django.core.wsgi import get_wsgi_application
from django.http import HttpResponse, JsonResponse
from django.urls import path


# A project of Django in one file
settings.configure(ROOT_URLCONF=__name__, ALLOWED_HOSTS=['*'], SECRET_KEY='insecure', TIME_ZONE='UTC')

# The user agent of the views, in the place of requests, and the API of GitHub, or the one of GITHUB_API_URL
ua = Pyjo.UserAgent.new()
api = os.environ.get('GITHUB_API_URL', 'https://api.github.com')


def index(request):
    return HttpResponse(f'Hello from Django at {request.path}!')


# A blocking request, which runs an event loop of its own, while the worker waits for it as for a query of the ORM, as
# the application sees no running event loop
def release(request):
    res = ua.get(f'{api}/repos/pyjoyment/Pyjoyment/releases/latest').result
    return JsonResponse({'status': res.code, 'release': res.json('/tag_name')})


urlpatterns = [path('', index), path('release', release)]

plugin('Mount', {'/': get_wsgi_application()})

app.start()

ASGI

The ASGI application of the project runs on the default reactor, of asyncio, with the same script, which mounts it in the place of the WSGI one.

plugin('Mount', {'/': 'config.asgi:application'})

Its async def views run in the event loop, where they await anything of asyncio and the promises of Pyjo.UserAgent, while the event loop serves the other requests. Its synchronous views and middleware run in a thread, which sync_to_async of Django gives them, one thread for all of them, and so do the queries of its async ORM, as await Note.objects.acount(), as the backends of databases of Django are synchronous. A project of synchronous views and queries serves one request at a time in each worker, and runs better on WSGI.

django_asgi.py
import Pyjo.UserAgent

from Pyjoyment.Lite import *

import os

from django.conf import settings
from django.core.asgi import get_asgi_application
from django.http import HttpResponse, JsonResponse
from django.urls import path


# A project of Django in one file
settings.configure(ROOT_URLCONF=__name__, ALLOWED_HOSTS=['*'], SECRET_KEY='insecure', TIME_ZONE='UTC')

# The user agent of the views, in the place of httpx, and the API of GitHub, or the one of GITHUB_API_URL
ua = Pyjo.UserAgent.new()
api = os.environ.get('GITHUB_API_URL', 'https://api.github.com')


def index(request):
    return HttpResponse(f'Hello from Django at {request.path}!')


# A request awaited in the event loop of asyncio, which serves the other requests meanwhile
async def release(request):
    res = (await ua.get_p(f'{api}/repos/pyjoyment/Pyjoyment/releases/latest')).result
    return JsonResponse({'status': res.code, 'release': res.json('/tag_name')})


urlpatterns = [path('', index), path('release', release)]

# An ASGI application, on the reactor of asyncio, the default one
plugin('Mount', {'/': get_asgi_application()})

app.start()

The ASGI application of Django has no lifespan, which Pyjo.App.ASGI notes in the log on debug and serves it without.

Tests

The tests of the project, which manage.py test runs, test the static files, the probes and the routes of Pyjoyment in front of it too, with Pyjo.Test.Pyjo of the application of config/server.py in a TestCase of Django, as Pyjo.Test describes the tests of unittest.

notes/tests.py
import Pyjo.Test
import Pyjo.Test.Pyjo

from notes.models import Note

import unittest

from config.server import app
from django.core.signals import request_finished, request_started
from django.db import close_old_connections
from django.test import TestCase


def setUpModule():
    unittest.enterModuleContext(Pyjo.Test.raise_failures())


# The requests of Pyjoyment reach the WSGIHandler of Django in the transaction of the test, whose handlers of
# request_started and request_finished would close its connection, so they are detached, as the test client of Django
# detaches them for its own requests
class ServerTests(TestCase):
    @classmethod
    def setUpClass(cls):
        super().setUpClass()
        cls.t = cls.enterClassContext(Pyjo.Test.Pyjo.new(app))
        for signal in (request_started, request_finished):
            signal.disconnect(close_old_connections)
            cls.addClassCleanup(signal.connect, close_old_connections)

    def test_notes(self):
        Note.objects.create(text='first')
        self.t.get_ok('/').status_is(200).content_like(r'^1 notes')
        self.t.post_ok('/notes', form={'text': 'second'}).status_is(200).json_is('/notes', ['first', 'second'])

    def test_probes(self):
        self.t.get_ok('/livez').status_is(200)
        self.t.get_ok('/readyz').status_is(200)
        self.t.get_ok('/static/notes/notes.css').status_is(404)
        self.assertEqual(Note.objects.count(), 0)

A request of the test reaches the WSGIHandler of Django, as one of a server does, which sends request_started and request_finished, and so does the check of the database. Their handler close_old_connections closes a connection that is not in the autocommit of the settings, as the one of the transaction of a TestCase, so a query of the view and of the check fails after it. The test client of Django detaches the handler for its own requests, and the test detaches it for its class. The test database of SQLite in memory, the default, hides it, as Django does not close its connection, and one of PostgreSQL, or of SQLite in a file, shows it.

More

The page of Examples has both examples, which request the API of GitHub, or the one of GITHUB_API_URL.

You can continue with Guides now.