Tips for debugging Django deployments to Azure App Service

I've been working on deploying various Django apps to Azure App Service this week and have run into a few issues since I'm customizing the app configuration quite a bit. Both as a reminder for myself and a resource for others, I figured I'd write up my tips for debugging Django app deployments and App Service.

After you finish deploying, first visit the app URL to see if it loads. If it does, amazing! If it doesn't, here are steps you can take to figure out what went wrong.

Check the deployment logs

Select Deployment Center from the side navigation menu, then select Logs. You should see a timestamped list of recent deploys:

Check whether the status of the most recent deploy is "Success (Active)" or "Failed". If it's success, the deployment logs might still reveal issues, and if it's failed, the logs should certainly reveal the issue.

Click the commit ID to open the logs for the most recent deploy. First scroll down to see if any errors or warnings are reported at the end. This is what you'll hopefully see if all went well:

Now scroll back up to find the timestamp with the label "Running oryx build". Oryx is the open source tool that builds apps for App Service, Functions, and other platforms, across all the supported MS languages. Click the Show logs link next to that label. That will pop open detailed logs at the bottom. Scroll down.

Here's what a successful Oryx build looks like for a Django application:

Command: oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform python --platform-version 3.10 -p virtualenv_name=antenv --log-file /tmp/build-debug.log  -i /tmp/8dad4af021551b0 --compress-destination-dir | tee /tmp/oryx-build.log
Operation performed by Microsoft Oryx, https://github.com/Microsoft/Oryx
You can report issues at https://github.com/Microsoft/Oryx/issues

Oryx Version: 0.2.20220825.1, Commit: 24032445dbf7bf6ef068688f1b123a7144453b7f, ReleaseTagName: 20220825.1

Build Operation ID: |BrSlkWT7Cgo=.afae917a_
Repository Commit : 0aed8982-e84a-4573-9ed8-82d46e9056c4

Detecting platforms...
Detected following platforms:
  python: 3.10.8

Using intermediate directory '/tmp/8dad4af021551b0'.

Copying files to the intermediate directory...
Done in 0 sec(s).

Source directory     : /tmp/8dad4af021551b0
Destination directory: /home/site/wwwroot

Python Version: /tmp/oryx/platforms/python/3.10.8/bin/python3.10
Creating directory for command manifest file if it does not exist
Removing existing manifest file
Python Virtual Environment: antenv
Creating virtual environment...
Activating virtual environment...
Running pip install...
[21:49:24+0000] Collecting Django==4.1.1
[21:49:24+0000]   Using cached Django-4.1.1-py3-none-any.whl (8.1 MB)
[21:49:25+0000] Collecting psycopg2
[21:49:25+0000]   Using cached psycopg2-2.9.5-cp310-cp310-linux_x86_64.whl
[21:49:25+0000] Collecting python-dotenv
[21:49:25+0000]   Using cached python_dotenv-0.21.0-py3-none-any.whl (18 kB)
[21:49:25+0000] Collecting whitenoise[brotli]
[21:49:25+0000]   Using cached whitenoise-6.2.0-py3-none-any.whl (19 kB)
[21:49:25+0000] Collecting asgiref<4,>=3.5.2
[21:49:26+0000]   Using cached asgiref-3.5.2-py3-none-any.whl (22 kB)
[21:49:26+0000] Collecting sqlparse>=0.2.2
[21:49:26+0000]   Using cached sqlparse-0.4.3-py3-none-any.whl (42 kB)
[21:49:26+0000] Collecting Brotli
[21:49:26+0000]   Using cached Brotli-1.0.9-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.7 MB)
[21:49:27+0000] Installing collected packages: Brotli, whitenoise, sqlparse, python-dotenv, psycopg2, asgiref, Django
[21:49:38+0000] Successfully installed Brotli-1.0.9 Django-4.1.1 asgiref-3.5.2 psycopg2-2.9.5 python-dotenv-0.21.0 sqlparse-0.4.3 whitenoise-6.2.0

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip

Content in source directory is a Django app
Running collectstatic...

132 static files copied to '/tmp/8dad4af021551b0/staticfiles', 643 post-processed.
Done in 21 sec(s).
Not a vso image, so not writing build commands
Preparing output...

Copying files to destination directory '/tmp/_preCompressedDestinationDir'...
Done in 22 sec(s).
Compressing content of directory '/tmp/_preCompressedDestinationDir'...
Copied the compressed output to '/home/site/wwwroot'

Removing existing manifest file
Creating a manifest file...
Manifest file created.
Copying .ostype to manifest output directory.

Done in 103 sec(s).

Look for these important steps in the Oryx build:

• Detected following platforms: python: 3.10.8
  That should match your runtime in the App Service configuration.
• Running pip install...
  That should install all the requirements in your requirements.txt - if it didn't find your requirements.txt, then you won't see the packages installed.
• Content in source directory is a Django app
  This message means that Oryx detected your app is a Django app, based on the presence of Django in requirements.txt, and will now run the next step.
• Running collectstatic... This step is running `manage.py collectstatic`, a necessary step for most Django apps. It can be disabled via the DISABLE_COLLECTSTATIC environment variable if desired.

If you see all those steps in the Oryx build, then that's a good sign that the build went well, and you can move on to checking the App Service logs.

Check the log stream

Under the Monitoring section of the side nav, select Log stream. Scroll to the timestamp corresponding to your most recent deploy.

The logs should start with pulling Docker images:

Here are the full logs for a Django app successfully starting in an App Service container:

2023-01-04T18:20:46.723Z INFO  - 3.10_20221128.12.tuxprod Pulling from appsvc/python
2023-01-04T18:20:46.731Z INFO  -  Digest: sha256:03576ed3edbc1bd69db8826f2cb5bbbdcfd483da4dab4f51f5050972491afef3
2023-01-04T18:20:46.733Z INFO  -  Status: Image is up to date for mcr.microsoft.com/appsvc/python:3.10_20221128.12.tuxprod
2023-01-04T18:20:46.760Z INFO  - Pull Image successful, Time taken: 0 Minutes and 0 Seconds
2023-01-04T18:20:47.273Z INFO  - Starting container for site
2023-01-04T18:20:47.285Z INFO  - docker run -d --expose=8000 --name djangocc-vssonnwxjqvw2-app-service_2_4555c318 -e WEBSITE_SITE_NAME=djangocc-vssonnwxjqvw2-app-service -e WEBSITE_AUTH_ENABLED=False -e PORT=8000 -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=djangocc-vssonnwxjqvw2-app-service.azurewebsites.net -e WEBSITE_INSTANCE_ID=37c77572be775c3da9e718e75bdd3dcd6c69246656e6f77972070ce4bfef6455 -e HTTP_LOGGING_ENABLED=1 -e WEBSITE_USE_DIAGNOSTIC_SERVER=True appsvc/python:3.10_20221128.12.tuxprod python manage.py migrate && gunicorn --workers 2 --threads 4 --timeout 60 --access-logfile '-' --error-logfile '-' --bind=0.0.0.0:8000 --chdir=/home/site/wwwroot config.wsgi
2023-01-04T18:21:01.494Z INFO  - Initiating warmup request to container djangocc-vssonnwxjqvw2-app-service_2_4555c318_msiProxy for site djangocc-vssonnwxjqvw2-app-service
2023-01-04T18:21:01.524Z INFO  - Container djangocc-vssonnwxjqvw2-app-service_2_4555c318_msiProxy for site djangocc-vssonnwxjqvw2-app-service initialized successfully and is ready to serve requests.
2023-01-04T18:21:01.531Z INFO  - Initiating warmup request to container djangocc-vssonnwxjqvw2-app-service_2_4555c318 for site djangocc-vssonnwxjqvw2-app-service
2023-01-04T18:21:17.476Z INFO  - Waiting for response to warmup request for container djangocc-vssonnwxjqvw2-app-service_2_4555c318. Elapsed time = 16.0211255 sec
2023-01-04T18:21:33.263Z INFO  - Waiting for response to warmup request for container djangocc-vssonnwxjqvw2-app-service_2_4555c318. Elapsed time = 31.8082536 sec
2023-01-04T18:21:37.548Z INFO  - Container djangocc-vssonnwxjqvw2-app-service_2_4555c318 for site djangocc-vssonnwxjqvw2-app-service initialized successfully and is ready to serve requests.
2023-01-04T18:21:00.181237975Z    _____
2023-01-04T18:21:00.181266077Z   /  _  \ __________ _________   ____
2023-01-04T18:21:00.181271277Z  /  /_\  \\___   /  |  \_  __ \_/ __ \
2023-01-04T18:21:00.181274977Z /    |    \/    /|  |  /|  | \/\  ___/
2023-01-04T18:21:00.181278277Z \____|__  /_____ \____/ |__|    \___  >
2023-01-04T18:21:00.181281678Z         \/      \/                  \/
2023-01-04T18:21:00.181284978Z A P P   S E R V I C E   O N   L I N U X
2023-01-04T18:21:00.181288278Z
2023-01-04T18:21:00.181291278Z Documentation: http://aka.ms/webapp-linux
2023-01-04T18:21:00.181294478Z Python 3.10.4
2023-01-04T18:21:00.181297678Z Note: Any data outside '/home' is not persisted
2023-01-04T18:21:01.465437465Z Starting OpenBSD Secure Shell server: sshd.
2023-01-04T18:21:01.683907061Z Site's appCommandLine: python manage.py migrate && gunicorn --workers 2 --threads 4 --timeout 60 --access-logfile '-' --error-logfile '-' --bind=0.0.0.0:8000 --chdir=/home/site/wwwroot config.wsgi
2023-01-04T18:21:02.113255325Z Starting periodic command scheduler: cron.
2023-01-04T18:21:02.115641859Z Launching oryx with: create-script -appPath /home/site/wwwroot -output /opt/startup/startup.sh -virtualEnvName antenv -defaultApp /opt/defaultsite -userStartupCommand 'python manage.py migrate && gunicorn --workers 2 --threads 4 --timeout 60 --access-logfile '-' --error-logfile '-' --bind=0.0.0.0:8000 --chdir=/home/site/wwwroot config.wsgi'
2023-01-04T18:21:02.513265532Z Found build manifest file at '/home/site/wwwroot/oryx-manifest.toml'. Deserializing it...
2023-01-04T18:21:02.518158007Z Build Operation ID: |83xYwqpdGr0=.fe7a9527_
2023-01-04T18:21:02.519884905Z Oryx Version: 0.2.20220825.1, Commit: 24032445dbf7bf6ef068688f1b123a7144453b7f, ReleaseTagName: 20220825.1
2023-01-04T18:21:02.520450236Z Output is compressed. Extracting it...
2023-01-04T18:21:02.533109549Z Extracting '/home/site/wwwroot/output.tar.gz' to directory '/tmp/8daee7f6ddb01b5'...
2023-01-04T18:21:22.545602410Z App path is set to '/tmp/8daee7f6ddb01b5'
2023-01-04T18:21:22.611376378Z Writing output script to '/opt/startup/startup.sh'
2023-01-04T18:21:24.124335826Z Using packages from virtual environment antenv located at /tmp/8daee7f6ddb01b5/antenv.
2023-01-04T18:21:24.125478895Z Updated PYTHONPATH to '/opt/startup/app_logs:/tmp/8daee7f6ddb01b5/antenv/lib/python3.10/site-packages'
2023-01-04T18:21:31.321664095Z Operations to perform:
2023-01-04T18:21:31.321750300Z   Apply all migrations: account, admin, auth, contenttypes, sessions, sites, socialaccount, users
2023-01-04T18:21:31.321758001Z Running migrations:
2023-01-04T18:21:31.321762401Z   No migrations to apply.
2023-01-04T18:21:33.977904663Z [2023-01-04 18:21:33 +0000] [80] [INFO] Starting gunicorn 20.1.0
2023-01-04T18:21:33.986985797Z [2023-01-04 18:21:33 +0000] [80] [INFO] Listening at: http://0.0.0.0:8000 (80)
2023-01-04T18:21:33.987019399Z [2023-01-04 18:21:33 +0000] [80] [INFO] Using worker: gthread
2023-01-04T18:21:34.016193616Z [2023-01-04 18:21:34 +0000] [81] [INFO] Booting worker with pid: 81
2023-01-04T18:21:34.093535167Z [2023-01-04 18:21:34 +0000] [82] [INFO] Booting worker with pid: 82
2023-01-04T18:30:36.727907506Z djangocc-vssonnwxjqvw2-app-service : [200ef84a-9498-4d8b-8023-11edbfe8251f] Incoming request on /healthcheck?api-version=2021-08-01
2023-01-04T18:30:36.740159926Z djangocc-vssonnwxjqvw2-app-service : [200ef84a-9498-4d8b-8023-11edbfe8251f] Request to TokenService: Endpoint 169.254.129.14:8081, Port 8081, Path /healthcheck, Query ?api-version=2021-08-01, Method GET, UserAgent HealthCheck/1.0
2023-01-04T18:30:37.201139217Z djangocc-vssonnwxjqvw2-app-service : [200ef84a-9498-4d8b-8023-11edbfe8251f] Returning response for Site , Endpoint 169.254.129.14:8081, Port 8081, Path /healthcheck, Method GET, Result = 200

A few notable logs:

• 2023-01-04T18:21:01.683907061Z Site's appCommandLine: python manage.py migrate && gunicorn --workers 2 --threads 4 --timeout 60 --access-logfile '-' --error-logfile '-' --bind=0.0.0.0:8000 --chdir=/home/site/wwwroot config.wsgi
  This log shows up because my app overrides the app's startup command (`appCommandLine` in a Bicep file). If your app doesn't override the startup command, then Oryx will generate a startup script for you that calls `gunicorn` to start the server. The default startup script does not call manage.py migrate, so that's why my app overrides it.
• 2023-01-04T18:21:31.321758001Z Running migrations:
  That proves that the container did indeed all `manage.py migrate` from my custom startup script.
• 2023-01-04T18:21:33.977904663Z [2023-01-04 18:21:33 +0000] [80] [INFO] Starting gunicorn 20.1.0
  That's the start of the gunicorn server serving the Django app. After it starts, the logs should show HTTP requests.

If you aren't seeing the full logs, it's possible that your deploy happened too long ago and the portal has deleted some logs. In that case, open the Log stream and do another deploy, and you should see the full logs.

Alternatively, you can download the full logs from the Kudu interface. Select Advanced Tools from the side nav:

When the Kudu website loads, find the Current Docker Logs link and select Download as zip next to it:

In the downloaded zip file, find the filename that starts with the most recent date and ends with "_default_docker.log":

Open that file to see the full logs, with the most recent logs at the bottom.

Running PostgreSQL in a Dev Container with Flask/Django

Not familiar with Dev Containers? Read my first post about Dev Containers.

I recently added Dev Container support to a Flask+PostgreSQL sample and a Django+PostgreSQL sample. I really wanted to be able to run PostgreSQL entirely inside the Dev Container, since 1) I've had a hard time trying to setup PostgreSQL on my laptop in the past, and 2) I'd like to access the database when running the Dev Container inside Github Codespaces on the web. Fortunately, thanks to some Docker magic, I figured it out! 🐳

The first step is to create a docker-compose.yml file inside the .devcontainer folder:

version: "3"

services:
  app:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
      args:
        VARIANT: 3.9
        USER_UID: 1000
        USER_GID: 1000

    volumes:
      - ..:/workspace:cached

    # Overrides default so things don't shut down after the process ends
    command: sleep infinity

    # Runs app on the same network as the database container,
    # allows "forwardPorts" in devcontainer.json function
    network_mode: service:db

  db:
    image: postgres:latest
    restart: unless-stopped
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: app_user
      POSTGRES_DB: app
      POSTGRES_PASSWORD: app_password

volumes:
  postgres-data:

That file declares an app service based off a Dockerfile (which we'll see next) as well as a db service based off the official postgres image that stores data in a mounted volume. The db service declares environment variables for the database name, username, and password, which must be used by any app that connects to that database.

The next step is the Dockerfile, also inside the .devcontainer folder:

FROM mcr.microsoft.com/vscode/devcontainers/python:0-3

RUN curl -fsSL https://aka.ms/install-azd.sh | bash

ENV PYTHONUNBUFFERED 1

RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends postgresql-client

This file is particular to the needs of my sample, so it builds on a pre-built Python-optimized Dev Container from Microsoft and includes the azd tool. Your own file could start from any Docker image that includes Python. However, you will want the final command in the file that installs a PostgreSQL client.

The final file inside the .devcontainer folder is devcontainer.json, which is the entry point for Github Codespaces and the VS Code Dev Containers extension, and describes all the customizations. Here's a simplified version of mine:

{
    "name": "Python 3 & PostgreSQL",
    "dockerComposeFile": "docker-compose.yml",
    "service": "app",
    "workspaceFolder": "/workspace",
    "forwardPorts": [8000, 5432],
    "extensions": [
        "ms-python.python",
        "mtxr.sqltools",
        "mtxr.sqltools-driver-pg"
    ],
    "settings": {
        "sqltools.connections": [{
            "name": "Container database",
            "driver": "PostgreSQL",
            "previewLimit": 50,
            "server": "localhost",
            "port": 5432,
            "database": "app",
            "username": "app_user",
            "password": "app_password"
        }],
        "python.pythonPath": "/usr/local/bin/python"
    }
}

Let's break that down:

• dockerComposeFile points at the docker-compose.yml from earlier.
• service matches the name of the non-postgres service from that file.
• workspaceFolder matches the location of the volume from that file.
• forwardPorts instructs the Dev Container to expose port 8000 (for the Django app) and port 5432 (for the PostGres DB).
• extensions includes a really neat extension, SQLTools, which provides a graphical UI for the database tables and allows you to run queries against the tables.
• Inside settings, sqltools.connections specifies the same database name, username, and password that was declared in the docker-compose.yml.

The final step is to make sure that the app knows how to access the PostgreSQL database. In my sample apps, the DB details are set via environment variables, so I just use a .env file that looks like this:

DBNAME=app
DBHOST=localhost
DBUSER=app_user
DBPASS=app_password

...And that's it! If you'd like, here's a video where I run a Flask+PostgreSQL Dev Container inside Github Codespaces and also highlight the changes described above:

Check out the sample repos for full code, and let me know if you run into issues enabling a PostgreSQL DB in your own Dev Containers.

Deploying a Django app to Azure with the Azure Developer CLI

Azure recently announced a new developer tool, the Azure Developer CLI, and after using it for the last month, I'm in love. ❤️

AZD makes it really easy to deploy applications that require multiple Azure services (which is most non-trivial applications!). Instead of running multiple commands or visiting multiple places in the Azure portal, I can run a single command and have it take care of everything. The azd commands rely on configuration files written in Bicep that describe all the resources, plus an azure.yaml file that describes what code needs to be deployed to which server. The azd tool also helps with CI/CD pipelines and app monitoring, so it's helpfulness goes beyond just the initial deployment.

The team is working on providing lots of templates that are already "azd-ified" so that you can get started quickly with whatever infrastructure stack you prefer. Since I'm on the Python team, I've been working on templates for the top Python frameworks, Django, Flask, and FastAPI.

My first finished template is for a Django app with PostgreSQL database that gets deployed to Azure App Service with a PostgreSQL flexible server. The App Service app communicates with the server inside a private VNet, to make sure the database can't be easily accessed from the outside.

Check out the sample here:
github.com/pamelafox/msdocs-django-postgresql-sample-app-azd/

The readme has instructions for local development, deployment, monitoring, and pipeline setup. If you run into any issues following the steps, let me know in the project discussions tab. The `azd` tool is still in preview, so this is a great time to get feedback about what is and isn't working.

I also put together this video showing me running azd up on the repo and setting up Django admin:

Preparing a Django app for deployment on Azure App Service

I recently went through the process of creating a simple Django app and deploying it to Azure App Service, and discovered I had to make several changes to get the app working in a production environment. Many of those changes are common across production environments and described in the Django deployment checklist, but a few details are Azure-specific.

Use environment variables

A best practice is to store settings in environment variables, especially sensitive settings like database authentication details. You can set environment variables locally with export on the command-line, but a more repeatable local strategy is to put them in a file and load them from that file using the python-dotenv package.

First, add python-dotenv to your requirements.txt. Here's what mine looked like:

Django==4.1.1
psycopg2
python-dotenv

Then create a .env file with environment variables and their local settings:

FLASK_ENV=development
DBNAME=quizsite
DBHOST=localhost
DBUSER=pamelafox
DBPASS=

Note that it's completely fine for this file to get checked in, since these are only the local values, and your production DB should definitely have a different password than your local DB. 😬

Now adjust your current settings.py to use these environment variables (if it wasn't already):

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql_psycopg2",
        'NAME': os.environ['DBNAME'],
        'HOST': os.environ['DBHOST'],
        'USER': os.environ['DBUSER'],
        'PASSWORD': os.environ['DBPASS']
    }
}

To make sure those environment variables are actually loaded in when running locally, you need to edit manage.py. Only the local environment should get its variables from that file, so add a check to see if 'WEBSITE_HOSTNAME' is a current environment variable. That variable gets set by the Azure build system when it deploys an app, so it will always be set on production and it should not get set locally.

import os
import sys

from dotenv import load_dotenv

def main():
    """Run administrative tasks."""
    
    is_prod = 'WEBSITE_HOSTNAME' in os.environ
    if not is_prod:
        print("Loading environment variables from .env file")
        load_dotenv('./.env')

    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)


if __name__ == '__main__':
    main()

Use production settings

After those changes, the app should run fine locally, but it's not ready for production. There are a number of settings that should be different in production mode, such as DEBUG, SECRET_KEY,ALLOWED_HOSTS, CSRF_TRUSTED_ORIGINS, and DATABASES.

A typical way to customize the settings for production is to add a new file that imports all the previous settings and overrides only the handful needed for production. Here's what my production.py looks like:

from .settings import *
import os

DEBUG = False
SECRET_KEY = os.environ['SECRET_KEY']

# Configure the domain name using the environment variable
# that Azure automatically creates for us.
ALLOWED_HOSTS = [os.environ['WEBSITE_HOSTNAME']]
CSRF_TRUSTED_ORIGINS = ['https://'+ os.environ['WEBSITE_HOSTNAME']] 


DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': os.environ['DBNAME'],
        # DBHOST is only the server name, not the full URL
        'HOST': os.environ['DBHOST'] + ".postgres.database.azure.com",
        'USER': os.environ['DBUSER'],
        'PASSWORD': os.environ['DBPASS']
    }
}

That file also uses the 'WEBSITE_HOSTNAME' environment variable, this time for setting the values of 'ALLOWED_HOSTS' and 'CSRF_TRUSTED_ORIGINS'.

Now you need to make sure the production settings get used when the app is running in a production environment.

First modify wsgi.py, since Azure uses WSGI to serve Django applications.

from django.core.wsgi import get_wsgi_application

is_prod = 'WEBSITE_HOSTNAME' in os.environ
settings_module = 'quizsite.production' if is_prod else 'quizsite.settings'
os.environ.setdefault("DJANGO_SETTINGS_MODULE", settings_module)

application = get_wsgi_application()

You also need to modify manage.py since Azure calls it to run Django commands on the production server, like manage.py collectstatic.

import os
import sys

from dotenv import load_dotenv

def main():
    is_prod = 'WEBSITE_HOSTNAME' in os.environ
    if not is_prod:
        print("Loading environment variables from .env file")
        load_dotenv('./.env')

    settings_module = "quizsite.production" if is_prod else 'quizsite.settings'
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', settings_module)
    
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)

if __name__ == '__main__':
    main()

Deploy!

Your app should be ready for deployment, code-wise. You can follow the instructions in a tutorial like Deploy a Python web app with PostgreSQL in Azure, but with your own app instead of the sample app. That tutorial includes instructions for setting environment variables on production, using either the VS Code extension or the Azure portal.

You may decide to store the SECRET_KEY value inside the Azure Key vault and retrieve it from there instead, following this KeyVault tutorial.

If you want to see all the recommended changes in context, check out the repository for my sample app .

Rewriting Django Admin in Backbone

Preamble:

This is an internal guide I wrote for Coursera about our site admin app, which is a Backbone "port" of Django admin. I've snapshotted it here in case it's interesting to other folks that are using Django admin and going down the same route, or just generally thinking about making an admin interface in Backbone.

A Bit of History

When Coursera first began advertising courses on www.coursera.org, the process to add a new course was quite manual: our Course Ops team would work with the university admins to draft up HTML describing the course, engineers would paste that HTML into the DB, and make edits to it as requested. Or, that's how the lore goes.

Here's how that data might look on the site:

As you can imagine, that process doesn't scale: it took unnecessary engineering honors to make incremental edits to the course descriptions, and it did not please university admins to have to wait on such a slow cycle. Ideally, they could log in, edit the description, and see the changes live, within a matter of minutes.

So we set about making an admin interface for the data. Since www.coursera.org runs off a Python/Django/MySQL backend, we first went down the expected route: Django Admin, an app that's built into Django for easy editing of database tables, complete with different permission levels, edit logs, and an extensibility mechanism.

Here's what it looked like on our data:

Django Admin was easy to set up and skin, but when we started implementing requests from Course Ops to improve the editing experience and workflow, we found we had to fight against Django Admin, to hack into its core in ways that felt wrong. We wanted a lot of different ways of editing data, which we could do via custom Django Admin widgets, but that often meant writing HTML and JavaScript into our Python code itself. We also wanted different buttons in our forms depending on the state of the model, like "open a session", and to do that, we had to modify the base templates with hacky HTML and javascript.

You can see a few of our widgets and buttons here:

We were making it work, sure, but I wasn't sure how long we could keep making it work, and if I could bring myself to look at the codebase later, knowing how we'd contorted it to meet our needs. Once I realized that I was the likely engineer to be making bug fixes and implementing feature requests, I decided it was time to move away, fast, before we got too deep into it.

Site Admin

So, over the course of the 3-day Labor Day weekend, I wrote "site admin", an admin interface that used our new approach to writing frontends, with a Backbone frontend and a RESTful Django API. It wasn't feature-complete with Django admin after those 3 days (and it still isn't), but it was built to be extendible on the client-side in a way that Django Admin was not, and that has made it much easier to build on.

Let's walk through the API and the frontend.

The API

The site admin API is designed to be exactly the sort of API that Backbone expects, a RESTful JSON API. It's based off an open-source project called Djangbone, but is now heavily modified.

In admin_api/views.py, the RestrictedAdminAPIView extends the generic Django View class, and defines get/post/put/delete methods that respond to the appropriate HTTP verb and understand how to generically fetch/create/edit/delete any model/collection. That file also contains AdminLogsView, which handles creation and retrieval of edit logs, and AdminSearchView, which is used by autocompletes in the frontend for finding models.

To set up an API for a particular set of models, we'd follow these steps, using the categories app and Category model as an example:

• Update categories/models.py: We add a static method to a Model class that returns back all of the models that the given user is allowed to administer.

  class Category(models.Model):
      @staticmethod
      def objects_administerd_by_user(user):
          if user.is_superuser:
              return Category.objects.all()
          else:
            return Category.objects.none()
  

• Create categories/admin_api.py: In it, we create a new view that extends RestrictedAdminAPIView, where we specify the base_queryset and base_model, corresponding to our Model, and we provide a list of fields that should be serialized into the JSON (we do not want to pass down all fields, particularly in the case of models with related fields, like course students). We also define get_add_form and get_edit_form, which return a Django Form subclass based on the request. We often serve different forms to different users, like when we want looser field validation for super users versus university admins.

  CATEGORY_FIELDS = (
                    'name',
                    'description',
                    )
  
  
  class CategoryAdminAPIView(RestrictedAdminAPIView):
      base_queryset = Category.objects.all()
      base_model = Category
  
      serialize_fields = CATEGORY_FIELDS + ('id', 'short_name')
  
      def get_add_form(self, request):
          if request.user.is_superuser:
              return NewCategoryAdminAPIForm
          else:
              return None
  
      def get_edit_form(self, request):
          if request.user.is_superuser:
              return EditCategoryAdminAPIForm
          else:
              return None
  
  
  class EditCategoryAdminAPIForm(AdminAPIForm):
  
      protected_fields = [
                          'name',
                          'short_name',
                          ]
  
      class Meta:
          model = Category
          fields = CATEGORY_FIELDS
  
      def clean(self):
          cleaned_data = super(EditCategoryAdminAPIForm, self).clean()
          short_name = cleaned_data.get('short_name')
          if short_name is not None and len(short_name) > 20:
              err = 'Please limit to 20 chars (currently %d).' % (
                  len(short_name))
              self._errors['short_name'] = self.error_class([err])
              del cleaned_data['short_name']
          if short_name is not None and not re.match('^[a-z0-9-\.]+$', short_name):
              self._errors['short_name'] = \
                  self.error_class(['Please limit to a-z,0-9.'])
              del cleaned_data["short_name"]
          name = cleaned_data.get('name')
          if name is not None and len(name) > 60:
              err = 'Please limit to 60 chars (currently %d).' % (
                  len(name))
              self._errors['name'] = self.error_class([err])
              del cleaned_data['name']
          return cleaned_data
  
  
  class NewCategoryAdminAPIForm(EditCategoryAdminAPIForm):
  
      class Meta:
          model = Category
          fields = CATEGORY_FIELDS + ('short_name',)
  

• Update admin_api/urls.py: We add 2 URL patterns for this model's API, to handle the model and collection verbs:

  url(r'^categories$',
    CategoryAdminAPIView.as_view(),
    name="api_categories"),
  url(r'^categories/(?P\d+)',
    CategoryAdminAPIView.as_view(),
    name="api_category"), 
      

• Update admin_api/tests.py: We add tests for the API, checking permissions and validation.

     def test_create_category(self):
      client = Client()
  
      cats_url = reverse('api_categories')
  
      # Test: super-user can create category
      create_and_login_as_superuser(client)
      data = {
          'name': 'Fake Category',
          'short_name': 'fake-cat',
      }
      response = self.post_json(client, cats_url, data)
      response_json = simplejson.loads(response.content)
      self.assertEqual(response.status_code, 200)
      self.assertEqual(response_json['name'], data['name'])
      self.assertEqual(response_json['short_name'], data['short_name'])
  
      # Test: cant edit short name after its created
      cat_url = reverse('api_category', args=[response_json['id']])
      data['short_name'] = 'fakecat2'
      response = self.put_json(client, cat_url, data)
      response_json = simplejson.loads(response.content)
      self.assertEqual(response.status_code, 200)
      self.assertEqual(response_json['short_name'], 'fake-cat')
  
      # Test: Cant use a shitty short name
      data['short_name'] = 'SoGreat OMG'
      response = self.post_json(client, cats_url, data)
      response_json = simplejson.loads(response.content)
      self.assertEqual(response.status_code, 400)
      self.assertEqual(response_json['short_name'],
          ['Please limit to a-z,0-9.'])
  
      # Test: instructors cant create categories
      create_and_login_as_instructor(client)
      response = self.post_json(client, cats_url, data)
      self.assertEqual(response.status_code, 400)
  

The Frontend

The site admin frontend is designed in a similar way as the backend: generic views that understand models and collections generally, with ways to specify the differences for each model.

ModelAdminPageView is responsible for creating a page with a header, banner, and then nesting a ModelAdminFieldsView which knows how to create a form with fields, buttons, and links to related models. To create that form, ModelAdminFieldsView calls upon a number of views which extend FieldView, like Select2View and HiddenInputView, and renders them depending on the attributes <-> fields mapping in a model.

CollectionAdminPageView is responsible for creating a page with a header, "new model" button, and then nesting a CollectionAdminListView, which knows how to create statistics charts via the nvd3 library and a tabular view of a collection via CollectionAdminTableView.

To add a model to the frontend, we follow these steps, using the category model as an example:

• Create models/CategoryAdminModel.js: This model extends AdminModel, an extension of Backbone.Model. It defines properties that are needed by Backbone, like the API endpoint, as well as custom properties that are needed by the views (and yes, that is not a perfect separation of data and presentation, but life must go on). The custom properties include a mapping of attributes to form field types, any custom buttons and modals, attributes to filter by in the table view, and more.

  define(["jquery",
            "backbone",
            "underscore",
            "js/core/coursera",
            "pages/site-admin/models/AdminModel"
            ],
  function($, Backbone, _, Coursera, AdminModel) {
  
    var model = AdminModel.extend({
  
      url: 'admin/categories',
      webUrlLabel: 'categories',
      label: 'Category',
  
      displayName: function() {
        return this.get('name');
      },
  
      fieldsets: function() {
  
        return [{
          name: 'name',
          type: 'text'
        }, {
          name: 'short_name',
          type: 'text',
          readonly: !this.isNew()
        }];
      }
  
    });
  
    return model;
  
  });
  

• Create collections/CategoriesAdminCollection.js: This extends AdminCollection, and specifies a handful of properties needed by Backbone and our views. The bulk of the custom logic is in the model, not the collection.

  define(["backbone",
            "js/core/coursera",
            "pages/site-admin/collections/AdminCollection",
            "pages/site-admin/models/CategoryAdminModel"
            ],
  
  function(Backbone, Coursera, AdminCollection, CategoryAdminModel) {
  
    var collection = AdminCollection.extend({
  
      url: 'admin/categories',
  
      webUrlLabel: 'categories',
  
      label: 'Categories',
  
      model: CategoryAdminModel
  
    });
  
    return collection;
  });
  

• Update site-admin/routes.js: This routes file extends Backbone.Router and defines generic URLs that can handle any AdminModel or AdminCollection. We add the new models and collections to modelAcls and collectionAcls, respectively, so that we know what collections to link a user to in the dashboard view. We enforce actual ACLs on the server-side, of course.

The future

The site admin API and frontend have served us reasonably well as we have grown to want more editing abilities and workflow improvements, but there is much to be improved upon.

Collaborative Editing

While developing site admin by myself in my living room, there was one thing that never occurred to me: I was building a collaborative editing interface. We have many different sorts of admins that edit course data, everywhere from super users to university admins to instructors to TAs, and sometimes, there could be multiple admins editing at once. I didn't realize this, however, until after we unleashed site admin for a big launch and I got emails at 4am about instructors losing data and freaking out. I quickly realized how easily that could happen, if two staff were working on a course on different machines. As a quick fix, I added a notion of "protected fields" - fields that weren't allowed to go from something to nothing - and that prevented the worst case of admins losing all the text they'd painstakingly inputted. It does add a problem, however, of admins legitimately wanting to clear fields sometimes, and engineers needing to manually make that change. It also doesn't protect against losing incremental data in a field.

To make site admin work better for multiple editors, there are a few approaches we've thought of, which could be combined in some optimal way:

• Partial updates: Currently site admin does a PUT of the full data of the model, and saves it wholly. An approach we take in many of our frontends now is to use Backbone's changedAttributes to track what changed, and only do an HTTP PATCH and partial update of the model. That would mean two admins could edit different fields and not worry about overriding eachother's changes.
• Real-time update: We could poll for updates to the model, updating the fields when we see changes. If the admin is currently editing an updated field, we could alert to confirm override or show them the changed version.
• Notifications: We could keep track of what admins are on a page, and alert them about the possibility of concurrent edits, which might encourage them to consult with eachother about what changes they are making.
• Change confirmations: We could detect that something had changed since the admin started editing, and prompt the admin to confirm that yes, indeed, they are okay with that change.

DRYer Permissions

We currently have code in the frontend that mirrors the permissions on the backend, like to figure out what collections a particular admin has access to at all, and to figure out what fields should be read-only for particular admins. This code is problematic since it means twice the code to change permission, and it's also dangerous because an engineer could fool themself into thinking that they'd secured something properly, if they did not put a test on the backend.

A better approach might be an API that sends down the permissions for an admin, perhaps using the HTTP OPTIONS verb. For example, an HTTP OPTIONS request to 'admin/api' might return the following:

{"collection": ["categories", "universities", "instructors"]}

Once loading the form for a particular model, the HTTP OPTIONS request might return the following, based on inspecting the Django Form instances:

{"fields": {
   "short_name": {"read_only": false, "restriction": "[a-zA-Z]", "max_length": "20"},
}

Those restrictions would also depend on whether the model was new or existing, and that would need to be represented in that API.

Soft Delete

When we do a delete in site admin now, it does a true delete, removing the rows and related rows from the database tables. That is a scary operation, of course, since it means that the only way to get back the data is to find it in an old database backup, provided we still have it around for the time of deletion.

We would prefer to do a soft delete, which would mean adding a "deleted" column to each model, setting that to true upon deletion, and changing all of our APIs to honor that deleted column when fetching data. This also makes it easier to do an undo. The biggest hurdle to this is change would be auditing all of the APIs that call upon that data to make sure they respect the change.

Drafts vs. Master

When an admin edits and saves their changes in site admin now, the changes are immediately live. Admins don't always want this; they often want to be able to preview their changes, feel confident in them, and then make them live. We have this in place for our course admin data, like quizzes and lectures, but it would require significant changes to the database tables, admin APIs, and user-facing APIs. A step in the right direction might be to make it possible for admins to preview their course pages with the unsaved data by sending it through an iframe and postMessage, perhaps. That is made more difficult by the fact that our admin APIs represent the data in a very different form than the user-facing APIs, however.

Improved Logs

We currently record who did what to which model, but the "what" is only whether it was a creation, edit, or delete. Ideally, we would also include exactly what fields were changed, and what the diff was between the fields. That would be made much easier to do by adding HTTP PATCH support. We also need better pagination and searching of the logs, and we may want to expose them to non-super-users.

Workflow-Based vs. Model-Based

Django admin revolves around models, assuming that the admin thinks to themself, "Yes, I'd like to edit such and such today." However, as it turns out, admins more often think in terms of goals or workflows, such as "Today I'd like to release grades." or "Today I'd like to open the course for enrollment and let all the subscribers know." These workflows often involve different models at each step, and it is non intuitive for the user to have to figure out that sequence. For a few of them, we've created "Checklist" views, with steps and links to the relevant models, anchor'ed at the form element that should be edited. But I think it would be worth it to re-think the admin interface from scratch with the idea of workflows being a first class citizen, instead of something we've tacked on at the end.