Part 2: A Tutorial on Optimizing Your Environment for Development and Production with Pydantic

This article is the second in a series exploring how to effectively use pydantic in Django projects. The first part focused on using pydantic’s Python type hints to improve the management of Django settings.

Developers often create inconsistencies between their development and production environments, leading to avoidable issues and increased workload. Maintaining consistent setups simplifies continuous deployment and streamlines the development process. To illustrate this, we’ll build a sample Django application using Docker, pydantic, and conda to create a consistent development environment.

A typical development environment consists of:

  • A local version control repository.
  • A Dockerized PostgreSQL database.
  • A conda environment for managing Python dependencies.

Both simple and complex projects can benefit from using pydantic and Django. The following steps demonstrate a straightforward solution for environment mirroring.

Git Repository Configuration

Before diving into coding or system setup, we’ll initialize a local Git repository:

1
2
3
4

mkdir hello-visitor
cd hello-visitor

git init

We’ll begin with a basic Python .gitignore file placed in the repository’s root directory. This file will be updated throughout the tutorial, and we’ll later add files that should be excluded from Git tracking.

Django PostgreSQL Configuration Using Docker

Django, by default, utilizes SQLite; however, we generally prefer more robust databases like PostgreSQL for handling mission-critical data, especially due to SQLite’s limitations with concurrent user access. Maintaining the same database for development and production is crucial, aligning with the principles of The Twelve-factor App.

Fortunately, running a local PostgreSQL instance is effortless with the help of Docker and Docker Compose.

To keep our root directory organized, we’ll place Docker-related files in dedicated subdirectories. Let’s start by creating a Docker Compose file for deploying PostgreSQL:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# docker-services/docker-compose.yml
version: "3.9"

services:
  db:
    image: "postgres:13.4"
    env_file: .env
    volumes:
      - hello-visitor-postgres:/var/lib/postgresql/data
    ports:
      - ${POSTGRES_PORT}:5432

volumes:
  hello-visitor-postgres:

Next, we’ll create a docker-compose environment file to configure the PostgreSQL container:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# docker-services/.env

POSTGRES_USER=postgres
POSTGRES_PASSWORD=MyDBPassword123

# The 'maintenance' database
POSTGRES_DB=postgres

# The port exposed to localhost
POSTGRES_PORT=5432

With the database server defined and configured, let’s start the container in the background:

1

sudo docker compose --project-directory docker-services/ up -d

It’s important to highlight the use of sudo in the command above. This is necessary unless specific steps are implemented within our development environment.

Database Creation

We’ll use a common toolset, pgAdmin4, to connect to and set up PostgreSQL, employing the same credentials defined earlier in the environment variables.

Now, let’s create a new database named hello_visitor:

A pgAdmin4 screen within a browser showing the General tab in a Create Database dialog. The database text field contains the value hello_visitor, the owner field displays the postgres user, and the comment field is blank.

With our database ready, we can proceed with setting up our programming environment.

Python Environment Management via Miniconda

We need an isolated Python environment with the necessary dependencies. We’ll use Miniconda for easy setup and management.

Let’s create and activate our conda environment:

1
2

conda create --name hello-visitor python=3.9
conda activate hello-visitor

Now, create a file named hello-visitor/requirements.txt to list our Python dependencies:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

django
# PostgreSQL database adapter:
psycopg2
# Pushes .env key-value pairs into environment variables:
python-dotenv
pydantic
# Utility library to read database connection information:
dj-database-url
# Static file caching:
whitenoise
# Python WSGI HTTP Server:
gunicorn

Next, instruct Python to install these dependencies:

1
2
3

cd hello-visitor

pip install -r requirements.txt

With the dependencies installed, we are prepared for application development.

Django Scaffolding

We’ll use django-admin to scaffold our project and app, followed by running the generated manage.py file:

1
2
3
4
5
6
7
8
9

# From the `hello-visitor` directory
mkdir src
cd src

# Generate starter code for our Django project.
django-admin startproject hello_visitor .

# Generate starter code for our Django app.
python manage.py startapp homepage

Next, configure Django to load our project. In the settings.py file, update the INSTALLED_APPS array to include our newly created homepage app:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# src/hello_visitor/settings.py

# ...

INSTALLED_APPS = [
    "homepage.apps.HomepageConfig",
    "django.contrib.admin",
    # ...
]

# ...

Application Setting Configuration

Following the pydantic and Django settings approach from the first installment, we need to create an environment variables file for our development environment. We’ll transfer our existing settings to this file as follows:

  1. Create src/.env to store our development environment settings.
  2. Copy settings from src/hello_visitor/settings.py to src/.env.
  3. Remove the copied lines from the settings.py file.
  4. Ensure the database connection string uses the previously configured credentials.

Our src/.env environment file should resemble this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

DATABASE_URL=postgres://postgres:MyDBPassword123@localhost:5432/hello_visitor
DATABASE_SSL=False

SECRET_KEY="django-insecure-sackl&7(1hc3+%#*4e=)^q3qiw!hnnui*-^($o8t@2^^qqs=%i"
DEBUG=True
DEBUG_TEMPLATES=True
USE_SSL=False
ALLOWED_HOSTS='[
    "localhost",
    "127.0.0.1",
    "0.0.0.0"
]'

Configure Django to read settings from environment variables using pydantic with this code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

# src/hello_visitor/settings.py
import os
from pathlib import Path
from pydantic import (
    BaseSettings,
    PostgresDsn,
    EmailStr,
    HttpUrl,
)
import dj_database_url

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent

class SettingsFromEnvironment(BaseSettings):
    """Defines environment variables with their types and optional defaults"""

    # PostgreSQL
    DATABASE_URL: PostgresDsn
    DATABASE_SSL: bool = True

    # Django
    SECRET_KEY: str
    DEBUG: bool = False
    DEBUG_TEMPLATES: bool = False
    USE_SSL: bool = False
    ALLOWED_HOSTS: list

    class Config:
        """Defines configuration for pydantic environment loading"""

        env_file = str(BASE_DIR / ".env")
        case_sensitive = True

config = SettingsFromEnvironment()

os.environ["DATABASE_URL"] = config.DATABASE_URL
DATABASES = {
    "default": dj_database_url.config(conn_max_age=600, ssl_require=config.DATABASE_SSL)
}

SECRET_KEY = config.SECRET_KEY
DEBUG = config.DEBUG
DEBUG_TEMPLATES = config.DEBUG_TEMPLATES
USE_SSL = config.USE_SSL
ALLOWED_HOSTS = config.ALLOWED_HOSTS

# ...

If you encounter issues after making these changes, compare your modified settings.py file with the version in our source code repository.

Model Creation

Our application will keep track of and show the number of times the homepage has been visited. To do this, we’ll need a model to store this count and then use Django’s object-relational mapper to initialize a single database row through a data migration.

First, let’s create the VisitCounter model:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# hello-visitor/src/homepage/models.py
"""Defines the models"""
from django.db import models


class VisitCounter(models.Model):
    """ORM for VisitCounter"""

    count = models.IntegerField()

    @staticmethod
    def insert_visit_counter():
        """Populates database with one visit counter. Call from a data migration."""
        visit_counter = VisitCounter(count=0)
        visit_counter.save()

    def __str__(self):
        return f"VisitCounter - number of visits: {self.count}"

Next, initiate a migration to create the database tables:

1
2
3

# in the `src` folder
python manage.py makemigrations
python manage.py migrate

We can use pgAdmin4 to view the database and confirm the existence of the homepage_visitcounter table.

Now, let’s add an initial value to our homepage_visitcounter table. We’ll create a new migration file using Django’s scaffolding to achieve this:

1
2

# from the 'src' directory
python manage.py makemigrations --empty homepage

Modify the generated migration file to utilize the VisitCounter.insert_visit_counter method defined earlier:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# src/homepage/migrations/0002_auto_-------_----.py 
# Note: The dashes are dependent on execution time.
from django.db import migrations
from ..models import VisitCounter

def insert_default_items(apps, _schema_editor):
    """Populates database with one visit counter."""
    # To learn about apps, see:
    # https://docs.djangoproject.com/en/3.2/topics/migrations/#data-migrations
    VisitCounter.insert_visit_counter()


class Migration(migrations.Migration):
    """Runs a data migration."""

    dependencies = [
        ("homepage", "0001_initial"),
    ]

    operations = [
        migrations.RunPython(insert_default_items),
    ]

We can now execute this modified migration for the homepage app:

1
2

# from the 'src' directory
python manage.py migrate homepage

By examining the contents of our table, we can ensure the migration was successful:

A pgAdmin4 screen within a browser showing a query "SELECT * FROM public.homepage_visitcounter ORDER BY id ASC". The Data Output tab shows that there is one row within that table. The surrogate key id field value is 1, and the count field value is 0.

We can see that our homepage_visitcounter table exists and has an initial visit count of 0. With the database set up, we can move on to building the UI.

Create and Configure Our Views

Our UI implementation consists of two main components: a view and a template.

We’ll create the homepage view, which will be responsible for increasing the visitor count, saving it to the database, and passing the updated count to the template for display:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# src/homepage/views.py
from django.shortcuts import get_object_or_404, render
from .models import VisitCounter

def index(request):
    """View for the main page of the app."""
    visit_counter = get_object_or_404(VisitCounter, pk=1)

    visit_counter.count += 1
    visit_counter.save()

    context = {"visit_counter": visit_counter}
    return render(request, "homepage/index.html", context)

Our Django application needs to handle requests directed at homepage. We’ll add the following file to configure this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# src/homepage/urls.py
"""Defines urls"""
from django.urls import path

from . import views

# The namespace of the apps' URLconf
app_name = "homepage"  # pylint: disable=invalid-name

urlpatterns = [
    path("", views.index, name="index"),
]

To serve our homepage application, we need to register it in a separate urls.py file:

1
2
3
4
5
6
7
8

# src/hello_visitor/urls.py
from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path("", include("homepage.urls")),
    path("admin/", admin.site.urls),
]

The base HTML template for our project will reside in a new file, src/templates/layouts/base.html:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

<!DOCTYPE html>
{% load static %}

<html lang="en">
  <head>
    <!-- Required meta tags -->
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <!-- Bootstrap CSS -->
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.1/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-+0n0xVW2eSR5OomGNYDnhzAbDsOXxcvSN1TPprVMTNDbiYZCxYbOOl7+AMvyTG2x" crossorigin="anonymous">

    <title>Hello, visitor!</title>
    <link rel="shortcut icon" type="image/png" href="{% static 'favicon.ico' %}"/>
  </head>
  <body>
  
    {% block main %}{% endblock %}

    <!-- Option 1: Bootstrap Bundle with Popper -->
    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.1/dist/js/bootstrap.bundle.min.js" integrity="sha384-gtEjrD/SeCtmISkJkNUaaKMoLD0//ElJ19smozuHV6z3Iehds+3Ulb9Bn9Plx0x4" crossorigin="anonymous"></script>

  </body>
</html>

We’ll extend the base template for our homepage app in a new file, src/templates/homepage/index.html:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

{% extends "layouts/base.html" %}

{% block main %}
  <main>
    <div class="container py-4">
      <div class="p-5 mb-4 bg-dark text-white text-center rounded-3">
        <div class="container-fluid py-5">
          <h1 class="display-5 fw-bold">Hello, visitor {{ visit_counter.count }}!</h1>
        </div>
      </div>
    </div>
  </main>
{% endblock %}

The final step in UI creation involves informing Django about the location of these templates. Let’s add a TEMPLATES['DIRS'] dictionary item to our settings.py file:

1
2
3
4
5
6
7
8

# src/hello_visitor/settings.py
TEMPLATES = [
    {
        ...
        'DIRS': [BASE_DIR / 'templates'],
        ...
    },
]

With the user interface implemented, we’re almost ready to test the application. But first, let’s set up the final component of our environment: static content caching.

Our Static Content Configuration

To ensure consistency with our production environment, we’ll configure static content caching in our development system.

All static files for our project will be placed in a single directory, src/static, and Django will be instructed to collect these files before deployment.

We’ll use Toptal’s logo as the favicon for our application and store it as src/static/favicon.ico:

1
2
3
4
5

# from `src` folder
mkdir static
cd static
wget https://frontier-assets.toptal.com/83b2f6e0d02cdb3d951a75bd07ee4058.png
mv 83b2f6e0d02cdb3d951a75bd07ee4058.png favicon.ico

Next, configure Django to collect static files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# src/hello_visitor/settings.py
# Static files (CSS, JavaScript, images)
# a la https://docs.djangoproject.com/en/3.2/howto/static-files/
#
# Source location where we'll store our static files
STATICFILES_DIRS = [BASE_DIR / "static"]
# Build output location where Django collects all static files
STATIC_ROOT = BASE_DIR / "staticfiles"
STATIC_ROOT.mkdir(exist_ok=True)

# URL to use when referring to static files located in STATIC_ROOT.
STATIC_URL = "/static/"

STATICFILES_STORAGE = "whitenoise.storage.CompressedManifestStaticFilesStorage"

We only want to keep the original static files in the source code repository, excluding the production-optimized versions. Let’s add the latter to our .gitignore file:

1

staticfiles

Now that our source code repository is correctly storing the necessary files, we can configure our caching system to handle these static files.

Static File Caching

To enhance the efficiency of serving static files for our Django application, we’ll utilize WhiteNoise in production, and consequently, in our development environment as well.

Register WhiteNoise as middleware by adding the following code snippet to the src/hello_visitor/settings.py file. The order of registration is critical, and WhiteNoiseMiddleware must be placed immediately after SecurityMiddleware:

1
2
3
4
5
6
7

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',
    # ...
]

STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'

Static file caching should now be configured in our development environment, enabling us to run the application.

Running Our Development Server

With a fully coded application, we can launch Django’s built-in development web server using the following command:

1
2

# in the `src` folder
python manage.py runserver

Each time we refresh the page at http://localhost:8000, the visit count will increment:

A browser window showing the main screen of our pydantic Django application, which says, "Hello, visitor!" on one line and "1" on the next.

We now have a functioning application that tracks visits with each page refresh.

Ready To Deploy

This tutorial outlined the steps to create a functional application within a robust Django development environment that mirrors a production setup. Part 3 will delve into deploying our application to its production environment. In the meantime, you can explore additional exercises showcasing the advantages of Django and pydantic in the code-complete repository accompanying this pydantic tutorial.

The Toptal Engineering Blog expresses gratitude to Stephen Davidson for reviewing and beta testing the code samples presented in this article.

Licensed under CC BY-NC-SA 4.0