This is the first installment in a series on leveraging pydantic for Django-based projects. Part 2 builds an app on this concept with Docker and conda to show how to align development and production environments; part 3 explores how to deploy a Django app to Heroku.

I used to find Django projects difficult because adding new environments in a reliable and scalable way was a challenge. However, combining pydantic with Python type hints provided the robust foundation I needed.

As explained in PEP 484, while type hints are primarily used for static analysis, they are also accessible during runtime. This allows third-party libraries like pydantic to leverage these annotations for runtime type checking. Pydantic utilizes Python type hints to effectively manage settings metadata and perform data validation during runtime.

This tutorial will illustrate the extensive benefits of using pydantic for settings management in Django.

Our configuration follows the best practices outlined on the Twelve-Factor App website:

1. Non-constant and sensitive configuration values should be stored as environment variables.
2. For development, use a .env file to store environment variables and exclude it from version control with .gitignore.
3. Utilize the cloud provider’s tools to manage (secret) environment variables in QA, staging, and production.
4. Employ a single settings.py file that dynamically configures itself based on environment variables.
5. Leverage pydantic to read, verify, validate, and typecast environment variables into Python variables used for Django configurations.

While some developers opt for multiple settings files (e.g., settings_dev.py, settings_prod.py), this method often leads to code duplication, ambiguity, elusive bugs, and increased maintenance overhead, hindering scalability.

Adhering to the best practices mentioned above allows for effortless, well-defined, and error-free environment additions. Although more complex configurations are possible, we’ll focus on development and production for simplicity.

Practical Implementation: Pydantic Settings and Environment Variables

Let’s examine a practical example in both development and production environments. We’ll illustrate the distinct configuration approaches and how pydantic supports them.

Our application requires a Django-supported database. Thus, we need to store the connection string securely. We’ll utilize the DATABASE_URL environment variable, formatted as follows, and managed using the dj-database-url Python package. This variable will be of type str:

1
2
3
4

postgres://{user}:{password}@{hostname}:{port}/{database-name}
mysql://{user}:{password}@{hostname}:{port}/{database-name}
oracle://{user}:{password}@{hostname}:{port}/{database-name}
sqlite:///PATH

In development, we can use a Dockerized PostgreSQL instance for convenience, while in production, we’ll connect to a provisioned database service.

Another crucial variable is the boolean flag DEBUG. In Django, the DEBUG mode should never be active in production. It’s designed for development purposes, providing detailed error pages during exceptions, among other features.

We can define different values for these environments:

The pydantic settings management module allows us to manage these distinct environment variable sets based on the current environment.

Setting the Stage: Preparatory Steps

Let’s start by configuring our development environment. Create a .env file with the following:

1
2

DATABASE_URL=postgres://postgres:mypw@localhost:5432/mydb
DEBUG=True

Next, add the .env file to your project’s .gitignore file. This ensures that sensitive information within the file is not tracked by version control.

While this method works seamlessly for development, our production environment necessitates a different approach. Following best practices, we’ll employ environment secrets for production environment variables. On Heroku, for instance, these are known as Config Vars, configurable through the Heroku Dashboard, and accessible to the deployed application as environment variables:

Our next step is to modify our application’s configuration to dynamically read these values from either environment.

Configuring Django’s settings.py

We’ll utilize a new Django project to provide the necessary structure. Execute the following terminal command:

1

$ django-admin startproject mysite

This generates a basic project structure for the mysite project, organized as follows:

1
2
3
4
5
6
7
8

mysite/
    manage.py
    mysite/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

The settings.py file contains default configurations that we’ll adapt to our needs.

At the beginning of the settings.py file, add the following code to incorporate environment variables and pydantic for settings management:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

import os
from pathlib import Path
from pydantic import (
    BaseSettings,
    PostgresDsn,
    EmailStr,
    HttpUrl,
)
import dj_database_url

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent

class SettingsFromEnvironment(BaseSettings):
    """Defines environment variables with their types and optional defaults"""

    DATABASE_URL: PostgresDsn
    DEBUG: bool = False

    class Config:
        """Defines configuration for pydantic environment loading"""

        env_file = str(BASE_DIR / ".env")
        case_sensitive = True

config = SettingsFromEnvironment()

os.environ["DATABASE_URL"] = config.DATABASE_URL
DATABASES = {
    "default": dj_database_url.config(conn_max_age=600, ssl_require=True)
}
DEBUG = config.DEBUG

Let’s break down what this code does:

• It defines a class named SettingsFromEnvironment that inherits from pydantic’s BaseSettings class.
• It defines two settings, DATABASE_URL and DEBUG, using Python type hints to specify their types and optional default values.
• It defines a Config class instructing pydantic to search for variables in the .env file if they’re not found in the system’s environment.
• It instantiates the Config class as config, making the desired variables accessible as config.DATABASE_URL and config.DEBUG.
• It defines the standard Django variables DATABASES and DEBUG using the values from these config members.

This code remains consistent across all environments, and pydantic handles the following:

• It searches for the environment variables DATABASE_URL and DEBUG.
  • If they exist, as in production, it utilizes them.
  • If not, it retrieves the values from the .env file.
  • If a value is still missing:
    • It raises an error for DATABASE_URL.
    • It assigns a default value of False to DEBUG.
• If an environment variable is found, pydantic performs field types and raises an error if they fail:
  • For DATABASE_URL, it verifies if it’s a valid PostgresDsn-style URL.
  • For DEBUG, it ensures it’s a valid, non-strict pydantic boolean.

The explicit setting of the operating system’s environment variable for DATABASE_URL, although seemingly redundant, enables pydantic to parse, verify, and validate it. If the DATABASE_URL environment variable is missing or incorrectly formatted, pydantic provides a clear error message, proving invaluable as the application transitions between environments.

If a variable is undefined, a default value is assigned, or an error prompts for its definition, specifying the required type. These prompts clarify which variables need definition, simplifying onboarding for new team members and DevOps engineers and preventing issues arising from undefined variables.

With this, we have implemented a maintainable settings management system using a single settings.py file. This approach’s strength lies in its flexibility, allowing us to define environment variables in a .env file or through our hosting environment’s mechanisms.

Towards Scalability

Using Django settings management with pydantic runtime typing has significantly improved the maintainability and reduced the size of my Django projects. This approach provides a well-structured, self-documenting, and scalable solution for managing environments.

The next article in this series provides a step-by-step guide to building a Django application from scratch with pydantic settings management and deploying it to Heroku.

The Toptal Engineering Blog thanks Stephen Davidson for reviewing the code samples in this article.

This article previously highlighted a specific Python release. As Python has supported type hints for some time, the introduction has been updated to reflect this.