Lazy Loading in Serializers and Templates #

The orders endpoint was fixed three months ago: one query with two eager loads, verified in the plan. It is slow again. The query layer is unchanged — but the serializer now emits a display_name field that calls a method on the customer, and that method falls back to the customer’s primary address when the name is blank. One query became 1 + 50 + 50.

The plan-level signature of lazy loading is covered in lazy loading plan artifacts. This page is about where those loads keep coming back from.

The Condition #

Eager loading is declared where the query is built. Rendering happens later, in code that usually has no idea which associations were preloaded:

Any of them can touch an association that was not preloaded, and the ORM will quietly satisfy it with a query per object. Because the loop is in the rendering layer, it does not appear in the code that was tuned, and it is invisible in the database as anything other than a high call count of a simple, fast statement.

The database side is the pattern described in detecting N+1 with pg_stat_statements: a statement with a very high calls value, a low mean_exec_time, and a call count proportional to another statement’s rows.

The durable fix is to make loading declarative and enforced: state the associations a representation needs next to that representation, and fail loudly when something else is touched.

The loop that lives in the rendering layer Four stages. The query layer runs one query with eager loads. The controller passes the objects to a serializer. The serializer calls a method per object that touches an unloaded association. The ORM issues one query per object, producing the N+1 pattern in the database. query layer 1 query, eager loads serializer walks each object computed field touches a lazy relation ORM one query per object the tuned code and the code causing the queries are in different files

Annotated Evidence #

From the database:

SELECT calls, round(mean_exec_time::numeric, 3) AS ms, round(total_exec_time::numeric) AS total_ms,
       left(query, 60) AS query
FROM pg_stat_statements ORDER BY calls DESC LIMIT 3;
  calls   |  ms   | total_ms |                       query
----------+-------+----------+----------------------------------------------------
  4120044 | 0.038 |   156561 | SELECT "addresses".* FROM "addresses" WHERE "addres…
  4120044 | 0.041 |   168921 | SELECT "customers".* FROM "customers" WHERE "custom…
    82401 | 1.902 |   156700 | SELECT "orders".* FROM "orders" WHERE "orders"."acc…
-- two statements with identical call counts, each 50× the orders query: two lazy loads per order

From the application, with a query-count guard around rendering:

with QueryBudget(3):                       # the pattern from strict-loading guards
    data = OrderSerializer(orders, many=True).data
# AssertionError: query budget 3 exceeded: SELECT "customers".* FROM "customers" WHERE …

And the plan of one of those queries — individually blameless:

Index Scan using customers_pkey on customers  (actual time=0.02..0.02 rows=1 loops=1)
  Index Cond: (id = $1)
  Buffers: shared hit=4
-- 0.04 ms is not a slow query; 4.1 million of them is a slow endpoint
How the pattern shows up Three items are annotated. Two statements with identical call counts, each a multiple of the parent query's row count. A per-call time far below a millisecond with a large total. A query-count guard failing inside serialization rather than inside the query layer. two statements, calls = 50 × parent rows one lazy load each per row mean 0.038 ms, total 156 s/day fast query, slow endpoint guard fails during serialization the loop is in rendering the database cannot tell you which code path issued the queries

Step-by-Step Resolution #

  1. Reproduce with a query-count guard around rendering only, so the failure points at the serializer rather than the query.

  2. Read the failing statement to identify which association is missing, then add it to the eager load where the query is built:

    orders = (Order.objects
          .select_related('customer')
          .prefetch_related('customer__addresses')
          .filter(account_id=account_id))

  3. Move the declaration next to the representation. Keep the list of required associations with the serializer or presenter that needs them, so a field added later is accompanied by its load:

    class OrderSerializer(...):
    required_select_related = ('customer',)
    required_prefetch_related = ('customer__addresses',)

    Then build the queryset from those attributes rather than duplicating them in views.

  4. Turn on strict loading for the request in development and tests so any unplanned access raises, as described in strict loading guards against lazy queries.

  5. Add an assertion to the endpoint’s test on the number of queries, with enough fixture rows that a per-object loop would exceed it.

  6. Watch for computed properties that fall back to another association; they are the most common hidden source.

  7. Re-check pg_stat_statements after deploying: the per-object statement’s call rate should drop to the number of requests, not the number of rows.

Queries per request After the original fix the endpoint issued three queries per request. After the serializer field was added it issued 103. After adding the prefetch for the new field it returns to four queries per request. after original fix 3 queries after new field 103 queries after prefetch added 4 queries 50 orders per page: two lazy loads per order

Before and After #

-- BEFORE: display_name touches customer.addresses per order
addresses by id: calls 4,120,044/day  total 156 s/day   endpoint p95 1.9 s

-- AFTER: prefetch_related('customer__addresses')
addresses by id = ANY(array): calls 82,401/day  total 9 s/day   endpoint p95 210 ms

Designing so the problem cannot recur #

Two structural habits prevent most recurrences. The first is to treat the set of associations a representation needs as part of that representation’s definition, not of the view that happens to use it — so adding a field forces the author to state its data requirement in the same file. Frameworks differ in how this is expressed, but the principle is identical in all of them.

The second is to make the rendering layer operate on plain data rather than on ORM objects: build a dictionary or dataclass in the query layer, with every value already materialised, and hand that to the serializer. A representation that cannot reach the ORM cannot lazily load anything. This costs a mapping step and pays for itself on any endpoint that renders more than a handful of objects.

Where neither is practical, the query-count assertion in tests is the safety net, and it should be part of the test for every list endpoint. It catches the regression at the moment it is introduced, which is the only time it is cheap to fix.

Common Pitfalls #

Fixing the query and not the representation. The next field reintroduces the loop. Diagnostic signal: the same endpoint regressing repeatedly. Fix: declare load requirements with the representation.

Guarding only the query layer. The loop is elsewhere. Diagnostic signal: guards passing while pg_stat_statements shows per-row calls. Fix: wrap rendering in the guard too.

Tests with one fixture row. A per-object loop looks like one query. Diagnostic signal: assertions passing on trivial data. Fix: enough rows that the loop is visible.

Nested serializers. Each level can add its own loads. Diagnostic signal: call counts that are multiples of multiples. Fix: prefetch the full path, such as customer__addresses.

Frequently Asked Questions #

Why do N+1 queries come back after I fixed the query? #

Because eager loading is declared where the query is built, while serializers, templates and computed properties access associations later. Any access to an association that was not preloaded issues its own query per object.

How do I find which code caused a per-object query? #

Wrap the rendering step in a query-count guard or enable strict loading in development, so the access raises at the exact line. The database alone only shows the statement, not the caller.

Should serializers know about eager loading? #

The association requirements should live next to the representation that needs them, and the query layer should build the queryset from that declaration. Otherwise the two drift apart every time a field is added.

Up: Lazy Loading Plan Artifacts