Free lesson · System Design Advanced

MongoDB History and the Document Model

Hard10 min readsystem-designadvanced

Every database is a bet on a data shape. Relational databases bet that your data fits into tables with fixed columns and that relationships between tables are the organizing principle. Key-value stores bet that your data is blobs you look up by a single key. Graph databases bet that relationships themselves are the interesting part, not the entities.

MongoDB bet that your data is documents - self-contained, hierarchical, variable-structure objects that map naturally to how applications actually represent things in memory. A user isn't a row in a table with foreign keys to addresses, phone numbers, and preferences. A user is a single document: {name: "Alice", address: {street: "123 Main", city: "Seattle"}, phones: ["206-555-0100", "206-555-0200"], preferences: {theme: "dark", notifications: true}}. One object. One read. No joins.

That bet turned out to be right for some workloads and catastrophically wrong for others. Understanding both sides is the point of this lesson.

The Origin Story

graph TD
    subgraph "MongoDB Origin"
        DWIGHT["Dwight Merriman & Eliot Horowitz<br/>──────────────────<br/>Previously built DoubleClick<br/>(the ad-serving platform that<br/>became Google DoubleClick)<br/><br/>At DoubleClick, they hit the<br/>limits of relational databases:<br/>• Rigid schemas slowed iteration<br/>• JOINs across 20+ tables for<br/>  a single ad decision were slow<br/>• Schema migrations on billion-row<br/>  tables took days<br/>• Object-relational mapping was<br/>  a constant source of bugs"]
        
        DWIGHT --> TENGEN["10gen (2007)<br/>──────────────────<br/>Started as a cloud platform<br/>(like early Heroku)<br/><br/>The platform needed a database<br/>that could handle dynamic,<br/>hierarchical data without<br/>rigid schemas<br/><br/>They couldn't find one,<br/>so they built it"]
        
        TENGEN --> MONGO_NAME["MongoDB (2009)<br/>──────────────────<br/>Name from 'humongous'<br/>(as in, humongous amounts of data)<br/><br/>Open-sourced the database<br/>component separately<br/><br/>The database became more<br/>popular than the platform<br/><br/>10gen renamed to MongoDB Inc."]
        
        MONGO_NAME --> NOSQL_ERA["The NoSQL Era (2009-2014)<br/>──────────────────<br/>MongoDB became the poster child<br/>of the NoSQL movement<br/><br/>Pitch: 'Schemaless! Web scale!<br/>No more JOINs! No more<br/>ALTER TABLE migrations!'<br/><br/>Teams adopted it for the marketing,<br/>not the architecture<br/><br/>Some workloads thrived<br/>Many workloads suffered"]
    end

This diagram shows MongoDB's origin. Dwight Merriman and Eliot Horowitz previously built DoubleClick, where they hit relational database limits - rigid schemas, slow joins across many tables, multi-day schema migrations, and constant object-relational mapping bugs. In 2007, they founded 10gen as a cloud platform that needed a database for dynamic hierarchical data. They couldn't find one, so they built MongoDB, named from "humongous." Released as open source in 2009, it became the poster child of the NoSQL movement with the pitch of schemaless and web-scale - but teams often adopted it for the marketing rather than the architecture, with mixed results.

The Document Model

graph TD
    subgraph "Relational Model (PostgreSQL/MySQL)"
        USERS_TABLE["users table<br/>──────────────<br/>id | name | email<br/>1  | Alice | alice@ex.com<br/>2  | Bob   | bob@ex.com"]
        
        ADDRESSES["addresses table<br/>──────────────<br/>id | user_id | street | city<br/>1  | 1       | 123 Main | Seattle<br/>2  | 1       | 456 Oak  | Portland<br/>3  | 2       | 789 Pine | Denver"]
        
        PHONES["phones table<br/>──────────────<br/>id | user_id | number<br/>1  | 1       | 206-555-0100<br/>2  | 1       | 206-555-0200<br/>3  | 2       | 303-555-0300"]
        
        QUERY_REL["To get Alice with addresses and phones:<br/>──────────────<br/>SELECT * FROM users u<br/>JOIN addresses a ON u.id = a.user_id<br/>JOIN phones p ON u.id = p.user_id<br/>WHERE u.id = 1<br/><br/>3 tables, 2 JOINs, result must be<br/>de-duplicated in application code"]
    end
    
    subgraph "Document Model (MongoDB)"
        DOC["users collection:<br/>──────────────<br/>{<br/>  _id: 1,<br/>  name: 'Alice',<br/>  email: 'alice@ex.com',<br/>  addresses: [<br/>    {street: '123 Main', city: 'Seattle'},<br/>    {street: '456 Oak', city: 'Portland'}<br/>  ],<br/>  phones: ['206-555-0100', '206-555-0200']<br/>}<br/><br/>ONE document. ONE read.<br/>No JOINs. No de-duplication.<br/>The object shape matches<br/>the application's data structure."]
    end

This diagram contrasts the relational model and the document model. In the relational model, Alice's data is spread across three tables - users, addresses, and phones - requiring two JOINs and application-level de-duplication to assemble. In the document model, Alice is a single document containing her name, email, an array of address objects, and an array of phone numbers. One document, one read, no joins, and the data shape matches the application's in-memory object structure.

When Embedding Works and When It Doesn't

The power of the document model is embedding - putting related data inside the parent document instead of in separate tables. But embedding has limits:

graph TD
    subgraph "Embedding Works When"
        GOOD1["One-to-few relationship<br/>──────────────────<br/>User → 2-3 addresses<br/>Product → 5-10 variants<br/>Article → 3-5 tags<br/><br/>Small, bounded arrays<br/>that are always read<br/>with the parent"]
        
        GOOD2["Data is read together<br/>──────────────────<br/>You always need the user<br/>AND their addresses<br/>AND their phones<br/><br/>Embedding means one read<br/>instead of multiple JOINs"]
        
        GOOD3["Child data doesn't exist<br/>independently<br/>──────────────────<br/>An address without a user<br/>is meaningless<br/>A comment without a post<br/>is meaningless<br/><br/>The child's lifecycle is<br/>bound to the parent"]
    end
    
    subgraph "Embedding Fails When"
        BAD1["One-to-many (unbounded)<br/>──────────────────<br/>User → 50,000 orders<br/>Post → 100,000 comments<br/>Store → 1,000,000 transactions<br/><br/>Document exceeds 16 MB limit<br/>Updating one comment requires<br/>rewriting the entire document<br/>Array grows indefinitely"]
        
        BAD2["Data is accessed independently<br/>──────────────────<br/>You need orders WITHOUT<br/>loading the user<br/>You need comments sorted<br/>across ALL posts<br/><br/>Embedded data can't be<br/>queried efficiently on its own"]
        
        BAD3["Many-to-many relationships<br/>──────────────────<br/>Students ↔ Courses<br/>Products ↔ Categories<br/>Users ↔ Groups<br/><br/>Which side do you embed?<br/>Both? Duplication nightmare<br/>Neither? You're doing JOINs<br/>in application code anyway"]
    end

This diagram explains when embedding works and when it fails. Embedding works for one-to-few relationships like a user with 2-3 addresses, when data is always read together, and when child data doesn't exist independently. Embedding fails for unbounded one-to-many relationships like a user with 50,000 orders which can exceed the 16 MB document limit, when data needs to be accessed independently of the parent, and for many-to-many relationships where choosing which side to embed leads to either duplication nightmares or application-level joins.

Important: The 16 MB document size limit is a hard wall, not a suggestion. MongoDB enforces a maximum BSON document size of 16 MB. An e-commerce system that embeds orders inside a user document will hit this limit when a user has enough orders. The fix - moving orders to a separate collection with a reference to user_id - is exactly the relational pattern you were trying to avoid. Design for bounded embedded arrays from day one. If the array can grow without bound, use a separate collection.

BSON: Binary JSON

MongoDB stores documents in BSON (Binary JSON), not raw JSON. Understanding the differences matters for performance and data modeling:

graph TD
    subgraph "JSON vs BSON"
        JSON["JSON (what you write)<br/>──────────────────<br/>{<br/>  'name': 'Alice',<br/>  'age': 30,<br/>  'joined': '2024-01-15T10:30:00Z',<br/>  'balance': 1234.56<br/>}<br/><br/>• Text format<br/>• Must parse to find any field<br/>• No native date type<br/>• No distinction between<br/>  int and float<br/>• No binary data support"]
        
        BSON["BSON (what MongoDB stores)<br/>──────────────────<br/>Binary encoding with:<br/><br/>• Length-prefixed fields<br/>  (skip to any field without<br/>   parsing the whole doc)<br/>• Native types:<br/>  - int32, int64, double, decimal128<br/>  - Date (64-bit ms since epoch)<br/>  - ObjectId (12-byte unique ID)<br/>  - Binary data<br/>  - Regular expressions<br/>  - Timestamp (internal)<br/>• Slightly larger on disk than JSON<br/>  (type tags + length prefixes)<br/>• Much faster to traverse"]
    end

This diagram compares JSON and BSON. JSON is text-based, requiring full parsing to find any field, with no native date type and no distinction between integers and floats. BSON is the binary format MongoDB actually stores - with length-prefixed fields allowing fast traversal without parsing the entire document, native types including 32-bit and 64-bit integers, dates, ObjectIds, binary data, and decimal128 for precise financial values. BSON is slightly larger on disk due to type tags but much faster to traverse.

ObjectId: MongoDB's Default Primary Key

Every MongoDB document has an _id field. If you don't provide one, MongoDB generates an ObjectId:

Bytes	Content	Purpose
0-3	Unix timestamp (seconds)	Roughly chronological ordering
4-8	Random value (unique per process)	Machine + process uniqueness
9-11	Incrementing counter	Ordering within same second on same process

Tip: ObjectId is time-sortable by construction. The first 4 bytes are a Unix timestamp, so sorting by _id gives you roughly chronological order. This means db.collection.find().sort({_id: -1}).limit(10) gives you the 10 most recent documents without a separate created_at field. However, "roughly" matters - two documents created on different servers in the same second may not be in strict order. For strict chronological ordering, use an explicit timestamp field.

Schema Flexibility: Power and Danger

MongoDB is "schema-flexible," not "schemaless." The distinction matters:

graph TD
    subgraph "Schema Flexibility Spectrum"
        RIGID["Rigid Schema (PostgreSQL)<br/>──────────────────<br/>CREATE TABLE users (<br/>  id SERIAL PRIMARY KEY,<br/>  name TEXT NOT NULL,<br/>  email TEXT UNIQUE NOT NULL,<br/>  age INTEGER CHECK (age > 0)<br/>);<br/><br/>Every row has exactly<br/>these columns, these types,<br/>these constraints<br/><br/>Changing schema requires<br/>ALTER TABLE (may lock table)"]
        
        FLEXIBLE["Flexible Schema (MongoDB)<br/>──────────────────<br/>Document 1: {name: 'Alice', age: 30}<br/>Document 2: {name: 'Bob', email: 'bob@ex.com'}<br/>Document 3: {name: 'Charlie', age: '25', status: 'active'}<br/><br/>Each document can have<br/>different fields<br/><br/>Same field can have<br/>different types across docs<br/>(age is int in doc 1,<br/>string in doc 3)<br/><br/>No ALTER TABLE needed<br/>for new fields"]
        
        VALIDATED["Schema Validation (Modern MongoDB)<br/>──────────────────<br/>db.createCollection('users', {<br/>  validator: {<br/>    $jsonSchema: {<br/>      required: ['name', 'email'],<br/>      properties: {<br/>        name: {bsonType: 'string'},<br/>        email: {bsonType: 'string'},<br/>        age: {bsonType: 'int', minimum: 0}<br/>      }<br/>    }<br/>  }<br/>})<br/><br/>MongoDB CAN enforce schema<br/>— most teams should use this"]
    end

This diagram shows the schema flexibility spectrum. PostgreSQL enforces rigid schemas where every row has exactly the defined columns, types, and constraints, requiring ALTER TABLE for changes. MongoDB's flexible schema allows each document to have different fields and even different types for the same field across documents, with no migration needed for new fields. Modern MongoDB supports schema validation with JSON Schema rules that can enforce required fields, types, and constraints - most production teams should use this to prevent data quality drift.

Warning: "Schemaless" is the most dangerous word in MongoDB's marketing history. Every application has a schema - the question is whether it's enforced by the database or by convention in application code. When the database doesn't enforce it, schema drift happens: one team writes age as an integer, another writes it as a string, a third writes it as null, a fourth doesn't include it at all. Over two years, you have four different shapes for the same field, and every query needs defensive code to handle all of them. Use MongoDB's schema validation in production. Always.

The Production War Story

At Amazon, I worked with a team that had chosen MongoDB for a product metadata service. The reasoning was sound at the time: product metadata varies dramatically by category. A laptop has processor, RAM, and screen size. A shirt has fabric, size, and color. A book has author, ISBN, and page count. In a relational database, you'd either have a massive sparse table (hundreds of columns, most NULL for any given product) or an EAV (entity-attribute-value) pattern that's performant nightmare to query.

MongoDB's document model seemed perfect. Each product was a document with a flexible set of attributes based on its category. The initial build was fast - no schema migrations, no ALTER TABLE, no EAV joins. The team shipped in 6 weeks.

The problems emerged 18 months later. Two hundred engineers across 30 teams were writing to the same collection. Without schema validation (this was pre-validation MongoDB), the price field was sometimes a number (99.99), sometimes a string ("99.99"), sometimes an object ({amount: 99.99, currency: "USD"}), and sometimes missing entirely. Queries that assumed price was a number silently returned wrong results for string-priced documents. Aggregation pipelines broke when encountering unexpected types. Nobody knew which format was "correct" because there was no schema to be correct against.

The fix took 6 months: implement schema validation for all collections, write a migration script to normalize the 800 GB of inconsistent data, and add application-level validation as a defense-in-depth layer. The team later evaluated whether PostgreSQL with JSONB columns would have been a better choice - and concluded yes, for their specific workload. The relational columns (id, title, category, created_at) enforced by the database, plus a JSONB column for category-specific attributes, would have given them both schema safety and flexibility.

The lesson isn't that MongoDB was the wrong choice - it's that MongoDB without schema validation and data governance is a time bomb for any team larger than five people.

Six Takeaways

MongoDB bets that your data is documents. Self-contained, hierarchical objects that map to application data structures. When this bet matches your data shape, reads are fast (no joins) and development is fast (no O/R mapping).
Embedding is the power move, but it has limits. Embed when data is one-to-few, always read together, and doesn't exist independently. Use separate collections for unbounded arrays, independently accessed data, and many-to-many relationships.
The 16 MB document size limit is a hard constraint. Unbounded embedded arrays will hit this wall. Design for bounded arrays from day one or use references to separate collections.
BSON is not JSON - it has native types. BSON supports dates, 64-bit integers, decimal128, ObjectId, and binary data. Use these types instead of storing everything as strings. ObjectId is time-sortable by construction.
Use schema validation in production, always. "Schemaless" means your schema is defined by convention in application code, which inevitably drifts when multiple teams write to the same collection. MongoDB's JSON Schema validation prevents this drift at the database level.
The document model doesn't eliminate schema design - it changes what you design. Instead of designing tables and foreign keys, you design document boundaries and embedding vs referencing decisions. This is equally complex and equally consequential.

Exercises

Exercise: Starter: You're building a blog platform with users, posts, and comments. (a) Design the MongoDB document structure. Should comments be embedded in the post document or in a separate collection? What's the maximum number of comments before embedding becomes problematic? (b) A post has tags (array of strings). A user has followers (array of user IDs). Which of these is safe to embed and which might grow unbounded? (c) Write the MongoDB schema validation rule that requires every post document to have title (string), author_id (objectId), created_at (date), and content (string), with an optional tags field that must be an array of strings.

Exercise: Intermediate: You're designing a product catalog for an e-commerce platform with 10 million products across 500 categories. Each category has different attributes (electronics: processor, RAM, screen_size; clothing: fabric, size, color). (a) Design the document structure. Use common fields at the top level and category-specific attributes in a nested object. Show example documents for three different categories. (b) Compare this design to three relational alternatives: single wide table, EAV pattern, and PostgreSQL with JSONB. For each, analyze: query performance for "find all laptops with > 16 GB RAM," schema enforcement capability, and development velocity. (c) Your team decides on MongoDB. Six months later, you need to add a "price_history" array to every product (one entry per price change). Some products have 500+ price changes. Evaluate the impact on document size and design the migration strategy.

Exercise: Advanced: You inherit a MongoDB deployment with 200 million documents across 50 collections. The system has been in production for 3 years with no schema validation. (a) Write a diagnostic query using $type aggregation to find fields with inconsistent types across documents in a collection (e.g., price is sometimes int, sometimes string, sometimes missing). (b) Design the schema validation migration plan: which collections to validate first, how to handle existing documents that violate the schema (validationAction: "warn" vs "error"), how to backfill and normalize inconsistent data, and the rollback plan if validation breaks writes from services you don't control. (c) The CTO asks: "should we migrate to PostgreSQL?" Evaluate for their specific workload (high-write product catalog, 500 categories, 200 engineers, 30 microservices). Build the honest pro/con matrix and make a recommendation with a decision framework, not a yes/no.

Back to System Design Advanced