How to get started with pymongo for MongoDB interaction in Python

All interaction with a running MongoDB server process begins with the instantiation of a class from the pymongo library. This class, aptly named MongoClient, is the conduit through which all commands and data flow. To create an instance of this class, one must provide it with a connection Uniform Resource Identifier, or URI. This string specifies the protocol, host, and port of the MongoDB instance to which we wish to connect.

For a standard MongoDB server running on the local machine, the default port is 27017. The client can be instantiated as follows:

from pymongo import MongoClient

# A connection to a MongoDB server on localhost, port 27017
client = MongoClient('mongodb://localhost:27017/')

It is important to understand what this client object represents. It is not a single, persistent network connection. Rather, it is a sophisticated manager for a pool of connections to the database. The pymongo driver handles the complexities of opening, closing, and reusing these connections as needed to fulfill your program’s requests efficiently. For the developer, the client is simply the gateway to the MongoDB server itself, which may host numerous independent databases.

Once a MongoClient instance is available, you can reference a specific database by name. PyMongo offers two convenient syntaxes for this purpose: attribute-style access and dictionary-style access.

# Attribute-style access to a database named 'catalog'
db = client.catalog

# Dictionary-style access to the same database
db = client['catalog']

The attribute-style access is often preferred for its brevity and clean appearance. However, should a database name contain characters that are not valid in Python identifiers (such as a hyphen) or if the name conflicts with a method of the MongoClient object, the dictionary-style syntax becomes necessary. For instance, a database named test-db could only be accessed via client['test-db'].

A crucial concept in MongoDB is that of implicit creation. Simply executing the line db = client.catalog does not create the catalog database on the server. The db object you receive is merely a handle, a local Python object that represents a potential database on the server. MongoDB operates on a “create-on-first-use” basis. The database itself, along with its constituent collections, will only spring into existence on the server when the very first document is inserted into a collection within that database. This behavior can initially be perplexing, as one cannot see the new database appear in administration tools immediately after acquiring the database object in code. This lazy-creation approach is fundamental to MongoDB’s design, promoting a flexible and dynamic development workflow. The client object, therefore, provides access to any database, whether it currently exists or not.

How documents come to be

With a handle to a database, the next logical step is to reference a collection within it. A collection in MongoDB is analogous to a table in a relational database; it is a grouping of documents. Much like accessing a database, PyMongo provides both attribute-style and dictionary-style access to collections.

# Attribute-style access to a collection named 'products'
products_collection = db.products

# Dictionary-style access to the same collection
products_collection = db['products']

Again, the collection object, products_collection, is merely a local representation. The products collection does not yet exist on the server. It will be created only when a document is first inserted into it.

A document in MongoDB is a structure composed of field-and-value pairs, conceptually similar to a JSON object. In the world of Python, the most natural representation for such a structure is the dictionary. To insert data, we first construct a Python dictionary that represents the document we wish to store.

# A Python dictionary representing a product
new_product = {
    "sku": "10-234-01",
    "name": "7-inch Pliers",
    "description": "A versatile tool for gripping and cutting.",
    "price": 12.99,
    "stock": 42,
    "tags": ["tools", "hand-tools", "steel"]
}

To insert this single document into our collection, we call the insert_one() method on the collection object. This method takes the dictionary as its argument.

result = products_collection.insert_one(new_product)

Upon successful execution, the insert_one() method returns an instance of the InsertOneResult class. This object is not the document itself but rather an acknowledgment from the server containing information about the operation. Of particular importance is the inserted_id attribute of this result object. It holds the unique identifier, or _id, of the newly created document.

print(f"Document inserted with _id: {result.inserted_id}")

If you inspect the document you provided, new_product, you will notice it lacks an _id field. If a document is inserted without a top-level _id key, MongoDB automatically adds one. This generated value is an ObjectId, a special 12-byte type that guarantees uniqueness across a distributed system. The value is constructed from a timestamp, a machine identifier, a process ID, and a counter, ensuring a very high probability of being globally unique. You can also specify your own _id. The value must be unique within the collection. For instance, using the product’s SKU as the _id might be a valid design choice.

product_with_custom_id = {
    "_id": "11-555-07",
    "name": "Adjustable Wrench",
    "price": 18.50,
    "stock": 25
}
result = products_collection.insert_one(product_with_custom_id)
# result.inserted_id will be "11-555-07"

For inserting multiple documents in a single operation, PyMongo provides the insert_many() method. This is far more efficient than calling insert_one() in a loop, as it requires only one round-trip to the database server. The method expects an iterable, typically a list, of dictionaries.

new_inventory = [
    {
        "sku": "99-001-22",
        "name": "Hammer",
        "price": 21.00,
        "stock": 15
    },
    {
        "sku": "99-002-87",
        "name": "Screwdriver Set",
        "price": 35.75,
        "stock": 30,
        "tags": ["tools", "set"]
    }
]

many_result = products_collection.insert_many(new_inventory)

The insert_many() method returns an instance of InsertManyResult. Instead of a single inserted_id, this object contains a list of identifiers in its inserted_ids attribute, corresponding to each document in the list that was provided. The order of the IDs in the inserted_ids list matches the order of the documents in the input list.

Asking questions of the database

Having populated a collection with documents, the next logical operation is to retrieve them. MongoDB offers a rich and expressive query language for this purpose. The simplest form of retrieval is to fetch a single document that matches a specific set of criteria. For this, PyMongo provides the find_one() method on the collection object.

The find_one() method accepts a single argument: a dictionary that specifies the query filter. The structure of this filter document mirrors the structure of the documents you wish to find. To locate the document for the product with the SKU “10-234-01”, you would construct a filter like so:

query_filter = {"sku": "10-234-01"}
found_product = products_collection.find_one(query_filter)

If a document matching the filter is found in the collection, find_one() returns it as a Python dictionary. This dictionary includes the _id field, whether it was generated by MongoDB or provided by the user. If no document matches the specified filter, the method returns None. This makes it straightforward to check for the existence of a result.

While find_one() is useful for locating a unique item, it is more common to retrieve a set of documents that match a query. The method for this task is find(). Like its singular counterpart, find() takes a filter document as its primary argument. However, its return value is fundamentally different.

Instead of returning a list of dictionaries, the find() method returns a Cursor object. A cursor is a pointer to the result set of a query. It is an iterable object, meaning you can loop over it, but it does not fetch all the matching documents from the server at once. Instead, it retrieves the documents in batches as your program iterates through the cursor. This lazy-loading behavior is highly efficient, especially for queries that might return thousands or even millions of documents, as it avoids loading the entire result set into the client’s memory.

To access the documents, one typically uses the cursor in a standard Python for loop:

# Find all documents in the collection by passing an empty filter
all_products_cursor = products_collection.find({})

for product in all_products_cursor:
    print(product['name'])

Passing an empty dictionary {} as the filter is the canonical way to specify that all documents in the collection should be matched.

The true power of MongoDB’s query language is revealed through its query operators. These special keys, which always begin with a dollar sign ($), allow for more complex comparisons than simple equality. For example, to find all products with a stock level greater than 20, we would use the “greater than” operator, $gt.

The syntax for using an operator involves a nested dictionary where the outer key is the field to be tested, and the inner dictionary contains the operator and its corresponding value.

# Find products where the 'stock' field is greater than 20
high_stock_query = {"stock": {"$gt": 20}}
high_stock_products = products_collection.find(high_stock_query)

for product in high_stock_products:
    print(f"{product['name']}: {product['stock']} units")

MongoDB provides a wide array of such operators, including $lt (less than), $lte (less than or equal), $ne (not equal), and $in (matches any value in a specified array). The $in operator is particularly useful for finding documents where a field’s value is one of several possibilities. This same logic applies to querying array fields within a document. To find all products tagged as “tools”, the query is remarkably simple:

# Find all documents where the 'tags' array contains the element "tools"
tools_query = {"tags": "tools"}
tools_cursor = products_collection.find(tools_query)

MongoDB intelligently understands that when you query an array field for a single value, you are asking for any document where that value is an element of the array.

Often, an application does not require every field from the documents it retrieves. Fetching unnecessary data consumes network bandwidth and client-side memory. To address this, MongoDB queries support the concept of projection. A projection is a second, optional argument passed to find() or find_one() that specifies which fields to include in or exclude from the result.

The projection document consists of field names as keys and either a 1 to include the field or a 0 to exclude it. You cannot mix inclusion and exclusion modes in a single projection, with one notable exception: the _id field. By default, the _id field is always included in the results. If you are specifying fields to include, you must explicitly exclude the _id if you do not want it.

For example, to retrieve only the name and price for all products, while also omitting the default _id, the query would be:

# Query for all products, but only return the 'name' and 'price' fields
name_and_price_projection = {"name": 1, "price": 1, "_id": 0}
price_list = products_collection.find({}, name_and_price_projection)

for item in price_list:
    # Each 'item' dictionary will only contain 'name' and 'price' keys
    print(item)

This combination of a query filter to select specific documents and a projection to shape the content of those documents provides a powerful and efficient mechanism for data retrieval. For instance, to get the SKU and stock count for all products priced under $20:

low_price_filter = {"price": {"$lt": 20.00}}
stock_projection = {"sku": 1, "stock": 1, "_id": 0}

inventory_check = products_collection.find(low_price_filter, stock_projection)