Comprehend MongoDB TTL Index Completely

In this article, we will examine the special type of indexing feature available in MongoDB. TTL indexes are single-field indexes that MongoDB can use to mechanically remove the documents from a collection after a certain amount of time or at a specific clock time.

Data expiration is useful for cocksure types of information like system-generated event data, session, and log information that only need to persist in a database for a finite amount of time.

Background Behaviour

Expiration of Data

TTL indexes expire documents after the specified number of seconds has passed since the indexed field value. The expiration threshold is the indexed field value plus the specified number of seconds.

If the field is an array, and there are multiple date values in the index, MongoDB uses the lowest (earliest) date value in the array to calculate the expiration threshold.

For time series collections, TTL indexes also remove a bucket of data when all documents inside it expire. This is equal to the upper timestamp limit of the bucket plus the expireAfterSeconds value. For example, if a bucket covers data up until 2023-03-27T18:29:59Z and expires after seconds is 300, the TTL index expires the bucket after 2023-03-27T18:34:59Z.

If the indexed field in a document is not a date or an array that holds one or more data values, the document will not expire.

If a document does not contain the indexed field, the document will not expire.

Delete operations

A background thread in Mongod reads the values in the index and removes expired documents from the collection.

When the TTL thread is active, you will see delete operations in the output of db.currentOp() or in the data collected by the database profiler.

Starting in MongoDB 6.1

To improve efficiency, MongoDB may batch multiple document deletions together.

The explain command results contain a new BATCHED_DELETE stage for batched document deletions.

If a time series collection contains documents with time field timestamps before 1970-01-01T00:00:00.000Z or after 2038-01-19T03:14:07.000Z, no documents are deleted from the collection by the TTL "time to live" feature.

Timing of the delete operation

MongoDB begins removing expired documents or time series buckets as soon as the index finishes building on the primary. For more information on the index build process, see Index Builds on Populated Collections.

The TTL index does not guarantee that expired data is deleted immediately upon expiration. There may be a delay between the time that a document expires and the time that MongoDB removes the document from the database.

The background task that removes expired documents runs every 60 seconds. As a result, documents may remain in a collection during the period between the expiration of the document and the running of the background task. MongoDB starts deleting documents 0 to 60 seconds after the index completes.

Because the duration of the removal operation depends on the workload of your Mongod instance, expired data may exist for some time beyond the 60-second period between runs of the background task.

The delete operations initiated by the TTL task run in the foreground, like other deletes.

Replica Sets

On replica set members, the TTL background thread only deletes documents when a member is in state primary. The TTL background thread is idle when a member is in state secondary. Secondary members replicate deletion operations from the primary.

Example

Let’s create a collection called logs. which having different data having expiration time after 10 and 15 min.

Review the created logs collection.

Create the TTL index on the collection field ‘CreatedDate’ using the below command. Make sure the field is date data type only.

db.logs.createIndex( { "CreatedDate": 1 }, { expireAfterSeconds: 30 })

Check data deletion after the expiration time. In the below screen print, it’s not yet deleted. Because we have set 30 seconds after the expiration time, and still, it’s not a gap of 30, so it’s not deleted. In the below case, it should be deleted after “2024-06-10T07:43:47.037Z”.

Now let’s check after the expiration time + expire After seconds. It’s deleted.

Restrictions

• TTL indexes are single-field indexes. Compound indexes do not support TTL and ignore the expireAfterSeconds option.
• You can only create TTL indexes for a time series collection on the collection time field.
• The _id field does not support TTL indexes.
• You cannot create a TTL index on a capped collection.
• If a non-TTL single-field index already exists for a field, you cannot create a TTL index on the same field since you cannot create indexes with the same key specification and differ only by the options. To change a non-TTL single-field index to a TTL index, use the collMod database command.