AWS DynamoDB: Automatically delete old records using TTL

AWS DynamoDB storage is inexpensive. AWS charges only for number of Reads and Writes performed on the DynamoDB tables. But, sometimes too much data will causes the Throughput exceptions when we perform a Scan operation.

We can remove the old records which are not required from the table by using some script which delete the records matched to some criteria. But delete operation consume Write capacity.

So for this scenario, AWS provides a concept called Time-To-Live (TTL). TTL for DynamoDB allows you to define when items in a table expire so that they can be automatically deleted from the database without using any Write capacity units. We need to specifty some timestamp in the TTL attribute, so once the timestamp expires, the corresponding item is marked as deleted and removed from the table in next 48 hours.

TTL is useful if you have continuously accumulating data that loses relevance after a specific time period. If you have sensitive data that must be retained only for a certain amount of time according to contractual or regulatory obligations, TTL helps you ensure that it is removed promptly and as scheduled.

Enabling TTL

To enable TTL for a DynamoDB table, there should be a numeric DynamoDB table attribute populated with unix/epoch time to specify the expiration criteria. Next we need to enable the TTL setting on the table. Click on ManageTTL against "Time to live attribute" property for the table in the Overview tab.

Next enter the attribute name in the specified column and click on continue. 

Note: 

• Enabling TTL requires no additional fees.
• The TTL attribute should be numeric and we should store the epoch time only in that. If format is wrong, the value is ignored and the item won’t be deleted. TTL attribute is like any other attributes means we care read it, create indexes etc.
• Enable TTL will takes 1 hour to apply. So, once you have enabled TTL for an attribute, you will not able to disable / change in that time period.

Expiring Items

DynamoDB runs a background scanner that monitors all the items. If the timestamp has expired, the process will mark the item as expired and queue it for subsequent deletion. The expired item will delete in 48 hours from the table permanently. So if you read data from the table, it will return the items which are marked for deletion but not deleted from table yet. We need to filter the items using filter expression that returns only items where the Time To Live expiration value is greater than the current time in epoch format.

Changing / Disable TTL   

If you want to disable TTL for a table, just click on ManageTTL -> Disable.

To specify a different TTL attribute, you need to disable TTL for the table first, then you can re-enable TTL with a new TTL attribute. But disable or enable TTL will takes almost 1 hour to effect.

We can also Enable or Disbale the TTL attribute using AWS CLI.
For more information on AWS DynamoDB TTL, click here.

Gopikrishna