miniguide

Data Lifecycle Cheatsheet: Managing Retention and TTL for Storage

Introduction

This cheatsheet offers commands and configuration snippets to manage the lifecycle of data in storage systems. It covers setting TTL for Cosmos DB, configuring log retention, and automating data cleanup. It is designed for DevOps professionals focused on resource management and cost optimization.

Table of Contents

• Data Lifecycle Cheatsheet: Managing Retention and TTL for Storage
  • Introduction
  • Table of Contents
    • 1. Set TTL on Cosmos DB Container
    • 2. Query Data Based on TTL
    • 3. Configure Automatic Cleanup (Cosmos DB)
    • 4. Set Retention Policy on Azure Storage
    • 5. Configure Log Disk Retention
    • 6. Export Log Data
    • 7. Delete Old Blobs Automatically
    • 8. Monitor Storage Metrics
    • 9. Update TTL Settings via ARM
    • 10. Audit Data Lifecycle Settings

1. Set TTL on Cosmos DB Container

az cosmosdb collection create --collection-name MyCollection --db-name MyDatabase --resource-group <rg> --name <cosmosAccountName> --ttl 3600

2. Query Data Based on TTL

SELECT * FROM MyCollection c WHERE c._ts < (UnixTimestamp() - 3600)

3. Configure Automatic Cleanup (Cosmos DB)

Using Azure Functions trigger on TTL expiry.

// Configure Azure Function to trigger deletion of expired items.

4. Set Retention Policy on Azure Storage

az storage account management-policy create --account-name <accountName> --resource-group <rg> --policy @"policy.json"

(policy.json should contain retention rules)

5. Configure Log Disk Retention

az monitor diagnostic-settings create --name "RetentionPolicy" --resource <resourceId> --logs '[{"category": "AuditLogs", "enabled": true, "retentionPolicy": {"days": 30, "enabled": true}}]'

6. Export Log Data

az monitor log-analytics query --workspace "<workspaceId>" --query "AzureDiagnostics | where TimeGenerated > ago(7d)" --output table

7. Delete Old Blobs Automatically

az storage blob delete-batch --source <containerName> --pattern "*.*" --if-unmodified-since "2022-01-01T00:00:00Z"

8. Monitor Storage Metrics

az monitor metrics list --resource <storageResourceId> --metric "Transactions" --output table

9. Update TTL Settings via ARM

az deployment group create --resource-group <rg> --template-file ttlTemplate.json

10. Audit Data Lifecycle Settings

az monitor diagnostic-settings list --resource <resourceId> --output table