How to cleanup Datasets based on expiry date?

info

• Follow the steps mentioned below.
• Total time taken for this task: 15 Minutes.
• Pre-requisites: User registration is completed, logged in to Amorphic and role switched

Tidbits

• Data cleanup option enables data deletion to save storage costs.
• This is based on the cleanup duration value set by the user.
• All the files which are past the expiration date will be removed permanently.
• Deleted files cannot be restored.

Create Dataset with cleanup option

• Click on 'DATASETS' --> 'Datasets' from left navigation-bar.
• Click on ➕ icon at the top right corner.
• Enter the following information

{
  "Dataset Name": "adp_ds_cleanup_<your_userid>"
  "Description": "This dataset demonstrates the file cleanup with a duration of 1 day."
  "Domain": "workshop(workshop)"
  "Data Classifications":
  "Keywords": "Retail"
  "Connection Type": "API (default)"
  "File Type": "jpg"
  "Target Location": "S3"
  "Update Method": "Append"
  "Enable Malware Detection": "No"
  "Enable AI Services": "No"
  "Enable Data Cleanup": "Yes"
  "Data Clean Up Duration": "1"
}

• Click on Files tab.
• Click on Upload Data icon at the top to upload few pics.
• Check the files next day. They will be permanantly deleted.

tip

• The expired dataset files are identified based on the upload date instead of the upload time.
• Example: If a file is uploaded on Aug 21, 4:55 PM and the data clean up duration is set to 2 days. The cleanup happens on Aug 23, 12:00 AM instead of Aug 23, 4:55 PM.