Archive Dynamodb based on date/days
<p>I want to archive dynamodb table, keeping data only for 90 days. I have a field called recorded_on in the table which I can use to track 90days. Looked at Datapipeline and it seems little overkill with EMR since we don't need it. Any better ways to do this? </p>

<pre><code>1. Cronjob that will continue to run everyday and match recorded_on + 90days &gt; today's date and put those rows in s3 and delete those rows.

2. Separate cronjob to put data from s3 to redshift everyday.
</code></pre>

<p>Why do you think using AWS data pipeline is overkill? You can use custom job but it will require additional work which pipeline does it for you automatically. </p>

<p>The fact that it uses EMR cluster behind the scenes shouldn't be a problem as its details are anyway abstracted away from you. Setting up pipeline to archive dynamoDb to s3 is very easy. For deleting data older than 90 days you can write a custom script &amp; use Data Pipeline ShellCommandActivity (<a href="http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-shellcommandactivity.html" rel="nofollow">http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-shellcommandactivity.html</a>) to execute it.</p>

<p>Here are some benefits of Data Pipeline over CRON:</p>

<ol>
<li>Retries in case of failures.  </li>
<li>Monitoring/Alarms.</li>
<li>No need to provision EC2, AWS takes care of everything behind the scenes.  </li>
<li>Control how much dynamoDb capacity the export can use, this is very important in preventing the export job from impacting other systems.</li>
</ol>

<p>Its also very cheap, <a href="https://aws.amazon.com/datapipeline/pricing/" rel="nofollow">https://aws.amazon.com/datapipeline/pricing/</a>. </p>

<p>Regards
Dinesh Solanki </p>

python amazon-s3 amazon-dynamodb boto3 amazon-data-pipeline