Push old data to S3
Use S3 to store data – makes sense
I love it when you find a whole sever and script that I totally forgot existed! This little script works best with its own dedicated server as it will mount a remote location (in my usage a NAS folder) which has all the old data that needs to be archived well that’s the point of it. So I use this on a weekly basis, I will put all the files I don’t need on my first tier storage into the s3upload folder (which can be around 500g) then I log onto the server with the script, run it and wait for the email confirmation. The reason I like having a ‘shove’ server for this is so - one I have enough space to offset all the data and secondly I can change the script to mount multiple locations also it can the upload process can be cpu intensive depending on the data being uploaded.
What you need to know about the script – you will need to have a s3 account and I strongly recommend getting one if you have lots of data that needs to be stored somewhere cheap and secure. I probably will write a blog post about amazon s3 and the advantages to be found. Right back to the script… once you have a S3 account you will need to install s3cmd (command line S3) and configure this with you private keys for your buckets, to get s3cmd apt-get update && apt-get install s3
The script works by mounting a remote location and this doesn’t have to be remote location at all, it can be any location on your disk so if that’s the case you will just need to get rid of the mount line. We then run s3cmd to push all data located in SOURCE to the s3 TARGET bucket also with have 2 log files being outputted to show output and errors then the script will email once the shove has been completed. If any errors are found then the error log will be emailed but warning on the errors found this can be false positives depending on the permissions of the files.
smbmount $SHARE $SOURCE -o username='shove',password='letmeout',rw,uid=shove
cat /dev/null > /tmp/s3-errors.txt
nohup s3cmd sync $SOURCE $TARGET >> /var/log/s3-raw-out.txt >> /tmp/s3-errors.txt
if [ -s '/tmp/s3-errors.txt' ]; then
mail -s "Shove Errors on $HOSTNAME - could just be group permissions" < /tmp/s3-errors.txt $MAIL_ADDR
echo "Shove successful, no errors reported." | mail -s "Shove Report for $HOSTNAME" $MAIL_ADDR