This incident helped us to think through how we should be using AWS and here is what we have come up with:
1. Always use elastic IP (behaves like a static ip) to point to your EC2 instance. You can then quickly swap instances without needing to make any DNS changes.
2. Always use EBS (elastic block storage) instead of the storage that comes with an EC2 instance to store your Database data . You can then quickly reattach your data store to a new instance as well as you will not loose your data in the event of EC2 instance failure and have your data automatically replicated.
3. Spread your EC2 instances to different availability zones and regions instead of putting them all in the same availability zone (a physical data center). You can then protect your application from single data center failure.
4. Backup (incremental snap shots) your database from EBS to S3 (Simple Storage Service), which is automatically replicated across multiple availability zones.
5. Create a custom AMI (Amazon Machine Image) for your application from a pre-configured public AMI. You can then bring up a new instance quickly (in less than 15 mins) on-demand and even completely automate server provisioning.
6. Rollout your new release to a new instance(s) when you need to change your AMI (i.e., when you upgrading your DB, OS, Application Server, etc.) and then reattach your elastic ip and EBS to this new instance. You can then quickly rollback to the old instance should you encounter any problem with the new release.Figure: How to transparently switch to a new instance
7. If budget permits, use cloud management tool like RightScale.
With a few practices in place, you can have your own virtual data center (on Amazon cloud) that was only available to large corporations.