After having run Hadoop on-premise in production for some time, we decided to build a Hadoop platform in AWS to extend the on-premise Hadoop cluster to a hybrid platform.
In this presentation, we first briefly state our motivation and requirements for building a cloud platform. Moving to the cloud not only offers new technical possibilities, it also helped us to make your way of working more agile. We explain how we setup a team of internal and external experts, defined an agile working mode and how this approach worked for us.
Then we explain our choice of tools (Terraform, Ambari Blueprints) and how we used them to automate the deployment of our cloud platform. We also take a quick glance into Cloudbreak. We share what worked well but we also mention some of the pitfalls we encountered.
Finally we look into how we connected the cloud and on-premise platform to build a hybrid platform. Here we look especially into user management and our Kerberos setup. We conclude our presentation by giving a glimpse into the next steps on our roadmap.