Why I stopped self-hosting Elasticsearch on Amazon EC2
Approximate time to read: 2 min
This is a sad story with a happy ending. Not the internet sort of happy ending but a traditional story where I’m happy at the end.
The story
The story starts with our need to create a centralised logging platform that allows us to search and filter results based on a log field (the user). Elasticsearch does searching and filtering really well and its integration with logstash makes the log aggregation side easy too. As our proxies are hosted on AWS EC2 instances I thought it made sense to spin up a few more instances to host our very own Elasticsearch cluster.
Elasticsearch is surprisingly easy to install, just follow a few internet tutorials and you’re the proud owner of a multi node search and indexing cluster. Pretty cool! With a bit of Ruby we had what we wanted, a personalised real time log viewer for all our Static IP users:
This is where the story gets sad. Elasticsearch is easy to install but a mystery to manage. During the setup process I had long hours of trying to work out why I had 5 Shards Unassigned and why 1 was initializing forever. My Cluster status was RED or YELLOW but there didn’t seem to be any way to get it GREEN without shutting it all down, losing all the data and starting again from scratch. I’m not an expert so there probably were ways to solve it but I couldn’t find them and I was seeing my time spiral down the drain hole. The final straw was when Amazon emailed me to say that one node on my cluster had been flagged as being part of a DDoS attack and I had to terminate it. I couldn’t face any more unassigned shards so called it a day on self-hosting.
Why be an expert in Elasticsearch when their are experts out there for a reasonable cost? Enter Elastic Cloud, hosted Elasticsearch by Elastic (the makers of Elasticsearch). I signed up and within 5 minutes had a two node cluster ready to go. I flipped my logstash config to point to this new cluster, bounced the service and watched the log messages start to roll in. No unassigned shards and my cluster health is GREEN. A happy ending.
The lesson
The lesson is if you can find an expert offering a service you need: use them. It might be cheaper in the short term to build and manage it yourself but when the pain starts and there’s nobody you can turn to its a very bad place to be. Are there other ways you could get a Static IP for your cloud app? Sure, but we’re the experts and we take the pain away for you just like Elastic Cloud have done for us and our Elasticsearch. Concentrate on building good products and let others take care of the rest.