Uncategorized

Porting my brain from Python to Go

Over the past few months, I have made it a goal to get down and dirty with Go. I have been getting more and more involved with Kubernetes lately and feel like its time that I pick up its native language. For the last 6 years, I’ve had Python as my go-to language with Ruby sneaking in when the time is right. I really enjoy Python and feel extremely productive in it. In order to get to some level of comfort with Go I knew that I would have to take a multi-faceted approach to my learning.

Obviously, the first thing I did was just get it installed on my laptop. Being a Mac and Homebrew user, I simply ran: brew install golang. So now I had the toolchain installed and could compile/run things on my laptop. Unfortunately ‘Hello World’ was not going to get me to properly port my existing programming skills into using Go as a primary/native language. I knew enough about Go to know that I need to understand how and why it was built. Additionally, I had been in interpreted language land my whole life (Perl->Python->Ruby) so was taking another leap outside of just changing dialects. At this point I took a step back from just poking at source code to properly study up on the language.

I’ve had a Safari Books Online subscription for quite some time and have leveraged it heavily when learning new technologies. This case was no different. I picked up Programming in Go: Creating Applications for the 21st Century. I found this book to be at the right level for me, existing programming experience but new to Go. I also found that there was enough context as to how to do things the Go way such that I wouldn’t just be writing a bunch of Python in a different language. After a few flights browsing the concepts, I started to read it front to back. At the same time, I found a resource specific to my task at hand, Go for Python Programmers. This was a good way to see the code I was accustomed to writing and how I might write it differently in Go. As I continued reading and studying up, I made sure to pay particular attention when looking at Go code in the wild. The ask of myself was really to understand the code and the idioms, not just glance at it as pseudocode.

There were a few things that I needed some more clarification on after reading through my study materials. I was still confused about how packaging worked in practice. For clarification, my buddy and Go expert Evan Brown pointed me to the Go Build System repository. The thing that clicked for me here was the structure of having a repository of code that is related to each other, splitting out libraries into directories, then using the cmd directory in order to make the binaries that tied things together. This repo also has a great README that shows how they have organized their code. Thanks Go Build Team!

The next thing that I needed to hash out was how exactly I would use my Object Oriented penchant in Go. For this I decided to turn to Go by Example which had a wealth of simple example code for many of the concepts I was grappling with. Things started to click for me after looking at  structsmethods, and interfaces again through a slightly different lens.

Sweet! So now I understood (more or less) how the thing worked but I hadn’t built anything with it. The next phase was figuring out how the rubber met the road and having a concrete task to accomplish with my new tool.

I didn’t have any projects off the top of my head that I could start attacking but remembered that a few moons ago I had signed up for StarFighters, a platform that promised to provide a capture the flag style game that you could code against. I looked back through their site and noticed that their first game, StockFighter, had been released. StockFighter provides a REST API that players can code against in order to manipulate a faux stock market. I didn’t know anything about the stock market but figured this would be as good a task as any to get started. I played through the first few levels by creating a few one off binaries. Then on the harder levels I started to break out my code, create libraries, workers and all kinds of other pieces software to help me complete the tasks that  StockFighter was throwing at me. One huge help in getting me comfortable with creating this larger piece of software was that I had been using Pycharm with the Go plugin. This made code navigation, refactoring, testing and execution familiar.

Shit had gotten real. I was building a thing and feeling more comfortable with each level I played.

After my foray with StockFighter, I felt like I could use a different challenge. It turns out if you start asking around if people need some software built at Google, there will be plenty of people who want to take you up on the offer. The homie Preston was working on an IoT demonstration architecture and needed a widget that could ingest messages from PubSub and then store the data in BigTable. As he explained the project, I told him that it should take me no more than 4 hours to complete the task. I hadn’t use the Go SDKs at the time so I figured that would eat up most of my time. I sat down that afternoon and started the timer. This was the litmus test. I knew it. My brain knew it. My fingers knew it.

I made it happen in just under 5 hours which to me was a damn good effort as I knew that generally estimates are off by 2x. After doing a little cross-compiling magic, I was able to ship Preston a set of binaries for Mac, Linux and Windows that achieved his task.

I’m certainly not done with my journey but I’m happy to have had small successes that make me feel at home with Go.

 

Standard
Uncategorized

Building a container service with Mesos and Eucalyptus

  marathon-blog
    Over the past few months, I’ve been digging into what it means to work with a distributed container service. Inspired by Werner Vogel’s latest post about ECS, I decided to show an architecture for deploying a container service in Eucalyptus. As part of my investigations into containers, I have looked at the following platforms that provide the ability to manage container based services:
Each of these provide you with a symmetrical (all components run on all hosts) and scalable (hosts can be added after initial deployment) system for hosting your containerized workloads. They also include mechanisms for service discovery and load balancing. Deis and Flynn are both what I would call a “lightweight PaaS” akin to a private Heroku. Mesos, however, is a more flexible and open ended platform, which comes as a blessing and a curse. I was able to deploy many more applications in Mesos but it took me far longer to get a working platform up and running.
     Deis and Flynn are both “batteries included” type systems that once deployed allow you to immediately push your code or container image into the system and have it run your application. Deis and Flynn also install all of their dependencies for you through automated installers. Mesos on the other hand requires you to deploy its prerequisites on your own in order to get going, then requires you to install frameworks on top of it to make it able to schedule and run your applications.
     I wanted to make a Mesos implementation that felt as easy to make useful as Deis and Flynn. I have been working with chef-provisioning to deploy clustered applications for a while now so I figured I would use my previous techniques in order to automate the process of deploying a functional and working N node Mesos/Marathon cluster. Over the last month, I have also been able to play with Mesosphere’s DCOS so was able to get a better idea of what it takes to really make Mesos useful to end users. The “batteries included” version of Mesos is architected as follows:
container-service
Each of the machines in our Mesos cluster will run all of these components, giving us a nice symmetrical architecture for deployment. Mesos and many of its dependencies rely on a working Zookeeper as a distributed key value store. All of the state for the cluster is stored here. Luckily, for this piece of the deployment puzzle I was able to leverage the Chef community’s Exhibitor cookbook which got my ZK cluster up in a snap. Once Zookeeper was deployed, I was able to get my Mesos masters and slaves connected together and was able to see my CPU, memory and disk resources available within the Mesos cluster.
mesos-resources
    Mesos itself does not handle creating applications as services so we need to deploy a service management layer. In my case, I chose Marathon as it is intended to manage long running services like the ones I was most interested in deploying (Elasticsearch, Logstash, Kibana, Chronos). Marathon is run outside of Mesos and acts as the bootstrapper for the rest of the services that we would like to use, our distributed init system.
     Once applications are deployed into Marathon it is necessary to have a mechanism to discover where other services are running. Although it is possible to pin particular services to particular nodes through the Marathon application definition, I would prefer not to have to think about IP addressing in order to connect applications. The preferred method of service discovery in the Mesos ecosystem is to use Mesos DNS and host it as a service in Marathon across all of your nodes. Each slave node can then use itself as a DNS resolver, wherein queries for services  get handled internally and all others are recursed to an upstream DNS server.
     Now that the architecture of the container service is laid out for you, you can get to deploying your stack by heading over to the README. This deployment procedure will not only deploy Mesos+Marathon but will also deploy a full ELK into the cluster to demonstrate connecting various services together in order to provide a higher order one.
Standard
Uncategorized

Introducing HuevOS+RancherOS

huevosArch

Today is an exciting day in Santa Barbara. We are very pleased to introduce our latest innovation to the world of DevOps. 

HuevOS – the Docker-based open-source operating system for tomorrow’s IT and Dev/Ops professional.  HuevOS 1.0 (SunnySide) is the open-source/free-range/gluten-free solution that forms the perfect complement to RancherOS.  In addition, we’re delighted to begin development on our proprietary blend of Services and Language Software as a Service (SaLSaaS) which, when overlayed atop a HuevOS+RancherOS stack, provides a complete and delicious solution around which your whole day can be centered.

Try HuevOS+RancherOS today and let us know which SaLSaaS we should work on first to ensure your hunger for DevOps is quenched thoroughly.  

To get your first taste visit the following repository which includes our Chef Recipe and a Vagrantfile to get you HuevOS in short order:

https://github.com/viglesiasce/huevos-cookbook.git

If you already have your RancherOS host up its easy to add in our HuevOS to the mix via the Docker Registry:

docker pull viglesiasce/huevos; docker run huevos

A huge thank you to all involved in getting us to this point and being able to ship a 1.0 version of the HuevOS+RancherOS platform.

Happy clucking!!

huevOS

Standard
Uncategorized

Installing Apache Hadoop on Eucalyptus using Hortonworks Data Platform stack and Apache Ambari

Get your #hortonworks #hadoop running on #eucalyptus

shaon's blog

This post demonstrates Hadoop deployment on Eucalyptus using Apache Ambari on Horton Data Platform stack. Bits and pieces:

  1. Eucalyptus 4.0.0
  2. Hortonworks Data Platform (HDP) stack
  3. Apache Ambari
  4. 4 instance-store/EBS-backed instances
    1. 2 vcpus, 1024MB memory, 20GB disk space
    2. CentOS 6.5 base image

For this is demo we tried to use very minimum resources. Our Eucalyptus deployment topology looks like below, 1x (Cloud Controller + Walrus + Storage Controller + Cluster Controller) 1x (Node Controller + Object Storage Gateway) To meet our instance requirement, we changed the instance-type according to our need.

Preparation Run an instance of m1.xlarge or any other instance type that meets the above requirement. When the instance is running copy the keypair that is used to run this instance at .ssh/id_rsa, we will be using this same keypair for all the instances.

Run three more instance of same type with same keypair and image and copy their private IPs for later use. Add security…

View original post 200 more words

Standard
Uncategorized

Pursuit of Quality at Eucalyptus

Overview of the extensive Eucalyptus quality toolchain.

shaon's blog

At Eucalyptus, if one thing we care about it is the quality of the product. Our goal is to deliver an AWS compatible software that just works. To ensure the highest level of quality we try to follow the optimum strategy possible in both Development and in QA.

We have adopted Agile Software Development model a while back and we love it. As a part of our strategy, there are several type of projects we open in Jira. After getting the green signal from Product Management, we get Epic and then after architectural discussion Story tickets are created with Sub-tasks for developers to work on. We also have Jira tickets for New Features which are basically similar to Story but comparatively smaller tasks and generally have no Sub-tasks. At this point one scrum master from each team works closely with the developers to keep track of the…

View original post 629 more words

Standard
Uncategorized

Hobos and Vagrants and Quality

Flattering blog post by the homie gregdek. Go get your micro-qa on!

Greg DeKoenigsberg Speaks

I don’t know how many people have actually met Vic Iglesias, our Quality Hobo.  Here’s what he looks like in his natural habitat:

It’s really super important not to make Quality Hobo angry. Let me assure you from personal experience: no one wants that.

Here are some things that make Quality Hobo angry (and that you should, therefore, avoid):

  • Stealing Quality Hobo’s cigarettes, electronic or otherwise.
  • Remarking upon Quality Hobo’s resemblance to a certain celebrity.
  • Wasting Quality Hobo’s time with questions about which test cases are most recent.
  • Wasting Quality Hobo’s time by writing tests that don’t integrate into Eutester.
  • Wasting Quality Hobo’s time with questions on how to build your QA environment.
  • Wasting Quality Hobo’s time in any way at all.

Here’s what I’m saying: it was only a matter of time before our Hobo got mixed up with a Vagrant.

Vagrant is incredible, and Mitchell Hashimoto is incredible for creating it.  It’s the…

View original post 391 more words

Standard