GSoC'19 Report - CNCF/Prometheus

2019/08/26

This blogpost summarizes the work done by me in the summer of 2019 as part of Google Summer of Code under the organization CNCF/Prometheus, I plan to keep contributing to Prometheus as a regular contributor after GSoC period.

Huge thanks Krasi Georgiev(Mentor) for being extremely helpful and supportive throughout this SoC. Most of the changes mentioned in this blogpost are not deployed to prombench.prometheus.io yet

Some silly Prometheus metrics, just for fun 😃 :

> gsoc_info{student="geekodour"}[12w]
> gsoc_info{student="geekodour",issues="9",prs="48"}

Just to recap, here are the tasks that were proposed to be worked on, but during the coding period I ended up working on various other issues due to which I might have missed some of the proposed goals:

prometheus/prometheus

prometheus/prombench

Things didn’t go as planned 🕥

Things didn’t go as planned in the proposal. The main goal with Prombench was to improve it infrastruture wise and enabling Prometheus developers to get more out of it and fix pending issues, there has been significant progress for this primary goal.

The 🐘 elephant in the room was to switch to GitHub Actions from Prow. This decision was taken because, even though Prow is an amazing CI system, it needed additional maintenance work and had a lot of moving parts.

Prombench currently runs a version of Prow that’s about one year old, using a custom built hook plugin: benchmark.go, I spent some time writing an external plugin for Prow which did not come of any use because we decided to switch to GitHub Actions anyway.

New Tools ✂️

With GitHub Actions we are no longer using that plugin, instead we built commentMonitor which I believe is an excellent tool for any project using GitHub Actions. It extracts the necessary variables needed by prombench.

Also added one missing part from Prombench, the Alertmanager. Since there’s no direct Github Reviver for AM, built a webhook server amGithubNotifier which takes in an alert and sends it to the respective PR on GitHub as comments. We’re using this to send alerts when any test is running for more than 3 days to save some cost.

New way to run Prombench tests 🌌

Looking at how the Kubernetes team is running their test-infra, I thought it would be good to try running Prombench tests as a StatefulSet, it turned out to be great! Now Prombench supports restarts with different version and with newer commits with the same command.

Oh Github Actions. 🎬

So after all this, we were ready to test everything we did with Github Actions but the real 🧱 brick on the head 🤕 fell on 9th of August’19 when GitHub decided to roll out a compeletely new Github Actions. Heck, they changed the expression language to YAML from HCL. It has quite a few bugs in it but I am not complaining because it is still in beta and the currently running features serves the usecases for Prombench.

So we had to rewrite everything related to GitHub Actions and currently we’re testing Github Actions and Prombench and are getting promising results. We plan to move this to prombench.prometheus.io soon.

Some miscellaneous additions to Prombench:

Well that was all I did for GSoC 😄