6 min read

Observability using Grafana - lessons learned


At GetInData, we understand the value of full observability across our application stacks.

In this article we will share with you our experience from running observability stacks on kubernetes hosted on public clouds and on premise environments. All presented with interesting use cases.

Continuous improvement

Each new implementation provides valuable experience which we use to improve the next one.

This allows us to continuously extend our stack to be more efficient, flexible and robust.


Currently, more than half of GetInData active projects are those where we manage observability stacks completely, meaning we design, implement and maintain the monitoring, logging and tracing of our application stacks:



Lessons learned

Now, let’s talk about some interesting use cases we have encountered along the way

We will divide them into the following areas: deployment, operation and performance.


How should you deploy your monitoring solution properly?

Thanos for highly available multi-cluster prometheus deployments

A single pane of glass for all your prometheus deployments? Yes, it’s possible!
Thanos can integrate multiple prometheus instances without any additional components

But there is more: 

  • it is 100% compatible with Prometheus API - no Grafana dashboards changes!
  • it can provide unlimited retention of historical data with downsampling capabilities
  • it supports all major object storage engines

✅ Choose reliable, feature reach tools able to support multiple architectures

But what does it mean exactly?
There are plenty of tools out there but as always, some of them are better than others.
Based on our experience, we recommend our version of LGTM stack:

  • L for logging using Loki
  • G for visualization using Grafana
  • T for tracing using Tempo
  • M for metrics using Prometheus

The Grafana users community is one of the largest out there for a reason:

  • it provides single pane of glass for metrics, logs, tracing andalerts
  • it supports all major data sources and of course, you can add custom ones
  • it supports all major federated user authentication standards
  • it can be easily deployed in a variety of configurations


How should you operate and/or upgrade your monitoring solution correctly? 

✅ Configuration changes should be verified and applied automatically

Making your configuration changes a part of your CI/CD pipeline is highly recommended.

Tasks to consider for your CI/CD pipeline are:

  • static code analysis
  • automatic documentation
  • security vulnerabilities scanning for container images
  • cloud cost changes before deployment
  • automatic deployment after successful verification

This way you can focus on the real value of your changes instead of wasting time on manual verification and applying your code. 

✅ Storage class with ReadWriteMany access mode (RWX) highly recommended

For example, Grafana publishes a new minor version every two weeks.

Having the possibility to continuously upgrade your apps without downtime is crucial.

In the kubernetes world, the storage class access mode heavily impacts this upgrade process.

As the name ReadWriteMany suggests: it supports read and write capabilities for multiple clients at the same time. 

And this is exactly what will be needed during the rolling upgrade of your Grafana pods with previous and new versions: the ability to write to the same volume at the same time.

Another use case where storage class with ReadWriteMany capabilities is recommended is during kubernetes node failure. When this happens, kubernetes will try to reschedule your Grafana pod to another node together with the corresponding persistent volume.

Unfortunately, for storage classes without the RWX capability, your Grafana pod won’t be able to start as kubernetes will still see its persistent volume as being used.

✅ Selinux for persistent volumes is available since k8s v1.24

Selinux is a security enhancement available by default on RedHat family distributions.

It increases the overall system security by decreasing the probability of a single operating system getting compromised. 

Because of that, the combination of running your kubernetes workloads on premise, together with Selinux enabled is still a common scenario for high security institutions like banking or government institutions.

While using kubernetes, Selinux installed on your nodes is also used in pods and persistent volumes. Unfortunately, a quite recent version of kubernetes is required in order to fully support Selinux for persistent volumes: v1.24 or higher.

✅ Grafana dashboards maintained by the code

Imagine your Grafana for all prometheus instances you deployed over all those years. Do you remember all the tiny configuration changes you made to quickly fix something?

Of course not! 

The solution is simple: include Grafana dashboards configuration in your CI/CD pipeline:

  • disable manual Grafana dashboard configuration changes
  • lint, test and verify your Grafana dashboard changes automatically before applying
  • enjoy your infrastructure synchronized with your code


How should you optimize your monitoring solution? 

✅ Storage performance is the key

Storage is crucial to overall system performance.

Even the best applications won’t perform while running on slow storage.

In the kubernetes world, the above statement still applies: each application pod using persistent volume will reflect the performance of the storage class configured underneath.

Surprisingly, using faster volumes doesn’t mean higher costs.

Many cloud providers offer attractive prices for faster storage, making it an easy choice from both cost and performance points of view.

For example, in AWS, you can reduce your storage costs up to 50% by simply migrating from slower General Purpose 2 (gp2) to new General Purpose 3 (gp3) SSD volume type.


In this post I have shared our experience with you in running observability stacks on k8s.

I hope you found it useful.

If you want to know more about our observability stack, please check the following blog post where I described its architecture in more detail: Running observability stack on Kubernetes.

11 May 2023

Want more? Check our articles

dbt machine learning getindataobszar roboczy 1 4

dbt & Machine Learning? It is possible!

In one of our recent blog posts Announcing the GetInData Modern Data Platform - a self-service solution for Analytics Engineers we shared with you our…

Read more
running observability kubernetesobszar roboczy 1 4

Running Observability Stack on Grafana

Introduction At GetInData, we understand the value of full observability across our application stacks. For our Customers, we always recommend…

Read more
Radio DaTa Podcast

Data Journey with Alessandro Romano (FREE NOW) – Dynamic pricing in a real-time app, technology stack and pragmatism in data science.

In this episode of the RadioData Podcast, Adama Kawa talks with Alessandro Romano about FREE NOW use cases: data, techniques, signals and the KPIs…

Read more
big data blog getindata from spreadsheets automated data pipelines how this can be achieved 2png

From spreadsheets to automated data pipelines - and how this can be achieved with support of Google Cloud

CSVs and XLSXs files are one of the most common file formats used in business to store and analyze data. Unfortunately, such an approach is not…

Read more
llm cluster hugging face gke autopilot getindataobszar roboczy 1 4

Deploy open source LLM in your private cluster with Hugging Face and GKE Autopilot

Deploying Language Model (LLMs) based applications can present numerous challenges, particularly when it comes to privacy, reliability and ease of…

Read more
noweobszar roboczy 1 3

GetInData in 2022 - achievements and challenges in Big Data world

Time flies extremely fast and we are ready to summarize our achievements in 2022. Last year we continued our previous knowledge-sharing actions and…

Read more

Contact us

Interested in our solutions?
Contact us!

Together, we will select the best Big Data solutions for your organization and build a project that will have a real impact on your organization.

What did you find most impressive about GetInData?

They did a very good job in finding people that fitted in Acast both technically as well as culturally.
Type the form or send a e-mail: hello@getindata.com
The administrator of your personal data is GetInData Poland Sp. z o.o. with its registered seat in Warsaw (02-508), 39/20 Pulawska St. Your data is processed for the purpose of provision of electronic services in accordance with the Terms & Conditions. For more information on personal data processing and your rights please see Privacy Policy.

By submitting this form, you agree to our Terms & Conditions and Privacy Policy