GetInData in 2022 - achievements and challenges in Big Data world

Time flies extremely fast and we are ready to summarize our achievements in 2022. Last year we continued our previous knowledge-sharing actions and launched new ones. Let’s not waste the time, dive into this summary and see what we did at GetInData last year!

• Published 44 blog posts about Big Data, ML/AI, streaming, cloud, modern data platform, events and more.
• Shared a lot of content about Machine Learning, Cloud and Artificial Intelligence. On our social media we promoted events and conferences. We also regularly posted Tech Facts with news from the Big Data world. Last but not least, there was plenty of content about life at GetInData. Find it on our social media channels that you can follow here.
• Started new formats: the Radio Data podcast and the Data Pill newsletter that you can join, to create a data community with us.
• Started Paper Talks - internal meetings, which in the end became public and now we are building a community in this area.
• Organized 2 Big Data conferences in Poland, the Big Data Technology Warsaw 2022 and the Data Mass Gdańsk Summit 2022, also taking part in Big Data conferences around the world.
• Continued our tradition and met more than 10 times for our internal Lunch & Learn sessions.
• Guilds and Labs are constantly growing. We created 4 areas that are focused on DevOps, Data Engineering, MLOps and Streaming.

If you want to know more about our achievements, below you can find a list of some of them. Enjoy the read!

Contributions

2022 was full of GetInData contributions to open-source.   

During the whole of last year, we  presented many solutions in the creation of which our big data experts participated, such as:

• Our DevOps Labs Team - Jakub Igła, Dominik Gniewek-Węgrzyn, Mariusz Wojakowski and Piotr Mossakowski, developed the Terraform module for Atlantis, which was highly acknowledged by the company and is now officially recommended as a way of installing Atlantis on Azure. Check it out here.
• Krzysztof Chmielewski put tremendous work into the release of Delta Connectors 0.6.0, which supports the Flink/Delta Connector on Apache Flink™ 1.15.3.
• Andrzej Dackiewicz recently worked on a new source-primetric connector for Airbyte.
• Mariusz Strzelecki had a hand in Apache Spark and Airflow.
• For months, the GetInData team including Maciej Obuchowski, Paweł Leszczyński, Jakub Dardziński and Tomasz Nazarewicz have been developing the OpenLineage project. We helped shape how Microsoft designed and implemented contributions to support Microsoft data sources and integrate with Azure Databricks. Also, our recent contribution supporting column level lineage has been the most anticipated feature for Microsoft. You can find an article written by Microsoft about the results of their work and our contributions here.
• GetInData credit in Delta Lake 2.0.0. by Grzegorz Kołakowski. This release's most exciting change from our point is Change Data Feed. Especially when we are able to implement it in Flink in Streaming.
• Apache Flink Source Connector for Delta Lake tables by Krzysztof Chmielewski.
• Contribution to Terraform provider for Snowflake and another one by Marek Wiewiórka 

Blog posts

During 2022, we were constantly posting on our blog. This means that one year later you can find here 44 published blog posts about big data, cloud, machine learning and more here. The top 5 most read are:

• Machine Learning model serving tools comparison - KServe, Seldon Core, BentoML written by Marcin Zabłocki
• Deploying secure MLflow on AWS  by Marek Jędraszewski
• Deploying serverless MLFlow on Google Cloud Platform using Cloud Run by Marcin Zabłocki
• Real-time ingestion to Iceberg with Kafka Connect - Apache Iceberg Sink by Grzegorz Liter
• A Step-by-Step Guide to Training a Machine Learning Model using BigQuery ML (BQML)  by Michał Bryś

That’s obviously not all. We published posts about new technologies like

• Introducing the Geoparquet data format by Paweł Kociński
• Introduction to GeoSpatial streaming with Apache Spark and Apache Sedona by Paweł Kociński

We also shared our knowledge in the field of Machine Learning (ML) and MLOps 

• Deep Learning with Azure: PyTorch distributed training done right in Kedro by Marcin Zabłocki 
• MLOps: 5 Machine Learning problems resulting in ineffective use of data and Feature Store comparison: 4 Feature Stores - explained and compared by Jakub Jurczak
• Kubeflow Pipelines up and running in 5 minutes by Marcin Zabłocki
• Feature store - managing multiple data sources with Feast by Kosma Grochowski
• What drives your customer’s decisions? Find answers with Machine Learning Models! H&M’s Kaggle competition by Adrian Dembek
• Online ML Model serving using MLeap and Flink with MLeap by Bartosz Chodnicki
• 2022 Big Data Trends: Retail and eCommerce become one of the hottest sectors for AI/ML by Adam Kawa 

Also, you could read more about our solution such as the GetInData Modern Data Platform

• Modern Data Platform - the what's, why's and how's? Demystifying the buzzword by Jakub Pieprzyk
• Announcing the GetInData Modern Data Platform - a self-service solution for Analytics Engineers
• GetInData Modern Data Platform - features & tool, both written by Michał Rudko.

Customer Stories

We also shared our success stories of working with clients with you:

• How we built a Modern Data Platform in 4 months for Volt.io, a FinTech scale-up by Paweł Kociński and Paweł Zalewski
• Platform focused on centralizing data sources and democratization of data with ING and Driving Customer Experience with scalable and secure Data Platform for Play  

We also started to share our content on Medium. Click if you want to follow us here. 

Webinars and Videos 

In 2022 we organized two live webinars. Were you not with us that day? That's not a problem at all! You can watch two of them here:

• Building ML pipelines with Kedro and Vertex AI on Google Cloud Platform where Michał Bryś demonstrated the way to operationalize Machine Learning models using open-source tools, like Kedro and deploy them using cloud computing.
• Data-Driven Fast Track: Introduction to data-drivenness with Piotr Menclewicz, who explained GetInData’s Data-Driven Fast Track, the 3-step framework for data transformation. In this one you can learn how to assess how data-driven your company is, how to generate ideas for new initiatives to push your company towards better decisions and how to think about implementing these initiatives to increase your chances of success.

On our YouTube channel you can also find videos with our experts:

• MLOps implemented - How we combine the cloud and open-source to boost data scientists' work. with Marek Wiewiórka and Krzysztof Zarzycki, who shared their experiences of the MLOps platform they built for our customers and how they boosted the way data scientists worked. They showed how customers’ projects of ML models training can go from zero to production in a much shorter time, while achieving superior performance, high code quality, training repeatability and governance. 
• How to run Kedro pipelines on Azure ML Pipelines service? - MLOps Tutorial by Marcin Zabłocki where he explained what exactly Kedro is, and what the benefits of this solution are, how to track the metrics using MLflow and how to install this plugin.
• How to become good Developer in Scrum Team? with Rafał Zalewski
• 7 Jupyter architectures for 7 different organizations with Mariusz Strzelecki, showed the different possible Jupyter setups with their pros and cons and shared the lessons we learned, also covering topics like culling (stopping inactive notebooks) and running spark-on-kubernetes sessions directly from notebooks.

Radio DaTa Podcast

We are also happy to share with you another project we started in 2022 - a Radio Data podcast! At Radio Data we talk about data, cloud, analytics and AI/ML/BI with different guest experts and different hosts in different segment formats. We have already started two segments:

• Data Journeys - episodes with special guests where our guest experts talk about how data moves around in their company, what technologies they use and why, and the value data brings to their products. These are the interviews. The host of the podcast is Adam Kawa. Some of the top listened-to episodes are:

  • Arunabh Singh (Willa) -  Building robust ML & Analytics capability very early at a FinTech, skills & competencies for data scientist, ML/AI predictions for the next decades
  • Alessandro Romano (FREE NOW) - Dynamic pricing in a real-time app, technology stack and pragmatism in data science
  • Victoria Bukta (Shopify) - Apache Iceberg and data ingestion
  • Wounter de Bie (Datadog) - Multi-cloud developer experience, Datadog vs. Spotify, areas to watch in the data landscape
  • Max Schultze (Zalando) - Data Mesh at Zalando, what it is, what it is NOT, how it helps Zalando to become even more data-driven company
  • Simona Meriam (Aidoc) - Unstructured data and data auditioning

• Data Updates - episodes with our GetInData experts where we share interesting news, or describe an interesting topic related to data, cloud, analytics, and AI/ML/BI. The top listened-to episodes were:

  • Paweł Leszczyński (GetInData) - Future-Aware Data Engineer
  • Adam Kawa (GetInData) - 5 current trends in the data and AI landscape (H1 2022)

The plan for the next year is to develop the existing formats and create new ones, so if you want to stay up to date, follow Radio DaTa on Spotify.

E-book

In 2022 we also released our eBook “MLOps: Power Up Machine Learning Process. Build Feature Stores Faster”.

What will you find there?

• How to eliminate the risk of the ineffective use of data in Machine Learning 
• How to reach the full potential of data-driven decision-making in real-time 
• A step-by-step guide to building a well functioning Feature Store 
• What MLOps is and the MLOps platform   

This eBook is divided into two parts. First from a business perspective of MLOps. Explaining the terms and dependencies necessary to making decisions in a business context like what the MLOps Platform is and whether you need it or not. The second one has a technical perspective with the advanced technical content necessary to implement the eBook knowledge.

Download the eBook for free.

Our Big Data Experts at conferences and meetups

Last year we organized two conferences: the Big Data Tech Warsaw Summit and DataMass Gdańsk Summit.

The 8th edition of the Big Data Technology Warsaw Summit was both on-site and online. If you weren't there you can still read the review of presentations and review of top 3 presentations, which will help you to decide to join us this year on the 29-30th March 2023!

There we had the pleasure of presenting:

• Bartosz Chodnicki and Linkier Seixas talked about the Benefits of a Homemade ML Platform.
• Mariusz Zaręba hosted a presentation called Let your analysts build data pipelines on Modern Data Platform using SQL.
• NetWorkS! project - real-time analytics that controls 50% of mobile networks in Poland - our Big Data Lead - our colleagues Maciej Bryński and Michał Maździarz from NetWorkS! described how we manage Flink jobs at scale using Ververica and Kubernetes, how we monitor the platform using Clickhouse and what problems we need to overcome in the project.

At the DataMass Gdańsk Summit, two presentations were given by our experts:

• Marek Wiewiórka gave a presentation named From first contact to a full charge... How we built a Modern Data Platform in 4 months for a FinTech scale-up.
• Also Adrian Dembek and Piotr Chaberski talked about From a Machine Learning competition to an enterprise analytics framework.

That's not all! Our experts had the pleasure of performing in other interesting Big Data Events, such as:

• During the Airflow Summit 2022, Maciej Obuchowski and Paweł Leszczyński gave a presentation entitled OpenLineage & Airflow - data lineage has never been easier.

• We were also at the Data Science Summit ML Edition 2022

  •  Mariusz Strzelecki talked about 7 Jupyter architectures for 7 different organizations. 
  • Adrian Dembek and Piotr Chaberski presented How NOT to win a Kaggle competition.

• During the Data Science Summit 2022 our experts gave few presentations:

  • Michał Rudko talked about Data Platform - what does it take to be called a modern one? A new stack with well-known best practices.
  • Piotr Menclewicz gave his presentation Data-driven fast-track - 3 steps to make your company data-driven.
  • Piotr Chaberski presented Prove your concept - faster, better, smarter.
  • Michał Stawikowski talked about Graph Neural Networks in Modern Recommendation Systems.

• at an IT Seminar organized by Veolia, Grzegorz Rycaj talked about why data likes the cloud and showed some success stories with the cloud from our portfolio.

• Lastly, we were at Warszawskie Dni Informatyki 2022 

  • Grzegorz Rycaj hosted a presentation “Excuse me, can I see the kitchen?”.
  • Marek Drob talked about “Have you been promoted to Team Leader or do you want to become one? Practical advice on how to succeed in your new role”.

What's more, we started our meetup called Paper Talks. We met for a few months to discuss new and interesting Machine Learning projects. At the end of the year we decided to make these meetings public. The next one will be in January, so if you want to talk or just listen to us then follow us on Linkedin to stay up to date with announcements.

Internal Knowledge Sharing

Lunch&Learn - we are continuing our meetings where our experts have the opportunity to share their knowledge with us. This is one of the most important internal initiatives at GetInData. During an online meeting, one of our specialists (or team) gives a presentation, and the rest of the group has the opportunity to ask questions and exchange experiences in this area. 

Some previous’ meetings subjects in 2022:

• Flink DBT Adapter
• Prove your concept - faster, better, smarter
• How to become a good developer in scrum
• Lookerstein Monster - why you shouldn’t be afraid of Looker
• Image-based CTR prediction & Google Tag Manager Webscraping

Guilds are a community of people who are passionate about the same topic. Anyone from GetInData can join a guild via slack and presence is voluntary. 

We have 5 Guilds working:

• MLOps
• DevOps
• Streaming (Real Time Data Processing)
• Data Engineering
• Advanced Analytics

At GetInData we also have Labs. The mission of Labs is to research and produce innovative solutions that develop our business and people to sustain our leadership position.

We currently have 5 work streams:

• DataOps Labs
• ML/MLOps Labs
• DevOps/Developer Labs
• Streaming Analytics Labs
• Advanced Analytics Labs

Data Pill Newsletter

During this year we developed new formats. You could read about our podcast but there is more. In June we released the first edition of our community newsletter called DATA Pill. It is a weekly newsletter sent every Friday morning with an overview of the best Big Data, Cloud, ML and AI content. 

Until now, we have released 33 editions of DATA Pill. We run it in two forms: as a traditional newsletter and a newsletter on LinkedIn (on Adam Kawa’s profile). 

Our community has almost 1500 people, 200 on the traditional mailing list and around 1300 on Linkedin.

You can read all previous DATA Pill editions and sign up here.

Plans for 2023

You can be sure we have a lot of new ideas to show you and develop existing ones in 2023. We are looking forward to other  experiences in the pipeline, opportunities and ways to share knowledge with you all. Stay up to date with us and follow our channels: Linkedin, Facebook, Twitter, and do not hesitate to subscribe to our channel on Youtube.