Launching a Data Engineering Tech Radar.

The world of data engineering is constantly changing. New tools emerge every week, architectural patterns fall in and out of favor, and the hype cycle churns endlessly. I’m sure we all at some point have come across variations of landscapes or state of data engineering diagrams packed with unreadable logos. Be assured, this is not one of those.

Keeping up feels like a full-time job already and we are usually looking to find something to help us cut through the noise and answer the following: How do you decide which technologies are genuinely worth your time?

To help, first myself, answer that question, I created this Data Engineering Tech Radar.

Inspired by the pioneering ThoughtWorks Tech Radar, this is my curated, opinionated, and practical guide to the data ecosystem. It’s a snapshot of my perspective on the tools, platforms, languages, and techniques that I have personally used or seen other teams adopt them in production. It is also a way to capture and keep track of what actually matters in our field right now and not feel overwhelmed.

Contents

Why a Tech Radar?

My motivation for creating this radar is threefold:

  1. To Navigate Complexity: The goal isn’t to list every tool, but to provide a filter. This radar helps separate the signal from the noise by offering a structured opinion on what’s production-ready, what’s promising, and what you might want to proceed with caution on.
  2. To Share Real-World Experience: Tutorials can show you how a tool works, but they rarely tell you if you should use it. This radar is built on hands-on experience, reflecting what has worked well in practice and the lessons learned along the way.
  3. To Track a Moving Target: The data landscape is not static, and neither is this radar. It’s a living document that I will update periodically to reflect new developments and evolving opinions, serving as a log of how our industry changes over time.

How It Works: Quadrants and Rings

To make sense of everything, the radar is broken down into four quadrants and four rings.

Data Tech Radar
Figure 1: Data Tech Radar

The Quadrants

The quadrants categorize items by their functional area in the data lifecycle.

  • Languages & Frameworks: This quadrant covers the foundational skills, languages and frameworks.
  • Transformation & Orchestration: Tools and practices for data transformation, ETL processes, and workflow orchestration.
  • Data Platforms & Storage: Technologies for data storage, databases, and data warehousing solutions.
  • Data Analytics: Tools and platforms for data analysis, visualization, and business intelligence.

The Rings

The rings represent my opinion on a technology’s maturity and my recommendation for its adoption.

  • Adopt: Technologies that are well-established and I have used or seen other teams adopt them in production and they’ve proven their value.
  • Trial: Emerging technologies that show promise and I have started exploring them or using on non-critical projects.
  • Assess: Technologies that caught my eye and are used or recommended by other teams that I think are worth exploring.
  • Hold: Technologies to proceed with caution on and I would not recommended for new projects.

This Is Just the Beginning

This is just the initial release and it might not have all the tools I have under the radar but this radar is a starting point, a snapshot in time, and it should be treated as such. My opinions will change as I learn, and new tools will emerge that demand a spot. I plan to revisit and update the radar periodically to keep it relevant.

But most importantly, this is meant to be a conversation starter.

What did I get right? What do you completely disagree with? What hidden gems are missing from the “Assess” ring? I’d love to hear your thoughts.

Check out the full Data Engineering Tech Radar and let me know what you think.

Stay tuned!