The recent developments in the AI space have impacted the way people live and work. This is particularly true for software and data engineers. Statements like “Coding is largely solved” and predictions that most code will be AI generated in some near future (timeline is different depending on whom you’re asking) have definitely contributed, among other things, to a challenging job market especially for entry-level and junior profiles.

In such an environment, and as the coding agents and models keep getting better, it is time to embrace the fact that change is inevitable and identify how exactly AI is affecting our jobs and what can we do about it.

So let’s get started!

Contents

• Why Business Context Matters More for Data Engineers
• Why Data Quality Still Matters in the Age of AI
• Same Data Engineering Principles, New Architectures
• Recap

Why Business Context Matters More for Data Engineers

Early in your career, it is natural to lean towards focusing on the technical skills in order to build a good foundation. However, this approach will only get you so far. Data engineers operate at the intersection of business strategy and technology. The role of the data engineer is not just building and maintaining pipelines. While that is still important, understanding the business context and being able to translate the requirements is what makes you a better data engineer.

Business stakeholders will ask you for example to provide a real-time dashboard. But what does it really mean real-time here? Maybe they meant the data needs to be up to date when they look at it every morning? Maybe they need the data to be accurate when they generate a report from it once a week? Jumping into solutioning mode without clearly understanding the requirements will result in an over-engineered and expensive solution. In the age of AI, this becomes even more important.

Data engineering is definitely not going away. It might be called differently at some point but a data pipeline is still a data pipeline. You just have to understand what it is feeding.

For example, training a large language model (LLM) requires a lot of data. This data is usually unstructured and spread across different sources and APIs. Processing the data, storing it and making it available across the different stages of the model training still requires data engineering expertise. As not every company will be training its own language model, we can take another emerging pattern as an example: RAG.

RAG stands for Retrieval-Augmented Generation. It is a cost-effective approach to overcome the limitation of outdated data and improve the output of the model or agent by feeding the correct internal data. Who knows better than the data engineering team which data to ingest? How to transform it and how to best store it for faster retrieval? This is all still data engineering work and has nothing specific to AI.

While I can keep iterating on examples, you might argue though that since coding agents are getting much better nowadays, they are able to generate code for data transformation workflows, write SQL queries and create dashboards. However, this does not necessarily imply removing the human from the loop. How a pipeline is built depends on a good understanding of the business and its unique needs. Without this context, these agents will just end up hallucinating and making things up.

You should treat them as tools that support and amplify your work by cutting down time on grunt work and boilerplate code. This will allow you to focus more on high-level design and optimizations.

As the creator of Claude Code said:

Therefore, understanding how these coding agents work and how to integrate them in your workflow is necessary. For a practical guide on AI coding agents, check out this article. You can also read about my own experience with these tools in Learnings from Vibe Coding.

Why Data Quality Still Matters in the Age of AI

As the volume of the data that we need to process is only going to increase, so are the stakes in your data quality. Flawed and poor quality input data inevitably produces similar output quality. This is what is known as: Garbage in, Garbage Out.

This is not a new problem. Data quality has been one of the biggest pain points for data teams fighting erroneous, missing or otherwise inaccurate data. The impacts of bad data go beyond drawing bad insights and sometimes even wrong conclusions. It can also result in wasted resources and money. If teams have to correct these failures all the time through backfilling pipelines, depending on the scale of data, this could take up many hours both in terms of compute and engineering capacity. In addition, it will affect product adoption and damage the overall organization’s reputation.

Now building AI systems also requires high-quality trustworthy data. If an organization is still struggling with data quality issues, any AI initiative will probably fail. AI adoption is accelerating and most companies are investing in it but without prioritizing the foundational building blocks, like having good data quality for starters, it will only lead to project failures. This is why I think the data engineer’s role will become even more important going forward. Supporting AI initiatives requires improving and maintaining good data quality and designing systems that meet the new scale, which we will discuss in the next section.

Same Data Engineering Principles, New Architectures

Thinking in abstractions has been helpful in navigating this crazy landscape. An example in the data field would be the Modern Data Stack. While a stack is just a combination of technologies and components, they do provide structure for building solutions. A recent similar emerging idea is the AI Stack. The purpose of this article is not to dive into the latter, but rather use it as a reference to highlight a key component around data engineering. There are multiple definitions out there for this new stack and its components but in each one of them we have a data layer. Given the specific requirements needed for building AI applications and the sheer amount of data, data engineers are faced with new challenges.

Continuing with the RAG use case mentioned earlier and in order to provide relevant context to the model, external data need to be fed in the form of numerical vectors called embeddings.

To achieve that, the high-level process involves splitting the data into smaller chunks, transforming them into embeddings and storing them in a specialized database called vector database. For a hands-on example of this process, check out how I built semantic search for this blog. These are optimized for storing, indexing and searching high-dimensional vector embeddings. While it may seem as a new type of architecture, underneath it is mainly about setting up pipelines, ingesting the data, managing the data flow, optimizing the embeddings and retrieval; all the things you would expect a typical data engineering team to be doing. As the volume of vector data grows, maintaining low-latency search and efficient indexing becomes challenging. In addition, ensuring fine-grained access control, encryption at rest and in transit, and audit trails are essential components of governance that need to be put in place.

Recap

The data engineering role is integral to the business and holds a central role in bringing AI initiatives to life. Having a strong data foundation is not optional for any AI initiative’s success and that puts the spotlight on data engineers who are no longer working behind the scenes. It goes without saying that while data engineering teams have seen AI-led productivity improvements, complexity has also increased. Data engineers need to embrace the change as the role will evolve significantly and the day-to-day activities will change.

With this we have reached the end of this post, I hope you enjoyed it!

If you have any remarks or questions, please don’t hesitate and do drop a comment below.

Happy learning!

AI coding tools have moved beyond simple suggestions. Today’s agents can read your codebase, execute commands, search the web, and complete multi-step tasks. As things are changing rapidly with new features and terminology appearing every week that each promises to revolutionize how we write code, I wanted to understand the bigger picture: What are the core building blocks that all these tools share? How do features interconnect? And most importantly, how can I use this knowledge to work more effectively?

This article maps out the anatomy of an AI coding agent, using Cline as the primary example while drawing connections to other tools in the ecosystem.

So let’s get started!

Contents

• Foundation: What is an Agent?
• Why Cline?
• Context Layer: What the Agent Knows
  • Context Engineering
  • Memory Bank
• Instructions Layer: How the Agent Behaves
  • Custom Instructions
  • Skills
• Tools Layer: What the Agent Can Do
  • Tools
  • MCP
• Recap
• Resources

Foundation: What is an Agent?

GitHub copilot was initially released in June 2021. A collaboration between GitHub and OpenAI resulted in an AI powered tool with the premise to help developers code faster leveraging OpenAI’s Codex model. The tool mainly offered what is now sometimes referred to as “autocomplete on steroids”. However, things evolved rapidly and by 2024 the AI coding tools got more advanced especially with integrations into IDEs to reach what we are currently witnessing as the emergence of AI coding agents. But what is an AI agent?

In order to understand what is an AI Agent and identify its foundational building blocks, we will first look at different definitions from AI research labs and technology companies.

According to OpenAI, an agent is An AI system that has instructions (what it should do), guardrails (what it should not do), and access to tools (what it can do) to take action on the user’s behalf.

Google has a similar definition with an emphasis on the capabilities of the model:

AI agents are software systems that use AI to pursue goals and complete tasks on behalf of users. They show reasoning, planning, and memory and have a level of autonomy to make decisions, learn, and adapt.

Looking to Anthropic’s definition, they have a more broad definition of an agentic system encompassing both:

• Workflows that are systems where LLMs and tools are orchestrated through predefined code paths.
• Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.

To simplify things and have a set of building blocks that we can use later as foundations to build upon and extend, we can consider an agent to be composed of the following:

Figure 1: AI Agent Building Blocks

It is worth highlighting that the agent operates in a continuous loop: it observes the current state, reasons about what to do next, takes actions through tools and observes the results. This cycle continues until the task is completed or the user intervenes.

With this understanding, let’s see why I consider Cline as my go-to AI coding assistant and explore later best practices and features that extend its functionality.

Why Cline?

The landscape of AI coding tools is evolving rapidly. On one hand, the underlying models powering these tools keep getting better with a new model at the top of the leaderboard every other month. On the other hand, integrated systems such as Claude Code limit your flexibility to switch models or explore other features that might not be available there.

Constantly switching and chasing the new kid on the block can be taxing. Of course this does not mean you shouldn’t try them out, but you might end up using multiple tools and managing multiple subscriptions. In an attempt to reduce this overhead, I have found Cline - a free VSCode plugin to be uniquely valuable.

What stood out to me is Cline’s philosophy; You get access to any model, you don’t have to choose a particular tool and you get the latest proven features from the industry without suffering the tool fatigue.

In the following sections, we will look into techniques and concepts that extend and improve the functionality of an agent and how they can be leveraged in Cline.

Context Layer: What the Agent Knows

Context Engineering

Context is a collection of information an AI agent uses to understand the user’s request. This is however a finite and limited resource because every AI model has a context window which is the maximum amount it can process, measured in tokens, in a single conversation.

What started as crafting optimized effective prompts, known as prompt engineering, has evolved to finding the best strategy for managing the entire context and choosing what to include in the limited context window.

Besides the model’s system prompt, when you start a new task or conversation, the context window will include the user’s query along with needed context, requirements, documentation and active files. As you interact more with model, the conversation history is also included. In addition, tools like Cline will include additional dynamic context needed to achieve the task at hand. For instance, Cline automatically discovers:

• project structure and file organization
• Import relationships and dependencies
• Code patterns and conventions
• Configuration files and settings
• Recent changes and git history (when using @git)

On complex projects and as the current conversation approaches the context window limit, a standard solution would be using summarization technique to compress context preserving important information and allows you to keep working. However, this is not ideal as you could still sacrifice important context.

Though things like documentation of your project, architectural decisions and design patterns are not persisted across different sessions but still required to get more accurate and relevant responses. This is where the memory bank comes in.

Memory Bank

Depending on the tools you use, files like CLAUDE.md or GEMINI.md, are used to provide persistent guidelines and preferences across sessions. Automatically loaded into the context, they act as a long term memory for the agent. An attempt to standardize this across many agents, resulted in this simple open format: AGENTS.md

Cline’s memory bank on the other hand, is not a specific feature but rather a methodology that takes this into a structured documentation that allows to maintain knowledge across sessions. It is simply a group of markdown files stored in your repository that goes beyond a single README.md file.

Instructions Layer: How the Agent Behaves

Custom Instructions

Setting custom instructions help the AI generate code that is more aligned with your specific conventions, frameworks and use cases. These project specific customizations lead to:

• Consistency across team members when receiving suggestions that adhere to the same coding standards and practices.
• More accurate and relevant suggestions knowing your tech stack, frameworks and tools.
• Reduce the time spent on refining AI-generated code

For Cline these custom instructions can be defined through Rules.

Rules are loaded when Cline starts a new task and are checked in the following order:

1. .clinerules/ folder (all .md files inside)
2. Single .clinerules file
3. AGENTS.md file

This highlights that workspace rules override global rules when both define the same guidance.

Rules provide consistent guidance for every interaction, but some tasks require more than general guidelines and would need step-by-step procedures. In addition, they are not necessarily invoked with every request. This is what skills are designed for.

Skills

Skills are folders of instructions that package repeated workflows, specialized knowledge, or new capabilities for your agent. It contains a SKILL.md file, along with instructions, scripts, assets and resources in the same folder that agents can discover to perform a specific task accurately.

The SKILL.md file contains 2 sections: metadata and instructions. The latter contains detailed instructions to follow for that particular skill. The metadata provides necessary information when to use this skill through specific fields of which 2 are mandatory: name and description.

Agent skills are now an open standard and has been adopted by a growing number of agent products.

Unlike rules which are always active, skills load on demand. Installing multiple skills does not affect the context as only the metadata is always loaded and the instructions and other files are loaded progressively when the skill is triggered.

Context and instructions shape what the agent knows and how it behaves, but an agent also needs the ability to read files, execute commands, and interact with external systems. This is the role of tools.

Tools Layer: What the Agent Can Do

Tools

Tools define what your AI assistant can do. They let an agent observe, decide and act in order to achieve goals.

Common built-in tools generally include tools to interact with the file system like listing directories, reading and writing files. They also include web tools for searching the web as well as shell tools to execute commands.

In order to extend the capabilities of the agent beyond common operations, agents use MCP servers, covered in the next section, also through a dedicated internal tool.

MCP

Model Context Protocol is an open-source standard for connecting AI applications to external systems. These systems include data sources such as local files and databases or tools like search engines. Through this standardization we reduce development time and complexity when integrating external systems and enhance the capabilities of our agents.

Essentially MCP servers act as API that the models can use with predefined set of tools or functions that can be executed through this interface. They offer modular integration where they can be easily added or removed without overhauling the entire system architecture.

With context management, persistent memory, custom instructions, skills, and extensible tools through MCP, we now have a complete picture of how modern AI coding agents work.

With this we have reached the end of this post, I hope you enjoyed it!

If you have any remarks or questions, please don’t hesitate and do drop a comment below.

Stay tuned!

Recap

We have covered a lot of ground: from the basic agent building blocks to advanced features like context engineering, memory banks, MCP and Skills.

This article is a snapshot of the current landscape focusing on foundational concepts and the challenges they address. The ecosystem is evolving rapidly, and there’s much we haven’t covered: sub-agents for delegating tasks, hooks for event-driven automation, workflows for chaining operations, and other patterns still taking shape.

The tools will keep changing and the models will keep improving but the challenges that led to these patterns and concepts like context engineering, persisting knowledge and extending capabilities will enable you to use tools interchangeably as the landscape evolves.

Happy learning!

Resources

https://cline.bot/blog/the-last-ai-coding-agent-youll-need

https://newsletter.pragmaticengineer.com/p/two-years-of-using-ai

https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

https://docs.cline.bot/prompting/understanding-context-management

https://cline.bot/blog/memory-bank-how-to-make-cline-an-ai-agent-that-never-forgets

https://claude.com/blog/skills-explained

https://agentskills.io/home

https://modelcontextprotocol.io/docs/getting-started/intro

We all have seen the exponential adoption of Large Language Models (LLMs) since the release of ChatGPT on November 2022 and the headlines that promise us something new every year; from Model Context Protocol (MCP) and CLI Coding Agents to autonomous multi-agents systems and Context Graphs, the list goes on. Taking a step back, this article is my first attempt to show you how to build AI use cases that actually work. Building is a great way of learning while also gaining insights into how these systems work under the hood.

So let’s get started!

Contents

• What is Semantic Search?
• The Architecture
• Tech Stack Choices
• Challenges and Optimizations
  • Building the FAISS Index
  • Automatic Index Updates
  • Cold Start
  • Cost Optimization
• What’s Next?

What is Semantic Search?

Semantic search is a way of finding information by meaning rather than exact keywords.
Unlike traditional keyword search where the literal words in your query are matched to the same words in documents, semantic search converts both the query and the documents into vectors, known as embeddings that capture their meaning. In essence, it is a nearest-neighbor search problem where, given a query, the top most semantically similar items or vectors are retrieved based on some heuristic used to calculate the likelihood. The naive solution is k-nearest neighbors (k-NN), which in the context of our blog would follow these steps:

1. Split all the blog posts into chunks, generate embeddings and save them into a vector store.
2. Given a new search query, convert it to a vector embedding.
3. Compute the similarity scores between the query embedding and all vectors in the database, using metrics such as cosine similarity.
4. Rank all vectors by their similarity scores.
5. Return k vectors with the highest similarity scores.

The Architecture

Before diving into implementation details, here’s the high-level architecture:

Figure 1: High Level Architecture

The flow is simple:

1. When I publish a new blog post, a GitHub workflow is triggered to rebuild the image of the backend serving the semantic search endpoint.
2. Building the image involves fetching posts from the website, generating embeddings, and building a search index
3. The image is pushed to Artifact Registry and deployed to Cloud Run

With the high level flow defined, in the following sections we look in more details into the constraints that influenced the tech stack choices, faced challenges and implemented optimizations.

Tech Stack Choices

With the objective of keeping cost to a minimum, yet still having fast deployments, minimal operations and easily scalable solution, choosing a fully managed serverless platform for deploying the backend service became the clear answer. In addition, the generous free tier of Google Cloud made the decision easier to go with Cloud Run. Cloud Run enables you to run stateless containers on top of Google’s scalable infrastructure.
The free tier includes 2 million requests/month and enough memory and compute time for the amount of traffic that I’m expecting. Crossing these limits would only cost pennies on the dollar given that you only pay for what you use and you can always scale it down to zero. For more information on free tier usage limits, you can check this page.

Considering the choice of Cloud Run and the number of blog posts and expected size of the generated embeddings, I decided to go with an in-memory vector database. FAISS is Facebook’s similarity search library. It is fast, in-memory and suitable for this scale. FAISS builds an index for fast similarity search and instead of comparing the query to every post which can be slow, it uses smart data structures to find the nearest neighbors quickly. Another required component that goes hand in hand with this is an embedding model. I used sentence-transformers library. It is a python framework for accessing and using state of the art embedding models. In addition, to keep the image size reduced, I used theall-MiniLM-L6-v2 model which is around 90MB and generates 384-dimensional vectors.

Regarding the backend, I used FastAPI; modern, fast and high-performance web framework for building python based APIs with automatic API docs, and excellent type safety with Pydantic.

Challenges and Optimizations

Building the FAISS Index

In order to have faster container startup, I build the FAISS index during Docker image creation, not at runtime. This ensures consistent results across all instances since the filesystem is ephemeral and no external dependencies at runtime.

Automatic Index Updates

We keep the blog and the search backend in separate repositories to keep concerns isolated and releases simple. The open question was: how do we update the search index whenever a new post goes live?

To solve this, I added a GitHub Actions workflow in the blog repo that triggers on publishing and calls the GitHub REST API to send a repository_dispatch event to the backend repo with a custom event_type=blog-updated. The backend repo has a workflow that listens for that event; when it matches, it builds and deploys the Docker image and refreshes the index.

Cold Start

When Cloud Run scales from zero, it takes between 5 and 10 seconds to start a container. This time includes the container startup time and loading of the embedding model as well as the FAISS index.

Cloud Run has the option to allocate extra CPU during startup and which can be activated by this flag:

--cpu-boost

In comparison to the size of the FAISS index, loading the embedding model, even a small one, on every cold start is slow. One solution to this would be caching the model to a specific location inside the container by adding the following line in your Dockerfile:

      
    RUN mkdir -p /app/models && \
    /app/.venv/bin/python -c "from sentence_transformers import SentenceTransformer; model = SentenceTransformer('all-MiniLM-L6-v2'); model.save('/app/models/all-MiniLM-L6-v2')"
   
   

While keeping the minimum number of running instances set to 1 would solve the issue of the cold start completely, it would also result in extra costs. Depending on your traffic and usage of the search functionality in your system, you can opt for this alternative. For me the objective was to build and test the sematic search end to end and even though I could still face timeout errors occasionally when searching it is still acceptable for now.

Cost Optimization

Since Cloud Run charges for the resources you use and in order not incur additional costs after the free tier limits, the service shuts down and scales back to zero when there’s no traffic and the Cloud Run service is idle. However, each time a new post is published, this results in a new image version. To avoid storage costs and stay within the free tier limit of Artifact Registry which is < 0.5 GB, I have applied the following cleanup policy to keep only the last 2 docker images:

      
    {
        "name": "keep-recent-2",
        "action": "DELETE",
        "condition": {
            "tagState": "UNTAGGED",
            "newerThan": "0s"
        },
        "mostRecentVersions": {
            "keepCount": 2
        }
    }
   
   

What’s Next?

Building a semantic search doesn’t require expensive infrastructure. With open-source tools and serverless platforms, you can build production-quality services that scale and cost almost nothing. Having the core architecture in place, adding new use cases such as a recommendations endpoint based on the existing similarity search and vector database or potentially combining the semantic search with keyword matching for better results are now easier. Avoiding over engineered solutions and solving the problems as they araise avoids pre mature unnecessary optimizations.

I’d love to hear your thoughts and know more about what AI projects you’re currently working on.

With this we have reached the end of this post, I hope you enjoyed it!

If you have any remarks or questions, please don’t hesitate and do drop a comment below.

Stay tuned!

In the data engineering capabilities and personas’ article, we looked at some emerging constellations of data teams and the general responsibilities associated with them. One area we mentioned was data platform engineering. While having already touched on some aspects needed for building a data platform, we will attempt in this article to further explore this topic starting with what exactly is a data platform and where AI will likely play a role in it.

So let’s get started!

Contents

• What is a Data Platform?
• Layers of a Data Platform
  • Ingestion Layer
  • Storage Layer
  • Processing Layer
  • Consumption Layer
  • Intelligence Layer
• Benefits of a Data Platform
• Recap
• Resources

What is a Data Platform?

For a variety of reasons, main one being marketing, the term “platform” has become overloaded with different meanings in different contexts and the term data platform is no exception.

The Data Warehouse was probably the first attempt at a centralized data platform. As we have seen here, things evolved in the big data era and the concept of the data lake emerged as a response to the previous limitations. A modern version or the modern data stack as it is often referenced to, represents a shift from focusing on a single technology to focusing on the entire data lifecycle. It is not a single tool, but an integrated ecosystem of tools designed to cover every step of the data flow.

Another natural evolution would then be the self-service data platform. Data engineering teams suffering through many “can you quickly pull this data for me?” kind of requests, the self service aspect is a process that makes it easier for the end users looking to “quickly” access some data to find it, understand its meaning and how they can use it and remove them from the process to some degree. This will enable non-technical users like marketing managers and sales leads to build their own views, filter their own data, and answer their own questions.

Before looking into the components of a data platform, other variations that would fit under the umbrella concept of the “data platform” include:

• Data Stack: a collection of various tools wired together to perform the platform’s functions.
• Data Fabric: A unifying technology layer that provides centralized access and governance across data that is physically stored in many different places.
• Customer Data Platform: A platform specifically focused on consolidating and unifying all customer data (sales, web, app, email) to create a single, actionable “360-degree view” of a customer.

Layers of a Data Platform

One way to think about the components of a data platform is through layers. Each layer will contain a set of tools that, stacked together cover the end to end lifecycle of data.

Ingestion Layer

This is the starting point of any data platform. Data within an organization is usually scattered across multiple systems and the objective is to consolidate it into one place. This usually starts with identifying the required data sources and how to access them. Patterns such as Change Data Capture (CDC) and automated pipelines would be part of this layer copying new data and moving it to the next storage layer.

Storage Layer

The storage layer provides a centralized and scalable way for storing and managing the data in a way that it can be accessed, queried, and updated easily. Choosing the right storage technology and data models are key considerations to keep in mind. The decision will also depend on the type of data, volume, size and maturity of the organization. Options include data warehouses, data lakes and recently the rising adoption of data lakehouses. It is worth mentioning that it’s not unusual to find organizations that have a combination of multiple data management solutions at once.

Processing Layer

While it might be tricky to draw a clear boundary around this layer, the main purpose is usually preparing data by executing a set of transformations on it. It can be thought of as an enabler to the other layers and hence used multiple times across the data lifecycle especially as we get closer to the consumption layer where the data needs to be accessible and easily consumable by different teams each having a different set of requirements. Therefore, similar to the ingestion layer for example, we also find complex data pipelines and orchestration tools to automate, manage and monitor the flow of data and execution of transformations.
Depending on which technique is used, this layer represents the T in ETL/ELT patterns. These transformations include data cleaning such as removing errors or duplicates, schema validation and other data quality checks, normalization, aggregation and enrichment with other sources.

Consumption Layer

This is where the business value happens. The consumption of the data happens in different forms because the users of this layer can be both technical and non technical. For technical personas, they would probably look for new datasets to enrich their models with new features. Others would be interested in a dashboard that helps them gain insights into trends and patterns or review certain KPIs to make informed decisions. Another form of presenting the data is through reports that are intended for a specific type of stakeholders looking for a structured and easy to understand format. In this layer, the problem of your data engineering team becoming the bottleneck as we mentioned earlier manifests the most. Typically, the process involves submitting a new request for a report to the data team. You wait two weeks. They bring it to you. If you want to change a filter, you create a new request and wait again. However, the modern self service approach leverages governance to make the data accessible and ready for consumption making the feedback loop shorter.

Intelligence Layer

This is the biggest shift in the last two or three years. With the rise of Large Language Models (LLMs) and autonomous AI Agents, new use cases have emerged and are now being integrated across the different layers of the data platform. Yet, in my opinion, it still makes sense to reason about them as a logically separate layer.

One of the patterns that have matured and became a default is Retrieval-Augmented Generation or shortly RAG. While LLMs are trained on large volumes of data and can generate an output for a variety of tasks, their responses will not be as effective against questions related to an organization’s internal information. This data is proprietary and not publicly available to be included in the training data which leads the LLMs to hallucinate and make up things in an attempt to provide an answer. RAG is a cost effective approach to overcome this limitation and improve outputs by providing accurate, relevant answers grounded in internal data.

On a high level, RAG works as follows:

1. Retrieval: When a user asks a question, the system first searches an external knowledge base (e.g., documents, databases) for relevant information.
2. Augmentation: The retrieved information is then added to the original prompt.
3. Generation: The LLM uses this combined information (original prompt + retrieved facts) to generate a more informed and factual response.

Going deeper on this pattern is not the scope of this article, and we must highlight that this is obviously a simplified explanation and rather basic as things have evolved over time but still needed to understand its implications from a data platform perspective. Making the enterprise knowledge RAG-ready introduces new pipelines: fetching documents, splitting them into small chunks, generating embeddings and saving them into vector databases are all required steps that are increasingly becoming platform native. As RAG matures, adoption of vector databases is accelerating and enterprises are adding vector-native storage and search features to enable high performance and scalable AI applications.

Beyond RAG powered chat assistants, enterprises are also shifting towards agents that plan, call tools/APIs, and execute workflows autonomously. In generic terms, agents can be defined as LLM powered systems that have access to a set of tools that allow them to take actions in a certain environments. Therefore, RAG based assistant can be considered as an agent with a single tool that is the retriever part that searches and fetches relevant documents depending on the user query. Now that you can quickly search and interact with your documentation, you want to give your system access to your project management tool. This allows you to fetch ongoing issues related to your discussion, check their status and create follow up issues if needed. You can then deploy an autonomous bot that can answer support requests, provide links to relevant documentation, check systems’ health and file bugs. These systems will continue to improve over time and become more capable but we should always bear in mind the risks that come with building a system where AI makes decisions automatically. That’s why human‑in‑the‑loop is non‑negotiable for high‑impact actions taken by the agents as well as the importance of platform observability. Moving beyond established capabilities, we need to also integrate agent specific monitoring. Things like understanding their interactions with external tools, RAG performance, multi-step agent execution, tracing user queries and analyzing model responses will help ensure reliability, debug issues and optimize costs for these complex AI pipelines.

Another clear trend is the move toward semantic, natural‑language access. Business users ask questions in plain language; assistants translate that into SQL, run governed queries, and return answers with citations. Modern platforms now embed this inside the warehouse/lakehouse and BI experiences, inheriting your RBAC/masking policies. This shortens feedback loops without bypassing governance.

These are some examples of new patterns and trends that I believe are probably here to stay. However, things will continue to evolve and we should expect more changes affecting all the layers of the data platform.

Benefits of a Data Platform

The main benefit of a data platform is to enable other teams. Your data team can’t grow as fast as your data and the former will quickly become a bottleneck if every question requires a dedicated data engineer. In addition, by handling common laborious tasks at the platform layer such as infrastructure setup and governance, teams can innovate and iterate more quickly. This will result in self perpetuation of the platform by making data quickly available for other teams to consume and generate value from.

In order to achieve this, the platform team needs to focus on the onboarding process making it easy for the other teams to join the platform. They need to take into consideration the maturity levels spectrum across the organization and account for when to enforce constraints and when to lower the entrance barrier. Faced with these challenges, a common pitfall is tool fatigue. It’s easy to buy 15 different “best-in-class” tools that don’t talk to each other trying to cover every aspect and accommodating every need in order to build the perfect platform. Depending on the platform adoption and perceived benefits, it will be very difficult to justify these costs down the line. The platform team should instead start small, gather user feedback and iterate quickly favoring simplicity over complex and over engineered solutions.

With this we have reached the end of this post, I hope you enjoyed it!

If you have any remarks or questions, please don’t hesitate and do drop a comment below.

Stay tuned!

Recap

Building a data platform is a journey. Teams should start small and aim to getting data out of silos and into a place where other teams can actually use it. The goal isn’t to have the fanciest technology. The goal is to stop arguing about whose spreadsheet is right, and start making better decisions.

Happy learning!

Resources

https://www.startdataengineering.com/post/self-serve-data-platform/

ttps://www.confessionsofadataguy.com/building-data-platforms-from-scratch/

https://developer.nvidia.com/blog/building-scalable-ai-on-enterprise-data-with-nvidia-nemotron-rag-and-microsoft-sql-server-2025/

Design patterns are standard solutions to common problems. They represent best practices and templates that can be reused in multiple situations. In this article, we will explore four common patterns that can be leveraged to ensure reliability, performance and observability in data pipelines.

So let’s get started!

Contents

• Write-Audit-Publish (WAP)
• Change Data Capture (CDC)
  • Log-based CDC
  • Trigger-based CDC
  • Timestamp-based CDC
• Dead-Letter Queue (DLQ)
• Cumulative Aggregate Table
• Recap
• Resources

Write-Audit-Publish (WAP)

Weather it is a schema evolution in the upstream data or third party APIs or incorrect join clause in your pipeline, preventing the resulting bad data and ensuring data integrity when writing to production systems requires rigorous validation.

The WAP (Write-Audit-Publish) pattern is a critical data strategy that prevents such data quality issues. This method involves writing data to a staging table, auditing it for quality, and only publishing it to production if it passes checks. This ensures that the end users or consumers of this data can trust it.

Figure 1: Write Audit Publish Workflow

In order to understand how the WAP pattern can be implemented we will use a key feature of the open table format Apache Iceberg: Branches.
For more information on Apache Iceberg and how it compares to the other well known formats, you can check this article.

The metadata of an Iceberg table stores a history of snapshots. These snapshots are changes applied to the table and are the basis for reader isolation and time travel. Iceberg branches are another named reference to a snapshot of a table. They can be used for handling GDPR requirements and retaining important historical snapshots for auditing or be part of an ETL pipeline enabling validation of new incoming data.

Implementing the 3 steps WAP pattern using Iceberg branches would result in the following:

1. Write: switch branch from main to an audit branch and commits updates there. Data is not yet accessible to downstream users who can only access main branch.
2. Audit: Run data quality checks on the audit branch.
3. Publish: The main branch can be fastForward to the head of the audit branch to update the main table state.

Using the WAP pattern, downstream pipelines or dashboards can intuitively depend on the production table directly. However, the main challenges is the introduced delays and increased latency by the multistep process which might not be idea for near real time use cases.

Change Data Capture (CDC)

Change Data Capture (CDC) is a data integration pattern that captures all changes in a source database; creates, updates and deletes and makes them available for downstream systems.

Organizations typically have a variety of operational data spread across different systems and applications needed for running the business. The purpose of CDC is to keep this data synchronized by capturing changes from source databases and moving it to your target data warehouse or data lake.

In contrast to batch processing, where a pipeline is scheduled periodically to replicate data from one system to another, CDC is considered more efficient. It eliminates the bulk load during specified windows and enables an incremental loading making the data available in near real time. In addition, it reduces the impact on the source database

CDC is typically implemented using two main approaches: push and pull. In the push approach, the source database pushes the updates to the downstream systems. This has the advantage that the target systems have the latest data in near real time. In the pull approach on the other hand, the source database only logs the changes and it is the responsibility of the target systems to continuously poll it. This results in a delay of the data availability. In both approaches, if the source or target systems are not available for some reason, the data is lost. To overcome this, a messaging system can be used in between to buffer these changes.

There are 3 common methods for how changes in data are detected using CDC:

• Log-based
• Timestamp-based
• Trigger-based

Log-based CDC

Transactional databases have internal log files where they record all changes committed against the database. These log files are primarily used for backup and disaster recovery purposes. By reading from these transaction logs we can propagate the changes to target systems without adding computational overhead. However, since each vendor’s logs have a different format, this method can’t be easily reusable.

Trigger-based CDC

Another feature that is widely supported by databases is trigger functions. These are stored procedures that are automatically executed once a specific event occurs on a table. We need to create per table one trigger for each operation and store the data changes in a separate table. This adds additional write operations and might impact the performance of the database. Also managing a large number of triggers can become challenging.

Timestamp-based CDC

This method requires changes to the database schema to include a timestamp column to record when was an entry last updated. Changes are captured by selecting items with a timestamp newer than the previous check timestamp. Being probably the easiest to implement, it fails at capturing delete operations and could add an additional overhead to the database.

Dead-Letter Queue (DLQ)

In streaming environments, a dead-letter queue is a special message queue used to store messages that were not successfully processed by the primary system. These include corrupted messages due to network or system failure or ones that failed the data quality checks.

By having a DLQ in place, we provide a mechanism for analyzing and identifying common patterns of errors enabling better observability in order to improve the reliability of the system. In addition, we can reduce the data loss by reprocessing messages that couldn’t be delivered from the DLQ.

Cumulative Aggregate Table

The cumulative aggregate table design pattern is very useful when we want to solve the performance issue that comes with calculating aggregates over a large rolling time window.

For example, if we want to calculate the number of active users for the last 30-days window, the straightforward solution would be to write a query that scans 30 days of raw event data for all users every single day. This query is slow and expensive especially if run regularly.

This design pattern proposes a solution to overcome this. The idea is to create a new table that stores the pre-computed state often using arrays or structs. Following the active users example, our table called user_activity_cumulative will have an array column that is updated daily by performing a FULL OUTER JOIN between today’s active users and yesterday’s cumulative table.

Table 1: Example of user_activity_cumulative Table

Using this new table like shown above, finding out the 30-day active users for November 30th can be done by running the following query:

      
   SELECT
     COUNT(user_id)
   FROM user_activity_cumulative
   WHERE
     date = '2025-11-13'
     -- Sum the 30 elements in the array.
     -- If the sum is > 0, they were active at least once.
    AND array_sum(activity_array_last_30_days) > 0
   
   

For more details and concrete hands on example, you can check this repository.

With this we have reached the end of this post, I hope you enjoyed it!

If you have any remarks or questions, please don’t hesitate and do drop a comment below.

Stay tuned!

Recap

In this article, we explored four essential data engineering design patterns that help build reliable and efficient data pipelines. The Write-Audit-Publish (WAP) pattern ensures data quality by validating data in a staging area before publishing to production and leveraged Apache Iceberg branches for an example implementation of this pattern. Change Data Capture (CDC) enables near real-time data synchronization by capturing incremental changes from source databases using log-based, trigger-based, or timestamp-based methods. Dead-Letter Queues (DLQ) provide a safety net for failed messages in streaming systems, improving observability and reducing data loss. Finally, the Cumulative Aggregate Table pattern optimizes performance for rolling window calculations by pre-computing and storing state, eliminating the need to scan large amounts of raw data repeatedly.

Happy learning!

Resources

https://www.confluent.io/learn/change-data-capture/#why-change-data-capture

https://aws.amazon.com/blogs/big-data/build-write-audit-publish-pattern-with-apache-iceberg-branching-and-aws-glue-data-quality/

https://lakefs.io/blog/data-engineering-patterns-write-audit-publish/

Following the AWS Lambda launch in 2014 and the release of Amazon’s API gateway in 2015, “serverless” grew in popularity to become a new buzzword in the industry. Today, many organizations are considering serverless architectures as a way to save costs and operational overhead in building and running their applications.

But what does it really mean to build with serverless, especially in data engineering? That is what we will try to cover in this article through concrete hands on example.

All the source code is available here

So let’s get started!

Contents

• What is Serverless?
• Simplified Workflow Example
• Serverless and Software Design
• Challenges and Limitations
• How to Choose?
• Resources

What is Serverless?

According to AWS, serverless computing is a set of “technologies for running code, managing data, and integrating applications, all without managing servers.”

To elaborate, let’s first differentiate serverless offerings from fully-managed services where some server configuration is required. A good example would be Amazon Aurora which offers two modes:

• Provisioned: a managed “instance-based” model where the instance size is configured by the developer and the cost model is usually a fixed hourly rate.
• Serverless: no servers are configured by the developer. The database scales automatically and the cost model is pay-per-use.

The core principles of a true serverless architecture are no server management and a pay-for-value cost model.

Simplified Workflow Example

The repository contains an example of an ETL pipeline for processing NYC Taxi Trip data using AWS services. The goal is to ingest raw CSV data, transform it into a clean, query-ready format, and make it available for analysis. The entire architecture is built on a foundation of serverless AWS services.

The entire infrastructure is defined as code using the AWS CDK making the deployment of the whole stack straightforward with a single command.

The solution implements the following architecture workflow:

• Data Ingestion (Amazon S3): Raw CSV files land in an S3 bucket. S3 acts as our scalable, serverless data lake.
• Orchestration (Amazon EventBridge & AWS Lambda): An Amazon EventBridge rule, a serverless cron job, kicks off the pipeline on a schedule (e.g., daily at 4 AM UTC). The rule triggers a Lambda function which in return starts the main ETL job with the correct parameters.
• ETL Processing (AWS Glue): The Lambda starts an AWS Glue job, which is the core of our pipeline. Glue is a serverless ETL service that automatically provisions compute resources to run our transformation script, and then shuts them down immediately after.
• Data Cataloging (AWS Glue Crawler): After the job completes, a Glue Crawler inspects the processed data (now in optimized Parquet format), infers its schema, and updates the AWS Glue Data Catalog.
• Querying (Amazon Athena): With the metadata available in the catalog, anyone can query the processed data in S3 using standard SQL with Athena, a serverless query engine.

Serverless and Software Design

Can you imagine a technology that lets you build automatically scaling, highly available software while optimizing for costs? This is the serverless proposition. But its real value is strategic, profoundly impacting how we approach software design. Specifically, how can you make any choices about your architecture, if your future use-cases aren’t known?

When you don’t know the future, a sacrificial architecture is invaluable. Martin Fowler defines sacrificial architecture as an architecture designed to be thrown away if the concept proves successful. In our context, cloud environments and especially serverless offerings make sacrificial architecture more attractive in order to build a Minimum Viable Product (MVP). Our taxi pipeline is a perfect MVP. We can deploy it quickly, gather feedback, and if requirements change, we can discard it with minimal sunk cost.

Now the success of a system depends on its ability to evolve. Designing a system isn’t a one-off operation. When the product direction, maturity, business requirements and team topologies change, then the success of a system will depend on its ability to evolve along with its demands. According to Martin Fowler’s foreword of the evolutionary architecture book: “The heart of doing evolutionary architecture is to make small changes, and put in feedback loops that allow everyone to learn from how the system is developing.”

A conjecture that can help us think about this is the well-known design stamina hypothesis, which stipulates that a system that has not been designed may be easy to develop in the beginning, but after a certain point (the so-called “design pay-off line”), a well-designed system will outperform it.

As shown in the diagram below, Serverless enables a “post-MVP redesign” strategy. You can start with a simple, serverless-first approach to ship quickly. As you learn and the system matures, you can refactor or redesign parts of the architecture without being locked into the initial design—a perfect example of an evolutionary architecture in practice.

The post-MVP redesign is comparably fast as the no-design approach delivering a lot of functionality and then shifts to the ideal scenario after redesigning to continuously deliver functionality in the long run.

Challenges and Limitations

That being said, it’s important to be aware of the challenges that come with serverless architectures.

• Cost Unpredictability: Unpredictable cost is a key aspect of serverless architecture that can be also a problem, especially for large enterprises operating on annually approved budgets. Cost unpredictability can make stakeholders reluctant to using a product where costs are mostly variable. For a scheduled workload like our ETL, the cost is quite predictable but that is not always the case and this is where active monitoring becomes a key to keep costs in check.

• Platform Limitations: Serverless platforms are always evolving. In the past, the AWS Glue service for example had limitations in terms of runtime choice and memory customization. While many of these have been addressed, it’s a reminder that you are dependent on the provider’s roadmap.

• Vendor Lock-in: Building on a specific cloud provider’s serverless offerings can tightly couple your application to their ecosystem, creating a trade-off between development speed and portability.

How to Choose?

So, should you go serverless? Consider these questions:

• Are you in a greenfield project where you need to iterate quickly and flexibly while building an MVP?
• Do you have a shortage of infrastructure developers in your team? Serverless empowers application developers to manage their own services.

If you answered yes to any of these, a serverless architecture is a powerful option. It’s not just a technical choice but a strategic one that prioritizes speed, agility, and the ability to evolve with changing business requirements.

With this we have reached the end of this post, I hope you enjoyed it!

If you have any remarks or questions, please don’t hesitate and do drop a comment below.

Stay tuned!

Resources

https://www.datadoghq.com/state-of-serverless/

https://martinfowler.com/articles/serverless.html

https://www.thoughtworks.com/radar/techniques/serverless-architecture

https://www.thoughtworks.com/en-gb/radar/techniques/lambda-pinball

https://www.freecodecamp.org/news/serverless-fully-managed-service-difference/

https://blogs.perficient.com/2021/06/17/aws-cost-analysis-comparing-lambda-ec2-fargate/

Contents

• What is Apache Spark
• Key features of Spark
• Resilient Distributed Datasets
  • Transformations
  • Actions
  • RDD vs DataFrame
• Fault Tolerance
• Spark Architecture
• Spark Components
• Lazy Evaluation
• DAG in Spark
• Caching
• Broadcast Variables
• Partitioning
  • Storage Partitioning vs Spark Partitioning
• Spark Applications Optimization

What is Apache Spark

Apache Spark is an open-source, distributed computing system designed for fast and general-purpose data processing. It was developed to address the limitations of Hadoop MapReduce, offering significant performance improvements and a more flexible programming model.

Key features of Spark

• Speed: Spark can be up to 100 times faster than Hadoop MapReduce for certain workloads, primarily due to its in-memory processing capabilities.
• Ease of Use: Spark provides high-level APIs in Java, Scala, Python, and R, making it accessible to a wide range of developers and data scientists.
• Unified Engine: Spark can handle diverse workloads including batch processing, interactive queries, streaming, machine learning, and graph processing, all within the same engine.
• Fault Tolerance: Spark achieves fault tolerance through the use of Resilient Distributed Datasets (RDDs) and their lineage information.
• Scalability: Spark can scale from one to thousands of nodes, allowing for efficient processing of large datasets.
• Flexibility: Spark can run in various environments, including Hadoop, Mesos, standalone, or in the cloud and it can access diverse data sources.

Resilient Distributed Datasets

Resilient Distributed Datasets (RDD) is the fundamental data structure in Apache Spark.

• Resilient: RDDs are fault-tolerant. If a partition of an RDD is lost due to node failure, it can be reconstructed using the lineage information.
• Distributed: Data in RDDs is divided into partitions and distributed across nodes in a cluster.
• Immutable: Once created, RDDs cannot be modified. Any transformation on an RDD creates a new RDD.
• Lazy Evaluation: Transformations on RDDs are lazily evaluated. They are not computed until an action is called.
• In-memory Computation: RDDs can be cached in memory for faster access in iterative algorithms.

RDDs support two types of operations: Transformations and Actions.

Transformations

These are operations that create a new RDD from an existing one and they are lazy (not computed immediately).

Examples of transformations:

• map(func): Apply a function to each element in the RDD. The result is an RDD with the same number of elements as the original.
• filter(func): Return a new RDD containing only the elements that pass the filter condition.
• flatMap(func): Similar to map, but each input item can be mapped to 0 or more output items. The results are flattened into a single RDD.
• groupByKey(): Group the values for each key in the RDD.
• reduceByKey(func): Combine values with the same key using the provided function.

Example of map vs flatMap usage:

      
    rdd =  sc.parallelize(["Hello World", "How are you"])
    
    map_result = rdd.map(lambda x: x.split())
    # Result: [["Hello", "World"], ["How", "are", "you"]]

    flatmap_result = rdd.flatMap(lambda x: x.split())
    # Result: ["Hello", "World", "How", "are", "you"]
   
   

Actions

These are operations that return a result to the driver program or write data to an external storage system. They trigger the execution of all the transformations that were called before it.

Examples of actions:

• collect(): Return all the elements of the RDD as an array to the driver program.
• count(): Return the number of elements in the RDD.
• first(): Return the first element of the RDD.
• take(n): Return an array with the first n elements of the RDD.
• reduce(func): Aggregate the elements of the RDD using a function.
• saveAsTextFile(path): Save the elements of the RDD as a text file.

RDD vs DataFrame

A Spark DataFrame is a distributed collection of data organized into named columns, similar to a table in a relational database or a data frame in R/Python. It’s built on top of RDDs and provides a more user-friendly API for structured and semi-structured data. It uses Catalyst Optimizer, which can significantly improve performance and has a defined Schema, allowing Spark to optimize query plans.

Fault Tolerance

Spark achieves fault tolerance primarily through the lineage of RDDs and the ability to recompute lost data.

• RDD Lineage:
  • Each RDD maintains information about its lineage (how it was derived from other datasets).
  • If a partition of an RDD is lost, Spark can rebuild it using this lineage information.
• Checkpointing:
  • For long lineage chains, Spark allows saving intermediate results to reliable storage (like HDFS).
  • This reduces recovery time in case of failures.
• Speculative Execution:
  • Spark can run multiple copies of slower tasks to reduce the impact of stragglers.
• Stage Retry:
  • If a task fails, Spark will retry it on a different executor.
  • If a whole stage fails, Spark can resubmit the entire stage.
• Data Replication:
  • When caching data, Spark can replicate it across nodes for added resilience.
• Driver and Worker Fault Tolerance:
  • Spark can recover from worker node failures.
  • For driver failures, Spark supports checkpointing of the driver’s state in some deployment modes.

Spark Architecture

• Driver Program: Contains the main() function and creates a SparkContext.
• SparkContext: The entry point for Spark functionality, representing the connection to the Spark cluster.
• Cluster Manager: Allocates resources across applications.
• Worker Nodes (Slaves):
  • Execute the tasks assigned by the driver.
  • Store data partitions.
• Executor
  • A process launched for an application on a worker node, that runs tasks and keeps data in memory or disk storage across them.
  • Each application gets its own executor processes, which stay up for the duration of the whole application and run tasks in multiple threads. This has the benefit of isolating applications from each other, on both the scheduling side (each driver schedules its own tasks) and executor side (tasks from different applications run in different JVMs). However, it also means that data cannot be shared across different Spark applications (instances of SparkContext) without writing it to an external storage system.
• Task: A unit of work that will be sent to one executor.
• Job: A parallel computation consisting of multiple tasks that gets spawned in response to a Spark action.
• Stage: Each job gets divided into smaller sets of tasks called stages that depend on each other.

Spark Components

• Spark Core: The foundation of the entire Spark system, providing distributed task dispatching, scheduling, and basic I/O functionalities.
• Spark SQL: Module for working with structured data. It allows querying data via SQL as well as the Hive Query Language (HQL) and supports various sources like Hive tables, Parquet, and JSON.
• Spark Streaming: Enables processing of live streams of data. It provides a high-level abstraction called DStream (discretized stream).
• MLlib (Machine Learning Library): A distributed machine learning framework on top of Spark Core. It provides various utilities for machine learning tasks, including classification, regression, clustering, and collaborative filtering.
• GraphX: A distributed graph-processing framework on top of Spark. It provides an API for expressing graph computation and can model user-defined graphs.
• SparkR: An R package that provides a light-weight frontend to use Spark from R.

Lazy Evaluation

Lazy evaluation is a key optimization technique used in Spark. When you apply a transformation on an RDD, Spark doesn’t compute the results immediately. Instead, it remembers the set of transformations applied to some base dataset. The transformations are only computed when an action requires a result to be returned to the driver program.

Importance of lazy evaluation:

• Optimization: Spark can optimize the execution plan by analyzing the full set of transformations before executing.
• Efficiency: It reduces the number of passes over the data by grouping operations.
• Reduced Computation: If the final action only needs to compute a small result, Spark can minimize the amount of data processed.
• Fault Tolerance: Lazy evaluation allows Spark to reconstruct lost data by recomputing only the lost partitions from the original data.

DAG in Spark

In Spark, a Directed Acyclic Graph (DAG) is a conceptual model of the execution plan for a set of operations on RDDs.

A DAG is a graph where each node represents an RDD partition, and the edges represent the operations to be performed on the RDD. The edges have a direction, indicating the flow of data from one operation to the next. There are no cycles in the graph, meaning the operations flow in one direction and don’t loop back.

Spark’s DAG scheduler optimizes the execution plan by analyzing the graph and combining operations where possible. The DAG is divided into stages. A stage is a set of tasks that can be executed together without shuffling data. If a node fails, Spark can reconstruct the lost partitions using the lineage information stored in the DAG.

The DAG is constructed when actions are called, not when transformations are defined.

Caching

Caching in Spark is a technique used to store the intermediate results of RDD computations in memory or disk. This allows faster access when the same RDD is used multiple times.

• Methods:
  • cache(): Stores the RDD in memory.
  • persist(): Allows specifying the storage level (memory, disk, or both).
• Storage Levels: Spark provides different storage levels like MEMORY_ONLY, MEMORY_AND_DISK, DISK_ONLY, etc.
• Lazy Evaluation: Caching is also lazily evaluated. The RDD isn’t cached until the first action that uses it.
• Automatic Memory Management: Spark automatically manages the cached data using LRU (Least Recently Used) eviction policy.
• Unpersist: You can manually remove cached data using the unpersist() method.

Broadcast Variables

Broadcast variables in Spark are read-only variables that are cached on each machine in the cluster rather than shipped with every task. They are used to efficiently share large read-only data across all nodes in a cluster. They are commonly used for lookup tables, machine learning models, or any large read-only data structure.

      
    lookup_table = sc.broadcast({1: "A", 2:"B", 3:"C"})

    def lookup_function(x):
        return lookup_table.value.get.(x, "Unknown")
    
    result = rdd.map(lookup_function)
   
   

Partitioning

Spark automatically partitions RDDs based on the input data source or the number of cores available. There are three types of partitioning:

• Hash Partitioning: Based on the hash code of the key.
• Range Partitioning: Based on ordered keys.
• Custom Partitioning: User-defined partitioning logic using methods like repartition() or coalesce().

Partitioning is important because:

• It affects the level of parallelism in data processing
• It influences the amount of data transfer during shuffles
• It impacts the efficiency of certain operations (e.g., joins, aggregations)

Storage Partitioning vs Spark Partitioning

These are two separate concepts, but understanding the difference is relevant as storage partitioning can impact Spark’s performance.

While storage partitioning refers to how data is organized on disk (e.g. in S3 or HDFS), Spark partitioning is about how data is distributed across executors for processing.

Storage partitioning allows partition pruning; Spark can skip reading irrelevant partitions based on query filters which leads to faster query processing times by reducing data scanned.

In order to read partitioned data, Spark performs partition discovery to identify partitions based on directory structure. It can also leverage partition information for optimizations like partition pruning. However, Spark still creates its own internal partitions for processing, which may differ from storage partitions.

Spark Applications Optimization

• Data Serialization: Use Kryo serialization instead of Java serialization for better performance.
• Proper Data Partitioning: Ensure data is well-distributed across partitions to avoid skew.
• Caching and Persistence: Use cache() or persist() for RDDs used multiple times.
• Avoid Shuffling: Minimize operations that cause data shuffling (e.g., groupByKey, reduceByKey).
• Use Broadcast Variables: For large shared data that needs to be distributed to all nodes.
• Optimize Data Formats: Use columnar formats like Parquet for better compression and query performance.
• Tune Spark Configurations: Adjust executor memory, number of executors, and other Spark parameters.
• Use Appropriate Join Strategies: Choose the right join strategy (broadcast joins for small-large table joins).
• Avoid UDFs When Possible: Use built-in functions instead of User Defined Functions for better performance.
• Monitor and Profile: Use Spark UI and other profiling tools to identify bottlenecks.

If you have any remarks or questions, please don’t hesitate and do drop a comment below.

Stay tuned!

Contents

• What is Vibe Coding?
• Expectations
• Key Takeaways
• Recap

What is Vibe Coding?

Vibe coding is a term coined by Andrej Karpathy in his X post that was shared on February 2, 2025. However, the term has definitely evolved in my opinion since the original description. It is indeed a new kind of coding where the developer has become more like a moderator of multiple chat sessions and agents running in the background resulting in a handful of new and updated files with new generated code by the large language model that are waiting for his review.

Expectations

This is not gonna be one of those I built a SaaS app in 15 minutes. It is easy and at times surprisingly good how fast you can generate a landing page for an idea or have a decent CRUD app running but things get more complicated when building complex SaaS.

In my experiment, I wanted to answer a simple question: can AI assisted development make me move faster through the backlog?

Obviously faster does not always mean better and the metric for measuring the outcome should not be the number of lines of code generated because if anything you won’t be disappointed: LLMs can truly quickly generate a lot of code!

As this was a personal side project, the stakes were not that high but the role of the moderator as mentioned earlier is still of huge importance. As you can see from the diagram below, I dealt with times where the model generated thousands of lines of code but almost always not all of it was accepted. You have to be a gatekeeper protecting your codebase from bugs or introducing weird behaviour and making sure to always follow clear design patterns and software development best practices.

So over the course of two weeks, I used Cursor to try to answer my question. I did not start from scratch and vibe coded my way into this project but rather build on top of an existing web application and extended it with new features. That means I already had a clear code structure with clearly defined interfaces and domain models that have greatly influenced the LLM’s code organization and structure.

When it comes to the features, some of them were straightforward but many weren’t. These were a collection of cards with merely a title in a Trello board that came from ideas I thought were cool to implement someday so I just wrote them down there quickly to not lose track of them. This point is important as we’ll see later because it influences how you approach building these features and how to draft your prompts for that.

A final thing to mention before moving on to the key takeaways is the distribution of the used programming languages in the project:

I’m not a frontend developer and my Javascript skills at this point were a bit rusty to say the least but surprisingly this is the part where I made most of the progress that I couldn’t have done without the AI assistance in such a short time.

Key Takeaways

Following are the observations and notes I took as I progressed in this experiment:

• Use user journeys and expected behaviour in your prompt when explaining a feature especially one that would require changes across both frontend and backend logic.
• Be specific and start with thin slices. Describing multiple features and expectations in the same prompt just because they are correlated will not result in better results but rather only confuse the model.
• Request analysis of the code structure and design by giving the whole codebase as context to the mode. This helps you reflect on the progress made so far, assess the list of features implemented and decide what to do next because it is easy to have decision paralysis in this honeymoon phase where you are in love with how productive you are and everything seems feasible that you just want to do it all at once.
• Asking the model to simplify any implementation is always good: rely on your judgment of assessing what looks good and what looks meh.
• If the model gets stuck with a particular implementation or request, expect to have duplicated code snippets and functions with similar logic but slightly different names resulting from multiple attempts at solving the issue. This means that the model was really bad at cleaning up dead code unless you specifically ask it to evaluate some code by highlighting it.
• After several interactions within the same chat, it is better to continue or start over in a new one: It helps you clear your thoughts with a fresh new prompt and therefore guide the model to better output.
• I can’t stress this enough but if you are not starting from scratch and you will be updating an existing codebase, have your unit tests ready as your defense mechanism.
• From time to time, it is useful to start your prompt by specifically asking the model not to make code changes. Ask it to give you multiple implementation options and review the suggestions before proceeding with any of them. If needed drill down on the one you found most appealing with follow up questions and clarifications.
• Commit changes frequently especially between requests when you are satisfied with the suggested changes. This way it is easier to review follow up code changes in particular those made to the same files.
• Review the whole codebase once you have committed new changes to keep your understanding of it in check and be able to track down potential issues or improvements.
• Some changes are really small and you can just make them in place using the tab functionality:

With this we have reached the end of this post, I hope you enjoyed it!

If you have any remarks or questions, please don’t hesitate and do drop a comment below.

Recap

Coming back to the original quest of this whole experiment: did AI supported me in doing things faster? for sure it did. Like any other tool when used properly and consciously it will help augment your abilities to do your work better and support you in many ways. Will programmers be replaced by AI? I’ll take that with a pinch of salt.

Contents

• What is Apache Kafka?
• Core Concepts
• Key Features
• Message delivery semantics
• Use Cases

What is Apache Kafka?

Apache Kafka is an open-source distributed event streaming platform. It consists of highly scalable and fault tolerant servers enabling real-time data ingestion and processing between clients that are decoupled (source and target) and can scale independently.

Core Concepts

• Cluster: A group of Kafka broker servers that work together to manage and distribute data.
• Broker: A Kafka server that stores data and serves client requests (producers and consumers). Multiple brokers form a cluster.
• Topic: A named stream of records to which producers send data and from which consumers read. Topics are split into partitions.
• Partition: Each topic is divided into partitions, which are ordered, immutable sequences of messages. Partitions enable parallelism and scalability.
• Producer: An application or service that sends (publishes) records to Kafka topics.
• Message Key: producers can choose to send a key with records then messages for that key will always go to the same partition.
• Consumer: An application or service that reads (subscribes to) records from Kafka topics.
• Consumer Group: A group of consumers that work together to consume data from a topic, ensuring each message is processed only once by the group.
• ZooKeeper: Used for managing and coordinating Kafka brokers (leader election, metadata, etc.). Note: Newer Kafka versions are moving toward removing ZooKeeper dependency.
• Offset: A unique identifier for each message within a partition, used by consumers to keep track of read messages.
• Replication: Each partition can be replicated across multiple brokers to ensure durability and high availability.

Key Features

• High Throughput: Capable of handling millions of messages per second.
• Low Latency: Designed for real-time streaming and processing.
• Scalability: Scales horizontally by adding brokers and partitions.
• Fault Tolerance: Data is replicated across brokers; if one fails, another can take over.
• Durability: Messages are persisted on disk and replicated.
• Decoupling: Producers and consumers are independent, enabling flexible architectures.
• Multiple APIs:
  • Admin API: manage and inspect topics, brokers, and other Kafka objects.
  • Producer API: Publish data to topics.
  • Consumer API: Subscribe to and process data from topics.
  • Streams API: Build stream processing applications.
  • Connect API: Integrate with external systems (databases, file systems, etc.)

Message delivery semantics

Apache Kafka provides three primary message delivery semantics:

• At-most-once: Messages are delivered zero or one time. Some messages may be lost, but never delivered more than once. Typical usage in applicaitons with high-throughput, low latency requirements and risk of data loss.
• At-least-once: Messages are delivered one or more times. No message is lost, but duplicates may occur. Typical usage in data pipelines where no data loss is acceptable, and duplicates can be handled.
• Exactly-once: Each message is delivered once and only once. No data loss or duplicate delivery. Ensures no message loss or duplication, but with increased latency and configuration overhead Typical usage in Financial transactions and critical data flows.

How kafka achieves these semantics:

• At-most-once:
  • Producer sends messages without waiting for acknowledgment (acks=0).
  • If a failure occurs before delivery, messages may be lost.
  • Consumer commits its offset before processing messages. If it crashes after committing but before processing, messages are lost.
• At-least-once:
  • Producer waits for acknowledgment (acks=1 or acks=all).
  • If acknowledgment is not received, the producer retries, which can result in duplicate messages.
  • Consumers must be idempotent to handle possible duplicates
  • Consumer commits offset after processing. If it crashes before committing, messages may be processed again after recovery (duplicates possible).
• Exactly-once:
  • Producer uses idempotence and transactions; each message is written once even if retried.
  • Consumers and producers must be properly configured for transactional processing. Offset commits and output are part of the same transaction, ensuring atomicity.

Use Cases

Some of the popular use cases for Apache Kafka include:

• Messaging: replacement to traditional message broker for decoupling data processing between producers and consumers.
• Website activity tracking: this is the original use case where a user’s site activity like page views, clicks and searches events are published to central topics and available for consumption from real time analytics and insights applications.
• Metrics: similarly apache kafka is used in aggregating statistics from distributed applications to produce centralized feeds of operational data.
• Log aggregation: used as a replacement for log aggregation solutions giving cleaner abstraction of log or event data as a stream of messages.

If you have any remarks or questions, please don’t hesitate and do drop a comment below.

Stay tuned!

Contents

• What is Apache Airflow?
• Core Concepts
  • Directed Acyclic Graphs (DAGs)
  • DAG Run
  • Tasks
  • Task Instances
  • Variables
• Architecture Components
• Architecture Components

What is Apache Airflow?

Apache Airflow is an open-source workflow management platform designed to programmatically author, schedule, and monitor complex data pipelines.

Core Concepts

Directed Acyclic Graphs (DAGs)

The fundamental structure in Airflow, representing a collection of tasks with defined dependencies. Each DAG is defined in Python code and dictates the order of task execution based on their relationships. A DAG is a graph structure where tasks are represented as nodes, and the dependencies between these tasks are represented as directed edges. The “directed” aspect indicates that tasks have a specific order of execution, while “acyclic” means there are no loops or cycles, preventing infinite execution paths.

DAG Run

A DAG Run is an object representing an instantiation of the DAG in time. Any time the DAG is executed, a DAG Run is created and all tasks inside it are executed. The status of the DAG Run depends on the tasks states. Each DAG Run is run separately from one another, meaning that you can have many runs of a DAG at the same time.

Tasks

Task is the individual units of work within a DAG. Each task represents a single operation, such as data extraction, transformation, or loading (ETL). The relationships between tasks are established using dependency definitions. This can be done through:

• Bitwise Operators: Using » to set downstream dependencies and « for upstream dependencies.
• Methods: Using set_upstream() and set_downstream() methods to explicitly define task relationships.

There are three common types of task:

• Operators, conceptually a template for predefined tasks that you can string together quickly to build most parts of your DAGs.
• Sensors, a special subclass of Operators which are entirely about waiting for an external event to happen.
• A TaskFlow-decorated @task, which is a custom Python function packaged up as a Task.

To pass data between tasks you have three options:

• XComs (“Cross-communications”), a system where you can have tasks push and pull small bits of metadata identified by a key as well as the task_id and dag_id it came from.
• Uploading and downloading large files from a storage service (either one you run, or part of a public cloud)
• TaskFlow API automatically passes data between tasks via implicit XComs

Task Instances

Much in the same way that a DAG is instantiated into a DAG Run each time it runs, task instances are specific executions of tasks at particular times, which can vary based on the DAG’s scheduling.

Variables

Variables are Airflow’s runtime configuration concept - a general key/value store that is global and can be queried from your tasks, and easily set via Airflow’s user interface, or bulk-uploaded as a JSON file. Variables are global, and should only be used for overall configuration that covers the entire installation; to pass data from one Task/Operator to another, you should use XComs instead.

Architecture Components

• Scheduler: The component responsible for scheduling tasks and determining when they should run. It checks the DAG directory for tasks that need to be executed.
• Executor: This defines how and where tasks are executed. Various executors are available. In the default Airflow installation, this runs everything inside the scheduler, but most production-suitable executors actually push task execution out to workers. Most executors will generally also introduce other components to let them talk to their workers - like a task queue - but you can still think of the executor and its workers as a single logical component
• Web Server: Provides a user interface for monitoring and managing workflows, allowing users to inspect DAGs and task statuses.
• A folder of DAG files, read by the scheduler and executor (and any workers the executor has)
• Metadata Database: Stores all metadata related to DAGs and tasks, typically using PostgreSQL or MySQL.

Architecture Components

• Task Management: Airflow manages task dependencies automatically, ensuring that tasks execute in the correct order.
• Scheduling: Airflow provides advanced scheduling capabilities, allowing workflows to run on defined schedules or trigger based on external events.
• Extensibility: Users can create custom operators and plugins to extend Airflow’s functionality, integrating with various data sources and services.
• Error Handling and Retries: Built-in mechanisms allow tasks to be retried automatically upon failure, enhancing workflow reliability.
• Scalability: Airflow can handle thousands of concurrent tasks across multiple workers, making it suitable for large-scale data operations.
• Rich Command Line Interface (CLI): The CLI provides utilities for managing DAGs and executing tasks directly from the command line.
• Integration with Other Tools: Airflow supports integration with various cloud services and data tools, including AWS, Google Cloud Platform, and many others.

If you have any remarks or questions, please don’t hesitate and do drop a comment below.

Happy learning!