Do you understand how Apache Iceberg actually works? I have tried to understand it multiple times, but it always felt hard to get into compared to basic Hive tables. When searching for references it often feels more like reading an advertisement with features like: ACID for open file formats, schema evolution and time travel. But even after using Apache Iceberg, I did not grok it before doing practical experiments and sitting down to read the official specification thrice. This post fills the gap of high-level overview and implementation and walks you through the most important features of Apache Iceberg by example, showing you how they work and why they are actually such an huge improvement over Hive tables.

Warning: This post is long and it could take you up to an hour to read it completely.

Apache Arrow is everywhere nowadays, and if you are a data engineer, your favorite tools/frameworks probably have one of its libraries as a dependency. The official list of products that use Apache Arrow is filled with big players in the data ecosystem and new systems built on top of the Apache Arrow based query engine Apache DataFusion pop up regularly. It seems that no other project is shaping the future of the data landscape quite like Apache Arrow does. So, as a data engineer, trying to understand what all the fuss is about came naturally to me.

Getting a high-level overview of Apache Arrow is easy. There are many good (and quite recent) blog posts on its history and the problems it solves, see this or that one. The Apache Arrow project page is also very well written. But for someone like me, who lacked a lot of basic computer science or system programming knowledge, Apache Arrow never really "clicked". This blog post aims to fill this gap. I will cover the concepts that are needed to not just believe that a standardized columnar memory layout makes sense, but to actually get that its creation was inevitable. But note, that I am not an expert on these topics and I will omit details for readability. To get the full picture please refer to the specifications.

Learning has always been an ambivalent topic for me. In school, it was cumbersome, and I didn’t find much joy in it. On the other hand, I’ve always enjoyed picking up new skills that interest me, like juggling or card magic. While I’ve never deeply developed any particular skill, I’ve become better at learning autonomously.

This autonomy helped me later in life, especially when I did a career switch from physics to software/data engineering three years ago. While I was skilled enough to land a job, I lacked knowledge in several areas and felt behind with a lot of ground to cover. My old, intuitive way of learning had worked before, but it felt inefficient and unsatisfactory. I didn't feel in control of the things I learned, and I reacted more to the things I randomly encountered rather than following a structure. Also, nothing really seemed to stick for long. I often had to relearn things, which felt more like starting from scratch rather than building on the past.

As I believe that learning is one of the most important skills in life, especially in areas that change quickly like software engineering, I decided to try to get better at it. I have read, viewed and studied some resources and tried various approaches and refined the ones that worked best for me. In the following, I present some of my key findings that I believe have been the most important in improving my way of learning. Note that all of this is non-scientific, highly opinionated and just a snapshot of my current thinking.

After I installed Proxmox on my Beelink SER 5 - Ryzen 7 5800H I wanted to start an Ubuntu VM with GPU passthrough that I could use as a daily driver. As I had zero prior experience with doing something like this my process was full of following guides I barely understood, frustrating trial and error and reading about the things I just tried to do. This journal describes this process. It should not be seen as a complete guide, as I am far too inexperienced to verify if I did everything correctly. But maybe it helps someone like me who is lost and googling error messages.

Before describing my journey I want to point out a few guides, with which I would not have been able to get it done:

• A guide that describes the process for the same machine I used but for a Windows VM: This guide saved me. Before I found it I was close to giving up. So big shout outs.
• Comprehensive guide for Ryzen 7000 series GPU passthrough: The guide above used this one for many steps.
• GPU passthrough to an Ubuntu VM, but with a NVIDIA GPU: This was the first guide I started with and it got me on track. Unfortunately it did not work out of the box for me. (Probably because I also have a different GPU setup.)