Modern Data Wrangling with Polars and Delta Lake in Microsoft Fabric

As a data engineer working in Microsoft Fabric, I’ve always been excited about combining performance with simplicity. In this article, I’ll walk you through how I load, transform, and write data to Delta tables using Polars — all within the Fabric ecosystem.

Why Polars?

Before diving into the implementation, let me tell you why I use Polars.

• Speed: It’s blazingly fast due to its Rust-based engine.
• Lazy Execution: Optimizes queries before execution.
• Ease of Use: The API feels familiar to anyone used to pandas or SQL.
• Seamless Integration: I can easily convert between Polars and PySpark or pandas when needed.

Step-by-Step Implementation

Let’s walk through how I work with Polars in a Fabric Python Notebook and write the data into a Delta table using two methods.

First, I have sales2015 to sales2025 CSV files loaded in my Fabric Lakehouse as seen below.

Install Polars Packages

In the Fabric Notebook, I am going to install the Polars packages and dependencies to perform the data processing. So, I executed the code as seen in the screenshot to install the packages.

Load Data from Lakehouse to Data into Polars

Next, I am going to load all the CSV files in the Sales Lakehouse Files section of Fabric. In addition, the has_header and inferSchema properties of Polars were set to true because the sales data container headers are in row one of each file and use the display method to display the content of the Polars DataFrame.

We should see the data displayed nicely, no need for Spark just yet.

Transform the Data Using Polars

Now I want to extract the Year from the OrderDate column.

Convert Polar DataFrame to a PySpark DataFrame

Fabric's Delta support works via Spark, so I need to convert my Pandas DataFrame to a PySpark DataFrame before writing it to a Delta table.

Write Data to a Delta Table using Two Methods

We can save the transformed data into a Delta table in Fabric using two methods. We can also see the respective names of the delta tables: transaction_data and sales_delta.

To conclude, working with Polars in Fabric Python Notebooks is a game changer is cool. We are assured of getting the speed and expressiveness of Polars for transformation, and the robustness of Delta tables for storage, scalability, and interoperability.