Viewing the Content of Spark DataFrames in Fabric Notebook

In this article, we will explore two ways to preview the content of the dataframe in Fabric Notebook – Show and Display.

In the example code below, I explicitly defined a schema for my sales data using StructType and StructField and imported the necessary pyspark libraries.

Then, we have the actual sales data with 50 rows and 8 columns.

Next, I created a Spark dataframe with sample sales data stored in the sales_df variable.

To view the content of the dataframe, I can use the show() method. As seen below, we have the result of the dataframe. By default, the top 20 rows are returned.

We can specify option arguments within the show method. For example, specifying sales_df.show(n=30) will display 30 rows as seen below.

We can also use the vertical argument to display the rows vertically. By default, the vertical argument is False. But when set to True, the content of the DataFrame is displayed vertically, as seen below.

The second way we can view the content of the Spark dataframe is to use the display function. To view, I executed display(sales_df).

The display function is a Fabric Notebook-specific function that offers a richer and more interactive view of the DataFrame compared to the show() method. The display function provides a unique and dynamic web-based table that allows users to use Data Wrangler, search, and create a chart visualization.

To quickly visualize the displayed dataframe, all I need to do is click on New chart, and there we go, as seen below.

To search for any value in the table, I can type that value in the search box. For example, I type elec in the search box, and the displayed dataframe is filtered in real-time, as seen below.

The display function also allows us to limit the number of rows returned. For example, I used the limit function to return just 5 rows, as seen below.

You can also download the table in different formats, such as CSV, JSON, and XML. To do that, click on Download and choose the preferred file format.

You can also inspect the table profile by clicking on the chevron icon to see the distribution of values in each of the displayed columns, as seen below.

Hope you find the article useful. See you in the next article.