Introduction To Pagination
Pagination is the process of displaying a huge number of records by splitting them into multiple pages. This can be better explained by an example. Consider you have 10000 records in a table and you want to show them to the end user upon request. The very simple thing is to display all of them by fetching from the database with a simple select * from <Table> query. OK fine you are done and your job is over. You have checked in the code and your Lead or some testing team person is verifying your features implementation. Suppose your manager saw the application and the particular feature. Can you imagine what might be his reaction to that.
I can explain the situation how it will be, as I have faced this situation in earlier days of coding.
- Question-I: What is this?
- Question-II: What have you done?
- Question-III: Is the end user able to read such a long list?
- Question-IV: How much time does it take to load all these records?
- Question-V: Did you consider the usability of this application when implementing this feature?
Yes your manager is right. Just think from the end user's point of view. If you see such a long list, how can you use it. The user might need to scroll to the end to see the last user. It takes a very long time to load the list. The process of fetching such a long list from a database is very time consuming and sometimes your application might get a timeout exception.
Here I will explain from the very beginning how to fully optimize the pages.
We need the following software to start our tutorial.
Microsoft SQL Server Express Edition or a trial version as at here.
Microsoft Visual Studio 2012 Ultimate trial version for 90 days here.
Fine; install the above software according to the guidelines provided.
Create Database
Create a new database in SQL Server 2012 as shown below and name the database as pagination.
Now add a new table to the database and name it as employeelist with the following columns:
EmployeeID:
FirstName
LastName
Department
Location
Experience
Salary
As show here:
Script to create EmployeeList Table for pagination:
USE [Pagination]
GO
/****** Object: Table [dbo].[EmployeeList] Script Date: 10/1/2012 2:54:39 PM ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [dbo].[EmployeeList](
[EmployeeID] [bigint] IDENTITY(1,1) NOT NULL,
[FirstName] [varchar](200) NOT NULL,
[LastName] [varchar](200) NOT NULL,
[Department] [varchar](200) NOT NULL,
[Experience] [decimal](18, 0) NOT NULL,
[Salary] [decimal](18, 0) NOT NULL,
CONSTRAINT [PK_EmployeeList] PRIMARY KEY CLUSTERED
(
[EmployeeID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
Now add 1000 rows with the simple following pl-sql program
DECLARE @intFlag INT
SET @intFlag = 1
DECLARE @Random INT
DECLARE @Upper INT
DECLARE @Lower INT
DECLARE @Exp INT
DECLARE @Sal INT
DECLARE @DEP INT
WHILE (@intFlag <=1000)
BEGIN
---- This will create a random number between 1 and 999
SET @Lower = 1 ---- The lowest random number
SET @Upper = 10 ---- The highest random number
SELECT @Exp = ROUND(((@Upper - @Lower -1) * RAND() + @Lower), 0)
SELECT @Random
---- This will create a random number between 1 and 999
SET @Lower = 10000 ---- The lowest random number
SET @Upper = 100000 ---- The highest random number
SELECT @Sal = ROUND(((@Upper - @Lower -1) * RAND() + @Lower), 0)
SELECT @Random
---- This will create a random number between 1 and 999
SET @Lower = 1 ---- The lowest random number
SET @Upper = 10 ---- The highest random number
SELECT @DEP = ROUND(((@Upper - @Lower -1) * RAND() + @Lower), 0)
SELECT @Random
INSERT INTO EmployeeList(FirstName,LastName,Department,Experience,Salary)
VALUES ('first name'+cast(@intFlag as varchar(5)),'last name'+cast(@intFlag as varchar(5)),'department'+cast(@DEP as varchar(2)),cast(@Exp as varchar(2)) ,cast(@Sal as varchar(7)))
SET @intFlag = @intFlag + 1
--IF @intFlag = 4
--BREAK;
END
GO
Create a procedure to get the unique department names from the EmployeeList Table, as in:
USE [Pagination] GO /****** Object: StoredProcedure [dbo].[GetDepartments] Script Date: 11/6/2012 1:42:55 PM ******/ SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO -- ============================================= -- Author:<Author,,Name> -- Create date: <Create Date,,> -- Description:<Description,,> -- ============================================= CREATE PROCEDURE [dbo].[GetDepartments] -- Add the parameters for the stored procedure here AS BEGIN -- SET NOCOUNT ON added to prevent extra result sets from -- interfering with SELECT statements. SET NOCOUNT ON; -- Insert statements for procedure here SELECT Distinct(Department) from EmployeeList order by department END
Yes you are now ready with the database and it is totally functional. We will now start with the ASP.Net side.
Creating New Application
Open Visual Studio 2012 and click on new project. It will open the already existing templates. Under Web templates select empty website for our case and name it Pagination.
Now add a new aspx page to the application as show in the picture below and name it employeelist.
Add quickly a ListView and name it lvEmployeeList and design the template for showing the employee list the page will look like this:
<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="EmployeeList.aspx.cs" Inherits="Pagination.EmployeeList" %>
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
<title></title>
</head>
<body>
<form id="form1" runat="server">
<div>
<asp:ListView ID="lvEmployeeList" runat="server">
<LayoutTemplate>
<table style="width: 100%; text-align: left;">
<tr>
<td style="width: 20%; text-align: center;">First Name</td>
<td style="width: 20%; text-align: center;">Last Name</td>
<td style="width: 20%; text-align: center;">Department</td>
<td style="width: 20%; text-align: center;">Experience</td>
<td style="width: 20%; text-align: center;">Salary</td>
</tr>
<tr runat="server" id="itemPlaceHolder"></tr>
</table>
</LayoutTemplate>
<ItemTemplate>
<tr id="itemPlaceHolder">
<td style="width: 20%; text-align: center;"><%#Eval("FirstName") %></td>
<td style="width: 20%; text-align: center;"><%#Eval("LastName") %></td>
<td style="width: 20%; text-align: center;"><%#Eval("Department") %></td>
<td style="width: 20%; text-align: center;"><%#Eval("Experience") %></td>
<td style="width: 20%; text-align: center;"><%#Eval("Salary") %></td>
</tr>
</ItemTemplate>
</asp:ListView>
</div>
</form>
</body>
</html>
So we have made a front end to render and show the employee details. Now we need to write a method to get rows from SQL Server.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Data;
using System.Data.Sql;
using System.Data.SqlClient;
namespace Pagination
{
public partial class EmployeeList : System.Web.UI.Page
{
string connectionstring;
protected void Page_Load(object sender, EventArgs e)
{
connectionstring = System.Web.Configuration.WebConfigurationManager.ConnectionStrings["PaginationConnectionString"].ConnectionString;
if (!IsPostBack)
GetDetails();
}
private void GetDetails()
{
DataTable dtEmployeeList = new DataTable("EmployeeList");
using (SqlConnection con = new SqlConnection(connectionstring))
{
SqlCommand cmd = new SqlCommand("select * from employeelist");
cmd.Connection = con;
con.Open();
using (SqlDataAdapter sda = new SqlDataAdapter(cmd))
{
sda.Fill(dtEmployeeList);
}
}
lvEmployeeList.DataSource = dtEmployeeList;
lvEmployeeList.DataBind();
}
}
}
Ok everything is ready and we have 1000 records in our database. We have code to fetch and show them. We quickly run and see the result.
Wow! Awesome; we got the page loaded in less than a second, it's great. Ok we will try with many more of these records. In my future parts I will explain each individual section with the time loading and performance optimization features.
We saw that with 1000 records the page is loading in less than a second; that works fine. Now I have added more than 2,00,000 records and when I ran the same page my system got hung; the reason is simple. The application memory is not sufficient to handle such a huge bunch of record sets. And the processing fetching time for those records from SQL Server is pathetic.
So what is the solution?
Points to Consider
Optimize the database query
Optimize indexing
Implement client-side pagination
Less use of Server Controls
I will explain each and every step here with the time taken to process records sets.
Step: I calculate the current performance of the page by means of the page loading time with the help of Jetbrain's Dotrace profiling tool. The dotrace tool makes profiling applications very easy with less overhead. I have been using this for the last 4 years. This gives each and every line of ____ and the method execution times perfectly. So according to the dotrace report you can tune, optimize or re-factor your code accordingly.
To load the initial 10000 records the time taken is 765 ms. Check the figure below:
Now we will try to run with the 100000 records to check the performance. The performance is recorded as in the following:.
See, the difference is that as the record count grows, the loading time also increases. Here I am showing only the Page load method execution time. So the times seen here are very less, like less than half of the full page rendering and method execution. Suppose the time taken to load and render 100000 records for me is 3 minutes but the dotrace report I am showing here is only the execution time of the method.
Step-I: Optimize The Database Query
We can optimize the database query in a number of ways. The very first possibility is to check what columns are required.
Consider that we need only Name, Department and Experience. Change the query and see the result here:
SELECT [FirstName]+' '+[LastName] Name,[Department],[Experience] FROM EmployeeList
You can see the change here, 3000 milliseconds has been reduced in rendering the results to the page. So the content to be rendered onto a page and the set of results that should pass over the network, everything got reduced; that is why the change.
In the same way, we can reduce the number of rows for a page by applying a filter which will provide the best usability for the end user as well as provide the best performance for the applications. So let's take three columns to filter the records; Name, Department, and Experience.
Change the Stored Procedure code accordingly to use these parameters.
USE [Pagination]
GO
/****** Object: StoredProcedure [dbo].[GetEmployeeList] Script Date: 10/13/2012 4:09:34 PM ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE [dbo].[GetEmployeeList]
@name varchar(200)=null,
@departmanet varchar(200)=null,
@expstart int = null,
@expend int=null
AS
BEGIN
SET NOCOUNT ON;
SELECT top 100000 [FirstName]+' '+[LastName] Name,[Department],[Experience] FROM EmployeeList
where
(FirstName like '%'+@name+'%' or @name is null)
and (LastName like '%'+@name+'%' or @name is null)
and (Department =@departmanet or @departmanet is null)
and ((Experience >@expstart or @expstart is null) and (Experience<@expend or @expend is null ) )
END
This will provide you the exact results for your query with the best application performance. Modify your screen so that the end user can input the the parameter values as shown in the following figure:
And modify the Codebehind as in the following:
private void GetDetails()
{
DataTable dtEmployeeList = new DataTable("EmployeeList");
using (SqlConnection con = new SqlConnection(connectionstring))
{
SqlCommand cmd = new SqlCommand("GetEmployeeList");
cmd.Connection = con;
cmd.CommandType = CommandType.StoredProcedure;
con.Open();
cmd.Parameters.Add(new SqlParameter("@name", txtName.Text.Trim()));
if (ddlDepartment.SelectedIndex > 0)
cmd.Parameters.Add(new SqlParameter("@departmanet", ddlDepartment.SelectedValue));
if (ddlStart.SelectedIndex > 0 && ddlStart0.SelectedIndex > 0)
{
cmd.Parameters.Add(new SqlParameter("@expstart", ddlStart.SelectedValue));
cmd.Parameters.Add(new SqlParameter("@expend", ddlStart0.SelectedValue));
}
using (SqlDataAdapter sda = new SqlDataAdapter(cmd))
{
sda.Fill(dtEmployeeList);
}
}
lvEmployeeList.DataSource = dtEmployeeList;
lvEmployeeList.DataBind();
}
Fine now run your application and provide the details as in the figure and let's run your dotrace tool to measure the application performance now.
And here the performance overview:
Wow!! we got amazing performance just 416 milliseconds to fetch the records from SQL Server, with the filters though they are filtered from 1 lac records. Ok still we have a problem with this approach because today they are just 13000+ records. What if in the future more records are added to the same cirteria? We need to optimize more to get same performance. Ok we will see them in my next article in the series.

Dinesh BeniwalPosted Jan 4, 2013, 11:19 AM
Great Work.
Areeg SamirPosted Jan 2, 2013, 6:31 AM
Thank you for your effort it is very helpful
Srinivasu PemmaPosted Dec 30, 2012, 12:18 AM
Thank You! Mahesh
Mahesh ChandPosted Dec 29, 2012, 10:55 PM
Srinivasu, Nicely done and well detailed explained.