Migrate documents and folders from old SharePoint servers to New SharePoint using .Net CSOM

Introduction

As we all know, SharePoint has been evolving since 2001 from its initial release to SharePoint online and now the SharePoint Subscription edition. Every organization that has been using SharePoint for years also has/had to upgrade its version. You can upgrade or migrate the content from the old version to the newer version of SharePoint in various ways.

Problem

I came across a problem of migrating folders and files from the SharePoint 2010 version to the SharePoint Subscription edition (successor of SharePoint 2019).

My requirement was simple: migrate around 0.5 million files and folders from one specific site on SP 2010 to a specific site on SPSE, retaining author and timestamps as is.

Solution

Of course, there are plenty of tools available in the market which can migrate from any old version to any new version of SharePoint. Still, it comes at the cost of thousands of dollars, which I think no one wants to spend, especially when you are a core developer.

So the first free tool that came into my mind was SharePoint Migration Tool (SPMT), but when I explored it in detail, I found out that it supports migration from on-premise versions to only SharePoint Online.

The second thought that came to mind was about site backup restore, taking site backup using a PowerShell command and restoring it to a new server. But this will not work because of the database version mismatch.

The third solution I thought about was – the database upgrade approach, taking a backup of the whole database from SQL Server 2008 R2 and restoring it on SQL Server 2019. But again, this won't work as Microsoft recommends upgrading your databases from 2010 to 2013 to 2016 and then 2019 – that's a very hectic process and needs many server versions to be installed on different machines.

Final solution

So, the last option left was to build my tool to migrate the folders and files from one server to another.

Prerequisites to make this solution works,

Have the latest version of Visual Studio installed on a machine from which you can access both old and new Sharepoint sites. I am using VS 2019.
Ensure the account you are using to run the tool has at least contributed access to both the source and destination site.
Ensure you have installed "SharePoint Online Client Components SDK" on your machine.
Make sure all the users who have access to the old Sharepoint site are already granted access to the new SharePoint site – this is a must-do step before migration. Without this, Authors and editors won't get copied as is.

Approach of migration

If you have huge content at the source site, then plan to migrate it to multiple destination sites and have separate content databases for each site.
We will migrate the content site-wise and then library wise
1. Migrate all folders first
2. Then migrate all the files
Identify a service account at the new SharePoint server, which can be used to replace the authors/editors who are not part of the organization anymore. This is required. Otherwise, the author and editor will appear as blank as those users are no more available in the active directory.

Build the tool

We are going to create a simple console application. You can make it fancier by creating forms applications and building a nice UI. It's up to you.

Finally, here is the C# CSOM code; First, get all the folders migrated, and then migrate all the files.

Function to migrate the folders

static void MigrateAllFolders()
{
	try
	{
		// Source and destination site and library URLs
		string sourceSiteUrl = "http://intranet/site1";
		string destinationSiteUrl = "https://spse/site1";
		string sourceLibraryName = "Unit Sale";
		string destinationLibraryName = "Unit Sale";
		// Source and destination SharePoint contexts
		ClientContext sourceContext = new ClientContext(sourceSiteUrl);
		ClientContext destinationContext = new ClientContext(destinationSiteUrl);
		// Source and destination SharePoint libraries
		List sourceLibrary = sourceContext.Web.Lists.GetByTitle(sourceLibraryName);
		List destinationLibrary = destinationContext.Web.Lists.GetByTitle(destinationLibraryName);
		// Retrieve all items from the source library
		CamlQuery query = new CamlQuery();
		// FSObjType with value 1 will give only folders
		// Scope as RecursiveAll will give all folders in the library
		query.ViewXml = "<View Scope='RecursiveAll'>"
					  + "  <Query>"
					  + "    <Where>"
					  + "      <Eq><FieldRef Name='FSObjType' /><Value Type='Integer'>1</Value></Eq>"
					  + "    </Where>"
					  + "  </Query>"
					  + "  <ViewFields><FieldRef Name='Title' /><FieldRef Name='Created' /><FieldRef Name='Modified' /><FieldRef Name='Author' /><FieldRef Name='Editor' /></ViewFields>"
					  + "</View>";
		ListItemCollection sourceItems = sourceLibrary.GetItems(query);
		sourceContext.Load(sourceItems);
		sourceContext.ExecuteQuery();
		var destRootWeb = destinationContext.Site.RootWeb;
		// Loop through all items in the source library and copy them to the destination library
		foreach (ListItem sourceItem in sourceItems)
		{
			Console.WriteLine(sourceItem["FileLeafRef"].ToString());
			#region Logic to validate Author - check it exists in destination site or not
			var authourUser = destRootWeb.EnsureUser(((FieldLookupValue)sourceItem["Author"]).LookupValue);
			try
			{
				destinationContext.Load(authourUser);
				destinationContext.ExecuteQuery();
			}
			catch
			{
				authourUser = destRootWeb.EnsureUser("DomainName\\FirstName.LastName");
				destinationContext.Load(authourUser);
				destinationContext.ExecuteQuery();
			}
			var editorUser = destRootWeb.EnsureUser(((FieldLookupValue)sourceItem["Editor"]).LookupValue);
			try
			{
				destinationContext.Load(editorUser);
				destinationContext.ExecuteQuery();
			}
			catch
			{
				editorUser = destRootWeb.EnsureUser("DomainName\\FirstName.LastName");
				destinationContext.Load(editorUser);
				destinationContext.ExecuteQuery();
			}
			#endregion
			//To create the folder
			ListItemCreationInformation itemCreateInfo = new ListItemCreationInformation();
			itemCreateInfo.UnderlyingObjectType = FileSystemObjectType.Folder;
			itemCreateInfo.LeafName = sourceItem["FileLeafRef"].ToString();
			ListItem destinationItem = destinationLibrary.AddItem(itemCreateInfo);
			destinationItem["Title"] = sourceItem["FileLeafRef"].ToString();
			// Code to retain Timestamp and Authors as is
			destinationItem["Created"] = sourceItem["Created"];
			destinationItem["Modified"] = sourceItem["Modified"];
			destinationItem["Author"] = authourUser.Id.ToString() + ";#" + authourUser.LoginName.ToString();
			destinationItem["Editor"] = editorUser.Id.ToString() + ";#" + editorUser.LoginName.ToString();
			destinationItem.Update();
			destinationContext.ExecuteQuery();
		}
		Console.WriteLine("All folders have been copied successfully.");
	}
	catch (Exception ex)
	{
		Console.WriteLine(ex.Message);
	}
}

Function to migrate the files

static void MigrateAllFiles()
{
	try
	{
		// Source and destination site and library URLs
		string sourceSiteUrl = "http://intranet/site1";
		string destinationSiteUrl = "https://spse/site1";
		string sourceLibraryName = "Unit Sale";
		string destinationLibraryName = "Unit Sale";
		// Source and destination SharePoint contexts
		ClientContext sourceContext = new ClientContext(sourceSiteUrl);
		ClientContext destinationContext = new ClientContext(destinationSiteUrl);
		// Source and destination SharePoint libraries
		List sourceLibrary = sourceContext.Web.Lists.GetByTitle(sourceLibraryName);
		List destinationLibrary = destinationContext.Web.Lists.GetByTitle(destinationLibraryName);
		// FSObjType with value 0 will give only folders
		// Scope as RecursiveAll will give all files in the library
		// Retrieve all items from the source library
		CamlQuery query = new CamlQuery();
		query.ViewXml = "<View Scope='RecursiveAll'>"
					  + "  <Query>"
					  + "    <Where><And>"
					  + "      <Eq><FieldRef Name='FSObjType' /><Value Type='Integer'>0</Value></Eq>"
					  + "      <Geq><FieldRef Name = 'ID' /><Value Type = 'Number'>1</Value></Geq>"
					  + "    </And></Where>"
					  + "  </Query>"
					  + "  <ViewFields><FieldRef Name='Title' /><FieldRef Name='Created' /><FieldRef Name='Modified' /><FieldRef Name='Author' /><FieldRef Name='Editor' /></ViewFields>"
					  + "</View>";
		ListItemCollection sourceItems = sourceLibrary.GetItems(query);
		sourceContext.Load(sourceItems);
		sourceContext.ExecuteQuery();
		// This is required to avoid time out error, this is required to copy large size files
		sourceContext.RequestTimeout = -1;
		destinationContext.RequestTimeout = -1;
		// Loop through all items in the source library and copy them to the destination library
		foreach (ListItem sourceItem in sourceItems)
		{
			//This ill help us to track last successfully migrated file in case or error
			Console.WriteLine(sourceItem["ID"]);
			#region Get Valid User
			var destRootWeb = destinationContext.Site.RootWeb;
			var authourUser = destRootWeb.EnsureUser(((FieldLookupValue)sourceItem["Author"]).LookupValue);
			try
			{
				destinationContext.Load(authourUser);
				destinationContext.ExecuteQuery();
			}
			catch
			{
				authourUser = destRootWeb.EnsureUser("DomainName\\FirstName.LastName");
				destinationContext.Load(authourUser);
				destinationContext.ExecuteQuery();
			}
			var editorUser = destRootWeb.EnsureUser(((FieldLookupValue)sourceItem["Editor"]).LookupValue);
			try
			{
				destinationContext.Load(editorUser);
				destinationContext.ExecuteQuery();
			}
			catch
			{
				editorUser = destRootWeb.EnsureUser("DomainName\\FirstName.LastName");
				destinationContext.Load(editorUser);
				destinationContext.ExecuteQuery();
			}
			#endregion
			string sourceFileRelativePath = sourceItem["FileRef"].ToString();
			// Create a new file in the destination library with the same name as the source file
			FileInformation fileInfo = File.OpenBinaryDirect(sourceContext, sourceFileRelativePath);
			FileCreationInformation fileCreateInfo = new FileCreationInformation();
			using (var sourceStream = fileInfo.Stream)
			{
				var copyStream = new System.IO.MemoryStream();
				sourceStream.CopyTo(copyStream);
				copyStream.Position = 0;
				fileCreateInfo.ContentStream = copyStream;
			}
			sourceFileRelativePath = sourceFileRelativePath.Replace("/sourcesite/sourcelibrary/", "");
			fileCreateInfo.Url = destinationSiteUrl + destinationLibraryName + "/" + sourceFileRelativePath; 
			fileCreateInfo.Overwrite = true;
			Microsoft.SharePoint.Client.File destinationFile = destinationLibrary.RootFolder.Files.Add(fileCreateInfo);
			ListItem destinationItem = destinationFile.ListItemAllFields;
			// Set the created and modified fields of the destination item to match the source item
			destinationItem["Created"] = sourceItem["Created"];
			destinationItem["Modified"] = sourceItem["Modified"];
			destinationItem["Author"] = authourUser.Id.ToString() + ";#" + authourUser.LoginName.ToString();
			destinationItem["Editor"] = editorUser.Id.ToString() + ";#" + editorUser.LoginName.ToString();
			// Save the changes to the destination item
			destinationItem.Update();
			destinationContext.ExecuteQuery();
		}
		Console.WriteLine("All files and folders have been copied successfully.");
	}
	catch (Exception ex)
	{
		Console.WriteLine(ex.Message);
	}
}

Error handling

If in case the tool fails for some reason, we know at which item it has failed. Observe the error, fix it at the code level, change the query with the last item ID, which was migrated successfully, and run again.

Summary

Thanks for reading! It takes hardly a few hours to build your tool to migrate the content, and in today's world of ChatGPT, it become easier to find the exact code chunk. So don't fall into the trap of paid tools and save the cost to your organization. I hope this will help you guys.