SharePoint Online/2016/2013 - How To Upload Large Files Using PowerShell Automation

Uploading large files to SharePoint On-Premise or Online is an obvious problem during data migration from any external systems like Lotus Notes.

Here is one of such errors which we might encounter while trying to upload a file of size greater than 250 MB-

1

In this article, I will explain a data upload strategy where we can split a large file into multiple chunks of a smaller size.

Solution Architecture Diagram

For better understanding, we can refer to the following solution architecture diagram-

2

Based on this diagram we can conclude the following facts,

  1. This solution can be hosted on multiple servers to launch parallel uploads
  2. This solution can consume data from Network File Shares
  3. Once data file is retrieved (say of size 300 MB), this solution will split the file (100 MB) automatically based on the pre-configured chunk size (which should not exceed the size limit of 250 MB)
  4. Each chunk then appended to the file uploaded in multiple iterations

In order to start with this demo, we would need a SharePoint Document Library in SharePoint Online (or On-Premise) Site as shown below,

3

Another prerequisite to demo is to have files of various sizes that we can use to upload to the document library.

I made use of following command line utility to generate files of various sizes. This utility takes destination folder path and size of the file in KBs as input.

Here is the usage example of the command line utility.

fsutil file createnew "C:\Prashant\Self Paced Training\Sample Files\2GB.txt" 2147483648

Similarly, I have generated other files too, as shown below.

4

Now, let’s dive down into the code to understand the actual implementation.

Step 1

Declare a variable to hold the document library name & folder path. For production use, I recommend having these values in an external configuration file.

Step 2

Reading files from the folder specified in path variable in Step 1.

Step 3

Loop through all the files and pass each file to the “UploadLargeFiles” function along with the document library name.

5

Step 4

Generate unique upload id & get file name of the file to be uploaded.

Step 5

Get the handle on document library object and load the root folder (or any target folder) within the document library.

Step 6

Calculate the block size to be uploaded and total file size (as shown in the architecture diagram).

Step 7

Read the bytes from the source file and set the read buffer based on the block size.

6

Step 8

Read the bytes based on the buffer limit that we set in the earlier steps.

7

Step 9

Check if this is the first chunk that is being uploaded, if yes then add a new file to SharePoint Document Library, get the file content based on the buffer size for the chunk and call “StartUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will add the file to the document library but with a small bunch of content only.

Step 10

Check if this is not the first chunk that is being uploaded, if yes then find the file in document library and get the handle on it

Step 11

If this is another chunk of data which is not the last chunk, this chunk will be appended to the same file by using “ContinueUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will append the content to the file identified by Upload Id that we have initialized in earlier steps.

Step 12

If this is the last chunk of data, this chunk will be appended to the same file by using “FinishUpload” function that is defined under “Microsoft.SharePoint.Client.File” class. This will append the content to the file identified by Upload Id that we have initialized in earlier steps and commits the changes to the file. Once this function completes successfully the changes will be made persistent to the file.

8

Step 13

Perform exception handling and call the “UploadLargeFileToLibrary”

9

I recommend reading the documentation on Microsoft.SharePoint.Client.File class and understand functions carefully before using it.

Once we execute this script we can see the following information,

  1. File Name to be uploaded
  2. Chunk size
  3. Total Time taken to upload the files

It is important to note that total time taken to upload the files may vary depending on the hosting environment.

File Size to be uploaded: 10 MB

10

File Size to be uploaded: 50 MB

11

File Size to be uploaded: 500 MB

12

File Size to be uploaded: 2 GB

13

Once the script is executed successfully we can see the respective files uploaded to the SharePoint Online Site as shown below,

14

Here is the complete code.

  1. function UploadLargeFileToLibrary() {  
  2.     $docLibraryName = "ProductsDocuments";  
  3.     $driveLocation = "C:\Prashant\Documents to Upload";  
  4.     $filesInRoot = Get - ChildItem - Path $driveLocation | ? {  
  5.         $_.psIsContainer - eq $False  
  6.     }  
  7.     Foreach($File in ($filesInRoot)) {  
  8.         $fileName = "C:\Prashant\Documents to Upload\$($File.Name)";  
  9.         $startDTM = (Get - Date)  
  10.         Write - Host $fileName  
  11.         Write - Host "Uploading file "  
  12.         $File.Name - ForegroundColor blue  
  13.         UploadLargeFiles $fileName $docLibraryName  
  14.         $endDTM = (Get - Date)  
  15.         Write - Host "Total Elapsed Time: $(($endDTM-$startDTM).totalseconds) seconds"  
  16.     }  
  17. }  
  18.   
  19. function UploadLargeFiles() {  
  20.     param($fileName, $docLibraryName)# $fileName = "C:\Prashant\Documents to Upload\1gb.txt";  
  21.     $UploadId = [GUID]::NewGuid()  
  22.     $UniqueFileName = [System.IO.Path]::GetFileName($fileName)  
  23.     $Docs = $clientContext.Web.Lists.GetByTitle($docLibraryName)  
  24.     $clientContext.Load($Docs)  
  25.     $clientContext.Load($Docs.RootFolder)  
  26.     $clientContext.ExecuteQuery()  
  27.     $fileChunkSizeInMB = 1  
  28.     $BlockSize = $fileChunkSizeInMB * 1024 * 1024  
  29.     $FileSize = (Get - Item $fileName).length  
  30.     $ServerRelativeUrlOfRootFolder = $Docs.RootFolder.ServerRelativeUrl[Microsoft.SharePoint.Client.File] $upload  
  31.     $BytesUploaded = $null  
  32.     $Fs = $null  
  33.     Try {  
  34.         $Fs = [System.IO.File]::Open($fileName, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read, [System.IO.FileShare]::ReadWrite)  
  35.         $br = New - Object System.IO.BinaryReader($Fs)  
  36.         $buffer = New - Object System.Byte[]($BlockSize)  
  37.         $lastBuffer = $null  
  38.         $fileoffset = 0  
  39.         $totalBytesRead = 0  
  40.         $bytesRead  
  41.         $first = $true  
  42.         $last = $false  
  43.         while (($bytesRead = $br.Read($buffer, 0, $buffer.Length)) - gt 0) {  
  44.             $totalBytesRead = $totalBytesRead + $bytesRead  
  45.             if ($totalBytesRead - eq $FileSize) {  
  46.                 $last = $true  
  47.                 $lastBuffer = New - Object System.Byte[]($bytesRead)[array]::Copy($buffer, 0, $lastBuffer, 0, $bytesRead)  
  48.             }  
  49.             If($first) {  
  50.                 $ContentStream = New - Object System.IO.MemoryStream  
  51.                 $fileInfo = New - Object Microsoft.SharePoint.Client.FileCreationInformation  
  52.                 $fileInfo.ContentStream = $ContentStream  
  53.                 $fileInfo.Url = $UniqueFileName  
  54.                 $fileInfo.Overwrite = $true  
  55.                 $Upload = $Docs.RootFolder.Files.Add($fileInfo)  
  56.                 $clientContext.Load($Upload)# $s = [System.IO.MemoryStream]::new($buffer)  
  57.                 $s = new - object System.IO.MemoryStream(, $buffer)  
  58.                 $BytesUploaded = $Upload.StartUpload($UploadId, $s)# $BytesUploaded = $Upload.StartUpload($UploadId, $buffer)  
  59.                 $clientContext.ExecuteQuery()  
  60.                 $fileoffset = $BytesUploaded.Value  
  61.                 $first = $false  
  62.             }  
  63.             else {  
  64.                 $Upload = $clientContext.Web.GetFileByServerRelativeUrl($Docs.RootFolder.ServerRelativeUrl + [System.IO.Path]::AltDirectorySeparatorChar + $UniqueFileName);  
  65.                 If($last) {#  
  66.                     $s = [System.IO.MemoryStream]::new(, $lastBuffer)  
  67.                     $s = new - object System.IO.MemoryStream(, $lastBuffer)  
  68.                     $Upload = $Upload.FinishUpload($UploadId, $fileoffset, $s)  
  69.                     $clientContext.ExecuteQuery()  
  70.                     Write - Host "File upload complete"#  
  71.                     return $Upload  
  72.                 }  
  73.                 else {#  
  74.                     $s = [System.IO.MemoryStream]::new(, $buffer)  
  75.                     $s = new - object System.IO.MemoryStream(, $buffer)  
  76.                     $BytesUploaded = $Upload.ContinueUpload($UploadId, $fileoffset, $s)  
  77.                     $clientContext.ExecuteQuery()  
  78.                     $fileoffset = $BytesUploaded.Value  
  79.                 }  
  80.             }  
  81.         }  
  82.     } catch {  
  83.         Write - Host $_.Exception.Message - ForegroundColor Red  
  84.     }  
  85.     Finally {  
  86.         if ($Fs - ne $null) {  
  87.             $Fs.Dispose()  
  88.         }  
  89.     }  
  90. }  
That is all for this demo. This article is equally applicable to both SharePoint Online & On-Premise Versions.

Hope you find it helpful.