Automation of Archiving Large Libraries

Introduction

Hi guys, let's try to get a better run process of my developed CSOM + PnP PowerShell scripts for seamless execution, and then move it to automation for a weekly scheduled process to handle all kinds of large libraries.

The main focus today on this Blog is to make the user avoid entering the Login Credentials during the run time of the PS scripts for a better high-level automation! So let's jump into the topic! This is a slightly better approach to my previous blog, so please follow all the pre-requisites mentioned there!

My auto-archival process includes 4 steps.

Document Library Scan of all the Files to be Archived

The library scan is performed with all Nested Folders and Files. It affects whichever has the Archive Flag manually set to 'True' OR scanned if they are 1 year old/n number of days old as per your Archival Strategy[LibScan.ps1].

#Getting NestedFolder and Files
for Archival Process on a CSV File
#Load SharePoint CSOM Assemblies
write - host - f Yellow "Scanning Large Library with Nested Folder and Files Started"
Add - Type - Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll"
Add - Type - Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"
#Config Parameters
$SiteURL = "https://sampleharenet.sharepoint.com/sites/classictest"
$ListName = "PnPCopytoLib"
$CSVPath = "D:\LibraryDocumentsInventory.csv"
#Config Login Details with Password Protection
$User = "[email protected]"
$PWord = ConvertTo - SecureString - String "**Contra#0987**" - AsPlainText - Force
$Credential = New - Object - TypeName System.Management.Automation.PSCredential - ArgumentList $User, $PWord
Try {
#Setup the context
$Ctx = New - Object Microsoft.SharePoint.Client.ClientContext($SiteURL)
$Ctx.Credentials = New - Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($Credential.UserName, $Credential.Password)
#Get the Document Library
$List = $Ctx.Web.Lists.GetByTitle($ListName)
#Define CAML Query to Get All Files
$Query = New - Object Microsoft.SharePoint.Client.CamlQuery
$Query.ViewXml = "@<View Scope='RecursiveAll'> <
Query >
<
Where >
<
And >
<
Eq > < FieldRef Name = 'FSObjType' / > < Value Type = 'Integer' > 0 < /Value></Eq >
<
Or >
<
Eq > < FieldRef Name = 'ArchivalFlag' / > < Value Type = 'Choice' > Yes < /Value></Eq >
<
Lt > < FieldRef Name = 'Created' / > < Value Type = 'DateTime'
IncludeTimeValue = 'True' > " + (get-date).adddays(-180).ToString("
yyyy - MM - ddTHH: mm: ssZ ") + " < /Value></Lt >
<
/Or> <
/And> <
/Where> <
/Query> <
/View>"
#powershell sharepoint online list all documents
$ListItems = $List.GetItems($Query)
$Ctx.Load($ListItems)
$Ctx.ExecuteQuery()
$DataCollection = @()
#Iterate through each document in the library
ForEach($ListItem in $ListItems) {
#Collect data
$Data = New - Object PSObject - Property([Ordered] @ {
FileName = $ListItem.FieldValues["FileLeafRef"]
RelativeURL = $ListItem.FieldValues["FileRef"]
CreatedBy = $ListItem.FieldValues["Created_x0020_By"]
CreatedOn = $ListItem.FieldValues["Created"]
ModifiedBy = $ListItem.FieldValues["Modified_x0020_By"]
ModifiedOn = $ListItem.FieldValues["Modified"]
FileSize = $ListItem.FieldValues["File_x0020_Size"]
})
$DataCollection += $Data
}
$DataCollection
#Export Documents data to CSV
$DataCollection | Export - Csv - Path $CSVPath - Force - NoTypeInformation
Write - host - f Green "Documents Data Exported to CSV!"
}
Catch {
write - host - f Red "Error:"
$_.Exception.Message
}

Output

We will get all the List of Files to be Archived as per the above-highlighted conditions using a Where Clause at your Local Path, D:\LibraryDocumentsInventory.csv

It should contain the following columns:

FileName
RelativeURL
CreatedBy
CreatedOn
ModifiedBy
ModifiedOn
FileSize

It copies the Files marked for Archival whose details are captured on the CSV report.

It copies all the Scanned Files with their Relative Folder Paths from the Source Library to the Archive Target Library[CopyFiles.ps1].

#Load SharePoint CSOM Assemblies
write - host - f Yellow "Nested Folder and Files Copying Started"
Add - Type - Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll"
Add - Type - Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"
#Config Login Details with Password Protection
$User = "[email protected]"
$PWord = ConvertTo - SecureString - String "Contra#0987" - AsPlainText - Force
$Credential = New - Object - TypeName System.Management.Automation.PSCredential - ArgumentList $User, $PWord
#Function to Copy a File
Function Copy - SPOFile([String] $SourceSiteURL, [String] $SourceFileURL, [String] $TargetFileURL) {
Try {
#Setup the context
#$Ctx = New - Object Microsoft.SharePoint.Client.ClientContext($SourceSiteURL)
#$Ctx.Credentials = New - Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($Credentials.Username, $Credentials.Password)
#Setup the context
$Ctx = New - Object Microsoft.SharePoint.Client.ClientContext($SiteURL)
$Ctx.Credentials = New - Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($Credential.UserName, $Credential.Password)
#Copy the File
$MoveCopyOpt = New - Object Microsoft.SharePoint.Client.MoveCopyOptions
$Overwrite = $True[Microsoft.SharePoint.Client.MoveCopyUtil]::CopyFile($Ctx, $SourceFileURL, $TargetFileURL, $Overwrite, $MoveCopyOpt)
$Ctx.ExecuteQuery()
Write - host - f Green $TargetFileURL " - File Copied Successfully!"
}
Catch {
write - host - f Red "Error Copying the File!"
$_.Exception.Message
}
}
$RootSiteURL = "https://crosssharenet.sharepoint.com"
$SourceSiteURL = "https://crosssharenet.sharepoint.com/sites/classictest"
$TargetSiteURL = "https://crosssharenet.sharepoint.com/sites/testsitearchival"
$SourceSitePath = "/sites/classictest/"
$TargetSitePath = "/sites/testsitearchival/"
$SourceDocLibURL = "/PnPCopytoLib"
$TargetDocLibURL = "/FlowArchiveLib2"
#$TargetDocLibURL = "/FlowArchiveLib2/April_3rdWeek"
#$Credentials = Get - Credential
Connect - PnPOnline - Url $TargetSiteURL - Credentials $Credential
$CSVPath = "D:\LibraryDocumentsInventory.csv"
Import - Csv $CSVPath | ForEach - Object {
$SourceFileURL = $RootSiteURL + $_.RelativeURL
$temp = ($_.RelativeURL).Replace($SourceDocLibURL, $TargetDocLibURL)
$temp = ($temp).Replace($SourceSitePath, $TargetSitePath)
$TargetFileURL = $RootSiteURL + $temp
$temp = ($temp).Replace($TargetSitePath, "")
$temp = ($temp).Replace("/" + $_.FileName, "")
if ($TargetDocLibURL - ne "/" + $temp) {
Resolve - PnPFolder - SiteRelativePath $temp
}
#Call the
function to Copy the File
Copy - SPOFile $SourceSiteURL $SourceFileURL $TargetFileURL
}

Precautions

Use Only Site Collection Admin/Global Admin Login details for logging-in while the script running is on progress.

Try to hard code with password-protected security string oriented Token Management on the above scripts for no End User manual inputting involvement.

Output

You will find all the listed files from that CSV report which have been selected for Archival created with Meta Data properties preserved on the Target Archive Library.

Remove all the Archived Files from Source Library after the above copying process is finished:

#Config Login Details with Password Protection
$User = "[email protected]"
$PWord = ConvertTo - SecureString - String "Contra#0987" - AsPlainText - Force
$Credential = New - Object - TypeName System.Management.Automation.PSCredential - ArgumentList $User, $PWord
$SourceSiteURL = "https://crosssharenet.sharepoint.com/sites/classictest"
Connect - PnPOnline - Url $SourceSiteURL - Credentials $Credential
$CSVPath = "D:\LibraryDocumentsInventory.csv"
Import - Csv $CSVPath | ForEach - Object {
Remove - PnPFile - ServerRelativeUrl $_.RelativeURL - force
}

Unify all the above 3 Scripts to Automatically run them on a Scheduled PowerAutomaton Flow/ Azure Web Job/ Azure Functions

For Flow Approach >> Simply go to flow.microsoft.com >> Build a Scheduled Weekly/Monthly running Azure Automated Job[Created in Azure with all the above PS codes tested on Run Books] >> Deploy it once to successfully run as per the Business needs.

For the Azure Web Job Approach, follow here.

For Azure, Function Approach, follow here.

You can apply the above seamless Automated Process with Large Lists too that needs to be Archived.

Note

Since we have a buffer size limit of 100 MB for a Complete Flow-based Copying of Files we are using PnP + CSOM PowerShell to run evergreen unlimited calls no pricing/no size limitation/metadata preserving, etc.

Happy Modern SharePoint Large Library Automated Archiving for your business needs and inside your respective organizations!

Cheers!