SharePoint On Prem - Large Data Storage

We have a large SP 2019 farm with over 30 million documents which is around ~20TB size.

Over the next few years, the data would grow exponentially and we would be around ~150 million documents and around ~100TB in size. This would become a super large farm and hard to maintain.

We need to search only the metadata of the document and no search is needed on the contents, so we are ok with moving the documents to external storage and just storing metadata in SP.

Any suggestions on external storage of documents and just storing the reference in metadata or other alternative approaches are highly appreciated

There are several options you can consider for storing documents externally and only storing metadata in SharePoint:

Cloud Storage: You can use a cloud storage service such as Azure Blob Storage, AWS S3, or Google Cloud Storage to store the actual documents. You can then store the file URLs or references in SharePoint metadata columns.
Network Attached Storage (NAS): You can set up a NAS device on your network to store the documents externally. SharePoint can then store references to the document locations on the NAS.
Content Delivery Network (CDN): You can use a CDN service to store and deliver the documents externally. SharePoint can store the CDN URLs or references in metadata.
External Database: Instead of storing documents in SharePoint, you can store them in an external database and store references to the documents in SharePoint metadata.
Hybrid Approach: You can use a combination of the above methods to optimize storage and retrieval of documents based on their size, frequency of access, etc.

It's important to also consider security, access controls, and performance when choosing an external storage solution. Make sure to thoroughly test any solution before implementing it in a production environment.

1 Reply

Know the answer? Post it — somebody with the same question will find it here.

Abhishek Yadav2y ago
There are several options you can consider for storing documents externally and only storing metadata in SharePoint:
1. Cloud Storage: You can use a cloud storage service such as Azure Blob Storage, AWS S3, or Google Cloud Storage to store the actual documents. You can then store the file URLs or references in SharePoint metadata columns.
2. Network Attached Storage (NAS): You can set up a NAS device on your network to store the documents externally. SharePoint can then store references to the document locations on the NAS.
3. Content Delivery Network (CDN): You can use a CDN service to store and deliver the documents externally. SharePoint can store the CDN URLs or references in metadata.
4. External Database: Instead of storing documents in SharePoint, you can store them in an external database and store references to the documents in SharePoint metadata.
5. Hybrid Approach: You can use a combination of the above methods to optimize storage and retrieval of documents based on their size, frequency of access, etc.
It's important to also consider security, access controls, and performance when choosing an external storage solution. Make sure to thoroughly test any solution before implementing it in a production environment.