how to Get Millions of files from folder and bulk insert in database ?

Ankur Patel

I have some folders having files, its count like 40Lac, 70Lac, 1Cr, 1.5cr... and filetype like .png, .xlsx, .txt, .msg, .ico, .jpg, .bmp etc...

Now I want to insert this filename into my database table with its size... but when I implement it takes too much time to scan the folder and then after it throws a memory exception.

can anyone please help me out here, how can I implement this scenario in a better manner?
how can insert faster in the table?

I am using C#.net with the PostgreSQL database...

3 Replies

Know the answer? Post it — somebody with the same question will find it here.

Rijwan Ansari4y ago
Accepted answer
Hi,

You can use Directory.EnumerateFiles with batch.
You can process 500 files at a time then repeat.

Sample:
public class WorkerImproved
{
    //omitted for brevity

    public async Task DoWork()
    {
        const int batchSize = 1000;
        var files = Directory.EnumerateFiles(Path);
        var count = 0;
        var filesToProcess = new List(batchSize);

        foreach (var file in files)
        {
            count++;
            filesToProcess.Add(MapToDto(file));
            if (count == batchSize)
            {
                await _processingService.Process(filesToProcess);
                count = 0;
                filesToProcess.Clear();
            }

        }
        if (filesToProcess.Any())
        {
            await _processingService.Process(filesToProcess);
        }
    }
}
More details: https://www.c-sharpcorner.com/article/batch-processing-with-directory-enumeratefiles/
+4
Ankur Patel4y ago
Thanks Rijwan,

But, Can you please provide me proper code, it will help me more.

Thanks
+1
Srinivasan Ramamoorthi4y ago
You can implement using massive parallel processing.
+2