• 1.9k
  • 15
  • 856

processing large file connecting to database

Jun 24 2020 1:49 PM


I have a large file with many millions of rows per day. I need to lookup a column in the input file and get related information from the Database and regenerate the same file with all same column values as the original file but one column transformed. There is no database load involved, just a lookup.

Could you please suggest the performance considerations such as splitting the file into smaller chunks and parallel processing, because there is a Database involved. What C# .Net programming features should I be exploring about (multi threading?)  

thank you

Brought to you by: JavaScript SDK for Bold BI dashboard and analytics embedding. Free trial.