Task Parallelism

Anurag Malani

11y1.6kC#

Hi All,
I have billions of records and i have to do some long calculation on those records. If I am using a sequential for loop then it takes around 7hr to complete that task which is not affordable. To overcome with this, I'm using Parallel.For but the catch is concurrency is occurred in jobs.

Example -
1. Run Parallel.For on 10 records, it creates 10 threads for each record.
2. While executions, First Thread is done with all the calculation and clear the data.
3. Here Second or Reaming Threads getting error in the calculation says - data in not available

How can we resolve this concurrency probleam?

The ultimate goal of my code is to create a new instance for that record and run separately in these block and should not affect the others.

4 Replies

Know the answer? Post it — somebody with the same question will find it here.

Benjamin Scharbach9y ago
Set the MaxParall in the Parallel.ForEach call to control the number of threads or workers used to complete your job. I discuss the Parallel.For on YouTube, if interested.

Ben
https://www.youtube.com/channel/UCih2aBHyGR5cJHApqkwAXBA
Sujeet Singh9y ago
Hi Anurag,
I think I have got you question, so solution for the same may be...
1. The logic of calculation and deletion should be seperate.
2. You can use arrays of Tasks and Use Wait all method to complete all tasks logic and then execute the logic of deletion of records.

Or
You can also use QueueUserWorkItem.

If possible to elaborate more about your logic of processing.

Thanks and Good Luck!!!
Andres Gomez9y ago
Hi ...
1. Is more easy help to you if you give us a little piece of your code.
2. I recommend to you , special atention to CPU in large Threads execution (A bad thread execution raises processor level to 100%)
3. You must ensure that all resources used between threads have an expected behavior, i reccomend use Lock instruction to guarantee a little wait time in the CPU and have exclusive acces to some objects an variables during Parallel foreach execution.
4. If you will use Lock, use an adicional object to block , not the object with you work with data modifications and storage.
sahana radder9y ago
Hi Anurag,
I think you have to use the "lock" in the for loop to avoid the data consequences. 2d record is looking for the value which is already changed in the 1st thread. Try to add the Lock() concept in the loop.
Regards - Sahana