Memory exception when handling very long Lists

Thank you guys. I've changed the concept, so no need in long lists.

I just want to add to what Vulpes has mentioned. Basically the memory allocation in .NET is done in continuous blocks. If there is memory available as chunks scattered across the blocks then it will not be allocated and you will be getting the out of memory exception.

Without code it is hard to follow exactly what you are doing. I think many of the values are more than 85,000 byte threshold and so your objects endup on LOH and so the GC was not collecting the objects and as you continue processing objects you soon end up out of address space. FYI LOH is the normal source of OutOfMemoryException.

Or you might creating a string objects over and over in your code and so it is creating memory issue.

Or you might be performing many operations while adding data to list and processing it over and over.

If the reason is not known and could not resolve this I would advise you to use sequences and forget about lists. Rather than List or List, try for IEnumerable or IEnumerable.

Do you know the approximate size in bytes of the objects you're adding to the list since, with 16 million entries, it's feasible that you may actually be running out of memory, depending on what else you may be doing?

If the objects are not too large, then one thing that may help is to set the List's initial capacity (in the constructor) to whatever number of items you think will eventually be reached.

If you do that then sufficient memory will be allocated on the 'large object heap' to accommodate this number of items in the List's internal array and no more allocations will be needed.

If you don't do it, then the List's capacity will be doubled from time to time as more space is needed by creating a new larger internal array and copying the old items into it. This means that at around 16 million items, a new array of 32 million items will need to be created and, even after a GC to get rid of the old one, it may not be possible to find sufficient memory on the LOH to accommodate the new array because (for performance reasons) the LOH is never compacted.