Slow Array Parsing (using Lists)

Hi All,

I'm loading in some data from an Oracle DB and finding it's reaaaaaally slow. It's about 65,000 customers I'm loading in and it takes about 30 mins to run.

I've tried a few different array types and settled on List (I read that the strongly typed collection was one of the best), but it's bloody awful.

Any ideas what I'm doing wrong?

class Customer : IComparable

{

//create a constructor

public Customer()

{

}

public string Code;

public string Name;

public string GenericPriceListName;

public string GenericPriceListType;

public string PriceList;

public string DiscountList;

public string ExceptionList;

public string PromotionList;

// implement IComparable interface

public int CompareTo(object obj)

{

if (obj is Customer)

{

return this.Code.CompareTo((obj as Customer).Code); // compare user names

}

throw new ArgumentException("Object is not a Customer");

}

class Program

{

private static List GetCustomerList(OleDbConnection myOleDbConnection)

{

OleDbCommand myOleDbCommand;

OleDbDataReader myOleDbDataReader;

List listOfCustomers = new List();

myOleDbCommand = myOleDbConnection.CreateCommand();

StringBuilder sqlStatement = new StringBuilder();

sqlStatement.Append("Select * from Customers");

myOleDbCommand.CommandText = sqlStatement.ToString();

myOleDbDataReader = myOleDbCommand.ExecuteReader();

while (myOleDbDataReader.Read())

{

Customer currentCustomer = new Customer();

currentCustomer.Code = myOleDbDataReader.GetString(0);

currentCustomer.Name = myOleDbDataReader.GetString(1);

currentCustomer.GenericPriceListName = myOleDbDataReader.GetString(2);

currentCustomer.GenericPriceListType = myOleDbDataReader.GetString(3);

listOfCustomers.Add(currentCustomer);

Console.WriteLine(string.Format("Extracting account {0}", currentCustomer.Name));

}

myOleDbDataReader.Close();

myOleDbCommand.Dispose();

return listOfCustomers;

}

I don't know but I would guess that the type of collection would not make a big difference. Try constructing the List using the constructor that takes a capacity, that will eliminate or reduce the amount of memory allocation and such. I doubt it will make a difference but it is easy enough to do.

Are you sure the computer has ample resources? Is there enough main memory and is the processor not maxed out? The problem is more likely due to input speed rather than the processing required for the List. If the processor is not maxed out then the collection is not the problem.

Try removing the Console.WriteLine. If you must see progress, do a Console.WriteLine for each 100 records or 1000 records.

1 Reply
Know the answer? Post it — somebody with the same question will find it here.

Sign in to answer this question
It is the same account you read, post and publish with — and you will come straight back to this page.
Sign inCreate an account

Sam Hobbs16y agoPosted Nov 24, 2010, 5:45 PM I don't know but I would guess that the type of collection would not make a big difference. Try constructing the List using the constructor that takes a capacity, that will eliminate or reduce the amount of memory allocation and such. I doubt it will make a difference but it is easy enough to do. Are you sure the computer has ample resources? Is there enough main memory and is the processor not maxed out? The problem is more likely due to input speed rather than the processing required for the List. If the processor is not maxed out then the collection is not the problem. Try removing the Console.WriteLine. If you must see progress, do a Console.WriteLine for each 100 records or 1000 records.