Design Advice

Thanks Sam and Mahesh for your input. Not 100% sure on the direction I will be taking at this point but will investigate further to ensure the right choices are made now.

cheers Matt.

Mahesh: yes, if the only purpose of converting to a database is to save the data, then it is not necessary. A database would be too much overhead or whatever. We need more clarification of the requirements. I am guessing that the requirements are sufficient to justify a database, but that is a guess.
 
We don't know how often the data will be processed. We don't know if data from more than one log file will be processed together. We don't know what relationships would or could exist among the data. A database can make data access more efficient when portions of each of many files need to be accessed. I don't know the data well enough to be sure, but the customers might want to do ad-hoc analysis and reporting. They can load the CSV into a database or spreadsheet themselves, but they would probably appreciate having that done for them.
 
I am not saying that either way is a mistake, and I am saying that both ways have advantages.

It really depends on what you need to do. Loading into a database is something that I am sure many experienced developers would consider very useful. Certainly if you were going to just process one file, it is not necessary to load into a database. My suggestion to load into a database is based on assumptions. If you need to process multiple CSV files multiple times and in various ways, a database can make everything more efficient for you and the computer. It really depends on what you need to do. The conversion to a database can be done by a tool without writing any code.
 
You can start out without a database, but my guess is that your requirements will expannd to the point where you will understand the advantage of a database. I have seen that type of thing happen way too much, and management is afraid to invest in fixing the problem. I agree that that is not likely to happen to you; you will likely be able to improve your design later, but I have seen many cases where that was not the case.

Matt/Sam,
Even though it is easy to load data from a database, I would not recommend saving data in a database. This is like you are doing twice work and now you will need to manage a database. Data is already in flat files, that are much faster accessing than from a relational database.

I suggest write a parser and develop a multi-threaded application to parse multiple log files simultaneously. You may want to search this site for Text parsing and threading. Here is an example:
Multi Threaded File Reader and Generator

You need to build a collection of objects. For instance, you can create a class and in this class each public property will be the column name in a row of text file. So when you done loading a log file, you should have a collection of objects. Once you have a collection, you can easily bind it to data-bound controls.

Post here if you have more questions.

Good luck!

Have a look at SQL Server Online Resources. I think you can use either SQL Server Express Edition or SQL Server Compact Edition. I am not sure what the advantages are of each, except I think that code developed for the SQL Server Express Edition is compatible with the full edition of SQL Server, so the customer probably has the option of using the full edition of SQL Server when they have it. The CE is smaller and compatible with Windows Mobile and such, but the code you write will be slightly different from the regular SQL client code.
 
There are other database systems, such as MySQL. There is another database system that other developers around here recommend, but I forget what it is.
 
If you use CSV without a traditional database system, then look at LINQ; it is something that .Net provides that will do most of the things that a database system will do, except in code. You need to learn about objects as a data source and things like that.

 
Something else worth looking at is business logic. It is a design methodology in which software is split into three layers; a Data Access Layer (DAL), a Business Logic Layer (BLL) and a User Interface. I don't know a lot about it but it is something I intend to learn about. The advantage for you is that you can create a DAL separate from the BLL and then you can create an alternative DAL with little or no modification to the BLL or the UI.

Thanks Sam I'll take a look at your article.

I have no experience with SQL server databases so not sure about this approach!
Can you embedded a SQL database into an application? Like if this application was to ship with our hardware it would have to be a complete analysis package. So simple is best... not only for me creating the application but for the end user(s) as well.

I think the best approach is to load the data into a SQL Server database or some other database. You can do that with little or no code.

Look at my article Using CSV and Other Delimited Files in C#, in case that helps. That can at least show you an easy way to read the CSV data.