I have CSV file of around 30M dala I want to serch lines from this fil

Hello Ramanath,

there is faster way for searching trough 30 MB of data, stored inside CSV document. One way is to load it to RAM and search trough.

I am using simple one dimensional Array to load whole NASA NEO Asteroid data base from downloaded CSV document, currently about 133 MB of data, and search trough with many different criteria. When I search for all data about an asteroid by its name, search is finished almost immediate.

I have declared List Lista, type of string that will hold all data from CSV document, structure Asteroid_BP that represents record of single Asteroid data from CSV document and Array type of Asteroid_BP that will contain all records from CSV document:

List <string> Lista;
int N;
struct Asteroid_BP
{
public string Datum;
public string ID_Asteroida;
public string Naziv;
public string Telo;
public string Opasan;
public string Nadzor;
public decimal Distanca;
public decimal Brzina;
public decimal Precnik;
public decimal Magnituda;
public string MOID;
public string Link;
};
Asteroid_BP [] Lista_Asteroida;

After that I have loaded data from CSV document into Array :

void Ucitaj_Listu_Asteroida()
{
Lista = new List<string>(File.ReadAllLines(Radni_Diretorijum +
Lista_Asteroida_Dokument,
Encoding.UTF8));
N = Lista.Count;
Lista_Asteroida = new Asteroid_BP[N];
decimal x = 0;
for(int i=0;i
{
string [] L = Lista[i].Split(',');
Lista_Asteroida[i].Datum = L[0];
Lista_Asteroida[i].ID_Asteroida = L[1];
Lista_Asteroida[i].Naziv = L[2];
Lista_Asteroida[i].Telo = L[3];
Lista_Asteroida[i].Opasan = L[4];
Lista_Asteroida[i].Nadzor = L[5];
x = decimal.Parse(L[6].Replace('.',','));
Lista_Asteroida[i].Distanca = x;
x = decimal.Parse(L[7].Replace('.',','));
Lista_Asteroida[i].Brzina = x;
if(L[8]!="null")
{
x = decimal.Parse(L[8].Replace('.',','));
Lista_Asteroida[i].Precnik = x;
}
else
Lista_Asteroida[i].Precnik = -1;
x = decimal.Parse(L[9].Replace('.',','));
Lista_Asteroida[i].Magnituda = x;
Lista_Asteroida[i].MOID = L[10];
Lista_Asteroida[i].Link = L[11];
}
Lista.Clear();
}

After this step You can search trough Array holding records of data from CSV document in any manner You need.

All the best,

Željko Peric

4 Replies

Know the answer? Post it — somebody with the same question will find it here.

Asma Khalid5y ago
Import your data into SQL Server than use SQL queries to get your require result set. I am using SQL Server with 4 million data of 3GB size and SQL server loading data speed is takes 5 to 10 minutes without any filtration and 1 to 2 minutes with filtration.

Mark answer as accepted if helpful

Thanks & Regards,
Asma Khalid

For More Visit: www.asmak9.com or www.bytezaar.com
+1
Rajanikant Hawaldar5y ago
use pandas library then you can apply a function, wanting to find a specific word in your record, on your data frame.
+2
Željko Perić5y ago
Hello Ramanath,
there is faster way for searching trough 30 MB of data, stored inside CSV document. One way is to load it to RAM and search trough.

I am using simple one dimensional Array to load whole NASA NEO Asteroid data base from downloaded CSV document, currently about 133 MB of data, and search trough with many different criteria. When I search for all data about an asteroid by its name, search is finished almost immediate.

I have declared List Lista, type of string that will hold all data from CSV document, structure Asteroid_BP that represents record of single Asteroid data from CSV document and Array type of Asteroid_BP that will contain all records from CSV document:
List <string> Lista;
int N;
struct Asteroid_BP
        {
            public string Datum;
            public string ID_Asteroida;
            public string Naziv;
            public string Telo;
            public string Opasan;
            public string Nadzor;

            public decimal Distanca;
            public decimal Brzina;
            public decimal Precnik;
            public decimal Magnituda;

            public string MOID;

            public string Link;

        };


        Asteroid_BP [] Lista_Asteroida;
After that I have loaded data from CSV document into Array :
void Ucitaj_Listu_Asteroida()
        {
            Lista = new List<string>(File.ReadAllLines(Radni_Diretorijum +
                                                       Lista_Asteroida_Dokument,
                                                       Encoding.UTF8));

            N = Lista.Count;

            Lista_Asteroida = new Asteroid_BP[N];

            decimal x = 0;
            for(int i=0;i
            {
                string [] L = Lista[i].Split(',');
                Lista_Asteroida[i].Datum        = L[0];
                Lista_Asteroida[i].ID_Asteroida = L[1];
                Lista_Asteroida[i].Naziv        = L[2];
                Lista_Asteroida[i].Telo         = L[3];
                Lista_Asteroida[i].Opasan       = L[4];
                Lista_Asteroida[i].Nadzor       = L[5];

                x = decimal.Parse(L[6].Replace('.',','));
                Lista_Asteroida[i].Distanca     = x;

                x = decimal.Parse(L[7].Replace('.',','));
                Lista_Asteroida[i].Brzina       = x;

                if(L[8]!="null")
                {
                    x = decimal.Parse(L[8].Replace('.',','));
                    Lista_Asteroida[i].Precnik  = x;
                }
                else
                    Lista_Asteroida[i].Precnik  = -1;

                x = decimal.Parse(L[9].Replace('.',','));
                Lista_Asteroida[i].Magnituda    = x;

                Lista_Asteroida[i].MOID         = L[10];

                Lista_Asteroida[i].Link         = L[11];

            }


            Lista.Clear();
        }
After this step You can search trough Array holding records of data from CSV document in any manner You need.
All the best,
Željko Peric
+1
Amit Gupta5y ago
Dictionary will help you to search faster https://stackoverflow.com/a/37837602. However, you can also use library too https://www.codeproject.com/Articles/5255318/fastCSV-A-Tiny-Fast-Standard-Compliant-CSV-Reader
+1