I have CSV file of around 30M dala I want to serch lines from this fil

Question

Hi team I have CSV file of around 30M dala I want to serch lines from this file which having specific keyword is there any way to do this in faster manner as I am doing simple grep and it is taking hours

Rajanikant Hawaldar · Answer

use pandas library then you can apply a function, wanting to find a specific word in your record, on your data frame.

Asma Khalid · Answer

Import your data into SQL Server than use SQL queries to get your require result set. I am using SQL Server with 4 million data of 3GB size and SQL server loading data speed is takes 5 to 10 minutes without any filtration and 1 to 2 minutes with filtration. Mark answer as accepted if helpful Thanks & Regards, Asma Khalid For More Visit: www.asmak9.com or www.bytezaar.com

Željko Perić · Answer

Hello Ramanath, there is faster way for searching trough 30 MB of data, stored inside CSV document. One way is to load it to RAM and search trough. I am using simple one dimensional Array to load whole NASA NEO Asteroid data base from downloaded CSV document, currently about 133 MB of data, and search trough with many different criteria. When I search for all data about an asteroid by its name, search is finished almost immediate. I have declared List Lista, type of string that will hold all data from CSV document, structure Asteroid_BP that represents record of single Asteroid data from CSV document and Array type of Asteroid_BP that will contain all records from CSV document: List string > Lista; int N; struct Asteroid_BP { public string Datum; public string ID_Asteroida; public string Naziv; public string Telo; public string Opasan; public string Nadzor; public decimal Distanca; public decimal Brzina; public decimal Precnik; public decimal Magnituda; public string MOID; public string Link; }; Asteroid_BP [] Lista_Asteroida; After that I have loaded data from CSV document into Array : void Ucitaj_Listu_Asteroida() { Lista = new List string >(File.ReadAllLines(Radni_Diretorijum + Lista_Asteroida_Dokument, Encoding.UTF8)); N = Lista.Count; Lista_Asteroida = new Asteroid_BP[N]; decimal x = 0; for ( int i=0;i { string [] L = Lista[i].Split( ',' ); Lista_Asteroida[i].Datum = L[0]; Lista_Asteroida[i].ID_Asteroida = L[1]; Lista_Asteroida[i].Naziv = L[2]; Lista_Asteroida[i].Telo = L[3]; Lista_Asteroida[i].Opasan = L[4]; Lista_Asteroida[i].Nadzor = L[5]; x = decimal .Parse(L[6].Replace( '.' , ',' )); Lista_Asteroida[i].Distanca = x; x = decimal .Parse(L[7].Replace( '.' , ',' )); Lista_Asteroida[i].Brzina = x; if (L[8]!= "null" ) { x = decimal .Parse(L[8].Replace( '.' , ',' )); Lista_Asteroida[i].Precnik = x; } else Lista_Asteroida[i].Precnik = -1; x = decimal .Parse(L[9].Replace( '.' , ',' )); Lista_Asteroida[i].Magnituda = x; Lista_Asteroida[i].MOID = L[10]; Lista_Asteroida[i].Link = L[11]; } Lista.Clear(); } After this step You can search trough Array holding records of data from CSV document in any manner You need. All the best, Željko Peric

Amit Gupta · Answer

Dictionary will help you to search faster https://stackoverflow.com/a/37837602. However, you can also use library too https://www.codeproject.com/Articles/5255318/fastCSV-A-Tiny-Fast-Standard-Compliant-CSV-Reader

I have CSV file of around 30M dala I want to serch lines from this fil

Answers (4)