C# regex file parser

Thanks for all your help :)

I've manage to fix it using that simple, albeit crude, fix using the sleep. It works now, and its does what it is supose to thankfuly.
I would consider this matter closed =)

regards

Sorry for the delay in responding but I've only just noticed your last post.

The only other suggestion I have is to try and quicken up the processing of each file by compiling the regular expressions. So, in all your Regex methods, I'd try replacing RegexOptions.IgnoreCase with:

RegexOptions.IgnoreCase | RegexOptions.Compiled

This will increase start up time because the regular expressions need to be compiled 'on the fly' to a separate assembly but should improve execution time.

Excellent work, been using my time trying to understand it and i get it. I did want to make a few modifications and they all work, except for one.

When the program is running, it ofcourse has to accept incomming files, we talked about this earlier, and you suggesting simply increating the buffer, however that does not seem to work. Or I am doing it wrong, which is highly possible, but im pretty sure its just a matter of increasing the current
            watcher.InternalBufferSize = 16384;
to something along the lines of
            watcher.InternalBufferSize = 65536;
or something higher, however it does not have an effect. So i tried something else, namely instead of it reacting on a 'created' event, i had it react to a 'changed' event, but that seems to make it react everytime the file is written - which is not what i need, i need it to wait for the file to finish writing before it does anything.

I also changed it to istead of only checking .txt files it checks everyhting, which is easy enough, and instead of suplying the arguments in -folderin now its just
program.exe folderIn folderOut cat subcat replacementcat

currently i've fixed this problem with the writing by introducing sleep, but its a poor fix in my oppinion, using sleep is 'bad' practice as far as i can see, atleast in this situation, also it slows the program down rather alot.

is there another way to do this?  I've looked on google for the watcher.Changed instead, as stated above, I also look at notifer.LastWrite, but I dont think its what I am looking for.

Anyway here is what Its become :


--Code--
using System;
using System.IO;
using System.Text.RegularExpressions;
using System.Security.Permissions;
using System.Threading;

namespace TextParser
{
    class Program
    {
        static string folderIn;
        static string folderOut;
        static string format;
        static string IPTCExp1;
        static string IPTCExp2;
        static string IPTCExp3;
        static string NITFExp1;
        static string NITFExp2;
        static string NITFExp3;

        static void Main(string[] args)
        {
            if (args.Length != 6 && args.Length != 5)
            {
                Console.WriteLine("There must either be 6 arguments supplied, or 5 without the format, if the format is not specified, it will default to NITF.");
                Console.WriteLine("For instance:");
                Console.WriteLine("pArser.exe c:\\sourcefolder c:\\destinationfoler ");
                return;
            }

            folderIn = args[0];
            folderOut = args[1];
            //determine the number of arguments and if 4, set the fifth to the default format NITF
            format = (args.Length == 5) ? "NITF" : args[6];

            IPTCExp1 = @"\b" + args[2] + @"\b"; // exp to find cat
            IPTCExp2 = args[3]; // exp to find subCat
            IPTCExp3 = args[4]; // exp to replace cat with catReplace

            NITFExp1 = "" + args[2] + ""; // exp to find cat
            NITFExp2 = "" + args[3] + "*" + ""; // exp to find subCat
            NITFExp3 = "" + args[4] + ""; // exp to replace cat with catReplace

            DirectoryInfo dInfoIn = new DirectoryInfo(folderIn);
            // check if the directory doesn't exist, if so then create it
            if (!dInfoIn.Exists)
            {
                dInfoIn.Create();
            }
            DirectoryInfo dInfoOut = new DirectoryInfo(folderOut);
            // check if the directory doesn't exist, if so then create it
            if (!dInfoOut.Exists)
            {
                dInfoOut.Create();
            }
            ProcessExistingFiles();
            Run();
        }


        [PermissionSet(SecurityAction.Demand, Name = "FullTrust")]
        public static void Run()
        {
            FileSystemWatcher watcher = new FileSystemWatcher();
            watcher.InternalBufferSize = 16384;
            watcher.Path = folderIn;
            watcher.Filter = "*.*"; // say
            watcher.Crated += new FileSystemEventHandler(OnCreated);
            // tried using this, but still it does not wait for the writing to be finished with the file
            //watcher.Changed+= new FileSystemEventHandler(OnChanged);
            watcher.EnableRaisingEvents = true;
            Console.WriteLine("Press \'q\' to quit the program.");
            while (Console.Read() != 'q') ;
        }
       
        static void ProcessExistingFiles()
        {
            // get all existing .txt files in the input folder
            string[] paths = Directory.GetFiles(folderIn, "*.*");
            // process each one
            foreach (string path in paths)
            {
                ProcessFile(path);
            }
        }

        static void ProcessFile(string fullPath)
        {
            string name = Path.GetFileName(fullPath);
            if (format == "IPTC")
            {
                string[] lines = File.ReadAllLines(fullPath);
                if (lines.Length > 2)
                {
                    // look for subCat first
                    bool found = Regex.IsMatch(lines[2], IPTCExp2, RegexOptions.IgnoreCase);
                    if (found)
                    {
                        // now look for cat and replace with catReplace
                        lines[1] = Regex.Replace(lines[1], IPTCExp1, IPTCExp3, RegexOptions.IgnoreCase);
                    }
                }

                File.WriteAllLines(folderOut + "\\" + name, lines);
               
            }
            else
            {
                string text = File.ReadAllText(fullPath);
                // look for subCat first
                bool found = Regex.IsMatch(text, NITFExp2, RegexOptions.IgnoreCase);
                if (found)
                {
                    // now look for cat and replace with catReplace
                    text = Regex.Replace(text, NITFExp1, NITFExp3, RegexOptions.IgnoreCase);
                }

                File.WriteAllText(folderOut + "\\" + name, text);
            }
            Thread.Sleep(1000); // allow time for handle to be released after reading
            File.Delete(fullPath);
        }

        static void OnCreated(object source, FileSystemEventArgs e)
        {
            Console.WriteLine(e.FullPath + " created");
            /* useless fix to the problem with reading a file that is being written.
             this will slow the program down too much, find another way to do this.
             tried using the OnChanged instead, but it did not wait for the files
            to be written before firing the event. */
            Thread.Sleep(500);
            // useless fix end

            ProcessFile(e.FullPath);
        }
        /*
        //Attempted fix - does not work, why not?
        //This change, should enable the file to be finished with being written,
        //as the OnChanged is fired not when the file is crated but when it is finished with being written.
       
        static void OnChanged(object source, FileSystemEventArgs e)
        {
            Console.WriteLine(e.FullPath + " changed");
            ProcessFile(e.FullPath);
        }

        */
    }

}

--code end --

Any ideas or help is appreciated
Regards
The Aspiring C# developer

It's not a problem to have a default value of NITF if the user doesn't specify the -format parameter. However, the args array would only include 10 elements rather than 12 in this scenario.

To process the files already in the input folder, I'd introduce another method and copy the common code which it shares with the OnCreated method into a third method.

As I said in my previous post, if the files are arriving too quickly and some are being missed, I'd try doubling the buffer (8K) by default to 16K.

Here's the revised code incorporating these points (untested):

using System;
using System.IO;
using System.Text.RegularExpressions;
using System.Security.Permissions;
using System.Threading;

namespace TextParser
{
   class Program
   {
       static string folderIn;
       static string folderOut;
       static string format;
       static string IPTCExp1;
       static string IPTCExp2;
       static string IPTCExp3;
       static string NITFExp1;
       static string NITFExp2;
       static string NITFExp3;

       static void Main(string[] args)
       {         
           if(args.Length != 12 && args.Length != 10)
           {
               Console.WriteLine("There must be 12 arguments, these are -folderIn -folderOut -cat -subCat -catReplace -format (either IPTC or NITF)");
               Console.WriteLine("For instance:");
               Console.WriteLine("pArser.exe -folderIn c:\\sourcefolder -folderOut c:\\destinationfoler -cat I -subCat VEJR -catReplace IV -format IPTC");
               Console.WriteLine("However, the -format argument is optional and, if not present, defaults to NITF");
               return;
           }
                    
           folderIn = args[1];
           folderOut = args[3];
           format = (args.Length == 10) ? "NITF" : args[11];

           IPTCExp1 = @"\b" + args[5] + @"\b"; // exp to find cat
           IPTCExp2 = args[7]; // exp to find subCat
           IPTCExp3 = args[9]; // exp to replace cat with catReplace

           NITFExp1 = "" + args[5] + ""; // exp to find cat
           NITFExp2 = "" + args[7] + ""; // exp to find subCat
           NITFExp3 = "" + args[9] + ""; // exp to replace cat with catReplace

           ProcessExistingFiles();
           Run();
       }

      
       [PermissionSet(SecurityAction.Demand, Name="FullTrust")]
       public static void Run()
       {        
            FileSystemWatcher watcher = new FileSystemWatcher();
            watcher.InternalBufferSize = 16384;
            watcher.Path = folderIn;
            watcher.Filter = "*.txt"; // say
            watcher.Created += new FileSystemEventHandler(OnCreated);
            watcher.EnableRaisingEvents = true;
            Console.WriteLine("Press \'q\' to quit the program.");
            while(Console.Read()!='q');
       }

       static void ProcessExistingFiles()
       {
            // get all existing .txt files in the input folder
            string[] paths = Directory.GetFiles(folderIn, "*.txt");
            // process each one
            foreach (string path in paths)
            {
               ProcessFile(path);
            }               
       }

       static void ProcessFile(string fullPath)
       {
            string name = Path.GetFileName(fullPath);
            if (format == "IPTC")
            {
                 string[] lines = File.ReadAllLines(fullPath);
                 if (lines.Length > 2)
                 {                           
                    // look for subCat first
                    bool found = Regex.IsMatch(lines[2], IPTCExp2, RegexOptions.IgnoreCase);
                    if (found)
                    {                    
                      // now look for cat and replace with catReplace
                      lines[1] = Regex.Replace(lines[1], IPTCExp1, IPTCExp3, RegexOptions.IgnoreCase);
                    }
                 }
              
                 File.WriteAllLines(folderOut + "\\" + name, lines);
             }
             else
             {
                 string text = File.ReadAllText(fullPath);
                 // look for subCat first
                 bool found = Regex.IsMatch(text, NITFExp2, RegexOptions.IgnoreCase);
                 if (found)
                 {
                    // now look for cat and replace with catReplace
                    text = Regex.Replace(text, NITFExp1, NITFExp3, RegexOptions.IgnoreCase);
                 }
              
                 File.WriteAllText(folderOut + "\\" + name, text);
             }
             Thread.Sleep(500); // allow time for handle to be released after reading
             File.Delete(fullPath);
       }

       static void OnCreated(object source, FileSystemEventArgs e)
       {
            Console.WriteLine(e.FullPath + " created");
            ProcessFile(e.FullPath);
       }
   }

}
     

Wauw that worked really well, I am wondering about a few things now though, I tried correcting the program to allow for the format to the empty, and then it would default to NITF, but I cant seem to get it to work I tried in the following way.

using System;
using System.IO;
using System.Text.RegularExpressions;
using System.Security.Permissions;
using System.Threading;

namespace TextParser
{
   class Program
   {
       static string folderIn;
       static string folderOut;
       static string format;
       static string IPTCExp1;
       static string IPTCExp2;
       static string IPTCExp3;
       static string NITFExp1;
       static string NITFExp2;
       static string NITFExp3;

       static void Main(string[] args)
       {

           if(args.Length > 12)
           {
               Console.WriteLine("There must be 12 arguments, these are -folderIn -folderOut -cat -subCat -catReplace -format (either IPTC or NITF)");
               Console.WriteLine("For instance:");
               Console.WriteLine("pArser.exe -folderIn c:\\sourcefolder -folderOut c:\\destinationfoler -cat I -subCat VEJR -catReplace IV -format IPTC");
               return;
           }
         
          
           folderIn = args[1];
           folderOut = args[3];
           format = args[11];
         //Inserted code to set format to NITF as default, if format is empty or not given
           if (args[11].Length = String.Empty)
           {
               format = "NITF";
           }

           IPTCExp1 = @"\b" + args[5] + @"\b"; // exp to find cat
           IPTCExp2 = args[7]; // exp to find subCat
           IPTCExp3 = args[9]; // exp to replace cat with catReplace

           NITFExp1 = "" + args[5] + ""; // exp to find cat
           NITFExp2 = "" + args[7] + ""; // exp to find subCat
           NITFExp3 = "" + args[9] + ""; // exp to replace cat with catReplace

           Run();
       }

     
       [PermissionSet(SecurityAction.Demand, Name="FullTrust")]
       public static void Run()
       {       
            FileSystemWatcher watcher = new FileSystemWatcher();
            watcher.Path = folderIn;
            watcher.Filter = "*.txt"; // say
          
            watcher.Created += new FileSystemEventHandler(OnCreated);
            watcher.EnableRaisingEvents = true;
            Console.WriteLine("Press \'q\' to quit.");
            while(Console.Read()!='q');
       }

       static void OnCreated(object source, FileSystemEventArgs e)
       {
            Console.WriteLine(e.FullPath + " created");
            if (format == "IPTC")
            {
                 string[] lines = File.ReadAllLines(e.FullPath);
                 if (lines.Length > 2)
                 {                          
                    // We look for the subCategory
                    bool found = Regex.IsMatch(lines[2], IPTCExp2, RegexOptions.IgnoreCase);
                    if (found)
                    {                   
                      // Now we look for category and replace with catReplace
                      lines[1] = Regex.Replace(lines[1], IPTCExp1, IPTCExp3, RegexOptions.IgnoreCase);
                     
                    }
                 }
             
                 File.WriteAllLines(folderOut + "\\" + e.Name, lines);
             }
             else
             {
                 string text = File.ReadAllText(e.FullPath);
                 // We look for the subCategory first
                 bool found = Regex.IsMatch(text, NITFExp2, RegexOptions.IgnoreCase);
                 if (found)
                 {
                    // Now we look for the category and replace it with the replacement category
                    text = Regex.Replace(text, NITFExp1, NITFExp3, RegexOptions.IgnoreCase);
                 }
             
                 File.WriteAllText(folderOut + "\\" + e.Name, text);
             }
             Thread.Sleep(500); // Introduce timing to allow for the "windows handle" to be released after reading the file
             File.Delete(e.FullPath);
       }
   }

}
However, that only generates an error message, I also tried checking simply if it was set to be 0 or "", but that does not work either comparing the args[11] == 0 just gives an outofRangeexception in args[1], no clue about why.

Instead I tried making it so the order of the arguments given was irrelevant, but that did not work at all. And the resulting code was nothing short of incomparable to the way it is now. A giant mess basically.

I was wondering how to enable it to read files already in the folder? not to mention how to delay the reading of incomming files as well, I got numerous crashes because whatever is writing the file is not done when the program starts to read the file to make the comparison and so forth.

Any help is appreciated.

Regards

Excellent, works perfectly.
Right now, I am still working on understand it, but it is quite clear :)

Thanks for the help =)

OK, here's some code to do that.

As you'll see, in addition to the other difficulties, I've had to introduce a FileSystemWatcher object to watch for files being added to 'folderIn'. You may have a problem with this if the files are arriving too fast though you could try increasing the buffer size. Another problem is deleting a file immediately after it has been read as the handle may not be released straightaway - I've introduced a 0.5 second delay before deleting in an attempt to deal with that:

using System;
using System.IO;
using System.Text.RegularExpressions;
using System.Security.Permissions;
using System.Threading;

namespace TextParser
{
   class Program
   {
       static string folderIn;
       static string folderOut;
       static string format;
       static string IPTCExp1;
       static string IPTCExp2;
       static string IPTCExp3;
       static string NITFExp1;
       static string NITFExp2;
       static string NITFExp3;

       static void Main(string[] args)
       {

           if(args.Length != 12)
           {
               Console.WriteLine("There must be exactly 12 arguments!");
               return;
           }

           folderIn = args[1];
           folderOut = args[3];
           format = args[11];

           IPTCExp1 = @"\b" + args[5] + @"\b"; // exp to find cat
           IPTCExp2 = args[7]; // exp to find subCat
           IPTCExp3 = args[9]; // exp to replace cat with catReplace

           NITFExp1 = "" + args[5] + ""; // exp to find cat
           NITFExp2 = "" + args[7] + ""; // exp to find subCat
           NITFExp3 = "" + args[9] + ""; // exp to replace cat with catReplace

           Run();
       }

      
       [PermissionSet(SecurityAction.Demand, Name="FullTrust")]
       public static void Run()
       {        
            FileSystemWatcher watcher = new FileSystemWatcher();
            watcher.Path = folderIn;
            watcher.Filter = "*.txt"; // say
            watcher.Created += new FileSystemEventHandler(OnCreated);
            watcher.EnableRaisingEvents = true;
            Console.WriteLine("Press \'q\' to quit the program.");
            while(Console.Read()!='q');
       }

       static void OnCreated(object source, FileSystemEventArgs e)
       {
            Console.WriteLine(e.FullPath + " created");
            if (format == "IPTC")
            {
                 string[] lines = File.ReadAllLines(e.FullPath);
                 if (lines.Length > 2)
                 {                           
                    // look for subCat first
                    bool found = Regex.IsMatch(lines[2], IPTCExp2, RegexOptions.IgnoreCase);
                    if (found)
                    {                    
                      // now look for cat and replace with catReplace
                      lines[1] = Regex.Replace(lines[1], IPTCExp1, IPTCExp3, RegexOptions.IgnoreCase);
                    }
                 }
              
                 File.WriteAllLines(folderOut + "\\" + e.Name, lines);
             }
             else
             {
                 string text = File.ReadAllText(e.FullPath);
                 // look for subCat first
                 bool found = Regex.IsMatch(text, NITFExp2, RegexOptions.IgnoreCase);
                 if (found)
                 {
                    // now look for cat and replace with catReplace
                    text = Regex.Replace(text, NITFExp1, NITFExp3, RegexOptions.IgnoreCase);
                 }
              
                 File.WriteAllText(folderOut + "\\" + e.Name, text);
             }
             Thread.Sleep(500); // allow time for handle to be released after reading
             File.Delete(e.FullPath);
       }
   }

}
     

These arguments will always be present, and there will be no defaults.

These arguments must allways be given in order to sort the incomming files into the correct places, and one must always specify categories and the folder for which incoming files are stored in and where to move them two, and ofcourse the fileformat, so all the arguments will always be present, and they will always be given in that order, while one could switch them around, for the purpose of this program it will suffice to say that they will be in this order.

Thanks :)

In that case the problem is much more complicated as you are receiving 12 command line arguments, albeit in 6 pairs.

Before I try to do something with that, can you confirm whether:

1.  There will always be exactly 12 arguments or could some pairs be missing, in which case defaults would be used.

2.  The order of the arguments will always be the same or could  the pairs be presented in any order?

First, Thank you for your reply :) I'm just starting out with this C# and I can do some things but I am having a hard time getting my head around many things.

Your reply gave way to alot of things, and I did learn a few things, but I have been unable to get it to working.

Background: Its for my site, I'm doing a news-service, but I may have gone over my head, but I would still like to get this working.

I'll expand the previous explanation abit.

The program is for a 'sorting-machine' I have a high number of files comming into a specific folder, and was hoping to make a windows service or an exe file i could run, as the following.

program.exe -folderIn c:\folderIn -folderOut c:\folderOut -cat I -subCat Weater -catReplace IV -format IPTC


where format is either IPTC or NITF where the IPTC file has the following format,

The first line in IPTC format, consists of a header which I can safely ignore, and the second linie, contains the category "RB0725 4 U 0130 ORD/z31", where the U in that line is the actual category (it could be other letters as well, I for instance).
The third line contains the subCategory, e.g. "210908:NOTE/AFGHANISTAN/NATO" in this example the line "NOTE/AFGAHNISTAN/NATO" is the subcategory

NITF format consist of an XML-like format.
The category is given in the tag I, in this case I is the category, as before it can be other letters as well. For the subcategory the tag Weather, here the subcategory would be Weather.

In regards to the subcategory, it doesnt have to match 100%, the subcategory simple has to contain the the -subCat argument.

When the program is done, it is supose to write the file to another folder, using the same filename, with the replaced text-lines in the file, and then delete the original file.

So basically the program must take the following input:
- Folder to 'listen' to
- Folder to write modified files to
- Main category (-Cat argument)
- Sub category (-subCat argument)
- Category to replace main category (-catReplace argument)
- Format (-format argument) argument being either IPTC or NITF


Any help is greatly appreciated.
Regards

Here's some basic code to do that using regular expressions. I've assumed that you want the first two arguments to be whole words (i.e. not just part of words) and that case should be ignored when looking for them:

using System;
using System.IO;
using System.Text.RegularExpressions;

namespace TextParser
{
   class Program
   {
       static void Main(string[] args)
       {
           if (args.Length != 4)
           {
              Console.WriteLine("There must be exactly 4 arguments!");
              return;
           } 
           string text = File.ReadAllText(args[3]); // read file
           string exp1 = @"\b" + args[0] + @"\b"; // exp to find 1st arg
           string exp2 = @"\b" + args[1] + @"\b"; // exp to find 2nd arg
           // look for 2nd arg first
           bool found = Regex.IsMatch(text, exp2, RegexOptions.IgnoreCase);
           if (!found)
           {
              // no need for any replacements
              return;
           }
           // now look for 1st arg and replace with 3rd arg
           text = Regex.Replace(text, exp1, args[2], RegexOptions.IgnoreCase);
           File.WriteAllText(args[3], text); // write back to file
       }
   }

}