Read Hidden characters from a text file

muharrem aker

I have a text file which I attached in this post.I read this file using StreamReader in c# and insert into SQL. When I try to insert the data,I get error.I was trying to find the problem for that i tried removing the text word by word from the text file and trying the code again.Finally I found the problematic part , removed it and successfully inserted into SQL . When I copy the text and paste into notepad ,there is nothing I could observe. . After spending some time I decided putting into word. I copied the text and paste into word 2007 and found the problem.The problematic text has hidden characters in it. The character is "--" . Two hyphens or dashes. According to my research , SQL database does not accept this Character sequence against SQL injection .After finding the issue, I tried to fix it. What I tried is , read the text file and if i see that sequence replace with new characters . I tried stream reader to read the text file but i can not find the hidden characters with stream reader . what is the best way to read hidden characters with c# .
I attached the file if you open in word you see the characters also in notepad++ . But not in notepad

4 Replies

Know the answer? Post it — somebody with the same question will find it here.

Sam Hobbs16y ago
It is useless to call them hidden characters; there is nothing we can do to help you if you say they are hidden. Trying to guess what is meant by "--" or two hyphens or dashes does notg help either unless you are absolutely sure that that is what the data is. Please be srue what the data is. First, are you sure it is two characters? Is there any possibility it is one character that looks like two? Please be sure. A better way to communicate what the data is is to tell us the hexadecimal or decimal value of the character or characters.
theLizard16y ago
As I said, they are unlikely to be hyphens as they are not characters that would normally be used as hidden identifiers, if you can see them in word 2007 they may be the Soft Hyphens used as word separators, when you know what the character code is, you can easily filter them out but until you know the character code there is not much more anyone can do.

Note, "--" by the looks is a double Soft Hyphen what you should be looking for is a single "-" but it may also depend on the language of the region the email came from.
muharrem aker16y ago
I dont know if they are hyphens or control characters. It is hidden in notepad however i could see in word 2007. The text is an outlook email signature and came to our system from external user. If it is control character , it should be a way to catch that sequence i guess . If i could find "--" "__" these and replace it with c# that would be wonderful.

thanks again
theLizard16y ago
It is unlikely that they are hyphens,they are more likely to be control character in which case they would be hidden to normal view in most text editors, what font is the text that you are having this problem with.

Looking at the character map, what you describe are most likely Soft Hyphens U+00AD but can also be Macrons or Hebrew Point Patah U+05B7 or any number of other similar characters so depending on the region you are in and the language of the region.

Once you know exactly what they are you can filter these out by looking for the control character code in your C# code.