Poor Man's Web Monitoring Tools

This article shares the idea on how to utilize free tools to download log files from server, upload IIS log files into the database, display the results and compare the baseline and production files.

Background

This project was initially started back in the year 2012. Then, one day, my computer hard drive crashed. I was lucky enough to be able to recover most of the files. After spending some time, I was able to get it to compile again. After that, I worked on it maybe once or twice a year. There is no doubt, lots of improvements can be made to it. Lately, I've made some changes to the project and decided to share it now, instead of letting it sit on the shelf for another couple of years. Hopefully, someone will find this project useful and continue to make enhancements to it.

Introduction

Like many other bloggers or website owners out there, most of us host our website on a shared server instead of dedicated server hosting due to the cost. The hosting company shall not be named here. Based on my own experiences, once in a while, some malicious folders and files/malware were being injected into the website. Every time I submitted a ticket about it, I would get a response like "your computer was compromised", "you're using outdated software", "you need to change your password", etc. All kind of this nonsense but the hosting company never took responsibility or did their due diligence. Some hosting companies even offered to clean up the vulnerability and monitoring EACH site on the hosting for a yearly subscription. Imagine if you have more than one website, the cost will quickly add up due to someone else's negligence.

Since I refused to pay some company every year to monitor all my websites, why not build my own? The title sys it all, "Poor Man's Web Monitoring Tools". If you're poor, then you need to work hard and put together all the available free tools yourself. It might look like a Frankenstein tool now, but I’m positive it will look better once we pour some more thought into it.

Shown below is the brief description of some of the components in the solution. In this article, I'll not go into every detail on how the solution was being implemented as I believe some more work is needed to optimize it. But I will share what each component will do and how to set it up.

  1. DownloadIISLog

    • Download IIS Log files thru FTP (WinSCP)
    • Insert the log files data into the SQL Database using Log Parser

  2. SimpleMonitor.Web

    • Web UI to display the IIS Log data using jqGrid
    • Option to mark certain Ip Addresses "Banned"

  3. SimpleMonitor.BlockIpAddress

    • Example of HTTP Module to block banned Ip address from accessing the Website

  4. SimpleMonitor.ScanFile

    • Download latest files
    • Compare Baseline vs latest files using WinMerge
    • Send email notification of the results
DownloadIISLog

One of the responsibilities of this console application is to download the log files from the FTP Server using WinSCP. Please note that I only tested this on two different shared hosting environments. Here, what I would suggest is that if you get a permission denied error while accessing the log folder, first, log into your hosting account, navigate to the file manager, look for the logs folder, and assign read-only permission to the folder. The second option is to submit a ticket.

Shown in listing 1 is the method to download the log files. The initial run will take a while depending on the number of the log files residing in the hosting log folder. The subsequent execution will be a lot faster because this method will download only files not previously downloaded before. To achieve this, the method will first get the list of log files metadata from the server using the WinSCP ListDirectory method and store it in the directoryInfo object. Next, it will get the most recent log file name stored in the database. After that, the module will try to query the directoryInfo object for the LastWriteTime by the log file name. Finally, the module will download all the log files where the LastWriteTime is after the LastWriteTime that was determined previously.

Listing 1

  1. static internal void WinScpGetLog() {  
  2.  using(Session session = new Session()) {  
  3.   // Connect  
  4.   session.Open(GetWinScpSession());  
  5.   
  6.   string remotePath = ApplicationSetting.FtpRemoteFolder;  
  7.   string localPath = ApplicationSetting.DownloadPath;  
  8.   
  9.   // Get list of files in the directory  
  10.   RemoteDirectoryInfo directoryInfo = session.ListDirectory(remotePath);  
  11.   
  12.   //latest file name in the table by application name  
  13.   var latestLogFileInDb = Path.GetFileName(unitOfWork.IISLogRepository.LatestFileName(ApplicationSetting.ApplicationName));  
  14.   
  15.   //get the date of the latest file from FTP  
  16.   var logFileDate = directoryInfo.Files  
  17.    .Where(w => w.Name.ToLower() == latestLogFileInDb ? .ToLower())  
  18.    .Select(s => s.LastWriteTime).FirstOrDefault();  
  19.   
  20.   // Select the files not in database table  
  21.   IEnumerable notInLogTable =  
  22.    directoryInfo.Files  
  23.    .Where(file => !file.IsDirectory && file.LastWriteTime > logFileDate).ToList();  
  24.   
  25.   //// Download the selected file  
  26.   foreach(RemoteFileInfo fileInfo in notInLogTable) {  
  27.    string localFilePath =  
  28.     RemotePath.TranslateRemotePathToLocal(  
  29.      fileInfo.FullName, remotePath, localPath);  
  30.   
  31.    string remoteFilePath = RemotePath.EscapeFileMask(fileInfo.FullName);  
  32.   
  33.    //download  
  34.    TransferOperationResult transferResult =  
  35.     session.GetFiles(remoteFilePath, localFilePath);  
  36.   }  
  37.  }  

Listing 2 shows the logic employs the Microsoft Log Parser tool to query the log files and insert them into the database. The query used by the Log Parser in the CallLogParser() method is very straightforward. It will connect to the SQL database using the provided username and password. Then read all the log files in the folder and then insert them into a table. The ApplicationNameparameter was added recently as an option to allow storing log files data for more than one website. If you have ten websites, you can duplicate this console application ten times and configure different ApplicationName values in the configuration file.

Listing 2

  1. static void CallLogParser() {  
  2.  ProcessStartInfo startInfo = new ProcessStartInfo(ApplicationSetting.LogParserPath);  
  3.  startInfo.WindowStyle = ProcessWindowStyle.Minimized;  
  4.   
  5.  Process.Start(startInfo);  
  6.   
  7.  startInfo.Arguments = string.Format("\"SELECT '{7}', *,1 INTO {0} FROM {1}{2}*.log\" -o:SQL -createTable:OFF -server:{3} -database:{4} -username:{5} -password:{6}",  
  8.   ApplicationSetting.LogParserSqlTable, ApplicationSetting.DownloadPath, ApplicationSetting.LogFilePrefix, ApplicationSetting.LogParserSqlserver,  
  9.   ApplicationSetting.LogParserSqlDatabase, ApplicationSetting.LogParserSqlUserName, ApplicationSetting.LogParserSqlPassword,  
  10.   ApplicationSetting.ApplicationName);  
  11.   
  12.  Process.Start(startInfo);  

Shown in figure 1 is the data imported into the database table through theCallLogParser()method.

Figure 1

Poor Man Web Monitoring Tools

DownloadIISLog - Configuration

Table 1 shows the descriptions of the settings in the app.config.

Table 1

KeyDescription
FtpHostThe address of the server. Example: poormantool.arr or 123.123.123.123
FtpRemoteFolderThe path to the logs files on the server. Example: /virtualFolder/logs/W3SVC999/
FtpUserNameFTP account
FtpPasswordFTP password
FilePrefixThe log file prefix, if any.
DownloadPathLocation to store the downloaded logs file. Example: c:\temp\app1\download\
LogPathLogPath The path to the log file for logging purposes like when the console run, stop, error, etc.
Example: c:\temp\app1\log\log.txt
LogParserPathThe path to the log parser. Example: C:\Program Files (x86)\Log Parser 2.2\LogParser.exe
LogParserSqlTableThe table to use in the SQL database to store the log files data. Right now, it can only be "iislog" unless you update the entity and code
ApplicationNameThe name of the application. Example: app1
LogParserSqlserverThe SQL server address
LogParserSqlDatabaseThe SQL server database
LogParserSqlUserNameThe SQL server username
LogParserSqlPasswordThe SQL server password
connectionStringsReplace the XXX with your SQL server information

SimpleMonitor.Web

The purpose of this web application is to display the log file data. Now that we have the data in the database, is up to you, what technology you want to use to interface with the data. This web application was developed by using MVC 5, EntityFramework v6.0, jqGrid, Bootstrap v4.0 and jQuery v3.3.

Figure 2 shows how the Interface looks from the web browser. All the columns are sortable and filterable. The blocked column indicates if the IP address is blocked from accessing the site. The goal of the blocked Hit column is to show the number of times a blocked IP address attempted to access the website.

Figure 2

Poor Man Web Monitoring Tools

To block an IP address, click on the green icon. A confirmation dialog will appear. Click Continue. Refer to figure 3. The application will insert the selected IP address into the dbo.BlockedIp table, then mark all the blocked IPs in the grid with a red ban circle icon.

Figure 3

Poor Man Web Monitoring Tools

To unblock an IP address, click on the red circle icon and continue button. Refer to figure 4.

Figure 4

Poor Man Web Monitoring Tools
 
SimpleMonitor.Web - Configuration

Make sure to update the connection string in the web.config to mirror your SQL server environment.

SimpleMonitor.BlockIpAddress

The purpose of this module is to demonstrate how to utilize the data in dbo.BlockedIp table. This module will check if the requester IP address exists in the table, if yes, redirect the request to an error page and then increase the blocked hit count. Listing 3 shows how to register the httpModules in the web application web.config file. To test it, I'll add my IP address into the table and re-run the web application. Again, this is just an example, is up to you how you want to implement the detection and prevention control.

Listing 3

  1. <system.web>  
  2.    <httpModules>  
  3.       <remove name="BlockIpHttpModules" />  
  4.      <add type="SimpleMonitor.BlockIpAddress.BlockIpHttpModule, SimpleMonitor.BlockIpAddress" name="BlockIpHttpModules" />  
  5.     </httpModules>  
  6.   </system.web>  
  7.   <system.webServer>  
  8.     <modules>  
  9.       <add type="SimpleMonitor.BlockIpAddress.BlockIpHttpModule, SimpleMonitor.BlockIpAddress" name="BlockIpHttpModules" preCondition="managedHandler" />  
  10.     </modules>  
  11.   </system.webServer> 
SimpleMonitor.ScanFile

The main purpose of this console application is to compare the folders and files between the baseline and the production. Initially, the application will download and save the files to the destination folder using WinSCP. Then it will run the WinMerge command to compare the files with the baseline. Listing 4 shows the WinMerge command line to compare the files.

Listing 4

  1. static void CompareFiles() {  
  2.  var tempFileName = $ "{Guid.NewGuid()}.html";  
  3.   
  4.  ProcessStartInfo startInfo = new ProcessStartInfo(ApplicationSetting.WinMergePath);  
  5.  startInfo.WindowStyle = ProcessWindowStyle.Minimized;  
  6.   
  7.  startInfo.Arguments = $ " {ApplicationSetting.BaselineFilesPath} {ApplicationSetting.LatestFilesPath} -minimize " +  
  8.   "-noninteractive -noprefs -cfg Settings/DirViewExpandSubdirs=1 -cfg Settings/DiffContextV2=2 " +  
  9.   "-cfg ReportFiles/ReportType=2 -cfg ReportFiles/IncludeFileCmpReport=1 -cfg Settings/ShowIdentical=0 " +  
  10.   $ " -r -u -or {ApplicationSetting.FileCompareOutputPath}{tempFileName}";  
  11.   
  12.  var process = Process.Start(startInfo);  
  13.  process.WaitForExit();  
  14.   
  15.  Email.SendEmail($ "{ ApplicationSetting.FileCompareOutputPath}{ tempFileName}");  

Table 2 shows the WinMerge parameters in listing 4 and its descriptions.

Table 2

KeyDescription
-minimizestarts WinMerge as a minimized window.
-noninteractiveExit WinMerge after compare / report generation
-noprefsDo not read / write setting information from registry (use default value)
-cfg Settings/DirViewExpandSubdirs=1Expand folder tree after comparison 0: do not expand the folder tree
-cfg Settings/DiffContextV2=20: shows only the different
1: shows the different and one line from above and below
2: shows the different and two lines from above and below
-cfg ReportFiles/ReportType=2Generate HTML-format report
-cfg ReportFiles/IncludeFileCmpReport=1Include file comparison report in folder comparison report, 0: do not include
-cfg Settings/ShowIdentical=0Do not show Identical files in the result
-rcompares all files in all subfolders
-uprevents WinMerge from adding either path (left or right) to the Most Recently Used (MRU) list
-orPath to the output file

At the end of the comparison, the CompareFiles() method will send a summary to the specified email address using Gmail. Figure 5 shows how the comparison summary looks.

Figure 5

Poor Man Web Monitoring Tools

Figure 6 shows how the comparison details look when clicking on the Filename to drill down from the computer where the comparison reports are saved.

Figure 6

Poor Man Web Monitoring Tools
 
SimpleMonitor.ScanFile – Configuration

Table 3 shows the descriptions of the settings in the app.config.

Table 3

Key Description
FtpHostThe address of the server. Example: poormantool.arr or 123.123.123.123
FtpRemoteFolderThe path to the logs files on the server. Example: /virtualFolder/wwwroot/
FtpUserNameFTP account
FtpPasswordFTP password
BaselineFilesPathThe path to the application production files
LatestFilesPathThe path to the latest files downloaded by using the WinSCP from the FTP Server
FileCompareOutputPathThe path where the comparison results will be stored
WinMergePathThe path to the WinMerge application
SmtpHostThe email SMTP host
SmtpToThe recipient
SmtpPortThe SMTP Port number
SmtpUserNameThe email account username / email
SmtpPasswordThe email account password
SmtpSubjectThe email subject

Require software

Download and install the following software.

  • WinMerge v2.15.2 (https://sourceforge.net/projects/winmerge/files/alpha/2.15.2/)
  • Log parser v2.2 (https://www.microsoft.com/en-us/download/details.aspx?id=24659)
How to setup

Execute the database_objects.sql script on an existing / new database instance. The script will create the following objects:

  • [BlockedIp] table
  • [iislog] table
  • [vwIISLog] view
  • [InsertUpdateBlockedIp] Stored Procedure
Conclusion

First of all, I would like to thanks the new WinMerge owner for continuing to maintain the WinMerge software. Please keep in mind that the solution provided here is not a preventive control. Whatever you see in the logs and comparison results are after the fact. But you can create your own preventive control such as IP blocker after reviewing and analyzing the information.

Here are other topics of interest that you can extract out of this project.

  • How to Use jqGrid with server-side paging, filtering, sorting
  • How to use Log Parser to import IISLog into SQL database table
  • How to use WinSCP to download files from FTP server
  • How to use WinMerge to compare files
  • How to create and utilize HTTP Modules
  • How to send email using Gmail

I hope someone will find this project useful. If you find any bugs or disagree with the contents or want to help improve this article, please drop me a line and I'll work with you to correct it. I would suggest visiting the demo site and exploring it in order to grasp the full concept because I might miss some important information in this article. Please contact me if you want to help improve this article.

History

  • 08/12/2018 - Initial version
Download

Resources

Feedback/issue