Web Service Design for Chunked Transfer of Large Quantities of Data

do
{
n = rnd.Next(1000);
name = string.Format("_part{0:000}.dat", n);
}
while (dir.GetFiles(name).Length > 0);

// At this point we have obtained unique name which is identified by number n

FileStream fs = new FileStream(dir.FullName + "\\" + name, FileMode.CreateNew);
fs.Close();

// Now we have created a file of size zero, which is server-side representative of the file handle

// Finally, return handle to the caller
return n.ToString("000");

}

This method basically creates a file under the FileUpload directory within the temporary files directory. The file name contains a part which is randomly generated, and that part is actually returned as new file handle. It is very important that handle is stripped off from all meaningful information other than trivially available, such as timestamp and similar. Handle, as well as session IDs, should be such that client cannot intentionally generate valid value in order to try to overtake other client's session and use it to do damage. In our demonstration handle is a three-digit random number. In more serious applications string of length 20 characters or so would probably be more appropriate, and more secure.

Next method exposed by the Web service is named UploadChunk. As its name implies, this method will upload part of the file by effectively appending data to partial file on the local file system. Method receives three parameters: file handle previously obtained from the BeginUploadFile method, array of bytes which is the content of part to append to file and position at which block of data starts in the complete file. Since we are appending data to partially completed file, this position would have to match current length of the file. Here is the source code:

[WebMethod]
[SoapDocumentMethod(OneWay=true)]
public void UploadChunk(string fileHandle, byte[] data, long startAt)
{

FileInfo fi = new FileInfo(Path.GetTempPath() + "FileUpload\\_part" + fileHandle + ".dat");

// Perform validation
if (!fi.Exists || fi.Length != startAt)
{
// Do whatever needs to be done when validation fails
}
else
{

// When validation has passed,
using (FileStream fs = new FileStream(fi.FullName, FileMode.Append))
fs.Write(data, 0, data.Length);

}
}

Method first processes the file handle by simply constructing the path to the file. Next step is to validate input data - in our case just to verify that file exists and that startAt matches current file length, in which case buffer content would be appended at the end of the file, as expected by design. When verification has passed, method performs the operation by appending content of the buffer at the end of the file.

Note that this method is declared as one-way, which means that client will not wait for the returning SOAP message, but it will consider the request completed as soon as it gets low-level TCP/IP confirmation that its request has been successfully posted to the server. This measure ensures that client will get onto sending the next chunk as soon as possible. But on the downside, this feature cuts off any feedback from the server through this method - there is no way for the client to find out whether storing data went well or not. Even if server threw the exception, that would not be sent back to the client.

To have the issue resolved Web service can expose a method which indicates current status. Client may invoke this method once in a while and, if status is not correct, it could jump back through data and repeat last couple of upload operations. Calling this status method rarely enough would ensure that performance will not diminish due to waiting for server to respond. This little complication may be afforded in practice because setting the UploadChunk method to be one-way would typically double the upload speed because half of the round trip is saved with every chunk. Here is the GetStatus method, which can be used to read current upload status:

[WebMethod]
public long GetStatus(string fileHandle)
{
FileInfo fi = new FileInfo(Path.GetTempPath() + "FileUpload\\_part" + fileHandle + ".dat");
long pos = (fi.Exists ? fi.Length : 0);
return pos;
}

This implementation simply returns current length of the partial uploaded file, which indicates total amount of data successfully uploaded so far.

Last method exposed by the Web service is named EndUploadFile and it must be invoked every time when a file is uploaded. This method receives file handle, so that server knows to which file to turn its attention, and a flag indicating whether file upload has been complete or client wishes to cancel further upload.

[WebMethod]
public void EndUploadFile(string fileHandle, bool quitUpload)
{

FileInfo fi = new FileInfo(Path.GetTempPath() + "FileUpload\\_part" + fileHandle + ".dat");

// Validate handle
if (!fi.Exists)
throw new System.ArgumentException();

if (quitUpload)
fi.Delete();
else
{
FileInfo targetFile = new FileInfo(Path.GetTempPath() + "FileUpload\\file" + fileHandle + ".dat");
if (targetFile.Exists)
targetFile.Delete();
fi.MoveTo(targetFile.FullName);
}
}

Again first step in the method is to determine file location from file handle and to verify that file exists. When verification passes, method will either delete temporary file (if quitUpload flag is set to true), or rename the file into permanent form (overwriting any existing file with the same name). These last steps mimic the cleanup logic which would be implemented in a real world system when data upload ends.

These methods complete design of the upload server. Following section will present the client design.

Upload Client Design

Client can be designed in quite a simple way, for example like this:

FileTransferServer srv = new FileTransferServer();
int maxChunkSize = 0;
string fileHandle = srv.BeginUploadFile(out maxChunkSize);

byte[] buffer = new byte[maxChunkSize];

using (FileStream fs = new FileStream("<file path here>", FileMode.Open))
{
int bytesRead = 0;
do
{
long pos = fs.Position;
bytesRead = fs.Read(buffer, 0, buffer.Length);
if (bytesRead < buffer.Length)
Array.Resize<byte>(ref buffer, bytesRead);
srv.UploadChunk(fileHandle, buffer, pos);
Console.WriteLine("Uploaded {0} of {1} bytes (handle={2})", fs.Position, fs.Length, fileHandle);
}
while (bytesRead > 0);
}
srv.EndUploadFile(fileHandle, false);

This implementation first instantiates the proxy class and then makes a request to the server's BeginUploadFile method, as to initiate the file transfer. Method's return value is remembered in the fileHandle string variable, and that is the handle which will be provided in all subsequent calls until this particular file upload completes.

After this introduction, series of calls to the UploadChunk method are made until all data contained in the file are sent to the server. Once done, a single call to EndUploadFile is made to indicate that upload is complete. Beyond this line, current value of fileHandle variable is no more valid, because server has already forgotten it.

Conclusion

In this article we have presented server and client code for chunked file upload. Techniques used to code these two entities can be employed in many other solutions. Advantages of chunking technique are that it can be applied in virtually any communication protocol (HTTP, XML Web services, sockets, etc.) and that it is very simple to implement. If one system is capable to send data in one segment, then it is certainly capable to send it sliced into pieces. Hence only a small change in design is required to introduce chunking into an existing system. On the negative side, chunking does not resolve the issue of responsiveness completely. It is so because client and server have opportunity to exchange control messages only between the chunks. If sending of one block of data gets stuck, then all communication gets stuck.

This problem can be overcome by putting the complete communication on a separate thread. As long as control messages sent between chunks only affect present session (e.g. sending of one file), outer world can assume whatever it wants about that session - e.g. that it has ended instantaneously when outer entity has instructed the client on a separate thread to cancel the sending. The fact that effective cut off will occur only after current chunk is sent, which might not be instantaneous at all, does not affect the rest of the system. This particular design is discussed in the next article titled "How to Improve Responsiveness of Objects that do not Guarantee Responsiveness" (http://www.c-sharpcorner.com/uploadfile/b81385/8914/).