WCF: Streaming Large Data Files

Andrew Fenster

WCFstreaming in WCFStreaming Large Data FilesWCF streaming

3 Comments

Join the conversation! Your thoughts help the community grow.

virat raj joshi15y ago
i am using REST WCF, I want to pass around 50000 Records to the wcf but Bad Request Error Is Occured. I Have Some settings in the service side Web.config file.... <behaviors> <serviceBehaviors> <behavior >   <dataContractSerializer maxItemsInObjectGraph="2147483647" ignoreExtensionDataObject ="true" /> </behavior> </serviceBehaviors> </behaviors> ........................................................... <standardEndpoint transferMode="Streamed" helpEnabled="true" automaticFormatSelectionEnabled="true" maxBufferPoolSize="9147483647223234233" maxBufferSize="2147483647" maxReceivedMessageSize="2147483647"/> <standardEndpoint name="DirectoryEndpoint"/> is there any other settings in service side web.config file if yes then please let me know. thanks in advance.....Virat Raj joshi
- Andrew Fenster15y ago
  I'm sorry, I'm not an expert on REST services, but I suspect your problem lies in the message getting sent by the client, not in the binding. I noticed that your maximum buffer size and maximum message size are the same. That is not the source of your problem, but normally the buffer size would be smaller than the message size.
Abhishek Bhat15y ago
I have a requirement to encrypt data as well as to stream the response...which binding do you feel is the best for the above scenario..For encryption to work, the content has to be buffered and not streamed. Let me know your thoughts.

WCF can be used to transfer image files, .pdf files or other large documents. The most common way to do this is to use streaming.

Normally WCF processes messages in buffered mode. This means incoming messages are not processed until the entire message has been received and buffered in memory. Likewise, outgoing messages are not sent until the entire message has been buffered in memory.

Buffered transfer mode is also blocking. This means the client is blocked until the entire message has been received and processed and the client receives a response.

Normally buffered transfer is fine. When the messages contain large files, however, buffering creates serious performance problems. The solution is to stream large messages instead of buffering. Streaming allows the message recipient (which could be client or the service) to start processing the message before the entire message has been received.

To use streaming transfer, you need to configure the binding:

<system.serviceModel>
<services>
<service name="MyDocumentService">
<endpoint
binding="basicHttpBinding"
bindingConfiguration="DocumentServiceBinding"
...
</endpoint>
</service>
</services>
<bindings>
<basicHttpBinding>
<binding
name="DocumentServiceBinding"
messageEncoding="Mtom"
transferMode="Streamed"
maxBufferSize="65536"
maxReceivedMessageSize="5242880"
...
<binding/>
</basicHttpBinding>
</bindings>
</system.serviceModel>

By default transferMode is Buffered. As shown above, transferMode is Streamed, which means all requests and responses are streamed. Other options include StreamedRequest (= streamed requests with buffered responses) and StreamedResponse (= buffered requests and streamed responses).

Restrictions

There are a number of important restrictions when you use streaming.

The message body cannot have a digital signature. A digital signature cannot be checked until the entire message has been received. That would require buffered transfer. Likewise, the message cannot be encrypted. You can still use transport level security such as SSL, but the message itself cannot be encrypted.

WCF reliable messaging requires buffering in order to guarantee that all messages are processed in the correct order.

Therefore, if you are streaming, there can be no digital signature, no message encryption and no reliable messaging. This also rules out many binding types. The only bindings (in .Net 4.0) allowed are:

BasicHttpBinding
BasicHttpContextBinding
NetTcpBinding
NetNamedPipeBinding

NetTcpBinding and NetNamedPipeBinding include reliable delivery and session support, so the lack of reliable messaging is not as important.

A common technique is to put all the operations which require streaming into one service endpoint which uses streaming, and to put all the other operations in a different endpoint which does not use streaming.

Contracts

When you're streaming, the message body must contain the stream and nothing else. Any other data has to go in the message header. So if you want to use a web service to upload documents and you also want to provide information about the document such as the author, the document stream goes in the message body and all other information goes in the header.

It should look something like the following. Notice the document name and author go in the message header, while the document Stream itself is in the message body.

[MessageContract]
public class DocumentUpload
{
[MessageHeader]
public string author;
[MessageHeader]
public string documentName;
[MessageBodyMember]
public Stream data;
}
[ServiceContract]
interface IMyDocumentService
{
[OperationContract]
SaveDocument(DocumentRequest upload);
[OperationContract]
RetrieveDocument(DocumentRequest download);
}

MTOM

In the sample binding above, message encoding is set to MTOM. This is not the default message encoding.

MTOM creates a complex message header, and there is a good bit of processing required to do so. Once this is done, however, the data itself is transferred very efficiently. So MTOM makes sense only when the documents being transferred are large. On the MSDN website, MTOM is only recommended when messages include more than 1 KB of binary data. See Large Data and Streaming.

Maximum Message Size and Maximum Buffer Size

It's important to set a maximum message size. Without it, extremely large files can simply overwhelm the service. The default maximum message size is 64 Kb. In the sample binding above, maximum message size is set to 5242880 bytes (= 5 Mb). The larger the maximum message size, the more vulnerable your service becomes to denial of service attacks or simple server overload.

It's also important to set a maximum buffer size. This is the maximum amount of memory that the service is permitted to devote to a single call. Even when streaming is enabled, the service still buffers the message header. Without setting a maximum buffer size, your service may still be vulnerable to denial of service attacks which include extremely large message headers.