SIGN UP MEMBER LOGIN:    
ARTICLE

WCF: Streaming Large Data Files

Posted by Andrew Fenster Articles | WCF with C# January 23, 2011
Web services that transfer large data files can use streaming to improve efficiency and throughput.
Reader Level:

WCF can be used to transfer image files, .pdf files or other large documents.  The most common way to do this is to use streaming. 

Normally WCF processes messages in buffered mode.  This means incoming messages are not processed until the entire message has been received and buffered in memory.  Likewise, outgoing messages are not sent until the entire message has been buffered in memory.  

Buffered transfer mode is also blocking.  This means the client is blocked until the entire message has been received and processed and the client receives a response.

Normally buffered transfer is fine.  When the messages contain large files, however, buffering creates serious performance problems.  The solution is to stream large messages instead of buffering.  Streaming allows the message recipient (which could be client or the service) to start processing the message before the entire message has been received.  

To use streaming transfer, you need to configure the binding:

    <system.serviceModel>
        <services>
            <service name="MyDocumentService">
                <endpoint 
                     binding="basicHttpBinding" 
                     bindingConfiguration="DocumentServiceBinding" 
                    ...   
                </endpoint>
            </service>
        </services>

        <bindings>
            <basicHttpBinding>
                 <binding 
                    name="DocumentServiceBinding" 
                    messageEncoding="Mtom" 
                    transferMode="Streamed" 
                    maxBufferSize="65536" 
                    maxReceivedMessageSize="5242880" 
                    ...  
                    <binding/>
            </basicHttpBinding>
        </bindings>

    </system.serviceModel>

By default transferMode is Buffered.  As shown above, transferMode is Streamed, which means all requests and responses are streamed.  Other options include StreamedRequest (= streamed requests with buffered responses) and StreamedResponse (= buffered requests and streamed responses).

Restrictions

There are a number of important restrictions when you use streaming.

The message body cannot have a digital signature.  A digital signature cannot be checked until the entire message has been received.  That would require buffered transfer.  Likewise, the message cannot be encrypted.  You can still use transport level security such as SSL, but the message itself cannot be encrypted.

WCF reliable messaging requires buffering in order to guarantee that all messages are processed in the correct order.

Therefore, if you are streaming, there can be no digital signature, no message encryption and no reliable messaging.  This also rules out many binding types.  The only bindings (in .Net 4.0) allowed are:
  • BasicHttpBinding
  • BasicHttpContextBinding
  • NetTcpBinding
  • NetNamedPipeBinding
NetTcpBinding and NetNamedPipeBinding include reliable delivery and session support, so the lack of reliable messaging is not as important.

A common technique is to put all the operations which require streaming into one service endpoint which uses streaming, and to put all the other operations in a different endpoint which does not use streaming.

Contracts

When you're streaming, the message body must contain the stream and nothing else.  Any other data has to go in the message header.  So if you want to use a web service to upload documents and you also want to provide information about the document such as the author, the document stream goes in the message body and all other information goes in the header.

It should look something like the following.  Notice the document name and author go in the message header, while the document Stream itself is in the message body.

[MessageContract]
public class DocumentUpload
{
    [MessageHeader]
     public string author;

    [MessageHeader]
     public string documentName;

    [MessageBodyMember]
     public Stream data;
}

[ServiceContract]
interface IMyDocumentService
{
     [OperationContract]
     SaveDocument(DocumentRequest upload);

    [OperationContract]
     RetrieveDocument(DocumentRequest download);
}

MTOM

In the sample binding above, message encoding is set to MTOM.  This is not the default message encoding.  

MTOM creates a complex message header, and there is a good bit of processing required to do so.  Once this is done, however, the data itself is transferred very efficiently.  So MTOM makes sense only when the documents being transferred are large.  On the MSDN website, MTOM is only recommended when messages include more than 1 KB of binary data.  See http://msdn.microsoft.com/en-us/library/ms733742.aspx.

Maximum Message Size and Maximum Buffer Size

It's important to set a maximum message size.  Without it, extremely large files can simply overwhelm the service.  The default maximum message size is 64 Kb.  In the sample binding above, maximum message size is set to 5242880 bytes (= 5 Mb).  The larger the maximum message size, the more vulnerable your service becomes to denial of service attacks or simple server overload.

It's also important to set a maximum buffer size.  This is the maximum amount of memory that the service is permitted to devote to a single call.  Even when streaming is enabled, the service still buffers the message header.  Without setting a maximum buffer size, your service may still be vulnerable to denial of service attacks which include extremely large message headers.


erver'>
Login to add your contents and source code to this article
share this article :
post comment
 

I'm sorry, I'm not an expert on REST services, but I suspect your problem lies in the message getting sent by the client, not in the binding. I noticed that your maximum buffer size and maximum message size are the same. That is not the source of your problem, but normally the buffer size would be smaller than the message size.

Posted by Andrew Fenster Jan 03, 2012

i am using REST WCF, I want to pass around 50000 Records to the wcf but Bad Request Error Is Occured. I Have Some settings in the service side Web.config file.... <behaviors> <serviceBehaviors> <behavior > <!--<serviceMetadata httpGetEnabled="false" />--> <!--<serviceDebug includeExceptionDetailInFaults="False" httpHelpPageEnabled="False"/>--> <dataContractSerializer maxItemsInObjectGraph="2147483647" ignoreExtensionDataObject ="true" /> </behavior> </serviceBehaviors> </behaviors> ........................................................... <standardEndpoint transferMode="Streamed" helpEnabled="true" automaticFormatSelectionEnabled="true" maxBufferPoolSize="9147483647223234233" maxBufferSize="2147483647" maxReceivedMessageSize="2147483647"/> <standardEndpoint name="DirectoryEndpoint"/> is there any other settings in service side web.config file if yes then please let me know. thanks in advance.....Virat Raj joshi

Posted by virat raj joshi Jan 03, 2012

I have a requirement to encrypt data as well as to stream the response...which binding do you feel is the best for the above scenario..For encryption to work, the content has to be buffered and not streamed. Let me know your thoughts.

Posted by Abhishek Bhat Sep 18, 2011

I have a requirement to encrypt data as well as to stream the response...which binding do you feel is the best for the above scenario..For encryption to work, the content has to be buffered and not streamed. Let me know your thoughts.

Posted by Abhishek Bhat Sep 18, 2011

I have a requirement to encrypt data as well as to stream the response...which binding do you feel is the best for the above scenario..For encryption to work, the content has to be buffered and not streamed. Let me know your thoughts.

Posted by Abhishek Bhat Sep 18, 2011
6 Months Free & No Setup Fees ASP.NET Hosting!
Become a Sponsor
PREMIUM SPONSORS
  • Finally – a virtual platform that delivers next-generation Windows Server 2008 Hyper-V virtualization technology from a managed hosting partner you can truly depend on. Visit www.maximumasp.com/max for a FREE 30 day trial. Hurry offer ends soon. Climb aboard the MaxV platform and take advantage of High Availability, Intelligent Monitoring, Recurrent Backups, and Scalability – with no hassle or hidden fees. As a managed hosting partner focused solely on Microsoft technologies since 2000, MaximumASP is uniquely qualified to provide the superior support that our business is built on. Unparalleled expertise with Microsoft technologies lead to working directly with Microsoft as first to offer IIS 7 and SQL 2008 betas in a hosted environment; partnering in the Go Live Program for Hyper-V; and product co-launches built on WS 2008 with Hyper-V technology.
    The leading .NET charting control now features PDF, Flash and Silverlight export, visualization of large datasets and more. Deliver true charting functionality to your BI, Scorecard, Presentation or Scientific apps. Download evaluation now.
Become a Sponsor