Message compression on WCF

Windows Communication Foundation (WCF) is a message-based communications infrastructure. So developers are concerned about the wire footprint (size) of message sent across the network, the text-based encoding of XML poses challenges for efficient transfer of binary data because to fit binary data into XML text, the common approach is to encode them using the Base64 encoding.

In a Base64-encoded string, each character represents 6-bits of the original 8-bit data, which results in a 4:3 encoding-overhead ratio for Base64, not counting extra formatting characters (carriage return/line feed) that are commonly added by convention.

To avoid this encoding overhead, the Message Transmission Optimization Mechanism (MTOM) standard allows for externalizing large data elements that are contained in a message and carrying them with the message as binary data without any special encoding. Still, as with Base64, MTOM also comes with some necessary overhead for the MIME format, so that advantages of using MTOM are only seen when the size of a binary data element exceeds about 1 KB. Due to the overhead, MTOM-encoded messages might be larger than messages that use Base64 encoding for binary data, if the binary payload remains under that threshold.

Wire-footprint aside, transferring a binary file of 500-MB payload also poses a great challenge at for the service and the client. By default, WCF processes messages in buffered mode. This means that the entire content of a message is present in memory before it is sent or after it is received. So, large messages might easily end up exhausting system’s resources.

The strategy to deal with large payloads is Streaming. On streaming, a message might be multiple gigabytes in size and resemble a continuous data stream more than a handy data package. When data is transferred in streaming mode instead of buffered mode, the sender makes the contents of the message body available to the recipient in the form of a stream, and the message infrastructure continuously forwards the data from sender to receiver as it becomes available. But streaming imposes many serious restrictions like

  • Digital signatures for the message body cannot be performed because they require computing a hash over the entire message contents. With streaming, the content is not fully available when the message headers are constructed and sent.
  • Encryption depends on digital signatures to verify that the data has been reconstructed correctly.
  • Reliable sessions must buffer messages on the client for redelivery if a message gets lost in transfer and must hold messages on the service before handing them to the service implementation to preserve message order in case messages are received out-of-sequence.

When streaming is enabled to handle large data the MaxReceivedMessageSize may be set to an extremely large value which provokes a Denial of Service by causing data to be buffered when the receiver expects it to be streamed. For example, WCF always buffers the SOAP headers of a message, and so if an attacker may construct a large malicious message that consists entirely of headers to force the data to be buffered, a memory overflow occurs.

Because of these functional constraints, you can use only transport-level security options for streaming. Moreover streaming is available only on selected Bindings like BasicHttpBinding, NetTcpBinding, NetNamedPipeBinding.

So, when message size is at a premium, we need to consider compressing the message contents before encoder serialize the message and passes it to the transport.

There are two ways to implement message compression before the message is serialized and passed to transport.

  • Writing custom compression MessageEncoder
  • Inspecting & compressing message body

Writing custom compression MessageEncoder

A MessageEncoder normally resides inside of a Transport Channel. MessageEncoder convert the bytes coming over the wire to a WCF Message class. So we need to compress the byte array, before converting them to message. Here, we will leverage DeflateStream class to compress and decompress the buffered array of bytes.

Custom MessageEncoder

When we build a custom MessageEncoder, we also build an accompanying MessageEncodingBindingElement and MessageEncoderFactory. Also, as with other WCF classes, the binding class handles configuration and the factory class handles creation. Following is the class declaration for CustomCompressionMessageEncoderFactory.

public class CustomCompressionMessageEncoderFactory :
        MessageEncoderFactory{

Following is the class declaration for CustomCompressionBinding.

public class CustomCompressionBindingElement :
        MessageEncodingBindingElement{

For compression and decompression we use CompressBufferArray, DecompressBufferArray helper functions. Internally these functions use DeflateStream class to compress and decompress the buffered array of bytes. Following are functions signature

static ArraySegment CompressBufferArray (ArraySegment buffer, 
BufferManager bufferManager, int messageOffset) {
static ArraySegment DecompressBufferArray(ArraySegment buffer, 
BufferManager bufferManager){

MessageEncoder contains overridable versions of the ReadMessage and WriteMessage functions, which are entry points into the CustomCompressionMessageEncoder. WCF runtime calls the overridden WriteMessage function to encode Message into buffered byte array. Here we use CompressBufferArray helper function to compress the resulting buffered byte array. Similarly, ReadMessage is called to decode a buffered byte array into a Message and DecompressBufferArray helper function is used to decompress the buffer received from ReadMessage function.

Inspecting & compressing message body

When an application calls an operation, the ClientOperation class translates the call and outbound parameters into a message, processes interceptors, confirms that the outbound call conforms to the target contract, and passes the outbound message to the ClientRuntime class, which is responsible for creating and managing the needed channels, handling extra outbound message processing (such as header modification), processing message interceptors in both directions. While, at the server side all message processing is done by the channel and endpoint dispatcher classes (ChannelDispatcher and EndpointDispatcher, respectively) that receive messages and invoke operations. The channel dispatcher pulls messages out of the underlying channel and passes them to the respective endpoint dispatchers.

custom message inspector

On client side IClientMessageInspector allows attaching a new behavior to a specific operation (using ClientOperation class) or a whole contract (using ClientRuntime class) and on server side IDispatchMessageInspector allows attaching a new behavior to whole contract (using DispatchRuntime class) or to a specific operation (using DispatchOperation class).

For compression and decompression, we use CompressMessage, DecompressMessage helper functions. These functions extracts message body node with help of XmlDocument and XmlTextReader classes and internally use DeflateStream class form compression and decompression. Following are functions signature

public string CompressMessage( string messageBody)
{
public string DecompressMessage( string messageBody)
{

On client and dispatch inspectors, we use CompressMessage and DecompressMessage helper functions to compress and decompress the message body.

5 thoughts on “Message compression on WCF

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s