by Derek Du

Configuring WCF web service’s security is just tedious. Microsoft has been trying to make it simple by removing many configuration settings in .NET 4.5, but it can still get messy if you need to touch the security part. There are many bindings and there are Message, Transport, and TransportWithMessageCredentials security modes, each with their own client credential types, not to mention all those authentication modes for Custom Bindings such as AnoymousForCertificate, IssuedTokenForCertificate, IssuedTokenOVerTransport, etc.

Developers are developing WCF web service on the platform from Microsoft. Developers are the users of the platform in this sense. It’s supposed to be user friendly and intuitive. But, I find it is quite difficult to select the right security configuration in different scenarios. Even after you read the documentation from Microsoft carefully, you sill have a very limited idea of how these security modes differentiate from each other.

This post is aimed to cover one small scenario of using WCF security: using a WCF web service with cert-based Message security behind a front-end SSL handler. Often times, your web servers are behind a load balancer that handles all SSL requests and pass in HTTP requests to your IIS. Below is diagram showing the infrastructure.

At first glance, it seems pretty straightforward – Transport security mode covers SSL security, Message security mode handles message encryption. There is even a security mode just for the two modes combined: TransportWithMessageCredentials. We should be able to just use that on both client and service side to achieve what we want, right? However, you just can’t be so optimistic in the world of software development.

This configuration isn’t working. First of all, since the service is not really receiving HTTPS requests, Transport mode should not be used. We just need Message security. So here is the correct configuration on service side:

The client side is where it gets tricky. I can’t really explain why TransportWihtMessageCredential doesn’t work. Something goes wrong under the hood. But this configuration worked for me. Use MutualSSLNegotiated mode and CustomBinding!

I spent a long time researching and trial and error to finally figure this out. Many development teams don’t have the time to mirror the environments of their clients, which makes finding and troubleshooting issues like this difficult. But hopefully this post can help you out.

This blog post originally appeared in Derek's blog. Stuff.