In Part 1 of this article, we discussed symptoms of Mac Mail unable to connect to Exchange 2016 servers through KEMP LoadMaster load balancers. We also discussed troubleshooting steps that addressed the issue of ExRCA.com unable to pass EWS tests, however, we stopped there since Mac Mail failed to connect despite ExRCA EWS tests passed.

Let’s see what we can do to get the Mac Mail to connect successfully to Exchange servers.

Since we addressed the 100-continue handling in Part 1 of this article, ExRCA tests did not indicate any more issues. Microsoft Outlook for Mac did not have issue connecting to Exchange even before we changed 100-continue handling to be RFC-7231 compliant.

At this point, I decided to look at the network trace. Looking at the trace captured using Wireshark on the client machine, I noticed the following HTTP dialog:

POST /ews/exchange.asmx <-> 401 Unauthorized <-> POST /ews/exchange.asmx <-> 200 OK

POST /ews/exchange.asmx <-> 401 Unauthorized <-> POST /ews/exchange.asmx <-> 401 Unauthorized <-> POST /ews/exchange.asmx <-> 400 Bad Request

Notice how initial dialog included challenge response for NTLM authentication and ended with 200 OK. At this point Mac Mail client started with another POST operation to EWS. However, this time the pattern followed multiple 401 responses from server and ended with 400.

When comparing that with Outlook for Mac, the pattern looked like the following:

POST /ews/exchange.asmx <-> 401 Unauthorized <-> POST /ews/exchange.asmx <-> 401 Unauthorized <-> POST /ews/exchange.asmx <-> 200 OK

Notice how the pattern looks similar to Mac Mail’s second POST request, however, it ends with 200 OK. The response included data that was requested by POST operation.

I needed to correlate this with the data being received on the Exchange server. Due to newer ciphers which use PFS (Perfect Forward Secrecy) I couldn’t decrypt data being received from load balancer by the server despite having the private key used to encrypt the information. I was able to decrypt client side traffic due to load balancer configuration of ciphers that didn’t use PFS. I don’t recommend this configuration in production and KEMP LoadMaster default setup uses more secure ciphers. I had it configured this way only for troubleshooting purpose in my lab and no production data was in flight during testing.

I was now facing the issue of capturing the headers from the server side. One way to address this is to configure list of ciphers used by server similar to what I had done on KEMP LoadMaster. However, in my test that did not work for some reason. I decided to use failed request tracing in IIS (big thanks to my Microsoft colleague Brad Hughes for the hint). Here’s how to set it up:

– Using IIS Manager, select the “Default Web Site” and select “Failed Request Tracing Rules”

– Create a new rule and select “All Contents” in the “Specify Content to Trace” dialog

– In “Trace Conditions” dialog, select “Status Code(s)” and type 400

– Accept the default selection for trace providers and click finish

– Click “Edit Site Tracing” option under Actions and click “Enable”

– Change trace location or accept default and click “OK”

Now that the trace is enabled and rule configured to capture HTTP 400 error condition, I launched Mac Mail client and reproduced the issue. When the error occurred, numerous xml files were created by IIS. Here’s the most relevant piece of data from the trace file (I have removed unnecessary headers for brevity):

Via: HTTPS/1.1 load.balancer.ip.address:443
Authorization: NTLM

Cookie: X-BackEndCookie=S-1-5-21-…;

ClientId=GALCMQFEEUEGNHCEWFGEVQ; exchangecookie=196574d09f7f419897032917045ee4fb
Host: mail.contoso.com
User-Agent: Mac OS X/10.11.1 (15B42); ExchangeWebServices/6.0 (242); Mail/9.1 (3096.5)
X-ClientSide: client.ip.address:49792 -&gt; load.balancer.ip.address:443

While the headers gives you idea of what is included in a normal POST request from the client, for me, the first and last headers are very important. The “Via” header indicates presence of a proxy between client and server. In this case it indicates the IP address of the load balancer. The X-ClientSide (or X-Forwarded-For) header is used by proxies and load balancers to preserve original client IP address when SNAT is in use. SNAT is recommended when load balancing Exchange servers. KEMP LoadMaster uses X-ClientSide header by default.

When I saw these headers, I remembered the discussion I had with my talented support engineers at KEMP. At some point in our discussions, someone had mentioned that inserting “Via” headers are known to negatively impact Mac Mail client.

I also found an old reference on our knowledge base which provided some more details on the issue: https://support.kemptechnologies.com/hc/en-us/articles/200503159-Exchange-2010-LoadMaster-and-OS-X-mail-issues

The issue stems from how Exchange server responds to the EWS request when “Via” headers are present and how different clients respond to it. When “Via” header is added by the load balancer, Exchange server sets value of Persistent-Auth header to “false”, indicating that client must reauthenticate when sending another EWS request:

This seems to be a protective mechanism to prevent man in the middle attacks.

When “Via” header is not present, Exchange server sets the value of Persistent-Auth to “true”, indicating that client doesn’t need to include authorization information in another EWS request. Client will need to reauthenticate eventually when authentication expires:

Outlook for Mac handles these responses as expected, including authentication information when Persistent-Auth is set to false by Exchange server. Which explains why Outlook worked even when “Via” headers were present.

Mac’s Mail client, however, seems to have its own ways. It authenticates intially, but after first “200 OK” response from Exchange server, which also indicates Persistent-Auth is set to false, Mail client follows up with another EWS request without including required authentication information. This results in dreaded “400 Bad Request” error from the Exchange Server.

Now the fix, if you ask me, should be to ask Apple to fix the behavior of their Mail client. But I won’t hold my breath for that to happen. Instead, I set “Additional L7 Header” from default value of “X-ClientSide” to “None”:

This stops the KEMP LoadMaster from inserting “Via” header in the HTTP stream to the Exchange Servers. Exchange servers respond accordingly, setting Persistent-Auth to “true”. While it’s the Mac’s built-in Mail client that is behaving badly, it seems the load balancer is at fault because changing something on load balancer fixed the issue.

Is it? You be the judge!