Periodic Issues Connecting to O365 with Teams and Outlook clients

[robling]

I have looked around and there are a number of threads such as this such as this one which are similar but do not match exactly, so I figured I would start another thread just in case there are others out there.

While piloting Zscaler Client Connector/ZApp (herein ZCC) with our I.S. department, I received periodic complaints from remote users that Outlook would not connect to their mailbox to send/receive email. If you “Turn Off” ZCC, the problem will resolve itself immediately. And once Outlook and/or Teams has connected, you can turn ZCC back on and everything will work properly going forward. These are the only apps I have received complaints or noticed the issue. If I open up Excel or Word, I seem to be logged in/connected properly.

Unlike the link above, I DO NOT see tons of entries in the event logs indicating a problem. I have opened up a support ticket and the first round provided some subtle change(s) which may or may not have fixed the issue. It is hard saying since it is intermittant.

I am able to replicate the issue with a VMware VM running Windows on my MacBook, but this could also have something to do with the way things are networked from Fusion, so I am waiting to hear back from my testing pool. I would love to get this problem behind us so it doesn’t bite us as we roll out to 3,350 more users. So if anyone else has had this issue and has fixed it, I would love to hear how. We are only using ZIA right now. And this is with MS Teams and Outlook clients connecting to O365. It almost seems like the clients are unable to authenticate.

Thanks for your time -
-Rob

[jkelly (Jamille Kelly)]

Hi Rob,

A few questions:
-Is this happening on windows or mac?
-are you using a vpn client? If so, split or full tunnel?
-If applicable, is this happening when your vpn client is connected?
-do you have O365 one-click configuration enabled? Policy–>URL & Cloud App Conrtrol–>Advanced Policy Settings–>Scroll down to the bottom under Office 365 Configuration

[Jones_Leung]

Please also check if you have manually add your “autodiscover.” URL to the SSL bypass list as well.

Best Regards,

Jones Leung

[robling]

Going to answer the questions for both Jamille & Jones here;

@jkelly

1. So far, this has only been reported from Windows machines. I have never personally experienced the issue nor had it reported in the Mac environment.
2. We use the Cisco AnyConnect VPN client - split tunnel.
   3a. When reported by users, it has been when they were connected to VPN, but these users are almost always connected to VPN.
   3b. As mentioned, I am able to replicate the issue using a Windows VM (NAT or Bridged mode it doesn’t matter), and it doesn’t matter if I am connected to VPN or not, the behavior is the same. From a clean boot after which ZCC loads and connects and is running, if I open up MS Teams (or Outlook), it will be unable to connect with Microsoft to load the current conversation and presence information. Here is a picture of the two messages you will see at the top of the screen:
   
   
3. Yes. O365 One-Click configuration is enabled.

@Jones_Leung - If you are talking about under Policy | SSL Inspection under the POLICY FOR SSL INSPECTION section, no our autodiscover. URL has not been added to the bypass list. Is this recommended? This is the first I have heard this. Let me know, and I will add it in.

Thanks for your responses.
-RL

[Jones_Leung]

Hi Rob,

Though it is not consistently well documented, but bypass the Autodiscover URL from SSL and auth (if you are not using ZAPP or don’t have IP surrogate enabled) will be more like a best practice, or first thing you can check with to isolate the issue further.

Best Regards,

Jones Leung

[robling]

Thanks for the explanation, Jones, I have added the autodiscover URLs to be bypassed for SSL inspection. While it hasn’t resolved this issue, a best practice setting never hurts, and is appreciated.

-RL

[jkelly (Jamille Kelly)]

-have you added your vpn’s ip address or domain name to the VPN Gateway bypass section with the app profile?

Within The Zscaler Client Connector portal, what are your settings for your forwarding profile? In the ZCC portal, go to Administration–>Forwarding Profile. Click the pencil icon to view the settings.

What do you have listed for “Windows Driver Selection”, and “Forwarding Profile Action for ZIA”?

[robling]

We have the FQDNs for our different VPN gateways in the bypass section of the app profile.

Windows Driver Selection is Packet Filter Based. This was changed a few days ago at the recommendation of Support as being a best-practice for ZPA which we plan on continuing a PoC once we are have ZCC deployed. Prior to the case, it was Route Based.

Forwarding Profile Action for ZIA is configured as pictured below:

Thanks,
-Rob

[jkelly (Jamille Kelly)]

Rob,

I would set the VPN Trusted Network & Off Trusted Network to Tunnel as well, and make sure Packet-Filter Based is selected. You could just select “Same as On Trusted Network” for both options.

Other option: Bypass the url that is causing the issue. You could take a pcap to try and hunt down the url, or you can add all off the O365 url’s (60+) to the ssl exemption list. See the attached file for the list of url’s. It’s a simple copy and paste to the SSL exemption list.

O365 Bypasses.docx (13.5 KB)

[robling]

Thanks Jamille. I have been going through Wireshark captures as I get time today, and so far have my VM reliably receiving chat updates, where it was otherwise failing before. I will continue at it and see if I can get the presence info coming through. I appreciate all of the info!

[manuel (Manuel Bock)]

Rob, did you also tried to create an exception for the login URLs? Solved issues with Teams and Outlook here, at least for one of our testusers. See also here and here.

[robling]

Support asked that I remove login.microsoftonline.com from the bypass list so that way the traffic all appeared to be coming from the same place, so I removed that entry. The other three are still in the list.

I am just going to have to wait and see if it gets reported again. I have done a bunch of drastic changes, and those didn’t even seem to help out my situation with my VMware VM. So rather than dwell on that rare scenario, I will see if we have maybe helped the normal end-user scenario out.

On a side note, I noticed there was a service degradation which was resolved last night/this morning too:

Issue accessing Microsoft applications
STATUS: Resolved
SEVERITY: Service Degradation

We are aware of networking issue at Microsoft end that may be preventing some users from accessing Microsoft applications. We will post additional information on this incident as it is available.

Update - Wed, 28 Oct 2020 05:38:41 UTC

Hopefully something else was fixed somewhere.

[Constantin (Constantin Lauterbach)]

Hi Rob,

did the bypass solved your problems?
Because some users of us are also facing the same issues as you described.

The difference is that we are already Using ZPA productively.

So i would be very interested in your current situation after inserting the bypass as recommended, except the http://login.microsoftonline.com/.

Kind regards

[robling]

Constantin,

Unfortunately, the issues have not gone away entirely. They still persist intermittently. As I get time I look further into this trying different server names/bypasses I come across as part of the Exchange Online and Teams services (especially those connected with MFA authentication). I will be sure to post anything I find that stands out as a potential solution and ask that anyone else do the same. This appears to be a common problem, or if rare the Zscaler name keeps coming up in association with this problem.

In the meantime, I have instructed our Help Desk to open up support tickets if they have time, otherwise, the (temporary) solution seems to be to disable the client connector while logging in to Teams and/or Outlook, and then re-enable the client connection again. Hopefully, they are re-enabling the client connector.

Thanks,
RL

[Constantin (Constantin Lauterbach)]

Rob,

thank you very much for the update!
So we will do the same at our end and also update this post here if we can find a solution.

But I have to say that I am little bit disappointed from Zscaler, as they always talk about the benefits of the Office365-One-Click Function. Even so, it seems like we still need to bypass URL and or IP ranges manually.

[auto_mate (Dan)]

@Constantin @robling are you still seeing these issues?

We have tickets open with both Zscaler and Microsoft for the same, it seems the issue is less to do with what URLs are by-passed and more the local proxy service in our case (we have by-passed all, and nothing, and found both exhibit the same behaviour)

Zscaler are telling us its likely to do with how Microsoft products behave when the proxy service is active (the web versions all work) and there is nothing being blocked in any of the logs we can produce.

[Constantin (Constantin Lauterbach)]

Hi Dan,

I don´t know your environment or if you even use Z-Tunnel 2 but:
We got the information to change the Z-Tunnel 2.0 Primary Transport Selection from DTLS to TLS and since that the problems with teams are solved.

[robling]

Dan - I have not received any additional reports from our users in the last two weeks or so. I will double-check with our help desk, but I don’t think we have experienced any of the issues. While I have hope, I am not confident the problem has magically resolved itself.

I appreciate you logging your experiences here, it is unfortunate but I am glad to see we are not the only ones who have experienced this problem. I wish I could reproduce it on-demand - then I could get support (both Zscaler and MS Premier) focused on the issue.

I have seen other people mention the problem going away after changing to Z-Tunnel 2.0 but I will have to look into it deeper. Then again, I haven’t had anyone report the issue in the last couple of weeks, either.

[scott_bristowe (Scott)]

I’m still seeing these issues pop up, we’ve 300+ incidents over the last 3 weeks or so. Same feedback, nothing visible in the logs.

[GordonWright]

Interesting comment above about changing Tunnel 2.0 config from DTLS to TLS. I will try that. In some locations we have to run DTLS as TLS is rate limited/blocked by government agencies.