Zscaler Private Access - Active Directory

Brief
Microsoft Active Directory is used extensively across global enterprises. Even with the migration to Azure Active Directory, companies continue to utilise Active Directory in a Hybrid environment where workstations may be joined solely to AD, or both AD joined and WorkPlace joined to AAD.
Companies deploying Zscaler Private Access should consider the connectivity workstations need to Active Directory to retrieve authentication tokens, connect to file shares, and to receive GPO updates.
This document describes some of the workings of Microsoft Active Directory, Group Policy and SCCM. The document then covers how Zscaler Private Access should be configured to work transparently with it with these Microsoft Services.
This document is NOT intended to be an exhaustive description of Active Directory, however it will describe the key services, and how Zscaler Private Access functions to utilise them.

Active Directory
Active Directory is used to manage users, devices, and other objects in an organization. It can be utilised as a data structure to store configuration data for Active Directory objects and applications such as SCCM. It is a tree structure exposed via LDAP and DNS, with a security overlay. The security overlay could be a simple password, NTLM Authentication Blob, Kerberos authentication token, or Client Certificate, where these credentials are stored securely in the user object in Active Directory.
The structure and schema for Active Directory is irrelevant for the functioning of Zscaler Private Access, however it is important to understand it to ensure Application Segmentation functions correctly.
Since Active Directory is based on DNS and LDAP, it’s important to understand the namespace. A good reference guide is available from Microsoft (https://docs.microsoft.com/en-us/azure/active-directory-domain-services/concepts-forest-trust) , and we’ll use this to describe Forests and Trusts.

image

In this diagram there is an Active Directory domain tailspintoys.com, with child domains (sub domains) europe and asia, which form europe.tailspinsoys.com and asia.tailspintoys.com. There is a separate Active Directory Domain wingtiptoys.com which has a child domain usa.wingtiptoys.com. There is an Active Directory Trust between tailspintoys.com and wingtiptoys.com, which creates an Active Directory Forest.
Any client within the forest should be able to DNS resolve any object within the forest, and should be able to connect to them. A user account in tailspintoys.com would have the format user@tailspintoys.com , and similarly a user account in wingtiptoys.com would have the format user@wingtiptoys.com .

Kerberos Authentication
Consider the process for a user in europe.tailspintoys.com domain to access a resource in usa.wingtiptoys.com :-

  1. User requests resource http://app.usa.wingtiptoys.com
  2. App.usa.wingtiptoys.com challenges for Kerberos authentication (401 Authenticate-Negotiate)
  3. User connects to europe.tailspintoys.com Domain Controller (TCP/88)
  4. User receives Kerberos ticket for user@europe.tailspintoys.com (Ticket Granting Ticket –krbtgt/Europe.tailspintoys.com/Europe.tailspintoys.com)
  5. User requests resource (Service Ticket) HTTP/app.usa.wingtiptoys.com sending TGT from europe.tailspintoys.com Domain Controller
  6. Europe.tailspintoys.com DC returns unable to issue ticket - directs user to wingtiptoys.com domain and issues “Cross Realm” ticket krbtgt/wingtiptoys.com/Europe.tailspintoys.com
  7. User requests resource (Service Ticket) HTTP/app.usa.wingtiptoys.com from wingtiptoys.com DC sending TGT krbtgt/wingtiptoys.com/Europe.tailspintoys.com
  8. User receives Service Ticket HTTP/app.usa.wingtiptoys.com from wingtiptoys.com
  9. User requests resource http://app.usa.wingtiptoys.com sending Service Ticket HTTP/app.usa.wingtiptoys.comUser
  10. receives content from resource

There may be many variations on this depending on the trust relationships and how applications are resolved. Problems occur with Kerberos authentication if there are issues with NTP (Time), DNS (Domain Name Services resolution) and trust relationships which should be considered with Zscaler Private Access. Additional issues may occur regardless of ZPA, such as Kerberos ticket size, and SID complications for cross-domain authentication. As a best practice, using A Records rather than CNAME records (aliases) is best for Kerberos authentication.

It’s also clear from the above that it’s important for all domains to be resolvable across trusts for Kerberos Authentication to function.

In the example above, Zscaler Private Access could simply be configured with two application segments
● *.wingtiptoys.com TCP/1-65535 and UDP/1-65535
● *.tailspintoys.com TCP/1-65535 and UDP/1-65535

This would also cover *.europe.tailspintoys.com and *.asia.tailspintoys.com as well as *.usa.wingtiptoys.com since the wildcard includes two subdomains resolution.

In steps 3 & 4 the client requests/receives the TGT from the Domain Controller, and subsequently requests/receives service tickets and TGT for the cross-realm. However there is a deeper process for resolving the Active Directory Domain Controllers.

Domain Controller Enumeration & Group Policy
A workstation is domain joined, and therefore exists in an Active Directory domain (e.g. workstation.Europe.tailspintoys.com). The workstation needs to ascertain which domain controller(s) it should connect to for authentication and how to retrieve it’s Group Policy.

Group Policy controls how a workstation should function in an Active Directory – this could be as simple as restrictions for administrators, or could control numerous aspects of applications on the workstations. To start at first principals – a workstation has rebooted after joining a domain. After logon it will identify the domain based on the FQDN and enumerate the domain controllers via DNS, CLDAP, LDAP, and then use Remote Procedure Calls (RPC) and Endpoint Mapper (EPM) to retrieve the Group Policy Objects (GPO) from the domain controller.

  1. Workstation is in Europe.tailspintoys.com
  2. DNS SRV lookup for _ldap._tcp.europe.tailspintoys.com
  3. SRV SRV Response returns multiple entries domaincontroller1.europe.tailspintoys.com:389, domaincontroller2.europe.tailspintoys.com:389, domaincontroller3.europe.tailspintoys.com:389, …
  4. For each entry in the DNS SRV response, CLDAP (UDP/389) connection and query Netlogon Service (LDAP Search), returning
    a. Server AD Forest
    b. Server AD Domain
    c. Server AD HostName
    d. Server NetBIOS Domain Name
    e. Server NetBIOS Name
    f. AD Username
    g. Server AD Site
    h. Client AD Site (based on Client IP Address)
  5. Client builds DNS query based on Client AD Site, and performs DNS lookup – e.g.
    a. DNS SRV lookup for _ldap._tcp.ENGLAND._sites._dc._msdcs.tailspintoys.com
  6. DNS SRV Response returns multiple entries domaincontroller10.europe.tailspintoys.com:389, domaincontroller11.europe.tailspintoys.com:389,
  7. Client looks for response where Server AD Site and Client AD Site are the same (i.e. most efficient)
  8. Client performs LDAP query to Domain Controller requesting capabilities
  9. Client performs NTP Sync (UDP/123)
  10. Client requests Kerberos LDAP Service Ticket from AD Domain Controller
  11. Client performs LDAP bind using Kerberos (SASL)
  12. Client makes RPC call to Domain Controller (TCP/135) which returns unique port to connect to for GPO (high port range 49152-65535 – configurable through registry)
  13. Client connects to high port for EPM/DCE
  14. Client requests Group Policy Object for workstation via LDAP (SASL authenticated). Return Group Policy Object ID
  15. Client connects to Domain Controller using SMB2 (TCP/445) and retrieves Machine Group Policy Objects
  16. Client requests Kerberos user TGT and Service Ticket from AD Domain Controller for CIFS
  17. Client connects to Domain Controller using SMB2 (TCP/445) and retrieves User Group Policy Objects

Through this process, the client will have

  1. DNS resolved Domain Controllers
  2. Connected to Domain Controller
  3. Received Kerberos tickets for machine and user, and Service Tickets for LDAP and CIFS
  4. Retrieved Group Policy Object descriptors via CLDAP, LDAP, DCE/RPC, and CIFS

From a connectivity perspective it’s important to

  1. Understand all DNS Domains
  2. Allow Access to
    ● UDP/389
    ● TCP/389
    ● UDP/123
    ● TCP/88
    ● TCP/445
    ● TCP/135
    ● TCP/49152-65535

Active Directory Sites and Services

In the Domain Controller Enumeration, the AD Site is key to ascertaining the closest domain controller. This is controlled in the AD Sites and Services control panel for Active Directory. A site is simply a label provided to a location where Domain Controllers exist. Subnets are defined and associated with the site, and inter-site transport controls the cost of utilizing the link. In this way Active Directory creates priorities for Domain Controller usage and how replication works across WAN/LAN links.

image

This is then automatically propagated toActive Directory DNS to enable the AD Site Enumeration. Note the default-first-site which gets created as the “catch all” rule.

image

SCCM

SCCM can be deployed in two modes – IP Boundary and AD Site.
In the IP Boundary mode, the client assesses its own IP interfaces and returns this data to the SCCM Management Point. The SCCM Management Point uses this data to determine the SCCM Distribution Point which will serve the installer packages.

In the AD Site mode, the client uses the Active Directory Site data returned in the AD Enumeration (CLDAP) process and returns this data to the SCCM Management Point. The SCCM Management Point uses this data and the AD Sites & Services and Inter-Site Link data to ascertain the SCCM Distribution Point which will serve the installer packages. An important difference is that this method effectively uses the connection’s source IP address (as seen by the CLDAP process) instead of the client communicating its interface addresses.

The decision to use IP Boundary or AD Site is largely dictated by customer preference and network topology. IP Boundary can be simpler to implement, especially in environments where AD replication may be problematic, or IP Overlaps / Address Translation may hamper AD Site implementation.
Similarly AD Site can be implemented where a robust replication policy exists, and a (relatively) flat/routed network exists. Microsoft will explicitly state that AD Site doesn’t suit networks with NAT, but specifically this is a problem with DNS and Address Translation.

DFS
Distributed File Services (DFS) is a mechanism for enabling a single mounted network share to be replicated across multiple file systems, and to simplify how shares are identified across the network. This provides resilience and high availability, as well as performance improvements where shares are replicated globally and users connect to the closest node.
DFS relies heavily on DNS with a dependency on DNS Search Suffixes, as well as Kerberos for Authentication. DFS uses Active Directory Site information and path weight costs to calculate the most efficient path to a share mount point.
Customers may have configured a GPO Policy to test for “slow link detection” which performs an ICMP (Ping) to the mount points. If the ICMP response is over a certain threshold, or fails to respond, then the link is deemed “slow” and fails to mount. See https://getadmx.com/?Category=Windows_10_2016&Policy=Microsoft.Policies.GroupPolicy::GPTransferRate_1 for more details.
A DFS share would be a globally available name space – e.g. \share.company.com\dfs . However, this is then serviced by multiple physical servers – e.g. \server1\dfs and \server2\dfs. A user mapping a drive to \share.company.com\dfs would be directed to connect to either \server1 or \server2. Depending on the client AD Site and the AD Site for the mount points, the client will establish a connection with the most efficient server.
In this example, it’s important to consider several items

  1. The mount point \share.company.com\dfs is a global namespace
  2. User would receive a Kerberos Service Ticket for CIFS/share.company.com
  3. User would retrieve mount points \server1\dfs and \server2\dfs – which would need to be completed to FQDN’s \server1.company.com\dfs and \server2.company.com\dfs
  4. Upon making the decision which mount point to connect to, the user would receive a Kerberos Service Ticket for CIFS/server1.company.com or CIFS/server2.company.com
  5. The mount points could be in different domains – e.g. server1.emea.company.com or server2.apac.company.com. Or even in different trusted domains server1.europe.tailspintoys.com or server2.usa.wingtiptoys.com.
    a. The DNS Search Suffixes would need to account for these mount points
    b. The Kerberos ticketing would need to account for the domain trusts
    c. For Kerberos ticketing to work, the AD Site enumeration needs to function
    As an example, this WireShark trace shows the client requesting the DFS Referrals in packet 4205, and receiving the referrals in packet 4210. Packet 4210 is displayed in the bottom pane detailing two of the four mount points servicing the DFS mount point.

image

Zscaler Private Access

Zscaler Private Access (ZPA) works with Active Directory, Kerberos, DNS, SCCM and DFS. It’s important to consider the implications Application Segmentation has when defining Active Directory, since ZPA effectively performs DNS proxy function (returned IP address is not the real IP address of the server) as well as DNAT for the client-side connection, and SNAT for the server-side connection. The DNS, DNAT and SNAT functions are dynamic and are an integral part of the ZTNA architecture.
Since an application request may be passed through multiple App Connectors serving the application, a user may be presented on the network from multiple locations. This has an effect on Active Directory Site Selection.
Domain Controller Enumeration & Group Policy
Consider the following, where domain.com is a globally available Active Directory. Connectors are deployed in New York, London, and Sydney. A roaming user is connected to the Paris Zscaler Service Edge.

image

The workstation goes through the AD Site Enumeration process, and issues the _LDAP._TCP.DOMAIN.COM query. This would return all Active Directory domain controllers (assuming there is one in every city) NYDC.DOMAIN.COM, UKDC.DOMAIN.COM, AUDC.DOMAIN.COM (say). For this lookup to function, an Application Segment must exist containing *.DOMAIN.COM, even if this Application Segment contains simply TCP/1

The workstation would then make the CLDAP requests to each of the domain controllers to identify which AD SITE they are in. At this point it’s imperative that the connector selected for these queries is the connector closest to the user. i.e. the London node should be used for the connection to NYDC.DOMAIN.COM:UDP/389, UKDC.DOMAIN.COM:UDP/389, and AUDC.DOMAIN.COM:UDP/389.
This is “counterintuitive” since you would expect to use the ZPA connector closest to each of them, however as far as AD Sites is concerned we need to pass through the closest connector to user for all these requests – since the source IP for any of these requests is used to identify the Client SITE for subsequent Active Directory request. The users Source IP would be London Connector for the request to AUDC.DOMAIN.COM, which would then return “SITE is London UK”. For this connection to succeed, an application segment must exist containing either *.DOMAIN.COM with UDP/389, or containing each of the domain controllers with UDP/389.

It’s clearly imperative that the ZPA App Connector can perform internal DNS resolution across the domain, and connect to the Active Directory Domain Controllers on the necessary ports – UDP/389 in particular. Any firewall/ACL should allow the App Connector to connect on all ports. It’s also imperative that the ZPA App Connector IP is part of the IP Subnets associated with the AD Site. (even if NATted behind a firewall). This article Zscaler Private Access - Active Directory Enumeration provides details of a script which can be run on the App Connector to ensure connectivity to the Domain Controllers, and identify the AD Sites and Services returned.

The workstation would issue a subsequent request for _LDAP._TCP.ENGLAND._sites._dc._msdcs.DOMAIN.COM which would return the UKDC.DOMAIN.COM which would process the remainder of the Netlogon and GPO requests.
As ZPA is rolled out through an organization, granular Application Segments may be created and policy written to control access. From an Active Directory perspective you may create an application segment for each regions or countries AD Servers – a company may have 1000 Domain Controllers across 100 countries, and a single Application Segment with 1000 entries may not be manageable. It is, however, imperative that ALL the Domain Controller application segments are associated with ALL connector groups capable of functioning for Active Directory Enumeration.

Considering a company with 1000 domain controllers, it is likely to support 1000’s of users. In the Active Directory enumeration process, an individual user will perform the DNS SRV lookup _LDAP._TCP.DOMAIN.COM and receive 1000 entries in the response. The client would then make UDP/389 connections to the servers in the response. These requests may pass through several ZPA App Connectors simultaneously to ascertain the AD Site. With 1000’s of users performing the same lookup at the same time, this may present an increase in traffic through ZPA App Connectors. It is therefore recommended to deploy ZPA App Connectors dedicated to Active Directory and ensure the App Connector performance improvements (Ephemeral Port increases) detailed here Zscaler App Connector - Performance and Troubleshooting

Summary
● Connector Groups dedicated to Active Directory where large AD exists
● App Connectors have connectivity to AD on appropriate ports AND their IP addresses are in the appropriate AD Sites and Services subnets.
● Dynamic Server Discovery group for Active Directory containing ALL AD Connector Groups
● Wildcard application segment *.domain.com for DNS SRV to function
o Application Segment contains AD Server Group
● Application Segments containing the domain controllers, with permitted ports
o TCP/88: Kerberos
o TCP/464: Kerberos Password Change
o TCP/3268: Global Catalog
o TCP/3269: Global Catalog SSL (Optional)
o TCP/135: MSRPC
o TCP/139: Common Internet File Service (CIFS)
o TCP/445: CIFS
o TCP/49152-65535: High Ports for RPC
o UDP/88: Kerberos
o UDP/123: NTP
o UDP/464: Kerberos Password Change
o UDP/389: LDAP
o UDP/445: CIFS
● Domain Controller Application Segment uses AD Server Group

Active Directory Authentication
It’s entirely reasonable to assume that there are multiple trusted domains for an organization, and that these domains are not internet resolvable – for example domain.intra or emea.company.
For Kerberos authentication to function, the wildcard application domains for SRV lookup need to be defined for the lookups of _kerberos._tcp.domain.intra. The Domain Controller Enumeration process occurs similar to how Site Enumeration occurs (previous section), however this time it will also look up across trust relationships.
Summary
● Wildcard application segments for all authentication domains
o *.domain.intra for DNS SRV to function
o *.otherdomain.local for DNS SRV to function
o *.emea.company for DNS SRV to function
● Application Segments containing the domain controllers, with permitted ports for Kerberos Authentication
o TCP/88: Kerberos
o TCP/464: Kerberos Password Change
o UDP/88: Kerberos
● Domain Controller Application Segment uses AD Server Group (containing ALL AD Connectors)
SCCM
SCCM can be deployed in IP Boundary or AD Site mode. In a traditional remote access solution (VPN) the user is provided an IP address on the network (VPN DHCP Pool), which would be registered as an IP Boundary, or which would be part of an AD Site. With ZPA the user is not presented on the network, and their IP address is invariably provided by their local router – e.g. 192.168.1.1 – which would be used by many users in many countries across the globe.
IP Boundary can be used with Zscaler Private Access, provided the RFC1918 ranges are configured as IP Boundaries. This may also have the effect of concentrating all SCCM requests on the same distribution point.
AD Site is a better way of deploying SCCM when using ZPA. The AD Site is ascertained based on the ZPA Connector’s IP address during the NetLogon process, and the user is directed to the better SCCM Distribution Point based on this.
In a scenario where the SCCM deployment is IP Boundary, it is conceivable to configure specific AD Sites for Zscaler Private Access App Connectors, and use these sites to control SCCM Distribution points. This way IP Boundary is used for users “on network” and AD Site is used for users “off network” via ZPA.
When a client connects to SCCM Management point to request a package, it is returned a list of Distribution Points which host the packages. The list returned may be unqualified shortnames, rather than FQDN’s – so it is important that DNS Domain Search Suffixes are configured in Zscaler Private Access.
Summary
● Domain Search Suffixes exist for domains where SCCM Distribution points exist.
o Ensure “Domain Validation in Zscaler App” is ticked for all domains. This ensures that search domains do not “leak” to the internet and ZPA is tried for all domains internally first.

image

● Application Segments containing all SCCM Management Points and Distribution Points with permitted SCCM ports
o TCP/80: HTTP
o TCP/443: HTTPS
o TCP/10123: HTTP Alternate
o TCP/8530: HTTP Alternate
o TCP/8531: HTTPS Alternate
o TCP/445: SMB
● Use AD Site mode for Client Distribution Point selection
o If IP Boundary is used – consider AD Site specifically for ZPA
● If IP Boundary ONLY is used (i.e. no ability to use AD Site) – configure IP Boundary with ALL RFC1918 addresses

DFS
DFS Uses Active Directory extensively for Site selection and Inter-Site path cost. Kerberos authentication is used for access.
When looking at DFS mount points, the redirects are often non-FQDN’s – i.e. they are shortnames. This relies on DNS Search Suffixes to complete the shortname to an FQDN – this also has an effect on how Kerberos Tickets are generated – so it is imperative that DNS Search Suffixes are created properly.
In the example above, where the DFS mount point was \company.co.uk\dfs, and the referrals were to servers \UK1234CSC123\dfs and \UK1923C4C780\dfs it would be necessary to have a domain search of company.co.uk in order for these to be completed to \UK1234CSC123.company.co.uk\dfs and \UK1923C4C780.company.co.uk\dfs

Once the DNS Search order is applied, the shares can appropriately be completed and the Kerberos ticketing can take place for the FQDNs. N.B. That they may not be in the same domain, and trust relationships/domain suffixes may need to be in place for multiple domains globally.
Summary
● Domain Search Suffixes exist for ALL internal domains, including across trust relationships
o Ensure “Domain Validation in Zscaler App” is ticked for all domains. This ensures that search domains do not “leak” to the internet and ZPA is tried for all domains internally first.

image

● Kerberos Authentication for all authentication domains is in place
o Regardless of DFS, Kerberos tickets should be accessible for all domains
● Active Directory Site enumeration is in place
o Ability to access all AD Sites from all ZPA App Connectors
o AD Site enumeration is necessary for DFS mount point calculation
● Application Segments containing DFS Servers
o TCP/445: SMB
o Single Segment for global namespace (e.g. \company.co.uk\dfs would have App Segment company.co.uk)
o Application Segments for individual servers (e.g. \UK1234CSC123.company.co.uk\dfs and \UK1923C4C780.company.co.uk\dfs could have a single segment containing UK1234CSC123.company.co.uk and UK1923C4C780.company.co.uk as they’re the same mount point)

Summary Recommendations

The following recommendations are made when deploying Active Directory, SCCM, and DFS with Zscaler Private Access

  1. Apply App Connector performance and troubleshooting improvements Zscaler App Connector - Performance and Troubleshooting
  2. Ensure Domain Search Suffixes cover all internal application/authentication domains
  3. Ensure Domain Search Suffix has “Domain Validation in Zscaler App” ticked
  4. Create a wildcard application segment for Active Directory SRV lookups, including all trusted authentication domains
  5. Deploy App Connectors within Active Directory Sites’ IP Subnets
  6. Create Application segments for
    a. Active Directory Domain Controllers – separate segments may be appropriate depending on number of Domain Controllers
    b. SCCM Management and Distribution Points
    c. DFS Name Spaces
    d. SMB2/CIFS servers hosting DFS Shares
  7. Associate Application Segments with Server Groups containing appropriate App Connectors
    a. Dedicated App Connectors for large AD deployments
    b. Active Directory Server Group contains ALL App Connectors capable of enumerating Active Directory (either dedicated for AD for large deployments, or shared with App Connectors for other deployments)
    c. Server Group for DFS should contain ALL App Connectors capable of connecting to DFS Shares
    d. Server Group for SCCM should contain ALL App Connectors capable of connecting to SCCM servers

e. Server Group for CIFS, SMB2 may contain ALL App Connectors, however it could be constrained geographically as necessary.
8. Server Groups should ALL be “Dynamic Discovery”
9. Ensure connectivity from App Connectors to all applications – ideally no ACL/Firewall should be applied. App Connectors will use TCP/UDP/ICMP probes to identify application health.
a. Use Script from here Zscaler Private Access - Active Directory Enumeration to test connectivity from Active Directory App Connectors to AD Site Enumeration.

Glossary of terms

RPC Remote Procedure Call - protocol to learn / request a service on a remote machine
EPM Endpoint Mapper - A client will call the endpoint mapper at the server to ask for a “well known” service. The server will answer the client at which addresses this service is available (if at all)
DCE/RPC Distributed Computing Environment - the API & protocol specs for RPC
GPO Group Policy Object - defines AD policy. Used by Kerberos to authorize access
TGT Ticket Granting Ticket - Proof of authentication and used to request SGTs
SGT
(Service Ticket) Service Granting Ticket - Proof of authorization to access a specific service

ZPA - Active Directory.pdf (2.9 MB)