What is the difference between PAC in Forwarding profile and APP profile?

Oucean · January 30, 2023, 8:53am

My customer is rolling out Zscaler ZIA with Zscaler Client Connector (Tunnel 2.0). But I’m fully confused about the PAC files in Forwarding profile and APP profile.

I’ve read this thread Difference in use of "App Profile PAC file" and "Forwarding Profile PAC file" and had an insight that PAC in Forwarding Profile will directly apply to Windows system proxy script, PAC in APP Profile will be used by Zscaler Client Connector only for determine how to deal with traffic.

But I still have some confusions:

In that thread, David_Cooper said “Forwarding profile PAC applies only to Tunnel with Local Proxy and Enforce PAC modes.” But what will happen if I select “Tunnel” and also define a PAC file? Actually this is what is it in our Zscaler Console.
According to my understanding, PAC in Forwarding Profile will directly apply to Windows system proxy script. So if I define a “DIRECT” action for some domains, those domains should be bypassed, right? But from the testing, it will not bypass but still send to Zscaler? Could you please guide me what did I mis-understand?
From Zscaler Help document. I saw one highlight “When using Tunnel mode, only use the PAC file to bypass traffic away from Zscaler Client Connector or to tunnel traffic to Zscaler Client Connector. Do not use it to tunnel traffic to the Zscaler cloud.” But could you please let me know what will happen if using the PAC file to tunnel traffic to Zscaler Service Edge? Since this is what our customer did in the forwarding profile’s PAC file. Will it due to we are using Regional Surcharge Location?

Thank you very much.

Andreas · January 30, 2023, 9:51am

Hi,
check out my “Update for my Tunnel 2.0 configuration…” section in the mentioned thread.

The forwarding profile pac is a must for “tunnel with local proxy” and can be used in tunnel mode. I used it in tunnel mode for other non-zscaler explicit proxies.
In tunnel mode, a DIRECT exception in the forwarding pac file is by default not bypassed from Zscaler. Direct means usually port 80&443 and in Tunnel mode ZAPP catches all port 80&443 traffic. So you need additional excpetions in App profile pac or app profile itself.
I my case the other explicit proxies used for example port 8080 and in tunnel 1 this is not catched by ZAPP. However in Tunnel 2 mode (all ports and protocols) also 8080 traffic is catched and therefore you need to place exceptions. For my explicit proxies I think I used “destination exclusions in app profile” and not in app-profile-pac.
I think I also experimented in the forwarding profile pac with something like "return muc1.sme.zscloud.net:dedicated proxy port. This is basically the same like “return myproxy.acme.net:8080” and will force traffic to the Datacenter in the Kingdom of Bavaria and you will get a Bavarian IP.
In tunnel 1 it is excluded (since only 80&443 is cached by ZAPP and I used the dedicated proxy port. If you do not have one, you can try to use other Ports supported by Zscaler. 9480 could be a good choice, check Writing a PAC File | Zscaler for a description of ports).
In tunnel 2 you need to create additional exceptions, in order to prevent ZAPP from catching all ports and protocols.

If you do so with muc1.sme.zscloud.net: this will cause the forwarding method in the logs to be e.g., “PAC” or “PAC over IPSec” and no longer ZAPP traffic. Also for this traffic a classic “SAML with cookie authentication” will be used, since traffic is not sent via Z-TUNNEL established by ZAPP. So using this may become burdensome, if cookies are missing or cannot be read due to ssl incryption. Therefore port 9480 (no-auth) will help, but this works only for known locations. Road Warriors connecting to 9480 will still require SAML authentication.

What I would do.

Check if there is a reason for tunnel 2 instead of tunnel 1. If you do not have Adv. Firewall, tunnel 1. might be better.
Check, why you need exceptions in the forwarding pac. For me, this was only the case, if other explicit proxies need to be used. An “other explicit proxy” might be a defined Zscaler DC, for example that certain traffic as a certain country specific IP address, while ZAPP has established a Z-Tunnel to a complete different Zscaler DC.
Always remember: tunnel 1 is only 80&443, while Tunnel 2 is all ports and protocols. Therefore the “exception approach” is different for T1 and T2.

Best regards
Andreas

Oucean · February 2, 2023, 3:14am

Hello Andreas

Thank you so much for your detail reply. It’s much clear now. Since I don’t have write access to my company’s Zscaler console and I can’t do any test…

According to your point and my research:

This point is clear now. Thank you very much again.
For this point, I also think DRIECT action means ZAPP will catch the kind of traffic according to my test. Thanks for your confirmation so I think it’s also clear now. And we are using Tunnel 2.0 also, according to Zscaler Document, it will catch not only port 80&443 but also all the traffic with DIRECT action. I believe that’s why we need to bypass it in APP Profile.
But in our company dashboard, I found a strange thing. We have few domains that has anti-proxy detection, those domains must be bypassed/excluded in Zscaler. And I saw the engineer added those domains in forwarding profile PAC and action is set to “PROXY ${ZAPP_TUNNEL2_BYPASS}”. And also added those domains in app profile PAC then set action is set to DRIECT. Do you also think it’s not necessary and not needed to add “PROXY ${ZAPP_TUNNEL2_BYPASS}” action in forwarding profile?
For this point, it’s still not clear actually. In my company’s configuration, I can see the engineer defines a Surcharge Location in PAC of forwarding profile and defines another Surcharge Location in PAC of app profile. And The result I tested is, the define in forwarding profile does not work at all. All the traffic is still sending to the one in APP profile. So I raised this query to see what will happen if there is a Zscaler Location defined in PAC of forwarding profile. Will it be directly ignored or…

Andreas · February 2, 2023, 8:23am

Hi,
to 2:
There are two options for direct exceptions in tunnel 2 situation_
If you have a forwardingprofile pac, you can use return "PROXY ${ZAPP_TUNNEL2_BYPASS} there and further you will need the same exception in the AppProfile.pac (or perhaps in the app profile itself.)
Other option:
For ZCC 3.8 or later, “return Direct” can be used instead of return “PROXY ${ZAPP_TUNNEL2_BYPASS}”;
If you configure under "Z-Tunnel 2.0 Transport Settings"the option “Redirect Web Traffic to ZCC Listening Proxy”. You may get rid of forwarding profile pac completely, if only “return direct” is remaining there.

Some infos here:
Best Practices for Adding Bypasses for Z-Tunnel 2.0 | Zscaler

To 3: That reads strange.
In the app-profile pac, you usually define where the Z-Tunnel established by ZAPP is terminating. You can of course use if clauses (example in pseudocode), like

if $COUNTRY==“Germany” return muc1.sme.zscloud.net,
if $COUNTRY==“Australia” return tyo4.sme.zscloud.net // to avoid RS ZENs.
default: return ${GATEWAY_FX}:443;

This will force to terminate the tunnel in Tokio (if you are in AU) or to Munich (if you are in Germany) or to the closest ZEN if you are in a different country.
This should not (or must not) be done in the forwarding profile. Also I do not think that ZAPP is capable of establishing two Z-tunnels at the same time to different ZENs.

However… as mentioned before, you can define different explicit proxies in the forwarding profile. Those explicit proxies must also be defined as exclusions in the app profile (or perhaps app profile pac) to ensure that this traffic is not catched by tunnel2 (all ports and protocols).
So I used the IP addresses of the explicit proxy in the destination exclusions of the app profile.
If you like, you can use a certain ZEN node as explicit proxy, too. But I do not think that this is a good idea, because of potential side effects of the required exclusions in app-profile.pac or app-profile.

Also it violates the documentation:
In Tunnel mode, you must only use the forwarding profile PAC file to bypass traffic away from Zscaler Client Connector or to tunnel traffic to Zscaler Client Connector. Do not use it to tunnel traffic to the Zscaler cloud.

I guess, what happens if you do so anyway:
ZAPP establishes a Z-tunnel2 to the ZEN defined in app-profile.pac

Browser accesses a URL that requires a special explicit proxy (in this case a ZEN), as defined in forwarding.pac.
Traffic is “evaluated by ZAPP” and based on the config of App profile / app-profile.pac it is bypassed from Z-Tunnel2.
So traffic hits the explicit proxy (in this case a ZEN) “direct” not via ZAPP/Z-Tunnel/Zscaler DC.
If there is a local Firewall, this traffic must be allowed, of course.
If the explicit proxy is a ZEN,the ZEN will then behave like “pac file only” traffic forwarding and will try to “saml-authenticate” the user. Which might need more Bypass exceptions for the IDP, btw.
Therefore I would use this only in conjunction with 9480 (no-auth port works only from known locations) or dedicated proxy port (will reduce SAML authentication, but not avoid).

If you check the logs for this example you will see as forwarding method:
Client Connector / Tunnel 2: for “normal URLs”
PAC file /PAC over GRE/PAC over IPSec: for the URLs that use the explicit proxy.

Best regards
Andreas

Oucean · February 7, 2023, 8:00am

Hello Andreas.

Thank you so much for replying me so quick. I’m sorry due to Network limitation, I can access Zenith only when I’m in office. But in office I’m always busy with multiple other cases. So…Apologize for always replying you late…
Thanks again for your detail explanation. So point 2 is also clear now.

For point 3, Yeah! I totally agree with you. This also violates the documentation. But I’m confused why the engineer of my company did this configuration. I’m thinking the ZEN node defined in forwarding PAC file is not working, so it use DIRECT action. Let me share a sample similar like the PAC file in my company. Due to company policy I can’t share the real PAC file here but I can share one very similar.

PAC File in Forwarding Profile:

function FindProxyForURL(url, host) {
var privateIP = /^(0|10|127|192\.168|172\.1[6789]|172\.2[0-9]|172\.3[01]|169\.254|192\.88\.99)\.[0-9.]+$/;
    var resolved_ip = dnsResolve(host);
   var country = "${COUNTRY}";

    /* Don't send non-FQDN or private IP auths to us */
    if (isPlainHostName(host) || isInNet(resolved_ip, "192.0.2.0","255.255.255.0") || privateIP.test(resolved_ip))
        return "DIRECT";

    /* FTP goes directly */
    if (url.substring(0,4) == "ftp:")
        return "DIRECT";

    /* Google Auth URL directly accessible */
	if ((shExpMatch(host, "trust.zscaler.com")) ||
	    (shExpMatch(host, "*.zscaler.net"))&& 
	    (url.substring(0,5) == "http:" || url.substring(0,6) == "https:"))
       return "PROXY ${ZAPP_TUNNEL2_BYPASS}";

        if (shExpMatch(country, "China")) {
    return "PROXY bjs1.sme.zscaler.net:443; DIRECT";
 	}
}

PAC File in APP Profile

function FindProxyForURL(url, host) {

    var privateIP = /^(0|10|127|192\.168|172\.1[6789]|172\.2[0-9]|172\.3[01]|169\.254|192\.88\.99)\.[0-9.]+$/;
    var resolved_ip = dnsResolve(host);
    var country = "${COUNTRY}";

    /* Don't send non-FQDN or private IP auths to us */
    if (isPlainHostName(host) || isInNet(resolved_ip, "192.0.2.0","255.255.255.0") || privateIP.test(resolved_ip))
        return "DIRECT";

    /* FTP goes directly */
    if (url.substring(0,4) == "ftp:")
        return "DIRECT";

    	if ((shExpMatch(host, "trust.zscaler.com")) ||
    	(shExpMatch(host, "*.zscaler.net")) && 
	    (url.substring(0,5) == "http:" || url.substring(0,6) == "https:"))
        return "DIRECT";

   if (shExpMatch(country, "China")) {
    return "PROXY bjs3.sme.zscaler.net:443; DIRECT";
 	}

    /* Default Traffic Forwarding. Forwarding to Zen on port 80, but you can use port 9400 also */
    return "PROXY ${GATEWAY_FX}:80; PROXY ${SECONDARY_GATEWAY_FX}:80; DIRECT";
}

You can see there is a DIRECT action as default action in the PAC file of Forwarding profile. So I’m thinking will it be the PROXY action is actually not working. And the DIRECT action is working?

Andreas · February 9, 2023, 1:27pm

Hi,

in your forwarding-pac example I think the “default return direct” is missing.
I would expect in the original something like:

if (shExpMatch(country, “China”))
return “PROXY bjs1.sme.zscaler.net:443; DIRECT”;

else
return “DIRECT”;

Since the China setting is in forwarding.pac and in app-profile.pac, it looks for me, that the engineer did not understand the difference of both pac files or is quite a Zscaler-magician.
But to be honest, my first pac files for ZCC looked the same and it took quite a long time for me and some discussions with our Zscaler contacts to understand the difference. That is, why I posted the other thread in the community.

And in short it is:
Pac in forwarding profile is only useful, if you need to use other explicit proxies. This is due to the fact that the browser must decide when to use the other explicit proxy, since ZCC cannot do this.

Pac in app-profiles defines the ZEN to connect to and the bypasses.

BTW: I would not use the “DIRECT” Fallback in the app-profile pac.
return “PROXY ${GATEWAY_FX}:80; PROXY ${SECONDARY_GATEWAY_FX}:80; ~~DIRECT~~”;
Best regards
Andreas

Oucean · February 16, 2023, 9:39am

Hello Andreas.

Again, thank you so much for your detail explanation. And I believe we are ini the same page now. I also think our engineer does not understand the difference of both PAC files and I also don’t think there should be a “DIRECT” fallback in the APP Profile. So I think we have the same opinion now.

But I have something else to share. I’m troubleshooting a slowness issue so I did a packet capture on my workstation. And then I found an interesting thing. It seems all the traffic is encapsulated to bjs1.sme.zscaler.net with port 443 in a proxy way. And then encapsulate again to ZCC Tunnel and send to bjs3.sme.zscaler.net. Maybe this explains the “PROXY bjs1.sme.zscaler.net:443; DIRECT” in the Forwarding profile. And I believe this should not be like that.

Andreas · February 17, 2023, 12:15pm

But I have something else to share. I’m troubleshooting a slowness issue so I did a packet capture on my workstation. And then I found an interesting thing. It seems all the traffic is encapsulated to bjs1.sme.zscaler.net with port 443 in a proxy way. And then encapsulate again to ZCC Tunnel
[/quote]

Hi,
yes, I expect exact this behavior. The browser is in your case unaware of ZCC and evaluates “its” pac file, which is the forwarding pac. This pac tells the browser to CONNECT to bjs1:443.
In the forwarding.pac example all traffic (except the usual exceptions for internal traffic) will be sent to BJS1 via Z-Tunnel.

ZCC in T1 will capture 80&443 traffic (which matches) and in T2 will capture all ports and protocols (which matches). So ZCC captures this traffic and sends it via Z-Tunnel to the DC. It could be interesting which DC processes the traffic, if any. I would expect that the Z-tunnel destination DC processes traffic, since one Zscaler-DCs do not re-route traffic to other Zscaler-DCs.
Behavior could only be different, if there are exceptions in the App-Profile for BJS1, to exclude this traffic from the Z-tunnel (aka. bypass).

Also behavior would be different in a “tunnel with local proxy” setup.

Best regards
Andreas

Oucean · February 21, 2023, 5:58am

Hello Andreas

Yes. This is the expected behavior and confirmed our opinion about PAC file in forwarding profile. This query is clear also. Thank you so much for your continuous help.
But I think I have a different opinion about which DC processes the traffic. In my opinion, both DCs process the traffic. For Tunnel 2.0, the traffic is send to BJS3 DC (configured in APP Profile) first. And then BJS3 DC processes the traffic. And then BJS3 DC found this is a traffic to TCP 443 port of BJS1 DC. Then it was send to BJS1 DC and then reach the destination. BJS3 DC may not care if the destination IP belongs to BJS1 DC. I think it only processes the traffic and then send to the destination, no matter where the destination is.
I found this issue during a access slowness troubleshooting, so I’m thinking this two DCs traffic played a role about our network slowness. In my opinion, no matter it plays a role or not, this should be corrected…But unfortunately, our engineers don’t believe my point…

Andreas · February 23, 2023, 7:22pm

Hi,

to determine the proccessing DC could be really tricky.
China is special, anyway.

To test I would use for example MUC1 in the forwarding pac and TOKYO4 in the AppProfile.

Forwarding pac:
if ((shExpMatch(host, “whatismyip.com”)
return “PROXY muc1.sme.zscaler.net:443”
if ((shExpMatch(host, “wieistmeineip.de”)
return “PROXY cph2.sme.zscaler.net:443”
return “DIRECT” // default return for all other URLs

AppProfile.pac
return “tyo4.sme.zscaler.net:443” // try to access showip.net

Access different pages that show the ip address, as well as ip.zscaler.com
In Tunnel2 condition: Add Destination exclusions for the ip range of muc1 and cph2 and repeat test.

BTW: I sometimes had the effect that an IPSec tunnel was connected to Sao Paulo2 (confirmed in tunnel logs) and ip.zscaler.com showed “Your are connected to Sao4”. I think this was special for Sao… but could perhaps happen for other DCs. This caused lots of discussions.
I know for sure, that if you are behind a GRE or IPSec that connects to MUC1 (edit: and it is configured to send any Zscaler Hub IP Addresses except pac server IP into the tunnel) - and you have a browser pac (pac only) or app-Profile.pac that should connect to Copenhagen, than the traffic is processed by MUC1. (Client → Edge device → IPSec (all Zscaler hub addresses) → MUC1)

Best regards
Andreas

Thomas · February 23, 2023, 7:57pm

If there is a GRE tunnel in between client and ZS cloud the CENR the tunnel is connected to (say muc1) will always take it. At least i’m not aware about any option avoid that (as long as you have a default route sending ANY destination into the tunnel).
If you do NOT route traffic to say sao4’s IP ranges into the tunnel then you can send traffic for say whatsmyip.com to sao4 and it will show up with an IP from BR.

Oucean · February 24, 2023, 2:58am

Hello Andreas

Emmmmmm…It’s really tricky. As per my point, when I access those websites, it should show my public IP as BJS1 DC’s IP. But as I tested, it’s showing BJS3 IP only.
So have you done the test you mentioned in the thread? What’s the result? Will your public IP always be TYO4 DC? I don’t have access to my company’s Zscaler Dashboard and I can’t do any test. And there is no any tunnel in our environment, so I think there will be a little bit more simple for my network.
From Legacy Network based on TCP/IP, the traffic should go to BJS3 and then to BJS1 and then to Internet. Because from the packet capture result, it’s showing destination IP of BJS1 DC’s IP in LWF result and destination IP of BJS3 DC’s IP in Adapter result. So the packet outside my computer should be like blow.

So I’m confused why it’s showing my public IP is still BJS3. Do you know if there is any way to raise this query to Zscaler Support?

Andreas · February 24, 2023, 7:38am

Hi,

I have basically stopped being a Zscaler consultant as I moved to a new company in January. Therefore I have no longer a test environment available.
Of course you can open a ticket and ask why you are connecting to bjs1 but ip.zscaler.com shows bjs3 as processing DC. I did this for SAOx and it took quite a while to get an answer.

I do not know how multiple DCs in one location / city work. It still could be the case that they have some cluster or internal routing from a bjs1 ingress to bjs3 egress. It should not matter anyway - unless certain websites block specific DC ranges or GeoLocation does not work correctly. I saw this for some SAOx ranges that had a wrong GeoLoc and therefore some BR government websites were not accessible, as they demand a BR source IP. But this is a different topic.
Also remember that slowliness issues in China may have multiple reasons and Zscaler has a bunch of options available to mitigate. Deploying Zscaler Internet Access in China | Zscaler

Best regards
Andreas

A_Y · February 28, 2023, 7:35am

For tunnel 2.0, the re-direct to specific Towers based on Geographical location is to be placed inside the APP profile only.

Oucean · March 1, 2023, 6:35am

Hello Andres

OK. Thank you all the same. Will try if I can have a chance to reach Zscaler support…
Yup. China is always a special exception for cross-border companies. I was working with network in China for more than 10 years and I have the confidence to say I know how to deal with China Network. But the thing is my company doesn’t want to give me the privilege to make the decision. Even like I said, I don’t have access to Zscaler Dashboard. So what I can do is only to assistant…Definitely it’s a difficult scenario. Anyway…
Still please accept my sincerely thanks. Really very much appreciate for your continuous help on this topic for this long time.

Oucean · March 1, 2023, 6:38am

Yup. So I think our engineer is not making a correct configuration…