FTP or FTPS Working under Zscaler Private Access

Charles · October 14, 2022, 10:21pm

Has anyone gotten FTP or FTPS working successfully under Zscaler? I have read some of the user comments concerning FTP(S) and it seems the only true way to get it to work is bypassing Zscaler and letting the underlying network process the request. Unfortunately my company wants to get rid of the underlying network that my 10.x.x.x type traffic is using and only use Zscaler. I have had several troubleshooting sessions with Zscaler support and we are still at a loss for how to get this to work. If some account has this working can you message me and let me know how you got it to work. Again, I can bypass Zscaler for now, but that is not going to be a long term option.

I know what the problem is, but I am trying to get my company’s Zscaler support team to make changes to test my theory, but that is taking time.

Here is the issue in a nutshell. When doing FTP(S), the control connection works fine. Even when doing FTPS, the control connection works and the TLS Handshake is also successful.
The problem is the Data Connection. FTP(S) starts a new connection for the data. This is where the problem starts. When using Zscaler the App Connectors both try to write to the same data control port. This will not work. According to my company’s Zscaler support team, there are two App Connectors for redundancy. I have explained only one of the App Connectors works to setup the Control Connection of FTP(S), so why do both App Connectors try to work with data connection. So I have been trying to get my company’s Zscaler team to setup a Zscaler policy with only one App Connector to eliminate the second App Connector.

While I am working with my Zscaler support team, I wanted to know if any other company was using FTP(S) through Zscaler’s Private Access route? This is usually your 10.x.x.x traffic. The non 10.x.x.x traffic works fine since it is bypassing Zscaler. Thanks in advance for any feedback.

mryan · October 17, 2022, 8:04am

There is a help article here Supporting FTP Applications | Zscaler which details configuring FTP through ZPA.
To provide a bit more detail - ZPA will, by default, perform a healthcheck on an application port to check reachability and to provide path selection for the user transaction. You should also be aware that in a connector group (whether 2 or 10 connectors), one connector will perform a healthcheck, and the client connection is then load balanced across the other connectors in the group (you may get lucky and the same connector is used for the healthcheck and the data connection, but it’s never guaranteed). Additionally where you have a single application segment with multiple FQDN’s and IP addresses in it, a single users connection to any of the FQDN’s and IP’s will be anchored to the same App Connector - this is to ensure session state for applications with multiple connections to multiple IP’s.

I’ll provide some clarity on how FTP functions with ZPA/ZTNA. There are two modes for FTP to function.

Active mode has the server initiate a data connection back to the client - this will not work with ZPA since the user is not placed on the network in a Zero Trust environment, and there is therefore no path back to the client.
Passive mode, the server tells the client which ephemeral port to connect to for the data connection. It’s therefore imperative to use passive mode for data transfer in ZPA.

The server data port is ephemeral, it is only available for a single TCP connection and then the port is torn down.
The server returns its IP address and the TCP port to connect to in the data packet
This is why disabling healthcheck on the application segment is important for FTP to function. The healthcheck consumes the port, and therefore the port is no longer available for the true data connection from the client.

So - PASV FTP returns the IP address and PORT to connect to in the data connection. To configure FTP for server ftp.company.com which resolves to 192.168.1.1, you would create an application segment :-

Name - FTP Server
Application Names - ftp.company.com, 192.168.1.1
PORTs - 21, 1024-65535 (or whatever the FTP server has been configured for as listening ports)
Healthcheck - Never
ServerGroup - AppConnector Groups containing 2 or more connectors

You cannot get away from the IP address requirement here, since the servers real IP address is provided in the data packet. You can, however, configure the single IP address and appropriate ports in the application segment along with the FQDN. The user will only ever connect to the FQDN, but the data transfer “under the hood” will be to the IP address.

Charles · October 19, 2022, 5:11am

Thanks for the reply. We are getting closer. I had my Zscaler support team implement your Passive suggestion by setting “Healthcheck – Never”. We are still not getting a data connection established, but I think it might be with some other trouble shooting suggestions we had previously done. We noticed early on that the second App Connector was also trying to establish a data connection on the same port the first App Connector had initiated the data connection. Since there were two different IP’s going to the same port it was failing. So the original suggestion we got from the Zscaler Vendor was to create a Source IP FW rule that would give both App Connectors the same IP when going to certain IP’s and ports. But it still failed, because the second App Connector (even with the same address as first App Connector) was still trying to go to the same data port. I think this is the Health check.

What we will test next is to use a connection that doesn’t have the Source IP FW rule and turn off Health Check on that connection. One thing that is different from your suggestion is we are using an IP range. Here is a facsimile of our setup

• Name - Healthcare
• Application Names – 10.238.8.x
• PORTs - 21, 1024-65535 (or whatever the FTP server has been configured for as listening ports)
• Healthcheck - Never
• ServerGroup - AppConnector Groups containing 2 or more connectors
So is there a problem using a range or do we have to have a specific IP? That was what we were not clear about.

mryan · October 19, 2022, 8:54am

The range is not a problem. As long as you have the IP address of the server and the port ranges in the same segment, and have disabled health-check, then this should work. Make sure when you connect to the FTP server you’re using passive mode for data transfer.

A PCAP often helps understand what’s going on. See below showing my client (using Filezilla) connecting to the FTP server 192.168.1.2. You see the QUOTE PASV being sent and the port information returned. The data connection is then established to the server on 60781, where the file is then transferred.

Charles · October 20, 2022, 9:36pm

Thanks for your WireShark display, I can see you are working. You know for sure you are using ZPA? We can bypass ZPA and use the Internet or underlying network and everything works.

Here is my trace so you can see the difference in the two scenario’s. One under ZPA and one without Zscaler.

Under ZPA we can logon to the remote site with no problems and even get past the Secure Handshake. The problem always happen when we start the data channel. Packet #23 is the correct SYN we expect to see to initiate a connection to our data port (61,000 – 61,099). It is using 61063 in this example. What happens is the 2nd App Connector comes in on packet #24 going to the same data port and the mainframe rejects that attempt and the connection fails.

FTP Client Displays:

From looking at the client FTP displays, you can see a passive connection comes in to port 61063 initially and then when the 2nd App Connector comes in, the mainframe will reject it because it was expecting a TLS Client Hello for the data port. At first we thought it was just a TLS issue, but this same double connection comes in on a non-secure connection.

WinSock 2.0

Mem – 16,139,652 KB, Virt – 137,438,953,344 KB

220-- By using this system, you understand and consent to the following:

220- The Government may monitor, record, and audit your system usage,*

220-including usage of personal devices and email systems for official

220-duties or to conduct HHS business. Therefore, you have no reasonable

220-expectation of privacy regarding any communication or data transiting

220-or stored on this system. At any time, and for any lawful Government

220-purpose, the government may monitor, intercept, and search and seize

220-any communication or data transiting or stored on this system.

220- Any communication or data transiting or stored on this system may be*

220-disclosed or used for any lawful Government purpose.

220 Connection will close if idle for more than 5 minutes.

AUTH TLS

234 Security environment established - ready for negotiation

TLSv1.2, cipher TLSv1/SSLv3 (ECDHE-RSA-AES256-SHA384) - 256 bit

USER zzdmtb

331 Send password please.

**PASS ************

230 ZZDMTB is logged on. Working directory is “ZZDMTB.”.

SYST

215 MVS is the operating system of this server. FTP Server is running on z/OS.

Keep alive off…

CWD ‘ZZDMTB.’

250 “ZZDMTB.” is the working directory name prefix.

PBSZ 0

200 Protection buffer size accepted

PROT P

200 Data connection protection set to private

PASV

227 Entering Passive Mode (10,5,100,84,238,135)

LIST

Connect socket #1648 to 10.238.8.53, port 61063…

SSL/TLS error - 0, SSL error - 5, error:00000005:lib(0):func(0):DH lib

SSL failed

Connection terminated

Error loading directory…

So do you think my Zscaler support team has not turned off Health Check? Does it require some kind of recycle for it to go into effect?

Here is the working copy. I picked up at the beginning of the opening of the data port, packet #57. In this case it is 61002. The non Zscaler device does not have the double connection to port 61002. And everything comes up :

Here is the FTP Client logs showing it connected successfully and also using PASV as highlighted.

20-- By using this system, you understand and consent to the following:
220-* The Government may monitor, record, and audit your system usage,
220-including usage of personal devices and email systems for official
220-duties or to conduct HHS business. Therefore, you have no reasonable
220-expectation of privacy regarding any communication or data transiting
220-or stored on this system. At any time, and for any lawful Government
220-purpose, the government may monitor, intercept, and search and seize
220-any communication or data transiting or stored on this system.
220-* Any communication or data transiting or stored on this system may be
220-disclosed or used for any lawful Government purpose.
220 Connection will close if idle for more than 5 minutes.
AUTH TLS
234 Security environment established - ready for negotiation
TLSv1.2, cipher TLSv1/SSLv3 (ECDHE-RSA-AES256-SHA384) - 256 bit
USER zzdmtb
331 Send password please.
PASS **********
230 ZZDMTB is logged on. Working directory is “ZZDMTB.”.
SYST
215 MVS is the operating system of this server. FTP Server is running on z/OS.
Keep alive off…
CWD ‘ZZDMTB.’
250 “ZZDMTB.” is the working directory name prefix.
PBSZ 0
200 Protection buffer size accepted
PROT P
200 Data connection protection set to private
PASV
227 Entering Passive Mode (10,5,100,84,238,74)
LIST
Connect socket #1664 to 10.238.8.53, port 61002…
TLSv1.2, cipher TLSv1/SSLv3 (ECDHE-RSA-AES256-SHA384) - 256 bit
125 List started OK
250 List completed successfully.
Transferred 6,294 bytes in 0.563 seconds

Another thing I noticed about your connection, you are not going through a NAT FW as I am. I need to use the NAT address (10.238.8.53) in both cases to connect to the mainframe. Since it is getting through the control connection with no issues, I don’t see how NAT can be an issue.

I am still waiting for my Zscaler Support Team to turn off Health Check on another company we are having the same issues with. I want to see if it might work with that company since we are not using the Source IP Policy to give each App Connector the same IP address for this company.

Charles · October 20, 2022, 9:37pm

Here is the working WireShark. I was not allowed to put in the previous post. Only one image is allowed:

mryan · October 21, 2022, 9:14am

Hi Charles,

I think support is the right place to troubleshoot this. If you can provide the case number and advise the account details with support access enabled, we can take a look.
I’m interested in the App Connector pool and NAT. I would imagine that with SNAT occurring for two app connectors, there is likely a TCP SourcePort change which might affect the FTP connection. I’m not 100% sure without seeing PCAP’s from the serverside.
There should not be a change from one connector to another in a connector group if the segment is defined and includes both the data and control connection. Healthcheck must be disabled to prevent the H/C consuming the TCP port for the Passive connection. The PCAP serverside (i.e. from the App Connectors) would help understand what is occuring here.
You can use the support function in the Admin UI to take a PCAP from both connectors at the same time. You can set a filter for the destination of the FTP server. Start the PCAP, generate the traffic, and then stop/download the PCAP from the admin interface. That way you can correlate the client side connection with the server side connection and see what is going on. There could be a firewall/NAT state problem occurring also (if the app connector is changing, this could likely be the cause).

Charles · October 25, 2022, 8:27pm

Thank you so much. I went back to my Zscaler team and had them backout all the non-working fixes we tried and I had them follow your suggestions. Once we did that and disabled health check everything started working. So we are good.

I think are main issue besides the health check was the lan segment definition. We created an application segment that was only for the mainframe instead of having multiple lan segments within the mainframe application segment. So we are working great now. Thanks for all your help.