Installing TLS / SSL ROOT Certificates to non-standard environments

Over 90% of websites now use TLS encryption (HTTPS) as the access method. Enterprises utilise TLS inspection for Advanced Threat Protection, Access controls, Visibility, and Data-Loss Prevention. Zscaler App is deployed on Windows and Mac devices and the Zscaler certificate is installed in the appropriate system Root Certificate Store so that the system/browser trusts the synthetic certificate generated during TLS Inspection.

However, a number of applications do not read the system certificate store – for example Python – and moreover developer tools such as Docker need to have the Root certificate installed in order for the applications which run there to trust the synthetic certificates.

This document describes techniques to deploy the Zscaler Root Certificate into these applications. It can be used as a basis to expand the certificate deployment into other applications. This document assumes you are using the Zscaler Intermediate certificate for TLS / SSL Inspection – if you are using a custom certificate for TLS / SSL Inspection, then you should replace all references to Zscaler Root with your custom Root certificate.

Select your OS from the list below:


Windows

Most applications will utilise the Windows certificate store, however many more especially those ported from Linux, start to use their own certificate stores.

GIT

From Powershell, run

git config -l

This will output

core.symlinks=false
core.autocrlf=true
core.fscache=true
color.diff=auto
color.status=auto
color.branch=auto
color.interactive=true
help.format=html
rebase.autosquash=true
http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
http.sslbackend=openssl
diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
credential.helper=manager

The CA Certificate store is identified at http.sslcainfo . This should be updated to include the Zscaler certificate by running the following command as an administrator in PowerShell which appends the Zscaler certificate to the bundle.

gc .\ZscalerRootCertificate-2048-SHA256.crt| ac $(git config --get http.sslcainfo)

Python

Python on Windows automatically includes PIP and Certifi which is the default certificate bundle for certificate validation. Similar to GIT, the bundle needs to be updated – replacing the python directory with your own

gc .\ZscalerRootCertificate-2048-SHA256.crt| ac C:\Python37\Lib\site-packages\pip\_vendor\certifi\cacert.pem


Linux

Base Operating System

Linux variants invariably use OpenSSL for their CA Trust. You can find the OpenSSL directory through the following command

openssl version -d

Which outputs

OPENSSLDIR: "/etc/pki/tls"

In this directory structure, you can add the Zscaler certificate into the certs directory by simply copying the file in.

cp ZscalerRootCertificate-2048-SHA256.crt $(openssl version -d | cut -f2 -d \")/certs

Alternatively you can place the file into the anchors directory and run the update-ca-trust command to push the certificate into the CA-Trust files. This is more effective since the CA-Trust file could be directly referenced by other applications

cp ZscalerRootCertificate-2048-SHA256.crt /etc/pki/ca-trust/source/anchors/ && update-ca-trust

Python

Python will (again) typically use it’s own CA store. You can identify the store if certifi package is installed

python -m certifi

Which will output

/usr/lib/python2.7/site-packages/certifi/cacert.pem

You can update the Zscaler certificate into this CA Store by doing the following

cat ZscalerRootCertificate-2048-SHA256.crt >> $(python -m certifi)

Similarly, you can configure system variables to point to this CA Store (or point to the OpenSSL store you’ve updated previously)

export CERT_PATH=$(python -m certifi)
export SSL_CERT_FILE=${CERT_PATH}
export REQUESTS_CA_BUNDLE=${CERT_PATH}


MacOS

Base Operating System

MacOS behaves very similar to Linux, but has it’s own configurations and directories. MacOS will mostly use the keychain, which should keep the OpenSSL CA Store in sync. However it may still be necessary to update the OpenSSL CA Store to include the Zscaler certificate.

sudo cat ZscalerRootCertificate-2048-SHA256.crt >> /usr/local/etc/openssl/cert.pem

Python

Python will (again) typically use it’s own CA store. You can identify the store if certifi package is installed

python -m certifi

Which will output

~/Library/Python/3.7/lib/python/site-packages/certifi/cacert.pem

You can update the Zscaler certificate into this CA Store by doing the following

cat ZscalerRootCertificate-2048-SHA256.crt >> $(python -m certifi)

Similarly, you can configure system variables to point to this CA Store (or point to the OpenSSL store you’ve updated previously)

export CERT_PATH=$(python -m certifi)
export SSL_CERT_FILE=${CERT_PATH}
export REQUESTS_CA_BUNDLE=${CERT_PATH}


Docker

Docker – on Windows, MacOS, and Linux, will use the OpenSSL CA Trust for it’s connections – ensure these are configured to allow Docker to download packages as you instantiate them in your Dockerfile

Once the Dockerfile is loaded and being processed, containers will make their own connections which will need to trust the Zscaler certificate. It’s therefore important to combine the above approaches to ensure your Docker container has the Zscaler certificates installed.

This example uses three files. The .env file controls whether the build is being run in production (no-Zscaler) or development (Zscaler). The docker-compose.yaml file reads the BUILD_ENV variables and passes to the Dockerfile

.env

BUILD_ENV=production

OR

BUILD_ENV=development

docker-compose.yaml

version: '3.1'

services:

dotnetconf19:
image: dockersamples/dotnetconf:19
build:
context: .
args:
- BUILD_ENV=${BUILD_ENV:-production}
- CERT_FILE=${CERT_FILE:-/etc/ssl/certs/ca-certificates.crt}
environment:
- BUILD_ENV=${BUILD_ENV:-production}
- CERT_FILE=${CERT_FILE:-/etc/ssl/certs/ca-certificates.crt}

Dockerfile

FROM mcr.microsoft.com/dotnet/core/sdk:3.0.100-preview9 AS builder

#No need to install certificates here – no Internet requests made

WORKDIR /src
COPY src/WebRequests.csproj .
RUN dotnet restore

COPY src/ .
RUN dotnet publish -c Release -o /out WebRequests.csproj

FROM mcr.microsoft.com/dotnet/core/runtime:3.0.0-preview9

#Image runs internet requests over HTTPS – Install Certs if dev environment
#Set ARG BUILD_ENV default = production
ARG BUILD_ENV=production

#Assign the $BUILD_ENV the BUILD_ENV ENV so that it can be accessed
ENV BUILD_ENV $BUILD_ENV
#Add the CA Certificate to the container
ADD src/ZscalerRootCertificate-2048-SHA256.crt /tmp/ZscalerRootCertificate-2048-SHA256.crt
#Use BUILD_ENV variable within the container to copy the CA certificate into the certificate directory and update
RUN if [ "$BUILD_ENV" = "production" ] ; then echo "production env"; else echo "non-production env: BUILD_ENV"; CERT_DIR=(openssl version -d | cut -f2 -d \")/certs ; cp /tmp/ZscalerRootCertificate-2048-SHA256.crt $CERT_DIR ; update-ca-certificates ; fi

#Continue the build where the HTTPS Connections are made
WORKDIR /app
ENTRYPOINT ["dotnet", "WebRequests.dll"]
ENV DotNetBot:Message="docker4theEdge!"

COPY --from=builder /out/ .

16 Likes

If I could give more than one thumbs up I would :slight_smile:

5 Likes

Great post!

One additional thing to mention.

On Debian based Linux systems, to add CAs you may need to do the following:

  1. cp filename.crt /usr/local/share/ca-certificates
  2. sudo update-ca-certificates --fresh
1 Like

Thanks for documenting this. I talk about this all of the time with customers as an issue they face with the developers but it is great that you not only talked about the issue, but showed people how to solve it. Thank you.

This is great. One of my prospects was looking for it today and here we have.
Is this something shareable outside Zscaler?

Thanks
Brijita

Sure, you can share it. Nothing confidential in there,

Edit for MacOS/Python it’s certifi, not certify.

**>>>** python -m certifi
/usr/local/lib/python3.7/site-packages/certifi/cacert.pem

I also found this command was necessary for PIP to work properly:

pip config set global.cert <your private cert file>

However, where I’m still stuck, when I use PIP the scripts within may call urllib2 and there doesn’t seem to be a way to bypass cert verification without editing the source code of the package installer and setting the context. Thoughts?

1 Like