888-804-1717

Internet Edge: Security Tool Insertion

Author
Peter Welcher
Architect, Operations Technical Advisor

Written by Peter Welcher and Fausat Ogunsanya

This is the ninth blog in an Internet Edge series.

Links to prior blogs in the series:

 

This blog is loosely based on some joint work NetCraftsmen has being doing with a customer. Real-world stuff!

My thanks to my colleague Fausat Ogunsanya for providing the background info and rationale behind this blog and checking my explanations. She’s the security expert, and I’m the design aficionado who asked about the design supporting the security tools. Any errors are probably mine.

The Encrypted Traffic Problem

Here’s the driving factor: With the increase in the malware propagation via encrypted traffic and the increase in the need to protect user’s transactions on the Internet, many websites, including popular search engines, have deployed SSL encryption. The transmission of encrypted traffic through the network makes it difficult for security tools to identify, analyze and protect the infrastructure network from malicious content.

In the past, most security tools and firewalls inspection of encrypted traffic involves analysis of traffic patterns and comparing of IP addresses in the packet headers against known bad players. Deep packet SSL inspection goes beyond traffic pattern and header IP address inspection. This article is focused on deep packet SSL inspection.

Deep packet SSL inspection tools can see both the header IP, traffic pattern, and the traffic payload. This is achieved by the tool being a “man-in-the-middle” between the source and destination hosts.

More and more network traffic is encrypted. Encrypted traffic is a big problem for security tools that need to examine traffic, whether for correct protocol behavior or for inspection of the traffic contents.

What has changed design-wise is the following: In most network environments, security appliances/tools are deployed serially, with traffic flowing from one device to another device in the path enforcing the organization’s security policies. Each security tool/device in the path performs a specific security function and hence applies security policy based on their function.  As an example, there is might be a Web Application Firewall (WAF) deployed in front of the firewall.

If you have several security tools doing different traffic-based analysis or inspection, setting them up as SSL proxies with appropriate certificates, etc., can be painful.

The alternative? Well, one alternative is to use another device as a central SSL decryption point which sends decrypted traffic to multiple tools. For this device to decrypt SSL outbound traffic from the enterprise to remove a destination, all browsers on the network must trust and add Certificate Authority (CA) certificate installed on the device to their trusted certificate store.

The SSL decryption devices will maintain two sessions for each SSL session – one session between itself and the traffic initiator(client) and the second session between itself and destination server, hence the term acting as SSL Man-In-The-Middle (MITM). Copies of decrypted traffic are sent to a set of other security tools. That potentially simplifies the certificate management issues that otherwise arise. It does mean the MITM devices need sufficient capacity to handle the aggregate flows the other tools need as input. But centralizing the decryption function also may help you centrally prioritize what you inspect and help you manage not decrypting sensitive traffic.

The following diagram shows this option, with the Gigamon (highlighted in orange!) being used in the role of the MITM device.

(There may well be a switch involved between the inside and outside firewalls, with VLANs putting the Gigamon virtually inline. Or it may be cabled as shown. I’m more interested in the big picture here rather than the details.)

If you *really* want to heavily use the MITM (Gigamon) device, you can even run the inside network to inside firewall connection through it. That’s a design choice, depending on how you intend to use the Gigamon (specifically, does it need to see packets prior to inside FW or not). I chose not to show that to keep this discussion a little simpler.

Maybe all of what we’ve just covered is obvious; maybe not. I thought it worth noting explicitly as a new design meme, hence this blog. If you already knew this, fine, you get an A+ and can skip the rest of this blog!

I haven’t seen or heard of other customers doing anything like this (yet) – but NetCraftsmen is big enough that I might not have heard about it.

For the function the Gigamon is doing (copying packets from various sources to various destinations), we will use the term I first heard a couple of years ago for this: Network Packet Broker (“NPB”).

How SSL Proxy Decryption Works

I like to understand flows and the big picture of how things work. If you have that, then you can usually figure out how to troubleshoot them. In this case, there’s some cleverness involved in how firewalls and other devices can do SSL decryption for outbound HTTPS and other traffic.

The process is explained well in two references, listed below. The following is my description of how this works and my diagrams. See also the references, e.g., for simpler basic diagrams.

We need a name for the “MITM” (Man In The Middle) box doing the decryption and re-encryption. Web or SSL proxy works. In this particular case, it was a Gigamon NPB. I’ll refer to this device as the MITM for conciseness below.

Please note that MITM describes the positioning. I am not implying the NPB device is malicious or hacking the traffic or anything like that.

The user starts the SSL proxy process by pointing their web browser at an URL starting with HTTPS or by performing some other activity that requires SSL encryption.

The MITM box sees the SSL request, makes a copy, and forwards the copy onto the Internet or other path next hop. To the destination website, that copy makes it look like the MITM wants to initiate an SSL session with the destination.

Then the destination replies, with a reply signed by its certificate.

The MITM receives that and forwards a copy back to the originator but signed with the MITM’s certificate.

The result is that you have SSL running from the originator to the MITM and a different SSL session between the MITM and the destination website.

Gigamon encrypts and decrypts traffic between the 2 TLS sessions (Gigamon-2-client and Gigamon-2webserver)

A copy of the decrypted traffic is sent to inline and non-inline tools.

Example: Leveraging FireEye

Leveraging an inline tool such as FireEye is shown in the following diagram.

Caveat

Note that some (all?) security tools may well be happier if you send them both packets and replies, i.e., both sides of a session conversation.

With an NPB device like a Gigamon, that may require a smidge of configuration. I see that as just one more configuration factor to consider, one that is easily forgotten.

One Possible Pain Point

Security and network teams don’t always talk to each other as much as they might. Hey, we’re all overly busy, and there’s no time, plus that other team is a bit weird, or rude, or hostile, … or not. Friction happens.

Note that the economics of higher-speed links in the datacenter has worked very well so that depending on how you connect a device, the optical or “twinax” connection costs for 10, 25, 100, and 400 Gbps may not be all that different – the cost does not scale linearly! You’ll still be paying for the higher speed ports in the router or switch or other device, but that too has become fairly cost-effective.

For “process switching” devices, like some firewall / IPS and other security functions, the same economics may not apply.

So be sure to communicate plans, especially about budgeted devices and capacity, “speeds and feeds.” Well in advance, too!

For example, if you put a 10 Gbps MITM device in between devices that can do N x 40 Gbps, you’ve created a bottleneck and also will have to be rather careful not to overload the MITM device.

Said differently, with MITM and security devices in general, care has to be taken with planning and budgets to keep the planned link speeds in synch with the networking/datacenter team’s plans.

References

I found two fairly readable descriptions of how the “SSL Magic” works:

I’m sure there are more out there. The diagrams in these two were what I wanted for review/reference purposes.

Topics of note in these references (outside the intended scope of this blog):

  • Security use cases
  • What NOT to decrypt (privacy), and setting priorities
  • Alternative ways to decrypt selected traffic

Conclusion

Decrypting SSL on multiple devices may be “plumb silly.” Instead, consider centralizing the SSL decryption MITM function on a Network Packet Broker.

 

Disclosure statement

Related Topics

Contact Us
To get a tailored solution.