Would you like an open proxy with your firewall?
UPDATED 5/8/2014: Cisco/Meraki has updated their documentation to state that a separate firewall is needed to protect MX firewall configured in passthrough mode. However, their document still fails to state that enabling web caching in passthrough mode will make it an open web proxy.
When I started working on our datacenter project, we had to decide what hardware we will use for network requirements, including which firewall to use for our needs. Given my previous experience with Cisco PIX and reputation of Cisco in networking world, I decided to go with Cisco Meraki MX series firewall. The idea was simple. Make management of firewall as easy as possible. With cloud management features offered by Meraki, it made a lot of sense to go with brand that is well recognized. What could go wrong, right?
Well, it wasn’t until I got a call from my datacenter manager who said my external IP address seemed to be spamming others and they have been getting complaints of abuse. At first, it was easy for me to dismiss this complaint as a spoof. Primarily because I hadn’t setup any services on two of my servers that I happened to have just installed Windows Server OS on. I had applied all recent updates and left them at that. Eventually I will have servers running Lync server Edge role among other services which need public IP addresses assigned directly to them. Due to this requirement, I had decided to configure Meraki firewall in passthrough mode. Since I also wanted to reduce bandwidth usage where possible, I enabled web caching on Meraki as well.
There is usually a detriment to write off a warning such as this and I happened to have fell for just that! That didn’t last very long as more reports of abuse rolled in and datacenter folks reached out to me again. This time they again repeated that traffic appeared to be coming from IP address I had assigned to firewall. It did not make any sense because since firewall was in passthrough mode, servers must use their own public IP address if they wanted to connect to internet. I should be seeing a server IP if they were infected with malware. And why would firewall send traffic on its own if servers weren’t sending it? This just got bizarre. I had not seen anything like this before.
No matter how confident, I decided to shut down my servers leaving nothing to chance and started looking at traffic patterns systematically. Since I had only two servers, not configured with anything specific just yet, it was easy to shut them down. No production workloads, no users effected, easy! Now, I only had firewall up, nothing else. Well, network switches ofcourse but they had no IP that can communicate to outside since there was no NAT device in the environment.
To my surprise, datacenter folks still saw outbound traffic! I had them capture traffic from their side, outside of my firewall. When I looked at the traffic capture in wireshark, I saw unknown IP addresses, making web requests and sending emails to different recipients using SMTP! It just didn’t make sense. It looked as if the firewall itself was acting as a proxy. A wide open public proxy infact! How could a firewall do that without any clients connected to it? How can it generate traffic on its own?
It certainly was time to get some answers. A call to Cisco Meraki was in order. Customer service was quick to respond. I got on the phone with a support engineer and explained the behavior. He was just as surprised as I was and mentioned he had never seen such bizarre behavior before. After looking around a little bit, we had ruled out VPN as no VPN had been configured. There wasn’t much configured in terms of options because it was passthrough configuration. The only item that stood out was web caching. After a brief hold, engineer came back beaming instructions to disable web cache. And I obliged. We were keeping eye on bandwidth usage all the while which was pushing north of 12Mb/s. That was a lot of traffic with no clients connected to firewall. As we disabled web caching, I saw the bandwidth graph fall off the cliff. It went from more than 12Mb/s to almost zero! We had just found our culprit!
It now was time for Cisco to come clean. I was told that since we didn’t collect any data before switching web proxy off, the logs were effectively lost. However, SE offered to get back to me after some research, which I agreed to. After few hours, I received an email with verbatim I am going to include below:
Because the MX was in pass-through mode with HTTP Content Caching enabled, the Caching engine was exposed to the outside internet effectively acting as an open web proxy, which was discovered by a 3rd party and utilized as such. Our documentation recommends that any MX device in passthrough mode be placed behind another NAT/Firewall device in order to prevent scenarios like this one. I apologize for any confusion there may have been in this regard.
So according to Cisco, by putting the firewall in passthrough mode and not protecting with YET ANOTHER firewall, I had made my firewall an open proxy that anyone in the world can connect to and do anything they wish on web!
Now, it is one thing if I had not read Meraki documentation but I am a believer of RTFM! I actually did read their documentation before implementing what I did and their documentation has NO mention of such behavior that I can find. Don’t believe me? Take a look: https://docs.meraki.com/pages/viewpage.action?pageId=15728727. So, despite claims by SE that their documentation recommends that an MX in passthrough mode be protected by another firewall device, their documentation falls short. The net net is someone like me who falls victim to an open proxy that they happened to have baked into their product. I can’t wait to explain my bandwidth bill to my management at end of first month with no productivity from the environment at all. I am so looking forward to explaining why we were almost blacklisted by many ISPs with no servers working! Thanks Cisco!