Tunneling corporate firewalls for developers

When you have a project, online service or WebApp that you manage and deploy, you usually have something that you SSH into. It maybe a real server, a VPS, a container, a Kubernetes node and what have you.

...unless your project is purely serverless and built with a bunch of (Insert Random Letter) as a Service bricks.

Being able to setup a connection you trust and where your dev tools work is important. How you connect to the server where you deploy your projects isn’t always straight forward though, when there are proxies, packet sniffing firewalls and network monitoring in-between you, the internet and the target server.

The "correct" answer is: setup a VPN! But that's sometimes not possible. Endpoint management may forbid it client-side. Server-side infrastructure may be managed by a third party, kicking off costly service requests.

Ultimately, this is what the article is about - how to SSH into machines, when there is stuff in the way preventing that and make sure that your tools like git, scp, rsync or editing files directly on the server via VSCode’s SSH integration work, with no new software installed and the absolute minimum of modifications to your server.

I find it fascinating what seemingly simple tools can do if you look closely. Did you know Git for Windows comes with tunneling software? How do these tools interact with network security on a per-packet basis? That’s what I will investigate along the way.

Tunneling - So many flavors #

If you control both Source and Destination, then you can tunnel everything through anything in complete secrecy and ultimately there is nothing anyone can do about it. This shouldn’t be news to anyone working with networks. There are countless articles and videos going over a multitude of tunneling combinations.

• Internet and SSH over ICMP (Pings)
• Internet and SSH over DNS requests
• Internet and SSH over WebSockets

As for this article, we’ll deep-dive ✨SSH over HTTP(S)✨. Be it Linux, Mac or Windows, we will look at how to setup everything up, what the underlying network traffic looks like and most importantly: how your digital infrastructure is already capable of all this … even if it wasn’t supposed to.

Basic SSH Connection Scenarios #

We’ll go through all the ways you may SSH into your server, with increasing levels of filtering, monitoring and connection blocking. Our context is web development, so how your main WebApps are reached is covered as well. Here, let’s assume your WebApps are served or reverse proxied with Nginx, Caddy or Apache / httpd.

In modern web deployments, your service may sit behind an application gateway, potentially with multiple micro-services at play. We are going to simplify and consider no such factors in this article.

Simple, direct connection #

Let’s start with a classic default setup: SSHD, the SSH daemon of OpenSSH is listening on Port 22, your WebApp is accessed via HTTP and HTTPS on Port 80 and 443 respectively. Server-side, these ports are port-forwarded and accessible by anyone via a static IP address or a domain name.

You have the basics of SSH security: fail2ban to prevent password brute forcing and/or configured SSHD to reject logins via password and only allow key based authentication. Whilst security through obscurity is bad, we don’t have to make it too obvious for random port scans and SSHD on ports like 59274 is just as valid.

This “direct” connection also covers the case, that any intermediate corporate proxy has whitelisted this connection to be direct, is part of a company internal subnet not going through a proxy to the outside or that the target is within your company VPN.

Network capture #

Let’s take a look at what happens inside the network. All captures are performed with wireshark. Source 💻 is a Laptop attempting ssh user@example.com. Target 🌍 is the server with port 22 open. The capture concerns just this specific TCP connection. As there is no intermediary yet, the capture is performed on Source 💻.

Each individual packet’s Direction is determined by the Source and Destination IP address, Protocol is judged by wireshark based on packet contents and connection history, Length is the packet size in bytes and Info is wireshark’s quick summary of what the packet is or does. IPs and ports are left our for brevity.

Rows with 💻 → 🌍 mean outgoing packets, aka Source → Target. Rows with 🌍 → 💻 and a darker background indicate incoming packets, aka Target → Source.

[SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM

[SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1460 SACK_PERM WS=128

[ACK] Seq=1 Ack=1 Win=131328 Len=0

Protocol (SSH-2.0-OpenSSH_for_Windows_9.5)

[ACK] Seq=1 Ack=34 Win=64256 Len=0

Protocol (SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.9)

Key Exchange Init

Key Exchange Init

Elliptic Curve Diffie-Hellman Key Exchange Init

[ACK] Seq=1098 Ack=1514 Win=64128 Len=0

Elliptic Curve Diffie-Hellman Key Exchange Reply, New Keys, Encrypted packet (len=228)

New Keys

[ACK] Seq=1606 Ack=1530 Win=64128 Len=0

Encrypted packet (len=44)

[ACK] Seq=1606 Ack=1574 Win=64128 Len=0

Encrypted packet (len=44)

Encrypted packet (len=60)

[ACK] Seq=1650 Ack=1634 Win=64128 Len=0

Encrypted packet (len=44)

Encrypted packet (len=500)

The first 3 packets are the standard TCP handshake. After that SSH begins its authentication, key change and encryption setup. You can kinda imply this to be SSH, as the communication happens on port 22. More obviously, anyone looking at the traffic can easily clock this as SSH, as the setup phase loudly proclaims to be SSH.

Even after the communication became fully encrypted, we can still infer this communication to be SSH, as this is still one specific connection, that we know previously talked with a “SSH smell”. Inferring the connection type by looking at the packets and connection history is what’s known as packet-sniffing.

Blocked by a “dumb” firewall #

Click the image for fullscreen, or finger zoom on mobile. The illustrations will get wider and wider going forward.

A “dumb” firewall, which performs no packet sniffing, is unable to block SSH specifically. These firewalls control which type (UDP, TCP, etc.) of packet can go from and to which port, address or application. This applies to both stateless firewalls and stateful firewalls, a distinction which we’ll ignore going forward.

Of course, no such thing as "dumb" and "smart", it's all about rule-sets and policies. Specific security products, specific firewall configs, placement in the OSI Model, is a can of worms this article will not touch.

For now we look at client-side firewalls only. A popular “set and forget” firewall ruleset to allow internet access but block users from doing other stuff is to only permit outbound TCP Port 80 for HTTP, TCP Port 443 for HTTPS and block everything else. We are ignoring things like DNS, E-Mail, etc. here.

Our default SSH connection attempt will error out in such an environment. Provided the domain itself isn’t blacklisted, DNS resolves correctly and the OS isn’t aware of the firewall rule giving you a ssh: connect to host example.com port 22: Permission denied, you’ll get this timeout error after a waiting period:

$ ssh user@example.com
ssh: connect to host example.com port 22: Connection timed out

Network capture #

Source 💻 is a Laptop attempting ssh user@example.com. Target 🌍 is the server with port 22 open. Here is what traffic looks like if a firewall is blocking the outgoing communication via a simple port block. Capture is again on Source 💻.

[SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 WS=128

[TCP Retransmission] [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 WS=128

As you might have expected, SSH sends a TCP Handshake SYN, which never reaches the server, prompting a bunch of [TCP Retransmission] before giving up with a timeout error.

SSH on port 443 #

Let’s start with the most obvious workaround for our “dumb” firewall case: Set the SSH port to 443. As two services cannot share the same port, we must either disable our HTTP and HTTPS service or set one of them to another port.

This will get us past our dumb firewall case, but will obviously prevent our WebApp from being reachable. Except for the port number, the network capture will look identical to the direct connection case and any packet sniffer will still be able to tell that this is an SSH connection.

SSLH: HTTP and SSH on the same port #

One popular solution to this is the connection multiplexer SSLH. It’s usually setup to listen on port 443, checks whether the incoming connection is SSH or HTTPS and establishes the connection to SSHD or your HTTP service respectively. SSLH can do more, but that’s the gist of it.

To be clear, the connections themselves are not mixed protocol and the clients need no modification. SSLH only determines where the connection is established to.

From the perspective of the client, nothing changed as compared to a direct connection. However, it requires a new piece of software to sit in front of your entire server-side communication stack. One more thing to maintain, one more thing to fail. Still subject to be blocked by corporate proxies and packet sniffing firewalls.

Blocked by a “smart” firewall #

A “smart” firewall can additionally look inside packets and block connections based on what traffic it sees. When faced with a smart firewall setup to block either SSH specifically or only allow “normal” web traffic, you will get a timeout error like this:

$ ssh user@example.com
kex_exchange_identification: read: Connection timed out
banner exchange: Connection to example.com port 22: Connection timed out

Network capture #

Source 💻 is a Laptop attempting ssh user@example.com. Target 🌍 is the server with port 22 open. Here is what traffic look like if it’s not going through and being filtered by a smart firewall, which inspects traffic. Capture performed on Source 💻 Laptop.

[SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM

[SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1452 SACK_PERM WS=128

[ACK] Seq=1 Ack=1 Win=132096 Len=0

[PSH, ACK] Seq=1 Ack=1 Win=132096 Len=33, Payload: SSH-2.0-OpenSSH_for_Windows_9.5

[TCP Retransmission] [PSH, ACK] Seq=1 Ack=1 Win=132096 Len=33, Payload: SSH-2.0-OpenSSH_for_Windows_9.5

[RST, ACK] Seq=34 Ack=1 Win=0 Len=0

Wireshark wasn't smart enough to label the failed banner exchange packets as SSH, even though the packets clearly indicate it was. I changed it manually in this section for completeness.

The target successfully answers our call for a TCP connection, but never responds to our requests, before our clients gives up with the RST signal. In the case of TCP connections, “smart” firewalls allow the initial connection to take place, look at what comes next and judge based on that.

After initial TCP connection, the next thing to be sent is the SSH identification string, something that unequivocally flags our connection as SSH, which the firewall blocks. That’s the reason SSH doesn’t timeout outright, but rather with this specific banner exchange timeout. The identification strings never make it to the other side.

Let’s reverse our perspective and take a look at server-side. Source 🖥️ is the server with SSHD on port 22 and Target 🌍 is a Laptop attempting ssh user@example.com, the very same connection attempt as above. Capture performed on Source 🖥️ Server, which hosts SSHD.

[SYN] Seq=0 Win=64240 Len=0 MSS=1452 WS=256 SACK_PERM

[SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1460 SACK_PERM WS=128

[ACK] Seq=1 Ack=1 Win=132096 Len=0

[PSH, ACK] Seq=1 Ack=1 Win=64256 Len=21, Payload: SSH-2.0-OpenSSH_9.9

[TCP Retransmission] [PSH, ACK] Seq=1 Ack=1 Win=64256 Len=21, Payload: SSH-2.0-OpenSSH_9.9

[FIN, ACK] Seq=22 Ack=1 Win=64256 Len=0

Same deal here. We observe the initial TCP connection being established, but with a sudden stop of communication, followed by a slow whimper of server-side identification string retransmission call-outs into the void.

You may see server-side Not allowed at this time packets instead, as OpenSSH is rate-limiting by default via the MaxStartups and penalty system. Either way, these packets are never heard.

Before we get out our hammer drill to circumvent this, let’s introduce another character of today’s journey…

The corporate proxy #

A corporate proxy is an exit point to the internet, deployed for security and compliance reasons within a company to monitor for threats and forbid anything that isn’t explicitly allowed. Furthermore it’s supposed to curb data exfiltration and ensure employees don’t setup infrastructure, without clearing it with IT prior.

Far beyond any malicious hacking, employees misconfiguring things, uploading sensitive data to 3rd parties and similar faux pas are the prime reason for data leaks.

These proxies are usually part of an overarching IT and endpoint security package sold by companies like ThreatLocker, Forcepoint and Cisco, among others. In Windows land, these are usually setup by Group policies pre-configuring a system-wide proxy and in *Nix land these are usually pre-configured with an initial OS image.

Among these corporate proxies, it’s standard to have packet sniffing capabilities, preventing connections that don’t pass the sniff-test of “looks like normal internet access”. Going forward, we will circumvent this and do so in a way that makes dev tools happy. Before that, let’s clear the elephants in the room…

Probably the time to mention, that as much as any other post, my blog's disclaimer applies.

…why would you need to? Can’t you simply open a ticket at your IT department? Certain situations may make a deeper architectural solution impossible on the timescale that a project needs delivering, happenings need to happen and things need to thing.

There may be intermediary companies which are responsible for digital infrastructure, kicking-off complicated inter-contract reviews; engineer access gateways may be on unreliable subnets; or simply, the present digital infrastructure may be such a mess, that trust just can’t be established in the first place.

Deep consideration is needed on whether such a setup is actually required and whether or not this may violate existing security policies. And the previous passage was of course hyperbolic. Infrastructure decisions are made as a team. Let’s find out what’s possible, with seemingly simple, standard tooling:

Honoring the proxy #

Proxies are such a vital piece of infrastructure, that we expect the operating system’s proxy settings to be honored by default, built proxy settings into most network connected software and have additional defacto standards to specify them like the environment variables http_proxy, HTTPS_PROXY, NO_PROXY and friends.

But what may come as a surprise, is that such a fundamental piece of infrastructure like OpenSSH doesn’t support it, with the exception of SSH as a proxy itself. Unix philosophy and all that. SSH clients like Putty do, but we’ll stick with OpenSSH. It’s FOSS, other tools rely on it and it comes pre-installed on every OS by now.

With OpenSSH instead of tools like Putty we also get to use Terminals like the new Windows Terminal or GhostTTY with modern design, good font support and copy-paste that doesn't make you blow your brains out

OpenSSH supplies ProxyCommand and relies on other tools proxying for it. Many ways to do this. For now, let’s start with the simplest ones, with no extra encryption at play. There are FOSS connect.c aka ssh-connect and corkscrew, working on Linux, BSD, Mac OS and Windows.

Linux, BSD and MacOS have nc preinstalled, which can also be used. But we are ignoring it for brevity, as it's more of a general purpose tool and the windows builds are flagged by Windows Defender.

ssh-connect created by @gotoh recently moved to GitHub. Most online documentation now points to a dead bitbucket repo. On Windows specifically it comes as connect.exe, installed by default with Git for Windows and can also be installed via Sccop or MSYS.

Though not quite the same, in the context of the article corkscrew does the same. It is more well known as a project, just 260 lines of C, but in contrast to ssh-connect has no widely distributed Windows build. For x64 Windows, I have compiled it myself and here it is as a shortcut for testing: corkscrew.zip.

Baby’s first tunnel #

Both corkscrew and ssh-connect are very simple and can authenticate if the proxy uses basic auth. To confirm connection to SSHD you don’t need to configure SSH. All proxy commands are supposed to return the SSH identification string, which is how you can check if a connection is established.

Let's use 198.51.100.4 and port 8080 for our corporate proxy.

$ corkscrew 198.51.100.4 8080 example.com 22
SSH-2.0-OpenSSH_9.9

/* or */

$ connect -H 198.51.100.4:8080 example.com 22
SSH-2.0-OpenSSH_9.9

If those proxy commands successfully return the server-side SSH identification string, we are good to go and can use it with SSH. If not, you can diagnose and look for the reason at this stage, without SSH spewing errors at you. Now to use it with SSH, we can supply the ProxyCommand like:

$ ssh -o ProxyCommand="corkscrew 198.51.100.4 8080 example.com 22" user@

/* or */

$ ssh -o ProxyCommand="connect -H 198.51.100.4:8080 example.com 22" user@

This annoyingly long command is not how you use SSH. We skip proper SSH configuration until later in the article.

Using this and optionally -i for the keyfile, we can connect to our target server. Note how ProxyCommand now determines destination and port. Whilst we can use placeholders %h %p, ssh itself is not involved in matters of routing anymore, so neither -p for non-default ports, nor destination after user@ is required.

By using corkscrew or ssh-connect, we instruct the intermediate proxy to handle the connection to SSH port 22 for us and OpenSSH now gets SSH packets delivered by ProxyCommand. But why would a corporate proxy do this? How do you make HTTP talk in SSH? That’s the neat thing, you don’t…

Network capture #

Source 💻 is a Laptop performing ssh -o ProxyCommand="corkscrew 198.51.100.4 8080 example.com 22" user@. Target 🏢 is an intermediate proxy sitting in between a private subnet and the internet. The capture is performed on this intermediate proxy Target 🏢.

[SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1

[SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1460 SACK_PERM=1 WS=128

[ACK] Seq=1 Ack=1 Win=131328 Len=0

CONNECT  example.com:22 HTTP/1.0  [TCP segment of a reassembled PDU]

[ACK] Seq=1 Ack=36 Win=64256 Len=0

CONNECT  example.com:22 HTTP/1.0

[ACK] Seq=1 Ack=38 Win=64256 Len=0

HTTP/1.0 200 Connection established

Protocol (SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.9)

Protocol (SSH-2.0-OpenSSH_for_Windows_9.5)

[ACK] Seq=81 Ack=71 Win=64256 Len=0

Key Exchange Init

[TCP segment of a reassembled PDU]

[ACK] Seq=1137 Ack=1095 Win=64128 Len=0

Key Exchange Init, Diffie-Hellman Key Exchange Init

[ACK] Seq=1137 Ack=1551 Win=64128 Len=0

Diffie-Hellman Key Exchange Reply, New Keys, Encrypted packet (len=228)

New Keys, Encrypted packet (len=44)

Encrypted packet (len=44)

Encrypted packet (len=60)

Encrypted packet (len=44)

Encrypted packet (len=500)

We communicate by SSH (a TCP protocol) over the corporate proxy, which speaks HTTP (also a TCP protocol). Though that term is a bit murky, this classifies our setup as tunneling. The center point of all this is packet number 6 - ✨ HTTP CONNECT ✨

corkscrew and ssh-connect instruct the corporate proxy to connect to our target server’s SSH port and relay back what it hears via the command HTTP CONNECT. HTTP CONNECT works by relaying RAW TCP. Similar to the dumb firewall previously, HTTP CONNECT doesn’t understand what TCP it’s actually relaying.

So why not forbid HTTP CONNECT? - Because that would break proxy connections with HTTPS, basic building block of modern web. Ignoring the unencrypted SNI, the point of HTTPS is that the communication is encrypted. For the corporate proxy to work, it has to relay TCP blindly. That’s what HTTP CONNECT is for.

We will cover DPI / TLS stripping later. There are network setups which can prohibit HTTP CONNECT. In such cases tunneling has to run deeper with projects like wstunnel, but we won't be covering that here.

If you take a look at the traffic, you still see the connection being clocked as SSH. The intermediate proxy talks to the destination server example.com:22 and thus can see the unencrypted SSH setup. If the corporate proxy performs packet sniffing, it will have no issues blocking this. Let’s fix that.

So far, our tunneling method doesn't hide anything, but I've seen this setup work more often than not, even in environments where outgoing SSH was supposedly blocked.

✨ HTTPS Tunneling ✨ #

Now we get into the meat and potatoes. If we want to hide from packet inspection based filtering, we need encryption on top of our communication, so the setup phase doesn’t give away the SSH nature of our connection. Luckily there is a super convenient and universally compatible way: Just like HTTPS - TLS.

Instead of talking to SSHD directly, we let the main HTTP server NGINX, Caddy or Apache on our server handle the routing to and from SSHD. All the automated encryption, certificate handling and IP range whitelisting, without the need for an open SSH port. Implemented by, once again: ✨ HTTP CONNECT ✨

Both corporate proxies and smart firewalls trying to packet-inspect, will be met with HTTPS encryption. Just like visiting the website of a bank to conduct online financial transactions, intermediates will have no insight into the connection, and most importantly for us, no insight into what type of connection it is.

The client-side tool for this is the FOSS proxytunnel, which is available for Linux, Mac, BSD and has automated builds for Windows. Actively maintained, it just had a new release this month. proxytunnel can use windows specific NTLM authentication and fully covers previously mentioned corkscrew and ssh-connect capabilities.

It's not available on Scoop, though I'm hoping to change that with this Pull Request

proxytunnel instructs our HTTP server to HTTP CONNECT us to our server’s SSHD. With no intermediate proxy, something corkscrew and ssh-connect can as well! But proxytunnel can chain mutliple HTTP CONNECT if there is a corporate proxy in-between and most importantly: with HTTPS (TLS) encryption on top.

But first, our HTTP server has to play along…

Server-side setup #

The neat thing is, we can utilize the very same HTTP server our WebApp is most likely served or reverse proxied over. To do so, we need to configure HTTP CONNECT, restrict it to localhost port 22 only and setup optional authentication. Luckily all the popular FOSS HTTP servers, Nginx, Caddy and Apache / httpd can do so.

No new software server-side. No port-forwarding, besides pre-existing HTTP ones. No need to touch the SSHD config either.

Due to how HTTP CONNECT works, we can setup the connection on the granularity of a domain or subdomain. So we can do ssh.example.com only, but we don’t have to and can keep it on the same level as the main WebApp example.com. The presence of HTTP CONNECT does not impact standard HTTP routing in any way.

You can slap on basic auth and/or IP whitelisting, basic HTTP config stuff I won’t describe for brevity. It’s totally fine to leave this wide open though, as that corresponds to a port-forwarded OpenSSH. Totally normal, given the security mechanisms of OpenSSH and optional hardening mentioned at the beginning.

It should go without saying: if your HTTP server goes down, your SSH connection does too, so it's probably a good idea to keep the standard port-forwarded SSH as a backup, given no other means of access.

We will be setting up on port 443 with HTTPS here, though you may perform the same setup without HTTPS, which simply means moving the setup to the non-HTTPS block in the respective config file. I will cover HTTPS only going forward and, of course, the optional basic auth without HTTPS is meaningless.

Crucially, one must take care to only allow such a connection to localhost:(SSHD Port) A config slip-up here would be catastrophic, allowing bad actors to pose as oneself.

Apache / httpd #

Apache / httpd supports this by default, nothing more than a config addition needed. The module responsible for this is mod_proxy_connect and should be precompiled and included everywhere you can get Apache to my knowledge. The addition happens on the level of a virtualHost

<VirtualHost *:443>
	ServerName example.com

	SSLEngine on
	SSLCertificateFile /etc/nixos/proxy-selfsigned.crt # Don't actually handle certs manually, just enable ACME
	SSLCertificateKeyFile /etc/nixos/proxy-selfsigned.key # Don't actually handle certs manually, just enable ACME

	## ... Main WebApp Config ... ##

	LoadModule proxy_connect_module modules/mod_proxy_connect.so # Module for HTTP_CONNECT
	ProxyRequests On # Enable it
	AllowCONNECT 22 # Restrict HTTP_CONNECT to port 22
	<Proxy *>
		Require all denied # Disable proxying for other targets
	</Proxy>
	<Proxy 127.0.0.1:22> # See the article's comments below, you may have to allow for the 'ServerName' here, instead of 127.0.0.1
		Require all granted # Enable proxying to our localhost:22
	</Proxy>
</VirtualHost>

If you fell for the NixOS Meme, then your services.httpd config should look like this.

httpd = {
  enable = true;
  virtualHosts = {
    "example.com" = {
      listen = [
        {
          ip = "*";
          port = 443;
          ssl = true;
        }
      ];
      sslServerCert = "/etc/nixos/proxy-selfsigned.crt"; # Don't actually handle certs manually, just enable ACME
      sslServerKey = "/etc/nixos/proxy-selfsigned.key"; # Don't actually handle certs manually, just enable ACME
      documentRoot = "/var/www/static";
      extraConfig = ''
        LoadModule proxy_connect_module modules/mod_proxy_connect.so
        ProxyRequests On
        AllowCONNECT 22
        <Proxy *>
            Require all denied
        </Proxy>
        <Proxy 127.0.0.1:22>
            Require all granted
        </Proxy>
      '';
    };
  };
};

Nginx #

Nginx requires an additional patch applied and module compiled to enable HTTP_CONNECT: ngx_http_proxy_connect_module. Luckily, support goes way back to Nginx 1.4 in 2013 and the patch has remained identical for every version since 1.21.1 in 2021. Build steps are minimal and described on the GitHub page.

After the module is in nginx, you can simply insert proxy_connect at the server-block level (or higher).

proxy_connect;
proxy_connect_allow 22; # restrict to port 22
proxy_connect_address 127.0.0.1; # restrict to localhost

For NixOS and its services.nginx, module inclusion should be covered by services.nginx.additionalModules, but due to a version check, nix will refuse to compile with the newest nginx. I started fixing it in this PR and will finish it after releasing this article. In the meantime, here’s my config with an override in place:

{ config, lib, pkgs, ... }:

let
  http_proxy_connect_module = pkgs.stdenv.mkDerivation {
    pname = "http_proxy_connect_module";
    version = "4f0b6c2297862148c59a0d585d6c46ccb7e58a39";
    src = pkgs.fetchFromGitHub {
      owner = "chobits";
      repo = "ngx_http_proxy_connect_module";
      rev = "4f0b6c2297862148c59a0d585d6c46ccb7e58a39";
      sha256 = "sha256-Yob2Z+a3ex3Ji6Zz8J0peOYnKpYn5PlC9KsQNcHCL9o=";
    };
    patches = [ "${pkgs.fetchFromGitHub {
      owner = "chobits";
      repo = "ngx_http_proxy_connect_module";
      rev = "4f0b6c2297862148c59a0d585d6c46ccb7e58a39";
      sha256 = "sha256-Yob2Z+a3ex3Ji6Zz8J0peOYnKpYn5PlC9KsQNcHCL9o=";
    }}/patch/proxy_connect_rewrite_102101.patch" ];

    meta = {
      license = [ lib.licenses.bsd2 ];
      description = "Forward proxy module for handling CONNECT requests";
      homepage = "https://github.com/chobits/ngx_http_proxy_connect_module";
    };
  };
in
{
	services = {
		
		# Rest of the config

	    nginx = {
	      enable = true;

		  # Rest of the config

	      additionalModules = [ http_proxy_connect_module ];

	      virtualHosts = {
	        "example.com" =  {

	          # Rest of the config

	          extraConfig = ''
	            proxy_connect;
	            proxy_connect_allow 22;
	            proxy_connect_address 127.0.0.1;
	          '';
	        };

Caddy #

Caddy v2 gained HTTP_CONNECT capability last year with an update to module forwardproxy. Compiling and module inclusion is dead simple. In the caddyfile, you can insert the HTTP CONNECT config.

https://example.com {
        forward_proxy { # Enable HTTP_CONNECT
                ports 22 # Restrict to port 22
                acl {
                        allow example.com # Restrict to this domain. It should be 127.0.0.1, but doesn't quite work out, more on that below. Also extra context: https://github.com/proxytunnel/proxytunnel/issues/84#issuecomment-2463191906
                        deny all # Deny everything else
                }
        }
}

Normally, HTTP CONNECT and standard HTTP coexist independently. On Caddy it’s either or it seems. I’m not a Caddy user and can’t comment on how to make both the relay to port 22 and the preservation of the main WebApp on the same (sub)domain block work. I hope Caddy users can chime and I will correct it in an addendum.

With just a couple config lines, we gained the ability to publicly expose any TCP port, without access to any of the underlying infrastructure.

As long as the desired client, not just ssh, is being fed by proxytunnel or understands HTTP CONNECT itself, then we attained a de-facto port-forwarding to any TCP port we desire, bypassing server-side firewalls. There can be server-side blocks for HTTP CONNECT, but it’s trivial to go a level deeper with projects wstunnel.

Client-side connection #

Just like previously, we can first test the connection without OpenSSH. We have to specify our own HTTP server as a proxy, the final destination to localhost:22, as well the intermediate corporate proxy, if there is any. So One proxy if there is a direct connection to our Server, Two with an intermediate proxy.

A bit confusingly, proxytunnel uses the terms Local proxy (-p) for the first and Remote proxy (-r) for the second jump. Our HTTP server becomes Local proxy if there is a direct connection, like with the smart firewall case. If there is an intermediate proxy, our server becomes Remote proxy. Example with corporate proxy 198.51.100.4:8080:

$ proxytunnel -X -z -p 198.51.100.4:8080 -r example.com:443 -d 127.0.0.1:22
Via 198.51.100.4:8080 -> example.com:443 -> 127.0.0.1:22
SSH-2.0-OpenSSH_9.9

• -p 198.51.100.4:8080 specifies the corporate proxy
• -r example.com:443 specifies our server, with the (sub)domain of where we configured HTTP CONNECT
  • -X specifies, that there is HTTPS on this Remote Proxy
  • Without an intermediate proxy, our server would become -p
  • We would also have to use -E, not -X in that case to specify HTTPS on Local Proxy, again a bit confusing nomenclature
• -d 127.0.0.1:22 specifies the final destination to our SSHD service.
  • As the connection happens after our HTTP server receives the information, this is regardless of port-forwarding port 22.
• -z tells to ignore HTTPS certificate checks, especially if you self-sign
  • Provided there is no basic auth, we don’t actually need to trust our connection, as encryption is handled by OpenSSH regardless
  • Making the certificate check work on Windows is a bit of a pain.

You can additionally specify -v for more debug information on the connection.

proxytunnel answers us with a nice arrow -> illustrated connection path and finally with the OpenSSH identification string, provided everything worked correctly. If there is no intermediate proxy, then the command will look like this:

$ proxytunnel -E -z -p example.com:443 -d 127.0.0.1:22
Via example.com:443 -> 127.0.0.1:22
SSH-2.0-OpenSSH_9.9

If you see SSH-2.0-OpenSSH_X.X, you hit the jackpot

Pit falls #

If you don’t see a SSH-2.0-OpenSSH_X.X reply, there could be a multitude of reasons. First: try your (sub)domain:(ssh port) as a destination. This is technically wrong, but due to how HTTP CONNECT works, you may need to tell HTTP CONNECT that the destination is example.com:22, even if there is no such thing.

$ proxytunnel -X -z -p 198.51.100.4:8080 -r example.com:443 -d example.com:22
Via 198.51.100.4:8080 -> example.com:443 -> example.com:22
SSH-2.0-OpenSSH_9.9

You can fix this for nginx, by specifying that server-block as default_server, which will make 127.0.0.1:22 work as a destination. If that is not desired, keep using the -d workaround. To my understanding, in Caddy you have to use the workaround. Apache should have no such issues, by I never fully confirmed it.

proxytunnel, nginx and apache only support up to HTTP/1.1 with HTTP CONNECT. If the corporate proxy initiates a HTTP/2 connection, you’ll error out. You can restrict it to HTTP/1.1 server-side or corporate-proxy-side. Caddy can also do HTTP/2 and HTTP/3, but again, proxytunnel can’t.

Performance wise, none of this matters. The theoretical benefits of HTTP/2 and HTTP/3 don't apply to tunneled SSH.

Finally: Caddy is a cheery lad ヽ(*・ω・)ﾉ and really likes to give a thumbs up, even if everything goes wrong. Even if proxytunnel’s -v debug info says Tunnel established., but there is no SSH-2.0-OpenSSH_X.X, it indicates a missing HTTP CONNECT, incomplete routing or block due to acl.

SSH Config #

We could already connect like this:

$ ssh -o ProxyCommand="proxytunnel -X -z -p 198.51.100.4:8080 -r example.com:443 -d 127.0.0.1:22" user@

For many reason, like compatibility with dev tools, we’ll be specifying a proper SSH config. In your user directory is an ssh config directory. /home/<username>/.ssh aka ~/.ssh on *NIX and C:\Users\<username>\.ssh aka %userprofile%\.ssh on Windows. Inside is extension-less file config. If not, create both.

For clarity let’s use user name kiwi for the user we want to login as at our server and user name frost for the user on the client laptop. For reasons we’ll get into shortly, if you are on Windows: use absolute paths for everything starting from C:\ and re-specify the default UserKnownHostsFile.

UserKnownHostsFile 'C:\Users\frost\.ssh\known_hosts' # Re-specify on Windows with absolute path, Quotes if path contains spaces
ServerAliveInterval 30 # Keep-Alive Packet every 30 seconds to ensure the connection doesn't terminate

## Configuring 3 Hosts. We only need one, but let's lay them all out for clarity:
# Case with Corporate Proxy in-between
Host exampleCorporate
    User kiwi
    IdentityFile 'C:\Users\frost\.ssh\keyFile' # Not required, if password authentication used instead, Quotes if path contains spaces, keep path absolute on Windows! 
    ProxyCommand proxytunnel.exe -X -z -p 198.51.100.4:8080 -r example.com:443 -d 127.0.0.1:22
	# Proxy command takes care of of both HostName and Port

# Case to simply go around a packet-sniffing firewall
Host exampleFirewall
    User kiwi
    IdentityFile 'C:\Users\frost\.ssh\keyFile' # Not required, if password authentication used instead, Quotes if path contains spaces, keep path absolute on Windows! 
    ProxyCommand proxytunnel.exe -E -z -p example.com:443 -d 127.0.0.1:22

# Classic, Direct connection case without ProxyCommand
Host exampleDirect
	HostName example.com
	Port 22 # Only required if OpenSSH port-forwarded on non-standard port
    User kiwi
    IdentityFile 'C:\Users\frost\.ssh\keyFile' # Not required, if password authentication used instead, Quotes if path contains spaces, keep path absolute on Windows! 

Now we have 3 ssh settings. Simply typing ssh exampleCorporate should connect you to the target server as user kiwi, whilst tunneling along the way. ssh exampleFirewall will connect you via the tunnel if there is no intermediate proxy and ssh exampleDirect will do the classic direct connection.

Network capture #

Source 💻 is a Laptop performing ssh exampleCorporate aka ssh -o ProxyCommand="proxytunnel -X -z -p 198.51.100.4:8080 -r example.com:443 -d 127.0.0.1:22" kiwi@. Target 🏢 is a corporate proxy sitting in between a private subnet and the internet. The capture is performed on proxy Target 🏢.

[SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1

[SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1460 SACK_PERM=1 WS=128

[ACK] Seq=1 Ack=1 Win=131328 Len=0

CONNECT example.com:443 HTTP/1.1

[ACK] Seq=1 Ack=104 Win=64256 Len=0

HTTP/1.1 200 Connection established

Client Hello

[ACK] Seq=40 Ack=430 Win=64128 Len=0

Server Hello, Certificate, Server Key Exchange, Server Hello Done

[ACK] Seq=430 Ack=2652 Win=131328 Len=0

Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message

[ACK] Seq=2652 Ack=588 Win=64128 Len=0

Change Cipher Spec, Encrypted Handshake Message

Application Data

[ACK] Seq=2703 Ack=718 Win=64128 Len=0

Application Data

Application Data

Application Data

Application Data

The proxy is none the wiser! Now there is no way for the proxy to know what’s going on and block us. Except with specialized tooling mostly relegated to research, which looks at the encrypted traffic and squints really hard, packet-inspection cannot tell the difference between a normal HTTPS Website and our SSH.

Off-topic: LLMs can guess somewhat accurately what LLMs are outputting based purely on encrypted packet length, if the chat frontend sends token by token as packet by packet.

Now, so far we have made an assumption: HTTPS won’t betray us. This is an assumption that online banking, financial transaction, medical institutions make, that unfortunately cannot be made in general. Advanced IT company security packages may perform…

Deep Packet Inspection #

DPI aka TLS decryption is a technique that involves forcing your machine to trust an intermediate certificate, allow an intermediate to decrypt your connection for inspection and re-encrypt it after inspection.

In the context of HTTPS or corporate connections, this involved pre-installing a Trusted Root Certificate Authority on the user’s machine (i.e. via Windows’ group policy). Such dangerous idea, that even the NSA issued an advisory and the Cypersecurity & Infrastructure Security Agency CISA outright cautions against it.

A compromise of that intermediate root certificate would mean a whole company’s connections being readable in clear text. DPI is easily detectable, as re-encryption rewrites certificate fingerprints. You can check against known fingerprints in the browser’s certificate detail view or this command:

$ openssl s_client -proxy <Corporate Proxy IP>:<Corporate Proxy Port> -connect <Site which is not blocked>:443 -servername <Site which is not blocked> | openssl x509 -noout -fingerprint -sha1

Ultimately, on the networking side you can sandwich another layer of encryption in-between, making DPI’s HTTPS stripping meaningless. Ultimately Ultimately, everything we talked about is network side and a security package sitting on your PC at kernel level can intercept truly anything.

What not to do #

Before we finish by making dev tools sing, let’s quickly go over what to avoid. Some projects go the jackhammer route of exposing the system shell as an HTTP service. The more popular ones are Wetty and Shell in a Box. These are excellent projects in their own right and there are situations where they are a good solution…

…managing public web apps is not one of them. With their reliance on HTTPS for encryption, even ignoring potential DPI, it takes only one config mishap to expose full console access. Especially when web apps change management, things like this are easily forgotten. One misunderstanding away from disaster.

Dev tools #

Now that we can ssh exampleCorporate to automatically connect via our encrypted tunnel, let’s make sure our dev tools are happy. If git is used via SSH, it should work automatically. Thanks to Unix philosophy, everything mostly works automatically. Mostly.

VSCode #

VSCode’s SSH integration should work automatically. Browsing, editing and downloading files with no extra settings required. The SSH config should be automatically detected by VS Code and be selectable when initiating a connection. Same goes for TRAMP in Emacs.

SCP #

The basic file copying scp works universally as well and is installed whenever the OpenSSH client is. Syntax remains also the same. Invocations like scp exampleCorporate:/path/to/source/file /path/to/target will automatically use the HTTPS tunnel. Nothing interesting here.

Rsync #

Provided your WebApp doesn’t have server-side CI/CD, you or your CI/CD system will be uploading files to your target server. A WebApp build usually contains many assets, font files, videos and scp -r just doesn’t cut it, retransferring everything. rsync is widely used to accelerate this task and file copies in general.

It copies folder structures, preserves their permissions, but does so whilst only copying what actually changed and applies compression for faster transfers. This makes rsync easily 100x faster when compared to scp -r when there are a bunch of assets, but you only changed the code a bit.

Honestly, rsync is a life saver.

Whilst rsync has no running service, it needs to be installed on the server as well. On the client side things are are just as simple, as with scp: rsync -avz --progress /path/to/source/folder exampleCorporate:/path/to/target/folder to upload a folder and show live progress will work automatically via the tunnel…

…unless you are on Windows…

Rsync on Windows #

This is a bit of an interesting story. On Windows there exist two versions of rsync. The standard FOSS rsync client with source code from the Samba project and cwRsync, which has a free client but is closed sourced and monetized via its server component by itefix.

cwRsync has a right to exist, though I do find it a bit scummy to capitalize on FOSS projects Cygwin and Rsync, considering the client is presumably mostly the FOSS Rsync source, compiled via cygwin.

Anyhow, connecting to proxies with cwRsync simply won’t work, if you don’t call it from a unix emulating environment. cwRsync will error out with /bin/sh not found, due to how itefix setup the compilation. Luckily the free open source way works, though the calling convention in windows is a bit of a mess.

The issue is, that rsync needs to be called with the OpenSSH that it was built with, as it is linked against a specific version. But OpenSSH in turn also needs to call our proxy command. The way this call-chain happens depends on compilation environment and settings. In Windows’ Shell, cygwin is responsible for translating these calls.

The call of rsync.exe → ssh.exe works, but the subsequent call of ssh.exe → proxytunnel.exe, connect.exe or corkscrew.exe fails, due to cygwin and its requirement of sh.exe to be present. Providing your own sh.exe won’t work due to binary incompatibility.

With itefix's cwRsync there is no way to fix it, since it's closed source. 👎

This calling convention needs the binary to be in a usr/bin/ subfolder with sh.exe present, due to how cygwin hardcodes things, otherwise you get a /bin/sh: No such file or directory. Unfortunately, the flexible windows package managers like scoop ships with cwRsync only, something I hope fix in a PR.

Without resorting to full WSL, we would need to install MSYS2, install rsync and make it available in PATH. Big fan of MSYS, but that’s too much of a mess. As a shortcut, I extracted rsync v3.4.1 from MSYS2 and the associated ssh. Beware that the usr/bin/ structure needs to be intact due to Cygwin hardcoding.

Here it is: rsync-3.4.1-windows.zip

Took me a while to figure this mess out.

We could just call it from inside MSYS2 or WSL and everything would be fine, but we would be referring to different SSH configs and you may have other tools in turn calling rsync, so this needs to work outside of MSYS and WSL. Let’s start by extracting rsync and exposing its usr/bin/ in PATH:

There are many tutorials showing how to set PATH, so I skip the details here. Restarting your terminal or closing all VSCode windows, if you use the VSCode integrated one, you will have gained access to command rsync. We can invoke rsync with a long command like this:

$ rsync -e 'C:\rsync\ssh.exe -F C:\Users\frost\.ssh\config' -avz --progress exampleCorporate:/path/to/source/folder /path/to/target/folder
bash.exe: warning: could not find /tmp, please create!
Via 198.51.100.4:8080 -> example.com:443 -> 127.0.0.1:22
Enter passphrase for key 'C:\Users\frost\.ssh\keyFile':
sending incremental file list
/path/to/target/folder/posts/tunneling-corporate-firewalls/rsync-3.4.1-windows.zip
      7,765,500 100%    4.30MB/s    0:00:01 (xfr#2, to-chk=53/10810)
/path/to/target/folder/posts/tunneling-corporate-firewalls/tunneling-corporate-firewalls.md
         83,344 100%  135.20kB/s    0:00:00 (xfr#3, to-chk=52/10810)

sent 6,136,784 bytes  received 18,824 bytes  373,067.15 bytes/sec
total size is 420,857,408  speedup is 68.37

The reason we used absolute paths even in the ssh config, is that rsync is in an invisible *nix environment provided by cygwin and cygwin’s path translation won’t resolve correctly otherwise. Same reason we re-specified UserKnownHostsFile. It would work without, but you’d get annoying This key is not known by any other names. each login.

You can ignore the bash.exe error, irrelevant for our case. -e let’s us specify the correct ssh and the related config. If we don’t specify it and let the system ssh take over, it will seem to connect, but will fail with a confusing error:

$ rsync -avz --progress exampleCorporate:/path/to/source/folder /path/to/target/folder
Via 198.51.100.4:8080 -> example.com:443 -> 127.0.0.1:22
Enter passphrase for key 'C:\Users\frost\.ssh\keyFile':
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(232) [Receiver=3.4.1]
rsync: [sender] safe_read failed to read 4 bytes: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(283) [sender=3.4.1]

But that’s an awfully long command and other tools using rsync won’t know it. Luckily, there is a way to specify a default: environment variable RSYNC_RSH, which is equivalent to the parameter -e. Same deal as before, lot’s of online explanations on how to set environment variables in Windows.

And after all that, we can finally use rsync via HTTPS tunneling, like we would on any *Nix platform or environment:

$ rsync -avz --progress exampleCorporate:/path/to/source/folder /path/to/target/folder

It's kind of bananas what we have to go through on Windows to get basic tooling without resorting to WSL or MSYS 2. Makes me really appreciate what a fine piece of engineering MSYS 2 is.

Finishing up: #

I should clarify what level of power and potential for misuse we are dealing with

Everything we talked about applies to circumventing internet content filtering as well, as SSH supports Dynamic port forwarding, allowing you to browse the web, if you connect via ssh -D 8888 exampleCorporate and point your browser to localhost:8888 as a SOCKS5 proxy in the network settings.

Your server turns into a quasi VPN, with you browsing the web from the perspective of the server, no corporate content filters. It’s standard practice to block changes to proxy settings company wide. But it’s up to the browser to enforce such policies, making it one of the more easily circumvented and useless security measures.

And there we go. We configured, drilled and circumvented, clawing back power to create digital infrastructure in situations where normally we wouldn’t be able to.

With great power comes great responsibility. Hope this article gave insight into what's possible, besides the classic ways of server access.