SSH all the way down!

This is my first technical post! It's a long one, for sure, but if you find the subject interesting, I hope you enjoy the read. Comments are more than welcome!

Table of contents

A journey of self-host discovery

So I've been getting into self-hosting some of my content, as evidenced by this gallery itself. I've never been much of a DevOps/infrastructure guy, but a full-stack developer. Still, I figured: why not try my hand at it? I have a bunch of stuff that I wanted to make available on the web:

• A Git service, to self-host my adult-oriented source code without worries about other sites' ToS.
• An image board, to have a single place from where I can share the artwork that I got from others.
• A privacy-first search engine, since the instance that I normally use often crashes or gets rate-limited.

And more. Plus, I had a Raspberry Pi lying around, and it was already online all of the time in order to run a Discord bot. So how hard could it be?

Well... almost impossible, actually. Never mind that the Raspberry Pi struggled running all of these services at once, that's the least of my issues. No, the REAL problem is that, even if I have a better computer plugged into my router 24/7, I can't self-host my websites.

And I don't mean buying a VPS. I mean actually self-hosting Internet services from my home.

If you've had any experience trying to do the same, you are probably wondering if the issue is Network Address Translation – in which case, you'd be correct!

If you're unfamiliar with NAT, in short, it means that the address that I access the Internet with doesn't match my router's address. So if I try and expose a server through port forwarding on my router, there's no way for any services to listen to it.

Well, bummer. What can we do about it?

One solution is to only use IPv6. Unfortunately, I have this pesky requirement where I want anyone to access my site. I like IPv6, but it's not as widespread as it should be.

Another solution – provided that you're still committed to running your services in your home server – is to use a proxy server. Instead of our IP, we'll use someone else's IP, which is already exposed to the Internet without any address translation. Then any traffic to our websites gets redirected through a tunnel, and our server transparently responds to them as they come through the network.

That's where something like a VPN really excels at. In fact, I've used Mullvad VPN in the past partially for that reason, while they still had port forwarding from the local computer to the Internet.

And I say had, because Mullvad removed support for port forwarding, alleging legal reasons for the decision.

Crap. And I guess Mullvad lost a customer. Back to square one...

To their credit, VPNs are excellent for this kind of setup. It's common practice to share data channels between servers through a secure tunnel, over a virtual network. But I personally have three issues with public VPNs (I'll leave the third one as a secret for later; see if you can guess it as a challenge):

1. You generally have to pay for them, or set them up properly, where any mistake can break your security.
2. It requires installing software I'm unfamiliar with, so there's a good chance I'll break stuff irrecoverably.
3. ???

That's not what I want. Half of the reason I took on this challenge of self-hosting stuff was for fun, to see what I could pull off with minimal effort. Plus, I want results in an afternoon, not over a week. What else can I do...?

How may I Serveo you?

Some options that I've immediately found were ngrok and Cloudflare Tunnel, both of which are API gateways. In short, they provide you with a tool to expose local servers to the Internet, by connecting to their service through a private tunnel. It's not so different from a VPN, except that it requires using their specific application.

I was a bit on the fence, since ngrok seemed to be a paid service, and Cloudflare requires full control over your domain. But it was interesting to know that things like this already existed. And in my research on Ngrok specifically, that's how I ended up learning about Serveo.

Chances are that when you click the link to Serveo in the last paragraph, it won't work. Serveo seems to get DDoS'd all the time. So here's how the page normally looks like:

To explain the gist of it, unlike ngrok, you don't pay any fees or install any special tools to use their proxy service. Instead, all you need is an SSH client.

Wait, really?!

Yes really. As it turns out, the secure shell protocol supports what is called a reverse SSH tunnel (definitely read this article if you wanna learn more). In fact, SSH can do much more than simply create a secure session to display a shell from a remote machine – it's loaded with functionalities that you might not expect it to have!

I was familiar with SSH already, or at least its common use cases. Creating keys, adding them to a server for authentication, pushing Git commits, and so on. It's always been a part of my dev career, and arguably, it's one of the fundamental skills you invariably pick up on.

And I'd heard of port forwarding through SSH, but never gave it much thought. But is it really that easy to set up?

Well, I gave it a go, trying out the simple single-line command that the Serveo website told me to follow:

ssh -R gitbadmanners:80:localhost:3000 serveo.net

And, to my delight, it worked! No extra configuration, no tool installation, no questions asked – my RPi server was live on gitbadmanners.serveo.net, and I could access it from any device connected to the Internet!

With a bit more fiddling, I even got my custom domain to work. Pointing the CNAME record to their service, adding a special TXT with my key's fingerprint, and adjusting the command slightly, I got both HTTP and SSH (with a caveat*) to work, too.

ssh -R git.badmanners.xyz:80:localhost:3000 -R git.badmanners.xyz:22:localhost:22 serveo.net

So that's pretty cool, huh? It even supports HTTPS! This feature is specific to Serveo (and most reverse proxy solutions). But to any developer, having HTTPS is not merely a huge plus; TLS support is kind of expected out of the box for anything Internet-facing nowadays.

In fact, there's nothing unique about Serveo doing what it does. localhost.run works very similarly, by also leveraging an SSH connection as a way to both authenticate the server and pass incoming data through a secure tunnel.

But how is SSH able to do all of this? It might be worth it to go over the fundamentals.

A SSHort-ish primer

I'll be using the terms OpenSSH and SSH interchangeably, although the former is an implementation of the latter – the most common implementation, in fact. I'll also make other assumptions and simplifications throughout, so feel free to check the links if you wanna learn more.

When two computers need to communicate over a network, they do so over a connection. That connection happens through sockets, an abstraction at the level of the operating system over finer networking details (such as addressing and sending electrical bits).

For any application, sending and reading data through a socket is not so different from interacting with files. Whether you connect to the Internet to play a game, access webmail, access non-web mail, and so on, your computer is connecting on a network socket (usually TCP) and sending data to your router, before it traverses the electricity-powered spaghetti that we call the Internet.

The main aspects distinguishing each application on the Internet is what kind of protocol is used for communication, as well as the code running on the server and your machine (usually called "client"). With a previously-agreed API, both parties can communicate, usually with the server being the authority over the client – and that model is how most of the modern web has been built.

One such application that runs over the web is SSH, short for Secure Shell Protocol. It's a cryptographic-based protocol, designed to replace less secure shell protocols from its time, by making sure that all traffic between a server and a client is end-to-end encrypted.

That's a lot of terms in a single sentence, so I'll explain them one by one.

Cryptography is a way to secure communication between two parties, or in our case, two computers over a network. There are mainly two reasons to encrypt (i.e. secure) this communication:

1. To prevent others from spying on the transmitted data. Without a secure channel, things like your password or credit card data would be passed in plaintext, for anybody on the way of your data to see and steal!
2. To prevent others from tampering with the transmitted data. For example, your ISP might decide to inject ads on the pages you access (that's a real thing!), or modify/censor the pages you access as it sees fit, without you even realizing that it's happening.

Whenever you access a link that starts with https://, you're using the secure version of HTTP. While it doesn't mean that you're protected from anything a malicious actor might do (or even that the other side necessarily is who they claim to be), it ensures that your traffic will be protected in the two ways mentioned above. When only the two ends (i.e. the client and the server) can understand the data, we say that the channel is end-to-end encrypted.

Now, what's a shell? It's a kind of application that comes with every operating system, allowing you to interact with the machine on a high level through text commands, rather than a graphical interface. It requires a terminal, like those you see in hacker stock footage, and isn't much different from running any other program, except that all communication is done with individual characters and escape codes.

Typically, a shell runs on the same machine that it's on, but with SSH or telnet, you can access a remote shell – that is, a shell on a remote server.

In order to keep its security guarantees, SSH handles authentication of users (which usually map to users in the operating system) with methods such as a password, keyboard prompts, or most importantly for us, public keys. I won't go over public key cryptography in this post, but suffice it to say that it's a special file that proves that you're really the one accessing the system. For accessing a remote shell, it is pretty useful, and more secure than a simple password.

And that's the way I learned how to use SSH, just a way to use my private key to access a remote server or transfer files without giving it much thought. But as we've seen with Serveo, SSH can do more. Much more. For instance, it is a viable way of exposing services to the wider Internet, as long as somebody else is willing to proxy the connection for us.

But is it really secure?

So we've seen that reverse SSH forwarding solves two of the problems I had before. I don't need any fancy setup to expose a service, and with Serveo, it's all painless and free. But it still comes at a cost, mainly (and here's the reveal of the third issue that I also had with other solutions):

3. All traffic is passed unencrypted through the proxy.

And, as it turns out, a service like Serveo runs into that same issue.

But wait – you say –, I thought all traffic in both HTTPS and SSH was end-to-end encrypted!

And it is, hypothetical reader. But we have to consider which ends are encrypted.

It makes sense, if you think about it. Our hole punching solution doesn't do anything special, and all handling of HTTPS traffic is delegated to the proxy server. That means that the proxy server also handles encrypting and decrypting any messages from HTTP or any other TCP connection, thus making us lose the guarantees that the data won't be inspected or modified in some way. This means that any data like passwords shouldn't be sent over our tunnel, as they could be stolen either by Serveo or a hacker who has taken over the service. I knew this going in, so I didn't make the mistake of ever authenticating over the exposed service – and you shouldn't either!

The fix for this is obvious, as much as it pains me to admit it: I'd have to buy a VPS and host my own solution.

At least this way, I can guarantee that I'm the only one able to interact with the transmitted data, be it encrypted or plain. And before you say "that's still insecure", consider that it's literally what happens with any web server. Plus, even with encryption, HTTPS doesn't mitigate malice. But in this case, I'm hosting stuff for myself, and I can trust that guy for sure.

But then, I wonder if there's even an open-source version of Serveo that I can use...

sish happens

It was easy enough to stumble upon sish through a search engine. It is essentially an open-source version of Serveo (or ngrok, or localhost.run), complete with a range of configurations. And as you can see from the image that they host on their website, they offer exactly the same interface:

After buying a new domain and VPS (I like Namecheap and Hetzner for those, respectively), and setting up an instance of sish through Docker Compose (which simplifies the deployment process a lot), I migrated all my Raspberry Pi services to that. Of course, since "migrating" is just changing the SSH command to point to a different URL, as well as updating some DNS entries, it was a pretty simple process!

Now I have all my services proxied through the sish instance, which can handle HTTPS for them – same as Serveo before. And I can guarantee that the server won't go down sporadically, unless I turn off the virtual machine myself (or it crashes), so that's another bonus.

But my RPi still isn't handling the load of so many services at once. Hm. Maybe now that I have a VPS, it's worth offloading some of the services to run on that, instead...

Less self-hosting is more hosting

So I have a bit of a weird setup. I use Fastmail to manage my domain. It lets me manage my e-mail addresses, but it means that I have to set up DNS records manually, for things hosted outside of Fastmail like this gallery and my personal site. That's fine. I just use CNAME and ANAME to point those to my NearlyFreeSpeech.NET websites.

Then there are also services like my Forgejo instance that I mentioned earlier, which was running on my RPi and being exposed through sish. But it often causes the mini computer to struggle, whenever it processes too much data or too many requests at once. In this case, it made sense to also move Forgejo to the VPS as well.

And so, I started looking at how to set up some kind of proxy server in front of sish. I'd heard of Traefik and Caddy, two reverse proxies that do some of the annoying stuff like managing TLS for you. Still, I couldn't find a trivial way to make them work well with sish. Aside from that, despite what I'd been led to believe, they still require a non-minimal amount of effort to set up and maintain – and I'm still intent on being maximally and efficiently lazy for this whole project.

I even started making a diagram, while trying to explain the architecture that I had in mind to a friend:

I wasn't really seeing a way to make this work... But hmm. What Traefik is doing isn't so different from what sish is doing. They are both reverse proxies, in a way, although Traefik is a traditional one, while sish does it in a roundabout way with SSH.

So then I tweaked the diagram a bit...

Aha! It turns out that exposing services through sish is always the same, whether it's running on the same machine or halfway across the globe. We just need to set our credentials and start a permanent shell session that does reverse port forwarding for us. With Docker Compose, that's both trivial and safe: no ports get exposed outside of the container network. And sish even supports advanced features, such as multiple domains and load balancing. It's an SSH-powered reverse proxy!

With everything configured just as before, and the appropriate private keys created to authenticate each service, we are able to finally expose multiple services from different sources with a single reverse proxy server. It's all transparent, too, despite how we're using several distinct domains and services, and despite how some of these things are running at my home instead of the VPS. Never mind that I ended pretty much back where I started, it all just works!

It's SSH all the way down, baby!

All in all, this project made me realize just how powerful SSH can be. Reverse port forwarding in particular is a great way to expose local servers remotely, quickly and easily, as long as you bind a socket for HTTP.

...Although, you don't even need to expose a socket on your machine to do this!

In the next post, I'll go over the actual nitty gritty details of how SSH does this and some code I wrote. I'll also show how I've deployed an SSH-proxied, HTTP-based game without any need for binding a TCP socket.

* A caveat about recurSSHive forwarding

I mentioned way earlier that Serveo (and sish) are able to forward SSH traffic, even through the same port where you bind services. This is what SSH internally refers to as a proxy, i.e. the sish instance serves as a proxy to our Git service.

In order to be able to direct traffic to our underlying service, we must tell our client to expect the sish instance to reply to us first. With the ProxyJump option, standard input and output on the client will be forwarded over the secure channel to establish TCP forwarding.

The details aren't completely clear even for me, but we don't need to care about the details. If we just want to connect to our Git server for pushing, we'd simply set the following contents in our SSH config:

Host git.badmanners.xyz
	ProxyJump my.sish.instance

Alternatively, if we were exposing a real SSH server on ssh.badmanners.xyz, we might use the following command:

ssh -J my.sish.instance ssh.badmanners.xyz