One of the most common fears about running your own services is that you don’t trust yourself to keep up with security.

It’s a valid fear, but instead of succumbing to the fear, I have decided to tackle it. After all – the reason we run our own services is because we

• Can
• Like it
• Don’t like to be to dependent upon external entities, that
• …may decide to sell your data.
• …or even have security holes themselves!

A couple of weeks ago, we saw a couple of serious 0-day vulnerabilities that would hit a lot of homelabbers, including me. It’s time to up the security game, also at home.

In this blog post, I’ll provide some good advices that will make your life simpler, as well as some things you, like I have to do, probably can do better than today.

Maintain a good outer firewall policy

This is the first and foremost step. You need to understand what you expose, how, and whether you expose it to the world or limit it further

Many people advertise not exposing anything at all,but relying on things like tailscale, where you in fact open ports outwards, and then lets tailscale forward traffic to your services over this open connection.

It’s actually pretty neat, and it also makes sense if you are behind things like CGNAT, where you don’t even have a public ip address at home on which you cand do port forwarding.

At the end of the day, if you have public IP to your home and control over your gateway, the difference isn’t that great. However, you can probably get by with less networking knowledge and setup, which is a valid tradeoff for sure. For me,who has dabbled into networking and firewalls for decades, self-control over this is more important. However, I have lately started using some of this for some of my workloads – but more on that later.

Whatever you do, make sure you limit the attack surface, and expose only what you intend and need to expose. It’s not a replacement for keeping your non-exposed services secure, but it will make sure the baddies need to penetrate one other layer of defense before it gets to your internal stuff, so it buys you time.

Cloud vendors lets you manage the outer firewall policy in code, and I made my unifi network operator, which gives me exactly that for my home lab. With this, I fully integrate my kubernetes workload with my Unifi gateway, and make sure that my firewall openings stay up to date with the changes in my cluster. It’s pretty neat, and one of the days I will get to releasing it as open source….

With a good firewall, you can also do things like rate limiting, geo blocking and other advanced mechanisms, but this will vary heavily on what you run at the edge, and I’ll not go that way in this post.

Your reverse proxy – it can compliment the firewall

I run traefik as ingress gateway. It is the TLS endpoint for all my workloads, and make sure to route, based on incoming hostname, the traffic to the correct workload.

It can do a lot more, though. Even though my blog is exposed, it’s admin interface is not, and that magic lives in traefik.

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: wordpress-ingressroute
  namespace: traefik-external
  annotations:
    kubernetes.io/ingress.class: "traefik-external"
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(vegard.blog.engen.priv.no) && (Path(/wp-login.php) || (PathPrefix(/wp-admin) && !Path(/wp-admin/admin-ajax.php)))
      kind: Rule
      priority: 11000
      middlewares:
        - name: internal-ipallowlist
      services:
        - name: wordpress
          namespace: wordpress
          port: 80
    - match: Host(vegard.blog.engen.priv.no) && HeaderRegexp(Content-Type, (?i)multipart)
      kind: Rule
      priority: 10000
      middlewares:
        - name: internal-ipallowlist
      services:
        - name: wordpress
          namespace: wordpress
          port: 80
    - match: Host(vegard.blog.engen.priv.no)
      kind: Rule
     services:
       - name: wordpress
         namespace: wordpress
         port: 80
  tls:
    certResolver: letsencrypt

As you can see, for two of my routes, I have a middleware internal-ipallowlist that limits acess to those routes to my internal ip addresses. The first of these limited routes is for the admin interface of wordpress, the second one is actuallty to block a 0-day vulnerability in the ingress. This allowed me to block a security issue even before a new docker image for wordpress was published.

The third route is for everything else i..e. for my blog readers.

Middlewares can be quite powerful, I also have one that does OIDC authentication, which I use for the workloads that doesn’t have that possibility themselves.

Update your workloads

Even with good outer protections, you are not secure. from bugs in the software itself. Manually keeping up with new versions of docker images etc is a pretty daunting task as your home lab grows. For this, automation can be pretty useful. I use renovate for this, a piece of software specifically meant for tracking dependency updates. In essence, ti scans my ArgoCD repositories and looks for refeences to software components and checks for updated versions. For these updates, it creates PR to update to newer versions of the software. At home, I just glance at the changelog for them and merge them on sight. In a more professional setting, you obviously want to make sure they don’t come with any breaking changes before you merge them.

Separate your workloads from each other – and limit their access to the internet?

One of the triggers for me to upgrade from plain docker to kubernetes was that it came with a richer set of infrastructure tool. One of them is the networking layer, that comes with traffic rules which limits traffic inside your cluster.

I have set a global policy which actally just denies all traffic. For any networking to happen at all, I need to apply rules that override them.

For example this blog is built on wordpress. It has wordpress itself, but it also needs a database. I need to explicity allow wordpress to talk to its database. I wrote about this in Kubernetes Security 101, so I am not going to dive into this a lot. But, I have moved further than this, I am now trying to limit also outgoing traffic more. The reasoning behind this is that if your workloads are breached, i.e. that someone gains shell access in a container, limiting outgoing traffic might just prevent them from downloading the malware they were going to install.

Login security, Identity and access restrictions

Insecure login mechanisms and brute forcing passwords is of course still popular. To make sure my workloads use good mechanisms there, I try to enforce OIDC wherever I can – and sometimes I protect my workloads with OIDC in traefik if the workload itself doesn’t support it.

I have chosen Keycloak for this. This means that wherever I use keycloak, I can enforce password policies, 2 factor authentication and other good practise logon security mechanisms.

It also makes my life simpler by creating roles for my home. Enforcing this in one place and not scattered all around makes it easier to maintain the policy and not leave around weak «test passwords» or other bad practises I might have left around when playing with stuff.

More advanced security tools

Doing the above, you have come a long way. But there exists more specific security tools, of course. The below are the ones I have tried so far, and am running in my home lab.

Falco – intrusion detection and prevention

Software comes in a lot of forms and a lot of complexity, with various behaviors. Falco is meant for detecting when your software starts doing more unusual things or more risky things than it should.

A plain old web server will just read the files it provides and serve them to the world. If your web server suddenly launches a shell in the container, it might be a sign that there are some shenigans around. If it starts tryng to talk to the Kubernetes API, you probably have reason to worry. In the latter case, I have network policies that would prevent it, but I’d definitely still like to know about someone trying to break in.

And this is where falco comes to your aid. Out of the box, it comes with a set of rules that would detect things like this. However, without tuning it will be quite noisy.

Things like a Kubernetes Operator is meant to contact the API, all the time, so you woudn’t want a warning each time it does. Some software will also launch other processes inside the container. You might decide to just allow that outright, but you can also have more control of it, by decideing which process in the container can launch what other processes.

Going even further, you can decide to actually take action about this, and not only warn about it. Falco Talon can take action when a rule triggers, and decide to label your workload suspicious or even outright prevent it from talking to the world!

Me, I mainly send a slack notification whenever a falco rule has triggered. Without proper tuning of your rule set, there will be a lot of notifications, so you’ll need tuning and exceptions.

I also use AI a lot to analyze the triggers, and have it propose exception rules for supressing the false positives.

I’d consider falco and talon next level in home lab security, but for someone like me, it’s a challenge I take.

Trivy – scanning your workloads for security issues.

Keeping up with the renovate PRs is definitely a good start. But is it enough? Maybe you are running self-built containers which you have to maintain yourself, or maybe you run badly maintained software.

Trivy is a security scanner you can run in your cluster. It will scan all your running container images – and more – for security issues.

One class of security issues it can detect is vulnerabilities. It will look up public vulnerability databases (CVEs etc), and check if your running versions are flagged as vulnerable.

It can also detect certain misconfigurations and secrets exposure, although no such issues have so far come up in my usage of it.

I started running Trivy a few weeks ago, when two common open soure components had severe 0-day issues in the span of just a few days.

So, can you actually use this for something? That varies, of course.

Php2Shell was one of the vulnerabilities. I had a couple of workloads running Php, and while waiting for upstream fixes, I made a Traefik Middleware that blocked the features the vulnerability was exposing (or, rather, limit access to it from the internet), by using this snippet in my ingressroutes for the workloads with PHP:

- match: Host(vegard.blog.engen.priv.no) && HeaderRegexp(Content-Type, (?i)multipart)
  kind: Rule
  priority: 10000
  middlewares:
    - name: internal-ipallowlist
  services:
     - name: wordpress
       namespace: wordpress
       port: 80

…and of course we can use AI!

A lot of this leaves around logs and metadata for my consumption later. As I can not be a 24/7 security team, I have decided to run a few AI agents to help me.

• One AI agent scans the trivy runs for issues and send me a notification if there is any serious issues.
• One agent is a general «health check» agent, but I also check some basic security in it

In addition to this, I use Claude with a few MCPs and scheduled jobs. One I particulary like is a «daily improval suggestion» job, which will just scan through a lot of my kubernetes cluster and come with a small task I can choose to to if I have the time, Being a geek, I love improving my services!

Summary

Kubernetes comes with a rich set of features and an abundant security ecosystem. With a little bit of initial work, you too can run home services with better security than many companies out there. The tools exist, are largely free and standardized, and not too daunting to set up.

This was by no means an extensive list of what you can do, but it’s a more than adequate start. Maybe you have some tips for good security tools you are using? Leave me a comment!