So I had a bit of a scare this morning.
I'm in a hotel in North Carolina, hundreds of miles from my server, when I try
to log in and get a "Connection refused" error. Phooey. I hope the hotel wifi
is just filtering port 22 for some terrible reason, and hop to a friend's server
over a non-standard port and I'm in. Great! I type out the connection again and...
Hmmm. This is starting to get concerning. I briefly freak out that my server might
have had a terrible accident, when I realize that all of the other services I rely
on all the time are still running. Phew. Still, I don't have ssh access, and that
is very troubling. I start poking around with other things and eventually
unconsciously try to ssh back in to check something and to my surprise, it works!
As soon as I realize what I've done, I pull up journalctl to see if sshd is going
crazy and I'm met with hundreds of lines of Chinese IPs plugging away at my server.
But wait! I have fail2ban installed. That should be stopping these, right? Well
maybe something broke when the server was upgraded to Debian 9... and now I'm
off trying to figure out what is running and where the config is.
Some time later, I finally have a solution. The problem (ultimately) was
systemd, though not directly. I had moved this system to systemd some time early
in Debian 7, which meant that the standard logging locations were still there,
but no loger being written to. So fail2ban was looking for /var/log/auth.log,
found a file, read it and found no problems, completely ignoring the fact that
the file hadn't been written to since 2013. This isn't really fail2ban's fault,
it has support for ingesting the systemd journal, but on Debian, the default
backend still tries to use those files. I could set up rsyslog to start
writing those files again, but I have no particular need or desire to do that
outside of this program, especially as fail2ban knows how to read the journal
on its own, if it's configured to do so.
So the solution is pretty simple, though it took me a while to get there. First,
I had to apt-get install python3-systemd, to get the proper libraries to
actually use the journal. Then, I had to have the following in my jail.local:
backend = systemd
I also put a few other tweaks in there as my config hadn't actually moved over
from the previous version, but since it's now in the jail.local instead of
editing the system config, this shouldn't be a problem again.