- A cautionary tale
of amateur webhosting -
Maybe amateurs shouldn't host their own sites. I do anyway, because it was a fun little project, and does continue to be, even though it has been on the back burner for a very long time.
So, confession time. For several years, my only method of SQL backups was a cron job on the server that saved a mysqldump locally, with a fixed filename (so it'd overwrite on the weekly schedule it was on). I would manually copy that over to my desktop every few months, or much less often. In February 2016, for whatever reason, I wisened up. Maybe I was bored, or maybe after several years of neglect, the slowly mounting fear of losing all my data due to an inadequate backup scheme had finally reached the point of prompting some kind of action. Probably both. I ended up researching a very quick way to set up a timed job on my Windows10 desktop that pulled the mysqldump file over to the desktop, and in the process named the file based on the copy date. It was possible without installing any new software, just vanilla Windows10, so that made me happy.
After 3 ½ weeks of running this setup, I checked up on how it was working. I was delighted to learn it had worked precisely as intended, but I also learned of an incredibly alarming trend in the filesize of the dump file.
A file from October 2013 that I had kept around as reference was around 11 megabytes. A manually pulled file from beginning of February was 17 megabytes, which I didn't have reason to doubt, it had been more than 2 years after all. But the pattern of the next weeks was jarring:
2016-02-15, 20 Mb
2016-02-22, 31 Mb
2016-02-29, 62 Mb (!!!)
This was the point at which I realized something was terribly wrong. It was apparent that something was flooding the server with spam. I run several very old unlicensed IPB forums (the version I run does allow it, so I shouldn't be breaching their terms for my noncommercial use), and I had to lock them down several years ago due to spambots having their way with them. It had been a sufficient strategy to stop new member registrations and disable guest posting, but now I got a bit worried if perhaps new exploits had been found that circumvented those controls.
I could have researched how to analyze the database to find out where the spam is, but I wanted to first try my luck with something as simple as a look into the access log. Typically the spambots try all kinds of different attacks at the IPB or the server in general, which tend to leave errors in the access log. This happened to be the case here as well. Upon simply glancing at the tail of the access log, I immediately spotted suspect behavior. Now, errors left by spambots are not a red flag as such - like said, they are a commonplace thing to have all the time. But the part that caught my attention was that the errors were arising from a subdomain that I did not honestly remember even existing.
That's a huuuge red flag. I realize I have an IPB there, and log on. There are thousands of threads and hundreds of "members" - on a board which, from what I could tell by looking at the oldest available threads, was completely empty until around December 2015.
I start with the usual lockdown, removing posting privileges from non-admin member groups. Since the board had been empty, I also use the IPB board on/off switch to turn it off altogether - no point even leaving behind a view access since this was an unused board. Luckily the IPB admin panel, while a bit crude in a 2003-dated IPB version, did have a few interesting statistics I could check out before commencing with the great purge.
The flow of members and threads had started late December 2015, with low volumes throughout January. I guess the bots had propagated the address over time slowly into their network, after discovering their spam was getting through. From February the volumes surged to hundreds of posts daily. There's no reason to suspect anything less than exponential growth would have continued to take place, had I not gotten incredibly lucky and set up my weekly backup task at just the right time to reveal the build-up of a tidal wave of spam, before it severely compromised my server.
It's not just that my bandwidth and storage would have been overwhelmed, it's scary to imagine what kind of crap could have ended up on the site that I could potentially have been liable for. Now I managed to plug the hole within just over 2 months of exposure.
I was ever so happy that an "empty board" feature existed in the admin panel, which got rid of all the spam in one fell swoop. Pruning the userbase took a little more effort, and I once accidentally removed the admin user with the IPB controls (yeah, oops...). Still, after restoring the admin user manually by way of ad-hoc sql, and a few more bulk runs on the Delete User tool in the IPB, I had cleaned up every user except the admin. Problem solved.
At the end of it, I pulled a manual sqldump. 13 megabytes.
On the bright side, seems the server as a whole is still not too exposed to crap, despite running on quite old software - after all, the leak was really mostly a configuration error on my part. Still, I've known I should probably rebuild the server and recode everything in the not-so-distant future. You know, maybe pick up some new but inexpensive, fault-tolerant, passively cooled, energy-efficient hardware. Get a new O/S and infrastructure (apache, php, sql) and finish by rewriting the sites not to run on a more than a decade old IPB.
I still need to do all of those things. I don't consider it just a chore, mind you. It should actually be reasonably fun. As long as I can do it in a way that the old system runs on the side, so I can take my time with it. But I sure have been putting it off, no doubt. Guess this was a reminder that I really can't put it off indefinitely.
- Fox Stevenson is amazing
It suddenly occurs to me that Fox Stevenson / Stan SB is amazing. I already knew he was great, but I just upgraded him to amazing after Spotify's Discover Weekly kindly threw "Get What I Can" my way today. Much love & respect.
That is all.
- A monstrous release from Monstercat this month
also:London Elektricity(after-post edit) -
New Monstercat compilation is out again and holy crap it is ON POINT. So many awesome tunes - I'm liking it more overall than the last couple ones. I mean, I always find a few really great tracks from each compilation but it seems there's just more of those this month. 024 - Vanguard streams on Spotify as usual.
I will update my Flavor of the Month playlist with some of my picks like I always do, over the coming weeks. I already have a few adds from the release on it.
Aside from that, there were a couple of interesting surprises on the album that I want to highlight. I was never too fond of Pegboard Nerds previously. I just found their style a little bit too dark for my tastes (thinking of tracks like "BADBOI" or "Bring the Madness" here). Imagine my surprise when I listen to "Pink Cloud" and then see Pegboard Nerds on the artist credit! They coming out with the feels on this one. Showing versatility I think. Totally in love with the track of course, nice to be surprised by these guys trying something different and rocking it.
And that wasn't the only moment I had with the album that made me do a double take. I've kind of liked Stephen Walking's style, found it pretty charming. A little silliness, a little retro, very distinctive. Still I wouldn't say I'm his biggest fan, for my taste the hit/miss factor usually landed ever so slightly on the "miss" side - but I do have a lot of respect for what he does. So here I am, casually listening to "One Man Moon Band". Suddenly I hear this mad, totally obscene bassline lead into a segment with just the most adorable, simple hook with a kind of a horn-like synth? The one at just past 1:45. I'm going like, "what the hell am I listening to???" and for the rest of the track my attention is undivided. There is simply a lot of awesome packed into the 4-odd minutes of music here. I love the demostyle in the last half of the track and the ending. By the time it was over, I just went, "OK, let's listen to this again right away" and it seems the song gets better with every listen. Knocked it out of the park, Stephen Walking. And the cowbell just cracks me up. Delightful music.
[Edit] While I'm posting. London Elektricity is also back with new music on Are We There Yet?. I'm really happy with the album, it sounds fresh and inspired. I already featured the uplifting "Why Are We Here?" on the above mentioned FOTM playlist when they released it as a single a couple of weeks ago but the full album is out now. Definitely a worthy addition to the London E catalogue. I have no idea what is up with the song name "Telefunken Lizard Filter" but somehow... the song sounds just like what a telefunken lizard filter should sound like? Fucking brilliant actually, love it.
- Was it down for everyone?
Was the server just unreachable for 3 days? I think I botched the network settings on it after replacing a PSU. I was just happy when I could connect from my local network, and it didn't occur to me to check with an external connection... @#%£ me. Oh well, still alive and kicking!