As some of you already noticed, the FFB game server has been acting a bit flaky recently. I've been hard at work trying to figure it out, and am still working on it.

I truly apologize for the inconvenience this instability has caused you, and continues to cause.

For those of you who are not interested in technical details, you can stop reading now as I'll simply be explaining the problems and background for the rest of this post.
A couple of weeks ago, the game server all of a sudden became unstable. People had trouble starting games and the server was asking for passwords and all sorts of craziness it shouldn't do.

This has happened before, and was an effect of a back-end Java server running out of memory due to a slow memory leak. Because of the incredible complexity of Java enterprise edition, I didn't bother to identify and fix this memory leak so instead I simply restarted the java service every now and again and things were fine. At the time, I didn't think much of it. Simple restart of the Back-end service and all would be good.

Wrong.

It turned out that a restart didn't fix the problem. Now, I've long since wanted to migrate the java back-end stuff to a less annoying platform to deal with (node.js), so I took the opportunity to rewrite this part of the back-end services to node.js. I already had most components for it in place and it only took like a day to get that sorted out. Things were back to normal again.

For a while.

Before this last weekend, the same types of problems appeared again, and it was worse than ever. The server was running fine for a few minutes and then all of a sudden it died again, going into this broken state where no games could be played. No error messages, no warnings, no nothing. It just stopped and kicked people from the games they were in.

So, I have now spent most of my time this weekend on doing another rewrite of this subsystem (it's a one-time password process used to authenticate you to the FFB server) in PHP. This eliminates some parts of the process (communication to a back-end service), but is less elegant.

This is what I did last night until way too late (3am). I got it functioning, restarted the servers. People were playing games again. Great.

Or so I thought.

At around 7am, the server went into this broken state again. 4 hours. Better than before, but way worse than what it has been in the past (weeks if not months of uninterrupted functionality).

So. I am now at a point where I've done a quick little system update (changing the number of open files allowed on the system), and will evaluate if that makes any difference. In parallel, I am going to be installing a new virtual machine that runs the game server instead of the existing physical machine (currently on older hardware).

If that doesn't work, I will end up having to dig into the FFB Server code and try to identify what is happening. At this point, I am not receiving any kind of error or warning from the system. It simply stops responding without anything in the log output.

All this comes at a pretty bad time, where I am both lacking in time available and am in need to do other updates for the site (GDPR related stuff), and a pretty hefty load in my day job. Rest assured, though, I am working as hard as I can on this and want to have it resolved as soon as possible.

Again, I am truly sorry for messing up your games.

Edit: We've gone past the most critical part of this issue and there's a blog about the bug explaining it in detail.