49 coaches online • Server time: 13:28
* * * Did you know? Up until now, 1231988 players have died on the pitch.
Log in
Recent Forum Topics goto Post SWL Season LXXIIIgoto Post Best Metagroup on FU...goto Post New rosters in R&...
Posted by: Christer on Monday, May 21, 2018 - 17:50
As some of you already noticed, the FFB game server has been acting a bit flaky recently. I've been hard at work trying to figure it out, and am still working on it.

I truly apologize for the inconvenience this instability has caused you, and continues to cause.

For those of you who are not interested in technical details, you can stop reading now as I'll simply be explaining the problems and background for the rest of this post.

A couple of weeks ago, the game server all of a sudden became unstable. People had trouble starting games and the server was asking for passwords and all sorts of craziness it shouldn't do.

This has happened before, and was an effect of a back-end Java server running out of memory due to a slow memory leak. Because of the incredible complexity of Java enterprise edition, I didn't bother to identify and fix this memory leak so instead I simply restarted the java service every now and again and things were fine. At the time, I didn't think much of it. Simple restart of the Back-end service and all would be good.

Wrong.

It turned out that a restart didn't fix the problem. Now, I've long since wanted to migrate the java back-end stuff to a less annoying platform to deal with (node.js), so I took the opportunity to rewrite this part of the back-end services to node.js. I already had most components for it in place and it only took like a day to get that sorted out. Things were back to normal again.

For a while.

Before this last weekend, the same types of problems appeared again, and it was worse than ever. The server was running fine for a few minutes and then all of a sudden it died again, going into this broken state where no games could be played. No error messages, no warnings, no nothing. It just stopped and kicked people from the games they were in.

So, I have now spent most of my time this weekend on doing another rewrite of this subsystem (it's a one-time password process used to authenticate you to the FFB server) in PHP. This eliminates some parts of the process (communication to a back-end service), but is less elegant.

This is what I did last night until way too late (3am). I got it functioning, restarted the servers. People were playing games again. Great.

Or so I thought.

At around 7am, the server went into this broken state again. 4 hours. Better than before, but way worse than what it has been in the past (weeks if not months of uninterrupted functionality).

So. I am now at a point where I've done a quick little system update (changing the number of open files allowed on the system), and will evaluate if that makes any difference. In parallel, I am going to be installing a new virtual machine that runs the game server instead of the existing physical machine (currently on older hardware).

If that doesn't work, I will end up having to dig into the FFB Server code and try to identify what is happening. At this point, I am not receiving any kind of error or warning from the system. It simply stops responding without anything in the log output.

All this comes at a pretty bad time, where I am both lacking in time available and am in need to do other updates for the site (GDPR related stuff), and a pretty hefty load in my day job. Rest assured, though, I am working as hard as I can on this and want to have it resolved as soon as possible.

Again, I am truly sorry for messing up your games.

Edit: We've gone past the most critical part of this issue and there's a blog about the bug explaining it in detail.
Game server instability | Login/Create an account | 13 Comments
Comments are owned by the poster. We aren't responsible for their content.
Re: Game server instability
by Kekskommander on May 21, 2018 - 19:52

(User info | Send a message)
No worries. You are doing a great job. If it takes time, it is like it is.


Re: Game server instability
by PeteW on May 21, 2018 - 21:01

(User info | Send a message)
I was going to suggest turning it off and then on again. But I see you did that already. ;P

You are a prince among men, Christer. We can be patient! Don't burn yourself out.


Re: Game server instability
by Gonzas1000 on May 21, 2018 - 21:28

(User info | Send a message) http://
Thanks for all Christer.


Re: Game server instability
by Dietz on May 21, 2018 - 23:15

(User info | Send a message) http://
Don't worry we all understand that this stuff isn't simple, especially as just one person. Fumbbl is still the best fantasy football client in our group's books. We greatly appreciate what you're doing.


Re: Game server instability
by realmalteser2 on May 22, 2018 - 08:47

(User info | Send a message) http://
Take your time :)
and thanks for everything mate!


Re: Game server instability
by Gartch on May 22, 2018 - 14:57

(User info | Send a message)
Don't be sorry, and thank you so much for all the time you spend on this site.


Re: Game server instability
by Guardikai on May 22, 2018 - 15:27

(User info | Send a message) http://
You're a star Christer ^^ please don't overdo it!!!


Re: Game server instability
by NairoD on May 22, 2018 - 19:38

(User info | Send a message)
Christer drop us info what amount of Money are needed for New hardware.
You are mailing marvelous Job. Dont bother with us. We dont deserved it


Re: Game server instability
by DrDeath on May 22, 2018 - 22:10

(User info | Send a message)
That sounds like a real headache, thanks for taking it on! Was going to say you are a star but no you are a LEGEND :)


Re: Game server instability
by magowes on May 26, 2018 - 23:51

(User info | Send a message) http://
Hi,

I wanna say thank you for the effort you are doing.


I don't know if it is here the threat to ask my question. If it isn't, i am sorry.

Due to the server fail, i have got 2 open games that neither me or my oponent can play it.
So, how do i get to close them to be able to play again with these teams?

Thanks a lot