50 coaches online • Server time: 01:02
* * * Did you know? 189 games were played yesterday.
Log in
Recent Forum Topics goto Post Grotty Little Tourna...goto Post The Light and The Da...goto Post TournyS(3!):Slaughte...
Christer
Online
Khemri Tomb Kings
Star
Khemri Tomb Kings
Record
59/24/37
Win Percentage
59%
Shambling Undead
Super Star
Shambling Undead
Record
51/5/10
Win Percentage
81%
Overall
[R]
Star
Overall
Record
229/56/79
Win Percentage
71%
Archive

2019

2019-04-14 23:33:08
rating 6
2019-04-07 16:59:39
rating 6
2019-04-07 00:55:26
rating 6
2019-01-08 15:27:38
rating 5.9
2019-01-05 02:58:18
rating 5.8

2018

2018-08-17 17:28:31
rating 6
2018-08-15 00:05:40
rating 6
2018-07-17 20:17:40
rating 6
2018-06-28 14:28:08
rating 5.9
2018-05-23 17:55:10
rating 6
2018-05-10 22:42:46
rating 6
2018-05-09 19:42:28
rating 6
2018-04-30 10:44:23
rating 5.8
2018-04-23 12:33:02
rating 5.8

2017

2017-04-23 18:06:35
rating 6
2017-04-06 23:00:56
rating 6
2017-04-03 19:06:00
rating 6
2017-03-29 22:35:46
rating 6
2017-03-25 16:18:39
rating 6
2017-03-11 21:24:26
rating 6
2017-02-14 14:23:58
rating 6
2017-02-10 14:54:03
rating 6

2016

2016-11-30 00:04:21
rating 6
2016-11-27 23:40:04
rating 6
2016-11-17 18:18:07
rating 6

2015

2015-09-06 23:59:26
rating 6
2015-01-24 15:56:29
rating 6
2015-01-22 13:10:32
rating 6
2015-01-19 21:20:53
rating 6
2015-01-10 19:03:45
rating 6

2014

2014-09-09 15:35:53
rating 6

2013

2013-04-26 11:48:40
rating 5.7

2012

2012-12-18 17:37:29
rating 5.9
2012-11-18 18:19:19
rating 6
2012-09-25 13:47:16
rating 5.6
2012-08-15 12:31:53
rating 5.9
2012-08-10 23:12:22
rating 5.9
2012-06-27 22:53:48
rating 5.9
2012-04-10 11:56:38
rating 5.9
2012-03-07 13:52:00
rating 5.9
2012-02-16 16:59:56
rating 5.9
2012-02-04 19:00:41
rating 5.3

2011

2011-07-25 23:32:43
rating 5.6
2011-05-23 13:12:52
rating 5.6
2011-02-04 14:26:18
rating 5.4

2010

2010-03-26 11:38:41
rating 5.1
2010-03-01 12:16:53
rating 5.6

2009

2009-12-08 16:40:30
rating 5.8

2008

2008-09-11 14:47:19
rating 4.1
2008-02-26 21:16:54
rating 5.3
2008-01-21 01:01:58
rating 5.6

2007

2007-11-06 21:23:14
rating 5.1
2007-10-16 00:26:11
rating 5.4
2007-09-30 17:10:03
rating 5.4
2007-09-30 12:01:42
rating 5.3
2007-08-09 12:14:57
rating 4.5
2007-08-06 12:02:52
rating 4.9
2007-08-03 17:56:21
rating 5.4
2017-03-11 21:24:26
55 votes, rating 6
Maintenance musings
At the time of writing, I've been working on the server migration for roughly 10 hours straight. I figured I'd give you an idea of what I've been up to during these hours :)

I won't do a full build log like I did with willow as it's generally speaking much the same process in terms of hardware. I took the old server out of the rack case, cleaned it, moved the new server into its place. This particular build had a slight snag, which caused a delay. Basically, deeproot has a lot of network interfaces (5 of them). Given that the x99 platform doesn't have integrated graphics at all, I also needed a graphics card for this setup.

Now, the GPU I had (NVidia GT 710), is passively cooled but has a pretty large heatsink. Large enough to actually stop me from installing it in the PCI-e slot I needed to use (the last one in order to not block another slot), because the server case is built a bit differently than a tower.

So, I spent some time trying to bend the cooling fins out of the way, but eventually simply stop wasting time with that and used the GPU from the old deeproot. If that one goes bad at some point, it's easy enough to buy a new one.

Either way, during this process deeproot got restless and decided to attack me, drawing blood. (What really happened was I accidentally bumped my head into a sharp corner on one of the rack rails). No niggling injury or significant damage was done though. :)

I moved the case back into the rack, and it had now been 3.5 hours. Wow, time goes fast.

Next up, I spent some time reorganising the power cords, which were a messy tangle of cables and extension cords. I had picked up a "Power Distribution Unit" just for this purpose. Effectively a power strip with a fancy name because it's related to 19" server gear.

After this, the hardware work was completed:




Next up, software. Going in to this migration, I know that I had a bunch of stuff remaining to do. In the old setup, deeproot had three primary purposes:

1. Firewall - Keeping things out that don't belong in the server network, and redirecting good traffic to the correct place
2. DNS - Making sure the servers know each-other's names and addresses
3. Mail - Sending outgoing mail to people who sign up, or get PMs with the notification enabled, or forum thread notifications.

The short of it is that I've spent a lot of time on all three of these, for different reasons.

Let's start with mail, because it should have been easy.

Before I even started this migration, I had outgoing mail configured and working on a new virtual machine (puggy). After the migration, I wanted the web site to use the new virtual server as its "forwarder", but for some reason it didn't work. I spent maybe 2 hours troubleshooting this, thinking that there was a DNS problem, thinking there was a firewall issue, thinking I mixed up the cables, tracing MAC addresses, looking at interface statu... And I notice one interface saying "Status: Disconnected". This was the network card that was supposed to carry backend traffic between the servers (including email to be sent out). Go back and look at the bottom right of the image above. That white network cable in the corner. Does it appear to be sticking out a bit too much? How about on this closeup?



Yup. That was it. Pushed it in a couple of millimeters and the NIC status went to "Up", and shortly thereafter, email worked. I had to change some configuration on postfix (the software I use for email) to make it listen on the proper interface, but the major part was the cable. Go figure.

DNS then. Oh my what a mess DNS is. I won't go through all my problems here; safe to say I got to spend quite some time on configuring BIND (the DNS software I, and most of the Internet, use). Compared to the previous configuration (also with BIND), I now have ACLs and Views set up to be able to separate my different networks (I'll detail the network setup in another blog post).

Finally, the firewall. This has been a long journey over the last week. I've spent all my spare time with firewall configuration, trying to get things set up. The core problem came from me switching what Firewall software I use. Before, I was using a Linux package called "Shorewall", a configuration interface on top of the built-in "IP Tables" that comes with Linux. Icepelt, the new virtual machine that does firewalling (and DNS and DHCP) is running pfSense, a FreeBSD based very very popular firewall suite.

pfSense works very differently from Shorewall (or IPTables in general), which has taken me many many hours (40+) to get a grip of, trying various configurations and restarting over and over again until I ended up with a setup I'm comfortable with.

It's not a bad platform; in fact, it's nice to have a GUI this time around instead of having to look at an endless list of firewall rules. It's quite different though, and has plenty of strange quirks.

Even after the hours on end of preparation, I still had to do the final config in the actual environment of the site. This took a while to sort out, but this "live work" has simply reinforced my belief that my understanding of pfSense is sound and works well.

So what's next?

Well, I still have some residual stuff to work on. My primary concern for now is that "Hyper-V Manager" refuses to connect to (new) deeproot. This is something that worked before I moved deeproot into place, but doesn't anymore. I have no clue why at this point but will be poking around with until I get it working again. It's technically possible to do it all using powershell, but I'd rather not :)

After that, I need to document this new network setup I have, so I know where things are connected and what's what.

And then it's back to FUMBBL code; getting BowlBot on Discord to announce Blackbox draws is high on my list.

(Ok, so that took like an hour to write.. Why can't things be quicker? :)
Rate this entry
Comments
Posted by Dominik on 2017-03-11 21:35:36
Thank you for your neverending commitment for this game. Running a website with steady workload for 15 years with no end in sight is absolutely amazing.
Posted by Arktoris on 2017-03-11 22:32:21
I have seen the Matrix...

and I want to swallow the blue pill.

Posted by DonnyRainboe on 2017-03-11 23:21:53
Yes, thank you for the commitment.
Posted by grunth on 2017-03-11 23:32:30
Thank you so much Christer... All the work done for our pleasure !
Posted by Endzone on 2017-03-11 23:36:23
Great work Christer, what a legend.
Posted by SpecialOne on 2017-03-11 23:43:39
thank you doesn't cover. But it is all I can say. Oh and I must get that silver donation tag away!
Posted by Sir_Vivor on 2017-03-11 23:46:37
awesome!
Posted by mister__joshua on 2017-03-11 23:54:50
If you worked in IT support, cable not plugged in is the first thing you'd have looked for ;)
Posted by Subs on 2017-03-12 00:04:02
Nice job! Thx for time and the blood you put in it! :]
Posted by Christer on 2017-03-12 00:11:52
The reasons it took so long before I checked the cable:

1. I connected them all in the first place. Who fails to connect a network cable anyway? :)
2. There are so many other things that could be wrong, that felt more likely than a cable problem.
3. The rack had been rolled into its place. Getting to the cables means rolling the (~200kg) rack out of the corner where it is in order to get access to the rear of the computers. It's on wheels, but still a bit of a pain to do..
Posted by Eisenherz on 2017-03-12 00:20:16
"its definitely not the cable, i just repaired it yesterday!" Ah, good old LAN-Party memories! I have been playing on this site for astonisihing 6 years (on and off) and i enjoy it now more than ever. THank you so much!
Posted by Rabe on 2017-03-12 00:34:33
Crazy stuff. Kudos!
Posted by Stanley on 2017-03-12 00:41:53
And all this pretty much just so we can play a game?
not enough thankyous could justify the amount of work you put into this.
Posted by awambawamb on 2017-03-12 03:09:10
so tidy <3
Posted by MattDakka on 2017-03-12 05:22:35
Infinite thanks for the work, passion, commitment and effort you put into this dream site!
Posted by Malmir on 2017-03-12 07:16:56
Awesome, just awesome.
Posted by RedMaul on 2017-03-12 12:12:45
Thank you Christer, I appreciate it.
Posted by Gartch on 2017-03-12 12:12:53
Thank you for all the time you spent for the site.
Posted by krom72 on 2017-03-12 15:08:05
looks shiny ;-)
Posted by Miceycraft on 2017-03-12 15:24:50
Amazing. Can't thankyou enough for your continued work
Posted by Throweck on 2017-03-12 17:15:21
Thanks Christer. Remember...if anything goes wrong in the future, heed the advice I gave you. Turn it off and on again. :P
Posted by Superlynx on 2017-03-12 18:55:36
You're literally bleeding for us and we owe you so much. I cannot say how much I thank you Christer. I also have the feeling that this site serves as a stress relief for many of us users. So what you're doing is nothing less than charity work.
Posted by Dhaktokh on 2017-03-12 21:12:33
First of all, thanks a LOT for all your work Christer! We all owe you!

2nd, It would be great to see a full view of the whole "heart of FUMBBL", with all stuff required to run the site (me is teh tech-noob). :-D
Posted by JellyBelly on 2017-03-12 23:18:19
The amount of time and effort you spend on the site is truly staggering, Christer. I propose that, once Deeproot is done, take a vacation and forget about FUMBBL for a couple of weeks. Bowlbot announcing box draws can wait!

Also, I have to ask ... that network cable labelled 'pub' ... Is that your direct line to order a beer? ;)
Posted by Christer on 2017-03-12 23:48:52
It'd be great if it was, right?

WAN = Wide Area Network, e.g. Internet
LAN = Local Area Network, my desktop computers
MGM = Management Network, for administrative stuff
PUB = Public Network, public traffic, such as web site and FFB Server stuff
BCK = Backend Network, for traffic between servers
Posted by Severedunit on 2017-03-13 02:57:14
Hire more e-squigs?
Posted by Roland on 2017-03-13 11:22:00
Do you have a link to previous build blog? I can't seem to find it.
Posted by Christer on 2017-03-13 12:28:52
https://fumbbl.com/p/notes?op=view&id=2192

That's the Borak upgrade.
Posted by Roland on 2017-03-13 14:35:39
Thx!
Posted by Badoek on 2017-03-13 14:56:00
e-gaping
Posted by Xeterog on 2017-03-13 15:55:10
great cable management. So many times I've seen in production set ups where the cables are all a tangled mess and not labeled at all.
Posted by JellyBelly on 2017-03-13 19:04:43
He he. I thought it was probably 'public', but the other option was more amusing :)
Posted by ceetee on 2017-03-13 20:22:49
Thanks Christer, epic work.