59 coaches online • Server time: 22:42
* * * Did you know? The best interceptor is Leena with 22 interceptions.
Log in
Recent Forum Topics goto Post Linux (Ubuntu) - can...goto Post GIFgoto Post Flings in the new ed...
Christer
Last seen 4 hours ago
Khemri Tomb Kings
Star
Khemri Tomb Kings
Record
59/24/37
Win Percentage
59%
Shambling Undead
Super Star
Shambling Undead
Record
51/5/10
Win Percentage
81%
Overall
[R]
Star
Overall
Record
228/56/79
Win Percentage
71%
Archive

2019

2019-04-14 23:33:08
rating 6
2019-04-07 16:59:39
rating 6
2019-04-07 00:55:26
rating 6
2019-01-08 15:27:38
rating 5.9
2019-01-05 02:58:18
rating 5.8

2018

2018-08-17 17:28:31
rating 6
2018-08-15 00:05:40
rating 6
2018-07-17 20:17:40
rating 6
2018-06-28 14:28:08
rating 5.9
2018-05-23 17:55:10
rating 6
2018-05-10 22:42:46
rating 6
2018-05-09 19:42:28
rating 6
2018-04-30 10:44:23
rating 5.8
2018-04-23 12:33:02
rating 5.8

2017

2017-04-23 18:06:35
rating 6
2017-04-06 23:00:56
rating 6
2017-04-03 19:06:00
rating 6
2017-03-29 22:35:46
rating 6
2017-03-25 16:18:39
rating 6
2017-03-11 21:24:26
rating 6
2017-02-14 14:23:58
rating 6
2017-02-10 14:54:03
rating 6

2016

2016-11-30 00:04:21
rating 6
2016-11-27 23:40:04
rating 6
2016-11-17 18:18:07
rating 6

2015

2015-09-06 23:59:26
rating 6
2015-01-24 15:56:29
rating 6
2015-01-22 13:10:32
rating 6
2015-01-19 21:20:53
rating 6
2015-01-10 19:03:45
rating 6

2014

2014-09-09 15:35:53
rating 6

2013

2013-04-26 11:48:40
rating 5.7

2012

2012-12-18 17:37:29
rating 5.9
2012-11-18 18:19:19
rating 6
2012-09-25 13:47:16
rating 5.6
2012-08-15 12:31:53
rating 5.9
2012-08-10 23:12:22
rating 5.9
2012-06-27 22:53:48
rating 5.9
2012-04-10 11:56:38
rating 5.9
2012-03-07 13:52:00
rating 5.9
2012-02-16 16:59:56
rating 5.9
2012-02-04 19:00:41
rating 5.3

2011

2011-07-25 23:32:43
rating 5.6
2011-05-23 13:12:52
rating 5.6
2011-02-04 14:26:18
rating 5.4

2010

2010-03-26 11:38:41
rating 5.1
2010-03-01 12:16:53
rating 5.6

2009

2009-12-08 16:40:30
rating 5.8

2008

2008-09-11 14:47:19
rating 4.1
2008-02-26 21:16:54
rating 5.3
2008-01-21 01:01:58
rating 5.6

2007

2007-11-06 21:23:14
rating 5.1
2007-10-16 00:26:11
rating 5.4
2007-09-30 17:10:03
rating 5.4
2007-09-30 12:01:42
rating 5.3
2007-08-09 12:14:57
rating 4.5
2007-08-06 12:02:52
rating 4.9
2007-08-03 17:56:21
rating 5.4
2012-11-18 18:19:19
58 votes, rating 6
IT infrastructure
10 years ago, when FUMBBL was started, the site ran on a single computer. This was a cheap standard desktop machine and didn't have any special hardware at all.

With the popularity of the site growing and the hardware wasn't up to par with the load, I upgraded the server to a more powerful machine, taking away the older low-powered one. The new machine was still desktop class (ie, no server-grade CPUs or memory, and this is still the case today), but much more powerful. As time progressed and the site got more and more popular and I added more and more features, this machine ended up not having enough power.

So, I purchased a second machine for the site and installed the database on it. This was a big upgrade in terms of hardware resources and I was at the time working hard on optimizing the code to run better (making sure DB queries had relevant indexes and things like that). The site continued to grow and I upgraded hardware as necessary to cope with the load. Although instead of completely getting rid of some of the old machines, they were instead delegated to take care of less resource intensive tasks.

In the end, we built up to the 4 server setup we use today. Web server, Database server, Game/IRC server and a firewall. Due to the way this build up organically, I never put any great thought into how the servers were networked. Until a couple of weeks ago, all servers were simply set up in a single network (LAN) for the servers and one separate LAN for the desktop machines I have in my home. There was simply a small home switch in the rack cabin where the servers reside, and each server had a single cable connected to that switch.

Now, a couple of weeks ago, I felt it was time to get rid of the small home-grade switch in the rack cabin (mostly because it was loose in the rack and it was starting to annoy me). So I went on a hunt for a relatively cheap rack-mounted switch. The switch I found (and bought) ended up being more enterprise grade than I expected to get for a reasonable price (the switch is the Netgear GS724T, a so called "smart switch").

One major feature of the switch is what's called VLAN support. In simple terms, this feature allows the switch to be split up into multiple parts and effectively make it function as if there are multiple, separated, switches instead of a single one.

With this feature available to me, I decided to take a step back and build a more enterprise-grade network for the servers. So I sat down with my graphing tool and planned what I wanted to do. This is what I came up with:



As you can see there are four networks in the chart in the network. The Public network is meant for web and game server traffic. The Backend network is meant for database traffic. The Management network is meant for management traffic (for example backups and my own administration). Finally, the LAN network is there to allow me to separate my own machines from the server networks.

To actually implement this, I needed to buy a few extra network cards and a bunch of network cable (for those of you who care, the server is strictly shielded (FTP) CAT.6 cable). Funny fact: The cables cost almost as much as the network cards these days...

Here's where I ended up on a tangent that cost me 14 hours of work... I bought the network cards I needed, and started to install them (one of the downtimes over this weekend)... only to notice that one of the servers needed a PCI-express card instead of the PCI cards I had. So, off to the store again to pick up a new card. So that done, I installed the network card (downtime #2).. only to find that it wasn't functioning properly. Le sigh... This was supposed to be the card for the cable between Deeproot and the Public network.

Ok, so I started to think "How do I get past this?", and came up with a plan involving combining the LAN and Public networks over a single cable (fancy feature normally called Trunking in network speak). Netgear's smart switch, however, does this a bit differently and I would need to do something called "VLAN tagging" on the traffic bound for at least one of these logical networks. Ok, so far so good. I had a plan. Unfortunately, there's a saying which goes like this: "No plan survives contact with the enemy". It turns out that the OS that was installed on Deeproot was old and the system was in a pretty weird state (functional, but not properly updated over the years and effectively I couldn't update things), meaning I could not enable the VLAN tagging feature. So I think "Ok then. I'll set up a virtual machine for the firewall and all will be well". But nope. The virtualization system was also not set up properly, and couldn't be installed.

And here comes downtime #3, a longer one. What I ended up doing was to reinstall the whole system (now running Ubuntu Server 12.04 LTS) to get it into a working state. An hour or so later, the machine was a freshly installed system and I spent another hours configuring the most basic things (network cards, firewall rules and mail server mainly). Funnily enough, the network card that was not working properly under the old OS works fine now with the new Linux kernel. This allowed me to skip the whole VLAN tagging thing entirely and go back to the original plan.

At this point, there was no real change from before. All traffic was still running on the same network (the management one). Now was the time to start moving the traffic over to these newly created networks.

So I start with the easiest traffic, the traffic from the internet (you guys) to the website. I went over to the firewall and changed the forwarding for the web traffic to go over the Public network. Easy, right? Wrong. Everything stopped, no-one could access the site from the outside.

So off I go on tangent #2. It turns out that Debian, which is what Ubuntu is based on, and also which is what is installed on fungus (the web server machine), changed the default behaviour of the firewall software I use (Shorewall) to drop packets that are considered "martians" (the name being the reason I'm writing this part :) ). In this context, a martian is a network packet coming in on a network card that originates from an IP which is not associated with that network card (yeah, it takes a while to wrap your head around that one). So, I loosened up the rules on that a bit, and things started to work properly again.

Back on track, the next thing I moved was the traffic between the web server and the database. This move was relatively pain-free and I want to think that the site became quicker by doing this (it feels quicker to me, but I don't have any actual benchmarks for this).

This is where things are now. There are a few more things I need to move (for example, game server database traffic hasn't been moved yet and currently goes over the management network). Also, I am contemplating limiting management traffic a bit further and setting up a VPN to get access to it in order to improve security.

In the end, this is all a big learning experience for me and something I wouldn't be likely to research in the details I have if I didn't have FUMBBL. At times, it's extremely frustrating to deal with but I truly am happy when things are working as I want them to and that makes all this stuff worth the effort to me. After this weekend, FUMBBL has a network that is pretty much enterprise-grade and that makes me happier. In theory, it should be faster as well, despite me always having had a full gigabit network.

This turned out to be a long blog entry.. Hope you enjoyed it :)
Rate this entry
Comments
Posted by PigStar-69 on 2012-11-18 18:25:13
none of this made any sense to me but sir i certainly appreciate the work you have put in.

where would we be without you ;-)
Posted by andr_e on 2012-11-18 18:25:43
It's simply too much for me :P
Well done :D
Posted by Jasfmpgh on 2012-11-18 18:28:56
Thanks for all the time and effort you put in Big C.
Posted by Endzone on 2012-11-18 18:32:10
I was with you all the way upto "10 years ago".

Great work - thanks!
Posted by MattDakka on 2012-11-18 18:39:59
Thanks for all your hard work and all the time spent for us!
Posted by Overhamsteren on 2012-11-18 18:40:16
You sir are a gentleman and a wizard
Posted by ClayInfinity on 2012-11-18 19:00:17
Rated 6 for the picture! Cool graphics Christer! And those Cyanide guys think they have graphics!!
Posted by Ehlers on 2012-11-18 21:32:22
One StarPlayer to rule them all, one StarPlayer to find them, One Starplayer to bring them all and in the darkness bind them

Awesome blog, awesome Deeproot rule them all.

Did not understand any of it, but nice with a storytelling before nighttime.
Posted by SavageJ on 2012-11-18 23:19:37
I'm no network expert, but I can follow this. It looks like a good design to me. Smart switches are cool. :-)
Posted by Qaz on 2012-11-18 23:23:25
Borak is the gaming server (if memory serves me right) And he has dirty player what does that tell you!

Great Work Christer the things you wrap your brain around.
Posted by Garion on 2012-11-18 23:43:56
Great stuff christer, also love the new improvement you made to the current games page.
Posted by Kelkka on 2012-11-19 00:12:56
You could easily put up your own company, manage the IT side and program stuff to make the profit. Absolutely impressive how one man can do it all, very nice :)
Posted by the_Sage on 2012-11-19 00:37:00
Rated 6 for Deeproot
Posted by Jeffro on 2012-11-19 01:01:59
Rated 6. There's a team theme in here somewhere, and as soon as I sober up...
Posted by Dhaktokh on 2012-11-19 11:08:38
Wizard!!
Posted by Badoek on 2012-11-19 15:58:50
all this and no plead for donations? amateur!
still rated 6 though :D