Be a Supporter!

Post #9: Load Balancer

  • 5,507 Views
  • 83 Replies
Respond to this Topic
Tim
Tim
  • Member since: Jul. 29, 2000
  • Offline.
Forum Stats
Member
Level 06
Blank Slate
Post #9: Load Balancer 2005-04-28 16:22:29 Reply

Short version
-------------------------------------
Load Balancer is up and running. Some people may have to wait a few hours for dns to update completely, although this should be < 10% of users.

Long Version
--------------------------------------

A long time ago in a galaxy far far away, I set out on a journey to rid the world of dust mites... Hmm, that's not right. Now that I think about it, it was about 9 months ago in Philly, it was more of a project than a journey, and getting a load balancer has nothing to do with dust mites. Ah well, so much for a good intro.

Anyway, in the fall of last year I talked to Tom about getting a load balancer for the site. For those of you who aren't familiar with such a product, I'll give a brief explanation. When a website has multiple webservers, there are several ways to distribute the traffic, or load. The simplest and most basic way to do this is with "round robin dns". In a RRDNS configuration, the ip's of several webservers are listed under the same DNS name. For example, previously you'd see something similar to the following:

# nslookup newgrounds.com

Name: newgrounds.com
Address: 127.53.32.232
Name: newgrounds.com
Address: 127.69.132.47
Name: newgrounds.com
Address: 127.231.39.49
Name: newgrounds.com
Address: 127.39.130.90
Name: newgrounds.com
Address: 127.74.38.203
Name: newgrounds.com
Address: 127.100.22.58
Name: newgrounds.com
Address: 127.240.28.22
Name: newgrounds.com
Address: 127.190.36.14

When your computer would connect to newgrounds, it would randomly select one of those IP's to connect to. As I said before, this solution is free to implement and very easy, however it has two major problems. First, if any one of the servers goes down or gets taken down for maintenance, a certain portion of traffic (1/8 in this example) would be going to a dead server and updating the DNS tables to remove the server can take hours. Second, anytime something is done "randomly", bad things can happen. In this example, every now and then too many people would "randomly" select the same address and overload the server. As a result, even though there was plenty of spare capacity on the other servers, whoever went to the overloaded one would receive an error. Obviously, this configuration is far from ideal.

The alternative to RRDNS is to use a load balancer. What the load balancer does is direct traffic to the appropriate server behind it. So, instead of having multiple IP's allowing people to connect directly to the webservers, you have a single IP which routes traffic to the least loaded server. This configuration has only one major problem; it's basically a single point of failure. That being said, it's generally more reliable than any software based server and is easily replaced if something bad were to happen; and since it eliminates both of the problems associated with RRDNS configurations, the overall result is a much more robust and capable website. Clearly, this is the way to go.

The idea was agreed upon and the wheels were set in motion. I've already written too much so I'll try to summarize the next 9 months with an analogy. Think of the project as a program where at the end is an "install Load Balancer" command with an endless list of nested if/then else statements before it. Now, imagine if all of those if/then statments resolved to "true". For example, if network switch is out of ports, replace network switch. Or, if fileserver is overloaded, replace fileserver. Or my personal favorite, if salesman misspoke about capabilities of initial Load Balancer, spend months returning the original and re-planning/re-ordering for the better model.

So here we are in the Spring of 2005 and I'm finally getting this thing up and running. I'm sorry it's taken so long and hope that this eliminates many of the connection problems the user community has had over the years. This was a major step but don't think we're done improving the site, it's constantly evolving (ordering up a 64bit DB server very soon). You may still have problems getting to the site from time to time, although these should be at off-peak times and related to maintenance rather than to an overloaded server. Now we just have to wait for the next Numa Numa to come along for a real test :)

Tim out

JackRabbitStudios
JackRabbitStudios
  • Member since: Oct. 3, 2003
  • Offline.
Forum Stats
Member
Level 23
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 16:26:19 Reply

Glad the whole thing worked out. Hope the sites next project to improve itself is as successful.


"Eggplant tastes like eggplant....but meat tastes like murder and murder tastes pretty God damn good, doesn't it?"-Dennis Leary (No Cure for Cancer)

BBS Signature
Chumbawamba
Chumbawamba
  • Member since: Aug. 13, 2004
  • Offline.
Forum Stats
Member
Level 48
Audiophile
Response to Post #9: Load Balancer 2005-04-28 16:27:01 Reply

So the best time to come on Newgrounds is when all of the kiddies are in school? Alright, but that will be a little difficult for me, seeing as I am a kid. =/

I hope Gary comes out with a new movie soon so some action can take place. >:)


BBS Signature
Chumbawamba
Chumbawamba
  • Member since: Aug. 13, 2004
  • Offline.
Forum Stats
Member
Level 48
Audiophile
Response to Post #9: Load Balancer 2005-04-28 16:29:21 Reply

Also, it looks like I'm not part of that 10% that needs to wait a few hours. Things are running swimmingly over here!


BBS Signature
RupeeClock
RupeeClock
  • Member since: Aug. 31, 2004
  • Offline.
Forum Stats
Member
Level 57
Animator
Response to Post #9: Load Balancer 2005-04-28 16:31:14 Reply

Hmm, so if each time we log onto the site we are given a different IP adress to access the site through, does this make it possible to vote on the same movie twice? I hope not.... Otherwise we could get one person voting on the same movie 8 times in one day...

Chumbawamba
Chumbawamba
  • Member since: Aug. 13, 2004
  • Offline.
Forum Stats
Member
Level 48
Audiophile
Response to Post #9: Load Balancer 2005-04-28 16:32:38 Reply

At 4/28/05 04:31 PM, FrOzEn_FoX wrote: Hmm, so if each time we log onto the site we are given a different IP adress to access the site through, does this make it possible to vote on the same movie twice? I hope not.... Otherwise we could get one person voting on the same movie 8 times in one day...

I think our own IP address will account for that, so no mass voting.


BBS Signature
Sk8erGirl14
Sk8erGirl14
  • Member since: May. 5, 2003
  • Offline.
Forum Stats
Member
Level 34
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 16:34:00 Reply

no, things seem to be running pretty good for me...glad to hear everything is back to normal and underway...


well, like, you couldn't, like, find it because, like, you're dumb, god
Follow me on twitter :3

BBS Signature
lightning
lightning
  • Member since: Apr. 12, 2002
  • Offline.
Forum Stats
Member
Level 41
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 16:41:41 Reply

Ng god has done it again. :)

RupeeClock
RupeeClock
  • Member since: Aug. 31, 2004
  • Offline.
Forum Stats
Member
Level 57
Animator
Response to Post #9: Load Balancer 2005-04-28 16:42:44 Reply

At 4/28/05 04:37 PM, jump3rboi wrote:
At 4/28/05 04:31 PM, FrOzEn_FoX wrote: Hmm, so if each time we log onto the site we are given a different IP adress to access the site through, does this make it possible to vote on the same movie twice? I hope not.... Otherwise we could get one person voting on the same movie 8 times in one day...
If that ever does happen, it'll be abused like fuck, just like when NG used to give referrals.

I wasn't here when that happened but referals only cause all sorts of trouble across the net, outwar encourages real bad spam, iPods encourage scams and possibly fill your inbox with more spam and of course the referal system at this site resulted in loads of alt accounts and abuse, this was of course a REAL bad way to get EXP, maybe there should be another method of getting exp but limiting it and making it fair....

KingOfTheKill
KingOfTheKill
  • Member since: Mar. 18, 2004
  • Offline.
Forum Stats
Member
Level 16
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 16:45:01 Reply

Awesome. Hopefully this will get rid of those pesky errors that I keep getting.

n0ir
n0ir
  • Member since: Nov. 7, 2003
  • Offline.
Forum Stats
Member
Level 18
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 16:49:38 Reply

I'd like to congratulate you on fixing all that server stuff.
And I would like to give you props in advance for making it into the double digits; post count!


BBS Signature
ramagi
ramagi
  • Member since: Apr. 6, 2001
  • Offline.
Forum Stats
Member
Level 59
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 17:01:29 Reply

Wow cool, that is great to hear, fun stuff.

RupeeClock
RupeeClock
  • Member since: Aug. 31, 2004
  • Offline.
Forum Stats
Member
Level 57
Animator
Response to Post #9: Load Balancer 2005-04-28 17:05:31 Reply

From what I understand anyway, the site should run better now because there will now be 8 different servers or something to handle the same ammount of users and they will be managed to have roughly the same ammount of users on each of them, ending up with having to share less traffic with other users and a better browsing experience, that right?

CodeRedClock
CodeRedClock
  • Member since: Jul. 22, 2003
  • Offline.
Forum Stats
Member
Level 31
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 17:09:31 Reply

At 4/28/05 04:27 PM, Kazuo_Kiriyama wrote: So the best time to come on Newgrounds is when all of the kiddies are in school? Alright, but that will be a little difficult for me, seeing as I am a kid. =/

I hope Gary comes out with a new movie soon so some action can take place. >:)

no thank you. log off.

deckheadtottie
deckheadtottie
  • Member since: Oct. 21, 2003
  • Offline.
Forum Stats
Supporter
Level 59
Programmer
Response to Post #9: Load Balancer 2005-04-28 17:22:18 Reply

At 4/28/05 05:09 PM, B_M_W wrote: no thank you. log off.

Better yet.

Post #9:  Load Balancer


#coys

BBS Signature
Chumbawamba
Chumbawamba
  • Member since: Aug. 13, 2004
  • Offline.
Forum Stats
Member
Level 48
Audiophile
Response to Post #9: Load Balancer 2005-04-28 17:27:27 Reply

At 4/28/05 05:22 PM, deck_head_tottie wrote:
At 4/28/05 05:09 PM, B_M_W wrote: no thank you. log off.
Better yet.

Even better (I don't care if I'm insulting myself).

Post #9:  Load Balancer


BBS Signature
KingOfTheKill
KingOfTheKill
  • Member since: Mar. 18, 2004
  • Offline.
Forum Stats
Member
Level 16
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 17:36:08 Reply

At 4/28/05 05:05 PM, FrOzEn_FoX wrote: From what I understand anyway, the site should run better now because there will now be 8 different servers or something to handle the same ammount of users and they will be managed to have roughly the same ammount of users on each of them, ending up with having to share less traffic with other users and a better browsing experience, that right?

Not exactly. There were already eight servers, but the program that controlled who went where when they logged in sent them to the diferent servers randomly. This was a problem because some of the servers would occasionally be sent too many people and crash. Now, it's been changed so that when you log in you are automatically sent to the server with the least people currently logged in on it, virtually eliminating the chances of the servers crashing. Okay?

DarkArchon
DarkArchon
  • Member since: Sep. 4, 2003
  • Offline.
Forum Stats
Member
Level 47
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 18:06:17 Reply

Good job, tim ;)

Great to see new and exciting server side things happening, and good to read and farmiliarize myself better with how the load balancer works.

Thanks for the quick teaching lesson ;D


My Email -- DarkArchon@Newgrounds.com ... LOL WOW this signature is ancient.

BBS Signature
VeryProudofYa
VeryProudofYa
  • Member since: Oct. 2, 2004
  • Offline.
Forum Stats
Member
Level 23
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 18:06:30 Reply

At 4/28/05 04:22 PM, Tim wrote:

Tim out

Seacrest is straight.
And if you say otherwise, I will deny it up and down


Everything is everything

BBS Signature
Miles
Miles
  • Member since: Jun. 23, 2002
  • Offline.
Forum Stats
Member
Level 28
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 18:58:52 Reply

Pretty sweet! NG is geting better and better


BBS Signature
Pilot-Doofy
Pilot-Doofy
  • Member since: Sep. 13, 2003
  • Offline.
Forum Stats
Member
Level 37
Musician
Response to Post #9: Load Balancer 2005-04-28 19:05:33 Reply

At 4/28/05 04:22 PM, Tim wrote: Tim out

Tim, you're so damned sexy.

KingPanther13
KingPanther13
  • Member since: Nov. 18, 2004
  • Offline.
Forum Stats
Member
Level 22
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 19:07:29 Reply

Alright go TIm! Finally, this site will start to speed up again. And, for Tim, you're posting up a storm! Already 6 posts this year! Amazing! Anyhow, thanks for all of the hard work you do, and for making NG a better place :)

Eldarion
Eldarion
  • Member since: Oct. 19, 2003
  • Offline.
Forum Stats
Member
Level 20
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 19:28:15 Reply

Thank you, Tim. Not only for getting the load balancer up and running, but taking the time to make such an informative post. When's the next lesson? :)

Steve84
Steve84
  • Member since: Mar. 25, 2000
  • Offline.
Forum Stats
Member
Level 42
Gamer
Response to Post #9: Load Balancer 2005-04-28 20:00:42 Reply

Tim that ais all awsome It's even faster then before keep it up man

asdfrasdfg
asdfrasdfg
  • Member since: Oct. 13, 2004
  • Offline.
Forum Stats
Member
Level 13
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 20:05:23 Reply

This is actually a really good thing for someone like me. Recently, Charter decided that they would make our service slower...well, they didn't actually make it slower, but for somehow they did something that caused everyone to move at 56k speeds from time to time. It really sucks, because that, in combination with a slow server, makes loading Newgrounds too frustrating to try and deal with, so I sometimes just end up leaving because it takes so long to load.
Hopefully, it won't be nearly as bad now.

joshuaost
joshuaost
  • Member since: Sep. 18, 2004
  • Offline.
Forum Stats
Member
Level 24
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 20:13:08 Reply

Nice, I've been waiting for something like this to happen.

krA
krA
  • Member since: Feb. 13, 2005
  • Offline.
Forum Stats
Member
Level 08
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 20:22:50 Reply

there's only 3 people in the general bbs!

KingPanther13
KingPanther13
  • Member since: Nov. 18, 2004
  • Offline.
Forum Stats
Member
Level 22
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 20:25:43 Reply

At 4/28/05 08:22 PM, krA wrote: there's only 3 people in the general bbs!

This is the news section. It almost fooled me, except for the fact, that every topic was news.

Stamper
Stamper
  • Member since: Feb. 13, 2000
  • Offline.
Forum Stats
Member
Level 05
Melancholy
Response to Post #9: Load Balancer 2005-04-28 20:46:47 Reply

At 4/28/05 04:22 PM, Tim wrote: Short version
-------------------------------------
Load Balancer is up and running. Some people may have to wait a few hours for dns to update completely, although this should be < 10% of users.

Long Version
--------------------------------------

Mmmm...

God I fucking love it when Tim gets all technical. Makes me all hot and shit.


YO MAMA GOT SAGGY TITTIES LMAO

BBS Signature
oc-fuzzy
oc-fuzzy
  • Member since: Nov. 7, 2004
  • Offline.
Forum Stats
Member
Level 14
Blank Slate
Response to Post #9: Load Balancer 2005-04-28 20:53:06 Reply

way to long confusing and boring post to read but if you wrote that much it must be important


You're so pretty, like a tree, or a high-class prostitute.

BBS Signature