Be a Supporter!

Database Stuff

  • 6,196 Views
  • 43 Replies
Respond to this Topic
egg82
egg82
  • Member since: Jun. 24, 2006
  • Offline.
Forum Stats
Supporter
Level 05
Game Developer
Response to Database Stuff 2012-07-31 16:34:46 Reply

At 7/31/12 11:37 AM, MikeyS9607 wrote: I don't understand from a programming point of view, like when you use an insert query in MySQL, does the server not pick it up till later?

it's a client -> server relation. Think of it as kinda like latency in a game. Latency could be caused by the server just not knowing that the packet was sent to it (CPU overload, not enough RAM, etc)

Usually a restart solves most server problems. Problem is, with large websites restarts are either impossible or near-impossible. Time is money. (Imagine if google went down for a day to restart their servers. Chaos!)

you won't understand it from a programming point of view because in all laws of programming, latency shouldn't exist. It's a networking thing :P


Programming stuffs (tutorials and extras)
PM me (instead of MintPaw) if you're confuzzled.
thank Skaren for the sig :P

BBS Signature
cryptacet
cryptacet
  • Member since: Jul. 15, 2000
  • Offline.
Forum Stats
Member
Level 45
Programmer
Response to Database Stuff 2012-07-31 16:42:43 Reply

At 7/31/12 11:37 AM, MikeyS9607 wrote: I don't understand from a programming point of view, like when you use an insert query in MySQL, does the server not pick it up till later?

If you can run an insert query, it's rather obviously NOT a read only database.

For read only, items are inserted into a normal database, then replicated to one or more read only ones. Read only being much faster when selecting data, which for a website is probably most of the queries.
When the replication process doesn't work properly or whatever, things appear to be missing even when they are not.

egg82
egg82
  • Member since: Jun. 24, 2006
  • Offline.
Forum Stats
Supporter
Level 05
Game Developer
Response to Database Stuff 2012-07-31 16:49:21 Reply

At 7/31/12 04:42 PM, cryptacet wrote: If you can run an insert query, it's rather obviously NOT a read only database.

It was a hypothetical question, he wasn't talking about a read-only.

Though I did forget about database mirroring somehow; probably because I haven't actually used it.


Programming stuffs (tutorials and extras)
PM me (instead of MintPaw) if you're confuzzled.
thank Skaren for the sig :P

BBS Signature
Chdonga
Chdonga
  • Member since: Mar. 24, 2008
  • Offline.
Forum Stats
Member
Level 46
Game Developer
Response to Database Stuff 2012-07-31 17:31:10 Reply

I've noticed a bug. My inbox says I have one unread message, but there's nothing new there when I look.


BBS Signature
egg82
egg82
  • Member since: Jun. 24, 2006
  • Offline.
Forum Stats
Supporter
Level 05
Game Developer
Response to Database Stuff 2012-07-31 17:33:13 Reply

At 7/31/12 05:31 PM, Chdonga wrote: I've noticed a bug. My inbox says I have one unread message, but there's nothing new there when I look.

That's been around for a while. I've never experienced it myself, but I think the solution is to send or receive a message. As it's an inbox, my money's on receive.


Programming stuffs (tutorials and extras)
PM me (instead of MintPaw) if you're confuzzled.
thank Skaren for the sig :P

BBS Signature
DJDela
DJDela
  • Member since: Jan. 22, 2011
  • Offline.
Forum Stats
Member
Level 12
Musician
Response to Database Stuff 2012-08-02 13:08:22 Reply

Everything works fine on the site, except the front page.
I still get the message that the site is being maintained, or that's just me?

ZiggyZack99
ZiggyZack99
  • Member since: Apr. 15, 2007
  • Offline.
Forum Stats
Member
Level 15
Melancholy
Response to Database Stuff 2012-08-02 13:59:32 Reply

At 7/31/12 11:37 AM, MikeyS9607 wrote: I don't understand from a programming point of view, like when you use an insert query in MySQL, does the server not pick it up till later?

They're using a cluster of servers for their database, several of them are read-only to speed up random views on "static" content. Normally changes made are replicated immediately however a fault has caused them to lag behind a bit.


n/a

PsychoGoldfish
PsychoGoldfish
  • Member since: Sep. 26, 2000
  • Online!
Forum Stats
Staff
Level 12
Programmer
Response to Database Stuff 2012-08-02 14:10:39 Reply

Basically the way it works is we have a master database server and a batch of slave servers. The idea is on pages where you only need to read data (especially slower lookups like keyword searches), we use the slave server. These servers are load balanced so if one is busy, it uses a different one, thus keeping page loads nice and efficient.

When we need to add any information, like posting a message, or submitting content, it writes to the primary database. We never directly write data to the slave servers, rather, they use a sort of query queue. When the master database gets a new item added, it adds that query to the slave queue and those all get run in order.

The problems we have been experiencing come from an issue where this queue on the slave servers is falling radically far behind the master. Typically they run immediately, but some of our big automated jobs take a little longer and the slaves fall behind. The site still runs fact because everything is doing what it's supposed to, but new data (such as a newly submitted movie) doesn't exist on the slave server yet, so the submission page throws an error. After a few minutes, the slave catches up and the submission magically works.

Anyway, this latency was something that obviously should not have been happening, so that's why we had the downtime yesterday.


Arch-Angel likes this sig!

BBS Signature
egg82
egg82
  • Member since: Jun. 24, 2006
  • Offline.
Forum Stats
Supporter
Level 05
Game Developer
Response to Database Stuff 2012-08-02 17:30:13 Reply

At 8/2/12 02:10 PM, PsychoGoldfish wrote: Anyway, this latency was something that obviously should not have been happening, so that's why we had the downtime yesterday.

so the big problem was a few larger things getting moved into the slaves?

hm. What was the fix, simply not allowing people to do anything until the servers caught up, or did you wait and then reboot the machines the DBs were on?


Programming stuffs (tutorials and extras)
PM me (instead of MintPaw) if you're confuzzled.
thank Skaren for the sig :P

BBS Signature
Ectohelix
Ectohelix
  • Member since: Jul. 30, 2012
  • Offline.
Forum Stats
Member
Level 01
Musician
Response to Database Stuff 2012-08-03 12:03:21 Reply

Crud, I was hoping it was maintenance so projects (I've made a couple I'd like to remove) could be deleted. Ah well...


Find me on: Facebook, Twitter, or Soundcloud.

Ericho
Ericho
  • Member since: Sep. 21, 2008
  • Offline.
Forum Stats
Member
Level 44
Movie Buff
Response to Database Stuff 2012-08-04 12:02:25 Reply

At 8/3/12 12:03 PM, Ectohelix wrote: Crud, I was hoping it was maintenance so projects (I've made a couple I'd like to remove) could be deleted. Ah well...

Well, there was another update recently, so that might count as maintenance.


You know the world's gone crazy when the best rapper's a white guy and the best golfer's a black guy - Chris Rock