The Thread Thread

[pieoncar]

I couldn't help myself. I had to make a stat thread of sorts for the BBS.

This project of mine started last June when I was on a ban, actually. I needed something to do with my free time, so I decided to write a bot to crawl the BBS and gather information about the threads on it. But right around that time, the BBS layout changed and I didn't feel it was worth my effort to fix up my crawler accordingly. I just this week finally took it out and reworked all the kinks.

So, here is what I have for pretty much every post -- I did make a couple changes in the middle of the run, so what I have doesn't apply to searches on the entire list of threads, but if you want to see all the threads you've made, I'll make sure those are up to date.

I have:
- the topic ID (i.e. /bbs/topic.php?this_number_here)
- the title of the topic
- the date and time the topic was posted
- the userid of whoever made the topic*
- the forum in which the post was made**
- the icon used for the topic
- whether the topic was locked when I crawled it
- the post ID of the first post in the topic (the number if you hit the reply with quote button)

*There was an error up until probably topic ID 180,000 or so, where if the first poster didn't have a profile, the parser said that the first person to post WITH a profile created that topic... but, when I make a list of any certain user's posts, I fix that. But do not expect me to go over the 167,399 topics of which I'm uncertain.

**Forums 9 and 11 both are restricted, so I assumed all restricted topics to be in forum 9.

Now I realize that this may be construed by some as a "spamometer" of sorts, and keeping this in mind, I will never post any statistics that I think will encourage spamming. In fact, to get this out of the way now, I'll just say that P-Bot has the most topics, and P-Bot is the only BBS member who should be putting out a thread a day anyway. I'm not sure offhand what else might be considered a spam statistic, but I am never obligated to give anyone the statistics they ask for.

The first statistics I'd like to get out of the way are the topics vs. point in time stats, so here are topics on given points in time. First, total topics per day of the week:

Sunday: 34252
Saturday: 32925
Monday: 32330
Tuesday: 31638
Thursday: 31205
Wednesday: 31126
Friday: 30299

It looks like BBS members do go out most often on Fridays after all :)

Posts by hour of the day:

21: 13642
22: 13390
20: 13113
19: 12860
18: 12784
17: 12724
16: 12492
0: 12451
23: 12349
15: 11590
14: 10554
13: 9813
1: 9496
12: 9358
11: 8227
2: 7609
10: 6946
3: 6226
9: 5724
8: 4977
4: 4598
7: 4437
6: 4217
5: 4198

That seems to somewhat accurately reflect BBS activity. Want your thread to get good exposure? I'd say you should probably post it in the early evening, around 4 or 5 PM EST. This way, there will be many people online but your topic will be less likely to get pushed down to the 2nd page by other posts.

Finally for the activity-relevant stats that I can think of, here are the topics by month, but only for 2002-2004, because 2001 and 2005 add bias to some of the months:

August: 18889
December: 18795
October: 17328
November: 16753
September: 16518
July: 15616
June: 13279
April: 13164
May: 13095
March: 12329
January: 12263
February: 11921

This set of data might not be so worthwhile, considering how Newgrounds' userbase is always increasing. But, August and December at the top makes sense for people being on break.

If you've read this far, I'd like to say thanks. But what next? Surely if this was the whole topic, it would hardly be worth an entire thread. So, I'll tell you what I'm going to do for you.

You can see the statistics I have up above, and you can figure out what I can make from them. I expect the most common request to be "give me a list of all the threads I've made." As long as the moderators say this is okay, then I'll gladly give you a list of your threads. No requesting lists of other people's threads. (Moderators may be an exception to this rule, but we'll worry about that later if it comes up at all.)

Next, you might need to find a thread that won't come up in the search box for whatever reason (<4 letters, too common of a word). I made a topic called "Tell me why" a while back, and it's impossible to find in the search because "tell" is too common of a word and is not returned as a result. Also, in case anyone is wondering, here are numbers on some poorly named threads:

wtf: 42
lol:37
omg: 31

If you can think of any other ways of organizing any of these stats, please make a request for them in this thread. I'll finish this post up with a couple of random other ideas I had.

I don't necessarily know how to use the text search in MySQL flawlessly, but as far as I can tell, Idiot-Finder is leading the Photoshop threads with 41 topics starting with the word Photoshop, with the next closest contender being TehBanStick with 9.

Last, of the 242,588 topics I have indexed, 18,813 have been deleted and 611 are in the moderators' forum. And disregarding any topics from the last week, while the mods can still lock them, we have 20,916 of our 221,068 topics locked. That's nearly 1 in 10 topics. Good work, moderators! (This is probably the only time I will post that stat because the lock information will need refreshing, which I do not plan on doing... right now was the freshest that will ever be.)

As far as closing notes go, don't expect there to be any kind of web interface for this. It won't be worth my effort to code. Don't expect open access to this database either, at least not without convincing me somehow :P

It is now going on 5 AM here, so I should go to sleep. Seriously, if you really read it all up to here, thank you. I hope to have some questions on this data when I wake up! :)

[Alkador]

Well, I don't really know what to say. It's all very interesting and there's a lot of stuff to read and all, but I'm not sure what to make of it. It seems to be filling of data, yet there's something missing. But I did like the omg, wtf and lol part, that was well done. This must have taken you a lot of work and effort to make!

Anyway, don't really want to give my overall opinion just yet, but I will say that is informing.

[gfoxcook]

At 3/13/05 05:51 AM, pieoncar wrote: I couldn't help myself. I had to make a stat thread of sorts for the BBS.

Just when they said people couldn't find a new sort of list to make or stat to keep track of....

But then again, with Dogma and RedCircle's HUGE (and long-time-in-the-making) flash portal submissions website... I suppose something like this was inevitable.

The first thing I think is... hm... very thorough... but... WHY, man, WHY? Then again, a lot of people have asked themselves (or spammed up my topics yelling about it out loud) why I ever made some of my 5 major stat lists on this forum, so...

*There was an error up until probably topic ID 180,000 or so, [snip] 167,399 topics of which I'm uncertain.

Fascinating. But wait, how are you CERTAIN on the 12,601 topics? #;-}>

**Forums 9 and 11 both are restricted, so I assumed all restricted topics to be in forum 9.

More than just 9 and 11. Methinks you didn't notice the new all-mods forum that was created much more recently than either of those two (the bbs mod forum and review mod forum).

Now I realize that this may be construed by some as a "spamometer" of sorts, and keeping this in mind, I will never post any statistics that I think will encourage spamming.

Nothing is spammy about this compilation of site-related statistics you've come up. It's the same thing as Dogma and RedCircle's stats about NG portal submissions, like I basically already said... that said... hang on.

First, total topics per day of the week:

Sunday: 34252

... (snip)

Friday: 30299

It looks like BBS members do go out most often on Fridays after all :)

Or maybe Fridays have the most topics made but also the most topics deleted? Do your methods account for that? I mean... any topics deleted prior to you starting to keep track of this stuff... you can't possibly know the date they were created, the user, etc. etc... all of that stuff. Just the topic id#. Which is enough to narrow down the DATE, but... from the sounds of your methods, that's not exactly how you did things.

Posts by hour of the day:

21: 13642
22: 13390
20: 13113

(snip)

That's a fascinating listing as well, but I kinda wish you'd ordered it by hour, not by postcount (and perhaps you should specify "topic-creating posts," though you mentioned that elsewhere.

That seems to somewhat accurately reflect BBS activity. Want your thread to get good exposure? I'd say you should probably post it in the early evening, around 4 or 5 PM EST. This way, there will be many people online but your topic will be less likely to get pushed down to the 2nd page by other posts.

Makes sense, but you can just use logic to get to the same conclusion.

A lot of people get home from work or school in the afternoon/early evening, and the best time for a topic to have been put up is RIGHT before this happens, because a lot of people will check the first page of a forum first, and if your topic name is interesting enough, you've got tons of viewers as they flood into the BBS. But later in the evening... there's going to be a ton of topics created BY those people (most of them, I'd wager, don't instantly make a new topic themselves, though some probably do). And at that point, stuff's getting pushed off the front page (of General, at least) every 20 minutes, if not even faster.

If you've read this far, I'd like to say thanks. But what next? Surely if this was the whole topic, it would hardly be worth an entire thread. So, I'll tell you what I'm going to do for you.

Actually, if that had been it, it would have still been worth its on thread, because... there's no OTHER thread about the above stats. It doesn't belong in the Submissions thread. It's the COUSIN of that thread, not the twin of it. And I think it's valid info and worth talking about. And you put it in the right place (on Wi/Ht? rather than General), too. So... no problems, man. You didn't have to justify its presence here on the stats' basis alone.

You can see the statistics I have up above, and you can figure out what I can make from them. I expect the most common request to be "give me a list of all the threads I've made." As long as the moderators say this is okay, then I'll gladly give you a list of your threads.

This is where the gray area begins. I don't want this topic turning into a topic like the "rate my ____" topics on General, nor do I want this topic turning into a human-powered version of the long-inactive BBS search-by-text option (only for topics instead of all posts).

But as a mod, all I can do is lock your topic if it gets out of hand or delete posts. I can't tell you if that's okay or not because I honestly don't know. You should ask an admin. liljim happens to be in Japan, Tom Fulp is in Austin (where I just met up with him last night, heh!), and Wade is busy with tons of review ban clearances and other things, no doubt. So this might not be the best timing...

I would discourage you, in the meantime, from offering this service to users, as tempting as it may be to them to ask for it... and clearly to you for offering it.

Also, if anyone was THAT interested in such a thing, they'd have kept track themselves. I personally have a shortcut in a folder to every topic I've ever created on NG, along with tons of important topics by other posters (like a lot of the topics around here, possibly including this one! #;-}>). I can tell you, therefore, that:

I've created 33 topics on the normal BBS areas.
Two of the 33 are are locked.
One topic has over 2000 posts in it.
Two topics have between 1000 and 2000 posts.
Three topics have between 500 and 1000 posts.
Five other topics (11 total) are over 1 page long.

Next, you might need to find a thread that won't come up in the search box for whatever reason (<4 letters, too common of a word). I made a topic called "Tell me why" a while back, and it's impossible to find in the search because "tell" is too common of a word and is not returned as a result.

That, OTOH, is a VERY interesting and useful sounding utility for the results of your efforts.

I, too, have searched for short words or commonly used words, both in the portal title search and in the BBS topic search... that won't let me find any results. So... yeah. Groovy. Great idea!

[Slightly-Crazy-Dude]

Wow, just when i thought all the lists where done in here, BAM! another pops up and its quite intresting.

Since no one has tested your ablities yet, i would like to ask you if you would indeed, see how many topics i have started. If you would email me them or post them here it wouldnt matter, though, i think email is the best option, i dont want "some" people knowing what i started a topic about 2 years ago, when i was a n00bling ;)

[Xiivi]

Haha, I don't really see the use, unless you wish to post a topic at the perfect time in order to get the most replies or something. Other than that I don't know too much of a use I could have such a list as this. :p

[ramagi]

Wow that was a lot of into to take in this morning. I have no doubt that took you a lot of time to put that all together. I think I goign to spend more time reviewing what you said.

[BananaBreadMuffin]

At 3/13/05 09:04 AM, Slightly_Crazy_Dude wrote: i would like to ask you if you would indeed, see how many topics i have started.

I would also like a list of all my topics, if you would be so kind.

[pieoncar]

At 3/13/05 07:09 AM, Alkador wrote: It seems to be filling of data, yet there's something missing.

If you could be a little more specific... can you maybe put your finger on what it is that is missing?

But I did like the omg, wtf and lol part, that was well done. This must have taken you a lot of work and effort to make!

Well, gathering all the topics took a lot of time, but it wasn't too much effort compared to other programs I write. The fact that I worked on it in the summer and just recently makes it hard to be exact, though, I suppose.

Anyway, don't really want to give my overall opinion just yet, but I will say that is informing.

Well, informing was one of the main objectives... :P

At 3/13/05 07:16 AM, gfoxcook wrote: The first thing I think is... hm... very thorough... but... WHY, man, WHY?

I was bored, and this was a decent way to practice coding something the likes of which I had never coded before. Then it at least had some informational value of sorts.

*There was an error up until probably topic ID 180,000 or so, [snip] 167,399 topics of which I'm uncertain.

Fascinating. But wait, how are you CERTAIN on the 12,601 topics? #;-}>

Oops, those parts made more sense to me when I was writing them. On the error up to about 180K, the problem was that if a user with a profile was the first person to reply in a thread where the topic creator did not have a profile, the replier was credited with creating the thread, in my database. This might be better shown with an example; I was looking through the threads I made while in the middle of the list, and I didn't remember making a thread on "Jupiter ownership."

My fix to that (after I patched my code to confirm that anyone was indeed the topic starter and not just a replier) was to add another variable that said whether I was sure of such; and when I'm pulling up data of who made what topics or when they made them, I re-check every entry that hasn't been confirmed as such.

More specifically on the numbers, as of this moment I can tell that I didn't start keeping track until 187,558, but because I have already rechecked some of them, there are 167,399 topics where the original poster may have posted without a profile and someone with a profile replied.

Methinks you didn't notice the new all-mods forum that was created much more recently than either of those two (the bbs mod forum and review mod forum).

Sure didn't. But since I can't distinguish from them anyway (the error message says "You do not have sufficient permissions to read this thread. Sorry.") on a lot of topics, I have no idea which forum they belong to.

Or maybe Fridays have the most topics made but also the most topics deleted? Do your methods account for that? ... Just the topic id#. Which is enough to narrow down the DATE, but... from the sounds of your methods, that's not exactly how you did things.

I could try re-setting up that query to estimate the dates of deleted topics. I had completely overlooked that while reporting that data.

That's a fascinating listing as well, but I kinda wish you'd ordered it by hour, not by postcount (and perhaps you should specify "topic-creating posts," though you mentioned that elsewhere.

I went back and changed posts to threads in a couple places, but I overlooked that by mistake. Anyway, here it is by hour rather than by post. This time I removed the first day and today to remove some bias (once again deleted topics are disregarded):

0: 12424
1: 9475
2: 7595
3: 6213
4: 4593
5: 4198
6: 4217
7: 4437
8: 4977
9: 5724
10: 6946
11: 8227
12: 9358
13: 9813
14: 10554
15: 11582
16: 12475
17: 12720
18: 12779
19: 12858
20: 13107
21: 13639
22: 13389
23: 12348

Makes sense, but you can just use logic to get to the same conclusion.

And now here's some hard data to back it up :P

I don't want this topic turning into a topic like the "rate my ____" topics on General

Understandably.

nor do I want this topic turning into a human-powered version of the long-inactive BBS search-by-text option (only for topics instead of all posts).

Maybe this is just because this is kind of my brainchild, but I'd like to know why, in more detail, you would disallow this function, when you went on to say

I, too, have searched for short words or commonly used words, both in the portal title search and in the BBS topic search... that won't let me find any results. So... yeah. Groovy. Great idea!

Unless you originally meant a full text search on the actual content of the first post, which I did not store. But, if finding topics with a certain word in the title is okay, then we're all okay :)

So this might not be the best timing...
I would discourage you, in the meantime, from offering this service to users, as tempting as it may be to them to ask for it... and clearly to you for offering it.

I'll shoot the admins an email, then, and wait for a response. Until then, I can't imagine why this wouldn't be allowed for discussion of topics vs. time, and things of that nature, so I'll continue with that.

Also, if anyone was THAT interested in such a thing, they'd have kept track themselves.

That was part of the dilemma. I could remember some of my topics when I'd finally decided to start bookmarking all of them, but I knew there were more, and now I know which ones these are (and why I didn't remember them :( )

I've created 33 topics on the normal BBS areas.

That's the only one I can confirm with certainty :0

So, as for some last words, I'm going to send an email to Tom, Wade, and Jim to see what they give the okay on. In the meantime, I would imagine that emailing people their personal stats would be okay (SCD and LFB, check your email in a couple minutes). Until the admins reply, I'll continue with the time statistics and helping out in finding topics with the "difficult" words.

[Slightly-Crazy-Dude]

Thanks alot Pieoncar, thats a very nice system. I made alot more topics than i thought, thats just fucking strange. Thanks for the info, its appreciated.

[Kaabi]

So you can find out how many topics someone made? Do me, Kaabi. And how did you make this bot? Very interesting.

[WadeFulp]

How long were you running your scripts to gather that information? This might explain why everyone is getting Internal Server Errors, because your scripts maybe over loading our database. We're getting fucking sick and tired of people running shit like this to find out stats. I can't say for sure what impact this has had, but similar things like this have fucked up the site for everyone. STOP WITH THE FUCKING STATS. If we want you to have certain stats we'll gather them in a manner that won't screw up the site. If we learn that your actions have been taxing our site it's nothing sort of a DOS attack and you can face legal action.

Some of you think you're doing no harm and are being helpful, well YOU ARE NOT. STOP RUNNING YOUR FUCKING SCRIPTS.

[ShittyKitty]

At 3/13/05 07:16 PM, WadeFulp wrote: STOP WITH THE FUCKING STATS. If we want you to have certain stats we'll gather them in a manner that won't screw up the site. If we learn that your actions have been taxing our site it's nothing sort of a DOS attack and you can face legal action.

Some of you think you're doing no harm and are being helpful, well YOU ARE NOT. STOP RUNNING YOUR FUCKING SCRIPTS.

Umm, does this include my NG Total Listing? If so, I'll stop the list as soon as I return to my machine next Sunday.

[WadeFulp]

At 3/13/05 09:01 PM, ShittyKitty wrote:

At 3/13/05 07:16 PM, WadeFulp wrote:

Umm, does this include my NG Total Listing? If so, I'll stop the list as soon as I return to my machine next Sunday.

How often is it banging away at our servers? Basically no one should be running anything that pulls info from our servers. LilJim will make pages to list the stats you guys want, but in a way that won't screw up the site. I know some of you don't intend harm, but some of these things do a lot of harm. You guys want NG to vanish from the web because you have to know all these stupid stats?

[ShittyKitty]

At 3/13/05 09:17 PM, WadeFulp wrote: How often is it banging away at our servers?

19,000 profiles on Wednesdays, about 90,000 profiles on Sundays, and all 1.2 million possible profiles every month.

[WadeFulp]

At 3/13/05 09:29 PM, ShittyKitty wrote:

At 3/13/05 09:17 PM, WadeFulp wrote: How often is it banging away at our servers?

19,000 profiles on Wednesdays, about 90,000 profiles on Sundays, and all 1.2 million possible profiles every month.

You know if we logged you doing something like that we could probably have you arrested? Please stop running those types of things. It's similar to a DOS attack which is illegal.

[idle]

At 3/13/05 09:51 PM, WadeFulp wrote: You know if we logged you doing something like that we could probably have you arrested? Please stop running those types of things. It's similar to a DOS attack which is illegal.

Just wanted to know if you are planning on stopping any of the current listings which are unauthorised? Are some of them allowed and some not?

I only ask because most of the listings make no secret about the fact that they are operating, and more often than not are using systems similar to a DOS attack.

[YoinK]

wow, I always thought ShittyKitty had permission to run his lists....
so if there is no more NG total listings... will there ever be the Top VP list which is made once a week? and the Penta List shouldn't do too much with the server when it is only done every 2 weeks... ramagi's list is only done once per month...
Would these other lists be fine Wade?

[WadeFulp]

LilJim is working on lists and will have bar graphs, etc, available to everyone soon. So in the meantime everyone should stop running these scripts. If you know people doing this tell them to stop. We are about to start logging these people and reporting them to their ISP's, and if they keep it up the FBI. Many of these people are basically carring out a DOS attack which is illegal. Hopefully most of these people mean NG no harm and once they realize it's harming NG they will stopo. Instead they should focus on what stats they want and prepare a list for LilJim to go through. We'll find a way to provide them with the stats they want that won't create uneccessary load on our servers.

Spread the news.

[Xiivi]

Looks like some of the lists will be ending up discontinued or people will have to start relying on manual pulls again. Tough luck list makers. :'( I guess with manual the lists would be updated less often and less people would be listed. ;_;

[DarkArchon]

I like how one or two people ruin it for the lists that were interesting, and didnt cause harm.

[YoinK]

At 3/13/05 10:49 PM, WadeFulp wrote: LilJim is working on lists and will have bar graphs, etc, available to everyone soon. So in the meantime everyone should stop running these scripts. If you know people doing this tell them to stop. We are about to start logging these people and reporting them to their ISP's, and if they keep it up the FBI. Many of these people are basically carring out a DOS attack which is illegal. Hopefully most of these people mean NG no harm and once they realize it's harming NG they will stopo. Instead they should focus on what stats they want and prepare a list for LilJim to go through. We'll find a way to provide them with the stats they want that won't create uneccessary load on our servers.

Spread the news.

hmm sounds fair enough. I always want this site to run good. Will these stat lists be updated automatically?
And any chance you can incoprorate the following lists....
Top 500 Experience list (updated once a month to show the change in exp points among the top 500 experience users.)
The Penta List (A list that adds experience, blams, saves, reviews, and posts)
The Top Voting Power List

[idle]

At 3/13/05 10:54 PM, YoinK_VineS wrote: And any chance you can incoprorate the following lists....
Top 500 Experience list (updated once a month to show the change in exp points among the top 500 experience users.)
The Penta List (A list that adds experience, blams, saves, reviews, and posts)
The Top Voting Power List

I think these three lists would make a great addition to the Users page on NG. Even though Wade said they have caused some trouble, I think they are important lists nonetheless. Instead of having a Top Voting Power list, there could just be a ranking in the profile (just a thought).

[phileeguy]

At 3/13/05 11:02 PM, -idle- wrote:

At 3/13/05 10:54 PM, YoinK_VineS wrote: And any chance you can incoprorate the following lists....
Top 500 Experience list (updated once a month to show the change in exp points among the top 500 experience users.)
The Penta List (A list that adds experience, blams, saves, reviews, and posts)
The Top Voting Power List

I think these three lists would make a great addition to the Users page on NG. Even though Wade said they have caused some trouble, I think they are important lists nonetheless. Instead of having a Top Voting Power list, there could just be a ranking in the profile (just a thought).

I think the Top Voting Power List as well as the Top Experience List would make good additions as well, but it's really up to them. The Penta List might cause problems because as it is configured now it incorporates Posts as an available way to be on it, something that might encourage spamming which just wouldn't do.

It's a shame that these lists are going to be put on hiatus, because like Wade said the creators didn't mean any harm. I am just a little depressed right now because I would have finally gotten a spot on the next Penta List update.

[ScottTowels]

What kind of lists should we expect?

-Voting power ranking + vp list?
-Extended exp, blam/protect and reviewing lists?
-Top blammers and top saviours?

As for the other lists, I expect any list that isn't implemented manually to disappear, even the quickdraw.

[ReconRebel]

At 3/13/05 10:50 PM, Inuzuka-Kiba wrote: Looks like some of the lists will be ending up discontinued or people will have to start relying on manual pulls again.

This place was becoming a list factory Inuzuka (way too many). I admit I'll miss some of the updates but it's for the good of Newgrounds like Wade said. Better to lose a bunch of lists than risk screwing up the entire site.

[WadeFulp]

If there is enough interest in a certain list i'm sure LilJim can put them together.

[cheesebizkit]

is ShittyKitty's Quickdraw be considered a list that would be included in this as well?

[Evark]

At 3/13/05 09:51 PM, WadeFulp wrote: You know if we logged you doing something like that we could probably have you arrested? Please stop running those types of things. It's similar to a DOS attack which is illegal.

I just hope you'll remain reasonable. All the list-makers have been operating for years now, most of them. ShittyKitty is relatively new, but his lister has made it much easier for all of them to maintain these lists. He originally posted his program here for anyone to download, but then revoked it as soon as he realized the potential it had to mess things up database-wise. As far as I know, it still is available only to those that SK deems actually need the lister to use for an actual legitimate reason.

As for pieoncar, I don't think he means any harm either. I think this mostly spawned out of my brother and myself's unknowingly similar interest in a search by topic author addition to the BBS search (by this I mean he might not have started the thread or fixed up his program had there been no reminder of it), and probably a lack of a way to find threads that contain words with less than three characters in them. People who are curious about something and don't see any problem with finding out the answer will go ahead and do it.

Just another case of a select few trying to make everyone's lives easier by taking the guesswork out of certain things, who didn't know they were causing harm. I realize by now that your post is several hours old, and that you probably aren't as mad as you were initially, but I still think you need to hear someone reinforcing the notion that nobody is out to harm Newgrounds.

[AngryAlt]

Maybe these guys collecting stats is what was causing alerts with my firewall?
Something along the lines of "MSSQL_NULL_PACKET" or is that something else?
I haven't got one of these alerts for several months though.

[Terminator]

At 3/14/05 12:03 AM, cheesebizkit wrote: is ShittyKitty's Quickdraw be considered a list that would be included in this as well?

He said for everyone to stop running all scripts.
It will be a shame to see some threads go but if it's slowing NG down, then there's no question they should stop.