Believe me, OCing is a bitch. 8 hours of prime, check back, bios, repeat. Since people are asking a lot, I might as well put together a guide as best I can. There are a few different stress testing programs that can be used, but I'll cover the two I'm familiar with.
USING AND ANALYZING A STABILITY TEST
Getting started
Before you open up that stability program, you need a temperature monitoring program. I recommend realtemp, as it works best with newer 45nm CPUs. Temperature is a huge limiting factor, and will probably be your first hurdle if you own a quad. Most higher OCs on quads require liquid cooling, while the Duos will still need some form of high end air.
Which do I use?
Traditional torture testing takes hours, but this is where IBT shines. In a half hour, your system has been so worked that stability can be confirmed. The problem is, it raises temperatures 10-15C higher than other tests. If you don't mind seeing numbers in a danger zone, and are short on time, use Intel Burn Test.
Prime95 is more thorough in its testing options. It has 3 different tests to choose from, each doing something different. Programmed to run in the background, it only uses idle clock cycles, meaning you can even game while running it. (Although I wouldn't recommend it.)
Learn2Prime95
When you open Prime, you will be confronted with 3 tests that you can run. If this is the first time you're opening it after new settings in the bios, then select small FFTs. This stresses the CPU very hard, but other components stay idle. You will need to run prime for 4-8 hours to confirm stability Process of elimination, if you fail this test, settings having to do with your CPU will need tweaking.
What, you passed 6 hours of small FFTs? Fuck you. Next is the blend test. This is more of a memory stress than the processor. Being good in small FFTs but not blend means the memory needs tweaking. 4-8 hours here too.
If you pass both of these, you can probably say you're stable. But for those with OCD, how about you run a pass of large FFTs. If you pass this with acceptable temperatures, then welcome to your new overclock. Prime95 temps will be 5-10C higher than gaming, so don't pull the plug just because you see 60s.
Learn2IBT
Intel burn test is much more straight-forward. Open it up, and you'll be faced with a firey red command prompt. Fitting. Turning error detection off gives you a few goodies, such as how many gigaflops you're achieving. For maximum stress, dial 1. For full stability, 20 passes is suitable.
If you turned error detection off, then you are the judge of stability. For each completed pass, compare the residual to the residual(norm). If they match, that test was passed. You will see very high temperatures on this program. Don't worry, as long as they don't break 80s. Then worry. Your computer will be almost completely bricked during IBT, because it uses almost every scrap of system resources. And yes, a BSoD does mean you failed IBT.
Congratulations, you now know how to test stability. Behold the endless cycle of trial and error that is overclocking. Anyone with anything else to add feel free, because I know I didn't cover everything.