Our brains function like multicore CPU, with each core being specialised at a particular task. When that task isn't required, that area of the brain shows decreased levels of activity. Because the areas seem to be somewhat specialised, the areas with decreased levels of activity don't, or more likely can't, pick up work from the one's that are busy.
So, when you image someone's brain, you can find that quite a lot of the brain tissue isn't actually doing much. It's not helped by the fact that a lot of activity scans are extremely specific, for instance, noting colours or smells and nothing else. In such a test, you'd expect to see only small areas of activity, and that's what they're after so's that they can try to pinpoint specific areas related to specific tasks. If you were walking down an unfamiliar street, much more of your brain would probably start coming on as each of the 'cores' began evaluating the new environment.