C++ fastest way to read iostream?

[Alphabit]

I have an iostream object and I need to read from it as fast as possible...
I'll have to break up the data into words (std::string) separated by certain characters... Currently, I'm just using stream.get(char) and fetching characters one at a time... I have to parse a file that contains over 800,000 characters and I have to try to get my algorithm to work under 5 seconds but I currently get about 9 seconds... What can I do to speed things up? I'm not too familiar with the iostream library.

[Glaiel-Gamer]

std::string buffer
streamin >> buffer;

will extract one word from the stream and place it into buffer, (words being delimited by whitespace)

std::string buffer
while(!streamin.eof()){
streamin >> buffer;
doStuffWith(buffer);
}

[thoughtpolice]

Use a profiler? Use a library that is designed to be highly performant for text processing?

Of course, it brings into question whether you really need to go from 9 seconds to 5 seconds - if you have to do a lot of contortions it might not be worth it, but that's a decision that's dependent on a lot of factors.

[DougyTheFreshmaker]

At 10/28/09 10:45 PM, Alphabit wrote: I have an iostream object and I need to read from it as fast as possible...
I have to parse a file that contains over 800,000 characters and I have to try to get my algorithm to work under 5 seconds but I currently get about 9 seconds...

File I/O kills performance. You should read the data in large "chunks" to avoid accessing the file as much as possible.

It sounds like you may understand that already, however, from your original question ('one character at a time...'). If that is the case, then Google should suffice. This C++ ifstream Reference is sure to offer something useful. A quick glance at the read function looks promising.

[CronoMan]

My advice would be to read the entire file into memory, and then parse from there.
I believe it should be plausible to get the time down to under a second
If you read the entire file, you make the hardware work in your favor. After all, the harddrive + our friend the DMA controller, can deliver 50MB per second, and your file is 800k?
Going through a linear char array of 800 000 elements should be fairly straightforward

At least in my spectacularly brilliant mind

[Alphabit]

Thanks guys.
The string buffer thing did give a slight performance boost.

[littleMonsterGames]

At 10/29/09 09:40 AM, CronoMan wrote: My advice would be to read the entire file into memory, and then parse from there.

That's what he doesn't want to do, I think.

[kiwi-kiwi]

At 10/31/09 05:01 PM, littleMonsterGames wrote:

At 10/29/09 09:40 AM, CronoMan wrote: My advice would be to read the entire file into memory, and then parse from there.

That's what he doesn't want to do, I think.

CronoMan is right.
Also if you need speed I'd advise you to use fread.
Relevant pic: