In 4 words: Surprisingly effective, but tedious.
It did have a great atmosphere, and I did feel genuinely unnerved, but the reel-in-and-swim-back times kept getting longer and longer, making it more immersion-breaking to get caught, especially with the hit box so large on those fish.
The reel-in time should have gotten quicker and quick with every few gems or so, just so I wasn't waiting increasingly longer periods of time to get back to the game. The swim sound the worm makes could also stand to be a few decibels quieter. It got annoying after a while.
Having collected all the gems, I'm quite disappointed that there wasn't much of a point to doing it. No secrets, no alternate ending, no special clue... Had I gotten to the end, been told I failed to get all the gems, and then replayed only to find out there was no point to it, I would have been furious.
Overall, not a complete waste of my time. However, I think the fact that this was for the ever-worthless Ludum Dare only held the game back. With more time and more heart, this could have been a lot more interesting and much better than it ended up being.