well the only option I can think of is this..
1) do your shape on a new layer
2) ctrl click the layer the shape is on to make a selection
3) go to the channels tab and click the new layer button, this will bring up an alpha channel
4) fill in your selection with white
5) deselect your selection and go filter --> blur --> gaussian blur set strength to about 2.0 ...do this about 3-4 times ( ctrl - F to apply the filter ) until it looks similar to "pic1"
6) ctrl L to bring up the levels window...start to bring the right and left slider closer to the middle...you will start to see the edges sharpen up, tweak until it looks right ( sliders too close to the middle makes the adges jaggy, sliders not close enough stays a bit blurry)
7) ctrl click the alpha 1 layer to select your new shape....go back into your normal layers palette, make a new layer and bucket fill
you can further smooth out the edges / change the shape with liquify. medium sized low strength brush
------------------------------
with careful use of this method you can probably iron out big rough line art.
