Well.... I'm rendering a scene now with the soft shadows turned on, and it is really, really slow.
I have also tested the same scene wihtout soft shadows enabled, and I think it was a lot faster.
I think this is because I have a large population of trees.