There is a smoothing filter, but it depends on what your main mask is made of. If it's a photoshop painted mask, e.g. that won't work. Blur in PS is then the best option. If you need hard edges for something again, you can always use the color adjust shader to harden those soft edges back.