Basically this is what you can do. Smooth any displacements to zero altitude before the compute terrain by the same mask.
Use a surface shader, mask it and set it to smooth. Only problem might be that if you use a PF to displace terrain, you'll have positive and negative displacement. So smoothing can also raise some ground that was lower than zero. In that case you can use the offset in the same surface layer to raise the whole lot again until you're satisfied with the smoothed area altitude.
For more variation on the shores/banks of your river, you could also use a PF to break up the transition between normal terrain and smoothed area. Play with values, and color roughness (!), and uncheck 'only breakup color'.