A plane object (even if displaced smoother than the DEM) will always have a hard edge when intersecting with the terrain as they are two separate objects. So for a smooth transition you need to use just the planet, and the intersect method. If you set the displacement intersection zone bigger, and the compute terrain as well, the transition will be smoother.
Here's a file for you that may be of help. If I understand what you need.
But with DEM's I don't really know if the intersection method will work, as they are sort of fixed displacement maps, I guess (I never use them).
Welcome, btw!