Thanks for all comments
Here are two more experiments where I made a small height field, eroded it with a height field erode and used that to supply some displacement to the same base stones. One uses positive displacement and one uses negative. The negative seems to confuse the merge by height shader since it allows the gravel layer to appear in the large stone where it is very close to the ground on the left.