It would be helpful to see some reference images of what you're trying to create, but here's a basic overview of what you should know to get started.
First, everything starts with a "flat" planet (i.e. normal planetary curvature, but locally it is essentially flat). You then displace upward. Unless you are using vector displacement, you cannot create overhanging surfaces without secondary displacement, and even truly vertical surfaces end up very stretched and problematic. For that reason it is best to displace upward with very steep but not truly vertical initial features. Then you apply secondary displacement laterally. The simplest way to do this is to use a Redirect Shader and only apply the X/Z direction, not Y (vertical).
You can get a good start on things by just adjusting a base noise function (Power Fractal) to have a high contrast (use the shader preview to accomplish this) with large scales (1000+ meters), then use that for high-amplitude displacement. This gets you sharp cliffs and rises. Then apply a second Power Fractal with much smaller scales and send it through a Redirect to get those lateral displacements. Obviously you need to find the right noise functions and settings to achieve your desired look, but this is the starting point you may need to iterate on that.
Aside from that I recommend checking out the canyon scenes and resources from both NWDA:
http://www.nwdastore.com/And Luc Bianco:
https://www.lucbianco.fr/store/For reference, starting points, and inspiration. Also look around in the forums here for "square rocks" threads and related.
- Oshyan
