It probably depends on what you use as displacement. If that's an image map on Y projection, it's logical, as that projects the greys downward. Or you should use a spherical projection, but that requires a different 2:1 map. If it's procedural, you may need a compute normal to get better calculation of lateral displacement. Or perhaps a transform shader set to world position will be enough. If you can post, it's easier to tell.