I just explained to Martin (T-U) in Dutch, so I'll do it in English as well. Paint in the color channels indeed, 16 bit RGB tif. Blue for Y, the other 2 for the other sides, whichever you choose (green=X and red=Z, or vice versa). Think where you want your cave, keeping in mind that it displaces to one side. On top of the lightest blue (white) is the top, so you don't want a cave there, paint it at the blackest edge. If you pant in the third channel on top or half on top of the green channel, you get a hook in the cave, which is what my example was about. If you draw a softer gray slightly higher than the cave (where the blue is lighter), it'll form a softer cave, and thus will produce a 'brow' above the cave. Lots to experiment with.
Then convert to 32bit and save. 16bit RGB won't open in TG, and 32bit can't be painted in in PS., hence the workaround. Set size appropriately in TG image map shader and check data is linear of course, like Jochen and/or paq mentioned. No need for an EXR. I couldn't even save an EXR I worked in as EXR.