That's actually not bad! You just need to apply the same displacement to the "stems" of the leaves. Hopefully it is not the same part of the UV map as the trunk, otherwise you would get the trunk bending as well, which is not desirable for a palm tree. But worth a try. You might also be able to mask it by height using a camera-projected gradient image, if necessary.
- Oshyan