What if you use much bigger values for the PF, maybe the smallest size like 25% of the tree height? Then perhaps the whole tree will bend and sway, instead of the small scale jittering.
I don't know when the displacement is getting its shape in the object or parts, but perhaps if you add a surface shader after the default for the trunk and set that to smooth, you could eliminate the former displacement. But perhaps (likely) it works differently. And you'll loose the bark displacement.
Another option is to make trees as two objects; trunks and leaves separately, populated by the same specs, but that's a hassle too.
But with far less displacement in this setup you might get quite far. Or 2 displacemnt PF's, one for the whole tree, as said with all big patches/sizes, and another with all small sizes, but very little displacement (2cm). Too bad it takes so long to render a sequence...