Your surface layer may apply displacement in various ways: in the Displacement tab, the Smoothing Effect, Intersect Underlying, or if it has any child shaders with displacement.
The fractal breakup shader usually has a lot of fine detail (the "smallest scale" is small and it has high roughness). Those fine details often create unwanted roughness if they are allowed to affect the displacement.
When "only breakup colour" is enabled, the fractal breakup shader does not affect displacement but only affects colour and luminosity. When "only breakup colour" is disabled, the fractal breakup shader affects displacement exactly the same as it affects colour and luminosity.
Matt