Those meteorites are a lot lighter than the huge planet-like rocks in the images. Meteorites will be easily accelarated like you describe (gravitational sling) but large bodies will accelarate considerably slower due to their huge mass. Consequently, their mass will also affect the earth's rotation.
If it wouldn't work that way then there wouldn't be starsystems I think as all celestial bodies would toss each other around.
In the end you're still right; Imagine two celestial bodies hitting eachother because their outer circumference of their orbit just slightly slightly overlapped. It's not a head on collission, more side by side, but both their speed is great already.