My guess here would be that you are noticing this near the end of a render. What is happening is that only one core works on each tile at a time. So when you only have 3 (2 or 1) tiles left in your image then only 3 (2 or 1) cores will still be working. One thing that you can do is to increase the AA value as that correspondingly will decrease the size of the render tiles. If you leave the AA value at the default 3, your render tiles are going to be quite large which is where you would notice the cpu falloff while there still may be a large portion of the image still to be rendered.