I see where you are now. I don't see anything about enlarging those holes, either. A few basic things to check (not to insult anyone's intelligence - we all make avoidable mistakes). Are the plates overlapping properly? All four holes have both the spar and a reinforcement plate. Is it possible that you didn't keep the four pairs together? Is it possible that the center hole for the rod bearing isn't aligned?
Did you use a K1000-6 nutplate for all four holes? Maybe the problem isn't the holes, but rather the wrong nutplate?
Looking briefly at my elevators, the fit isn't 3/1000 inch, like you suggest. There is enough of a margin for error that if my nutplate was off by a little bit from dead center (and I'm sure it is off a little), the bearings still would fit through the hole into the nutplate.
Or, maybe Van's CNC process made an error somewhere.
Be careful with a unibit or reamer widening that hole - since you have a nutplate installed in each hole, if the unibit pokes through too far, you will damage it.
If you need to increase the diameter of the holes without hurting the installed nutplates, one option would be to use a narrow sanding or grinding tip in a dremel - just poke the tip in there and sand out the edge until the hole is wide enough.