It should swing very freely with all attaching bolts torqued. However, mine had the same problem you are seeing.
Turned out there were two problems. First, the rod ends were rotating slightly as the AN365-624A lock nuts were torqued. The rod ends therefore tended to jam against their supporting lugs on the WD-807 control column weldment. Was able to fix this by using an aluminium wedge to hold the rod end faces parallel to the weldment lug faces while torquing.
Second, more important, problem was the longitudinal distance between the rod end mounting holes in F-849 forward control mount and F-837-1 aft control mount. With the F-837-1 aft control mount temporarily installed by AN3 bolts in all four attaching holes, the distance between mounting holes was about 1mm less than the distance between the rod end bearings as measured on WD-807 control column. Noting that Van's says the aft control mount is not a major structural element, and has allowed previous builders to slot holes in it, I extended the ⅜" hole about 1mm aft, then reinstalled and retorqued the aft rod end bearing. No more problems, very smooth and easy lateral movement of control column.
Hope this helps