Does your saw have an adjustment to change the centering of the blade on the top wheel? If so, small adjustments in how the blade centers on the upper wheel will affect what you are describing. When I first set up my saw, I made sure the table was perpendicular to the blade. I don’t even know if there is an adjustment for the table to affect the direction of the miter slot / fence. I handle that by the positioning on the upper wheel.
In a recent note from Paul (his blog, perhaps), he commented that he just makes sure that the blade is perpendicular to the table in both dimensions. In other words, test against the side of the blade (bevel angle of the cut) and test against the back of the blade. I’ve never checked against the back of the blade, but you might use that to get your starting point and see if you get the cut you want.