I’m glad you solved your problem! Looks like it was both stubborn grain and a wobbly workbench. I do think the angle of the bevel could be slightly increased to at least 25° for hardwoords. With a shallow angle, the edge is fragile and damages more easily.
PS: totally forgot I sold you a plough plane ☺ Glad to hear you already made several projects with it!