My first ever raised panel was done with Paul’s bench plane method in one of his in-person classes. The class was intense and things moved along…there was work to do and not much time to linger on things, so I made it as best I could and moved on. My point is that I didn’t have time to be perfect and simply tapered the panel until it would fit into the full length of the groove in the panel, testing with a cuttoff, if I recall correctly.
This is all just a prelude to say that this panel is on my tool chest, which is now about 10 years old and has never shown any signs of becoming loose. The concerns you raise all make perfect sense theoretically, but in practice I don’t think the issue is as worrisome as it might seem. What is probably more important, if you plan to use stain or dye, is to remember to get stain or dye onto the panel before you glue up to avoid showing a white line if / when the panel contracts.
Remember, you have many points of contact along each edge, wood has some elasticity, and the vertical dimension is not going to change, so you will always have the top and bottom of the panel snug and keeping the panel from rattling (as long as you don’t undercut them grossly).