How are you calculating size of your tx? I grab the cbor string for my end and have to add 3 bytes each time as Im assuming the 3 extra bytes are added when you actually submit it.
Also noticing on adaex which is what I use sometimes to check roughly the size of the tx that their estimates are lower then what I actually have from my cbor for the signed tx