After rereading this, I think my original comment probably was not the cause of your issue.
I find that connecting only to one or two relays during sync speeds things up, especially if they are iohk relays. Here is a good topology for syncing. You should also disable topology updater while syncing as you don’t want your node restarting.
{
"Producers": [
{
"addr": "relays-new.cardano-mainnet.iohk.io",
"port": 3001,
"valency": 2
}
]
}