You are mostly right, that’s a criticism of vanilla PoS.
All Proof-of-Stake consensus try to mitigate that.
For the DPoS implemented by ouroboros, you can delegate your vote, like a delegative democracy. So you can give someone you vote, which as, indeed, some level of power, but you could also revocate him.
From the paper of Uroboros Praos, they really try to have a set of stakeholder as fragmented as possible (as EOS, etc).
I think that with (D)PoS, you won’t have a 100% decentralised network, but you could have a certain degree of decentralisation which is fine.
With PoW, I also like the idea of having many small computer doing the computation which would result in a random choice of the miner, but the incentive took it, and you have nowadays way more security than we actually need…
Here some infos about delegation : https://cardanodocs.com/technical/delegation/