We have recently discussed that in another forum. It seems, that it is no user fault.
Instead, according to the Cardano Improvement Proposal CIP3, the method how your keys and addresses are calculated from the 24 words are different between Ledger and Trezor.
It seems that Ledger is following the Bitcoin standards BIP39/BIP32, while Trezor, Daedalus and Yoroi are following the CIP3 recommendation („Icarus“ procedure).
This is not related to a different derivation path, as you may know it from other wallets, but to the very first step, when the master root key is calculated from the seed phrase.
Thus, you have to use another Ledger for importing the seed phrase. Or one would need the knowledge and have to invest some time, to change the wallet code manually.
@SebastienGllmt has indicated that already in another thread. I think it would be nice to have an import function in Daedalus, which uses the standard BIP procedure, so that in worst case one could import the Ledger seed phrase.