While the hardware device should be protected, you won't lose your private keys if you lose the wallet. The seed that gets created when you initialize the wallet is what must be protected at all cost. In the case of the Ledger, it's an ordered, 24-word list that will reconstruct your accounts if your wallet gets lost, damaged, stolen or whatever. If that list is lost and your device is inaccessible, then you have lost your currencies for good. I believe all the hardware wallets use a seed like this. Remember, this is why informed investors are always saying to get your investments off the exchanges. The exchanges actually are the ones who hold the private keys, not you. You only have a journal entry in your account for the currencies you own, so you have no control over them. All the security measures exchanges take are good...2FA, email confirmations before finalizing transactions, etc., but these methods only address the marginal threat of individual account compromise. The more serious threat is hacking the system and obtaining the private keys, resulting in a much larger theft.