If I can get your private keys from your public keys, then I can spend/send as if I'm you. You would then no longer trust that currency.
If someone breaks the encryption, people will dump the currency because they will no longer trust it as secure. Your value being secure is based on the encryption.