That's the crux of it. Getting to all of our witnesses and all copies of the blockchain in order to modify the past would be quite the undertaking. Virtually impossible.
I get so excited about this stuff, there's so much potential here :)