Very true. The solution may sound simple. However, the life we live at a given moment (resulting from whatever decisions we took) tends to involve other people. To live the life we envision before those people were also part of it may involve leaving not only a crappy job or a filthy town, but also leaving people who may or may not be responsible for our sense of failure or unhappiness.
It can get very complicated.