I think you are thinking of the end result as opposed to the entire process.

I'm not sure I follow.

greater harm for those who don't obey (jail, shoot-out with the cops, police brutality for someone who doesn't respond quick enough)

To me, that sounds like you're thinking of the end result (a highly improbably one). I don't know about the US but I'm pretty sure that in most other countries there won't be a shoot-out with the cops for jaywalking.

It could also mean being late for work or not having time to play with the kids or not having time to see grandma on her death bed before she dies.

All hypothetical. We can go either way with hypotheticals. By not crossing at the properly designated place, you could get run over and die even before grandma does. Not only that, the accident causes the driver to miss seeing his grandma before she dies. We can justify anything we want with hypotheticals.

Besides, it's not the fault of the street-crossing that it's not located conveniently. That's the job of the city planner or the council or whoever is responsible. If there's no toilet where it's conveniently accessible to you, should we just abandon toilets altogether and let everyone "go" wherever they want? After all, it's their liberty, right? And they're not harming any other human beings.

Restrictive laws made on hasty lack of consideration could easily abound if they are tolerated.

I agree. We shouldn't tolerate laws made on hasty lack of consideration. But that doesn't mean we should abandon all laws altogether simply because some of them are idiotic. Why not just get rid of idiotic laws?

As for escalation, how exactly would you prevent me from crossing in the middle of the street?

I won't. Maybe I'll take your photo and publicly shame you. Maybe send the photo to the authorities and you pay a fine. Or not. That's the thing. You can't force anyone to do anything. It's about rules and consequences. If you don't want to follow the rules then you face the consequences. Or if you're lucky, maybe you don't. But the rules still need to be in place.

Again, same with taxes. You don't have to pay them but there may be consequences if you don't.

I should probably clarify that I include "loss of liberty" as a type of "harm"

Okay, now we're getting somewhere. I think this is where we fundamentally disagree. You see loss of liberty as harmful. In my case, I don't see the world in black and white. To me, some instances of loss of liberty can be harmful. Other instances not. It always depends.

If it depends, how do we decide which is which? Well, that's up to the society to decide by implementing rules and laws. Some won't be happy with them but tough luck. That's life. I don't subscribe to the idea that society as a whole needs to cater to every single individual. Because that's impossible and will only lead to complete chaos.

And again, if you don't like the society you're living in then either move to another one or start your own.

Edit: I forgot another option. You can of course work to change the society you're currently living. If it were me, I'd work towards removing all stupid laws. But if you want to remove all laws, well, good luck with that.

Edit 2: I will concede this: laws that are implemented for your own good (like sugary drinks in New York) infringe on a person's liberty and I'm not in favor of those. Laws that are implemented for the good of others, those I'm in favor of. Why? Because I don't trust all humans to act ethically. If they all did, then sure, by all means let's get rid of all rules and laws.