Naive's Bitcoin Scaling Question(s)

I am a web developer for quite a while now (years) but I am very new to bitcoin or blockchain-related development; I still consider myself "naive" on this field.

However, I have been watching for many months now, or maybe already a year or two, the hot issue about it's scaling dilemma and the solutions that has been presented. Segwit, Segwit2x, UASF, what else? Until now, there seems to be no consensus on which solution is the best for the entire community.

From what I have understood, as much as possible, the users would not want a solution that will result in splitting the bitcoin blockchain. And it seems like there is no 100% guarantee for that in any of the solutions presented (correct me if I'm wrong).

However, I asked myself, why is it so hard to get consensus to an upgrade?

I am just thinking from the common programmer's point of view. As a computer programmer in my own language, upgrading a software shouldn't cause so much harm if the upgrade support backward compatibility. If we still allow users to keep using an older or oldest version of the software if they wanted to. It will still be the same software, pointing to the same database. However, if they wanted to acquire the benefits of a newer version, they can just install it anytime.

Having said that, with regards to increasing the block size to 2MB for example, is it not possible to make such change in a way that bitcoin users (or miners) who don't want to upgrade can still use the old bitcoin software and still be part of the same blockchain, connecting to the same database or bitcoin ledger?

From the common programmer's point of view, I think something like that is very possible. But then again, I am still naive in this blockchain or bitcoin technology so most probably I am wrong. However, if something like that is possible, or if future changes or improvements to bitcoin technology can be implemented with backward-compatibility, there should be no reasons for the many of the bitcoin end-users to worry about an upgrade because they don't have to upgrade their own software version if they don't want to.

Your thoughts will be appreciated on this. I appreciate some first-hand education about this topic.