Comment by earlythesis
With closed software, the developer has power over the user that the user can’t inspect, understand or challenge. In any other part of life, unaccountable power over people is something we treat as ethically suspect, not neutral.So why have we allowed so much of our lives to be controlled this way?
Think about what runs on code you’ll never be allowed to see. Your bank. Your car. Your phone, your thermostat, your doctor’s records, the payment terminal at the supermarket, the logistics system that decides whether there’s anything on the shelves when you get there. We didn’t vote for this. It just happened, one convenient app and one “I agree” button at a time.
I think we’re at the beginning of a dark digital age, and it won’t stay digital. It will hit the physical world hard. Most of us would fall apart without access to our digital assets, products and comforts. Payments, food supply, transport and communication all run through systems we can’t inspect and couldn’t fix if we wanted to.
And now look at what’s happening. AI models have become powerful enough that finding and exploiting vulnerabilities is getting cheaper and faster. Systems are being broken into constantly. The attackers have new tools. The defenders are mostly the same small, overworked teams as before, protecting code that nobody else is allowed to look at.
That’s the part that keeps me up. Closed software was always a trust exercise: trust us, we’ve got it handled. That trust is now being tested by attackers who can work faster than any human team, against codebases where the only people allowed to audit them are the vendors themselves.
Humans are not prepared for this. We built a society that can’t function without software, and then made sure almost nobody can understand, verify or repair the software it runs on.
Don’t people realize how fragile this is? It doesn’t take a movie-style apocalypse. It takes a few critical systems going down at the same time, and suddenly the convenient digital life turns into standing in line for toilet paper to be handed out.
I’m not saying every app needs to be open source tomorrow. But for infrastructure we can’t live without, “you’re not allowed to see how it works” should no longer be an acceptable answer.
Am I overreacting, or are we sleepwalking into this?