Comment by pSYoniK

I took a note a few months ago and my point was I think more engineering specific, although it might be my own lacking abilities/skills that caused this realization. "Your capacity to learn/recall and map information is the new bottleneck. LLMs can act as learning amplifiers but correctness isn't as important for their output as critical thinking on the side of the consumer - YOU."

My point is that, I think if an LLM outputs 50,000 lines of code, your ability to go through what has changed, how it has changed and where the changes have occurred is the bottleneck. I see the approaches here, sure, "summarize the changes" or "draw me a picture" or the more recently observed "build me a city building simulator to understand this", but I feel that misses the point from an engineering perspective. The difference in understanding the weeds such as DB transactional boundaries or tenant isolation (which I believe was a topic in a recent data leak), those aren't summarized that easily in drawings or if they are, if you are working at this granularity, then your 50,000 line PR will yield 50,000 pages of crayon drawings you now have to understand.

I guess, my point is that understanding is the bottleneck, but low level understanding and the ability to read/map/connect is even more so. Any developer with some experience will agree that if changes are trivial you can scan and pick up mistakes or flaws easily. So most SOTA models won't necessarily even make these. So what you're reviewing now is going to be one level higher or more in terms of difficulty, mapping multiple components or touching multiple surfaces. Your ability to make the links, reason about them and attempt to find flaws or logic issues is the bottleneck. In the time it takes you to understand, another 50,000 line PR is up.

I'm not sure how we're going to be solving this. I don't know if in the current state it is a solvable issue, maybe another 6 months? Maybe another 6 years? Maybe this is fine and we will settle in a sort of place where your mediocre engineer will be responsible for tens of reviews a day signing off on method/functions/classes/interfaces being added, get paid 50k a year and doing the same non-thinking work day in day out while signing their name to the quality of the code being shipped while a senior/lead will be busy reviewing multiple of these. Think of the way an assembly line functions.

P.S. I hate to see this annoying tendency of transforming knowledge work into assembly line work. We keep trying to "fix" this without understanding what knowledge itself is. Maybe this technology will indeed yield software assembly lines, I don't wish to eat my words, but I'm still struggling to see how we will handle the nitty gritty of software work. Maybe the same way we handle building airplanes - as long as only a couple crash a year, we're sort of fine.