The Bitter Lesson

Software
1

This article from 2019 documents how scaling computation always outperforms building human reasoning into systems.

And some modern commentary:

I don’t believe humans are meant to be bound to screens at a desk writing boilerplate code for half of their waking hours.

I don’t believe humans are meant to sift through reams of legal documents searching for specific information.

I don’t believe humans are meant to painstakingly transcribe hundred year old documents into a digital format.

I do believe we’re meant to create.

I do believe we’re meant to inquire into the nature of truth.

2

What’s interesting is that heuristics can be useful for building the system in the short term. It seems like this is used initially to bootstrap the AI system. Over time, the “custom code” gets replaced by the “neutral network.”

I’m not confident that you can go straight to the “everything is a neutral network” from the start.