Just yesterday I talked with J. about startup ideas. I mostly complained about how LLMs still suck for enabling you to actually get work done, and that we needed better, more personal assistants.
I really think intellectual, white-collar, etc. types have way more tokens than they realize. It's 2026, not 1996. If you sit in front of a computer to do your work, or use computers to communicate with other people, you spend your days generating data that can be readily ingested and processed to help with training.
Then you have a lot of possible tricks for sample-efficiency - sample-efficiency is not actually pursued very hard in LLM research and it's not hard to get >10x. Then there's more active methods, like interactive quizzes, QA comparisons... it's not that hard to label hundreds or thousands of datapoints if it's worth it. No one does it right now with their own stuff because it's not worth it.
Further, ChatGPT can't make optimal use of the context it has because it's an incorrigible chatbot persona. Apparently they don't even train the LLM on the user statements. afaict, the gradients are always masked out, and the loss is computed only on the assistant's tokens.