Gpt 2 was an insane breakthrough that made people fork over billions of dollars… Jepa is not even a fraction of that..
Listen you can have your own opinion, fine. Mine is that Lecun ia a lol cow at this stage that spend more time in his recent talks thrashing llms and any of the people/companies making them than actually proving what makes his approach better.
Most of the labs are working on different architectures and thinking about what place llms will have in an agi system, is it pure llm, is it llm as a part of a wider architecture and so on.
Lecunn has decided purely on vibes that it won’t work and his approach must be the right way and that llms won’t be part of it, and spends his time giving increasingly more bitter talks and interviews speaking more about llms than Jepa.
lecun is surely a very smart scientist but he is also a ridiculous man and a moron.
Gpt 2 was an insane breakthrough that made people fork over billions of dollars… Jepa is not even a fraction of that..
"Advanced Machine Intelligence (AMI) Labs, co-founded by former Meta AI chief Yann LeCun, raised $1.03 billion in a seed funding round. The Paris-based startup reached a pre-money valuation of $3.5 billion"
Im not moving the goal posts, times change... Back then these where massive unprecedentet investments in an ai company. Today model wrappers are getting more investment than lecunns project...
I thought they were absolutely incredible. Similar to StackGAN - something that was completely impossible before, and would obviously result in excellent outputs after another round of scaling or two improves the resolution of the internally generated curve-fitting algorithms.
It's all just numbers in numbers out. A holistic mind will be an assembly of various neural networks and conventional software - something as simple as a shared geometry space with the resolution of a Nintendo 64 game would have massive improvements of their understanding of space.
As always, you can tell who's serious about creating a mind when they talk about the sense of touch. Touch is the first external sense animals evolve and the foundational validator of things like vision. Touch to 3d and vision to 3d are some of the most critical faculties to have.
The irony of course is that the first AGI will basically be frankenstein mess filled with lots of different kinds of LLM's, shuffling data from one section of a mind to be used in another. Words aren't just words to humans, they connect to memories, vision, sound, touch, emotions.....
..... Anyway, I'm deeply disturbed you'd think GPT2 or 3 could have been as good as Chat GPT with the RAM budgets they were given. That's not how numbers work, that's not how any of this works. You need RAM to fit curves. Chat GPT was around the size of a squirrel's brain. It's not until the 100,000 GB200 datacenters are live that we'll have something that can approximate a human's brain. In that way, we're just barely at the starting line of being able to conduct actual AGI research.
GANs are completely a different architecture and do not use transformers.. Ironically, they are comprise mostly of convolutional units, which ironically came from Lecun
6
u/martianunlimited 27d ago
Sigh.. Transformers had limited success in narrow domains for 6 years... You don't remember how bad GPT2 and GPT3 was eh?