動画数：17件
Intro into the growing LLM ecosystem
The audio modes demos at pm were simply amazing!
ChatGPT interaction under the hood
Basic LLM interactions examples
Exactly, I have observed this so many times!! Why do these chat platforms not have an option to branch out to a new chat (for exploring multiple ideas or something) from a particular answer point? Are there any technical challenges?
new chat --->for new topic
Be aware of the model you're using, pricing tiers
Personal note:
Thinking models and when to use them
Tool use: internet search
Is one Search one token in the context window?
- Casual sneeze making the video even more fun
Bless you, Andrej-!
search
We Vietnamese always cherish exceptional talents like you, Andrej.
Tool use: deep research
ChatGPT was not the first to offer Deep Research. Gemini made Deep Research available on December 11, 2024. ChatGPT added theirs February 2, 2025.
You missed Gemini Deep Research. That’s the original one.
What we would really need is ability to pass the response with all the provided references to another thinking + internet access AI system with a task "Does this article content match the provided references?". I'm pretty sure that different AI models do not accidentally hallucinate badly enough to fail this kind of verification task most of the time.
File uploads, adding documents to context
Re , accessing .epub in context would be a win.Imagine clicking Table of Contents Chapter inside of Cursor or ChatGPT Platform and having it ready for the selected LLM.. 📖 🙂
I think Copilot in Edge allows you to ask questions in a taskpane and also supports marking as i remember. . Thanks for your Insights!
you need the Highlight app. it literally takes into context whatever document you have opened in your system, so no copying is needed. very smooth
I suggest using kortex for large amount of pdf or books that can be using with an LLM. I am not sure about each LLMs limit in terms of document upload (MB) and how is connected with token input limits, I would like to know more about this
You could just have the ChatGPT floating window open while you read a book in full-screen. That way, you don’t have to keep switching between windows. 👍🏻
"don't read books alone"
Tool use: python interpreter, messiness of the ecosystem
Gemini's prediction is not actually close. It is lower by an order of 3. But another amazing video by Andrej ! Thank you :)
ChatGPT Advanced Data Analysis, figures, plots
keep in mind if you reading this, just because it uses an internet source, doesn’t mean it won’t hallucinate content it thinks it found in the source
0.1 is a heuristic to avoid 0, which may behave badly?
Claude Artifacts, apps, diagrams
This is pure gold. Andrej is the best teacher on all things AI. He teaches with such clarity and simplicity that the knowledge just sticks. I just wish that the part about coding between - 1. a disclaimer when there are high vulnerabilities in node dependencies (2. discusses the legal aspects of using code generated by llms or llm powered tools like cursor, windsurf, github copilot etc. I really wish such videos talk about this crucial aspect else most viewers will get a sense that software development is as simple as just prompting LLMs for code and they can use the code generated as it is. There are many cases when such LLMs spit out copyrighted code or code under licenses and using them without attribution is risky.
---> conceptual diagram
Love the conceptual diagram idea. Very very useful
Cursor: Composer, writing code
)
The confetti moment got me excited too. Amazing video, Andrej, thank you!
showed
talk to llms
Audio (Speech) Input/Output
What a gigachad. And yet for some reasons he doesn't seem to be aware that his Mac comes with Dictation feature (). Maybe he has an older model of MacOS. Maybe I'm missing something but this section of the video makes no sense to me. But again, what an amazing video by a generous genius!
The native ChatGPT app for macOS does have the mic icon.
Why don't you use mac dictate feature?
Advanced Voice Mode aka true audio inside the model
kind of how shazam works under the hood, by getting a graph made for the audio spectogram and by identifying the peak points in the graph with background noise minimized and then it those peak points being converted to audio fingerprints and at last based on the fingerprint it searches its database of millions of songs.
Your reaction at killed me lmao
NotebookLM, podcast generation
Image input, OCR
woke up in the middle of the night to find that I had been listening to this all night. If I magically know a bunch of shit about LLMs….im going to be shook
For those interested, the math problem at is not that tricky 🙃.
No Andrej, you failed me to trick😎😅
Image output, DALL-E, Ideogram, etc.
Video input, point and talk on app
Video output, Sora, Veo 2, etc etc.
ChatGPT memory, custom instructions
whenever you make a typo while typing, that should be a reminder to type with superwhisper instead
"I am Andrej Karpathy; Yes - the AI researcher" What an insane flex. Imagine confirming to an LLM that it's indeed talking to that guy you actually have training memory on.
Custom GPTs
Can you add a reverse (round-trip) button to your translator? It's a great way to test the "stability" of a translation.
agree 👍 going to use it
Summary
introduction
- Introduction
pretraining data (internet)
- LLM Pre-training
Atound, you explain a really interesting notion, that models need to "think" before producing a complex response, thats because each layer in a neural network has finite computation. I feel like its somewhat related to the notion of computational irreducibility Stephen Wolfram talks about. This is also why we humans need to spend some time thinking about complex issues before coming up with a good response.
But what if the ultimate joke about pelicans is actually 'the the the the the the,' but we simply don't have enough intelligence to understand it—just like an unusual move in the game of Go? XD
wow amazing hours so much in few hours .. Saved me hours of research and insprie me for more ..great work looking forward for new such interesting videos..
at , talks about eliminating racist sites during corpus preprocessing. This can introduce bias by eliminating candid discussion of, for example, average IQ test scores of racial subgroups. Claude refuses to answer this altogether, calling race a constructed concept. ChatGPT and Gemini, at the time I queried them, both produced valid, honest outputs, which aligned with the research. Those of you so enamored with Claude are still trapped in Dario's echo-chamber. But society has moved on, now (2025). Will you?
tokenization
neural network I/O
- Neural Net & Training
neural network internals
inference
GPT-2: training and inference
Somewhere around , you said something about training 1 million tokens. Do you mean you train chunks of 1 million tokens to generate output or you train different tokens that add up to a million to generate output?
- GPUs & Model Costs
Llama 3.1 base model inference
: Parallel universes !!! Just loving these analogies - awesome !
pretraining to post-training
post-training data (conversations)
- Build LLM Assistant
"something went wrong" 😂 lol I love that he left this in there!
his genuine laugh at ChatGPT error is so pure and spontaneous. How can someone not love Karpathy!!?? Sir you are pure Gold for humanity.
hallucinations, tool use, knowledge/working memory
The chapter about hallucinations was so insightful. Never heard about it as an issue of the dataset, i.e., it wasn't trained to say "I don't know" and how one can test the knowledge of the model. Thanks!
Observation: Approx. at , Andrej tests the question "Who is Orson Kovacs" using falcon-7b-instruct in HF playground, the temperature is still 1.0 which will make the model to respond in a balanced manner between randomness and deterministic. Although it makes up stuff to behave like hallucinations, it is good to test out with temperature less or more than 1.0 to understand how the factuality of the data varies.
you mentioned around mark - the reason why you allow the model to say i don't know, instead of augmenting it with the new knowledge, is it because there's infinite amount of knowledge to learn so that it's virtually impossible to learn knowledge, and thus it's better to train it to know when to refuse? In other words, say if somehow the model CAN learn ALL the knowledge of the world, we won't need to train it to stop hallucinating? Thanks.
Thanks for the informative video! I have a question about training language models for tool use, specifically regarding the process you described around
knowledge of self
models need tokens to think
@. Question. I was just reading a paper recently (I believe it was from Anthropic, but sadly I can't find it now) that when they have looked at "thinking models", it appears the final answer is generally already determined well before the reasoning process begins. Then the model just fills in the chain of thought to get from the question to where it wants to go. Isn't this exactly what you said is not the correct way to handle this? Can you comment on why, if this is the "wrong" approach, it seems to be what modern models are doing?
@ that is elucidating! This is the first time I’ve heard of this concept. Thank you Andrej.
This teacher is very good at giving cute examples Appreciate it and I agree it.
tokenization revisited: models struggle with spelling
Wow.. love this explanation about why these models fail at character related and counting related task
jagged intelligence
supervised finetuning to reinforcement learning
- Model Training in Practice
reinforcement learning
DeepSeek-R1
Deepseek says “$3 is a bit expensive for an apple, but maybe they’re organic or something” 😂
What a treat!!! At , haha when you say this is very busy very ugly because of google not being able to nail that was epic hahah
AlphaGo
Thank you for the video Andrej! One small note: at , the dashed line in the AlphaGo Zero plot is the Elo of the version of AlphaGo that *defeated* Lee in 2016 (not the Elo of Lee himself).
reinforcement learning from human feedback (RLHF)
Tiny typo "let's add it to the dataset and give it an ordering that's extremely like a score of 5" -> SHOULD BE "let's add it to the dataset and give it an ordering that's extremely like a score of 1"
preview of things to come
keeping track of LLMs
if you have come till this time stamp then finish the video and go and build something with LLMs.😊
where to find LLMs
grand summary
In principle these models are capable of analogies no human has had. Wow😮