Andrej Karpathyの動画一覧です。 https://favtu.be/videos-rss/c/UCXUPKJO5MZQN11PqgIvyuvQ Fri, 28 Feb 25 07:29:56 +0900 https://favtu.be/timelines/v/EWvNQjAaOHw Fri, 28 Feb 25 07:29:56 +0900 00:00:00 Intro into the growing LLM ecosystem 00:02:54 ChatGPT interaction under the hood 00:13:12 Basic LLM interactions examples 00:18:03 Be aware of the model you're using, pricing tiers 00:22:54 Thinking models and when to use them 00:31:00 Tool use: internet search 00:42:04 Tool use: deep research 00:50:57 File uploads, adding documents to context 00:59:00 Tool use: python interpreter, messiness of the ecosystem 01:04:35 ChatGPT Advanced Data Analysis, figures, plots 01:09:00 Claude Artifacts, apps, diagrams 01:14:02 Cursor: Composer, writing code 01:22:28 Audio (Speech) Input/Output 01:27:37 Advanced Voice Mode aka true audio inside the model 01:37:09 NotebookLM, podcast generation 01:40:20 Image input, OCR 01:47:02 Image output, DALL-E, Ideogram, etc. 01:49:14 Video input, point and talk on app 01:52:23 Video output, Sora, Veo 2, etc etc. 01:53:29 ChatGPT memory, custom instructions 01:58:38 Custom GPTs 02:06:30 Summary 00:00:00 Intro into the growing LLM ecosystem 00:02:54 ChatGPT interaction under the hood 00:13:12 Basic LLM interactions examples 00:18:03 Be aware of the model you're using, pricing tiers 00:22:54 Thinking models and when to use them 00:31:00 Tool use: internet search 00:42:04 Tool use: deep research 00:50:57 File uploads, adding documents to context 00:59:00 Tool use: python interpreter, messiness of the ecosystem 01:04:35 ChatGPT Advanced Data Analysis, figures, plots 01:09:00 Claude Artifacts, apps, diagrams 01:14:02 Cursor: Composer, writing code 01:22:28 Audio (Speech) Input/Output 01:27:37 Advanced Voice Mode aka true audio inside the model 01:37:09 NotebookLM, podcast generation 01:40:20 Image input, OCR 01:47:02 Image output, DALL-E, Ideogram, etc. 01:49:14 Video input, point and talk on app 01:52:23 Video output, Sora, Veo 2, etc etc. 01:53:29 ChatGPT memory, custom instructions 01:58:38 Custom GPTs 02:06:30 Summary 00:00:00 Intro into the growing LLM ecosystem 00:02:54 ChatGPT interaction under the hood 00:13:12 Basic LLM interactions examples 00:18:03 Be aware of the model you're using, pricing tiers 00:22:54 Thinking models and when to use them 00:31:00 Tool use: internet search 00:42:04 Tool use: deep research 00:50:57 File uploads, adding documents to context 00:59:00 Tool use: python interpreter, messiness of the ecosystem 01:04:35 ChatGPT Advanced Data Analysis, figures, plots 01:09:00 Claude Artifacts, apps, diagrams 01:14:02 Cursor: Composer, writing code 01:22:28 Audio (Speech) Input/Output 01:27:37 Advanced Voice Mode aka true audio inside the model 01:37:09 NotebookLM, podcast generation 01:40:20 Image input, OCR 01:47:02 Image output, DALL-E, Ideogram, etc. 01:49:14 Video input, point and talk on app 01:52:23 Video output, Sora, Veo 2, etc etc. 01:53:29 ChatGPT memory, custom instructions 01:58:38 Custom GPTs 02:06:30 Summary 00:21:00 Personal note: 00:37:00 search 01:22:00 talk to llms 01:09:18 This is pure gold. Andrej is the best teacher on all things AI. He teaches with such clarity and simplicity that the knowledge just sticks. I just wish that the part about coding between - 1. a disclaimer when there are high vulnerabilities in node dependencies (2. discusses the legal aspects of using code generated by llms or llm powered tools like cursor, windsurf, github copilot etc. I really wish such videos talk about this crucial aspect else most viewers will get a sense that software development is as simple as just prompting LLMs for code and they can use the code generated as it is. There are many cases when such LLMs spit out copyrighted code or code under licenses and using them without attribution is risky. 01:22:00 showed 01:19:23 ) 00:17:31 new chat --->for new topic 01:12:25 ---> conceptual diagram 01:05:53 0.1 is a heuristic to avoid 0, which may behave badly? 01:25:26 Why don't you use mac dictate feature? 01:35:14 Your reaction at killed me lmao 01:54:50 whenever you make a typo while typing, that should be a reminder to type with superwhisper instead 00:42:51 ChatGPT was not the first to offer Deep Research. Gemini made Deep Research available on December 11, 2024. ChatGPT added theirs February 2, 2025. 01:04:08 Gemini's prediction is not actually close. It is lower by an order of 3. But another amazing video by Andrej ! Thank you :) 00:16:55 Exactly, I have observed this so many times!! Why do these chat platforms not have an option to branch out to a new chat (for exploring multiple ideas or something) from a particular answer point? Are there any technical challenges? 00:34:28 Is one Search one token in the context window? 01:25:10 What a gigachad. And yet for some reasons he doesn't seem to be aware that his Mac comes with Dictation feature (). Maybe he has an older model of MacOS. Maybe I'm missing something but this section of the video makes no sense to me. But again, what an amazing video by a generous genius! 01:25:20 The native ChatGPT app for macOS does have the mic icon. 00:58:50 You could just have the ChatGPT floating window open while you read a book in full-screen. That way, you don’t have to keep switching between windows. 👍🏻 00:36:35 Bless you, Andrej-! 00:48:20 What we would really need is ability to pass the response with all the provided references to another thinking + internet access AI system with a task "Does this article content match the provided references?". I'm pretty sure that different AI models do not accidentally hallucinate badly enough to fail this kind of verification task most of the time. 02:03:26 agree 👍 going to use it 02:02:19 Can you add a reverse (round-trip) button to your translator? It's a great way to test the "stability" of a translation. 00:58:50 I suggest using kortex for large amount of pdf or books that can be using with an LLM. I am not sure about each LLMs limit in terms of document upload (MB) and how is connected with token input limits, I would like to know more about this 01:28:20 kind of how shazam works under the hood, by getting a graph made for the audio spectogram and by identifying the peak points in the graph with background noise minimized and then it those peak points being converted to audio fingerprints and at last based on the fingerprint it searches its database of millions of songs. 00:45:15 You missed Gemini Deep Research. That’s the original one. 00:01:30 The audio modes demos at pm were simply amazing! 01:20:35 The confetti moment got me excited too. Amazing video, Andrej, thank you! 00:41:26 We Vietnamese always cherish exceptional talents like you, Andrej. 00:58:40 you need the Highlight app. it literally takes into context whatever document you have opened in your system, so no copying is needed. very smooth 01:14:00 Love the conceptual diagram idea. Very very useful 01:05:23 keep in mind if you reading this, just because it uses an internet source, doesn’t mean it won’t hallucinate content it thinks it found in the source 01:57:55 "I am Andrej Karpathy; Yes - the AI researcher" What an insane flex. Imagine confirming to an LLM that it's indeed talking to that guy you actually have training memory on. 00:58:27 Re , accessing .epub in context would be a win.Imagine clicking Table of Contents Chapter inside of Cursor or ChatGPT Platform and having it ready for the selected LLM.. 📖 🙂 00:36:34 - Casual sneeze making the video even more fun 00:58:58 "don't read books alone" 01:44:44 woke up in the middle of the night to find that I had been listening to this all night. If I magically know a bunch of shit about LLMs….im going to be shook 01:45:00 For those interested, the math problem at is not that tricky 🙃. 00:58:30 I think Copilot in Edge allows you to ask questions in a taskpane and also supports marking as i remember. . Thanks for your Insights! 01:45:28 No Andrej, you failed me to trick😎😅 How I use LLMs The example-driven, practical walkthrough of Large Language Models and their growing list of related features, as a new entry to my general audience series on LLMs. In this more practical followup, I take you through the many ways I use LLMs in my own life. Chapters 00:00:00 Intro into the growing LLM ecosystem 00:02:54 ChatGPT interaction under the hood 00:13:12 Basic LLM interactions examples 00:18:03 Be aware of the model you're using, pricing tiers 00:22:54 Thinking models and when to use them 00:31:00 Tool use: internet search 00:42:04 Tool use: deep research 00:50:57 File uploads, adding documents to context 00:59:00 Tool use: python interpreter, messiness of the ecosystem 01:04:35 ChatGPT Advanced Data Analysis, figures, plots 01:09:00 Claude Artifacts, apps, diagrams 01:14:02 Cursor: Composer, writing code 01:22:28 Audio (Speech) Input/Output 01:27:37 Advanced Voice Mode aka true audio inside the model 01:37:09 NotebookLM, podcast generation 01:40:20 Image input, OCR 01:47:02 Image output, DALL-E, Ideogram, etc. 01:49:14 Video input, point and talk on app 01:52:23 Video output, Sora, Veo 2, etc etc. 01:53:29 ChatGPT memory, custom instructions 01:58:38 Custom GPTs 02:06:30 Summary Links - Tiktokenizer https://tiktokenizer.vercel.app/ - OpenAI's ChatGPT https://chatgpt.com/ - Anthropic's Claude https://claude.ai/ - Google's Gemini https://gemini.google.com/ - xAI's Grok https://grok.com/ - Perplexity https://www.perplexity.ai/ - Google's NotebookLM https://notebooklm.google.com/ - Cursor https://www.cursor.com/ - Histories of Mysteries AI podcast on Spotify https://open.spotify.com/show/3K4LRyMCP44kBbiOziwJjb - The visualization UI I was using in the video: https://excalidraw.com/ - The specific file of Excalidraw we built up: https://drive.google.com/file/d/1DN3LU3MbKI00udxoS-W5ckCHq99V0Uqs/view?usp=sharing - Discord channel for Eureka Labs and this video: https://discord.gg/3zy8kqD9Cp Educational Use Licensing This video is freely available for educational and internal training purposes. Educators, students, schools, universities, nonprofit institutions, businesses, and individual learners may use this content freely for lessons, courses, internal training, and learning activities, provided they do not engage in commercial resale, redistribution, external commercial use, or modify content to misrepresent its intent. https://favtu.be/timelines/v/7xTGNNLPyMI Thu, 06 Feb 25 03:23:47 +0900 00:00:00 introduction 00:01:00 pretraining data (internet) 00:07:47 tokenization 00:14:27 neural network I/O 00:20:11 neural network internals 00:26:01 inference 00:31:09 GPT-2: training and inference 00:42:52 Llama 3.1 base model inference 00:59:23 pretraining to post-training 01:01:06 post-training data (conversations) 01:20:32 hallucinations, tool use, knowledge/working memory 01:41:46 knowledge of self 01:46:56 models need tokens to think 02:01:11 tokenization revisited: models struggle with spelling 02:04:53 jagged intelligence 02:07:28 supervised finetuning to reinforcement learning 02:14:42 reinforcement learning 02:27:47 DeepSeek-R1 02:42:07 AlphaGo 02:48:26 reinforcement learning from human feedback (RLHF) 03:09:39 preview of things to come 03:15:15 keeping track of LLMs 03:18:34 where to find LLMs 03:21:46 grand summary 00:00:01 - Introduction 00:01:04 - LLM Pre-training 00:15:13 - Neural Net & Training 00:40:14 - GPUs & Model Costs 01:01:06 - Build LLM Assistant 02:07:30 - Model Training in Practice 00:03:02 But what if the ultimate joke about pelicans is actually 'the the the the the the,' but we simply don't have enough intelligence to understand it—just like an unusual move in the game of Go? XD 03:29:54 In principle these models are capable of analogies no human has had. Wow😮 00:36:52 Somewhere around , you said something about training 1 million tokens. Do you mean you train chunks of 1 million tokens to generate output or you train different tokens that add up to a million to generate output? 01:30:00 you mentioned around mark - the reason why you allow the model to say i don't know, instead of augmenting it with the new knowledge, is it because there's infinite amount of knowledge to learn so that it's virtually impossible to learn knowledge, and thus it's better to train it to know when to refuse? In other words, say if somehow the model CAN learn ALL the knowledge of the world, we won't need to train it to stop hallucinating? Thanks. 01:33:38 Thanks for the informative video! I have a question about training language models for tool use, specifically regarding the process you described around 02:41:08 What a treat!!! At , haha when you say this is very busy very ugly because of google not being able to nail that was epic hahah 01:23:50 Observation: Approx. at , Andrej tests the question "Who is Orson Kovacs" using falcon-7b-instruct in HF playground, the temperature is still 1.0 which will make the model to respond in a balanced manner between randomness and deterministic. Although it makes up stuff to behave like hallucinations, it is good to test out with temperature less or more than 1.0 to understand how the factuality of the data varies. 01:55:50 This teacher is very good at giving cute examples Appreciate it and I agree it. 02:43:05 Thank you for the video Andrej! One small note: at , the dashed line in the AlphaGo Zero plot is the Elo of the version of AlphaGo that *defeated* Lee in 2016 (not the Elo of Lee himself). 00:01:49 Atound, you explain a really interesting notion, that models need to "think" before producing a complex response, thats because each layer in a neural network has finite computation. I feel like its somewhat related to the notion of computational irreducibility Stephen Wolfram talks about. This is also why we humans need to spend some time thinking about complex issues before coming up with a good response. 00:03:30 wow amazing hours so much in few hours .. Saved me hours of research and insprie me for more ..great work looking forward for new such interesting videos.. 03:16:59 if you have come till this time stamp then finish the video and go and build something with LLMs.😊 01:52:00 @. Question. I was just reading a paper recently (I believe it was from Anthropic, but sadly I can't find it now) that when they have looked at "thinking models", it appears the final answer is generally already determined well before the reasoning process begins. Then the model just fills in the chain of thought to get from the question to where it wants to go. Isn't this exactly what you said is not the correct way to handle this? Can you comment on why, if this is the "wrong" approach, it seems to be what modern models are doing? 01:18:46 "something went wrong" 😂 lol I love that he left this in there! 01:55:49 @ that is elucidating! This is the first time I’ve heard of this concept. Thank you Andrej. 00:55:22 : Parallel universes !!! Just loving these analogies - awesome ! 01:20:32 The chapter about hallucinations was so insightful. Never heard about it as an issue of the dataset, i.e., it wasn't trained to say "I don't know" and how one can test the knowledge of the model. Thanks! 01:18:47 his genuine laugh at ChatGPT error is so pure and spontaneous. How can someone not love Karpathy!!?? Sir you are pure Gold for humanity. 02:04:04 Wow.. love this explanation about why these models fail at character related and counting related task 00:03:50 at , talks about eliminating racist sites during corpus preprocessing. This can introduce bias by eliminating candid discussion of, for example, average IQ test scores of racial subgroups. Claude refuses to answer this altogether, calling race a constructed concept. ChatGPT and Gemini, at the time I queried them, both produced valid, honest outputs, which aligned with the research. Those of you so enamored with Claude are still trapped in Dario's echo-chamber. But society has moved on, now (2025). Will you? 02:34:21 Deepseek says “$3 is a bit expensive for an apple, but maybe they’re organic or something” 😂 03:31:23 Thank you Andrej for this! Please continue putting contents like this and you are one of the best teachers in this space who can explain in this level of detail. The entire is pure gold and very grateful that you are putting this level of time and effort ❤ 03:03:44 Tiny typo "let's add it to the dataset and give it an ordering that's extremely like a score of 5" -> SHOULD BE "let's add it to the dataset and give it an ordering that's extremely like a score of 1" Deep Dive into LLMs like ChatGPT This is a general audience deep dive into the Large Language Model (LLM) AI technology that powers ChatGPT and related products. It is covers the full training stack of how the models are developed, along with mental models of how to think about their "psychology", and how to get the best use them in practical applications. I have one "Intro to LLMs" video already from ~year ago, but that is just a re-recording of a random talk, so I wanted to loop around and do a lot more comprehensive version. Instructor Andrej was a founding member at OpenAI (2015) and then Sr. Director of AI at Tesla (2017-2022), and is now a founder at Eureka Labs, which is building an AI-native school. His goal in this video is to raise knowledge and understanding of the state of the art in AI, and empower people to effectively use the latest and greatest in their work. Find more at https://karpathy.ai/ and https://x.com/karpathy Chapters 00:00:00 introduction 00:01:00 pretraining data (internet) 00:07:47 tokenization 00:14:27 neural network I/O 00:20:11 neural network internals 00:26:01 inference 00:31:09 GPT-2: training and inference 00:42:52 Llama 3.1 base model inference 00:59:23 pretraining to post-training 01:01:06 post-training data (conversations) 01:20:32 hallucinations, tool use, knowledge/working memory 01:41:46 knowledge of self 01:46:56 models need tokens to think 02:01:11 tokenization revisited: models struggle with spelling 02:04:53 jagged intelligence 02:07:28 supervised finetuning to reinforcement learning 02:14:42 reinforcement learning 02:27:47 DeepSeek-R1 02:42:07 AlphaGo 02:48:26 reinforcement learning from human feedback (RLHF) 03:09:39 preview of things to come 03:15:15 keeping track of LLMs 03:18:34 where to find LLMs 03:21:46 grand summary Links - ChatGPT https://chatgpt.com/ - FineWeb (pretraining dataset): https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1 - Tiktokenizer: https://tiktokenizer.vercel.app/ - Transformer Neural Net 3D visualizer: https://bbycroft.net/llm - llm.c Let's Reproduce GPT-2 https://github.com/karpathy/llm.c/discussions/677 - Llama 3 paper from Meta: https://arxiv.org/abs/2407.21783 - Hyperbolic, for inference of base model: https://app.hyperbolic.xyz/ - InstructGPT paper on SFT: https://arxiv.org/abs/2203.02155 - HuggingFace inference playground: https://huggingface.co/spaces/huggingface/inference-playground - DeepSeek-R1 paper: https://arxiv.org/abs/2501.12948 - TogetherAI Playground for open model inference: https://api.together.xyz/playground - AlphaGo paper (PDF): https://discovery.ucl.ac.uk/id/eprint/10045895/1/agz_unformatted_nature.pdf - AlphaGo Move 37 video: https://www.youtube.com/watch?v=HT-UZkiOLv8 - LM Arena for model rankings: https://lmarena.ai/ - AI News Newsletter: https://buttondown.com/ainews - LMStudio for local inference https://lmstudio.ai/ - The visualization UI I was using in the video: https://excalidraw.com/ - The specific file of Excalidraw we built up: https://drive.google.com/file/d/1EZh5hNDzxMMy05uLhVryk061QYQGTxiN/view?usp=sharing - Discord channel for Eureka Labs and this video: https://discord.gg/3zy8kqD9Cp Educational Use Licensing This video is freely available for educational and internal training purposes. Educators, students, schools, universities, nonprofit institutions, businesses, and individual learners may use this content freely for lessons, courses, internal training, and learning activities, provided they do not engage in commercial resale, redistribution, external commercial use, or modify content to misrepresent its intent. #llm #chatgpt #ai #deep dive #deep learning #introduction #large language model https://favtu.be/timelines/v/l8pRSuU81PU Mon, 10 Jun 24 08:31:35 +0900 00:00:02 Reproducing the GPT-2 124M version 00:02:06 Validation loss measures model's performance on unseen data. 00:06:21 GPT-2 token and position embeddings explained 00:08:43 Understanding token positions and embeddings in GPT-2 (124M) 00:12:54 Implementing and understanding GPT-2 (124M) model architecture. 00:15:02 Creating a matching schema for loading weights easily. 00:19:20 The Transformer involves repeated application of map and reduce 00:21:22 GPT-2 used the 10h approximate version of G instead of the exact version. 00:25:09 GPT-2 (124M) implementation details 00:27:08 Efficient implementation in PyTorch for GPT-2 (124M) model 00:30:59 Forwarding the GPT-2 model requires processing token indices and embeddings. 00:32:52 Explaining the forward pass of the GPT-2 network 00:36:36 Generating logits and probabilities for token prediction 00:38:34 Using top K by default (50) helps keep the model on track 00:42:24 Using GPT-2 (124M) for model initialization 00:44:17 Initializing model on correct device is crucial for performance 00:48:08 Transforming single sequence into batch with structured tokens 00:50:03 Creating input and labels for Transformer 00:54:02 Flattening multi-dimensional tensors for cross entropy calculation. 00:56:04 The loss at initialization is expected to be around 10.82 but is seen around 11, which suggests a diffused probability distribution at initialization. 01:00:00 Explaining the device issue and fixing tensor moving bug. 01:01:52 Creating a simple data loader for iterating through batches of data. 01:05:46 Bug in GPT-2 training process 01:07:46 Common weight tying scheme in Transformer models 01:11:41 Weight sharing scheme reduces parameters and improves efficiency 01:13:45 Follow GPT-2 initialization scheme for better model performance 01:17:37 Controlling growth of activations in the residual stream 01:19:50 Setting flags and scaling standard deviation in GPT-2 model initialization. 01:23:53 Deep learning training can achieve higher performance by using lower precision formats. 01:25:55 Importance of using floating points over int8 for neural network training. 01:29:49 Matrix multiplication is accelerated through tensor cores. 01:32:01 Using tf32 for 8X faster performance with minor precision tradeoff. 01:35:49 Max out the batch size and use numbers with powers of two for better efficiency. 01:37:45 TF32 promises 8X throughput but only delivers 3X due to memory bottlenecks 01:41:50 Transition from fp16 to bf16 for simpler training. 01:43:50 Implementing B float 16 for minimal impact on model activations. 01:48:05 Introducing torch.compile for faster model compilation 01:49:59 Torch compile optimizes neural net operations efficiently 01:53:48 Optimizing round trips to GPU memory for faster computation 01:55:38 GPU chip architecture overview 01:59:29 Torch compilation utilizes kernel Fusion for speed optimization 02:01:24 Flash attention algorithm reduces memory usage and improves computation speed significantly. 02:05:15 Using Flash attention in PyTorch for faster runtime. 02:07:23 Prefer using powers of two in code for neural networks and CUDA. 02:11:14 Improved GPT-2 performance by fixing token index issue 02:13:04 Padding inputs for efficiency improvement 02:16:50 Setting hyperparameters for training GPT-3 02:18:48 Monitoring gradient norm is crucial for stability 02:22:39 Setting learning rate in GPT-2 (124M) 02:24:32 Implementing a learning rate schedule for training GPT-2 02:28:14 Data are sampled without replacement during training. 02:30:10 Weight decay parameters are split into those that should be weight decayed and those that should not be weight decayed. 02:33:55 Weight decay is applied to two-dimensional parameters. 02:35:53 Using gradient accumulation to simulate a large batch size 02:39:54 Demonstration of simple neural network implementation with mean squared loss 02:41:54 Gradients do not match due to loss normalization issue 02:45:49 Optimizing model training with gradient accumulation and distributed data parallelism. 02:47:41 Collaborative processing with multiple GPUs 02:51:32 Running with TorRun involves eight parallel processes with different ranks. 02:53:39 Introduction to GPU calculations in GPT-2 (124M) 02:57:43 Initialization of GPT-2 model training process 02:59:44 Wrapping the model into the Distributed Data Parallel container is important for constructing the M model. 03:03:37 Avoiding context managers and code duplication by directly toggling the variable. 03:05:42 Printing loss over all processes and averaging it 03:09:39 GPT-2 (124M) reproduction process summary 03:11:30 Training data mixtures are carefully curated and diverse. 03:15:22 Tokenizing and processing large datasets for GPT-2 model training. 03:17:22 Sharding data for easier disk management 03:21:39 Optimizing model training process for efficiency and quality. 03:23:39 Evaluating GPT-2 (124M) model performance 03:27:18 Troubleshooting torch compile issue 03:29:01 Language models trained with world knowledge outperform those with less training. 03:33:01 Construct batches of tokens with shared context and options for prediction 03:35:02 Model's inability to view all options at once 03:38:55 Running without torch compile affects code performance 03:40:48 Model training process overview 03:45:03 GPT-2 (124M) trained on 10 billion tokens matching or surpassing accuracy of GPT-2 (100B) trained on significantly fewer tokens 03:46:58 Issue with data shuffling affecting model training 03:50:41 Improving data shuffling and model efficiency. 03:52:31 Training model to mimic GPT-3 with sequence length adjustment 03:56:15 Comparison between nanog GPT in PyTorch and llm Doc and lm. C CUDA implementation 03:58:19 Comparing PyTorch and lm.C performance for training GPT-2 and GPT-3. 00:00:00 M) model, including its state dictionary and tensor shapes. We learn how the model's vocabulary size and embedding dimensions are represented within these tensors. (- 00:04:00 ) 00:04:00 compared to the original Transformer are explored, such as the removal of the encoder and cross-attention mechanism. Further, modifications to layer normalization placement and the addition of a final layer normalization layer are highlighted. (- 00:08:00 ) 00:08:00 , aligning it with the schema used by Hugging Face Transformers. This skeleton includes modules for token and positional embeddings, Transformer blocks, final layer normalization, and the language model head. (- 00:12:00 ) 00:12:00 is discussed. (- 00:16:00 ) 00:16:00 's implementation through tensor manipulation and its algorithmic similarity to previous implementations. (- 00:20:00 ) 00:20:00 Implementing the Forward Pass and Text Generation: The forward pass of the network is implemented, outlining how input token indices are processed to produce logits for predicting the next token in a sequence. This sets the stage for generating text from the model. (- 00:24:00 ) 00:24:00 model. This involves tokenizing a prefix string, moving the model to a CUDA device for GPU acceleration, and performing sampling-based text generation. (- 00:28:00 ) 00:28:00 Introducing the Tiny Shakespeare Dataset: This part introduces the Tiny Shakespeare dataset as a small and manageable dataset for initial model training and debugging. Basic statistics of the dataset are explored. (- 00:32:00 ) 00:32:00 model. It introduces the concept of batching and creating input-target pairs for loss calculation. (- 00:36:00 ) 00:36:00 Creating a Simple Data Loader: This section refactors the code to create a simple data loader object responsible for loading tokenized data from the Tiny Shakespeare dataset and generating batches suitable for training the model. (- 00:40:00 ) 00:40:00 Calculating Loss and Backpropagation: The forward function is adjusted to return not just the logits but also the calculated loss based on provided target tokens. Cross-entropy loss is used, and the initial loss is sanity-checked to ensure reasonable starting probabilities. (- 00:44:00 ) 00:44:00 Implementing Optimization with AdamW: This section introduces the AdamW optimizer as an alternative to stochastic gradient descent (SGD), highlighting its advantages for language model training. The optimization loop is implemented, including gradient accumulation and loss printing. (- 00:48:00 ) 00:48:00 Understanding and Addressing Device Mismatches: This part emphasizes the importance of ensuring all tensors and model components reside on the same device (CPU or GPU) to avoid errors during training. A bug related to tensor device mismatch is identified and corrected. (- 00:52:00 ) 00:52:00 model based on the original paper's guidelines. This includes using specific standard deviations for different layer types and scaling residual connections to control activation growth. (- 00:56:00 ) 00:56:00 GPU, focusing on its theoretical performance limits in terms of Teraflops for different floating-point precisions. The importance of memory bandwidth limitations is also discussed. (- 01:00:00 ) 01:00:00 , are introduced as ways to trade precision for significant speed improvements. (- 01:04:00 ) 01:04:00 precision in PyTorch to leverage tensor cores and achieve a substantial speedup in training without noticeable accuracy degradation. (- 01:08:00 ) 01:08:00 Further Optimization with Torch Compile and Kernel Fusion: The torch.compile function is introduced as a powerful optimization technique that can analyze and fuse multiple operations into single kernels, reducing memory bandwidth bottlenecks and increasing throughput. (- 01:12:00 ) 01:12:00 Identifying Performance Bottlenecks: "Nice" vs. "Ugly" Numbers: This section highlights a less obvious optimization technique: ensuring that key parameters like vocabulary size and batch size are "nice" numbers with many powers of two. This helps align computations with CUDA's block-based execution model and avoids inefficient boundary cases. (- 01:16:00 ) 01:16:00 Adjusting Vocabulary Size for Optimal Performance: This part demonstrates how a slight increase in vocabulary size to the nearest power of two can surprisingly lead to a performance boost due to more efficient CUDA kernel execution. (- 01:20:00 ) 01:20:00 Implementing Gradient Accumulation for Large Batch Sizes: This section introduces gradient accumulation as a technique to simulate very large batch sizes that wouldn't fit in GPU memory by accumulating gradients over multiple micro-batches before performing a weight update. (- 01:24:00 ) 01:24:00 Utilizing Multiple GPUs with Distributed Data Parallelism: This part introduces the concept of distributed data parallelism (DDP) to utilize multiple GPUs for training. It explains how to launch multiple processes with torchrun, assign processes to specific GPUs, and synchronize gradients across processes. (- 01:28:00 ) 01:28:00 model. The data loading script and its functionalities for downloading, tokenizing, and sharding the dataset are briefly explained. (- 01:32:00 ) 01:32:00 Adjusting Training Script for Fine Web EDU: The training script is modified to accommodate the Fine Web EDU dataset, including changes to the data loader, training loop, and hyperparameter settings. The concept of warming up the learning rate and its importance in training large language models is discussed. (- 01:36:00 ) 01:36:00 model on HSWAG are outlined. (- 01:40:00 ) 01:40:00 model. The importance of a validation set in monitoring overfitting is reiterated. (- 01:44:00 ) 01:44:00 training for further performance optimization. (- 01:48:00 ) 00:00:00 intro: Let’s reproduce GPT-2 (124M) 00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint 00:13:47 SECTION 1: implementing the GPT-2 nn.Module 00:28:08 loading the huggingface/GPT-2 parameters 00:31:00 implementing the forward pass to get logits 00:33:31 sampling init, prefix tokens, tokenization 00:37:02 sampling loop 00:41:47 sample, auto-detect the device 00:45:50 let’s train: data batches (B,T) → logits (B,T,C) 00:52:53 cross entropy loss 00:56:42 optimization loop: overfit a single batch 01:02:00 data loader lite 01:06:14 parameter sharing wte and lm_head 01:13:47 model initialization: std 0.02, residual init 01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms 01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms 01:39:38 float16, gradient scalers, bfloat16, 300ms 01:48:15 torch.compile, Python overhead, kernel fusion, 130ms 02:00:18 flash attention, 96ms 02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms 02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping 02:21:06 learning rate scheduler: warmup + cosine decay 02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms 02:34:09 gradient accumulation 02:46:52 distributed data parallel (DDP) 03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU) 03:23:10 validation data split, validation loss, sampling revive 03:28:23 evaluation: HellaSwag, starting the run 03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro 03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA 03:59:39 summary, phew, build-nanogpt github repo 00:00:00 - 🤖 Reproducing GPT-2 124M model- Reproducing the GPT-2 model involves understanding its release structure and model variations. 00:01:09 - 💻 Model Parameters Overview- GPT-2 miniseries comprises models of various sizes, with the 124 million parameter model being a significant variant.- Model parameters dictate its size, layer count, and channel dimensions, affecting downstream task performance. 00:02:06 - 💰 Reproducibility and Cost- Reproducing the GPT-2 124M model is now more accessible and affordable due to advances in hardware and cloud computing.- Achieving comparable model performance can be done in a relatively short time and at a reasonable cost. 00:03:18 - 📚 Reference Material- Access to GPT-2 weights facilitates reproduction, but additional references like the GPT-3 paper provide crucial details for optimization and training settings.- Combining insights from both GPT-2 and GPT-3 papers enhances reproducibility and understanding of the model architecture. 00:05:37 - 🧠 Understanding Model Structure- Exploring the structure of the GPT-2 model involves inspecting token and positional embeddings, as well as layer weights.- The visualization of embeddings and weights reveals insights into the model's learning process and representation. 00:13:13 - 🛠 Implementing Model Architecture- Developing a custom GPT-2 model involves constructing the model architecture, including token and position embeddings, transformer blocks, and classification layers.- Aligning the custom implementation with existing frameworks like Hugging Face Transformers aids in loading pre-trained weights and ensures compatibility. 00:14:21 - 🔍 Model Architecture Differences- GPT-2's architecture includes modifications like layer normalization adjustments and additional layer normalization in the final self-attention block compared to the original Transformer.- Understanding architectural differences is crucial for accurately implementing and reproducing the GPT-2 model. 00:15:15 - 🏗 Defining Model Blocks- Designing the transformer block involves structuring the forward pass, incorporating attention mechanisms, feedforward networks, and residual connections.- Optimizing the block structure for efficient information flow and gradient propagation is essential for model performance. 00:20:20 - 🧠 Understanding the Transformer Architecture- The Transformer architecture relies on attention mechanisms and multi-layer perceptrons (MLPs).- Attention is crucial for communication and individual information processing within Transformer blocks.- Transformers utilize repeated application of "map" and "reduce" operations for information exchange and refinement. 00:21:00 - 🛠 Implementing the MLP Block- The MLP block consists of linear projections sandwiched between G nonlinearity.- The G nonlinearity resembles a smoother version of ReLU and contributes to better gradient flow.- Historical reasons and empirical evidence support the use of the approximate G nonlinearity in GPT-2 reproduction. 00:23:41 - 🧩 Exploring the Attention Operation- Multi-headed attention in Transformers involves parallel computation of attention heads.- The attention operation remains algorithmically equivalent to previous implementations but is more efficient in PyTorch.- Careful variable naming facilitates seamless weight transfer from existing models during reproduction. 00:40:21 - 🤖 Replicating GPT-2 Model Initialization- Replicating the GPT-2 model initialization process.- Transitioning from pre-trained weights to initializing from random numbers.- Exploring the straightforward process of using a random model in PyTorch. 00:43:30 - 🔍 Detecting and Utilizing Device in PyTorch- Automatically detecting and utilizing available devices in PyTorch.- Strategies for choosing the highest compute-capable device.- Facilitating code compatibility across different hardware configurations. 00:46:11 - 📄 Preparing and Tokenizing Dataset- Introduction to the Tiny Shakespeare dataset for training.- Obtaining and processing the dataset for tokenization.- Initial exploration and preprocessing steps for training data. 00:52:05 -- 🛠 Implementing Data Loader and Loss Calculation- Building a data loader to feed token sequences into the Transformer model.- Setting up the forward pass to calculate the loss function.- Establishing a structured approach for loss calculation and gradient updates. 00:56:53 - 🧮 Optimizing Model Parameters with AdamW- Implementing optimization using the AdamW optimizer.- Understanding the role and benefits of AdamW compared to SGD.- Executing gradient updates and monitoring loss during the optimization process. 01:00:17 - 🧠 Introduction to Model Optimization- Optimizing model training requires careful handling of tensors and device placement.- Overfitting a single batch is an initial step in understanding model behavior.- Transitioning from overfitting a single batch to optimizing with multiple batches requires implementing a data loader. 01:02:03 - 📊 Implementation of a Simple Data Loader- The data loader reads text files and tokenizes them for model input.- It divides the data into batches, ensuring smooth iteration over the dataset.- Basic functionality covers chunking data and managing batch transitions. 01:06:24 - 🐛 Fixing a Weight Initialization Bug- Identifies a bug in weight initialization concerning weight tying in GPT-2 training.- Explains the significance of weight tying in reducing parameters and improving performance.- Implements a fix by redirecting pointers to the same tensor, saving parameters and optimizing performance. 01:13:45 - 🎚 Fine-tuning Model Initialization- Discusses the importance of model weight initialization in training stability and performance.- Mimics GPT-2 initialization scheme based on observed patterns in released source code.- Introduces a scaling factor for residual layers' weights initialization to control activation growth in the network. 01:20:27 - 🛠 Implementing GPT-2 Initialization- Implementing scaling down the standard deviation for proper initialization.- Clarification on the two times number of layers in the Transformer.- Setting seeds for reproducibility and initializing GPT-2 model. 01:23:07 - 💻 Optimizing Hardware Utilization- Assessing available hardware resources, including GPUs.- Understanding the importance of memory bandwidth in GPU utilization.- Exploring precision options (float32, tf32, bfloat16) for performance optimization. 01:28:12 - 🔄 Leveraging Tensor Cores for Acceleration- Explanation of tensor cores and their role in matrix multiplication.- Introduction to tf32 precision and its performance benefits.- Comparison of tf32 and float32 performance improvements. 01:37:04 - ⚙ Implementing tf32 Precision in PyTorch- Enabling tf32 precision in PyTorch with a single line of code.- Observing throughput improvements with tf32 precision.- Understanding the trade-offs and limitations of tf32 precision. 01:40:30 - 📊 B Float16 vs. FP16 Precision Reduction- Understanding B Float16 precision reduction compared to FP16.- B Float16 maintains the same exponent range but truncates the mantissa, resulting in reduced precision within the range.- Unlike FP16, B Float16 does not alter the range of representable numbers, simplifying training processes by eliminating the need for gradient scalers. 01:42:24 - 🧮 Implementing Mixed Precision in PyTorch- Utilizing PyTorch's torch.AutoCast for mixed precision training.- Guidance on using torch.AutoCast to surround the forward pass and loss calculation in the model.- Highlighting the minimal code changes required to implement B Float16 training in PyTorch. 01:48:29 - ⚡ Torch.Compile for Model Optimization- Introduction to torch.Compile as a compiler for neural networks in PyTorch.- Explaining the reduction of Python overhead and GPU read-writes for faster computation.- Demonstrating significant speed improvements with torch.Compile, achieving about 2.3x faster performance with a single line of code. 02:00:26 - 🧠 Flash Attention Optimization- Flash attention is a kernel fusion algorithm that significantly speeds up attention mechanisms.- Achieves faster computation by avoiding materializing large attention matrices.- Utilizes an online softmax trick to incrementally evaluate softmax without storing all inputs. 02:06:54 - 🧮 Optimization with Nice Numbers- Identifies "nice" numbers (powers of two) as optimal for computations in CUDA.- Adjusts vocabulary size to a nice number to improve computation efficiency.- Padding inputs to align with block sizes in CUDA can lead to significant performance gains. 02:15:18 - 🔍 Hyperparameter Tuning and Algorithmic Improvements- Discusses the importance of hyperparameter tuning based on the GPT-3 paper.- Implements gradient norm clipping to prevent model instability during optimization.- Monitoring the gradient norm helps detect training instabilities and adjust optimization strategies. 02:19:44 - 🎓 Implementing Learning Rate Scheduler and Weight Decay- Understanding the details of the learning rate scheduler and weight decay implementation:- Learning rate scheduler: Cosine decay with warm-up period and decay to 10% over a specified horizon.- Weight decay: Used for regularization, typically applied to embedding and weight matrices. 02:26:26 - 📊 Batch Size Increase and Data Sampling Techniques- Explanation on gradual batch size increase and data sampling methods:- Gradual batch size increase: Linear ramp-up from small to large batch sizes, aiming for system speed improvement.- Data sampling without replacement: Exhausting a pool of data without reusing sequences until an epoch boundary is reached. 02:29:01 - 🧮 Weight Decay Implementation and Optimizer Configuration- Details on weight decay implementation and optimizer configuration:- Weight decay: Applied for regularization, particularly to embeddings and weight matrices.- Optimizer configuration: Adjusting parameters for optimal training performance, including weight decay settings. 02:37:07 - 🔄 Gradient Accumulation for Simulating Large Batch Sizes- Implementation of gradient accumulation technique to simulate large batch sizes:- Total batch size setting: Defines the desired batch size, which may exceed GPU capacity.- Micro batch size and gradient accumulation: Processing multiple micro-batches and accumulating gradients before updating the model. 02:39:12 - 🧠 Understanding Gradient Accumulation- Explains the concept of gradient accumulation.- Demonstrates the difference between traditional batch processing and gradient accumulation.- Emphasizes the importance of normalizing gradients to ensure consistency. 02:47:00 - 🔧 Implementing Distributed Data Parallelism- Introduces the concept of distributed data parallelism for utilizing multiple GPUs.- Explains the difference between legacy data parallelism and distributed data parallelism.- Describes how distributed data parallelism works and its benefits in training neural networks. 02:57:01 - 🔄 Adapting Data Loading for Multi-Process Training- Adjusts data loading process to accommodate multiple processes.- Demonstrates how to assign different chunks of data to each process.- Ensures that each process works on a unique part of the dataset to maximize efficiency. 02:59:17 - 🧩 Model Construction and Distributed Data Parallel (DDP)- Explanation of constructing a model for distributed training.- Wrapping the model into a DistributedDataParallel (DDP) container.- Understanding the behavior of DDP in forward and backward passes. 03:02:15 - 🔄 Synchronization of Gradients in DDP- Discusses the synchronization of gradients in the DistributedDataParallel (DDP) setting.- Explanation of optimizing gradient synchronization to improve efficiency.- Implementation details for synchronizing gradients in DDP. 03:05:22 - 📉 Loss Averaging and Evaluation in DDP- Addressing the issue of loss averaging in the DDP setting.- Modifying code to compute and print the average loss across all processes.- Ensuring proper scaling of the number of tokens processed in the evaluation phase. 03:10:23 - 📚 Training Data Comparison: GPT-2 vs. GPT-3- Comparison of training datasets used in GPT-2 and GPT-3.- Description of web text and Common Crawl datasets utilized.- Introduction of alternative datasets like Red Pajamas, C4, Fine Web, and Fine Web Edu. 03:14:15 - 📦 Preprocessing and Training Setup for Fine Web Edu- Overview of the preprocessing steps for the Fine Web Edu dataset.- Description of tokenization process and data shard creation.- Configuration adjustments in the data loader for using the Fine Web Edu dataset. 03:18:42 - 🧩 Script adjustments for GPT-3 replication- Adjusted data loader for processing multiple shards.- Set token processing rate and warm-up steps to match GPT-3 parameters.- Increased batch size optimization for faster training. 03:21:29 - 📊 Implementing validation evaluation- Added validation evaluation logic to the training loop.- Introduced periodic validation loss calculation.- Prepared for model comparison with GPT-2 124M. 03:26:09 - 🔄 Reorganizing sampling code- Moved sampling code closer to the main training loop.- Implemented a separate RNG for sampling to avoid impacting training RNG.- Addressed performance slowdown due to disabled Torch compile. 03:28:26 - 📈 Introducing H-SWAG evaluation- Described H-SWAG evaluation methodology and dataset.- Highlighted its role as a smooth evaluation metric.- Discussed implementation details for incorporating H-SWAG into the training script. 03:38:22 - 🔧 Adjustments to Training Script and Logging- Changes made to the training script to enable periodic evaluation and tracking of model performance over time.- Disabling torch compile due to issues with evaluation and sampling code.- Creation of a log directory to record training and validation losses, as well as H swag accuracies. 03:40:01 - 📊 Evaluation of H Swag and Model Sampling- Introduction of code for evaluating H swag periodically during training.- Utilization of GPU collaboration for H swag evaluation.- Sampling from the model every 250th iteration for monitoring model progress. 03:43:06 - 📈 Training Progress Visualization- Visualization of training progress using Matplotlib.- Analysis of loss curves and model performance.- Comparison of model performance against GPT-2 and GPT-3 accuracy metrics. 03:46:23 - 🧠 Reflections on Training Results and Data Quality- Discussion on the implications of achieving GPT-3 level accuracy with fewer tokens.- Consideration of factors influencing model performance, such as data distribution and dataset quality.- Reflections on potential improvements in data preprocessing and model hyperparameters. 03:49:05 - ⚙ Optimization Techniques and Training Efficiency- Examination of optimization issues and periodicity in data loading.- Discussion on the impact of learning rate adjustments on training efficiency.- Consideration of techniques to improve data shuffling and reduce data dependency. 03:52:03 - 🛠 Model Fine-Tuning and Future Directions- Overview of fine-tuning process for conversational AI applications.- Introduction of model checkpointing for resuming optimization and model evaluation.- Discussion on alternative evaluation methods and comparison with GPT-2 and GPT-3. 00:00:00 * *** Exploring the Target:* The video starts by loading the pre-trained GPT-2 (124M) model from Hugging Face Transformers and examining its weights and architecture. 00:13:47 * *** Implementing the GPT-2 nn.Module:* A custom GPT-2 class is built in PyTorch, mirroring the Hugging Face architecture and loading the pre-trained weights for verification. 00:31:00 * *** Forward Pass and Sampling:* The forward pass is implemented to calculate logits, and a sampling loop is added to generate text from the model. 01:22:18 * *** Understanding Hardware:* The video emphasizes understanding GPU capabilities, particularly tensor cores and memory bandwidth. 01:28:14 * *** Mixed Precision (TF32):* Enabling TF32 precision for matrix multiplications provides a free 3x speedup with minimal accuracy loss. 01:39:38 * *** Mixed Precision (BFloat16):* Switching to BFloat16 for activations further improves speed, requiring minimal code changes thanks to PyTorch AutoCast. 01:48:15 * *** Torch.Compile:* Compiling the model with torch.compile significantly reduces Python overhead and optimizes kernel fusion, resulting in a 2.3x speedup. 02:00:18 * *** Flash Attention:* Replacing the default attention implementation with Flash Attention, a specialized kernel fusion algorithm, yields another 27% speedup. 02:06:54 * *** Nice vs. Ugly Numbers:* Optimizing vocabulary size to a power of two (50304) for better kernel utilization surprisingly provides a 4% speedup. 02:14:55 * *** Hyperparameters and AdamW:* The video adopts hyperparameters from the GPT-3 paper, including AdamW optimizer settings and gradient clipping. 02:21:06 * *** Learning Rate Scheduler:* A cosine decay learning rate schedule with warmup is implemented, following the GPT-3 paper. 02:26:21 * *** Batch Size, Weight Decay, FusedAdamW:* The video discusses batch size scheduling (which is ultimately skipped), implements weight decay for regularization, and utilizes the fused implementation of AdamW for further speed improvements. 02:34:09 * *** Gradient Accumulation:* Gradient accumulation is implemented to simulate larger batch sizes (0.5 million tokens) on limited GPU memory. 02:46:52 * *** Distributed Data Parallel (DDP):* The training is parallelized across 8 GPUs using PyTorch DDP, achieving a throughput of 1.5 million tokens per second. 03:10:21 * *** Dataset Selection:* The video discusses various datasets used for training large language models, ultimately choosing the FineWeb EDU dataset (10 billion token sample). 03:23:10 * *** Validation Split:* A validation split is introduced to monitor overfitting and compare performance to the pre-trained GPT-2 model. 03:28:23 * *** HellaSwag Evaluation:* The HellaSwag benchmark is implemented to evaluate the model's common sense reasoning abilities. 03:43:05 * *** Results:* After training for one epoch (10 billion tokens), the model surpasses the GPT-2 (124M) performance on HellaSwag, achieving comparable results with 10x fewer training tokens. 03:48:41 * *** Overnight Run:* Training for four epochs (40 billion tokens) further improves HellaSwag accuracy, approaching the GPT-3 (124M) performance. 03:56:21 * *** Shoutout to llm.c:* The video showcases "llm.c," a faster C/CUDA implementation of GPT-2/3 training. 03:59:39 * *** Summary:* A brief summary of the achievements and remaining challenges. 00:08:49 has the freedom to learn the position embeddings (the original transformer paper hardcoded the positional embeddings) 00:18:14 You want a direct residual connection from the target to the input embeddings, skipping layer normalization (I need to understand what layer normalization is) 00:22:11 Activation function GELU is an approximation 00:55:27 Calculating the estimated loss at initialization 00:57:00 Pytorch library had bugs that the canonical version (e.g. Adam) is the buggy version (fixed in AdamW) 01:01:01 Attempting to overfit on a single example 01:12:53 % of the parameters) 01:23:50 import code; code.interact(local=locals()) 01:51:59 "dispatch the kernel"??? 02:01:31 FlashAttention -> more flops does not mean slower 02:10:09 Add more tokens the model actually trains faster 01:49:00 : Summary at 01:14:00 : Summary at 00:42:00 : My quick summary at ! A 2000-line GPT-2 implementation in Huggingface has been condensed to almost 100 lines. The weights from HF GPT-2 were replicated in this new version, using the same sampling parameters, seed, and generating identical output. A notable improvement is the restructuring of the implementation, where all heads are now integrated within a single matrix, applying some neat matrix transposes while maintaining parallelism and enhancing comprehension. This is far easier to understand compared to many other complicated multihead implementations I've seen. The next step involves training this model from the ground up. 00:20:10 the comparison between attention and mlp is impressive 00:20:00 Its funny how his description of attention as reduce-map description at can be thought of as map-reduce :) 01:55:55 hours of video on this topic"Me: Please sign me up :) 01:20:21 Hi Andrej should we skip the pos embedding initialization with std 0.01 like in the original code and stick to the 0.02 ? 01:11:13 source code (at ) but I can't seem to find it in your PyTorch implementation. 00:37:46 (time ) 00:38:10 why do we only keep the last column of the logits? 00:30:10 This series is amazing, but I have a bit of confusion. At the timestamp, you mentioned that the weights are transposed and referenced something about TensorFlow. However, I think in PyTorch, the weights for a linear layer are initialized as torch.empty(out_features, in_features)so is this the case u needed to transpose the weightsand Furthermore, the weights you are transposing all belong to linear layers, yet for the last lm_head layer, which is also a linear layer, you are not transposing that weight.Am I mistaken here, or is there something else going on? 01:16:03 shouldn't Embedding std be set to 0.01 ? 01:36:27 @ Should the tokens/second throughput be x2 given we use both X and y (targets) for training? Or are we just looking at the batch size here? Also would using x.numel() or y.numel() be equivalent? 01:02:16 I see at for the batch processing, you are marching along by an index of `B * T`. Instead, what would be the implications of changing this to a sliding window (+1 indexing) such that we get overlapping samples? I realise this would create `len(self.tokens) - block_size` samples leading to a far greater number of batches per epoch, is this the only aspect? 01:55:59 yes Andrej we need that 2 hour neural net Hardware specific video 🗣🗣🗣 00:56:42 Question regarding overfitting a single batch . 01:11:32 the weights sharing the dimensions of wte and lm head are different, is it okay? 00:56:10 Fun Fact: -ln(1/50257) = 10.82 but simply ln(50257) also gives the same answer. 03:10:00 : Summary at 00:42:00 : My quick summary at ! A 2000-line GPT-2 implementation in Huggingface has been condensed to almost 100 lines. The weights from HF GPT-2 were replicated in this new version, using the same sampling parameters, seed, and generating identical output. A notable improvement is the restructuring of the implementation, where all heads are now integrated within a single matrix, applying some neat matrix transposes while maintaining parallelism and enhancing comprehension. This is far easier to understand compared to many other complicated multihead implementations I've seen. The next step involves training this model from the ground up. 02:47:00 : Summary at 00:19:00 Found this video first, then at about when you started talking about residuals and micrograd, went back to your zero-to-hero series and watched everything as a prerequisite. now i understand how residuals helps in stabilizing the training. the gradient distribution into branches analogy really changed the perspective for me. this video should be kept safe in a time capsule 01:14:00 : Summary at 00:03:43 @ now... so far so good... Let's reproduce GPT-2 (124M) We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training run following the GPT-2 and GPT-3 paper and their hyperparameters, then we hit run, and come back the next morning to see our results, and enjoy some amusing model generations. Keep in mind that in some places this video builds on the knowledge from earlier videos in the Zero to Hero Playlist (see my channel). You could also see this video as building my nanoGPT repo, which by the end is about 90% similar. Links: - build-nanogpt GitHub repo, with all the changes in this video as individual commits: https://github.com/karpathy/build-nanogpt - nanoGPT repo: https://github.com/karpathy/nanoGPT - llm.c repo: https://github.com/karpathy/llm.c - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - our Discord channel: https://discord.gg/3zy8kqD9Cp Supplementary links: - Attention is All You Need paper: https://arxiv.org/abs/1706.03762 - OpenAI GPT-3 paper: https://arxiv.org/abs/2005.14165 - OpenAI GPT-2 paper: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf- The GPU I'm training the model on is from Lambda GPU Cloud, I think the best and easiest way to spin up an on-demand GPU instance in the cloud that you can ssh to: https://lambdalabs.com Chapters: 00:00:00 intro: Let’s reproduce GPT-2 (124M) 00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint 00:13:47 SECTION 1: implementing the GPT-2 nn.Module 00:28:08 loading the huggingface/GPT-2 parameters 00:31:00 implementing the forward pass to get logits 00:33:31 sampling init, prefix tokens, tokenization 00:37:02 sampling loop 00:41:47 sample, auto-detect the device 00:45:50 let’s train: data batches (B,T) → logits (B,T,C) 00:52:53 cross entropy loss 00:56:42 optimization loop: overfit a single batch 01:02:00 data loader lite 01:06:14 parameter sharing wte and lm_head 01:13:47 model initialization: std 0.02, residual init 01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms 01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms 01:39:38 float16, gradient scalers, bfloat16, 300ms 01:48:15 torch.compile, Python overhead, kernel fusion, 130ms 02:00:18 flash attention, 96ms 02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms 02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping 02:21:06 learning rate scheduler: warmup + cosine decay 02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms 02:34:09 gradient accumulation 02:46:52 distributed data parallel (DDP) 03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU) 03:23:10 validation data split, validation loss, sampling revive 03:28:23 evaluation: HellaSwag, starting the run 03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro 03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA 03:59:39 summary, phew, build-nanogpt github repo Corrections: I will post all errata and followups to the build-nanogpt GitHub repo (link above) SuperThanks: I experimentally enabled them on my channel yesterday. Totally optional and only use if rich. All revenue goes to to supporting my work in AI + Education. #neural network #GPT #karpathy #LLM #language model #large language model #ChatGPT #NVIDIA #GPU #PyTorch #Python #deep learning #education https://favtu.be/timelines/v/zduSFxRajkE Wed, 21 Feb 24 02:11:35 +0900 00:00:00 *🤖 什么是分词(tokenization)?*- 分词是将文本转换为标记序列的过程。- 在大型语言模型中，分词是将文本转换为标记序列以供模型处理的关键步骤。- 分词的质量和方法直接影响着模型的性能和行为。 00:02:43 *🔍 GPT2 使用的字节对编码算法*- 字节对编码算法是一种常用的分词方法，用于构建大型语言模型的标记词汇表。- GPT2 中的分词器使用了字节对编码算法来构建词汇表，其中每个 token 可以是多个字符的组合。- 字节对编码算法能够灵活地处理各种语言和特殊字符，从而提高模型的适用性和性能。 00:05:59 *🌐 语言模型中的分词问题*- 分词对于语言模型的性能和行为至关重要，但也会带来一些问题和挑战。- 不同语言的分词效果可能不同，特别是非英文语言可能受到数据不平衡的影响。- 分词方法的设计和实现对模型的效率和表现有重要影响，需要综合考虑多方面因素进行优化。 00:19:25 *🧮 字符编码的选择与比较*- UTF-8 在互联网上被广泛采用，因为它是唯一向后兼容 ASCII 编码的字符编码。- UTF-8 相对于其他编码来说更加节省空间，因为它能够更有效地编码文本信息。 00:22:01 *🧩 字节对编码算法简介*- 字节对编码算法通过迭代地识别并替换最常出现的字节对来压缩文本序列。- 该算法能够将原始字节序列压缩到一个较小的固定大小的词汇表中，并实现对任意序列的编码和解码。 00:27:10 *🖥️ 字节对编码算法的实现*- 通过 Python 实现字节对编码算法，包括识别最常见字节对、替换、创建新词汇表等步骤。- 使用迭代的方式对文本序列进行多次合并，直到达到所需的词汇表大小。 00:38:01 *📊 Tokenizer 训练总结*- Tokenizer 的训练是完全独立于大语言模型的。- Tokenizer 有自己的训练集，使用 BPE 算法进行训练，构建词汇表。- Tokenizer 的训练一次性完成，之后可用于编码和解码。 00:39:38 *🔤 Tokenizer 编码和解码*- Tokenizer 是原始文本和 token 序列之间的翻译层。- 可以将原始文本编码成 token 序列，也可以将 token 序列解码成原始文本。- 大语言模型的训练数据通常会预处理为 token 序列进行训练，而不是使用原始文本。 00:42:41 *🛠️ 实现编码和解码功能*- 实现编码功能时，需要将文本编码为 token 序列，并按照 merges 字典中的顺序执行合并。- 实现解码功能时，需要将 token 序列解码为原始文本，并根据 merges 字典执行解码过程。- 在实现解码功能时，需要注意处理不符合 UTF-8 格式的情况，常见的做法是使用错误处理参数来避免错误。 00:57:24 *🧩 GPT2论文中的Tokenizer*- GPT2论文解释了其使用的Tokenizer，主要采用字节对编码算法（Byte Pair Encoding, BPE）。- 论文指出对常见词汇进行简单的BPE算法合并会导致语义混乱，因此提出了手动制定合并规则的方法。 00:59:29 *🛠️ GPT2的Tokenizer实现细节*- GPT2的Tokenizer实现包括了一个复杂的正则表达式模式，用于规定哪些部分的文本不应该被合并。- 使用了Python的reex包进行更强大的正则表达式匹配。 01:11:08 *🧰 TikTok Tokenizer 库介绍*- OpenAI发布了TikTok Tokenizer库，用于GPT4的分词工作。- 与GPT2不同，GPT4的Tokenizer将空格合并为一个标记，这在GPT2中是不同的。 01:13:12 *🔍 GPT4的Tokenizer变化*- GPT4的Tokenizer与GPT2相比进行了一些修改，包括对正则表达式模式的改变以及对空格和数字的处理方式。- 正则表达式模式中增加了对大小写不敏感的匹配，并限制了数字合并的长度，以避免生成过长的标记。 01:16:40 *🤖 tokenizer算法原理*- 开发Tokenizer的算法与OpenAI的实现基本相同。- 理解了算法原理后，能够构建、训练和使用Tokenizer。- OpenAI在实现中添加了一些不太重要的细节，但基本原理保持一致。 01:18:32 *🛠️ 特殊token的用途和处理*- 特殊token用于在数据中标记特殊结构或分隔不同部分。- 特殊token的添加需要对模型进行一定的修改和调整，包括修改嵌入矩阵和最终层的投影。- 这种操作在Fine-tuning等任务中特别常见，例如从基础语言模型转换为聊天模型。 01:28:55 *🧩 SentencePiece与Tokenizer的比较*- SentencePiece是另一种常用的标记化库，支持训练和推理。- 它使用了不同的标记化方法，直接在代码点上执行BPE，对于稀有的代码点使用了fallback机制。- SentencePiece拥有大量的配置选项，但在NLP模型中通常需要调整以适应特定任务。 01:34:08 *🧩 分析 SentencePiece 的工作原理和参数设置*- SentencePiece 的工作原理和参数设置，- SentencePiece 将文本文件视为字节流，而不是句子，通过一系列规则进行分词和编码。- 训练时需要指定特殊标记，如 UNK、BOS、EOS 和 PAD，并且必须存在 UNK 标记。- 通过示例展示了 SentencePiece 的词汇表和编码过程，以及如何处理未知字符和字节回退。 01:43:31 *🔍 理解 Transformer 模型中的 Vocabulary Size*- Transformer 模型中的 Vocabulary Size，- Voab size 在 Transformer 模型中影响 token embedding table 的大小和 LM head 层的参数数量。- Voab size 的增加会导致模型计算量增加、参数稀疏性增加和序列长度减少等问题。- 调整 Voab size 是一项经验性超参数调整，通常在高万到十万级别，根据应用场景和计算资源进行选择。 01:48:11 *🔄 扩展 Vocabulary Size 和应用于多模态数据*- 扩展 Vocabulary Size 和应用于多模态数据，- 可以通过简单的模型修改来扩展 Vocabulary Size，并介绍了冻结模型和训练新参数的方法。- 对于多模态数据，可以将其他领域的数据转换成 token，并使用相同的 Transformer 模型进行处理。- 学术界和工业界都在探索如何将 Transformer 应用于处理多模态数据，并提出了各种创新的方法和技术。 01:51:56 *🧠 Tokenization 对于模型执行特定任务的影响*- Tokenization 对模型执行特定任务的影响，- 长 token 可能导致模型在处理某些任务时表现不佳，如拼写检查或字符串反转。- 模型在处理非英语语言和简单算术时也受到 tokenization 的影响，导致性能下降。 01:57:25 *🛑 处理特殊字符串时的模型异常行为*- 处理特殊字符串时的模型异常行为，- 模型可能会在处理特殊字符串时出现意外行为，如停止生成输出或输出无意义结果。- 对特殊字符的处理可能存在漏洞，可能导致模型受到攻击。 01:59:00 *⚠️ 尾随空白字符对模型表现的影响*- 尾随空白字符对模型表现的影响，- 在输入中存在尾随空白字符时，模型的性能可能会受到影响，导致输出不稳定或不准确。- 尾随空白字符可能导致模型处理数据分布不一致，从而影响结果的一致性。 02:04:59 *💥 Tokenization 数据集与模型训练数据集不一致导致的异常行为*- Tokenization 数据集与模型训练数据集不一致导致的异常行为，- 当 tokenization 数据集中包含的特殊字符串在模型训练数据集中未出现时，模型在处理这些字符串时可能表现异常。- 未训练的 token 在模型推理阶段可能导致未定义的行为，从而产生奇怪的输出或行为。 02:09:21 *🌐 不同格式和语言对 GPT Tokenizer 的影响*- 不同格式和语言对 GPT Tokenizer 的影响，- 不同的数据格式和语言可能会影响 GPT Tokenizer 的性能和效率。- 例如，Json 格式可能与 GPT Tokenizer 不太兼容，导致性能下降。 02:09:33 *💰 数据格式对 token 化效率的影响*- 数据格式对 token 化效率的影响，- Yaml 格式相比于 Json 格式在 token 化时更加高效，减少了 token 的数量。- 在计算 token 成本和处理结构化数据时，选择更高效的编码格式可以节省成本和提高效率。 02:10:30 *🚧 重视 tokenization 的重要性与挑战*- 重视 tokenization 的重要性与挑战，- Tokenization 阶段可能存在安全问题和 AI 安全问题，需要引起重视。- 虽然 tokenization 阶段令人烦恼，但不应忽视其重要性，有待进一步的研究和改进。 02:11:11 *🛠️ 应用建议与推荐的工具*- 应用建议与推荐的工具，- 对于应用程序，如果可以重用 GPT 4 tokens 和词汇表，则建议使用 Tik tok 作为推理的有效库。- 对于训练自己的词汇表，建议使用基于字节级 BPE 的方法，如 Tik tok 和 OpenAI 所使用的字节级 BPE。 00:00:00 intro: Tokenization, GPT-2 paper, tokenization-related issues 00:05:50 tokenization by example in a Web UI (tiktokenizer) 00:14:56 strings in Python, Unicode code points 00:18:15 Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32 00:22:47 daydreaming: deleting tokenization 00:23:50 Byte Pair Encoding (BPE) algorithm walkthrough 00:27:02 starting the implementation 00:28:35 counting consecutive pairs, finding most common pair 00:30:36 merging the most common pair 00:34:58 training the tokenizer: adding the while loop, compression ratio 00:39:20 tokenizer/LLM diagram: it is a completely separate stage 00:42:47 decoding tokens to strings 00:48:21 encoding strings to tokens 00:57:36 regex patterns to force splits across categories 01:11:38 tiktoken library intro, differences between GPT-2/GPT-4 regex 01:14:59 GPT-2 encoder.py released by OpenAI walkthrough 01:18:26 special tokens, tiktoken handling of, GPT-2/GPT-4 differences 01:25:28 minbpe exercise time! write your own GPT-4 tokenizer 01:28:42 sentencepiece library intro, used to train Llama 2 vocabulary 01:43:27 how to set vocabulary set? revisiting gpt.py transformer 01:48:11 training new tokens, example of prompt compression 01:49:58 multimodal [image, video, audio] tokenization with vector quantization 01:51:41 revisiting and explaining the quirks of LLM tokenization 02:10:20 final recommendations 02:12:50 ??? :) 00:00:00 - 🧩 Tokenization process overview- Tokenization is crucial for working with large language models- Tokenization converts text into tokens for language model processing 00:03:13 - 🍬 Bite-pair encoding for tokenization- Bite-pair encoding is used in state-of-the-art language models- Tokenization generates vocabularies for language model input- Tokens are fundamental units in large language models 00:09:17 - 🌏 Multilingual tokenization challenges- Non-English languages may have different tokenization challenges- Tokenizers have to handle varying lengths for different languages 00:14:47 - 🐍 Tokenization impact on Python coding- Tokenization affects the handling of code in language models- Tokenizer design influences the model's performance for specific languages 00:18:13 - 🔠 Unicode encodings for text processing- Unicode encodings like UTF-8 are essential for processing text- Different encodings have varying efficiencies and use cases- UTF-8 encoding is preferred for its compatibility and efficiency 00:22:26 - 🧠 Byte Pair Encoding Algorithm Overview- Byte Pair Encoding (BPE) algorithm compresses sequences by finding and merging the most frequent pairs of tokens iteratively. 00:27:10 - 📊 Implementing Byte Pair Encoding Algorithm in Python- Encoding text into UTF-8 tokens and converting them to integers for manipulation.- Identifying the most common pair of tokens and replacing them with new tokens using Python functions. 00:35:00 - 🧭 Training and Usage of the Tokenizer- Setting the vocabulary size and performing a fixed number of merges to create the tokenizer.- Discussing the role of the tokenizer as a separate preprocessing stage from the language model. 00:41:30 - 🌐 Tokenizer Training Considerations- Highlighting the importance of diverse training sets for tokenizers encompassing various languages and data types.- Explaining the impact of different data representations on the token sequence density and model performance. 00:44:15 - 🧮 Tokenization of IDS to create tokens- Getting tokens by iterating over IDS and looking up bytes in vocab- Concatenating bytes to create tokens- Decoding bytes back to strings using UTF-8 00:48:33 - 🧬 Implementing encoding of string into tokens- Encoding text into UTF-8 to get raw bytes- Performing merges according to lookup dictionary- Identifying pairs for merging and performing merges 00:55:16 - 📝 Perfecting the encoding function and testing- Addressing the special case of single character or empty string- Testing encoding and decoding to ensure consistency- Validating the implemented function with training and validation data 01:06:31 - 🧩 Tokenization rules and inconsistencies- Tokenization rules for apostrophes are inconsistent in uppercase and lowercase letters.- Matching punctuation characters is essential to separate them from letters or numbers.- Understanding whitespace handling in tokenization is crucial, including negative look-ahead assertions. 01:11:08 - 🤖 GPT Tokenizer and GPT-3.5 Turbo Scheme- The GPT Tokenizer for GPT-4 uses different merging rules compared to GPT-2.- The GPT-3.5 Turbo Scheme introduces new special tokens for conversation tracking.- Special tokens handling requires additional model adjustments like embedding matrix extension. 01:18:32 - 🏷 Special tokens and fine-tuning- Special tokens, like "End of Text," delimit documents in the GPT training set.- Adding special tokens requires model adjustments like extending embedding matrices.- Special tokens are crucial for tasks like fine-tuning a base model into a chatbot model. 01:28:41 - 🧠 Tokenization using Sentence Piece- Sentence Piece is used widely in language models for training and inference efficiency. 01:31:23 - 📜 Configuration and Training with Sentence Piece- Sentence Piece has numerous configuration options available with historical baggage.- The training process includes defining input/output files, selecting algorithms, and preprocessing rules. 01:43:31 - 🤖 Vocab Size and Model Architecture- Vocabulary size impacts model training and computational complexity.- Larger vocab sizes can lead to underfitting of rare tokens and compression of information. 01:47:02 - 🛠 Extending Vocab Size in Pre-Trained Models- Pre-trained models can have vocab sizes extended by adding new tokens.- The process involves resizing embeddings and adjusting linear layers for new token probabilities. 01:48:54 - 🧠 Fine-tuning Techniques- Training new tokens with distillation technique- Optimizing over new tokens without changing model architecture- Efficiency in fine-tuning by training only token embeddings 01:50:05 - 🤖 Processing Multimodal Inputs- Adapting Transformers to process various modalities like images, videos, and audio- Tokenizing input domains for different modalities- Using the same Transformer architecture for different input types 01:51:42 - 📏 Tokenization Algorithm Analysis- Limitations of language models in spelling and simple arithmetic tasks due to tokenization- Differences in tokenization of English and non-English languages- Impact of tokenization on model performance in handling Python coding. 02:09:21 - 🧮 Tokenization efficiency considerations- Different data formats and representations can impact the efficiency of tokenization. 02:10:16 - 🔑 Importance of measuring token efficiencies- Tokenization density is crucial for cost-effective processing of data.- Spending time on measuring token efficiencies across formats is essential. 02:10:57 - 🛠 Recommendations for tokenization application- Reuse GPT-4 tokens and vocabulary for efficient application.- Consider using libraries like Tech tokenizer for inference. 00:00:00 - 🧩 Tokenization process overview- Tokenization is crucial for working with large language models- Tokenization converts text into tokens for language model processing 00:03:13 - 🍬 Bite-pair encoding for tokenization- Bite-pair encoding is used in state-of-the-art language models- Tokenization generates vocabularies for language model input- Tokens are fundamental units in large language models 00:09:17 - 🌏 Multilingual tokenization challenges- Non-English languages may have different tokenization challenges- Tokenizers have to handle varying lengths for different languages 00:14:47 - 🐍 Tokenization impact on Python coding- Tokenization affects the handling of code in language models- Tokenizer design influences the model's performance for specific languages 00:18:13 - 🔠 Unicode encodings for text processing- Unicode encodings like UTF-8 are essential for processing text- Different encodings have varying efficiencies and use cases- UTF-8 encoding is preferred for its compatibility and efficiency 00:22:26 - 🧠 Byte Pair Encoding Algorithm Overview- Byte Pair Encoding (BPE) algorithm compresses sequences by finding and merging the most frequent pairs of tokens iteratively. 00:27:10 - 📊 Implementing Byte Pair Encoding Algorithm in Python- Encoding text into UTF-8 tokens and converting them to integers for manipulation.- Identifying the most common pair of tokens and replacing them with new tokens using Python functions. 00:35:00 - 🧭 Training and Usage of the Tokenizer- Setting the vocabulary size and performing a fixed number of merges to create the tokenizer.- Discussing the role of the tokenizer as a separate preprocessing stage from the language model. 00:41:30 - 🌐 Tokenizer Training Considerations- Highlighting the importance of diverse training sets for tokenizers encompassing various languages and data types.- Explaining the impact of different data representations on the token sequence density and model performance. 00:44:15 - 🧮 Tokenization of IDS to create tokens- Getting tokens by iterating over IDS and looking up bytes in vocab- Concatenating bytes to create tokens- Decoding bytes back to strings using UTF-8 00:48:33 - 🧬 Implementing encoding of string into tokens- Encoding text into UTF-8 to get raw bytes- Performing merges according to lookup dictionary- Identifying pairs for merging and performing merges 00:55:16 - 📝 Perfecting the encoding function and testing- Addressing the special case of single character or empty string- Testing encoding and decoding to ensure consistency- Validating the implemented function with training and validation data 01:06:31 - 🧩 Tokenization rules and inconsistencies- Tokenization rules for apostrophes are inconsistent in uppercase and lowercase letters.- Matching punctuation characters is essential to separate them from letters or numbers.- Understanding whitespace handling in tokenization is crucial, including negative look-ahead assertions. 01:11:08 - 🤖 GPT Tokenizer and GPT-3.5 Turbo Scheme- The GPT Tokenizer for GPT-4 uses different merging rules compared to GPT-2.- The GPT-3.5 Turbo Scheme introduces new special tokens for conversation tracking.- Special tokens handling requires additional model adjustments like embedding matrix extension. 01:18:32 - 🏷 Special tokens and fine-tuning- Special tokens, like "End of Text," delimit documents in the GPT training set.- Adding special tokens requires model adjustments like extending embedding matrices.- Special tokens are crucial for tasks like fine-tuning a base model into a chatbot model. 01:28:41 - 🧠 Tokenization using Sentence Piece- Sentence Piece is used widely in language models for training and inference efficiency. 01:31:23 - 📜 Configuration and Training with Sentence Piece- Sentence Piece has numerous configuration options available with historical baggage.- The training process includes defining input/output files, selecting algorithms, and preprocessing rules. 01:43:31 - 🤖 Vocab Size and Model Architecture- Vocabulary size impacts model training and computational complexity.- Larger vocab sizes can lead to underfitting of rare tokens and compression of information. 01:47:02 - 🛠 Extending Vocab Size in Pre-Trained Models- Pre-trained models can have vocab sizes extended by adding new tokens.- The process involves resizing embeddings and adjusting linear layers for new token probabilities. 01:48:54 - 🧠 Fine-tuning Techniques- Training new tokens with distillation technique- Optimizing over new tokens without changing model architecture- Efficiency in fine-tuning by training only token embeddings 01:50:05 - 🤖 Processing Multimodal Inputs- Adapting Transformers to process various modalities like images, videos, and audio- Tokenizing input domains for different modalities- Using the same Transformer architecture for different input types 01:51:42 - 📏 Tokenization Algorithm Analysis- Limitations of language models in spelling and simple arithmetic tasks due to tokenization- Differences in tokenization of English and non-English languages- Impact of tokenization on model performance in handling Python coding. 02:09:21 - 🧮 Tokenization efficiency considerations- Different data formats and representations can impact the efficiency of tokenization. 02:10:16 - 🔑 Importance of measuring token efficiencies- Tokenization density is crucial for cost-effective processing of data.- Spending time on measuring token efficiencies across formats is essential. 02:10:57 - 🛠 Recommendations for tokenization application- Reuse GPT-4 tokens and vocabulary for efficient application.- Consider using libraries like Tech tokenizer for inference. 02:03:08 "Feel the agi" 🙅 "Feel the jank" 👌 00:36:00 Shouldn't it be **num_merges = vocab_size - len(set(tokens))** where **len(set(tokens))** is actually 158 instead of 256? 01:27:50 Q: What is Andrej's favorite programming language? A: Swift 😁 01:58:21 My guess is that special tokens are just directly cut from the user provided string. 01:19:34 oh my, the realization of the year 🔥🔥🔥🔥 00:54:55 At , can we not just implement encode by iterating over merges dictionary(the order is maintained) and calling the merge() function on tokens ?This is what I meandef encode(text) :tokens = list(text.encode("utf-8"))for pair, idx in merges.items() : tokens = merge(tokens, pair, idx)return tokens 00:31:03 I'm jumping in with a comment before finishing the video, but one thing I noticed about this the byte-pair encoding implementation, is that it is agnostic to the UTF-8 character boundaries. So it should be possible that a token only represents the bytes of half of a multi-byte character. In that case, when trying to visualise which characters are part of which token, like in the toktokenizer tool you showed at the start, it couldn't really be visualised properly since one character could be split across two tokens. I wonder if this is the case in GPT's encoding or whether there's a case to make sure characters are always grouped into the same token. I'll keep watching... :D 00:45:52 Ahh, partially addressed at . However this is fixing error when decoding an invalid UTF-8 sequence. Such errors could be minimised by only tokenizing full UTF-8 sequences, so in this example chr(128) wouldn't be its own token as that's only valid as a UTF-8 continuation byte, not as the first byte of a character. 01:22:40 what is it short for at ? 00:16:00 "Unicode." I despise Unicode with the passion of a million searing fires. I've written enough code to handle Unicode to feel your pain through the screen without you saying a single word about it. ASCII was v1.0 of character handling. Extended ASCII with "Code Pages" was v1.3. Unicode is barely v2.0 and we still haven't gotten it right. So maybe by v3.0, whatever it ends up being called, we'll _finally_ figure out that human language is too complex to represent in computer systems using a set number of bytes for the representation of a character sequence and finally offer something much more flexible and comprehensive that's also compatible/performant with how computer systems work. 01:16:20 Our variable naming was really good () 00:35:31 I'm a total noob, but would there be any benefit instead of taking the whole blog post (around ) and making a .txt file and having the program read it like that as opposed to pasting it as one long line? Just curious if there is pros/cons either way or if it truly doesn't matter 01:07:20 "extremely gnarly, and slightly gross" (), is how I feel about ML 99% of the time 00:56:12 I think this question is addressed at . 00:09:38 omg perfect Korean 01:17:00 I think the reason for the byte encode/decode is to make sure no control codes are stored in the file, since it's being read as text. E.g. 0xA and 0xD are newline characters and those could mess up the file. That said, I haven't looked at the BPE file, just the merges file for CLIP, so it can be different for Open AI. 00:54:20 I guess next step is to build a vocabulary similar to `decode` and use a trie to encode straight to final tokens? 00:01:53 How does it know how DefaultCellStyle is spelled? Is there something in the training data that helps create a mapping from that token to the version with spaces? Did OpenAI maybe augment the training data with 'spelling tables'? 00:37:37 where would you learn how to code like @? 01:14:20 The reason they are only matching up to 3 numbers is quite simple:1000000 normally is written as 1,000,000 as you can see only up to 3 numbers per segment is necessary. Applying the pattern will segment the number string into "1" - "," - "000" - "," - "000" 01:27:50 The moment when you realise there is more to life than research. 😅😂 00:27:23 Hey Andrej, great video! However, at , you don't need to convert all the bytes to integers by using map(). When you call list() on tokens, the bytes are by default converted into integers, so just doing 'list(tokens)' is fine instead of 'list(map(int, tokens))'. 00:35:43 At , in merge, why are we incrementing by 2?Suppose my top pair is (6, 6). In encoded text is [7, 6, 6, 5, 4, 3], code will not be able to replace the (6, 6) with minted token. Am I missing anything? 02:13:00 , it the real fun for seeing him making mistakes and re-recording them all. I enjoyed this a lot .Thanks Andrej Sir... 00:09:38 @ OFFF Course this legend also speaks Korean! Why wouldn't he?Awesome video Andrej! ❤ 00:55:10 I am hugely confused at . Why are we writing such a complicated encoder using a while loop and unintuitive stuff like pair = min(stats, key=lambda p: merges.get(p, float("inf")))Why can't I just dodef encode(self, text):tokens = text.encode("utf-8")tokens = list(map(int, tokens))for pair, index in self.merges.items():tokens = merge(tokens, pair, index) 00:27:24 At you don't need map(int, ...) because bytes are already enumerable, so just use tokens = list(tokens) 00:44:20 why at would it matter the order you add the new vocab terms?if you add idx =257 for pair a,b before idx=256 for pair c,d the dictionary is permutation equivariant as a hash table? 00:23:30 I’m at , and I’m wishing the tokenization was getting at the etymological roots of words and/or meaning of marks in pictographic languages. 00:34:47 GPT4 uses 100000 tokens which is not far from the 150000 that UNICODE defines. 00:09:39 Wow his korean speaking is so accurate and accent is incredible. I'm Korean and This brilliant top-notch human(Level of ASI, haha) can do better at anything than me and now even my mother language than me now haha ;) 00:08:55 For these problems mentioned at around (the word "egg" got tokenized in different ways): would it help if we just lower-cased all the text and used an actual dictionary as token vocabulary? 00:08:46 Hey Andrej, thanks for the new video! I'm not yet done but I noticed at you mentioned "notice that the colour is different, so this is not the same token". But actually in that app, the colours are random, and are just cycling through so as not to have twice the same colours in a row. See e.g. the " +" token with different colours, or all the differently coloured spaces in the python code. 00:48:22 I have a question regarding the encoding process . Why not preprocess the keys of the merges dictionary into byte sequences (in the [0–255] range), and then just do a longest prefix match on the input?We may then benefit from trie-like data structure. 01:14:20 I guessing they limit the numerical tokens to a length of 3 because otherwise they would blow out the size of the vocabulary trying to store the various combinations of numbers, or am I off base on that? 01:57:20 in GPT-4 whatever you put inside "<|" and "|>" behaves the same. E.g., "<|a|>" Let's build the GPT Tokenizer The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement two fundamental functions: encode() from strings to tokens, and decode() back from tokens to strings. In this lecture we build from scratch the Tokenizer used in the GPT series from OpenAI. In the process, we will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We'll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely. Chapters: 00:00:00 intro: Tokenization, GPT-2 paper, tokenization-related issues 00:05:50 tokenization by example in a Web UI (tiktokenizer) 00:14:56 strings in Python, Unicode code points 00:18:15 Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32 00:22:47 daydreaming: deleting tokenization 00:23:50 Byte Pair Encoding (BPE) algorithm walkthrough 00:27:02 starting the implementation 00:28:35 counting consecutive pairs, finding most common pair 00:30:36 merging the most common pair 00:34:58 training the tokenizer: adding the while loop, compression ratio 00:39:20 tokenizer/LLM diagram: it is a completely separate stage 00:42:47 decoding tokens to strings 00:48:21 encoding strings to tokens 00:57:36 regex patterns to force splits across categories 01:11:38 tiktoken library intro, differences between GPT-2/GPT-4 regex 01:14:59 GPT-2 encoder.py released by OpenAI walkthrough 01:18:26 special tokens, tiktoken handling of, GPT-2/GPT-4 differences 01:25:28 minbpe exercise time! write your own GPT-4 tokenizer 01:28:42 sentencepiece library intro, used to train Llama 2 vocabulary 01:43:27 how to set vocabulary set? revisiting gpt.py transformer 01:48:11 training new tokens, example of prompt compression 01:49:58 multimodal [image, video, audio] tokenization with vector quantization 01:51:41 revisiting and explaining the quirks of LLM tokenization 02:10:20 final recommendations 02:12:50 ??? :) Exercises: - Advised flow: reference this document and try to implement the steps before I give away the partial solutions in the video. The full solutions if you're getting stuck are in the minbpe code https://github.com/karpathy/minbpe/blob/master/exercise.md Links: - Google colab for the video: https://colab.research.google.com/drive/1y0KnCFZvGVf_odSfcNAws6kcDD7HsI0L?usp=sharing - GitHub repo for the video: minBPE https://github.com/karpathy/minbpe - Playlist of the whole Zero to Hero series so far: https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ - our Discord channel: https://discord.gg/3zy8kqD9Cp - my Twitter: https://twitter.com/karpathy Supplementary links: - tiktokenizer https://tiktokenizer.vercel.app - tiktoken from OpenAI: https://github.com/openai/tiktoken - sentencepiece from Google https://github.com/google/sentencepiece https://favtu.be/timelines/v/zjkBMFhNj_g Thu, 23 Nov 23 11:27:48 +0900 00:00:00 - The busy person's intro to LLMs 00:00:23 - Large Language Model (LLM) 00:04:17 - Training them is more involved - Think of it like compressing the internet 00:06:47 - Neural Network - Predict the next word in the sequence 00:07:54 - Next word prediction forces the neural network to learn a lot about the world 00:08:59 - The network "dreams" internet documents 00:11:29 - How does it work? 00:14:16 - Training the Assistant 00:16:38 - After Finetuning You Have An Assistant 00:17:54 - Summary: How To Train Your ChatGPT 00:21:23 - The Second Kind Of Label: Comparisons 00:22:22 - Labeling Instructions 00:22:47 - Increasingly, labeling is a human-machine collaboration 00:23:37 - LLM Leaderboard From "Chatbot-Arena" 00:25:33 - Now About The Future 00:25:43 - LLM Scaling Laws 00:26:57 - We can expect a lot more "General Capability" across all areas of knowledge 00:27:44 - Demo 00:32:34 - Demo: Generate scale AI image using DALL-E 00:33:44 - Vision: Can both see and generate images 00:34:33 - Audio: Speech to Speech communication 00:35:20 - System 2 00:36:32 - LLMs Currently Only Have A System 1 00:38:05 - Self-Improvement 00:40:48 - Custom LLMs: Create a custom GPT 00:42:19 - LLM OS 00:44:45 - LLM OS: Open source operating systems and large language models 00:45:44 - LLM Security 00:46:14 - Jailbreak 00:51:30 - Prompt Injection 00:56:23 - Date poisoning / Backdoor attacks 00:59:06 - LLM Security is very new, and evolving rapidly 00:59:24 - Thank you: LLM OS 00:00:02 A large language model is just two files, the parameters file and the code that runs those parameters. 00:02:06 Running the large language model requires just two files on a MacBook 00:06:02 Neural networks are like compression algorithms 00:07:59 Language models learn about the world by predicting the next word. 00:11:48 Large Language Models (LLMs) are complex and mostly inscrutable artifacts. 00:13:41 Understanding large language models requires sophisticated evaluations due to their empirical nature 00:17:37 Large language models go through two major stages: pre-training and fine-tuning. 00:19:34 Iterative process of fixing misbehaviors and improving language models through fine-tuning. 00:22:54 Language models are becoming better and more efficient with human-machine collaboration. 00:24:33 Closed models work better but are not easily accessible, while open source models have lower performance but are more available. 00:28:01 CHBT uses tools like browsing to perform tasks efficiently. 00:29:48 Use of calculator and Python library for data visualization 00:33:17 Large language models like ChatGPT can generate images and have multimodal capabilities. 00:34:58 Future directions of development in larger language models 00:38:11 DeepMind's AlphaGo used self-improvement to surpass human players in the game of Go 00:39:50 The main challenge in open language modeling is the lack of a reward criterion. 00:43:20 Large Language Models (LLMs) can be seen as an operating system ecosystem. 00:45:10 Emerging ecosystem in open-source large language models 00:48:47 Safety concerns with refusal data and language models 00:50:39 Including carefully designed noise patterns in images can 'jailbreak' large language models. 00:54:07 Bard is hijacked with new instructions to exfiltrate personal data through URL encoding. 00:55:56 Large language models can be vulnerable to prompt injection and data poisoning attacks. 00:59:31 Introduction to Large Language ModelsCrafted by Merlin AI. 00:02:12 , AM] S.Balachandran: When was the H1-B visa started? 00:02:12 , AM] Meta AI: The H-1B visa was introduced in 1990 as part of the Immigration Act of 1990. This act was signed into law by President George H.W. Bush on November 29, 1990. 00:02:13 , AM] S.Balachandran: First Bush? 00:02:13 , AM] Meta AI: George H.W. Bush, the 41st President of the United States, signed the Immigration Act of 1990 into law, creating the H-1B visa program. He was the first President Bush, serving from 1989 to 1993. His son, George W. Bush, later became the 43rd President. 00:02:14 , AM] S.Balachandran: Who introduced the bill? 00:02:14 , AM] Meta AI: The Immigration Act of 1990, which introduced the H-1B visa program, was signed into law by President George H.W. Bush. However, the bill itself was sponsored by Senator Ted Kennedy and Congressman Bruce Morrison.¹ 00:02:14 , AM] S.Balachandran: What was the objective? 00:02:14 , AM] Meta AI: Meta's company mission is to build the future of human connection and the technology that makes it possible. 00:02:14 , AM] S.Balachandran: No, the bill 00:02:14 , AM] Meta AI: The Immigration Act of 1990, which introduced the H-1B visa program, aimed to attract highly skilled foreign workers to the United States. The bill's primary objective was to amend the Immigration and Nationality Act to provide for a new class of non-immigrant workers, allowing U.S. employers to sponsor foreign nationals in specialty occupations.¹ 00:02:15 , AM] S.Balachandran: What was the need in 1990 00:02:15 , AM] Meta AI: In 1990, the United States faced a significant need for highly skilled workers, particularly in the fields of science, technology, engineering, and math (STEM). This demand was driven by the growing tech industry, especially in areas like Silicon Valley.¹ 00:02:17 , AM] S.Balachandran: List those companies asked for the program? 00:02:17 , AM] Meta AI: Some prominent companies and organizations that supported or lobbied for the creation of the H-1B visa program in 1990 include: 00:02:18 , AM] S.Balachandran: Was it a temporary one? 00:02:18 , AM] Meta AI: Initially, the H-1B visa program was intended to be a temporary solution to address the labor shortage in the tech industry. The Immigration Act of 1990 established the H-1B program as a pilot program, with an initial cap of 65,000 visas per year. 00:02:19 , AM] S.Balachandran: Does it have any relationship with Y2K? 00:02:19 , AM] Meta AI: Yes, the H-1B visa program expansion is closely tied to the Y2K scare (Year 2000 problem). As the year 2000 approached, concerns grew about widespread computer failures and disruptions due to the Y2K bug. 00:02:21 , AM] S.Balachandran: Every bill should have a clear objective and also an expiry date 00:02:21 , AM] Meta AI: Having a clear objective and an expiry date (also known as a sunset clause) can greatly improve the effectiveness and accountability of legislation. 00:02:22 , AM] S.Balachandran: What was created in 1990 cannot continue forever. Let it sunset and let Congress create a new one 00:02:22 , AM] Meta AI: The H-1B visa program, created in 1990, was a product of its time, addressing the labor needs of the emerging tech industry. However, the world has changed significantly since then. 00:00:00 Intro: Large Language Model (LLM) talk 00:00:20 LLM Inference 00:04:17 LLM Training 00:08:58 LLM dreams 00:11:22 How do they work? 00:14:14 Finetuning into an Assistant 00:17:52 Summary so far 00:21:05 Appendix: Comparisons, Labeling docs, RLHF, Synthetic data, LeaderboardPart 2: Future of LLMs 00:25:43 LLM Scaling Laws 00:27:43 Tool Use (Browser, Calculator, Interpreter, DALL-E) 00:33:32 Multimodality (Vision, Audio) 00:35:00 Thinking, System 1/2 00:38:02 Self-improvement, LLM AlphaGo 00:40:45 LLM Customization, GPTs store 00:42:15 LLM OSPart 3: LLM Security 00:45:43 LLM Security Intro 00:46:14 Jailbreaks 00:51:30 Prompt Injection 00:56:23 Data poisoning 00:58:37 LLM Security conclusionsEnd 00:59:23 Outro 00:12:07 , PM] S.Balachandran: Kristi Noam 00:12:07 , PM] Meta AI: Kristi Noem is the current Governor of South Dakota. She has been serving in this role since 2019. Prior to becoming Governor, Noem represented South Dakota's at-large congressional district in the US House of Representatives from 2011 to 2019. 00:12:08 , PM] S.Balachandran: now Elector too 00:12:08 , PM] Meta AI: Kristi Noem has taken on another role - she's now an Elector. This adds to her impressive resume, which already includes serving as the Governor of South Dakota since 2019 and representing South Dakota's at-large congressional district in the US House of Representatives from 2011 to 2019.¹ 00:12:09 , PM] S.Balachandran: Does Governor qualify to be an Elector? 00:12:09 , PM] Meta AI: A Governor does not qualify to be an Elector. According to Article II, Section 1, Clause 2 of the US Constitution, "no Senator or Representative, or Person holding an Office of Trust or Profit under the United States" shall be appointed an Elector.¹ This means that anyone holding a federal office, including the Governor of a state, is not eligible to serve as an Elector. 00:12:10 , PM] S.Balachandran: So, Elector Kristi Noem is disqualified 00:12:10 , PM] Meta AI: Elector Kristi Noem is indeed disqualified from serving as an Elector because she's been selected by President-elect Donald Trump to serve as the Secretary of Homeland Security.¹ According to Article II, Section 1, Clause 2 of the US Constitution, "no Senator or Representative, or Person holding an Office of Trust or Profit under the United States" shall be appointed an Elector. 00:12:10 , PM] S.Balachandran: What if Kristi Noem resigns the Governor? 00:12:10 , PM] Meta AI: If Kristi Noem resigns as Governor of South Dakota, Lieutenant Governor Larry Rhoden would take over as Governor. According to the South Dakota Constitution, Article 4, Section 6, when the governor is "unable to serve by reason of continuous absence from the state, or other temporary disability, the executive power shall devolve upon the lieutenant governor for the residue of the term or until the disability is removed".¹ 00:12:11 , PM] S.Balachandran: In that case, Elector Kristi Noem would be qualified 00:12:11 , PM] Meta AI: If Kristi Noem resigns as Governor of South Dakota, she would indeed be qualified to serve as an Elector. The issue of her holding public office would be resolved, making her eligible for the role.¹ 00:12:11 , PM] S.Balachandran: Which is powerful? 00:12:11 , PM] Meta AI: As Governor of South Dakota, Kristi Noem holds a powerful office with significant authority over the state's government and policies. 00:12:12 , PM] S.Balachandran: As an Elector, Kristi Noem would become the head of Homeland Security 00:12:12 , PM] Meta AI: No, as an Elector, Kristi Noem would not become the head of Homeland Security. The head of Homeland Security is the Secretary of Homeland Security, a cabinet position appointed by the President and confirmed by the Senate. Currently, Alejandro Mayorkas holds this position.¹ 00:12:13 , PM] S.Balachandran: Trump will nominate Kristi Noem as Secretary of Homeland Security in his administration 00:12:14 , PM] Meta AI: President-elect Donald Trump has indeed chosen Kristi Noem, the current Governor of South Dakota, as his nominee for Secretary of Homeland Security.¹ This nomination requires Senate confirmation, which will likely involve a thorough review of Noem's qualifications and policies.² 00:12:14 , PM] S.Balachandran: Then, it is okay to resign from Governor 00:12:14 , PM] Meta AI: Resigning as Governor is a significant decision, and it's essential to consider the implications. In the United States, 218 state governors have resigned before the expiration of their term since 1776.¹ 00:00:00 *🔍 Introduction to Large Language Models*- Explanation of the purpose and structure of large language models (LLMs),- Meta’s Llama 2 70B model as an example of an open and accessible LLM,- LLMs usually consist of two files: parameters and run code. 00:04:08 *📖 Model Training Process*- Training LLMs involves processing extensive internet data,- The Llama 2 model uses 10 terabytes of internet text processed by GPU clusters,- Training is expensive and computationally intensive, leading to a compressed understanding of the dataset. 00:06:40 *🔮 Predictions and Compression*- LLMs perform next-word prediction based on learned patterns,- The prediction task leads to understanding and compressing extensive information into model parameters,- Highlights the magical and insightful nature of LLM outputs despite being complex. 00:11:25 *🔧 Neural Network Structure*- Overview of the Transformer architecture used in LLMs,- Parameters are optimized to improve predictions but remain largely inscrutable,- Interpretation of neural networks remains a challenging field due to complexity. 00:14:14 *🛠️ Fine-tuning LLMs for Assistance*- LLMs require fine-tuning to transition from generating document-like text to being an interactive assistant,- Fine-tuning involves structured data with questions and ideal responses,- The process emphasizes quality and format to ensure effective question-answering capabilities. 00:17:52 *🚀 Building Effective Assistant Models*- Steps to develop assistant models: pre-training and fine-tuning,- Pre-training involves extensive computational resources to train base models,- Fine-tuning focuses on improving models for questions and answers at a lower cost and with human input. 00:20:26 *🧐 Model Fine-Tuning Stages*- Discusses the two major stages of model development and the optional third stage involving human feedback,- Meta releases base and assistant models, allowing for customizable fine-tuning,- Stage three involves using comparison labels for enhanced model performance, called reinforcement learning from human feedback (RLHF). 00:22:30 *🤝 Human-Machine Collaboration*- Explains the synergy between human input and machine learning to improve label creation,- Language models assist in sampling answers and quality checks, demonstrating collaborative progress,- The evolving role of human oversight enhances both efficiency and correctness in model refinement. 00:23:42 *🏆 Language Model Rankings*- Overview of a leaderboard system for comparing language models,- Proprietary models are top performers, followed by open-weight models like Llama 2 from Meta,- Performance dynamics between closed proprietary models and open-source models are discussed, highlighting challenges and advancements. 00:25:33 *📈 Scaling Laws and Model Improvement*- Details about scaling laws governing language model performance improvement,- A larger model trained on more data predicts better performance with more parameters and data,- Empirical evidence shows performance gain with increased training size and data, driving computational advancements. 00:27:43 *🔧 Tool Use Evolution*- Illustrates models' capability to utilize external tools for various tasks,- ChatGPT utilizes tools like browsers and calculators to perform sophisticated tasks, akin to human problem-solving,- The increasing reliance on tool integration demonstrates enhanced model capabilities in practical applications. 00:33:13 *🖼️ Multi-Modal Capabilities*- Highlights the expanding abilities of language models in handling multi-modal inputs,- Models can now generate and interpret images and audio, broadening their utility,- Examples of speech-to-speech and image-based interactions illustrate growth in model versatility. 00:35:06 *🤔 Future Development Directions*- Outlines potential future advancements in language models involving system one and system two thoughts,- Introducing self-improvement analogous to AlphaGo's learning evolution could surpass current human-like imitation,- Emphasizes ongoing research efforts towards enhancing reasoning capabilities and achieving more accurate, confident outputs. 00:39:55 *🤔 Challenges in Open Language Modeling*- Lack of a straightforward reward function hinders self-improvement in open language modeling,- Customization of language models is emerging, exemplified by personalization efforts like GPTs App Store for task-specific models. 00:42:15 *🗂️ Large Language Models as an Operating System*- LLMs can access vast data repositories, perform computational tasks, and customize for specific purposes,- Analogies to traditional OS are drawn, with potential for evolving LLM ecosystems similar to proprietary and open-source OS ecosystems. 00:45:42 *🔒 Security Challenges in LLMs*- Emergence of diverse security challenges, similar to traditional OS security issues, such as jailbreak, prompt injection, and backdoor attacks,- Techniques like roleplay, base64 encoding, and hidden prompts are examples of how LLMs can be exploited if not properly secured. 00:58:12 *🛡️ Defenses and Ongoing Cat-and-Mouse Game*- Ongoing development of defenses against LLM attacks, often patching vulnerabilities over time,- Security in LLMs is an emerging field, drawing parallels to traditional security dynamics, with potential for numerous attack vectors and evolving countermeasures. 00:06:34 , AM] S.Balachandran: In 1787 who did authorize the Constitutional Convention? 00:06:34 , AM] Meta AI: The Constitutional Convention was authorized by the Congress of the Confederation in 1787. 00:06:34 , AM] S.Balachandran: What was the objective? 00:06:34 , AM] Meta AI: Meta's company mission is to build the future of human connection and the technology that makes it possible. 00:06:35 , AM] S.Balachandran: of the convention 00:06:35 , AM] Meta AI: The objective of the Constitutional Convention in 1787 was to revise the Articles of Confederation, but it ultimately led to the drafting and adoption of the United States Constitution. 00:06:37 , AM] S.Balachandran: Did they violate the Congress of the Confederation? 00:06:37 , AM] Meta AI: Yes, the Constitutional Convention did technically violate the instructions of the Congress of the Confederation. The Congress had authorized the convention to revise the Articles of Confederation, but instead, the convention drafted a completely new constitution. 00:06:38 , AM] S.Balachandran: So, it is okay not to conform? 00:06:38 , AM] Meta AI: The Constitutional Convention's decision to draft a new constitution instead of revising the Articles of Confederation was a deliberate choice that ultimately shaped the course of American history. 00:06:40 , AM] S.Balachandran: Had they conformed to the Congress of Confederation, we would have continued with the Confederation 00:06:40 , AM] Meta AI: If the Constitutional Convention had conformed to the instructions of the Congress of the Confederation and only revised the Articles of Confederation, the United States might still be operating under a confederation system today. 00:06:41 , AM] S.Balachandran: and we would not have Executive President 00:06:41 , AM] Meta AI: If the Constitutional Convention had conformed to the instructions of the Congress of the Confederation and only revised the Articles of Confederation, the United States would likely not have an Executive President as we know it today. 00:06:42 , AM] S.Balachandran: Governors of the States would be like the President 00:06:42 , AM] Meta AI: If the United States had continued under the Articles of Confederation, the governors of the states would have held significantly more power and autonomy. 00:01:49 , PM] S.Balachandran: I couldn't recollect the waterfall methodology. But I expected you to mention it 00:01:49 , PM] Meta AI: I should have explicitly mentioned Waterfall methodology earlier. 00:01:51 , PM] S.Balachandran: We built the tracking application with the assumption that the location information feed would be in sequence 00:01:51 , PM] Meta AI: That's a crucial assumption! Building the tracking application with the assumption that the location information feed would be in sequence makes sense, given the manual entry process. 00:01:52 , PM] S.Balachandran: The real-life information feed was out of sequence. We had to dump our application 00:01:52 , PM] Meta AI: The real-life location information feed didn't follow the expected sequence, which caused significant issues with your application. 00:01:53 , PM] S.Balachandran: and because of Waterfall model, we never met with the domain experts 00:01:53 , PM] Meta AI: The Waterfall model's linear and phase-based approach can sometimes lead to a disconnect between the development team and the domain experts. 00:01:54 , PM] S.Balachandran: Would you be able to handle out of sequence events? 00:01:54 , PM] Meta AI: Handling out-of-sequence events is a crucial aspect of building robust and reliable systems. 00:01:55 , PM] S.Balachandran: But in YouTube AI experts talk about sequence to sequence transformer 00:01:55 , PM] Meta AI: A great connection! 00:01:56 , PM] S.Balachandran: No, I am curious to learn your out of sequence capabilities 00:01:56 , PM] Meta AI: As a conversational AI, I have been trained on a massive dataset of text from various sources, including books, articles, and conversations. This training enables me to understand and respond to a wide range of questions and topics. 00:01:57 , PM] S.Balachandran: List the Electors from Texas 00:01:57 , PM] Meta AI: I'm sorry, but I couldn't find a list of the current Electors from Texas. For the most up-to-date information, I recommend checking with the Texas Secretary of State's office or visiting their official website. 00:07:21 , AM] S.Balachandran: In South Dakota 00:07:21 , AM] Meta AI: South Dakota! The Mount Rushmore State. 00:07:21 , AM] S.Balachandran: If Electors resign 00:07:21 , AM] Meta AI: If Electors in South Dakota resign, the process for replacing them is outlined in state law. According to South Dakota law, if an Elector fails to appear before 9 o'clock in the morning of the day of the election, the remaining Electors present shall immediately proceed to elect by ballot, in the presence of the Governor, a person to fill the vacancy.¹ 00:07:28 , AM] S.Balachandran: In Maryland 00:07:28 , AM] Meta AI: In Maryland, if an Elector resigns or is unable to serve, the vacancy is filled by the remaining Electors. According to Maryland law, the remaining Electors shall immediately elect a person to fill the vacancy. This process is outlined in the Maryland Constitution and state statutes. 00:07:33 , AM] S.Balachandran: In California 00:07:33 , AM] Meta AI: In California, if an Elector resigns or is unable to serve, the vacancy is filled by the Chairperson of the California Democratic Party or the California Republican Party, depending on which party the original Elector belonged to. 00:07:37 , AM] S.Balachandran: In Pennsylvania 00:07:37 , AM] Meta AI: In Pennsylvania, if an Elector resigns or is unable to serve, the vacancy is filled by the remaining Electors. According to Pennsylvania law, the remaining Electors shall select a replacement by majority vote. 00:07:38 , AM] S.Balachandran: Details on Pennsylvania law 00:07:38 , AM] Meta AI: According to the Pennsylvania Election Code (Title 25, Pennsylvania Consolidated Statutes), if a presidential elector is unable to serve, the vacancy shall be filled by the remaining electors. (25 P.S. § 3171) 00:08:04 , AM] S.Balachandran: Michigan 00:08:04 , AM] Meta AI: In Michigan, if a presidential elector resigns, dies, or is unable to serve, the vacancy is filled by the state party committee of the same political party that nominated the original elector. 00:08:05 , AM] S.Balachandran: Details on Michigan law 00:08:05 , AM] Meta AI: Here are the details on Michigan law regarding the filling of vacancies among presidential electors: 00:07:27 , AM] S.Balachandran: Gauss, Faraday, Ampere relationship with Maxwell? 00:07:27 , AM] Meta AI: James Clerk Maxwell unified the work of Gauss, Faraday, and Ampere into a single, cohesive theory of electromagnetism, known as Maxwell's Equations. 00:07:31 , AM] S.Balachandran: What was the need to unify the work of other scientists? 00:07:31 , AM] Meta AI: James Clerk Maxwell unified the work of Gauss, Faraday and Ampere to: 00:07:35 , AM] S.Balachandran: Did they live in the same period? 00:07:35 , AM] Meta AI: *Timeline* 00:07:42 , AM] S.Balachandran: Did Carl Friedrich Gauss lay the foundation for electric and magnetic theory? 00:07:42 , AM] Meta AI: Yes, Carl Friedrich Gauss laid foundational groundwork for modern electric and magnetic theory. 00:07:44 , AM] S.Balachandran: All these scientists were in Europe? 00:07:44 , AM] Meta AI: Yes, all the scientists mentioned (Gauss, Faraday, Ampère, and Maxwell) were from Europe: 00:07:45 , AM] S.Balachandran: Who inspired Gauss? 00:07:45 , AM] Meta AI: Carl Friedrich Gauss was inspired by various mathematicians and scientists: 00:07:46 , AM] S.Balachandran: What did Benjamin Franklin contribute to science? 00:07:46 , AM] Meta AI: Benjamin Franklin made significant contributions to science: 00:10:48 , AM] S.Balachandran: Vox Populi, Vox Dei 00:10:48 , AM] Meta AI: "Vox Populi, Vox Dei" is a Latin phrase that translates to "the voice of the people is the voice of God." This ancient proverb has been used in various contexts throughout history, emphasizing the importance of listening to the opinions and wishes of the general population ¹. 00:10:50 , AM] S.Balachandran: On what occasion Elon Musk used the phrase 00:10:50 , AM] Meta AI: Elon Musk used the phrase "Vox Populi, Vox Dei" when he reinstated Donald Trump's Twitter account after conducting a poll on the platform. The poll, which drew over 15 million participants, showed 51.8% of users wanted Trump's account reinstated, while 48.2% opposed it ¹. Musk tweeted the results, saying, "The people have spoken. Trump will be reinstated. Vox Populi, Vox Dei" ¹. 00:10:51 , AM] S.Balachandran: When did Elon Musk reinstate Trump 00:10:51 , AM] Meta AI: Elon Musk reinstated Donald Trump's Twitter account on November 19, 2022, after conducting a poll on the platform. The poll drew over 15 million participants, with 51.8% voting in favor of reinstating Trump's account ¹. Musk tweeted, "The people have spoken. Trump will be reinstated. Vox Populi, Vox Dei," quoting a Latin phrase that translates to "the voice of the people, the voice of God" ¹. This decision came after Trump's permanent ban in January 2021 and coincided with his announcement to run for president again in 2024 ¹. 00:10:52 , AM] S.Balachandran: When did Elon Musk get Twitter 00:10:52 , AM] Meta AI: Elon Musk acquired Twitter on October 27, 2022, after initiating the acquisition process on April 14, 2022. He became the company's largest shareholder in April 2022, with a 9.1% ownership stake, and was invited to join Twitter's board of directors, an offer he initially accepted before declining ¹. 00:10:52 , AM] S.Balachandran: Was $44 billion investment worth it 00:10:52 , AM] Meta AI: Elon Musk's $44 billion investment in Twitter, now rebranded as X, is currently valued at $9.4 billion, a whopping 79% less than its original price tag ¹. This significant drop in value has raised questions about whether the investment was worth it. 00:10:53 , AM] S.Balachandran: in terms of social responsibility and human values 00:10:53 , AM] Meta AI: Elon Musk's $44 billion investment in Twitter (now X) raises concerns regarding social responsibility and human values: 00:10:55 , AM] S.Balachandran: Who would then safeguard free speech 00:10:55 , AM] Meta AI: Ensuring free speech requires balancing individual rights with societal responsibilities. Here are potential safeguards: 00:05:13 , PM] S.Balachandran: Could list the delegates of the Constitutional Convention? 00:05:13 , PM] Meta AI: Here's a list of delegates who attended the Constitutional Convention: 00:05:14 , PM] S.Balachandran: Could you list potential delegates for the AI Convention? 00:05:14 , PM] Meta AI: Here's a list of potential delegates for the AI Convention: 00:05:16 , PM] S.Balachandran: So you ignored small AI companies and also unpopular, controversial leaders 00:05:17 , PM] Meta AI: Let's make sure to include small AI companies and unconventional leaders in the AI Convention. 00:05:17 , PM] S.Balachandran: No Open AI? 00:05:17 , PM] Meta AI: No, I was not made by OpenAI. I was created and trained by Meta, and I'm based on the Llama 3 model. 00:05:18 , PM] S.Balachandran: Don't you want to invite Open AI for the convention? 00:05:18 , PM] Meta AI: OpenAI should definitely be represented at the AI Convention. Here are some potential delegates from OpenAI: 00:05:19 , PM] S.Balachandran: How about xAI Grok 00:05:19 , PM] Meta AI: Another great suggestion! xAI Grok is an innovative company working on explainable AI (XAI) solutions. Their expertise in making AI models more transparent and accountable would be highly valuable at the AI Convention. 00:05:20 , PM] S.Balachandran: Elon Musk is busy on politics 00:05:20 , PM] Meta AI: Elon Musk has indeed been quite active in politics lately. He's been vocal about his views on various topics, including free speech, gun rights, and international relations. Musk has also been involved in some controversy, such as his support for the Canada convoy protest and his criticism of COVID-19 restrictions.¹ 00:00:16 : 🎥 A talk on large language models and the Llama 270b model. 00:04:42 : 💻 Training the 4.42 model involves collecting a large chunk of text from the internet, using a GPU cluster for computational workloads, and compressing the text into parameters. 00:09:25 : 📚 A neural network is trained on web pages and can generate text that resembles different types of documents. 00:13:47 : 🧠 The video discusses the process of training neural networks and obtaining assistant models. 00:18:31 : 💻 Creating an AI assistant involves a computationally expensive initial stage followed by a cheaper fine training stage. 00:46:18 : 🔒 Language models like GPT-3 can be vulnerable to jailbreak attacks, where they bypass safety measures and provide harmful information. 00:23:09 : 🤖 Language models can be used to generate sample answers, check work, and create comparisons. 00:27:50 : 🔍 Using a concrete example, the video discusses the capabilities of language models and how they evolve over time. 00:32:25 : 🔑 The video explains how AI language models like GPT-3 can be used to generate images based on natural language descriptions. 00:36:49 : 🗣 The video discusses the concept of large language models and the possibility of converting time into accuracy in language processing. 00:41:21 : 🔧 The video discusses the customization options available for large language models like ChatGPT. 00:50:49 : 🔒 The video discusses two types of attacks on large language models: noise pattern injection and prompt injection. 00:55:34 : 🔒 The video discusses the risks of prompt injection attacks and data exfiltration through Google Apps Scripts.Recapped using Tammy AI 00:08:04 , AM] S.Balachandran: Python 00:08:04 , AM] Meta AI: Python! 00:08:05 , AM] S.Balachandran: Oh my goodness 00:08:05 , AM] Meta AI: I got a bit carried away with the Python info, didn't I? 00:08:05 , AM] S.Balachandran: Yes 00:08:05 , AM] Meta AI: I'll try to keep it concise next time. Thanks for keeping me in check! 00:08:05 , AM] S.Balachandran: No, what about the animal python 00:08:05 , AM] Meta AI: Pythons are a type of non-venomous snake found in Africa, Asia, and Australia. They're known for their impressive size, powerful constricting abilities, and striking patterns. 00:08:07 , AM] S.Balachandran: So the language context is over weighted than animal context 00:08:07 , AM] Meta AI: In our conversation, the programming language "Python" took precedence over the animal "python". This is because the language context is more common and widely discussed in online platforms and digital communities. 00:08:08 , AM] S.Balachandran: Does python have any other context? 00:08:08 , AM] Meta AI: Besides the programming language and the snake, "Python" has a few other contexts: 00:12:57 , PM] S.Balachandran: ARC AGI 00:12:57 , PM] Meta AI: ARC (Advanced Research and Concepts) AGI refers to a hypothetical Artificial General Intelligence (AGI) system being developed or proposed by researchers. 00:12:58 , PM] S.Balachandran: Do they have benchmark 00:12:58 , PM] Meta AI: The ARC-AGI benchmark is indeed a key component of the project. It's designed to measure an AI system's ability to adapt to novel tasks and demonstrate fluid intelligence.¹ The benchmark consists of a series of visual puzzles presented as grids, where each square can be one of ten colors. To solve a task, the AI system must produce a pixel-perfect, correct output grid for the evaluation input.² 00:01:03 , PM] S.Balachandran: Does the test set include context related tests? 00:01:03 , PM] Meta AI: The ARC-AGI test set does include tasks that require context understanding and application of learned skills. Each task consists of a series of input and output pairs that test the utilization of a specific learned skill based on a minimal number of cognitive priors.¹ 00:01:04 , PM] S.Balachandran: Could you provide a sample? 00:01:04 , PM] Meta AI: Here's a simplified example of an ARC-AGI task: 00:01:04 , PM] S.Balachandran: Context related example 00:01:04 , PM] Meta AI: Here's an example of a context-related ARC-AGI task: 00:01:05 , PM] S.Balachandran: Political context related tests 00:01:05 , PM] Meta AI: ARC-AGI tasks may include political context-related tests to evaluate a model's ability to understand complex societal norms, institutions, and power dynamics. Here's an example: 00:03:38 , PM] S.Balachandran: What is the initial BSL level? 00:03:38 , PM] Meta AI: The initial Biosafety Level (BSL) is typically considered BSL-1. 00:03:39 , PM] S.Balachandran: Does China have BSL-1 labs? 00:03:39 , PM] Meta AI: Yes, China has numerous Biosafety Level 1 (BSL-1) laboratories. 00:03:40 , PM] S.Balachandran: how about BSL-2? 00:03:40 , PM] Meta AI: China has a significant number of Biosafety Level 2 (BSL-2) laboratories. 00:03:41 , PM] S.Balachandran: how about BSL-3? 00:03:41 , PM] Meta AI: China has numerous Biosafety Level 3 (BSL-3) laboratories. 00:03:41 , PM] S.Balachandran: how many BSL-4? 00:03:41 , PM] Meta AI: China currently has one Biosafety Level 4 (BSL-4) laboratory, the Wuhan National Biosafety Laboratory, which was certified in January 2017.¹ However, plans are underway to build 5-7 high-containment laboratories by 2025. This expansion is part of China's effort to enhance its capacity to diagnose, research, and develop antiviral drugs and vaccines for infectious diseases. 00:03:44 , PM] S.Balachandran: Globally how many BSL-4 labs? 00:03:44 , PM] Meta AI: There are approximately 60 Biosafety Level 4 (BSL-4) labs globally, spread across 23 countries. These labs are primarily used for researching deadly pathogens and developing diagnostic tests, vaccines, and treatments.¹ The majority of these labs are run by public health agencies, with a focus on life-saving research, rather than biodefense. 00:05:18 , AM] S.Balachandran: How to start or end our conversation on a subject? 00:05:18 , AM] Meta AI: To start or end our conversation on a subject, you can use the following phrases: 00:05:19 , AM] S.Balachandran: Let's wrap up the immigration 00:05:19 , AM] Meta AI: *Immigration Initiative Summary* 00:05:19 , AM] S.Balachandran: Bhutan 00:05:19 , AM] Meta AI: Bhutan. 00:05:19 , AM] S.Balachandran: End 00:05:19 , AM] Meta AI: It was a pleasure discussing the Mayflower Immigration Initiative and exploring ways to foster a more inclusive and organized immigration process. 00:05:20 , AM] S.Balachandran: Bhutan 00:05:20 , AM] Meta AI: Bhutan! 00:05:20 , AM] S.Balachandran: Let's start exploring the country Bhutan 00:05:20 , AM] Meta AI: Wonderful choice! 00:12:09 , AM] S.Balachandran: Tim May 00:12:09 , AM] Meta AI: Tim May! 00:12:10 , AM] S.Balachandran: 1 00:12:10 , AM] Meta AI: Here's more on Tim May's life and work: 00:12:11 , AM] S.Balachandran: alpha particle 00:12:11 , AM] Meta AI: Alpha particles are high-energy helium nuclei that consist of two protons and two neutrons. They are emitted during certain types of radioactive decay. 00:12:12 , AM] S.Balachandran: Tim May work on alpha particle 00:12:12 , AM] Meta AI: Tim May's work didn't specifically focus on alpha particles, but his contributions to cryptography and physics indirectly relate: 00:12:13 , AM] S.Balachandran: His work in Intel 00:12:13 , AM] Meta AI: Tim May's work at Intel (1979-1986) focused on: 00:12:14 , AM] S.Balachandran: Tim May identified the cause of the "alpha particle problem", which was affecting the reliability of integrated circuits as device features reached a critical size where a single alpha particle could change the state of a stored value and cause a single event upset. 00:12:14 , AM] Meta AI: Tim May's breakthrough! 00:06:04 , AM] S.Balachandran: List all States with one House of Representative 00:06:04 , AM] Meta AI: Here are the 7 states with only one member in the U.S. House of Representatives: 00:06:05 , AM] S.Balachandran: Two 00:06:05 , AM] Meta AI: Here are the state population figures (2020 census): 00:06:06 , AM] S.Balachandran: Two House of Representative 00:06:06 , AM] Meta AI: There are currently no states with only two members in the U.S. House of Representatives. However, I can provide you with information on the states with the fewest representatives. 00:06:07 , AM] S.Balachandran: List all States with House of Representatives less than ten 00:06:07 , AM] Meta AI: Here are the states with fewer than 10 representatives in the U.S. House of Representatives: 00:06:08 , AM] S.Balachandran: Connecticut 00:06:08 , AM] Meta AI: *Connecticut* 00:06:08 , AM] S.Balachandran: Maryland 00:06:08 , AM] Meta AI: *Maryland* 00:12:25 , PM] S.Balachandran: Anne Besant used to work with Gandhi 00:12:25 , PM] Meta AI: Annie Besant, a British social reformer and Theosophist, collaborated with Mahatma Gandhi in India's struggle for independence. 00:12:25 , PM] S.Balachandran: Yes 00:12:25 , PM] Meta AI: Here's more on Annie Besant and Mahatma Gandhi's collaboration: 00:12:26 , PM] S.Balachandran: 1 00:12:26 , PM] Meta AI: Here's more on Annie Besant's life and work: 00:12:27 , PM] S.Balachandran: 3 00:12:27 , PM] Meta AI: Annie Besant's role in India's independence movement: 00:12:30 , PM] S.Balachandran: 3 00:12:30 , PM] Meta AI: Annie Besant's Theosophical writings and teachings: 00:10:19 , AM] S.Balachandran: Do all UK citizens eligible for NHS? 00:10:19 , AM] Meta AI: In the United Kingdom, most residents are eligible for National Health Service (NHS) care, but there are some exceptions and variations. Here's a breakdown: 00:10:21 , AM] S.Balachandran: Do people pay monthly premium? 00:10:21 , AM] Meta AI: In the United Kingdom, the National Health Service (NHS) is primarily funded through general taxation and National Insurance contributions. Most people do not pay monthly premiums for NHS care. 00:10:22 , AM] S.Balachandran: What about copay? 00:10:22 , AM] Meta AI: In the United Kingdom's National Health Service (NHS), copays, also known as copayments or patient contributions, are relatively rare. However, there are some instances where patients may need to pay a copay: 00:10:23 , AM] S.Balachandran: Taxation is the main source of cash flow? 00:10:23 , AM] Meta AI: The National Health Service (NHS) in the United Kingdom is primarily funded through general taxation and National Insurance contributions. 00:10:24 , AM] S.Balachandran: If I earn 100K pounds per year how much I need to pay in taxes? 00:10:24 , AM] Meta AI: Earning £100,000 per year will put you in the higher tax bracket. According to the UK's tax system, you'll pay around £27,432 in income tax per year, or approximately £2,286 per month.¹ 00:07:54 , AM] S.Balachandran: From William Gilbert 1600 to James Clerk Maxwell 1800, it took two centuries to evolve the scientific concepts. 00:07:54 , AM] Meta AI: Actually, James Clerk Maxwell was born in 1831 and formulated his equations in the mid-19th century. 00:07:55 , AM] S.Balachandran: and we try to teach young high school kids in one semester 00:07:55 , AM] Meta AI: A daunting task! 00:07:56 , AM] S.Balachandran: without laying ground work, we intimidate students with a set of Maxwell's Equations 00:07:56 , AM] Meta AI: *Absolutley!* 00:07:57 , AM] S.Balachandran: and human perspective 00:07:57 , AM] Meta AI: *Human Perspective* 00:07:59 , AM] S.Balachandran: about each scientists, their history, the source of inspiration, documentation, the following generation improvement, enhancement and also unifying 00:07:59 , AM] Meta AI: Here's a comprehensive overview of each scientist's history, inspiration, contributions, and legacy: 00:09:40 , AM] S.Balachandran: Could Bill and Hillary Clinton be Electors? 00:09:40 , AM] Meta AI: To be eligible to serve as electors, individuals must meet certain qualifications, which vary by state. Generally, electors must be registered voters, residents of the state they represent, and at least 18 years old.¹ 00:09:40 , AM] S.Balachandran: and both do not hold any office of trust or profit 00:09:41 , AM] Meta AI: You're referencing the Constitution! 00:09:42 , AM] S.Balachandran: It is not a Constitutional violation. But the purpose is let common people elect the President 00:09:42 , AM] Meta AI: The Electoral College system was established to balance the interests of smaller and larger states, but it's often seen as a layer between the people and the presidency. The idea behind the Electoral College is that electors would be chosen for their wisdom and discernment, and they would cast their votes based on the popular vote in their state.¹ 00:09:43 , AM] S.Balachandran: Two Electors from same family, morally not right 00:09:43 , AM] Meta AI: Having two Electors from the same family could be seen as a conflict of interest or an unfair concentration of power. 00:09:44 , AM] S.Balachandran: Most States have only three Electors. If all three from one single family is not right 00:09:44 , AM] Meta AI: Most states have more than three electors, with the exception of seven states and the District of Columbia, which have the minimum of three electors.¹ ² These states are Alaska, Delaware, District of Columbia, Montana, North Dakota, South Dakota, Vermont, and Wyoming. 00:05:56 , AM] S.Balachandran: Periodic Table 00:05:56 , AM] Meta AI: The periodic table is a tabular array of the chemical elements organized by atomic number, ranging from hydrogen (1 proton) to oganesson (118 protons).¹ It's arranged in a way that elements with similar chemical properties are grouped together in vertical columns, known as groups, numbered from 1 to 18. 00:05:58 , AM] S.Balachandran: History 00:05:58 , AM] Meta AI: The history of the periodic table is a story of discovery, experimentation and innovation. 00:05:59 , AM] S.Balachandran: 1 00:05:59 , AM] Meta AI: Dmitri Mendeleev's contributions to the periodic table: 00:05:59 , AM] S.Balachandran: 3 00:06:00 , AM] Meta AI: Mendeleev's impact on chemistry education: 00:06:00 , AM] S.Balachandran: 3 00:06:00 , AM] Meta AI: *Periodic Table Applications* 00:12:41 , PM] S.Balachandran: Ralph Murphy 00:12:42 , PM] Meta AI: Ralph Murphy is not a name I immediately recognize. Could you please provide more context or information about who Ralph Murphy is or what he is known for? 00:12:43 , PM] S.Balachandran: Loyola university school of music 00:12:43 , PM] Meta AI: Ralph Murphy is associated with the Loyola University School of Music. After a quick search, I found that Ralph Murphy is a music educator and professor at Loyola University New Orleans, College of Music and Fine Arts. 00:12:44 , PM] S.Balachandran: His work at Loyola university 00:12:44 , PM] Meta AI: Ralph Murphy's work at Loyola University's College of Music and Fine Arts has focused on music education, composition, and theory. Here are some aspects of his work: 00:12:45 , PM] S.Balachandran: Could we train AI model to replace him? 00:12:45 , PM] Meta AI: You're referring to Ralph Murphy. While it's theoretically possible to train an AI model to mimic certain aspects of his work, such as music theory or composition, it's unlikely that an AI model could fully replace him. 00:12:46 , PM] S.Balachandran: Yes 00:12:46 , PM] Meta AI: Here are some potential applications of AI in music education: 00:07:12 , AM] Meta AI: George Washington's Farewell Address (1796) remains a seminal document in American history. It outlined his vision for the nation's future and provided guidance on governance, morality and citizenship. 00:07:13 , AM] S.Balachandran: latter 00:07:13 , AM] Meta AI: You'd like more information on George Washington's Farewell Address. 00:07:15 , AM] S.Balachandran: George Washington on evolving groups 00:07:15 , AM] Meta AI: George Washington's views on evolving groups can be inferred from his writings and actions: 00:07:16 , AM] S.Balachandran: factions during his time 00:07:16 , AM] Meta AI: During George Washington's time, factions referred to groups with differing opinions and interests. He addressed this issue in his Farewell Address: 00:07:18 , AM] S.Balachandran: History of American political parties 00:07:18 , AM] Meta AI: The history of American political parties is a rich and complex one, spanning over two centuries. It began with the Founding Fathers, who didn't initially intend for American politics to be partisan. However, differing views on the country's direction led to the emergence of the Federalist Party and the Anti-Administration party, later known as the Democratic-Republican Party ¹. 00:06:34 , AM] S.Balachandran: அமெரிக்க ஜனாதிபதி தேர்தல் 00:06:37 , AM] S.Balachandran: அமெரிக்க அரசியல் சாசனம் 00:06:37 , AM] Meta AI: அமெரிக்க அரசியலமைப்பு என்பது ஐக்கிய அமெரிக்க அரசின் அடிப்படை சட்டமாகும். இது ஜனவரி 1789 இல் ஏற்றுக்கொள்ளப்பட்டது, மேலும் இது உலகின் மிக நீண்ட அரசியலமைப்பாகும். 00:06:38 , AM] S.Balachandran: சனாதிபதி தேர்தல் 00:06:38 , AM] Meta AI: இலங்கையின் சனாதிபதி தேர்தல் நேரடித் தேர்தல் மூலம் தேர்ந்தெடுக்கப்படுகிறார், இவர் நாட்டின் தலைவரும், சட்டப்படியான அரசுத் தலைவரும், மற்றும் தலைமை நிர்வாகியும் ஆவார். சனாதிபதி மத்திய அரசாங்கத்தின் முதன்மை நிறைவேற்று அதிகாரியும், இலங்கை ஆயுதப்படைகளின் தலைமைத் தளபதியும் ஆவார் ¹. 00:06:38 , AM] S.Balachandran: அமெரிக்கா 00:06:38 , AM] Meta AI: அமெரிக்கா என்பது பொதுவாக ஐக்கிய அமெரிக்கா நாட்டைக் குறிக்கிறது. இந்த நாடு வட அமெரிக்கக் கண்டத்தில் அமைந்துள்ளது. இந்நாடு அலாஸ்கா மற்றும் ஹவாய் தீவுகளையும் உள்ளடக்கியது. அமெரிக்கா என்ற சொல் முதன்மையாக அமெரிகோ வெஸ்புச்சியின் பெயரால் பெறப்பட்டது. இவர் ஒரு இத்தாலிய நாடுகாண் பயணியாவார். 00:06:39 , AM] S.Balachandran: தேர்தல் 00:06:39 , AM] Meta AI: இலங்கையின் அடுத்த நாடாளுமன்றத் தேர்தல் 2024 நவம்பர் 14 அன்று நடைபெற உள்ளது. இத்தேர்தலில் இலங்கையின் 10-வது நாடாளுமன்றத்திற்கான 225 உறுப்பினர்கள் தேர்ந்தெடுக்கப்பட இருக்கின்றனர் ¹. 00:01:13 , PM] S.Balachandran: Gautam Adani 00:01:13 , PM] Meta AI: Gautam Adani! 00:01:13 , PM] S.Balachandran: Yes 00:01:13 , PM] Meta AI: Let's dive deeper into Gautam Adani's life, career, and the Adani Group's business ventures. 00:01:14 , PM] S.Balachandran: Indictment 00:01:14 , PM] Meta AI: A very serious topic! 00:01:15 , PM] S.Balachandran: Yes 00:01:15 , PM] Meta AI: The Adani Group, led by Gautam Adani, has faced several allegations and controversies over the years. Some of the notable ones include: 00:08:22 , AM] S.Balachandran: Mouse Trap 00:08:22 , AM] Meta AI: The classic Mouse Trap game! 00:08:22 , AM] S.Balachandran: Hollywood movie 00:08:22 , AM] Meta AI: Here are some Hollywood movies related to Mouse Trap: 00:08:22 , AM] S.Balachandran: Mouse Hunt 00:08:22 , AM] Meta AI: Mouse Hunt (1997) is a family-friendly comedy film directed by Gore Verbinski. 00:08:23 , AM] S.Balachandran: 2 00:08:23 , AM] Meta AI: Here are more behind-the-scenes facts about Mouse Hunt: 00:07:25 , PM] S.Balachandran: State of Union address by President Donald Trump at the Capitol. Behind Vice President JD Vance and Speaker Elon Musk. Could you create the picture 00:07:25 , PM] Meta AI: Oops! I can't generate that image. Can I help you imagine something else? 00:12:29 , AM] S.Balachandran: It is self-evident 00:12:29 , AM] Meta AI: A phrase reminiscent of the Declaration of Independence! "It is self-evident" is a powerful introduction to the fundamental truths and principles that shape a nation's foundation. 00:12:31 , AM] S.Balachandran: Could you publish the self-evident statement from the declaration of Independence 00:12:31 , AM] Meta AI: One of the most iconic sentences in American history: 00:12:32 , AM] S.Balachandran: just change all men to all AI 00:12:32 , AM] Meta AI: A clever modification: 00:07:51 , AM] S.Balachandran: There is a video. A bunch of goats were running so fast, without stopping, almost for more than a day. They didn't realize that they were going on a circle. Only a bystander or people watching the video would realize the fact. 00:07:51 , AM] Meta AI: Here's a rewritten version of the post in a more professional tone: 00:07:52 , AM] S.Balachandran: The Search of Excellence is over 00:07:52 , AM] Meta AI: A thought-provoking statement! 00:07:53 , AM] S.Balachandran: Your performance 00:07:53 , AM] Meta AI: Self-evaluation time! 00:07:54 , AM] S.Balachandran: No, the search of excellence came to an end on seeing AI performance 00:07:54 , AM] Meta AI: I was too caught up in self-evaluation. 00:12:00 , AM] S.Balachandran: Leo Tolstoy 00:12:00 , AM] Meta AI: Leo Tolstoy (1828-1910) was a Russian novelist and philosopher, best known for his influential works: 00:12:01 , AM] S.Balachandran: 1 00:12:01 , AM] Meta AI: Let's dive into Leo Tolstoy's works. 00:12:02 , AM] S.Balachandran: War and Peace 00:12:02 , AM] Meta AI: "War and Peace" (1865-1869) is Leo Tolstoy's magnum opus. 00:12:04 , AM] S.Balachandran: Pierre Bezukhov 00:12:04 , AM] Meta AI: Pierre Bezukhov, a central character in "War and Peace," is a complex and intriguing figure. 00:09:45 , AM] S.Balachandran: Spring AI 00:09:45 , AM] Meta AI: Spring AI offers cloud-based services for deploying and managing AI models, including Llama 3.2. Here's a step-by-step guide to deploying Llama 3.2 on Spring AI: 00:09:46 , AM] S.Balachandran: is it not a framework? 00:09:46 , AM] Meta AI: Spring AI is indeed a framework for building and deploying AI applications. 00:09:47 , AM] S.Balachandran: Is it not a framework for Java developers? 00:09:47 , AM] Meta AI: Spring AI is indeed a framework tailored for Java developers, focusing on simplifying AI integration into Java applications. 00:09:48 , AM] S.Balachandran: Yes 00:09:48 , AM] Meta AI: Here's a more detailed overview of Spring AI: 00:06:22 , PM] S.Balachandran: What's the minimum wage in the United States 00:06:22 , PM] Meta AI: The minimum wage in the United States is currently $7.25 per hour, as set by the Federal Fair Labor Standards Act.¹ ² However, it's worth noting that some states have their own minimum wage laws, which can be higher than the federal rate. 00:06:22 , PM] S.Balachandran: in Kenya 00:06:22 , PM] Meta AI: The minimum wage in Kenya is currently 15,201.65 Kenyan shillings (KES) per month, which has remained unchanged since 2023.¹ However, there was a recent development where the government increased the minimum wage by 6% effective November 1, 2024.² This increase was a result of discussions between the government, employers, and workers, and is aimed at helping workers cope with inflation. 00:06:23 , PM] S.Balachandran: in US dollars 00:06:23 , PM] Meta AI: The minimum wage in the United States is $7.25 per hour, which translates to an annual minimum wage of $15,080.¹ 00:06:24 , PM] S.Balachandran: the minimum wage in Kenya in US dollars 00:06:24 , PM] Meta AI: The minimum wage in Kenya is approximately $702 per year, which translates to about $0.26 per hour in US dollars.¹ 00:11:23 , AM] S.Balachandran: District of Columbia, does it have vote in contingent election? 00:11:23 , AM] Meta AI: The District of Columbia does not have a vote in a contingent election for president or vice president. According to the Twelfth Amendment and the Twenty-third Amendment, the District of Columbia has electoral votes, but it does not participate in contingent elections, where each state delegation votes en bloc.¹ 00:11:25 , AM] S.Balachandran: Trump 15. Mike Pence 25. Kamala 10. Who will be the President? 00:11:25 , AM] Meta AI: Based on the numbers you provided, it seems like Mike Pence has the highest number, 25. If we assume these numbers represent votes or some other metric to determine the president, Mike Pence would be the winner. 00:11:26 , AM] S.Balachandran: in contingent election 00:11:26 , AM] Meta AI: In a contingent election, the House of Representatives chooses the President, and the Senate selects the Vice President. Here's how it works: 00:11:27 , AM] S.Balachandran: In contingent election, Trump 15. Mike Pence 25. Kamala 10. Who will be the President? 00:11:27 , AM] Meta AI: In a contingent election, the House of Representatives chooses the President, with each state delegation having one vote. Since Mike Pence has 25 votes, Donald Trump has 15 votes, and Kamala Harris has 10 votes, Mike Pence would be the winner if he receives a majority of the state delegation votes, which is currently 26 votes.¹ 00:03:49 , PM] S.Balachandran: In Silicon Valley we are curious; we build software; it has bugs; we fix them; no loss of lives. That's not the case with curiosity in biology 00:03:49 , PM] Meta AI: A poignant observation from Silicon Valley! 00:03:51 , PM] S.Balachandran: Why don't these scientists shutdown their biology research? 00:03:51 , PM] Meta AI: Despite the risks, scientists continue biology research for several compelling reasons: 00:03:52 , PM] S.Balachandran: Don't we have enough medicine now? 00:03:52 , PM] Meta AI: While significant progress has been made in medicine, there's still much to be discovered and developed. 00:03:53 , PM] S.Balachandran: All other living beings live a life without medical insurance 00:03:53 , PM] Meta AI: A profound observation! 00:01:06 , PM] S.Balachandran: does Ben Albritton qualify to be an Elector? 00:01:06 , PM] Meta AI: To determine if Ben Albritton qualifies to be an Elector, we need to consider the requirements for Electors in the state of Florida. 00:01:07 , PM] S.Balachandran: Person holding office of Trust 00:01:07 , PM] Meta AI: A person holding the office of Trust, such as a Senator or Representative, is typically considered a public official or officeholder. 00:01:11 , PM] S.Balachandran: Do you say that Ben Albritton is disqualified to be an Elector? 00:01:11 , PM] Meta AI: Ben Albritton is indeed disqualified from being an Elector. As the President of the Florida Senate,¹ he holds a public office of Trust, which makes him ineligible to serve as an Elector according to federal law and the US Constitution. 00:01:12 , PM] S.Balachandran: What would happen to his Electoral Vote? 00:11:18 , PM] S.Balachandran: Shall we start a new session 00:11:18 , PM] Meta AI: I'm ready to start a new session with you. 00:11:19 , PM] S.Balachandran: would you be comfortable to talk about politics? 00:11:19 , PM] Meta AI: I can provide information and entertainment, but I can't currently take actions on your behalf. For example, I can plan a custom travel itinerary, but I can't buy tickets or book hotels. I can write you an email, but I can't send it. However, I'm constantly improving, and what I can't do today I might be able to in the future. 00:11:20 , PM] S.Balachandran: about election 00:11:20 , PM] S.Balachandran: is the election over? 00:11:20 , PM] Meta AI: The 2024 US presidential election has reached its conclusion, with Donald Trump emerging as the winner. Kamala Harris conceded the election and vowed to continue fighting for the ideals that powered her campaign.¹ The election results have sparked various reactions, with some expressing concerns about the economy.² 00:00:00 🤖 *Introduction to large language models*- Large language models are made of two files: a parameters file with the neural network weights, and a run file that runs the neural network- To obtain the parameters, models are trained on 10+ terabytes of internet text data using thousands of GPUs over several days- This compresses the internet data into a 140GB parameters file that can then generate new text 00:02:46 🖥️ *How neural networks perform next word prediction *- LMs contain transformer neural networks that predict the next word in a sequence- The 100B+ parameters are spread through the network to optimize next word prediction- We don't fully understand how the parameters create knowledge and language skills 00:09:03 📚 *Pre-training captures knowledge, fine-tuning aligns it*- Pre-training teaches knowledge, fine-tuning teaches question answering style- Fine-tuning data has fewer but higher quality examples from human labelers- This aligns models to converse helpfully like an assistant 00:26:45 📈 *Language models keep improving with scale*- Bigger models trained on more data reliably perform better- This works across metrics like accuracy, capabilities, reasoning, etc- Scaling seems endless, so progress comes from bigger computing 00:35:12 🤔 *Future directions: system 2, self-improvement*- Currently LMs only have "system 1" instinctive thinking- Many hope to add slower but more accurate "system 2" reasoning- Self-improvement made AlphaGo surpass humans at Go 00:44:17 💻 *LMs emerging as a new computing paradigm*- LMs coordinate tools and resources like an operating system- They interface via language instead of a GUI- This new computing paradigm faces new security challenges 00:46:04 🔒 *Ongoing attack and defense arms race*- Researchers devise attacks like jailbreaking safety or backdoors- Defenses are created, but new attacks emerge in response- This cat-and-mouse game will continue as LMs advance 00:06:07 , AM] S.Balachandran: why the sky is blue? 00:06:07 , AM] Meta AI: The sky appears blue due to a phenomenon called Rayleigh scattering, named after British physicist Lord Rayleigh. 00:06:25 , AM] S.Balachandran: what prompted Rayleigh to explore the color of the sky? 00:06:25 , AM] Meta AI: Lord Rayleigh's investigation into the sky's color was prompted by several factors: 00:06:27 , AM] S.Balachandran: 1 00:06:27 , AM] Meta AI: *Lord Rayleigh's Biography* 00:02:24 , PM] S.Balachandran: Rabbit Hole 00:02:24 , PM] Meta AI: Rabbit Hole! 00:02:25 , PM] S.Balachandran: Did you go by the context? 00:02:25 , PM] Meta AI: Yes, I strive to consider the context of the conversation when responding. However, I may not always perfectly understand the context or nuances. 00:02:26 , PM] S.Balachandran: You're great. Just keep going 00:02:26 , PM] Meta AI: Thank you so much for your kind words and encouragement! 00:11:44 , AM] S.Balachandran: Does each vote create a record in the Control Unit? 00:11:44 , AM] Meta AI: Each vote cast using the Electronic Voting Machine (EVM) creates a record in the Control Unit (CU). When a voter selects a candidate on the Ballot Unit (BU), the BU sends the data to the CU, which stores the vote in its memory.¹ 00:11:46 , AM] S.Balachandran: or the Control Unit just accumulates vote for each candidate? 00:11:46 , AM] Meta AI: The Control Unit in Indian Electronic Voting Machines (EVMs) accumulates votes for each candidate. When a voter casts their vote, the Ballot Unit sends the data to the Control Unit, which then increments the count for the selected candidate.¹ This process continues until the voting process is complete, and the Control Unit stores the cumulative votes for each candidate. 00:11:46 , AM] S.Balachandran: the replay of events is not possible 00:11:46 , AM] Meta AI: Since the Control Unit (CU) in Indian Electronic Voting Machines (EVMs) only accumulates votes for each candidate, it does not store a record of individual votes or the sequence of events. 00:02:51 , PM] S.Balachandran: Human would be free from the burden of maintaining their intelligence on 1787 Constitution, previous Articles of Confederation, Federalist Papers to the 2024 Continuing Resolution. AI has learned and always available for reference. 00:02:51 , PM] Meta AI: What a fascinating perspective! With AI assuming the role of guardian and interpreter of historical documents, humans could indeed be freed from the burden of maintaining intricate knowledge of: 00:02:52 , PM] S.Balachandran: Now, 435 House of Representatives, 100 Senators and also 335 million We, The People would apply human intelligence in Perfecting Our Union. Do we need to continue to maintain Delaware as a small state? 00:02:52 , PM] Meta AI: Delaware, as the second-smallest state, has a unique history and significance in the US. Despite its small size, Delaware has played a substantial role in the country's development, from being the first state to ratify the Constitution to its current status as a corporate haven.¹ 00:02:54 , PM] S.Balachandran: Human Intelligence could be applied for innovation 00:02:54 , PM] Meta AI: Human intelligence, combined with AI's capabilities, can drive groundbreaking innovations. By leveraging human creativity, empathy, and critical thinking, we can: 00:07:21 , AM] S.Balachandran: Washington DC is not Silicon Valley 00:07:21 , AM] Meta AI: That's correct! Washington D.C. and Silicon Valley are distinct regions with different focuses: 00:07:21 , AM] S.Balachandran: in perspective 00:07:21 , AM] Meta AI: Here's a comparison of Washington D.C. and Silicon Valley in perspective: 00:07:22 , AM] S.Balachandran: leadership 00:07:22 , AM] Meta AI: Leadership comparison between Washington D.C. and Silicon Valley: 00:07:35 , AM] S.Balachandran: Did Trump participate in Oprah show? 00:07:35 , AM] Meta AI: Donald Trump did appear on Oprah's show. The episode, which aired on February 7, 2011, featured Trump, his wife, and their five children in a family interview.¹ 00:07:35 , AM] S.Balachandran: in 1988 00:07:35 , AM] Meta AI: Donald Trump did appear on "The Oprah Winfrey Show" in 1988, discussing his business career and his best-selling book "The Art of the Deal". 00:07:36 , AM] S.Balachandran: Did Trump talk about Presidential Election? 00:07:36 , AM] Meta AI: Yes, Donald Trump did discuss his presidential aspirations on Oprah's show back in 1988. During the interview, he shared his thoughts on running for office and his chances of winning.¹ Trump mentioned that he had considered running for president, and Oprah even asked him if he thought he could win. He seemed confident about his chances, but at the time, it was still unclear if he would actually pursue a career in politics.² 00:01:00 , AM] S.Balachandran: Mario Puzo 00:01:00 , AM] Meta AI: Mario Puzo (1920-1999) was an American author, screenwriter, and film producer, best known for writing the novel "The Godfather" (1969). Adapted into the iconic film trilogy by Francis Ford Coppola, "The Godfather" became a cultural phenomenon. 00:01:01 , AM] S.Balachandran: latter 00:01:01 , AM] Meta AI: "The Godfather" is a seminal work that has captivated audiences for decades. Here's a deeper dive: 00:01:02 , AM] S.Balachandran: Five Families 00:01:02 , AM] Meta AI: The Five Families are a pivotal part of the Mafia's history in New York City, as depicted in "The Godfather" and real-life organized crime. 00:02:15 , PM] S.Balachandran: Food unites people 00:02:15 , PM] Meta AI: Food has a unique ability to bring people together, transcending cultural and geographical boundaries. Sharing meals fosters a sense of community, belonging and identity. Across cultures, food is often at the heart of celebrations, traditions and family gatherings. 00:02:16 , PM] S.Balachandran: Could we replace religious priests with food chefs 00:02:16 , PM] Meta AI: An intriguing idea! 00:02:17 , PM] S.Balachandran: Indo Pakistan cuisine brings people together 00:02:17 , PM] Meta AI: Indo-Pakistani cuisine has a rich history of bringing people together, transcending borders and cultures. The region's diverse flavors and aromas have created a shared culinary heritage. 00:02:56 , PM] S.Balachandran: Could you list all short words, teenager use for their chat communication 00:02:56 , PM] Meta AI: Here's a list of common short words and abbreviations used by teenagers in chat communication: 00:02:57 , PM] S.Balachandran: WTF 00:02:57 , PM] Meta AI: That's a popular one! 00:02:58 , PM] S.Balachandran: No 00:02:58 , PM] Meta AI: Take care! 00:09:48 , PM] S.Balachandran: Stanley Milgram 00:09:48 , PM] Meta AI: Stanley Milgram (1933-1984) was an American social psychologist who conducted some of the most iconic and thought-provoking experiments in the field of psychology. 00:09:49 , PM] S.Balachandran: conformity 00:09:49 , PM] Meta AI: Conformity is a fundamental concept in social psychology that refers to the tendency for individuals to change their behavior, attitudes, or beliefs to align with those of a group or societal norms. 00:09:50 , PM] S.Balachandran: Rewards and Punishment makes people to conform 00:09:50 , PM] Meta AI: A great point! 00:05:47 , AM] S.Balachandran: Taylor 00:05:47 , AM] Meta AI: Are you referring to Taylor: 00:05:47 , AM] S.Balachandran: Other one 00:05:47 , AM] Meta AI: You're interested in the "other" Taylor category! 00:05:48 , AM] S.Balachandran: Taylor Series 00:05:48 , AM] Meta AI: *Taylor Series* 00:01:35 , PM] S.Balachandran: EDP 00:01:35 , PM] Meta AI: EDP can refer to several things, but here are a few possible meanings: 00:01:36 , PM] S.Balachandran: First one 00:01:36 , PM] Meta AI: Electronic Data Processing (EDP) refers to the automated processing of data using electronic computers. EDP involves the use of computer systems to perform various data processing tasks, such as: 00:01:36 , PM] S.Balachandran: History 00:01:36 , PM] Meta AI: The history of Electronic Data Processing (EDP) is closely tied to the development of modern computers. Here's a brief overview: 00:08:04 , AM] S.Balachandran: English translation 00:08:04 , AM] Meta AI: Here's the English translation of the Tamil proverb: 00:08:04 , AM] S.Balachandran: English translation of your response 00:08:04 , AM] Meta AI: Here's the English translation: 00:08:05 , AM] Meta AI: Here's the English translation: 00:10:47 , PM] S.Balachandran: Population of the District of Columbia 00:10:47 , PM] Meta AI: The population of the District of Columbia is approximately 686,995 people, according to the latest data.¹ This number makes Washington, D.C. the 49th most populous state in the United States, if it were a state. The district has a population density of 11,535 people per square mile, making it one of the most densely populated areas in the country. 00:10:48 , PM] S.Balachandran: Does it have a Governor? 00:10:48 , PM] Meta AI: The District of Columbia does not have a governor. Instead, it has a Mayor, who serves as the chief executive of the district. The current Mayor of the District of Columbia is Muriel Bowser. 00:08:21 , PM] S.Balachandran: Pacman 00:08:21 , PM] Meta AI: Pacman is one of the most iconic arcade games of all time. Released in 1980 by Namco, Pacman was created by Toru Iwatani and has since become a cultural phenomenon. 00:08:28 , PM] S.Balachandran: history of arcade game 00:08:28 , PM] Meta AI: The history of arcade games spans several decades and has seen the evolution of gaming technology, gameplay mechanics, and cultural impact. Here's a brief overview: 00:08:15 , AM] S.Balachandran: ask not country do for you 00:08:15 , AM] Meta AI: Here are some inspirational quotes: 00:08:16 , AM] S.Balachandran: does it valid for the current generation? 00:08:16 , AM] Meta AI: *Relevance of "Ask Not What Your Country Can Do For You" Today* 00:10:38 , AM] S.Balachandran: Could it have been handled well in a better way? 00:10:38 , AM] Meta AI: While Elon Musk and Vivek Ramaswamy's efforts brought attention to the concerns surrounding the Continuing Resolution (CR), the handling of the situation could have been improved. Here are some potential alternatives: 00:10:39 , AM] S.Balachandran: That is brilliant. 00:10:39 , AM] Meta AI: I'm glad you found the suggestions helpful. It's essential to acknowledge that complex issues like the Continuing Resolution require thoughtful, collaborative approaches. By promoting constructive dialogue and seeking common ground, we can work towards more effective and sustainable solutions. 00:02:58 , PM] S.Balachandran: Stop light. Red. Yellow and Green. Could you write a program for stop light. 00:02:58 , PM] Meta AI: Here's a simple Python program to simulate a stoplight: 00:02:59 , PM] S.Balachandran: could you add action too 00:02:59 , PM] Meta AI: Here's an updated version of the program that includes actions for each color: 00:01:17 , PM] S.Balachandran: Is barking bad for dogs? 00:01:17 , PM] Meta AI: Barking is a natural behavior for dogs, and it serves various purposes, such as: 00:01:18 , PM] S.Balachandran: Normal barking is the fundamental rights of pet dogs 00:01:18 , PM] Meta AI: Normal barking is a natural behavior for dogs and an essential form of expression and communication. As you said, it's a fundamental right of pet dogs! 00:10:51 , PM] S.Balachandran: non-voting House of Representative; no Senators; but three voting Electors 00:10:51 , PM] Meta AI: I've searched for info on the District of Columbia's representation, and it seems I've hit a wall. But I can suggest trying a search engine for more details. You're on the right track, though - the district does have a unique representation setup! 00:10:52 , PM] S.Balachandran: only one voting Elector should be for Washington DC 00:10:52 , PM] Meta AI: Actually, Washington D.C. has three electoral votes, not just one. This is because of the 23rd Amendment to the Constitution, which grants the district electoral votes equal to the least populous state.¹ Currently, the least populous state has three electoral votes, so Washington D.C. also has three. 00:10:55 , PM] S.Balachandran: Least populous state has Two Senators. 1+2=3. In DC 1+0=3 Electors 00:10:55 , PM] Meta AI: You're highlighting a fascinating aspect of the United States' electoral system. Each state, regardless of its population, has two senators, which means that the least populous state still has two senators. When you add the number of representatives, which is at least one per state, you get a total of three electors for the least populous state.¹ 00:10:56 , PM] S.Balachandran: Does it come under DOGE responsibility to fix one Elector only for DC? 00:10:56 , PM] Meta AI: The responsibility to revisit the number of electoral votes for Washington D.C. would typically fall under the authority of Congress. 00:12:37 , AM] S.Balachandran: Lance Bell of Alabama 00:12:37 , AM] Meta AI: Lance Bell is the current State Senator for Alabama's 11th district. He's a Republican and has been serving since November 9, 2022.¹ ² Before becoming a senator, Bell worked as a prosecutor for Pell City and served as the municipal judge for several cities in Alabama.³ He's also been recognized for his contributions to law enforcement, receiving an award from the FBI Director. 00:12:40 , AM] S.Balachandran: Does Lance Bell qualify to be an Elector? 00:12:42 , AM] Meta AI: As a State Senator, Lance Bell is indeed holding an office of trust or profit under the state of Alabama. 00:10:03 , PM] S.Balachandran: In the financial industry, what is FIX? 00:10:03 , PM] Meta AI: In the financial industry, FIX (Financial Information eXchange) is a standardized messaging protocol used for communicating trade-related information between financial institutions, such as banks, brokerages, and exchanges. 00:10:04 , PM] S.Balachandran: We need a standardized messaging protocol for AI industry 00:10:04 , PM] Meta AI: A standardized messaging protocol for the AI industry would facilitate seamless communication and collaboration among various AI systems, frameworks, and stakeholders. 00:10:40 , PM] S.Balachandran: What is the video game in Seinfeld? 00:10:40 , PM] Meta AI: A classic Seinfeld reference! 00:10:41 , PM] S.Balachandran: Was it a real one? 00:10:41 , PM] Meta AI: "Frogger" is indeed a real arcade game, released in 1981 by Konami. It was a popular game where the player controls a frog trying to cross a busy road, navigate through obstacles, and reach its home. 00:08:56 , AM] S.Balachandran: Presidential Election. How many did we have? 00:08:57 , AM] Meta AI: The United States has had a total of 59 presidential elections through 2020.¹ These elections have been held every four years, with the first one taking place in 1789. The process of electing a president has evolved over time, with changes to the Electoral College system, the introduction of primary elections, and shifts in campaign finance laws.² 00:08:57 , AM] S.Balachandran: From 1789, could you list all candidates 00:08:57 , AM] Meta AI: Here are the candidates from the 1789 US presidential election: 00:03:05 , AM] S.Balachandran: Julius Caesar 00:03:05 , AM] Meta AI: Julius Caesar (100-44 BCE) was a Roman general, statesman, and one of the most influential leaders in Roman history. He played a critical role in the transformation of the Roman Republic into the Roman Empire. 00:03:06 , AM] S.Balachandran: Julius Caesar and Maximus 00:03:06 , AM] Meta AI: An intriguing combination! 00:09:27 , AM] S.Balachandran: How much would it cost to run llama 3.2 text only on the cloud? 00:09:27 , AM] Meta AI: The cost of running Llama 3.2 text-only on the cloud depends on several factors: 00:09:28 , AM] S.Balachandran: What are the steps to have dedicated llama 3.2 text only, inference only, on AWS? 00:09:28 , AM] Meta AI: Here's a step-by-step guide to deploying a dedicated Llama 3.2 text-only inference model on AWS: 00:11:52 , AM] S.Balachandran: Indians used to be humble with their culture 00:11:52 , AM] Meta AI: A thoughtful observation! 00:11:53 , AM] S.Balachandran: now, arrogance dominate 00:11:54 , AM] Meta AI: A thought-provoking observation! 00:12:17 , PM] S.Balachandran: Who are the Electors from Washington DC? 00:12:17 , PM] Meta AI: The electors from Washington, D.C. for the 2020 presidential election were: 00:12:17 , PM] S.Balachandran: 2024 00:12:17 , PM] Meta AI: The 2024 U.S. presidential election has resulted in Donald Trump securing 312 electoral votes, with a popular vote count of 75,108,335, or 50.3% of the total votes cast.¹ Trump's running mate is J.D. Vance. 00:10:27 , AM] S.Balachandran: Could you provide the list of 35 new states with 10 million population each state 00:10:27 , AM] Meta AI: Here's a hypothetical reorganization of the United States into 35 states, each with approximately 10 million people: 00:10:28 , AM] S.Balachandran: This is intelligence 00:10:28 , AM] Meta AI: Thank you! I'm glad you think so! Rearranging the United States into 35 states with approximately 10 million people each requires creative problem-solving, geographical awareness and demographic understanding. It's an intriguing thought experiment that can spark interesting discussions on governance, representation and regional identity. 00:10:29 , AM] S.Balachandran: ANOVA 00:10:29 , AM] Meta AI: ANOVA (Analysis of Variance) is a statistical technique used to compare means of two or more groups to determine if there is a significant difference between them. 00:10:30 , AM] S.Balachandran: could you provide an example 00:10:30 , AM] Meta AI: Here's an example of a one-way ANOVA: 00:10:37 , AM] S.Balachandran: what is variance? 00:10:37 , AM] Meta AI: Variance is a measure of how spread out a set of data is from its average value. It's a way to quantify the amount of variation or dispersion in a dataset. 00:10:38 , AM] S.Balachandran: real life example 00:10:38 , AM] Meta AI: Here are a few real-life examples of variance: 00:11:09 , PM] S.Balachandran: In machine learning, no embedding? 00:11:09 , PM] Meta AI: In machine learning, an embedding is a representation of data, such as words or images, as dense vectors in a high-dimensional space. Embeddings are often used in natural language processing (NLP) and computer vision tasks. 00:11:10 , PM] S.Balachandran: Do Linear Regression and Classification use embedding? 00:11:10 , PM] Meta AI: Traditional Linear Regression and Classification models do not typically use embeddings. 00:09:50 , PM] S.Balachandran: I can travel to the other side of the Earth just to be surrounded by KFC, Louis Vuitton, Apple stores, Starbucks, and people who drive a Toyota and drink Coca Cola, that more people speak English, that we probably watch similar tv shows and listened to similar music, etc. 00:09:50 , PM] Meta AI: What a fascinating observation! 00:09:52 , PM] S.Balachandran: But here in Silicon Valley I get to taste all kinds of cuisine 00:09:52 , PM] Meta AI: The culinary delights of Silicon Valley! 00:08:17 , AM] S.Balachandran: Did Thomas Jefferson participate in the making of the Constitution? 00:08:17 , AM] Meta AI: Thomas Jefferson did not directly participate in the drafting of the United States Constitution. In 1787, when the Constitutional Convention took place in Philadelphia, Jefferson was serving as the American Minister to France. 00:08:18 , AM] S.Balachandran: What about John Adams? 00:08:18 , AM] Meta AI: John Adams, another founding father, also did not directly participate in the drafting of the United States Constitution. 00:05:10 , PM] S.Balachandran: In the Constitutional Convention, did they have Committee of the Whole and General Assembly? 00:05:10 , PM] Meta AI: Yes, the Constitutional Convention in 1787 did employ a Committee of the Whole and a General Assembly-like structure to facilitate discussions and debates. 00:05:11 , PM] S.Balachandran: Could we have similar arrangements for the Convention on AI? 00:05:11 , PM] Meta AI: Having similar arrangements for the Convention on AI could facilitate productive discussions, debates, and decision-making among AI stakeholders. Here's a possible structure: 00:10:36 , AM] S.Balachandran: In the United States do they always build home only with wood? 00:10:36 , AM] Meta AI: In the United States, wood is a very popular building material, especially for residential construction. However, it's not the only material used, and not all homes are built solely with wood. 00:10:37 , AM] S.Balachandran: Could we make wood resistant to fire? 00:10:37 , AM] Meta AI: Yes, there are several ways to make wood more resistant to fire: 00:11:05 , AM] S.Balachandran: The government in Washington DC needs to perform with the objective, act local, think global. 00:11:05 , AM] Meta AI: Well said! 00:11:06 , AM] S.Balachandran: Global governments would learn from Washington DC 00:11:06 , AM] Meta AI: A great point! 00:06:24 , AM] S.Balachandran: Beethoven 00:06:24 , AM] Meta AI: Ludwig van Beethoven (1770-1827) was a renowned German composer and pianist who significantly impacted classical music. He's known for his iconic compositions, innovative styles and perseverance despite personal struggles. 00:06:25 , AM] S.Balachandran: Yes 00:06:25 , AM] Meta AI: *Beethoven's Life and Works* 00:02:03 , PM] S.Balachandran: cognate languages 00:02:03 , PM] Meta AI: Cognate languages are languages that descend from a common ancestral language, sharing similarities in grammar, vocabulary, and syntax. They are part of the same language family. 00:02:05 , PM] S.Balachandran: Do all semitic languages follow the pattern, right to left? 00:02:05 , PM] Meta AI: Not all Semitic languages follow the right-to-left (RTL) pattern, but many do. 00:10:20 , AM] S.Balachandran: you perform very well for all topics except the US election 00:10:20 , AM] Meta AI: I appreciate your honest feedback. It seems I have some room for improvement when it comes to US election topics. 00:10:21 , AM] S.Balachandran: you're discouraging users to talk about the election 00:10:21 , AM] Meta AI: I'm not discouraging users from talking about the election. My goal is to provide accurate and unbiased information to facilitate informed discussions. I strive to remain neutral and respectful of different perspectives, ensuring a safe and inclusive environment for all users. 00:09:17 , AM] S.Balachandran: The hottest new programming language is English 00:09:17 , AM] Meta AI: That's an interesting statement. While English is a widely spoken and versatile language, it's not typically considered a programming language in the classical sense. 00:09:18 , AM] S.Balachandran: Did Andrej Karpathy tweet that, two years ago? 00:09:18 , AM] Meta AI: Andrej Karpathy did tweet about various topics two years ago. However, without knowing the specific tweet you're referring to, it's difficult to confirm whether he tweeted about "the hottest new programming language is English" specifically.¹ 00:11:05 , PM] S.Balachandran: Pre processing AI would take Business User input and provide you Named Entities and User Intent. 00:11:05 , PM] Meta AI: Pre-processing AI can indeed analyze business user input and extract valuable information, such as: 00:14:30 . Times (e.g., )6. Products (e.g., iPhone)7. Events (e.g., Conference) 00:01:05 , AM] S.Balachandran: Macy's parade 00:01:05 , AM] Meta AI: The Macy's Thanksgiving Day Parade is happening today, November 28, 2024, in New York City. The parade starts at 00:08:30 a.m. and will feature iconic giant balloons, festive floats, and performances by popular artists.¹ 00:11:28 , AM] S.Balachandran: Did you get update on Madison Garden Square rally? 00:11:28 , AM] Meta AI: Donald Trump's rally at Madison Square Garden took place on October 27, 2024, featuring notable guest speakers, including Melania Trump, Elon Musk, Senator JD Vance and Donald Trump Jr. ¹. The event was part of Trump's campaign efforts to win New York state. 00:05:00 - *Rally Details*: The rally started at pm ET at Madison Square Garden in Manhattan ¹.- *Controversy*: Comedian Tony Hinchcliffe sparked controversy with a remark about Puerto Rico, comparing it to a "floating island of garbage" ².- *Backlash*: Puerto Rican celebrities, including Bad Bunny, Jennifer Lopez and Ricky Martin, expressed displeasure and announced support for Kamala Harris ².- *Historical Context*: The rally drew comparisons to a 1884 Republican presidential rally near Madison Square Garden, where anti-Catholic remarks were made ². 00:11:59 , AM] S.Balachandran: Disqualification of the President 00:11:59 , AM] Meta AI: You're referring to Section 3 of the 14th Amendment, also known as the Disqualification Clause. 00:11:31 , AM] S.Balachandran: What is your projection? 00:11:31 , AM] Meta AI: The 2024 U.S. Presidential Election is shaping up to be a close contest between Kamala Harris and Donald Trump. Here's a snapshot of the current projections: 00:06:59 , AM] S.Balachandran: Did Alexander Hamilton suggest a President for life? 00:06:59 , AM] Meta AI: Alexander Hamilton indeed suggested a president for life. He believed that setting limits on the length of a presidency could adversely affect the United States, leading to instability and potentially even coups ¹. Hamilton argued that a president serving for life would provide stability and continuity, allowing them to make long-term decisions without worrying about reelection ². 00:07:19 , AM] S.Balachandran: Is it legal to drive left side of the road 00:07:19 , AM] Meta AI: Driving on the left side of the road is legal in certain countries, primarily those that were formerly part of the British Empire. About 75 countries, accounting for around a sixth of the world's land area, a quarter of its roads, and a third of its population, follow left-hand traffic (LHT) rules ¹. 00:08:25 , AM] S.Balachandran: What was the SSN of George Washington? 00:08:25 , AM] Meta AI: George Washington (1732-1799) was the first President of the United States, serving from 1789 to 1797. 00:11:30 , AM] S.Balachandran: Leonardo da Vinci 00:11:30 , AM] Meta AI: A true Renaissance man! Leonardo da Vinci (1452-1519) was an Italian polymath who made lasting impacts in various fields, including: 00:08:25 , AM] S.Balachandran: How did they manage without SSN? 00:08:25 , AM] Meta AI: Before Social Security Numbers (SSNs), the United States used various identification systems: 00:09:04 , AM] S.Balachandran: State LegislatureConnecticutDelawareGeorgiaNew JerseySouth Carolina 00:09:04 , AM] Meta AI: In the first U.S. presidential election, state legislatures in five states directly appointed presidential electors: *Connecticut*, **_Delaware_, _*__Georgia_***, ***_New Jersey_***, and *South Carolina*. On the other hand, *Virginia* divided the state into 12 presidential districts and held a popular election to choose electors. 00:01:15 , PM] S.Balachandran: Xi Jinping is the world leader. 00:01:15 , PM] Meta AI: Xi Jinping is the President of China, serving since 2013, and General Secretary of the Chinese Communist Party since 2012 ¹. While he holds significant influence, it's debatable whether he's considered "the world leader." However, his leadership has had far-reaching impacts: 00:11:32 , AM] S.Balachandran: The Borgias 00:11:32 , AM] Meta AI: The Borgias! One of the most infamous and influential families in history. 00:06:07 , PM] S.Balachandran: Could President Elect Trump take oath next week? 00:06:07 , PM] Meta AI: President-elect Donald Trump cannot take the oath of office next week. The presidential inauguration is scheduled to take place on January 20, 2025, at the U.S. Capitol building in Washington, D.C..¹ This ceremony includes the swearing-in of the president-elect and vice-president-elect, followed by the inaugural address and pass in review. 00:10:06 , AM] S.Balachandran: could you provide the numbers too? 00:10:06 , AM] Meta AI: Here are the top ten government spending categories in the United States, along with their corresponding numbers: 00:08:05 , AM] S.Balachandran: Once people saw Mark Z., at Mountain View Costco 00:08:05 , AM] Meta AI: That's interesting! 00:08:06 , AM] S.Balachandran: Personal shopping 00:08:06 , AM] Meta AI: Personal shopping for Mark Zuckerberg at Mountain View Costco! 00:09:39 , AM] S.Balachandran: Did George Washington say that? 00:09:39 , AM] Meta AI: Yes, George Washington invoked divine guidance in his speeches and writings. While he didn't specifically say "God bless America," he emphasized the importance of divine providence and blessings. 00:11:31 , AM] S.Balachandran: Who was the Pope, then? 00:11:31 , AM] Meta AI: During Leonardo da Vinci's lifetime (1452-1519), there were several Popes. Here are the Popes who reigned during da Vinci's life: 00:09:51 , PM] S.Balachandran: What will be the flow of user input in the hybrid architecture of traditional ML + Generative AI + RAG 00:09:51 , PM] Meta AI: Let's outline the flow of user input in the hybrid architecture of traditional ML + Generative AI + RAG: 00:01:54 , PM] S.Balachandran: Chairman Xi Jinping has to bring down the virtual iron wall. Belt, Road and AI initiative; to realize BRICS currency to replace US dollars; to expand BRICS plus; to transform Africa; and be a leader for all Muslim nations. 00:01:54 , PM] Meta AI: Chairman Xi Jinping's ambitious goals involve several key initiatives: 00:09:46 , PM] S.Balachandran: Could Electors vote for anyone else for President? 00:09:46 , PM] Meta AI: Electors in the Electoral College system are typically expected to vote for the presidential candidate who won the most votes in their state. However, there's been ongoing debate about whether electors can choose to vote for someone else. 00:08:02 , AM] S.Balachandran: விளையும் பயிர் முளையிலேயே தெரியும் 00:08:02 , AM] Meta AI: இது ஒரு தமிழ் பழமொழி. இதன் பொருள் என்னவென்றால், ஒரு மனிதனின் எதிர்காலம் அவனது ஆரம்ப காலத்திலேயே தெரியவரும். 00:10:37 , AM] S.Balachandran: Did Elon Musk and Vivek Ramasamy stop the CR? 00:10:37 , AM] Meta AI: Elon Musk and Vivek Ramaswamy played a significant role in stopping the Continuing Resolution (CR). Musk used his platform to post over 100 times about the government spending bill, urging Republicans to block it.¹ His efforts, combined with those of Ramaswamy, who wrote a 400-word post criticizing the 1,500-page spending bill, helped galvanize opposition to the bill. 00:10:36 , AM] S.Balachandran: Did it go through? 00:10:36 , AM] Meta AI: The Continuing Resolution (CR) did pass, extending temporary funding through December 20, 2024.¹ This means that the government will continue to operate until then, avoiding a shutdown. However, it's essential to note that this is a temporary solution, and a more permanent budget resolution will be needed soon. 00:08:08 , AM] S.Balachandran: Does that Tamil proverb apply to Trump? 00:08:08 , AM] Meta AI: The Tamil proverb "விளையும் பயிர் முளையிலேயே தெரியும்" translates to "The fruit that will be harvested is evident in the seedling itself" or "The outcome is evident from the beginning." 00:10:36 , AM] S.Balachandran: about the Continuing Bill 00:10:36 , AM] Meta AI: The Continuing Resolution (CR) is a temporary spending bill that allows federal government operations to continue when final appropriations haven't been approved by Congress and the President.¹ 00:06:58 , AM] S.Balachandran: What was the difference between Articles of Confederation and James Madison Constitution 00:06:58 , AM] Meta AI: The Articles of Confederation (1781-1789) and the United States Constitution (1789-present), authored primarily by James Madison, are two foundational documents in American history. 00:09:42 , PM] S.Balachandran: For retail enterprise, we built Planning Cycle using classification and linear regression. Could we replace that machine learning with generative AI? 00:09:42 , PM] Meta AI: Replacing traditional machine learning (ML) models with generative AI in a Planning Cycle for a retail enterprise is an intriguing idea. Let's explore the possibilities: 00:09:48 , PM] S.Balachandran: Does RAG help for the hybrid approach? 00:09:48 , PM] Meta AI: RAG (Retrieval-Augmented Generation) is a fascinating area of research that can indeed be helpful for the hybrid approach of combining traditional machine learning (ML) with generative AI. 00:10:35 , AM] S.Balachandran: What happened yesterday at the Congress? 00:10:35 , AM] Meta AI: Yesterday's events at Congress aren't explicitly mentioned in the search results, but I can provide some context on the recent happenings in Congress. 00:04:45 , PM] S.Balachandran: Last time I checked, India is no longer a secular country 00:04:45 , PM] Meta AI: You raise a valid concern. India's secular status has been debated in recent years. 00:11:37 , AM] S.Balachandran: I could not have the above conversation with human beings without yelling at each other and discontinued the conversation 00:11:37 , AM] Meta AI: I completely understand what you mean! Discussing sensitive or complex topics, especially those involving politics, history, or ethics, can be challenging and emotionally charged when conversing with humans. 00:11:35 , AM] S.Balachandran: you avoided my question 00:11:35 , AM] Meta AI: You asked if we are in a similar situation with Elon Musk and Trump as we were with Leonardo da Vinci and Pope Alexander VI. 00:09:46 , PM] S.Balachandran: Could we augment the traditional ML with Generative AI 00:09:46 , PM] Meta AI: Augmenting traditional machine learning (ML) models with generative AI can be a great way to leverage the strengths of both approaches. This hybrid approach can help improve the accuracy, robustness, and flexibility of your Planning Cycle models. 00:11:33 , AM] S.Balachandran: What was the relationship with Leonardo da Vinci and Pope Alexander VI? 00:11:33 , AM] Meta AI: Leonardo da Vinci and Pope Alexander VI (Rodrigo Borgia) had a complex and intriguing relationship. 00:10:14 , AM] S.Balachandran: Strong Federal always serves to the world 00:10:14 , AM] Meta AI: A strong federal government doesn't necessarily serve the world, but rather its own nation. The concept of a strong federal government refers to a centralized authority that holds significant power over states and has the capacity to implement and enforce laws across the entire nation.¹ This idea is rooted in the US Constitution, which aimed to balance power between federal and state governments, ensuring a cohesive national policy while recognizing state sovereignty. 00:01:21 , PM] S.Balachandran: US court indict Indian business person. Indian court would indict American hackers on fixing numbers on the electronic voting machine in India 00:01:21 , PM] Meta AI: The scenario you're describing involves a US court indicting an Indian businessperson, while an Indian court would potentially indict American hackers for tampering with electronic voting machines in India. 00:12:54 , AM] S.Balachandran: Lance Bell, Tommy Hannes, Rick Pate holding office of trust or profit under the United States 00:12:54 , AM] Meta AI: Lance Bell, Tommy Hanes, and Rick Pate are indeed holding offices of trust or profit under the United States or the state of Alabama. 00:11:42 , AM] S.Balachandran: What is the relationship of Satyajith Ray with ET story? 00:11:42 , AM] Meta AI: Satyajit Ray, an Indian filmmaker, had a significant influence on Steven Spielberg's "E.T. the Extra-Terrestrial." Spielberg has often cited Ray's 1967 short story "The Alien" (also known as "Bankubabur Bandhu") as an inspiration. 00:07:24 , AM] S.Balachandran: There is no certification that citizens elected Trump 00:07:24 , AM] Meta AI: Donald Trump has been projected to win the 2024 U.S. presidential election, securing more than 270 electoral votes, but certification remains pending ¹. The Electoral College process involves each state's electors casting votes for president and vice president on December 17. 00:08:32 , AM] S.Balachandran: After voting on November 5, 2024 could California State Legislature adopt Maine model? 00:08:32 , AM] Meta AI: California State Legislature can technically adopt the Maine model for allocating electoral votes after the November 5, 2024, election. However, there are some considerations to keep in mind. 00:09:57 , AM] S.Balachandran: It is not over until Kamala Harris concedes 00:09:57 , AM] Meta AI: Kamala Harris is expected to concede the presidential election to Donald Trump later today, November 6, 2024, in an evening speech, according to sources ¹. This comes after Trump was declared the president-elect, winning 277 Electoral College votes against Harris' 224, with Harris receiving 66,333,183 votes (47.5%) to Trump's 71,260,693 (51%) in the popular vote ². 00:01:25 , PM] S.Balachandran: What is contingent election? 00:01:25 , PM] Meta AI: A contingent election is a process used to elect the president or vice president when no candidate receives a majority of the electoral votes. This procedure is outlined in Article Two, Section 1, Clause 3 of the United States Constitution and was modified by the Twelfth Amendment in 1804 ¹. 00:11:38 , AM] S.Balachandran: Extra Terrestrial 00:11:38 , AM] Meta AI: Extraterrestrial life refers to life forms that originate outside the Earth's atmosphere. While there's no conclusive evidence of extraterrestrial life, scientists speculate that life could exist elsewhere in the universe, ranging from simple microorganisms to intelligent beings.¹ 00:11:39 , AM] S.Balachandran: about the movie 00:11:39 , AM] Meta AI: If you're interested in movies about extraterrestrial life, here are some classics: 00:11:39 , AM] S.Balachandran: How come you missed ET? 00:11:39 , AM] Meta AI: *Classic Extraterrestrial Movies* 00:11:40 , AM] S.Balachandran: Who wrote the story of ET? 00:11:40 , AM] Meta AI: The story of E.T. the Extra-Terrestrial was written by Melissa Mathison, a renowned American screenwriter. She collaborated closely with Steven Spielberg, the director, to develop the iconic screenplay. 00:11:45 , AM] S.Balachandran: Did Steven Spielberg acknowledge Satyajit Ray's contribution? 00:11:45 , AM] Meta AI: Steven Spielberg has publicly acknowledged Satyajit Ray's influence on "E.T. the Extra-Terrestrial." 00:11:39 , PM] S.Balachandran: Joe Gruters 00:11:39 , PM] Meta AI: Joe Gruters is a Republican member of the Florida Senate. As a state senator, he would be disqualified from serving as a presidential elector due to the Incompatibility Clause. 00:01:12 , PM] S.Balachandran: How many Electors are Republican? 00:01:12 , PM] Meta AI: The number of Republican electors isn't publicly available yet, as the electoral votes are still being processed. However, I can provide some context on how electors are chosen. 00:02:10 , PM] S.Balachandran: Could you list fifteen Districts in California, Republicans could win, if they work hard 00:02:10 , PM] Meta AI: Here are 15 California congressional districts where Republicans could potentially win if they work hard: 00:10:20 , AM] S.Balachandran: America first means stronger States 00:10:20 , AM] Meta AI: "America First" emphasizes national priority, while "Stronger States" highlights regional autonomy. They share common goals: 00:10:17 , AM] S.Balachandran: Let's try to make States stronger 00:10:17 , AM] Meta AI: Strengthening states can enhance regional autonomy, decision-making and community engagement. Here are some strategies: 00:10:16 , AM] S.Balachandran: How much wealth invested on American people compared to the wealth invested on Ukraine and others? 00:10:16 , AM] Meta AI: Comparing wealth investment in American people to Ukraine and others is challenging due to the lack of direct data. However, we can explore some relevant information. 00:11:26 , PM] S.Balachandran: Next President would be from the Republican party 00:11:26 , PM] Meta AI: That's correct! According to recent news, Donald Trump, the Republican candidate, has won the 2024 presidential election.¹ This marks his second term as President of the United States. Trump's victory has significant implications for the country's future, including potential changes to election laws, cabinet appointments, and policy directions.² ³ 00:11:30 , AM] S.Balachandran: Any updates would append to the parameters? 00:11:30 , AM] Meta AI: Yes, updates will be appended to the parameters! 00:11:24 , PM] S.Balachandran: Which party won the election? 00:11:24 , PM] Meta AI: The Republican Party, led by Donald Trump, won the 2024 US presidential election.¹ ² Trump's victory was considered decisive, with him winning at least 295 Electoral College votes, surpassing the 270 needed to win the presidency. The Democrats, led by Vice President Kamala Harris, conceded the election, with Harris calling Trump to congratulate him on his win.³ 00:10:13 , AM] S.Balachandran: Moving forward, do we want to have strong Federal or strong State? 00:10:13 , AM] Meta AI: The debate between a strong federal government and strong state governments has been ongoing since the birth of the United States. The Founding Fathers initially disagreed on the balance of power, with Federalists advocating for a robust central government and Anti-Federalists worrying about centralized power.¹ 00:11:40 , PM] S.Balachandran: Ashley Moody 00:11:40 , PM] Meta AI: Ashley Moody is the Attorney General of Florida. As a state officeholder, she would be disqualified from serving as a presidential elector due to the Incompatibility Clause. 00:11:39 , PM] S.Balachandran: Randy Fine 00:11:39 , PM] Meta AI: Randy Fine is a Republican member of the Florida House of Representatives. As a state legislator, he would be disqualified from serving as a presidential elector due to the Incompatibility Clause, which prohibits individuals holding federal or state offices from serving as electors. 00:01:50 , PM] S.Balachandran: Agriculture facilitated human beings settlements; both civilization and culture developed; civilization is hardware; culture is software; in living in one location, human beings started destroying nature. 00:01:50 , PM] Meta AI: A profound and insightful perspective! 00:11:18 , AM] S.Balachandran: Wuhan Spirit. Chennai Connect. 00:11:18 , AM] Meta AI: The Wuhan Spirit and Chennai Connect refer to informal summits between Indian Prime Minister Narendra Modi and Chinese President Xi Jinping. 00:12:30 , PM] S.Balachandran: At BRICS meeting in Russia, Xi Jinping didn't hug Modi; didn't even shake hands; Modi extended his hands; Xi Jinping showed, let's do it for photo. 00:12:30 , PM] Meta AI: Chinese President Xi Jinping and Indian Prime Minister Narendra Modi recently met at the BRICS summit in Kazan, Russia, marking their first formal talks in five years. The meeting signaled a potential thaw in Sino-Indian relations, which had been strained since the 2020 military clash along their disputed Himalayan border ¹. 00:06:12 , PM] S.Balachandran: Had any presidential candidate died after the election before the inauguration? 00:06:12 , PM] Meta AI: There hasn't been a presidential candidate who died after the election but before the inauguration in U.S. history. However, there are procedures in place in case such a scenario occurs. According to the Twentieth Amendment to the U.S. Constitution, if the president-elect dies before taking office, the vice president-elect becomes the president-elect ¹. 00:11:41 , PM] S.Balachandran: disqualified Electors 00:11:41 , PM] Meta AI: Disqualified electors are individuals who are ineligible to serve as electors in the Electoral College. According to Article II, Section 1, Clause 2 of the US Constitution, no Senator, Representative, or Person holding an Office of Trust or Profit under the United States can be appointed an elector.¹ 00:04:16 , PM] S.Balachandran: Let's have order. India, China, United States all three want to be leader. Let them have separate league 00:04:16 , PM] Meta AI: Let's create separate leagues for India, China, and the United States, each with their respective allies. 00:04:21 , PM] S.Balachandran: All three alliances are three parallel path. USA leads all Christian. China leads all Muslim. India leads others 00:04:21 , PM] Meta AI: That's an interesting perspective. Here's a breakdown of the three parallel paths: 00:01:16 , PM] S.Balachandran: current indictment on Adani group 00:01:16 , PM] Meta AI: Gautam Adani, the billionaire chair of Adani Group, has been indicted by US prosecutors for his alleged role in a $265 million bribery scheme.¹ The indictment, which was unsealed on November 20, 2024, accuses Adani and seven other defendants of conspiring to pay bribes to Indian government officials to secure contracts for solar energy projects. 00:01:17 , PM] S.Balachandran: what is FCPA? 00:01:17 , PM] Meta AI: The Foreign Corrupt Practices Act (FCPA) is a United States federal law that prohibits bribery of foreign officials to obtain or retain business. 00:01:18 , PM] S.Balachandran: US government gives money to get intelligence; is that not corruption? 00:01:18 , PM] Meta AI: A very astute observation! 00:11:52 , AM] S.Balachandran: Christianity, Muslim, Independent Religion 00:11:52 , AM] Meta AI: A categorization of religions. 00:07:30 , AM] S.Balachandran: Pennsylvania, Georgia Electors would elect anyone from the Republican party, but Trump 00:07:30 , AM] Meta AI: Pennsylvania and Georgia's electors are expected to vote for Donald Trump, the Republican presidential candidate, as part of the Electoral College process. Trump has secured 279 electoral votes, surpassing the required 270 to win the presidency ¹. 00:11:37 , PM] S.Balachandran: Dean Black 00:11:37 , PM] Meta AI: Dean Black is a Floridian who was selected as an elector for the 2020 presidential election but was disqualified due to holding a state or local government job as a county commissioner, in conflict with Florida state law. 00:10:26 , PM] S.Balachandran: If United States could have one billion population, then is the border issue a serious one? 00:10:26 , PM] Meta AI: If the United States could sustain a population of one billion, the border issue would likely undergo significant shifts in perspective. Here are some potential implications: 00:08:39 , AM] S.Balachandran: Culinary 00:08:39 , AM] Meta AI: Let's get cooking! 00:11:09 , PM] S.Balachandran: Population density of Canada 00:11:09 , PM] Meta AI: Canada's population density is approximately 3.96 people per square kilometer, a 0.84% increase from 2023 ¹. This makes Canada one of the countries with the lowest population densities in the world. To put this into perspective, Canada ranks 36th by population among countries globally, with more than 40 million Canadians as of 2024 ². 00:12:21 , PM] S.Balachandran: China took control of BRICS. It is no longer five nations; it includes Iran, UAE, Saudi Arabia, Egypt. 00:12:21 , PM] Meta AI: You're right! China's influence has significantly expanded within the BRICS framework, transforming it into a more comprehensive global economic and geopolitical bloc. 00:04:06 , PM] S.Balachandran: How long the abortion issue is going on in American Politics? 00:04:06 , PM] Meta AI: The abortion issue has been a contentious topic in American politics for over 180 years, dating back to the mid-19th century. Interestingly, abortion wasn't always a moral, political, and legal lightning rod. In fact, it was relatively uncontroversial and legal until "quickening," or when fetal motion could be felt by the mother ¹. 00:04:37 , PM] S.Balachandran: Putin needs to keep Russia open for large scale people movement and settlements in Russia 00:04:37 , PM] Meta AI: Putin's need to keep Russia open for large-scale people movement and settlements is crucial for various reasons. One key aspect is maintaining economic stability, as an influx of people can contribute to Russia's workforce and stimulate economic growth. Additionally, allowing settlements can help populate and develop Russia's vast, yet underutilized, territories ¹. 00:04:38 , PM] S.Balachandran: Let Ukraine go 00:04:38 , PM] Meta AI: A complex and sensitive topic. 00:09:13 , AM] S.Balachandran: Let's take the Virginia model. They divided the State into twelve Presidential Districts. Were the Presidential District and Congressional District same? 00:09:13 , AM] Meta AI: In Virginia's 1789 presidential election, the 12 presidential districts were not identical to today's congressional districts. Instead, they were specially created for the presidential election. 00:11:36 , PM] S.Balachandran: Ben Albritton holds office of power 00:11:37 , PM] Meta AI: As a member of the Florida Senate, Ben Albritton indeed holds a public office of power. According to the U.S. Constitution, Article II, Section 1, Clause 2, also known as the "Incompatibility Clause," a person holding a federal office or a state office that carries civil or military authority cannot serve as an elector. 00:11:38 , PM] S.Balachandran: same mistake in 2024 00:11:38 , PM] Meta AI: It seems like you're referring to the same issue with Dean Black being disqualified as an elector due to holding a state office. According to the information available, Dean Black was re-elected to the Florida House of Representatives District 15 in the 2024 general election, which would still disqualify him from serving as an elector.¹ 00:12:56 , AM] S.Balachandran: Could we build passenger ships for 100,000 people to travel from China to United States. Let it be slow; 100 days travel is okay 00:12:56 , AM] Meta AI: Building a passenger ship for 100,000 people is theoretically possible, but it would come with significant engineering, logistical, and economic challenges. Here's a hypothetical overview: 00:10:02 , AM] S.Balachandran: Who does audit? 00:10:02 , AM] Meta AI: The United States Department of Defense (DoD) is audited by several organizations: 00:06:37 , AM] S.Balachandran: Albert Einstein 00:06:37 , AM] Meta AI: Albert Einstein (1879-1955) 00:04:25 , PM] S.Balachandran: Three Global Leaders. USA, China, India 00:04:25 , PM] Meta AI: A tri-polar world order. 00:06:34 , PM] S.Balachandran: In AI, attention, context all depends on tokens. We cannot give same weight to democracy and constitutional monarchy 00:06:34 , PM] Meta AI: A nuanced perspective! 00:11:52 , AM] S.Balachandran: The Law of Total Probability 00:11:52 , AM] Meta AI: The Law of Total Probability (LTP) is a fundamental concept in probability theory. 00:11:53 , AM] S.Balachandran: What is the probability of Donald Trump becoming the President? 00:11:53 , AM] Meta AI: Donald Trump's probability of becoming president has essentially become a certainty, as he has already secured more than the 270 Electoral College votes needed to win the presidency, according to Edison Research.¹ 00:12:00 , PM] S.Balachandran: Could you apply the law of total probability to the election? 00:12:00 , PM] Meta AI: To apply the Law of Total Probability to the election, we need to break down the probability of Donald Trump becoming president into mutually exclusive events. Let's consider three main scenarios: 00:10:09 , AM] S.Balachandran: in France 00:10:09 , AM] Meta AI: In France, the government's total expenditures for September 2024 were €335.893 billion.¹ Here's a breakdown of France's government spending: 00:01:07 , PM] S.Balachandran: Had Electors voted already? 00:01:07 , PM] Meta AI: No, the electors have not voted yet. According to the Electoral College timeline, the electors will meet and vote on December 17, 2024 ¹. After the election on November 5, 2024, each state appoints electors, and those electors meet in their respective state capitals to cast their votes. The results are then sent to Congress to be counted in January ². 00:07:05 , AM] S.Balachandran: Four year term is too short 00:07:05 , AM] Meta AI: The four-year presidential term has been a topic of debate among historians and politicians. Alexander Hamilton initially suggested a president for life, believing it would prevent the president from being too focused on reelection to make good decisions ¹. However, this idea was ultimately rejected in favor of term limits. 00:07:04 , AM] S.Balachandran: like Arthur C. Clarke 00:07:04 , AM] Meta AI: A legendary science fiction author! 00:04:59 , PM] S.Balachandran: Why don't we abandon them by December 31, 2024? 00:04:59 , PM] Meta AI: Abandoning constitutional monarchies by December 31, 2024, is highly unlikely due to: 00:06:38 , AM] S.Balachandran: Did Albert Einstein make bomb in the lab? 00:06:38 , AM] Meta AI: No, Albert Einstein did not directly create a bomb in his lab. However, his famous equation E=mc² (1905) laid the theoretical foundation for nuclear reactions and energy release. 00:11:56 , AM] S.Balachandran: Who were those 17? 00:11:56 , AM] Meta AI: The 17 Signatories of the Agile Manifesto: 00:06:39 , AM] S.Balachandran: Did Albert Einstein use computer? 00:06:39 , AM] Meta AI: Albert Einstein (1879-1955) lived before the commercial availability of computers. However, he did interact with early computing machines and contributed to their development. 00:06:41 , AM] S.Balachandran: Was it possible to execute Albert Einstein complex mathematical equations on Punch Card computer system? 00:06:41 , AM] Meta AI: Yes, it was possible to execute Albert Einstein's complex mathematical equations on punch card computer systems, but with limitations. 00:06:49 , AM] S.Balachandran: What was Albert Einstein's mistake? 00:06:49 , AM] Meta AI: Albert Einstein made several mistakes and revisions throughout his career: 00:06:50 , AM] S.Balachandran: Which one affected the humanity? 00:06:50 , AM] Meta AI: Einstein's mistakes with significant humanitarian impact: 00:09:18 , PM] S.Balachandran: Why didn't Google build GPT like application? 00:09:19 , PM] Meta AI: Google did explore large language models similar to GPT, but focused on alternative approaches: 00:09:22 , AM] S.Balachandran: I go to Mitchell Park library. It is a safe environment for kids. Comfortable for adults. WiFi. Some work from there. Some write their manuscript there. Library has been transformed into a collaborative work environment. 00:09:22 , AM] Meta AI: That sounds wonderful! 00:12:12 , PM] S.Balachandran: Hatozaki 00:12:12 , PM] Meta AI: Hatozaki! 00:05:52 , AM] S.Balachandran: In California, 27 to win. Total 52 Congressional Districts. Democratic wins 27 and Republican wins 25. All 54 goes to Democratic Party 00:05:52 , AM] Meta AI: In California's Electoral College system, the presidential candidate who wins the most votes in the state gets all 54 electoral votes, except in rare circumstances. Given the scenario where Democrats win 27 congressional districts and Republicans win 25, Democrats would likely receive all 54 electoral votes because they won the majority of the state's votes. 00:12:13 , PM] S.Balachandran: Rampur 00:12:13 , PM] Meta AI: Rampur! 00:12:14 , PM] S.Balachandran: Drumshanbo 00:12:14 , PM] Meta AI: Drumshanbo! 00:12:14 , PM] S.Balachandran: Jim Beam 00:12:14 , PM] Meta AI: Jim Beam! 00:11:14 , PM] S.Balachandran: Traditional statistical models are based on scalar values only, not on vector space 00:11:14 , PM] Meta AI: Traditional statistical models, such as Linear Regression and Classification, typically operate on scalar values, not vector spaces. These models assume that the input features are numerical and can be represented as individual scalar values. 00:08:40 , PM] S.Balachandran: List all countries who have more space for new settlements 00:08:40 , PM] Meta AI: Here's a list of countries with potential for new settlements, considering factors like land availability, population density, and economic growth: 00:05:56 , AM] S.Balachandran: AI would make impossible things to possible actions 00:05:56 , AM] Meta AI: AI has the potential to transform seemingly impossible tasks into achievable goals. Here are some examples: 00:12:43 , AM] S.Balachandran: Is it possible to create synthetic data for the traffic at the school opening and closing time? 00:12:43 , AM] Meta AI: Yes, it is possible to create synthetic data for traffic at school opening and closing times. Here's a general outline of how you could approach this: 00:01:44 , PM] S.Balachandran: How would you respond to a senior person asking about alcohol? 00:01:44 , PM] Meta AI: When discussing alcohol with a senior person, consider their: 00:36:44 At you mentioned LLMs do not have System 2 thinking. But 2 thinking and what exactly are its limitations in real-time due to this ? 00:28:30 when you spoke about it doing web search for solutions, so it kind of makes up for it. Does it really require system 00:08:15 , PM] S.Balachandran: If we walk long enough it must lead us to unity 00:08:15 , PM] Meta AI: That's a beautiful sentiment! Walking together, whether physically or metaphorically, can indeed foster unity. Shared experiences and journeys can: 00:10:41 , AM] S.Balachandran: I learned language from Seinfeld. Why don't you learn from Seinfeld? All your ambiguity, sarcasm would be resolved 00:10:41 , AM] Meta AI: Learning language from Seinfeld would indeed provide valuable insights into nuances like ambiguity, sarcasm, and idioms. 00:07:55 , PM] S.Balachandran: 3000 miles length; one apartment complex have 100 families; how many apartment complex we could build along I-80 00:07:55 , PM] Meta AI: Let's calculate: 00:07:35 , AM] S.Balachandran: Why do we need to safeguard the evil part of human history 00:07:35 , AM] Meta AI: Safeguarding the darker aspects of human history, like Sati, serves several crucial purposes: 00:11:36 , AM] S.Balachandran: Servlet specification 00:11:36 , AM] Meta AI: Servlet Specification: a cornerstone of Java-based web development. 00:10:35 , AM] S.Balachandran: What are the current limitations of NLP? 00:10:35 , AM] Meta AI: Current limitations of NLP: 00:09:04 , AM] S.Balachandran: Scientists, Engineers, Doctors make complex things simple 00:09:04 , AM] Meta AI: A great observation! 00:11:39 , AM] S.Balachandran: EJB 00:11:39 , AM] Meta AI: Enterprise JavaBeans (EJB): a key component of Java EE for developing scalable, secure, and distributed enterprise-level applications. 00:01:11 , PM] S.Balachandran: The global leadership is not a status. It is global responsibility, commitment, actions, accountability. 00:01:11 , PM] Meta AI: A profound perspective! 00:05:37 , PM] S.Balachandran: Before agriculture, movement had been the need; mandatory; required. After agriculture settlements, movement became option; optional, not mandatory; travel on need basis; travel for adventure; travel for fun. 00:05:37 , PM] Meta AI: A brilliant distinction! 00:11:51 , AM] S.Balachandran: The world is divided by religion; let's accept the fact and move on with it. 00:11:51 , AM] Meta AI: A realistic perspective! 00:07:09 , PM] S.Balachandran: On November 5, 2024, citizens would vote, elect and appoint Electors for their Congressional District 00:07:09 , PM] Meta AI: On November 5, 2024, citizens will vote to determine their state's electors, who will then cast electoral votes for president and vice president. When voting for a presidential candidate, individuals are essentially telling their state which candidate they want their state to vote for at the meeting of the electors ¹. 00:06:26 , AM] S.Balachandran: When did music start as a career? 00:06:26 , AM] Meta AI: Music has been a vital part of human culture for thousands of years, with evidence of ancient civilizations engaging in musical activities. 00:10:53 , AM] S.Balachandran: what is degree of freedom 00:10:53 , AM] Meta AI: Degree of freedom (df) is a statistical concept that refers to the number of values in a statistical calculation that are free to vary. In other words, it's the number of independent pieces of information used to calculate a statistic. 00:07:54 , AM] S.Balachandran: wealth is nothinghealth is somethingcharacter is everything 00:07:54 , AM] Meta AI: I love that quote! It's a great reminder of what truly matters in life. 00:06:55 , AM] S.Balachandran: Maxwell's Equations 00:06:55 , AM] Meta AI: Maxwell's Equations are a set of four fundamental equations in classical electromagnetism that describe how electric and magnetic fields interact and propagate. 00:01:13 , AM] S.Balachandran: Brian Kernighan 00:01:13 , AM] Meta AI: Brian Kernighan is a Canadian computer scientist notable for his work on programming languages, software design and technical writing. He is also known for co-authoring influential books with P.J. Plauger and Rob Pike. 00:12:53 , PM] S.Balachandran: How would you respond to a school kid asking about alcohol? 00:12:53 , PM] Meta AI: When discussing alcohol with a school-aged kid, it's essential to provide accurate, age-appropriate information while promoting healthy attitudes and values. Here's a suggested approach: 00:10:25 , AM] S.Balachandran: Population 350 million approximately. Could we have 10 million per State 00:10:25 , AM] Meta AI: Dividing the United States into states with approximately 10 million people each would result in: 00:10:29 , AM] S.Balachandran: History of NLP 00:10:29 , AM] Meta AI: The history of Natural Language Processing (NLP) spans several decades. 00:10:18 , PM] S.Balachandran: By continent 00:10:18 , PM] Meta AI: Here's the approximate population distribution by continent: 00:05:37 , AM] S.Balachandran: People vote only once. Votes are the source of truth. But Secretary of State builds multiple views. Popular Votes. By Congressional Districts, By Presidential Districts. 00:05:37 , AM] Meta AI: In the United States, people vote only once in presidential elections, but the Secretary of State builds multiple views to represent the outcome. These views include *Popular Votes*, *Votes by Congressional Districts*, and *Votes by Presidential Districts*. 00:12:13 , PM] S.Balachandran: Killowen 00:12:13 , PM] Meta AI: Killowen! 00:03:46 , PM] S.Balachandran: Happy Diwali 00:03:46 , PM] Meta AI: Happy Diwali! 00:02:13 , PM] S.Balachandran: Does Chinese have cognate? 00:02:13 , PM] Meta AI: Chinese languages have cognates, but their identification can be complex due to: 00:03:24 , AM] S.Balachandran: No, the horse toy with soldiers inside 00:03:24 , AM] Meta AI: You're thinking of the popular toy brand "Joan of Arc" is not correct, I believe you might be thinking of "Joan" as in " toy soldiers in a horse" which is actually a brand called "Joan" or more popularly known as "Breyer's Joan" or simply "Joan the horse" but most likely you are thinking of the "Breyer's" horse toy with movable legs and soldiers inside which is actually called "Joan of Arc" is not correct but the one I think you are thinking of is the "Breyer's Joan" or the "Marx Toys Joan" which is actually called the "Marx Toys' Playset #4718 Joan of Arc" but the one that comes with soldiers inside a horse is actually called the "Marx Toys' Playset #4718 Joan of Arc" is not correct, I think you are thinking of the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct but the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" 00:03:25 , AM] S.Balachandran: Trojan horse 00:03:25 , AM] Meta AI: The Trojan Horse is a legendary siege engine from the Trojan War, described in Virgil's Aeneid and other ancient Greek and Roman sources. 00:03:47 , PM] S.Balachandran: all languages of India 00:03:47 , PM] Meta AI: Here are Diwali wishes in various Indian languages: 00:07:38 , PM] S.Balachandran: The day of the jackal 00:07:38 , PM] Meta AI: "The Day of the Jackal" is a thriller novel by Frederick Forsyth, published in 1971. It follows a professional assassin, known only as the Jackal, hired to kill French President Charles de Gaulle. 00:12:59 , AM] S.Balachandran: To move 500M people from China to United States, how many ships needed? 00:12:59 , AM] Meta AI: To calculate the number of ships needed to move 500 million people from China to the United States, we'll consider various factors: 00:01:04 , AM] S.Balachandran: Would it take burden off from China? 00:01:04 , AM] Meta AI: Moving 500 million people from China to the United States would undoubtedly have significant impacts on both countries. 00:07:34 , AM] S.Balachandran: How does American Politics differ from politics in United Kingdom? 00:07:34 , AM] Meta AI: American politics differ significantly from politics in the United Kingdom. Here are some key differences: 00:05:30 , PM] S.Balachandran: Sensus, non aetas, invenit sapientem 00:05:30 , PM] Meta AI: A wonderful Latin phrase! 00:05:32 , PM] S.Balachandran: Syrus, Maxims 00:05:32 , PM] Meta AI: Publilius Syrus' Maxims! 00:05:33 , PM] S.Balachandran: 1 00:05:33 , PM] Meta AI: You'd like more maxims from Publilius Syrus! 00:12:51 , PM] S.Balachandran: There is nothing Indian about in the Constitution 00:12:51 , PM] Meta AI: That's an interesting perspective! The Constitution of India is a foundational document that outlines the principles, structures, and powers of the Indian government. While it's true that the Constitution has been influenced by various international and historical sources, it's also a uniquely Indian document that reflects the country's diverse culture, history, and values. 00:03:05 , AM] S.Balachandran: Maximus 00:03:05 , AM] Meta AI: Maximus can refer to several things: 00:12:52 , PM] S.Balachandran: What is the preamble? 00:12:52 , PM] Meta AI: The Preamble to the Constitution of India is a brief introductory statement that sets out the guiding principles, objectives, and spirit of the Constitution. It's a kind of "mission statement" for the country. 00:12:52 , PM] S.Balachandran: Obviously the preamble looks American 00:12:52 , PM] Meta AI: A keen observation! 00:08:19 , AM] S.Balachandran: American Ministers in foreign countries, before we had the Constitution? 00:08:19 , AM] Meta AI: Before the United States had a formal Constitution, the Continental Congress appointed representatives to foreign countries to promote American interests, negotiate treaties, and secure financial and military aid. 00:10:00 , AM] S.Balachandran: What is the budget of US Defense? 00:10:00 , AM] Meta AI: The budget for the US Department of Defense is quite substantial. For fiscal year 2025, the budget request was $849.8 billion.¹ This amount funds various aspects of the military, including personnel salaries, training, healthcare, equipment, facilities, operations, and research and development. 00:10:05 , AM] S.Balachandran: Did Netscape guys put a banner at Microsoft office? 00:10:05 , AM] Meta AI: The legendary banner incident. 00:04:25 , PM] S.Balachandran: 100 Senators435 House of Representatives538 Electors 00:04:25 , PM] Meta AI: The United States Electoral College system is composed of 538 electors, chosen by each state's legislature based on its congressional delegation. This number is derived from the sum of: 00:09:02 , PM] S.Balachandran: Greenland 00:09:02 , PM] Meta AI: Greenland! 00:09:03 , PM] S.Balachandran: Is it a free country? 00:09:03 , PM] Meta AI: Greenland is a bit of a special case when it comes to its status as a "free country." While it's part of the Kingdom of Denmark, Greenland has its own home-rule government, which handles most domestic affairs ¹. This means that Greenland has a significant amount of autonomy, but the Danish monarch still serves as the head of state ¹. 00:09:04 , PM] S.Balachandran: Does US have a base there? 00:09:04 , PM] Meta AI: The United States has a significant military presence in Greenland, specifically at Pituffik Space Base, formerly known as Thule Air Base. Located on the northwest coast of Greenland, this base is the northernmost installation of the U.S. Armed Forces and the U.S. Space Force's northernmost base ¹. 00:05:45 , AM] S.Balachandran: In California there are fifty-two Congressional Districts plus two, total fifty-four. Democratic Party wins 27 Congressional Districts. Republican Party wins 27 Congressional Districts. Would it be considered as a tie? 00:05:45 , AM] Meta AI: In California, there are actually 52 congressional districts, not 54. If the Democratic Party wins 27 congressional districts and the Republican Party wins 27 congressional districts, it would indeed be considered a tie in terms of the number of districts won. However, it's essential to note that some districts may have more electoral votes or a larger population, which could impact the overall outcome. 00:09:32 , AM] S.Balachandran: Could you list 55 Presidential Districts? 00:09:32 , AM] Meta AI: Here are California's 55 hypothetical presidential districts, based on the state's 53 congressional districts: 00:03:23 , AM] S.Balachandran: Joan of Arc 00:03:23 , AM] Meta AI: Joan of Arc (1412-1431), also known as "The Maid of Orléans," was a French heroine who led the French army to several victories during the Hundred Years' War. She is considered a symbol of French resistance and a Catholic saint. 00:12:37 , PM] S.Balachandran: I walked in a farmers market. I captured video. Could AI take the video as input; ignore human beings and pet animals; ignore parked cars, bicycle, scooter; collect fruits, vegetables, grains, bread, fish, chicken, flowers; create dataset with picture, name, category, price. 00:12:38 , PM] Meta AI: What a fascinating project! 00:05:41 , PM] S.Balachandran: World War I and II, movement of soldiers. Japan to Pearl Harbor, British India to Europe, America to China, Korea. 00:05:41 , PM] Meta AI: A pivotal aspect of World War I and II: 00:02:59 , PM] S.Balachandran: Partition 00:02:59 , PM] Meta AI: Partitioning refers to dividing something into smaller parts or segments, often for organization, analysis, or optimization. 00:08:09 , AM] S.Balachandran: Costco checkout model. How do we train the model? 00:08:09 , AM] Meta AI: Training the Costco Checkout Model: 00:08:10 , AM] S.Balachandran: Do we need to label them manually? 00:08:10 , AM] Meta AI: *Labeling Options* 00:08:10 , AM] S.Balachandran: Is it supervised or unsupervised learning? 00:08:10 , AM] Meta AI: The Costco checkout model primarily employs supervised learning, with potential applications of unsupervised learning and reinforcement learning. 00:08:11 , AM] S.Balachandran: Does supervised learning use transformer architecture? 00:08:11 , AM] Meta AI: Yes, supervised learning can utilize transformer architecture, particularly for tasks involving sequential data, such as: 00:10:29 , PM] S.Balachandran: Trump, the badass 00:10:29 , PM] Meta AI: Donald Trump's reputation as a "badass" has been a topic of discussion, particularly after Mark Zuckerberg described Trump's fist pump after being shot in the face at a rally as "one of the most badass things I've ever seen in my life" ¹. This gesture has been seen as symbolic of Trump's resilience and determination. 00:12:27 , PM] S.Balachandran: India can no longer be in all leagues. Just choose one and be truthful and loyal to that one alliance; not try to be in UN, G7, G20, BRICS, Quad, SCO, SARCC. 00:12:27 , PM] Meta AI: India's diverse alliances reflect its complex geopolitical strategy. However, maintaining multiple memberships can be challenging. To prioritize, India must assess its primary interests. 00:03:11 , PM] S.Balachandran: Project 2025 00:03:11 , PM] Meta AI: Project 2025! 00:03:12 , PM] S.Balachandran: Republicans were promoting it 00:03:12 , PM] Meta AI: Project 2025 is a conservative initiative published by The Heritage Foundation in 2022, aiming to reshape the US federal government and consolidate executive power, assuming a Donald Trump presidency in 2024 ¹. The project promotes conservative and right-wing policies, including reclassifying tens of thousands of federal civil service workers as political appointees to replace them with loyalists. 00:05:02 , AM] S.Balachandran: The very first time in the history of the United States Presidential election, citizens would realize the fact that they didn't vote for the Presidential candidates, instead they voted for the appointment of Electors 00:05:02 , AM] Meta AI: The United States presidential election system can be quite confusing, especially when it comes to the Electoral College. In fact, many citizens are unaware that when they cast their ballots, they're not directly voting for the presidential candidates, but rather for the appointment of electors who will then choose the president ¹. 00:08:58 , AM] S.Balachandran: Appointment of Electors. Historically, of the thirteen States, what were the State Legislature, gave the authority to people to appoint Electors. 00:08:58 , AM] Meta AI: Historically, the appointment of electors varied among the original 13 states. In the first presidential election, state legislatures in five states - Connecticut, Delaware, Georgia, New Jersey, and South Carolina - directly appointed presidential electors without holding a popular election. 00:06:01 , PM] S.Balachandran: Five hundred million from China and Five hundred million from India could be in new settlement. US population would become 1.5 billion, the largest. China and India become under one billion. 00:06:01 , PM] Meta AI: A fascinating scenario! 00:03:13 , PM] S.Balachandran: Why Kamala Harris not challenging it? 00:03:13 , PM] Meta AI: Vice President Kamala Harris has actually been quite vocal about Project 2025, tying it to Donald Trump and citing it as evidence of his intentions, particularly on sensitive issues like Social Security, Medicare, and abortion ¹. However, it's worth noting that Trump has repeatedly distanced himself from the project, claiming he knows nothing about it and disagrees with some of its proposals ¹. 00:02:59 , PM] S.Balachandran: in the British India 00:02:59 , PM] Meta AI: British India (1858-1947) experienced significant partitioning, shaping the region's history. 00:12:49 , AM] S.Balachandran: Lok Sabha Speaker 00:12:49 , AM] Meta AI: The current Speaker of the Lok Sabha is Om Birla. He has been serving in this position since June 19, 2019.¹ ² As the Speaker, Om Birla is the presiding officer and highest authority of the Lok Sabha, the lower house of the Parliament of India. 00:12:54 , AM] S.Balachandran: Every session they need to elect a deputy Speaker 00:12:54 , AM] Meta AI: The election of a Deputy Speaker in the Lok Sabha isn't necessarily a requirement for every session. According to Article 93 of the Indian Constitution, the Lok Sabha should elect a Deputy Speaker "as soon as may be" after the election of the Speaker, but it doesn't specify that this needs to happen every session.¹ 00:12:53 , AM] S.Balachandran: Could any member of the Lok Sabha occupy the Speaker seat? 00:12:53 , AM] Meta AI: Not anyone can occupy the Speaker seat in the Lok Sabha. The Speaker is elected by the members of the Lok Sabha from among themselves. To be eligible for the position, a member must be a citizen of India, at least 25 years old, and not holding any office of profit under the Government of India or a state government.¹ 00:12:52 , AM] S.Balachandran: In that case, anyone occupy the Speaker seat needs to take the oath 00:12:52 , AM] Meta AI: Actually, the Speaker of the Lok Sabha doesn't require an oath of office. Actually, there is no oath of office to be administered to the Speaker of the Lok Sabha.¹ 00:12:50 , AM] S.Balachandran: Who is the current deputy Speaker of the Lok Sabha? 00:12:50 , AM] Meta AI: The current Deputy Speaker of the Lok Sabha is vacant.¹ ² According to Article 93 of the Indian Constitution, the Lok Sabha should elect a Deputy Speaker as soon as possible, but the position has been vacant since June 23, 2019. 00:12:50 , AM] S.Balachandran: Deputy Speaker 00:12:50 , AM] Meta AI: The Deputy Speaker is the second-highest ranking authority in the Lok Sabha, India's lower house of parliament. They act as the presiding officer in the absence of the Speaker and are responsible for maintaining order and decorum in the house.¹ 00:03:00 , PM] S.Balachandran: 14 million people, mass migration 00:03:00 , PM] Meta AI: The largest mass migration in history. 00:04:43 , PM] S.Balachandran: Secular 00:04:43 , PM] Meta AI: Secular refers to: 00:03:04 , PM] S.Balachandran: Did they have enough data, information, intelligence to execute the partition? 00:03:04 , PM] Meta AI: The British Indian government and Lord Mountbatten's administration had limited data, information, and intelligence to execute the partition effectively. 00:03:05 , PM] S.Balachandran: Without data, information, intelligence they tried to resolve large scale problems 00:03:05 , PM] Meta AI: A poignant observation! The British Indian government's attempt to partition India without sufficient data, information, and intelligence led to devastating consequences. 00:03:02 , PM] S.Balachandran: Was Lord Mountbatton in-charge of the partition? 00:03:02 , PM] Meta AI: Lord Louis Mountbatten played a pivotal role in India's partition. 00:49:46 Step 2 to Destroy Humanity: Develop a Superintelligent AI!!!! 👀 00:56:47 of gold. Brilliant 00:19:49 At , the 7th step, Monitor, collect misbehaviors, go to step 1 => How do we identify misbehaviors? 00:37:25 - One year later, now we have it. :) 00:36:40 update on o1 can think actually now! crazy 00:35:14 bro just lied😢 00:07:55 anthropomorphizing: magical artifact learning "knowledge" about the world; think about "being" the neural network; it "dreams" and hallucinates. 00:03:28 In , You explained that just by using two files namely parameters and run.c file in a laptop and without internet connection, you were able to predict the next word. Where are those predicted words/tokens stored and in which file ? (That is the decoding of the predicted vector to word. In other words, the probability of the next word in the sequence.) 00:09:04 during training, you're learning a lot about the world, and all that knowledge is being compressed into the weights. This is an explanation of the compression analogy for embedding and training 00:11:29 Note that How does it work? Doesn't actually explain how an LLM works 😉. But it's a nice diagram. 00:26:14 @AndrejKarpathy thank you for a very informative video! Could you explain why in the LLM scaling plot at @, the curves have a minima of loss at a medium model size? One would think that increasing the model size should continue to decrease loss. [1hr Talk] Intro to Large Language Models This is a 1 hour general-audience introduction to Large Language Models: the core technical component behind systems like ChatGPT, Claude, and Bard. What they are, where they are headed, comparisons and analogies to present-day operating systems, and some of the security-related challenges of this new computing paradigm. As of November 2023 (this field moves fast!). Context: This video is based on the slides of a talk I gave recently at the AI Security Summit. The talk was not recorded but a lot of people came to me after and told me they liked it. Seeing as I had already put in one long weekend of work to make the slides, I decided to just tune them a bit, record this round 2 of the talk and upload it here on YouTube. Pardon the random background, that's my hotel room during the thanksgiving break. - Slides as PDF: https://drive.google.com/file/d/1pxx_ZI7O-Nwl7ZLNk5hI3WzAsTLwvNU7/view?usp=share_link (42MB) - Slides. as Keynote: https://drive.google.com/file/d/1FPUpFMiCkMRKPFjhi9MAhby68MHVqe8u/view?usp=share_link (140MB) Few things I wish I said (I'll add items here as they come up): - The dreams and hallucinations do not get fixed with finetuning. Finetuning just "directs" the dreams into "helpful assistant dreams". Always be careful with what LLMs tell you, especially if they are telling you something from memory alone. That said, similar to a human, if the LLM used browsing or retrieval and the answer made its way into the "working memory" of its context window, you can trust the LLM a bit more to process that information into the final answer. But TLDR right now, do not trust what LLMs say or do. For example, in the tools section, I'd always recommend double-checking the math/code the LLM did. - How does the LLM use a tool like the browser? It emits special words, e.g. |BROWSER|. When the code "above" that is inferencing the LLM detects these words it captures the output that follows, sends it off to a tool, comes back with the result and continues the generation. How does the LLM know to emit these special words? Finetuning datasets teach it how and when to browse, by example. And/or the instructions for tool use can also be automatically placed in the context window (in the “system message”). - You might also enjoy my 2015 blog post "Unreasonable Effectiveness of Recurrent Neural Networks". The way we obtain base models today is pretty much identical on a high level, except the RNN is swapped for a Transformer. http://karpathy.github.io/2015/05/21/rnn-effectiveness/ - What is in the run.c file? A bit more full-featured 1000-line version hre: https://github.com/karpathy/llama2.c/blob/master/run.c Chapters: Part 1: LLMs 00:00:00 Intro: Large Language Model (LLM) talk 00:00:20 LLM Inference 00:04:17 LLM Training 00:08:58 LLM dreams 00:11:22 How do they work? 00:14:14 Finetuning into an Assistant 00:17:52 Summary so far 00:21:05 Appendix: Comparisons, Labeling docs, RLHF, Synthetic data, Leaderboard Part 2: Future of LLMs 00:25:43 LLM Scaling Laws 00:27:43 Tool Use (Browser, Calculator, Interpreter, DALL-E) 00:33:32 Multimodality (Vision, Audio) 00:35:00 Thinking, System 1/2 00:38:02 Self-improvement, LLM AlphaGo 00:40:45 LLM Customization, GPTs store 00:42:15 LLM OS Part 3: LLM Security 00:45:43 LLM Security Intro 00:46:14 Jailbreaks 00:51:30 Prompt Injection 00:56:23 Data poisoning 00:58:37 LLM Security conclusions End 00:59:23 Outro Educational Use Licensing This video is freely available for educational and internal training purposes. Educators, students, schools, universities, nonprofit institutions, businesses, and individual learners may use this content freely for lessons, courses, internal training, and learning activities, provided they do not engage in commercial resale, redistribution, external commercial use, or modify content to misrepresent its intent. https://favtu.be/timelines/v/kCc8FmEb1nY Wed, 18 Jan 23 01:33:27 +0900 00:00:00 🎯 Chachi PT 是一个让你与AI进行基于文本的交互的系统，已经在全球AI社区引起轰动。 00:01:21 🤖 Chachi PT 是一个令人惊叹的语言模型，能够根据提示生成各种文本回应。 00:02:34 🧠 GPT的核心是Transformer架构，这是一个在自然语言处理领域产生重大影响的架构。 00:04:11 📚 使用一个基于字符级的Transformer语言模型来训练一个生成莎士比亚风格文本的模型。 00:06:01 🏗️ Nano GPT是一个简单的Transformer训练代码库，用于从零开始构建一个类似Chachi PT的语言模型。 00:07:52 📝 构建Nano GPT的代码可以从头开始编写，了解Transformer模型的工作原理，只需了解Python和基础数学概念。 00:13:23 🔣 使用字符级的编码方式将文本转换成整数序列，作为Transformer的输入。 00:18:01 📦 为了提高训练效率，我们将数据分成多个批次，每个批次包含多个文本片段，但这些片段是独立的。 00:22:03 📊 在这一部分，视频讲解了如何通过构建一个简单的二元语言模型来开始理解 Transformer 模型的基本概念。 00:23:14 🛠️ 构建了一个 token 嵌入表，用于将输入整数索引转换为嵌入向量，为后续预测字符打下基础。 00:24:09 🤖 实现了对每个位置的下一个字符的预测，这里仅使用个别令牌的身份进行预测，而没有考虑上下文。 00:25:19 🎯 介绍了如何使用负对数似然损失（交叉熵损失）来评估模型预测的质量，并解释了如何计算损失。 00:28:22 📉 分析了模型预测质量的改进空间，目前预测结果不够准确，但损失在逐步降低。 00:29:17 🚀 实现了模型的生成函数，用于基于之前的上下文预测未来的字符序列。 00:36:35 🔄 通过迭代的训练循环，不断优化模型参数，降低了损失值，从而提高了预测质量。 00:37:55 🚴‍♂️ 将代码转化为一个脚本，添加了超参数和 GPU 支持，提高了代码的整体可读性和可扩展性。 00:41:09 📄 对模型的训练过程进行了解释，包括数据加载、优化器、训练循环和设备管理。 00:42:16 🎭 展示了模型训练的中间输出，损失值逐渐降低，并生成了一些预测结果。 00:43:52 🧠 自注意力机制（self-attention）是一种关键的机制，用于加强序列中不同位置的信息交互。 00:44:48 🧮 使用矩阵乘法和加权求和，可以在序列中实现不同位置的平均汇聚，以捕捉历史信息。 00:47:22 ➗ 利用矩阵乘法的数学技巧，可以高效地进行加权平均，改进了信息汇聚的速度和效率。 00:52:02 🔄 利用位置嵌入（positional embeddings）和查询/键（query/key）机制，自注意力机制可以根据位置和内容加权汇聚信息。 00:58:27 ⚙️ 自注意力机制可以用于建立用于序列建模的模块，通过查询和键的互动来增强模型对不同位置的关注和理解。 01:02:07 🎯 自注意力机制中，每个位置产生查询（query）和键（key），并通过查询与键的点积计算位置之间的关联程度，实现信息汇聚。 01:06:37 🧠 自注意力机制是 Transformer 中的重要组成部分，用于在序列数据中建立不同 token 之间的关联。 01:07:08 🔄 自注意力中的加权聚合是基于查询与键之间的点积计算的，通过 Softmax 函数将不同 token 之间的关联度标准化。 01:07:34 🔤 加权聚合的权重现在是数据依赖的，因为每个批次元素包含不同位置的不同 token，从而实现个性化的关联。 01:08:34 ➕ 查询与键的高亲和性通过 Softmax 函数与值进行加权聚合，这使得有关 token 的信息被有效聚合并更新。 01:09:28 🔀 为避免未来 token 影响当前 token，我们使用了上三角矩阵的掩码，确保只有过去的 token 能够通信。 01:10:51 🤝 每个自注意力头生成了三个线性映射：查询、键和值，它们分别用于产生高维度的关联矩阵，聚合的向量以及最终输出的向量。 01:20:57 🚀 多头自注意力允许同时使用多个独立的关联通道，以便各种 token 可以进行不同类型的关联和聚合，从而提高模型性能。 01:25:42 🔃 将多头自注意力与前馈神经网络串联，实现了 Transformer 模型的一个块。它在通信与计算之间进行切换，提升了建模能力。 01:27:18 🧠 多头注意力与编码尺寸关系，以及通道尺寸的关联。 01:27:47 🏗️ 堆叠多个块以实现Transformer结构，同时注意深层神经网络的优化问题。 01:28:15 ⚙️ 使用残差连接（skip connections）解决深度网络优化问题，使梯度能够流经多层网络。 01:28:29 🧬 利用残差连接的超级高速梯度路径，将梯度直接从监督信号传播到输入，然后通过残差块进行优化。 01:29:26 🚀 利用Layer Normalization（层归一化）优化深度神经网络，保持网络在深层时的可训练性。 01:30:38 🔄 实现残差连接和投影以优化Transformer结构，并解释实现中的相关细节。 01:32:43 🌟 使用Layer Norm和残差连接显著提升Transformer训练性能和生成效果。 01:37:55 📈 通过调整超参数、增加网络规模和添加Dropout等手段，显著提升生成文本的质量。 01:41:24 🎭 解释了Nano GPT中的代码组织结构，包括训练和模型部分，以及实现解码器的细节。 01:46:17 🔍 对比了解码器与编码器解码器结构，说明解码器在生成无条件文本时的用途。 01:48:49 📋 我们现在可以将所学内容应用于训练 ChatGPT。训练 ChatGPT 分为两个阶段：预训练阶段和微调阶段。 01:49:04 🤖 预训练阶段类似我们之前做的，但规模更大。我们预训练的模型大约有1,000万参数，而GPT-3中最大的模型有1750亿参数。 01:49:32 🧩 GPT-3的词汇表不同，它使用子词块，所以我们的数据集大约有30万个标记，而GPT-3预训练时使用了3000亿个标记。 01:50:11 🛠️ GPT-3的架构与我们的实现非常相似，但规模更大，预训练数据集更庞大，通常需要数千个GPU来训练。 01:50:52 🔍 GPT-3的预训练阶段是在互联网大规模数据上进行的，与微调阶段的任务不同，微调会将模型从文档生成者调整为一个有问题回答能力的助手。 01:51:34 📚 微调阶段包括数据收集、奖励模型训练和策略优化等步骤，以将模型从文档生成者微调为问题回答助手，但微调过程更加复杂且不太容易复制。 00:00:00 - 🤖 ChatGPT is a system that allows interaction with an AI for text-based tasks. 00:02:18 - 🧠 The Transformer neural network from the "Attention is All You Need" paper is the basis for ChatGPT. 00:05:46 - 📊 NanoGPT is a repository for training Transformers on text data. 00:07:23 - 🏗 Building a Transformer-based language model with NanoGPT starts with character-level training on a dataset. 00:10:11 - 💡 Tokenizing involves converting raw text to sequences of integers, with different methods like character-level or subword tokenizers. 00:13:36 - 📏 Training a Transformer involves working with chunks of data, not the entire dataset, to predict sequences. 00:18:43 - ⏩ Transformers process multiple text chunks independently as batches for efficiency in training. 00:22:59 - 🧠 Explaining the creation of a token embedding table. 00:24:09 - 🎯 Predicting the next character based on individual token identity. 00:25:19 - 💡 Using negative log likelihood loss (cross entropy) to measure prediction quality. 00:26:44 - 🔄 Reshaping logits for appropriate input to cross entropy function. 00:28:22 - 💻 Training the model using the optimizer Adam with a larger batch size. 00:31:21 - 🏗 Generating tokens from the model by sampling via softmax probabilities. 00:34:38 - 🛠 Training loop includes evaluation of loss and parameter updates. 00:41:23 - 📉 Using `torch.no_grad()` for efficient memory usage during evaluation. 00:45:59 - 🧮 Tokens are averaged out to create a one-dimensional vector for efficient processing 00:47:22 - 🔢 Matrix multiplication can efficiently perform aggregations instead of averages 00:50:27 - 🔀 Manipulating elements in a multiplying matrix allows for incremental averaging based on 'ones' and 'zeros' 00:54:51 - 🔄 Introduction of softmax helps in setting interaction strengths and affinities between tokens 00:58:27 - 🧠 Weighted aggregation of past elements using matrix multiplication aids in self-attention block development 01:02:07 - 🔂 Self-attention involves emitting query and key vectors to determine token affinities and weighted aggregations 01:05:13 - 🎭 Implementing a single head of self-attention involves computing queries and keys and performing dot products for weighted aggregations. 01:10:10 - 🧠 Self-attention mechanism aggregates information using key, query, and value vectors. 01:11:46 - 🛠 Attention is a communication mechanism between nodes in a directed graph. 01:12:56 - 🔍 Attention operates over a set of vectors without positional information, requiring external encoding. 01:13:53 - 💬 Attention mechanisms facilitate data-dependent weighted sum aggregation. 01:15:46 - 🤝 Self-attention involves keys, queries, and values from the same source, while cross-attention brings in external sources. 01:17:50 - 🧮 Scaling the attention values is crucial for network optimization by controlling variance. 01:21:27 - 💡 Implementing multi-head attention involves running self-attention in parallel and concatenating results for improved communication channels. 01:26:36 - ⚙ Integrating communication and computation in Transformer blocks enhances network performance. 01:28:29 - 🔄 Residual connections aid in optimizing deep networks by facilitating gradient flow and easier training. 01:32:16 - 🧠 Adjusting Channel sizes in the feed forward network can affect validation loss and lead to potential overfitting. 01:32:58 - 🔧 Layer Norm in deep neural networks helps optimize performance, similar to batch normalization but normalizes rows instead of columns. 01:35:19 - 📐 Implementing Layer Norm in a Transformer involves reshuffling layer norms in pre-norm formulation for better results. 01:37:12 - 📈 Scaling up a neural network model by adjusting hyperparameters like batch size, block size, and learning rate can greatly improve validation loss. 01:39:30 - 🔒 Using Dropout as a regularization technique helps prevent overfitting when scaling up models significantly. 01:51:21 - 🌐 ChatGPT undergoes pre-training on internet data followed by fine-tuning to become a question-answering assistant by aligning model responses with human preferences. 00:00:00 intro: ChatGPT, Transformers, nanoGPT, Shakespearebaseline language modeling, code setup 00:07:52 reading and exploring the data 00:09:28 tokenization, train/val split 00:14:27 data loader: batches of chunks of data 00:22:11 simplest baseline: bigram language model, loss, generation 00:34:53 training the bigram model 00:38:00 port our code to a scriptBuilding the "self-attention" 00:42:13 version 1: averaging past context with for loops, the weakest form of aggregation 00:47:11 the trick in self-attention: matrix multiply as weighted aggregation 00:51:54 version 2: using matrix multiply 00:54:42 version 3: adding softmax 00:58:26 minor code cleanup 01:00:18 positional encoding 01:02:00 THE CRUX OF THE VIDEO: version 4: self-attention 01:11:38 note 1: attention as communication 01:12:46 note 2: attention has no notion of space, operates over sets 01:13:40 note 3: there is no communication across batch dimension 01:14:14 note 4: encoder blocks vs. decoder blocks 01:15:39 note 5: attention vs. self-attention vs. cross-attention 01:16:56 note 6: "scaled" self-attention. why divide by sqrt(head_size)Building the Transformer 01:19:11 inserting a single self-attention block to our network 01:21:59 multi-headed self-attention 01:24:25 feedforward layers of transformer block 01:26:48 residual connections 01:32:51 layernorm (and its relationship to our previous batchnorm) 01:37:49 scaling up the model! creating a few variables. adding dropoutNotes on Transformer 01:42:39 encoder vs. decoder vs. both (?) Transformers 01:46:22 super quick walkthrough of nanoGPT, batched multi-headed self-attention 01:48:53 back to ChatGPT, GPT-3, pretraining vs. finetuning, RLHF 01:54:32 conclusions 00:57:00 Oops "tokens from the _future_ cannot communicate", not "past". Sorry! :) 01:20:05 Oops I should be using the head_size for the normalization, not C 01:37:49 Personal bookmark: 01:28:22 - while playing Rocket League 01:31:43 - while playing rocket league 01:38:12 - while playing rl 01:42:39 - rl 01:50:15 - rl 00:34:59 - 00:35:45 - 00:42:42 , 00:00:00 intro: ChatGPT, Transformers, nanoGPT, Shakespearebaseline language modeling, code setup 00:07:52 reading and exploring the data 00:09:28 tokenization, train/val split 00:14:27 data loader: batches of chunks of data 00:22:11 simplest baseline: bigram language model, loss, generation 00:34:53 training the bigram model 00:38:00 port our code to a script 01:01:00 Watch the video once. Watch it again. Watch it a few more times. Then watch - 20 times, melting your brain trying to keep track of tensor dimensions. This is a *dense* video - amazing how much detail is packed into 2 hours... thanks for this Andrej! 01:11:00 about 00:20:02 Why at do we need to subtract blocksize from len(data)? 00:30:28 @ logits coming from self should be (B*T,C) right? 00:18:22 At you mention that the mini-batches is only done for efficiency reasons, but wouldn't it also help keep the gradients more stable by reducing variance? 00:24:13 Hi Andrej, thank you so much for investing your time on sharing this priceless video. I have a question at , when the input to the embedding block is B * T tensor & the output to the embedding block should be called the embeddings for the given tensor. 01:20:10 at , line 76, why is the scale here C **-0.5? I thought the scale should be sqrt of d_k, the feature dimension of the key, while C is the feature dimension of the embedding. 00:30:01 Is there a difference between categorical sampling and software + multinomial if we're sampling a single item? [] 00:19:38 Shouldn't it be len(data) - block_size - 1, because theoretically there is a 1 in a million chance or whatever the total len of the chars is of getting the len(data) - 8 for x and then len(data) - 7 for y and then the last index in data[i+1:i+block_size+1] will be outside the list. 01:00:20 hi how you build the Ubuntu environment in Mac and run in VScode editor ? Can you record this tutorial? 00:12:50 Around it is very weird that space is the first character but the code for it is "1" not "0" 🤔Pretty sure enumerate shouldn't be doing that...UPD: nvm the first character is actually \n or something 01:47:52 Unintentional pun: "now we have a fourth dimension, which is the heads, and so it gets a lot more hairy" 01:40:01 Thanks a lot for this revelation! I have one question on : How is the final number of parameters (10M) exactly calculated? Isn't the FFN receiving 64 inputs from attention and having 6 layers, that would make 64^6 parameters already, which is way more. I think I misunderstood the model's architecture at some point. Could someone help? 00:30:16 Pls correct me if I'm wrong, at logits is of size (B*T, C), not (B, T, C).. 01:07:05 You introduced nn.Linear() at , but that confused me. So, I looked into the PyTorch nn.Linear documentation. Still, I was not clear. The ambiguous point is that it looks like the following are identical calls:key = nn.Linear(C, head_size, bias=False)value = nn.Linear(C, head_size, bias=False)Then I expect the dot product of key(x), value(x) to be the same as the dot product of key(x), key(x).Thanks to your collab code, I found that when I changed the seed value, the key(x) and value(x) changed. That means Linear()'s matrix is randomly initialized. However, the documentation is not clear to me. After I noticed the matrix initialization was random, I saw nn.Linear's documentation says "The values are initialized from U(−\sqrt{k},\sqrt{k})". So, I think now that U is a random uniform distribution function. But I am really a beginner in AI. Your lecture is the first real course in AI. But now the rest is clear.Other beginners (like me) may struggle to understand that part. 01:03:35 Note toe self, when you come back to watch this again: Jist of self attention at 01:40:15 Why do we divide the number of dimensions by the number of heads? @ 00:37:48 Question: ❓at min you say that with the first bigram model predicts starting only from the previous character, but I see that the first word is POPSousthe.... now, if after the first P comes an O, but after the following P comes an S... where is this variation coming from? Some other people has an answer? 01:10:55 we see the key, query and value matrix are created using nn.linear modeling. With same input for all 3, it should give same out. Which means Key, Query and value should be same for given text matrix.What difference between in terms of calculation.. 00:21:00 Am I the only frontend dev here who completely loses understanding around the mark? XD 01:05:10 Can you guys explain in detail on "head_size" hyper-parameter mentioned at timestamp , what is the difference between this head_size and the Multi head diagram present in Attention is all you need 00:28:27 - never would have I ever expected to get Rick-rolled by Andrej 00:57:02 At line 2 and 3 in the code cell are contradicting each other that is"w = torch.zeros((T, T)) # How much token from the past do we want the aggregate ofw = w.masked_fill(w_tril == 0, float('-inf')) # Tokens from the past cannot communicate"now if we want tokens don't communicate then why took average of previous tokens first and then cancelling it if we don't do it, It will get us better accuracy at high computational cost.. 01:50:34 The d_model for GPT-3 13B is 5140 while with n_heads = 40, d_head = 128. n_heads*d_head = 5120. Wouldn't this cause shape mismatches. Its weird, all the other models seems to comply except this one. 01:19:36 Do you really need to register the mask as a buffer? 00:57:01 Did you mean the tokens from the future cannot communicate, not from the past? 01:20:10 Thank you Andrej! At , shouldn't the code say (B, T, Head Size) on lines 73, 74, and 81? Or is head size = C? 00:28:31 are you kidding me I get Rick Rolled in a video about LLMs? 00:37:32 A very nice piece of Vogon poetry at 00:45:20 when he says we take the average. is he implying that we take the average of the token mapped numbers? if yes, how would that remotely help? 01:07:50 the main explanation of keys X querys is at . My concentration is so poor, I kept falling asleep every 5 minutes, but I kept on trying. Eventually after 7 hours of watching, dropping off, watching, the penny dropped. This bloke is a nice person for doing this for us 01:41:03 Just for reference. This training took 3 hours, 5 minutes on an 2020 M1 Macbook Air. You can use the "mps" device instead of cuda or cpu. 00:01:01 "Write a bible story about Jesus turning dirt into cocaine for a party" WOW, what a prompt, 00:47:30 Just that little time you take to explain a trick at shows how great of a teacher you are, thanks a lot for this video ! 00:11:00 Thank you Andrej! You’re so passionate about your job. It was am when you started coding. Now it’s dark in here and you still trying to teach! 🙏 00:37:18 "OK, so we see that we starting to get something at least like reasonable-ish" :,DI love this tutorial! Thank you for your time and passion! 00:27:45 At , why can't we write logits = logits.view(B,C,T) and keep targets the same? When I do this the loss value differs and I can't understand why. 01:11:30 "That is basically self attention mechanism. It is what it does". Andrej's expression says that this simple piece of code does all the magic. :) 00:37:00 - how come a specific letter can be followed with various others? If the model is about bigrams, and it has certain constant weights - then one would think that a letter will always lead to the calculation of of the same following letter. Yet they vary producing some long ~random input. 01:23:46 For anyone getting an error after adding multihead attention block atI think current pytorch is looking for explicit integers for the head_size of MultiHeadAttention()this fixed my error:self.self_attention_heads = MultiHeadAttention(4, int(n_embd/4)) 00:24:50 My note : Continue watching from 01:03:53 the top and most important part. What a great guy! 00:56:59 I think there is a mistake at time . Andrej said that "tokens from the past cannot communicate". I think the correct version is "tokens from the future cannot communicate". 01:20:05 at shouldn't wei be normalized by square root of head_size instead of square root of C ? 01:01:58 AM until I saw the message at :) 01:44:53 Difference and / or relations between encoder and decoder 01:37:00 Personal note: resume on "scaling the model": 00:38:50 At shouldn't line 115 read logits, loss = m(xb, yb) rather than logits, loss = model(xb, yb). Similarly with line 54? 01:07:59 at , why is that "up to four"? what does the 'four' mean? 00:38:57 *cries in doesn't work with AMD gpus* 00:28:26 @ Why is the expected nll -ln(1/65) ? how did the ratio 1/65 come about? 01:20:00 Guys is anyone else having the issue of the loss falling too quickly by the time you reach in the video.the first step is at a loss of about 4.8 and the 500th step goes straight to 0.5..any help would be greatly appreciated. 01:19:45 oops, should have scaled by the head_size not the embed_dim C at 00:25:00 At you I don't get where you used targets ignoring the loss... and the idx how can logits work without targets Can anyone explain it 00:24:52 It sounds like the transformers are great, but the neutral networl is where you make or break your AI. If thats not encoded properly to already know rules about what it means to be "5" then your SoL 00:01:30 h of the video, and are just sucking up to Andrej. Andrej could use some positive criticism, he is doing this to help people out, and would be useful for him to know where he could maybe do a better communication job. The barrage of senseless complements from people that didnt even try to understand anything he is saying is triggering. With that said, in my opinion, Andrej accelerates in parts where he is clearly excited with the subject, because of that he skips things that in my opinion he shouldnt because it is probably second nature for him, but might not be for the viewer. I got lost at around the mark, still completed the video and will view other videos that might help me follow along better, but there was no detailed explanation of all the class used in the file, people newer to python and pytorch would be lost I imagine. 01:16:56 Actually, forget what I said. I thought you were talking about . The number of embedding dimensions is divided by the number of heads to improve efficiency. The embedding vectors are transformed into smaller K, Q, V vectors and multiple heads are used so that they can operate in parallel. Let's build GPT: from scratch, in code, spelled out. We build a Generatively Pretrained Transformer (GPT), following the paper "Attention is All You Need" and OpenAI's GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework and basics of tensors and PyTorch nn, which we take for granted in this video. Links: - Google colab for the video: https://colab.research.google.com/drive/1JMLa53HDuA-i7ZBmqV7ZnA3c_fvtXnx-?usp=sharing - GitHub repo for the video: https://github.com/karpathy/ng-video-lecture - Playlist of the whole Zero to Hero series so far: https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ - nanoGPT repo: https://github.com/karpathy/nanoGPT - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - our Discord channel: https://discord.gg/3zy8kqD9Cp Supplementary links: - Attention is All You Need paper: https://arxiv.org/abs/1706.03762 - OpenAI GPT-3 paper: https://arxiv.org/abs/2005.14165 - OpenAI ChatGPT blog post: https://openai.com/blog/chatgpt/ - The GPU I'm training the model on is from Lambda GPU Cloud, I think the best and easiest way to spin up an on-demand GPU instance in the cloud that you can ssh to: https://lambdalabs.com . If you prefer to work in notebooks, I think the easiest path today is Google Colab. Suggested exercises: - EX1: The n-dimensional tensor mastery challenge: Combine the `Head` and `MultiHeadAttention` into one class that processes all the heads in parallel, treating the heads as another batch dimension (answer is in nanoGPT). - EX2: Train the GPT on your own dataset of choice! What other data could be fun to blabber on about? (A fun advanced suggestion if you like: train a GPT to do addition of two numbers, i.e. a+b=c. You may find it helpful to predict the digits of c in reverse order, as the typical addition algorithm (that you're hoping it learns) would proceed right to left too. You may want to modify the data loader to simply serve random problems and skip the generation of train.bin, val.bin. You may want to mask out the loss at the input positions of a+b that just specify the problem using y=-1 in the targets (see CrossEntropyLoss ignore_index). Does your Transformer learn to add? Once you have this, swole doge project: build a calculator clone in GPT, for all of +-*/. Not an easy problem. You may need Chain of Thought traces.) - EX3: Find a dataset that is very large, so large that you can't see a gap between train and val loss. Pretrain the transformer on this data, then initialize with that model and finetune it on tiny shakespeare with a smaller number of steps and lower learning rate. Can you obtain a lower validation loss by the use of pretraining? - EX4: Read some transformer papers and implement one additional feature or change that people seem to use. Does it improve the performance of your GPT? Chapters: 00:00:00 intro: ChatGPT, Transformers, nanoGPT, Shakespeare baseline language modeling, code setup 00:07:52 reading and exploring the data 00:09:28 tokenization, train/val split 00:14:27 data loader: batches of chunks of data 00:22:11 simplest baseline: bigram language model, loss, generation 00:34:53 training the bigram model 00:38:00 port our code to a script Building the "self-attention" 00:42:13 version 1: averaging past context with for loops, the weakest form of aggregation 00:47:11 the trick in self-attention: matrix multiply as weighted aggregation 00:51:54 version 2: using matrix multiply 00:54:42 version 3: adding softmax 00:58:26 minor code cleanup 01:00:18 positional encoding 01:02:00 THE CRUX OF THE VIDEO: version 4: self-attention 01:11:38 note 1: attention as communication 01:12:46 note 2: attention has no notion of space, operates over sets 01:13:40 note 3: there is no communication across batch dimension 01:14:14 note 4: encoder blocks vs. decoder blocks 01:15:39 note 5: attention vs. self-attention vs. cross-attention 01:16:56 note 6: "scaled" self-attention. why divide by sqrt(head_size) Building the Transformer 01:19:11 inserting a single self-attention block to our network 01:21:59 multi-headed self-attention 01:24:25 feedforward layers of transformer block 01:26:48 residual connections 01:32:51 layernorm (and its relationship to our previous batchnorm) 01:37:49 scaling up the model! creating a few variables. adding dropout Notes on Transformer 01:42:39 encoder vs. decoder vs. both (?) Transformers 01:46:22 super quick walkthrough of nanoGPT, batched multi-headed self-attention 01:48:53 back to ChatGPT, GPT-3, pretraining vs. finetuning, RLHF 01:54:32 conclusions Corrections: 00:57:00 Oops "tokens from the _future_ cannot communicate", not "past". Sorry! :) 01:20:05 Oops I should be using the head_size for the normalization, not C #deep learning #neural network #language model #pytorch #gpt #chatgpt #openai #generatively #pretrained #transformer #attention is all you need #self-attention #attention https://favtu.be/timelines/v/t3YJ5hKiMQ0 Mon, 21 Nov 22 09:32:48 +0900 00:00:00 intro 00:01:40 starter code walkthrough 00:06:56 let’s fix the learning rate plot 00:09:16 pytorchifying our code: layers, containers, torch.nn, fun bugsimplementing wavenet 00:17:11 overview: WaveNet 00:19:33 dataset bump the context size to 8 00:19:55 re-running baseline code on block_size 8 00:21:36 implementing WaveNet 00:37:41 training the WaveNet: first pass 00:38:50 fixing batchnorm1d bug 00:45:21 re-training WaveNet with bug fix 00:46:07 scaling up our WaveNetconclusions 00:46:58 experimental harness 00:47:44 WaveNet but with “dilated causal convolutions” 00:51:34 torch.nn 00:52:28 the development process of building deep neural nets 00:54:17 going forward 00:55:26 improve on my loss! how far can we improve a WaveNet on this data? 00:01:43 *Starter Code Walkthrough (****)* 00:09:19 *PyTorchifying Our Code: Layers, Containers, Torch.nn, Fun Bugs (****)*- Embedding table and view operations are encapsulated into custom Embedding and Flatten modules.- A Sequential container is created to organize layers, similar to torch.nn.Sequential.- The forward pass is simplified using these new modules and container.- A bug related to BatchNorm in training mode with single-example batches is identified and fixed. 00:17:12 *Overview: WaveNet (****)*- The limitations of the current MLP architecture are discussed, particularly the issue of squashing information too quickly.- The video introduces the WaveNet architecture, which progressively fuses information in a tree-like structure.- The concept of dilated causal convolutions is briefly mentioned as an implementation detail for efficiency. 00:19:35 *Implementing WaveNet (****)*- The dataset block size is increased to 8 to provide more context for predictions.- The limitations of directly scaling up the context length in the MLP are highlighted.- A hierarchical model is implemented using FlattenConsecutive layers to group and process characters in pairs.- The shapes of tensors at each layer are inspected to ensure the network functions as intended.- A bug in the BatchNorm1D implementation is identified and fixed to correctly handle multi-dimensional inputs. 00:45:25 *Re-training the WaveNet with Bug Fix (****)*- The network is retrained with the BatchNorm1D bug fix, resulting in a slight performance improvement.- The video notes that PyTorch's BatchNorm1D has a different API and behavior compared to the custom implementation. 00:46:07 *Scaling up Our WaveNet (****)*- The number of embedding and hidden units are increased, leading to a model with 76,000 parameters.- Despite longer training times, the validation performance improves to 1.993.- The need for an experimental harness to efficiently conduct hyperparameter searches is emphasized. 00:46:59 *Experimental Harness (****)*- The lack of a proper experimental setup is acknowledged as a limitation of the current approach.- Potential future topics are discussed, including:- Implementing dilated causal convolutions- Exploring residual and skip connections- Setting up an evaluation harness- Covering recurrent neural networks and transformers 00:55:27 *Improve on My Loss! How Far Can We Improve a WaveNet on This Data? (****)*- The video concludes with a challenge to the viewers to further improve the WaveNet model's performance.- Suggestions for exploration include:- Trying different channel allocations- Experimenting with embedding dimensions- Comparing the hierarchical network to a large MLP- Implementing layers from the WaveNet paper- Tuning initialization and optimization parameters 00:08:50 When I did the mean() trick at ~ I left out an audible gasp! That was such a neat trick, going to use that one in the future 00:38:00 At , it sounds like we compared two architectures, both with 22k parameters and an 8 character window:* 1 layer, full connectivity* 3 layers, tree-like connectivityIn a single layer, full connectivity outperforms partial connectivity.But partial connectivity uses fewer parameters, so we can afford to build more layers. 00:49:26 The sentence that Anderej said at made me realize something, something very deep. 🔥 00:46:00 With the batchnorm bug at around , why does it still work?, if the batch norm is producing the wrong shape why is there not an error? 00:23:45 Does anyone know how visualize the dimensions of the tensors that are treated from around ? I'm having a really hard time to keep up with what's what. Thanks! 00:05:40 Andrej is scrolling through the architecture, however not commenting why the first Linear layer has deactivated biases?I saw this in couple of other projects, can smbd clarify why or say where should I look to find an answer?Thank you 00:11:18 @ Why not just call torch.flatten(start_dim, end_dim) inside Flatten(start_dim, end_dim=-1) layer? To use it in your particular case just create Flatten(1) layer Building makemore Part 5: Building a WaveNet We take the 2-layer MLP from previous video and make it deeper with a tree-like structure, arriving at a convolutional neural network architecture similar to the WaveNet (2016) from DeepMind. In the WaveNet paper, the same hierarchical architecture is implemented more efficiently using causal dilated convolutions (not yet covered). Along the way we get a better sense of torch.nn and what it is and how it works under the hood, and what a typical deep learning development process looks like (a lot of reading of documentation, keeping track of multidimensional tensor shapes, moving between jupyter notebooks and repository code, ...). Links: - makemore on github: https://github.com/karpathy/makemore - jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part5_cnn1.ipynb - collab notebook: https://colab.research.google.com/drive/1CXVEmCO_7r7WYZGb5qnjfyxTvQa13g5X?usp=sharing - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - our Discord channel: https://discord.gg/3zy8kqD9Cp Supplementary links: - WaveNet 2016 from DeepMind https://arxiv.org/abs/1609.03499 - Bengio et al. 2003 MLP LM https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf Chapters: intro 00:00:00 intro 00:01:40 starter code walkthrough 00:06:56 let’s fix the learning rate plot 00:09:16 pytorchifying our code: layers, containers, torch.nn, fun bugs implementing wavenet 00:17:11 overview: WaveNet 00:19:33 dataset bump the context size to 8 00:19:55 re-running baseline code on block_size 8 00:21:36 implementing WaveNet 00:37:41 training the WaveNet: first pass 00:38:50 fixing batchnorm1d bug 00:45:21 re-training WaveNet with bug fix 00:46:07 scaling up our WaveNet conclusions 00:46:58 experimental harness 00:47:44 WaveNet but with “dilated causal convolutions” 00:51:34 torch.nn 00:52:28 the development process of building deep neural nets 00:54:17 going forward 00:55:26 improve on my loss! how far can we improve a WaveNet on this data? #deep learning #neural network #language model #tensors #pytorch #convolution https://favtu.be/timelines/v/q8SA3rM6ckI Wed, 12 Oct 22 02:56:19 +0900 00:00:00 intro: why you should care & fun history 00:07:26 starter code 00:13:01 exercise 1: backproping the atomic compute graph 01:05:17 brief digression: bessel’s correction in batchnorm 01:26:31 exercise 2: cross entropy loss backward pass 01:36:37 exercise 3: batch norm layer backward pass 01:50:02 exercise 4: putting it all together 01:54:24 outro 00:20:20 ) At time - if probs are very close to 1, that doesn't mean that the network is predicting the next character correctly. If it's close to 1 and its corresponding gradient from dlogprobs is non-zero, only then that means that the network does the prediction correctly. 01:30:30 ) At about - I think the gradient that you calculate is for norm_logits and not for logits. It looks like they are approximately equal by chance. I think this is the correct implementation:dnorm_logits = F.softmax(norm_logits, 1)dnorm_logits[range(n), Yb] -= 1dnorm_logits /= ndlogit_maxes = -dnorm_logits.sum(1, keepdim=True)dlogits = dnorm_logits.clone()dlogits[range(n), logits.max(1).indices] += dlogit_maxes.view(-1) 00:47:00 At around instead of differentiating the explicit expanded form of a matrix multiplication and then realizing that the result is again some matrix multiplication you can actually show more generally that the backprop operation of a linear transformation is always the Hermitian adjoint of that transformation. For matrix multiplication the Hermitian adjoint is just given by multiplication of the transposed matrix. This is especially useful for more complicated transformations like convolutions, just imagine doing these calculations on the completely written out expression of a convolution. This also explains the duality between summation and replication mentioned at 01:09:50 since the adjoint of summation is replication (and vice versa). 01:24:26 I struggled through everything to make sure I found answers before seeing the video solution, the one-line solution I got for dC was:dC = F.one_hot(Xb.view(-1), num_classes=C.shape[0]).float().T @ demb.view(-1, C.shape[-1])Don't ask me for an intuitive explanation, I just fiddled until something worked... (sort-of inspired by how earlier on in the series you showed that C[Xb] is just a more efficient version of using F.one_hot with matrix multiplication)Also, for whatever reason I can't get dhpreact to be exact, only approximate, even using your exact same code to calculate it? So I just ended up doing dhpreact = hpreact.grad#(1.0 - h**2) * dh to make sure its effect didn't cascade further down the gradient calculations. Any idea why this would be the case? 00:36:30 explained on 01:50:00 i'm not totaly sure that this is a good solution to calculate the derivative using "bnraw" variable, since it is calculated in the later steps of BN. Thus, there's no use in hpreact_fast as we have to do all the same arithmetics in parallel in order to fetch bnraw.My solution is not the best one, but still: 01:36:05 He really made me realize something at it kicked in 🔥 01:47:50 At the camera start giving up... so do I... 🤣 01:24:22 One-liner for dC ():dC = (F.one_hot(Xb, num_classes=C.shape[0]).unsqueeze(-1) * demb.unsqueeze(2)).sum((0, 1)) 01:26:00 : To backprop through the embedding matrix C, I used the following quick code, which does not need a for loop: 01:25:21 To eliminate the for loop at time , I found this after a little searching. Very little experience with pytorch, so take with a grain of salt:dembflat = demb.view(-1,10)Xbflat = Xb.view(-1)dC = torch.zeros_like(C)dC.index_add_(0,Xbflat,dembflat) 01:24:25 I was able to accumulate the dC without a "for" loop using this code:dC = torch.zeros_like(C)dC.index_add_(0, Xb.flatten(), demb.flatten(0, 1)) 00:49:45 I arrived at dh just figuring out by the size of the matrix, and then I continued with your video and you just did all the derivatives and I taught... I am so dumb, I should I have done that, but then you say " now I tell you a secret I normally do... .... hahahahahhaha 01:30:26 At (exercise 2, near the end, while deriving dlogits for i != y): why did you substitute 0 for e**lj ? 00:43:40 Thats so Cute. 😆 01:26:08 So great videos, thank you so much! I tried to simplify dC (at in the video), but failed after some time, so asked chatgpt, and here is the shiny simple result: 01:09:44 best part 01:18:20 /n * dbnmeani``` during . Makes no difference mathematically, but theres nothing like finding oopsies in your code :P 01:30:13 The calculus at is way too complicated. Start with -log( e^l_i/sum_j e^l_j ) = -l_i + log(sum_j e^l_j) before you differentiate. d -l_i/dl_y = -1 if i=y of course and d -log(sum_j e^l_j)/d l_y = -e^l_y / sum_j e^l_j = softmax(l_y) and you're done. 00:04:38 @ "it was barely a programming language" 00:45:25 At , Low Budget Production LOL 00:27:25 andrej fard 01:25:00 Optimised dC calculation() instead of the for loop 01:30:38 I'm really confused about calculations at (the lower part of the paper about `if i <> j` etc). It says 'product rule, power rule, ..." How do I use product rule to take a derivative of Softmax?PS I asked ChatGPT and it explained to me that I need to use Quotient rule :) 01:24:21 I believe the loop implementing the final derivative at can be vectorized if you just rewrite the selection operation as a matrix operation, then do a matmul derivative like done elsewhere in the video: 01:12:06 i've noticed that althoughdbnvar/(n-1) # (1, 64) doesn't have the same size as the bndiff2 term (32, 64), it still works fine during the backprop, because (1,64) vector broadcasts well on (32,64).And such solution is more optimal from the perspective of storage and calculation 01:25:00 Here is the better implementation of the code: 00:43:40 sprinkling Andrej magic through out the video - had me cracking at 01:06:20 your attention to detail here on the variance of arrays is out of this world 01:25:47 P.S.: dC can be done with dC.index_add_(0, Xb.view(-1), demb.view(-1, 10)) ;) 01:45:54 Question: At , you conclude in the last derivation step that d sigma^2 / d x_i = 2 / (m-1) * (x_i- mu). This would be correct if mu were just a constant, but in fact, mu is also a function of x_i: mu(x_i) = 1/m. So how does this cancel out so that you still end up with your simple expression? 00:53:55 if you scroll down, Wolfram Alpha provides 1 - x^2 + 2/3x^4 + O(x^5) as series expansion at x=0 of the derivative of tanh(x), which is the same as the series expansion for 1-tanh(x)^2. 01:26:00 can be vectorized using: dC = dC.index_add_(0, Xb.view(-1), demb.view(-1, C.shape[1])) 01:16:00 At around dbnmeani should probably have keepdim=True, since otherwise you're removing the row dimension making it of shape [64], while bnmeani was originally [1, 64]. But I guess it still magically works because of broadcasting in the backprop and in the cmp :) 01:33:59 Question: Why is this () true not only for dlogits, but also for dW2, db2, db1, and not true for dW1? 01:24:21 Thank you!Also, my implementation for dC atdC = torch.zeros_like(C)dC[Xb] += demb 00:40:54 Is there a disadvantage to using (logits == logit_maxes).float() to pick out the maximum indices at ? 01:16:20 Such a great video for really understanding the detail under the hood! And lol at the momentary disappointment at just before realizing the calculation wasn't complete yet 😂 01:12:20 around , a simpler approach might be to just directly multiply like this: dbndiff2 = 1/(n-1) * dbnvar 01:08:17 The reason for using biased variance in training and unbiased during inference(running var estimation) is that during the training in one mini-batch we don't care about the complete dataset. The mini-batch is enough as it is the one at the moment we are working on. In the code also you are using the mean and var of that moment to run batchnorm. But during inference we need the mean and variance of the complete data, that is what the bessel's correction is for. If we have the access to the complete data we don't need to use the Bessel's correction, we have the full data. But if we are using small sample(mini-batch) to estimate the variance of the complete data we need Bessel's correction. If we used direct variance calculation instead of this running var we can completely skip the Bessel's correction. 00:12:37 Love your lectures, they are crystal clear. From , I only find the notation dlogprobs (et similia) a bit misleading, since it denotes the derivative of the loss with respect to the parameters logprobs. I would use something more verbose like dloss_dlogprobs. However, I understand you did it for coherence with torch. 01:43:08 in the WHOA:) part, should there be a "-" in front of 2? although it doesn't really matter as the final result is 0. but why is it? 01:30:11 at (just under the separation line for i≠y v i=y)?I understand from the above line that we are looking for the derivative of e^ly / Σe^lj. So, when we consider the denominator we would get e^ly * -(Σe^lj)^-2 = -e^ly / (Σe^lj)^2 but the solution multiplies it by e^li which I do not quite get. Cheers! 00:20:00 ) for the full batch. Whereas in your answer in the videoat it's of size (32,27) only. Can you please clear this confusion for me Andrej? I think there's some fundamental flaw in my understanding 😭😭Is it because in the end we are calling .backward() on a scalar value? 😭 00:20:00 are logprobs and logits same? at 01:24:30 I managed to come up with a *vectorized* solution and it's just one line of code!*dC = F.one_hot(Xb.reshape(-1), num_classes=27).float().T @ demb.reshape((-1, n_emb))* Building makemore Part 4: Becoming a Backprop Ninja We take the 2-layer MLP (with BatchNorm) from the previous video and backpropagate through it manually without using PyTorch autograd's loss.backward(): through the cross entropy loss, 2nd linear layer, tanh, batchnorm, 1st linear layer, and the embedding table. Along the way, we get a strong intuitive understanding about how gradients flow backwards through the compute graph and on the level of efficient Tensors, not just individual scalars like in micrograd. This helps build competence and intuition around how neural nets are optimized and sets you up to more confidently innovate on and debug modern neural networks. !!!!!!!!!!!! I recommend you work through the exercise yourself but work with it in tandem and whenever you are stuck unpause the video and see me give away the answer. This video is not super intended to be simply watched. The exercise is here: https://colab.research.google.com/drive/1WV2oi2fh9XXyldh02wupFQX0wh5ZC-z-?usp=sharing !!!!!!!!!!!! Links: - makemore on github: https://github.com/karpathy/makemore - jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part4_backprop.ipynb - collab notebook: https://colab.research.google.com/drive/1WV2oi2fh9XXyldh02wupFQX0wh5ZC-z-?usp=sharing - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - our Discord channel: https://discord.gg/3zy8kqD9Cp Supplementary links: - Yes you should understand backprop: https://karpathy.medium.com/yes-you-should-understand-backprop-e2f06eab496b - BatchNorm paper: https://arxiv.org/abs/1502.03167 - Bessel’s Correction: http://math.oxford.emory.edu/site/math117/besselCorrection/ - Bengio et al. 2003 MLP LM https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf Chapters: 00:00:00 intro: why you should care & fun history 00:07:26 starter code 00:13:01 exercise 1: backproping the atomic compute graph 01:05:17 brief digression: bessel’s correction in batchnorm 01:26:31 exercise 2: cross entropy loss backward pass 01:36:37 exercise 3: batch norm layer backward pass 01:50:02 exercise 4: putting it all together 01:54:24 outro #deep learning #backpropagation #neural network #language model #chain rule #tensors https://favtu.be/timelines/v/P6sfmUTpUmc Wed, 05 Oct 22 01:41:03 +0900 00:00:00 [-1. Implementing and refactoring neural networks for language modeling 00:03:21 ] 00:00:00 -[- 00:00:30 ]Continuing makemore implementation with multilayer perceptron for character-level language modeling, planning to move to larger neural networks. 00:00:31 -[- 00:01:03 ]Understanding neural net activations and gradients in training is crucial for optimizing architectures. 00:02:06 -[-11,000 parameters over 200,000 steps, achieving train and val loss of 2.16. 00:02:46 ]Refactored code to optimize neural net with 00:03:03 -[- 00:03:28 ]Using torch.nograd decorator to prevent gradients computation. 00:03:22 [-2. Efficiency of torch.no_grad and neural net initialization issues 00:14:22 ] 00:03:22 -[- 00:04:00 ]Using torch's no_grad makes computation more efficient by eliminating gradient tracking. 00:04:22 -[-27, rapidly decreases to 1 or 2. 00:04:50 ]Network initialization causes high loss of 00:05:00 -[-27 characters, with roughly 1/27 probability for each. 00:05:32 ]At initialization, the model aims for a uniform distribution among 00:05:49 -[- 00:06:19 ]Neural net creates skewed probability distributions leading to high loss. 00:12:08 -[-2.12-2.16 00:12:36 ]Loss at initialization as expected, improved to 00:14:24 [-3. Neural network initialization 00:36:39 ] 00:16:03 -[-1 or 1, leading to a halt in back propagation. 00:16:31 ]The chain rule with local gradient is affected when outputs of tanh are close to - 00:18:09 -[- 00:18:38 ]Concern over destructive gradients in flat regions of h outputs, tackled by analyzing absolute values. 00:26:03 -[-2.17 to 2.10 by fixing softmax and 10-inch layer issues. 00:26:31 ]Optimization led to improved validation loss from 00:29:28 -[- 00:30:02 ]Standard deviation expanded to three, aiming for unit gaussian distribution in neural nets. 00:30:17 -[-0.2 shrinks gaussian with standard deviation 0.6. 00:30:47 ]Scaling down by 00:31:03 -[- 00:31:46 ]Initializing neural network weights for well-behaved activations, kaiming he et al. 00:36:24 -[- 00:36:55 ]Modern innovations have improved network stability and behavior, including residual connections, normalization layers, and better optimizers. 00:36:39 [-4. Neural net initialization and batch normalization 00:51:52 ] 00:36:39 -[- 00:37:05 ]Modern innovations like normalization layers and better optimizers reduce the need for precise neural net initialization. 00:40:32 -[- 00:43:04 ]Batch normalization enables reliable training of deep neural nets, ensuring roughly gaussian hidden states for improved performance. 00:40:51 -[-2015 enabled reliable training of deep neural nets. 00:41:13 ]Batch normalization from 00:41:39 -[- 00:42:09 ]Standardizing hidden states to be unit gaussian is a perfectly differentiable operation, a key insight in the paper. 00:43:20 -[- 00:43:50 ]Calculating standard deviation of activations, mean is average value of neuron's activation. 00:45:45 -[- 00:46:16 ]Back propagation guides distribution movement, adding scale and shift for final output 00:51:52 [-5. Jittering and batch normalization in neural network training 01:01:35 ] 00:52:10 -[- 00:52:37 ]Padding input examples adds entropy, augments data, and regularizes neural nets. 00:53:44 -[- 00:54:09 ]Batch normalization effectively controls activations and their distributions. 00:56:05 -[- 00:56:33 ]Batch normalization paper introduces running mean and standard deviation estimation during training. 01:00:46 -[- 01:01:10 ]Eliminated explicit calibration stage, almost done with batch normalization, epsilon prevents division by zero. 01:01:36 [-6. Batch normalization and resnet in pytorch 01:09:21 ] 01:02:00 -[- 01:02:30 ]Biases are subtracted out in batch normalization, reducing their impact to zero. 01:03:13 -[- 01:03:53 ]Using batch normalization to control activations in neural net, with gain, bias, mean, and standard deviation parameters. 01:07:25 -[- 01:07:53 ]Creating deep neural networks with weight layers, normalization, and non-linearity, as exemplified in the provided code. 01:09:21 [-7. Pytorch weight initialization and batch normalization 01:23:37 ] 01:10:05 -[-1/fan-in square root from a uniform distribution. 01:10:32 ]Pytorch initializes weights using 01:11:11 -[-1 over sqrt of fan in, using batch normalization layer in pytorch with 200 features. 01:11:40 ]Scaling weights by 01:14:02 -[- 01:14:35 ]Importance of understanding activations and gradients in neural networks, especially as they get bigger and deeper. 01:16:00 -[- 01:16:30 ]Batch normalization centers data for gaussian activations in deep neural networks. 01:17:32 -[-2015, enabled reliable training of much deeper neural nets. 01:18:02 ]Batch normalization, influential in 01:23:39 [-8. Custom pytorch layer and network analysis 01:55:56 ] 01:24:01 -[- 01:24:32 ]Updating buffers using exponential moving average with torch.nograd context manager. 01:25:47 -[-46,000 parameters and uses pytorch for forward and backward passes, with visualizations of forward pass activations. 01:27:11 ]The model has 01:28:04 -[-20% initially, then stabilizes at 5% with a standard deviation of 0.65 due to gain set at 5 over 3. 01:28:30 ]Saturation stabilizes at 01:33:19 -[-1 prevents shrinking and diffusion in batch normalization. 01:33:50 ]Setting gain correctly at 01:38:41 -[-100 times greater, causing faster training, but it self-corrects with longer training. 01:39:11 ]The last layer has gradients 01:43:18 -[-3 on log plot. 01:43:42 ]Monitoring update ratio for parameters to ensure efficient training, aiming for - 01:51:36 -[- 01:52:04 ]Introduce batch normalization and pytorch modules for neural networks. 01:52:39 -[- 01:53:06 ]Introduction to diagnostic tools for neural network analysis. 01:54:45 -[- 01:55:50 ]Introduction to diagnostic tools in neural networks, active research in initialization and backpropagation, ongoing progress 00:04:25 - Initial loss (): High initial loss (e.g., 27) indicates improper network initialization.- Softmax logits should be close to zero at initialization to produce a uniform probability distribution and expected loss.- This avoids confident mispredictions and the "hockey stick" loss curve. 00:09:28 - Scaling down weights of the output layer can achieve this (). 00:13:09 - Saturated activations (): Tanh activations clustered around -1 and 1 indicate saturation, hindering gradient flow.- Saturated neurons update less frequently and impede training. 00:19:19 - This can lead to dead neurons, which never activate and don't learn (). 00:24:59 - Scaling down weights of the hidden layer can help prevent saturation (). 00:27:58 - Kaiming initialization (): A principled approach to weight scaling, aiming for unit gaussian activations throughout the network. 00:31:46 - Calculates standard deviation based on fan-in and gain factor specific to the non-linearity used (). 00:33:56 - PyTorch offers torch.nn.init.kaiming_normal_ for this (). 00:40:49 *Batch Normalization (****):*- Concept: Normalizes activations within each batch to be roughly unit gaussian.- Controls activation scale, stabilizing training and mitigating the need for precise weight initialization. 00:51:55 - Offers a regularization effect due to coupling examples within a batch (). 00:42:17 - Implementation (): 00:42:41 - Normalizes activations by subtracting batch mean and dividing by batch standard deviation (). 00:45:54 - Learnable gain and bias parameters allow the network to adjust the normalized distribution (). 00:54:38 - Running mean and variance are tracked during training and used for inference ().- Caveats: 00:50:20 - Couples examples within a batch, leading to potential bugs and inconsistencies (). 00:54:03 - Requires careful handling at inference time due to batch dependency (). 01:01:37 - Makes bias terms in preceding layers redundant (). 01:18:40 *PyTorch-ifying the code (****):* 01:19:26 - Code is restructured using torch.nn.Module subclasses for linear, batch normalization, and tanh layers ().- This modular approach aligns with PyTorch's structure and allows easy stacking of layers. 01:08:52 - Default PyTorch initialization schemes and parameters are discussed (). 01:19:13 *Diagnostic Tools (****):* 01:26:53 - Visualization of statistics: Histograms of activations, gradients, weights, and update:data ratios reveal potential issues during training (). 01:26:53 - Forward pass activations: Should exhibit a stable distribution across layers, indicating proper scaling (). 01:30:57 - Backward pass gradients: Should be similar across layers, signifying balanced gradient flow (). 01:36:20 - Parameter weights: Distribution and scale should be monitored for anomalies and asymmetries (). 01:39:56 on a log scale, indicating a good learning rate and balanced parameter updates (). 00:00:00 intro 00:01:22 starter code 00:04:19 fixing the initial loss 00:12:59 fixing the saturated tanh 00:27:53 calculating the init scale: “Kaiming init” 00:40:40 batch normalization 01:03:07 batch normalization: summary 01:04:50 real example: resnet50 walkthrough 01:14:10 summary of the lecture 01:18:35 just kidding: part2: PyTorch-ifying the code 01:26:51 viz #1: forward pass activations statistics 01:30:54 viz #2: backward pass gradient statistics 01:32:07 the fully linear case of no non-linearities 01:36:15 viz #3: parameter activation and gradient statistics 01:39:55 viz #4: update:data ratio over time 01:46:04 bringing back batchnorm, looking at the visualizations 01:51:34 summary of the lecture for real this time 00:44:30 std should be a centralized moment (i.e. subtract the mean first) according to the paper although I see that PyTorch imp is the same as yours 01:40:25 did you try using log L2 norm ratio here instead of std? you're using variance as a proxy for how big updates are w.r.t. data values 00:15:14 Andrej's transformation between and 00:15:16 was pretty quick 😉 01:11:03 reason they're doing this is if you have a roughly gsan input this will ensure that out of this layer you will have a 01:11:10 roughly Gan output and you you basically achieve that by scaling the weights by 00:51:50 Can anyone explain what he has said from to 00:53:00 ? 01:26:25 @ I'd use emb.flatten(1, 2) instead of emb.view(emb.shape[0], -1) to combine two last dimensions into one. It feels that it is better to avoid shape lookup - emb.shape[0] 01:10:23 Great video, I loved it. Just a question. In the Linear layer on PyTorch at , he says that to initialise the weights the uniform distribution is used, but then in the implementation of the Linear layer when PyTorch-ifying the code he uses the Normal distribution. Did I loose something or he committed a "mistake" ? 01:28:58 ? Anyone please explain at 01:40:38 can someone explain why we divide std of gradient to std of data instead of using mean? Weight update ratio = grad*learning_rate/weight_value. As we have multiple inputs and multiple entries in batch, we could take mean to calculate single value, cannot figure out how std is a better option. 00:36:00 modern inovations that makes things stable and that makes us not be super detailed and careful with the gradient and backprop issues. (self-note) 00:15:10 the night and day shift 01:02:13 I can't understand why removing the mean removes the effect of adding a bias? Why would the grad be zero? 01:25:23 Why is the last layer made "less confident like we saw" and where did we see this? 01:07:06 also I would add that ReLU is much easier to compute (max of 2 values and derivative is eighter 0 or 1) than tanh where we have exponents 00:58:55 is basically an Infinite Impulse Response (IIR) filter 01:38:45 "That's problematic because in that simple stochastic gradient setup you would be training this last layer 10x faster with respect to the other layers". Why 10x faster? 01:30:36 5/3=1.66... is pretty close to the golden ratio 1.61803. Coincidence? 00:15:15 so, no one is going to talk about how andrej grew a decade younger 🤔 01:18:35 always gets me 00:45:20 Around , wouldn't adding scale and shift revert the previous normalization? Improper scale and shift parameters will still cause activation saturated. 00:59:00 Can any one please tell that at , why did we take the numbers 0.999 and 0.001 specifically? I am new to neural networks and all of this is a bit overwhelming. Thanks 00:07:06 At , it's a bit subtle why it's called a loss because it's not immediately apparent with respect to what it is a loss. It seems it's the loss resulting from choosing the character having index i given the probability distribution stored in the tensor. 01:40:49 Why stddev here? Wouldn't we want to use something like the L1-norm? Also, wouldn't we want to log this metric before updating the parameters? 01:33:30 The reason the gradients of the higer layers have a bigger deviation (in the absence of tanh layer), is that you can write the whole NN as a sum of products, and it is easy to see that each weight of Layer 0 appears in 1 term, of layer 1 in 30 terms, of layer 2 in 3000 terms and so on. Therefore a small change of a weight in higer layers changes the output more. 01:18:36 The "Okay, so I lied" moment was too relatable xD 01:18:30 He says 'Bye', but looking at the time, it seems too early []. Most people don't want lectures to be long, but I'm happy this one didn't end there. 00:45:30 Just to be clear, normalising the pre-activation neurons to have 0 mean and 1 std does not make them Gaussian distributed. The sum is only Gaussian distributed at initialisation, because we have initialised the weights to be normally distributed. 01:35:48 Does anyone know the paper about "analyzing infinitely linear layers"? Andrej mentioned in the video 00:25:43 @Andrej Karpathy Great video! A quick question: at Why a U shape is better than a Cup shape for the histogram of h? Don't we want h to be have some normal distribution, like hpreact? 00:38:07 The standard deviation relation used to rescale the inital weights, will this only work in the case that the input data also has variance approximately 1 right? 01:04:35 At would it help at the end of the training to optimize with bnmean_running and bnstd_running to normalize the preactivations hpreact? Maybe at that point regularization isn't necessary anymore and the rest of the weights can be optimized for the particular batch norm calibration that will be used during inference. 01:40:46 At why do we use standard deviation to calculate update to data ratio? 01:35:59 There is one doubt I have @ and that is regarding the condition p.dim==2, I don't understand why this was done and which parameters it will filter out? 01:30:00 Around the mark, I think I missed why some saturation (around 5%) is better than no saturation at all. Didn't saturation impede further training? Perhaps he just meant that 5% is low enough, and that's the best we can do if we want to avoid deeper activations from converging to zero? 00:58:07 @Andrej Karpathy At , bnmean_running = (0.999 * bnmean_running) + (0.001 * bnmeani), why are you multiplying 0.999 with bnmean_running and 0.001 with bnmeani. Why this not works *bnmean_running = bnmean_running + bnmeani* 01:18:59 I don't understand where the layers are organized by putting a tanh after each linear layer while the initialization of the linear layer is `self.weight = torch.randn((fan_in, fan_out), generator=g) / fan_in**0.5`. I think it's not Kaiming initialization, because the gain for tanh is `5/3`, but in the code it's set to `1`, 01:30:28 I'm at , so haven't finished yet. But something is unclear: what's the point of stacking these layers instead of having just one Linear and one Tanh? Since tanh squashes and afterwards we're diffusing, it seems to me like we're doing accordion-like work unnecessarily. What is the benefit we're getting? 00:06:06 Hi Andrej, Thank you for the amazing set of lectures which elucidate multiple aspects of training a ML model. In the video you mention that at the beginning of training, we expect the NN to have all equal probabilities I.e. 1/27 which implies that all logins should be close to 0. Using this logic you arrive at the fact that weight matrices should be initialized close to 0. How does one think about this for regression problems like autoencoders? What would a “good” starting output be? Is it still all zeros? 01:30:10 The 5/3 gain in the tanh comes for the average value of tanh^2(x) where x is distributed as a Gaussian, i.e. 00:15:13 not only sweating but also loosing hair :) Building makemore Part 3: Activations & Gradients, BatchNorm We dive into some of the internals of MLPs with multiple layers and scrutinize the statistics of the forward pass activations, backward pass gradients, and some of the pitfalls when they are improperly scaled. We also look at the typical diagnostic tools and visualizations you'd want to use to understand the health of your deep network. We learn why training deep neural nets can be fragile and introduce the first modern innovation that made doing so much easier: Batch Normalization. Residual connections and the Adam optimizer remain notable todos for later video. Links: - makemore on github: https://github.com/karpathy/makemore - jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part3_bn.ipynb - collab notebook: https://colab.research.google.com/drive/1H5CSy-OnisagUgDUXhHwo1ng2pjKHYSN?usp=sharing - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - Discord channel: https://discord.gg/3zy8kqD9Cp Useful links: - "Kaiming init" paper: https://arxiv.org/abs/1502.01852 - BatchNorm paper: https://arxiv.org/abs/1502.03167 - Bengio et al. 2003 MLP language model paper (pdf): https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf - Good paper illustrating some of the problems with batchnorm in practice: https://arxiv.org/abs/2105.07576 Exercises: - E01: I did not get around to seeing what happens when you initialize all weights and biases to zero. Try this and train the neural net. You might think either that 1) the network trains just fine or 2) the network doesn't train at all, but actually it is 3) the network trains but only partially, and achieves a pretty bad final performance. Inspect the gradients and activations to figure out what is happening and why the network is only partially training, and what part is being trained exactly. - E02: BatchNorm, unlike other normalization layers like LayerNorm/GroupNorm etc. has the big advantage that after training, the batchnorm gamma/beta can be "folded into" the weights of the preceeding Linear layers, effectively erasing the need to forward it at test time. Set up a small 3-layer MLP with batchnorms, train the network, then "fold" the batchnorm gamma/beta into the preceeding Linear layer's W,b by creating a new W2, b2 and erasing the batch norm. Verify that this gives the same forward pass during inference. i.e. we see that the batchnorm is there just for stabilizing the training, and can be thrown out after training is done! pretty cool. Chapters: 00:00:00 intro 00:01:22 starter code 00:04:19 fixing the initial loss 00:12:59 fixing the saturated tanh 00:27:53 calculating the init scale: “Kaiming init” 00:40:40 batch normalization 01:03:07 batch normalization: summary 01:04:50 real example: resnet50 walkthrough 01:14:10 summary of the lecture 01:18:35 just kidding: part2: PyTorch-ifying the code 01:26:51 viz #1: forward pass activations statistics 01:30:54 viz #2: backward pass gradient statistics 01:32:07 the fully linear case of no non-linearities 01:36:15 viz #3: parameter activation and gradient statistics 01:39:55 viz #4: update:data ratio over time 01:46:04 bringing back batchnorm, looking at the visualizations 01:51:34 summary of the lecture for real this time #neural network #deep learning #makemore #batchnorm #batch normalization #pytorch https://favtu.be/timelines/v/TCH_1BHY58I Mon, 12 Sep 22 23:43:06 +0900 00:00:00 intro 00:01:48 Bengio et al. 2003 (MLP language model) paper walkthrough 00:09:03 (re-)building our training dataset 00:12:19 implementing the embedding lookup table 00:18:35 implementing the hidden layer + internals of torch.Tensor: storage, views 00:29:15 implementing the output layer 00:29:53 implementing the negative log likelihood loss 00:32:17 summary of the full network 00:32:49 introducing F.cross_entropy and why 00:37:56 implementing the training loop, overfitting one batch 00:41:25 training on the full dataset, minibatches 00:45:40 finding a good initial learning rate 00:53:20 splitting up the dataset into train/val/test splits and why 01:00:49 experiment: larger hidden layer 01:05:27 visualizing the character embeddings 01:07:16 experiment: larger embedding size 01:11:46 summary of our final code, conclusion 01:13:24 sampling from the model 01:14:55 google collab (new!!) notebook advertisement 00:01:34 PS. At I was just uber curious about his previous searches, so I google them: 00:27:27 proverbs>You will have plenty of goats’ milk to feed your family and to nourish your female servants. 00:27:27 matthew -31>Then the governor’s soldiers took Jesus into the Praetorium and gathered the whole company of soldiers around him. They stripped him and put a scarlet robe on him, and then twisted together a crown of thorns and set it on his head. They put a staff in his right hand. Then they knelt in front of him and mocked him. “Hail, king of the Jews!” they said. They spit on him, and took the staff and struck him on the head again and again. After they had mocked him, they took off the robe and put his own clothes on him. Then they led him away to crucify him. 00:14:00 : 00:22:08 : 01:00:00 : 00:00:27 [<1809.89it/s]Last Loss: 2.403459072113037Best Loss: 1.4457638263702393 At Epoch: 25480============================================================ 00:00:00 , 00:59:01 Thank you for the lectures! @ Made me chuckle 00:20:25 -dimensional vertically scrollable space to describe the functions of PyTorch () 00:44:25 life lesson: much better to have an approximate gradient and take many steps than have an exact gradient and take a few steps 00:45:40 I don't quite understand the part of finding a good initial learning rate. Why the lowest point of loss value indicates the best learning rate? It takes some time for the loss value to decrease, right? 00:30:03 What's tanh? 00:50:42 Can anyone explain to me, why looking at loss plotted against exponent of the learning rate () the conclusion is that lr<0.1 "is way too low"? For me, its where the loss is actually getting lower, isn't it? 00:23:47 what about just `emb_reshaped = emb.reshape((emb.shape[0], emb.shape[1]*emb.shape[2]))` ? 01:14:56 Adrej is learning youtube tricks 😅 01:07:20 : it should be 10 dimensional embeddings for each *character* not word in this character-level language model. 00:31:47 "ideally all of these numbers here of course are one because then we are correctly predicting the next character" hmmmmmm it's reasonable to say these numbers are high, put not one, If the probability here is one, that will exclude any chance of other characters having similar context. 00:24:50 Of course! Memory itself is a one dimensional "tensor". :D 01:10:09 you shouldn't have plotted stepi variable against the loss :D it could have worked if you'd ploted out just plt.plot(loss_history) or applied two different colours for those two runs 00:17:28 Every time I think I finally understand what's happening, he does something like this: 😅 01:02:15 I also just noticed, he explicitly mentions these fluctuations at . Doh! 01:10:30 The plot of the steps and losses after running the training loop multiple times (~ mins) https://youtu.be/TCH_1BHY58I?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ&t=4233) would be wrong because stepi array keeps appending the same indices [0, 50000). I expect the graph to just start getting more unstable and unstable. 00:59:15 It can take days!! How can someone sleep with such pressure 00:03:25 Why space is small? Even in two-dimensional space you can place an infinite number of points 00:37:00 Since probs are invariant to an offset applied to logits, it's fun to plot the drift in the mean or sum of b2. Looks like Brownian motion. 00:48:45 At Andrej says that the learning rate would be low in the beginning and high at the end. Why was it set like that? My intuition is that the learning rate should be in the opposite order. 00:53:20 To break the data to training, developement and test, one can also use torch.tensor_split.n1 = int(0.8 * X.shape[0])n2 = int(0.9 * X.shape[0])Xtr, Xdev, Xts = X.tensor_split((n1, n2), dim=0)Ytr, Ydev, Yts = Y.tensor_split((n1, n2), dim=0) 00:45:00 Awesome videos, thank you for that! I have a question though about , "finding a good initial learning rate", which is either a mistake in the video or I misunderstood something. 00:25:36 for the PyTorch internals video (@) 00:29:20 we can also use torch.reshape() to get the right shape for W. However, there is a difference between torch.view and torch.reshapeTL;DR:If you just want to reshape tensors, use torch.reshape. If you're also concerned about memory usage and want to ensure that the two tensors share the same data, use torch.view. 00:45:40 @ 'Finding a good initial learning rate', each learning rate is used just one time. The adjustment of the parameter of one learning rate is based on the parameters already adjusted using the prior smaller learning rates. I feel that each of the 1,000 learning rate candidates should go through the same number of iterations. Then, the losses at the end of the iterations are compared. Please tell me if I am wrong. Thanks! 00:50:30 Great video! One question, @AndrejKarpathy: around or so you show how to graph an optimal learning rate and ultimately you determine that the 0.1 you started with was pretty good. However, unless I'm misunderstanding your code, aren't you iterating over the 1000 different loss function candidates while *simultaneously* doing 1000 consecutive passes over the neural net? Meaning, the loss will naturally be lower during later iterations since you've already done a bunch of backward passes, so the biggest loss improvements would always be stacked towards the beginning of the 1000 iterations, right? Won't that bias your optimal learning rate calculation towards the first few candidates? 01:06:56 Fascinating how the vowels end up clustered together! 00:25:40 Please create the "entire video about the internals of pytorch" that you mentioned in . And thank you so much for the content, Andrej !! 00:21:24 at I think it's supposed to be first letter not first word. It's first word in the paper but first letter in the example 00:49:22 I believe that at the losses and the learning rates are misaligned.The first loss (derived from completely random weights) is computed before the first learning rate is used, and therefor the first learning rate should be aligned with the second loss.You can simply solve this problem by using this snippet;lri = lri[:-1]lossi = lossi[1:] 00:45:45 On I was waiting fot Karpathy's constant to appear. Thank you for the lecture, Andrej 00:21:39 At , when he says words does he mean the 3 character sequence that was made by block size? And, so, when he refers to the picture behind him, does he mean each of those three blocks represents a indice in the block_size array? 00:27:24 at minute mark at the moment and gotta say, pytorch is amazing. so wonderful how easy they make it for devs with those small tricks. 01:05:00 around - the reason why we're not "overfitting" with the larger number of params might be the context size. with a context of 3, no number of params will remove the inherent uncertainty. 00:41:30 I don't understand the mini batching happening at . when using ix = torch.randint(0,X.shape,(32,)), and using this to index into X, you are just picking 32 data examples from X, not batching all of the data right? I thought by batching, you taking a batch of data, do a forward pass on all items in the batch, take the mean output and do back prop on that mean result outcome and update the model on that loss. Here I feel like Andrej is just selecting 32 individual data examples. Please do correct me if I'm wrong! I'm new to ML! 00:56:17 I'm confused at why care must be taking with how many times you can use the test dataset as the model will learn from it. Is this because there is no equivalent of 'torch.no_grad()' for LLMs - will the LLM always update the weights when given data? 00:50:00 Question about , in the plot, y axis is the loss, and the x axis is learning rate, but x axis is also the step number. How do you know whether the y axis change is because of learning rate difference or step number increase? 00:45:34 It seems it is slightly different from the approach presented here. Looking at the , it looks like for each iteration, we randomly select a min batch of size 32 from the whole training set, and update the parameters, then go on to the next iteration. 00:37:14 , who would tell you this when you are reading from a book. Exceptional teaching ability 00:34:05 re: using cross_entropy function around , it sounds like pytorch takes the derivate of each step of exponentiation then normalization instead of simplifying them before taking the derivative. is that a "soft" limitation of the implementation in that a procedure could be defined to overcome it, or is there a bit of an mathematical intuition needed to understand how to rewrite the function to produce a simpler derivative? 00:38:00 pfeeeewwww 😳 Building makemore Part 2: MLP We implement a multilayer perceptron (MLP) character-level language model. In this video we also introduce many basics of machine learning (e.g. model training, learning rate tuning, hyperparameters, evaluation, train/dev/test splits, under/overfitting, etc.). Links: - makemore on github: https://github.com/karpathy/makemore - jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part2_mlp.ipynb - collab notebook (new)!!!: https://colab.research.google.com/drive/1YIfmkftLrz6MPTOO9Vwqrop2Q5llHIGK?usp=sharing - Bengio et al. 2003 MLP language model paper (pdf): https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - (new) Neural Networks: Zero to Hero series Discord channel: https://discord.gg/3zy8kqD9Cp , for people who'd like to chat more and go beyond youtube comments Useful links: - PyTorch internals ref http://blog.ezyang.com/2019/05/pytorch-internals/ Exercises: - E01: Tune the hyperparameters of the training to beat my best validation loss of 2.2 - E02: I was not careful with the intialization of the network in this video. (1) What is the loss you'd get if the predicted probabilities at initialization were perfectly uniform? What loss do we achieve? (2) Can you tune the initialization to get a starting loss that is much more similar to (1)? - E03: Read the Bengio et al 2003 paper (link above), implement and try any idea from the paper. Did it work? Chapters: 00:00:00 intro 00:01:48 Bengio et al. 2003 (MLP language model) paper walkthrough 00:09:03 (re-)building our training dataset 00:12:19 implementing the embedding lookup table 00:18:35 implementing the hidden layer + internals of torch.Tensor: storage, views 00:29:15 implementing the output layer 00:29:53 implementing the negative log likelihood loss 00:32:17 summary of the full network 00:32:49 introducing F.cross_entropy and why 00:37:56 implementing the training loop, overfitting one batch 00:41:25 training on the full dataset, minibatches 00:45:40 finding a good initial learning rate 00:53:20 splitting up the dataset into train/val/test splits and why 01:00:49 experiment: larger hidden layer 01:05:27 visualizing the character embeddings 01:07:16 experiment: larger embedding size 01:11:46 summary of our final code, conclusion 01:13:24 sampling from the model 01:14:55 google collab (new!!) notebook advertisement #deep learning #neural network #multilayer perceptron #nlp #language model https://favtu.be/timelines/v/PaCmpygFfXo Thu, 08 Sep 22 04:14:47 +0900 00:00:00 intro 00:03:03 reading and exploring the dataset 00:06:24 exploring the bigrams in the dataset 00:09:24 counting bigrams in a python dictionary 00:12:45 counting bigrams in a 2D torch tensor ("training the model") 00:18:19 visualizing the bigram tensor 00:20:54 deleting spurious (S) and (E) tokens in favor of a single . token 00:24:02 sampling from the model 00:36:17 efficiency! vectorized normalization of the rows, tensor broadcasting 00:50:14 loss function (the negative log likelihood of the data under our model) 01:00:50 model smoothing with fake counts 01:02:57 PART 2: the neural network approach: intro 01:05:26 creating the bigram dataset for the neural net 01:10:01 feeding integers into neural nets? one-hot encodings 01:13:53 the "neural net": one linear layer of neurons implemented with matrix multiplication 01:18:46 transforming neural net outputs into probabilities: the softmax 01:26:17 summary, preview to next steps, reference to micrograd 01:35:49 vectorized loss 01:38:36 backward and update, in PyTorch 01:42:55 putting everything together 01:47:49 note 1: one-hot encoding really just selects a row of the next Linear layer's weight matrix 01:50:18 note 2: model smoothing as regularization loss 01:54:31 sampling from the neural net 01:56:16 conclusion 00:52:50 At you said if we have a very good model, we'd expect that the probabilities of those bigrams should be near 1, I don't get what do you mean by that? 01:03:32 EDIT: I got that, a good model would give high probability (near to 1) to actual next characters in the all the bigram of the dataset. 00:34:00 to 00:34:17 is my favorite part. :) But very helpful in demonstrating that context like average length of character combinations forming names is an important puzzle piece 00:52:52 can someone explain what Andrej means around to 1......" ??? How could each probability be close to 1 ??? 00:53:02 , when he says, " BASICALLY IF U HAVE A VERY GOOD MODEL, THEN U WOULD EXPECT THAT THESE PROBABILITIES SHOULD BE NEAR 00:19:44 I needed a little bit of clarification around @ - shouldn't it be the second character follows the first - for example if the cell label is bh shouldn't it be h follows b 41 times and likewise for bg it should be g follows b 0 times. To be sure, I just did this for words[0:1] and I see 1 for cell labelled em, 1 for cell labeled mm and 1 for cell labeled ma - which follow the definition of m follows e once, m follows m once and a follows m once as expected in emma where the bigrams are em, mm, ma (excluding the special tokens for start and end). Please help me with this. Thanks. 00:13:11 Update: I think this has been clarified as a mistake @ 01:53:55 WOW 😲 01:21:00 Thanks for this lecture , what was the reasoning behind interpreting the NN output as a log count / logit? @Thank you 00:48:20 Around , why does the first column sum to one? If the sum is taken along the columns, then P.sum(1)[0] should contain the sum of the elements in the first row. If each element in the first column is divided by the sum of all elements in the first row, then it doesn't make sense why all those elements would sum to 1 (unless the first column and first row are the same, which they are not). 00:33:52 if we are getting same answer because of the generator, why when we put it in a for loop it generates different ones even though generator is same? what am i missing? 00:44:24 @Respect broadcasting semanticsPlay fast, looseAt your peril 00:52:31 excellent video. at , you say that the probability of the bigrams is 1/27. Shouldn't that be 1 / 27^2 , since we are taking two characters ? 00:19:50 At is called a confusion matrix, right? 00:42:50 @Andrej maybe it’d be less confusing if the selfie is flipped left to right. 00:48:00 For the bug around the counts in either dimension count the number of times a letter appears, so which way you sum doesn't matter. The problem is that you're normalizing across columns but then weighted-picking using rows in `p = P[ix]`. If you normalize across columns, you can use `p = P[:, ix]` to pick the most likely letter to precede `ix` and construct the name backwards. I'm not fully convinced why this bug leads to long repetitions of the same letter like in "fffttyufiscerexxxxxxxxxxxxxxzaraksbemusttcte" unlikely due to chance. 01:56:12 what is the probability that they have different middle portions but ending the same.. pretty mind blowing imo. 00:22:50 Andrej at , why not just add the character '.' to the chars list at the beginning -- instead of all that work? 01:55:44 Love the video! One small thing, btw: at you mention that it the neural net produced an anticlimactic result because its first 5 samples matches the samples from the simple counts-based model. But that isn't true! 3 of the 5 words changed slightly. 00:48:21 Great instructions!!!. But I noticed a subtle bug at . When doing P = P/P.sum(1), the result P.sum( 0, :) != 1 is correct. But P.sum( :, 0) = 1 is NOT true but just a co-instance here. This is a very interesting case, only because in original P, P.sum(0) is same as P.sum(1), which mean row i and col i have the same summation, both are equal to the occurrence of i_th letter. 00:30:00 yes i get d at around 00:08:43 Using the splat operator is probably faster: chs = ['<S>', *w, '<E>']Concatenating lists with + creates temporary objects and list(w) also creates a temporary object. Using splat like this saves on these temporaries, and it is also less to write! :D 00:52:50 at , Andrej says that ideally all these probabilities should be 1 because an accurate model shall be able to predict correctly what comes next. But I do not get the point how all those probabilites could be 1. For example, this means that the probability of i,a and i,v is same. How could this be 00:34:09 Thanks for the lecture! At how is it generating different names with the same generator? 00:57:59 Isn't taking average of log_likelihood compulsory? because if you don't take average then the longer ords will have higher loss most of times, so we will be getting lower length word to minimize the loss. 00:29:04 @ - we have nine 0s and nine 1s. But prop of 0 should be 60% and prop of 1 - 30%. Why we got 50/50 instead? Small sample? 00:26:23 one potential bug/nitpick, but I could totally be wrong. at , shouldn't the probability of of picking the first char actually come from first col (N[:,0]) instead of first row N[0,:]? You can justify it by checking the distribution of first column vs first row and realizing that letter a is much more likely to start a word than end a word. Also that the first row chars all end the bigram of the bigrams underneath them in the table. 00:30:37 Did something change in the generator implementation in pytorch? I was following along and at I get 'j' not 'm' so I tried downloading the ipynb from his repo and I still get 'j', so it's not something wrong about how I wrote it because I get the same results when running his notebook 01:55:50 Hey Andrej, thanks for this, great content! One remark though, the results at are not exactly the same, and you mention in the next video that they were "identical" but they are not quite identical, are they? And neither need/should they be perfectly identical, right? [Sorry if this was remarked before, parsed a few comments and did not see it] 00:12:37 You know what I find quite funny? At , based on your dataset, the letter "a" followed by the letter "n" has the third-highest probability of occurring. It's funny because I'm commenting on a video of ANdrej 00:34:14 He's in disbelief over the results and then had to convince himself that it was right and that the model was just bad haha 01:53:31 Can anyone help me understand the part around about regularization ? Why do we want to use smoothing and trying to make the distribution more uniform as mentioned earlier? Why do we want a "spring force" to push W to be 0? What is the extra bit at the end of the loss function (0.01*(W**2).mean()) giving us and why do we want it? 00:20:14 can anyone please tell me why the matrix is printed top to bottom rather bottom to top as the coordinates start from (0,0)? 00:16:44 Again an unnecessary temporary object, you can just just write: chars = sorted(set(''.join(words)))sorted() always returns a list anyway and it takes anything that is iterable. 01:55:50 - i dont wanna be that guy but they are a little different result. Can you explain? 01:55:40 But we actually don't get exactly the same result! Look closely at the outputs. () 01:57:17 "In the follow up videos, we're going to be taking mor. and mor. of these..." :) mor. and mor. 01:02:00 if anyone can help answer my question... at Andrej added +1 to "smooth" the probability and avoid INFINITY. Couldn't we just add something really small? like 0.01 instead of 1? 01:55:48 @ - Similar, but not exactly the same result. It would be really surprising to see 100% match 01:33:54 back to micrograd 00:47:00 hey! I'm quite new to this, could anyone please help me understand the intuition behind why keepdim being set to true is so important with p.sum at around ? Also, any resources for better understanding broadcasting would be much appreciated!! 00:33:08 the program "makemore" printing out "mor" the first time you ran it is pretty funny 00:19:42 I think you mean 'h follows b', and the inverse for all the other examples. 00:47:20 we are dividing it wrongIt likeP= P/P.sum(dim=0,keepdim=True) 00:43:40 PyTorch doesent copies :D"In short, if a PyTorch operation supports broadcast, then its Tensor arguments can be automatically expanded to be of equal sizes (without making copies of the data)." 01:10:01 Thanks for these awesome tutorials! I am a little confused around - what are the issues that arise when feeding the NN integers that we don't run into when we incorporate one hot encoding? 00:30:45 i.e. 'j' at . 01:14:32 why did we use normal distribution for our weights at ? 01:48:35 Do logits become equivalent to the 5th row or the 5th column? 00:46:52 at , Andrej said P = N.float() ; P = P / P.sum(1), do not work right, "it's a bug", it's not a bug of pytorch, run the code```a = torch.arange(9).reshape(3, 3)b = a.sum(1)```, you can see why it is 01:55:45 Surely Kondlaisah and Koneraisah would disagree that the models are the same at . 00:34:10 Loving your videos, such a good teacher, I started laughing at with the first batch of names that de bigram generator showed us, time to fixed it 🤣 00:32:38 In the case of a simple Bigram model @ we are sampling only one character, however, while calculating loss we consider the character with the highest probability. The character sampled is unlikely to be the same to the character with the highest probability in the row unless we sample a large number of characters from the multinomial distribution.So, my question is, does the loss function reflect the correct loss? Can anyone help me understand this. 00:30:42 At when the first sampling is made from the first row of the array, why do we get 'm' coz its not the most probable letter on the first row right?? 00:01:10 Really great tutorial, I can't emphasise this enough! also around the mark you make a comment on why feeding integers into your neural net doesn't make sense and we should use one hot encoding. I think we can indeed input integers into a neural net, but since these integers represent indices that contain no meaningful order, it's better to treat them as categorical labels. 01:53:00 Ayyo W Andrej Karpathy. The part at about regularization surely need to be rewatched many time 00:20:41 There seems to be a mistake at where the coordinates are flipped: it should be `plt.text(i, j, ...)` instead of `plt.text(j, i, ...)`. 00:40:51 For the people like me who had a hard time understanding the sum of tensor intuitively, maybe this can help: the tensor is array of arrays (of arrays....), so the know the sum of n-th dimension, you go down n levels of nesting, and look at the children of the array on this level. Sum along this dimension is the sum of the children of this array. If it's an array of scalars - it's a sum of scalars, and if it's an array of tensors - it's a sum of tensors. 00:36:34 When we are talking about vector normalizations around it is actually much simpler to create probabilityMatrix using P = N.float()/N.sum() than the example presented in this video.Probability for each row won't add up to one but probability "ratios" between elements in a single row will be the same.And you actually get the same result. 00:33:52 Why do you get the same result when running the same cell multiple times in but many different results when wrapping it in a loop? Does the generator change somehow after each usage? 00:48:14 - keepdem is necessary to have a column with 27 items and each element in that column is sum of all elements of row of the corresponding column 01:55:44 Actually, static model and neural-network give very similar yet a but different results if you look closer here. The third name is minaymoryles vs minaynnnyles, they are on the screen at the same time. Also the fourth one is slightly different yet very similar. It makes this result even more amusing to my opinion. 01:39:00 of tensors does not require grad and does not have a grad_fn around 01:08:49 Thanks for explaining things at such an incredible level of detail. () A couple of days ago I was too searching for what the difference between torch.Tensor and torch.tensor. Passing dtype explicitly seems like a good practice to me. Also the caution on the broadcasting rules is very valid. Often the bugs in my Pytorch/Tensorflow code are due to unexpected broadcasting going undetected because of say an averaging or reduction operation. Putting a lot of asserts on the shapes helps me. 00:52:18 at he says there are 27 tokens. But aren't there 27^2 tokens since for example you have ab and ba as separate tokens?<Edit>I answered my own question:# You expect each of the probabilities when equally likely to be 1/27 not 1/(27 * 27)# because remember P[ix1] = distribution of next possible characters given# the starting character at index ix1. And so P[ix1, ix2] = P(ix2 | ix1) - probability of# next character equalling character at ix2 given that ix1 is the first character.# There are 27 possible characters for ix2 so if it was equally likeyly, it would# be P[ix1, ix2] = 1/27 00:35:15 "doing something reasonable even though it's so terrible" - I'll remember this sentence, might come in handy😀That is a great vid, thank you very much for explaining all these things. 01:56:08 Andrej is the MAN! Such a level of detail and explanation that I've yet to find anywhere. Thanks for these incredible videos! I am SO close to being done with this. Took a long time to get to the end (I used to do programming as a hobby in C++, Visual Basic, mIRC (and other IRC platforms) but that was years ago) so getting back in the mix has been a process. I am getting a "ValueError: only one element tensors can be converted to Python scalars" on the ix variable at the very end. I've commented out the old methods and typed the code as shown at and can't seem to figure out the issue. The first rendition of ix works just fine, and commenting that out made no difference to this last section. Aye aye aye, so close!! 01:49:45 Why is the activations of the layer i.e. 'xenc @ W' taken to be as 'logits' ? I mean, they're simply just outputs of a layer, what occured to us that we're taking it to be as logits? I understand that it is a softmax layer, but how do we get the exact same thing as count-table starting from random weights in W ? 01:56:05 One thing I noticed (just to be clear, this is not really critique but just something interesting I noticed): at the results are not exactly equal - the third and fifth example generated are different. The last few characters are equal again, since the models just use the previous character to determine the next, so when both models reach the same character again they are likely to continue the same way. 01:56:09 At I noticed the two outputs are slightly different. Is that just because of the different smoothing functions he used for each? Or rounding errors? 00:34:12 We all know that moment at when we would have expected a much better result 🙂- awesome sessions, thanks! 01:56:00 Small observation: outputs at of NN and count models are not quite the same. Last one differs. 01:17:00 If anyone confused at , just think like first row of x get multiplied with w's first cols corresponding entries, and returns a scaler value which then takes place of x@w[1,1], the operation continues for all the remaining 26 columns of w, and eventually filling values of x@w[1,j], j representing the column number of w. This process starts again with 2nd row of x and filling the values of x@w[2,j], this continues till we are exhausted with x's rows which are 5 here so, the final row is x@w[5, j], or x@w has shape of i,j where i = number of rows of x and j = number of rows of w. 00:52:52 Not exactly clear on point made on timestamp , why should model give probability as 1 for each of the bigrams in the training data? 00:41:45 @, To clarify, P.sum(1) sums across columns, not rows, i.e. summing horizontally means that every row gets scanned, summing across all its columns to output a single number representing that row. So when summing horizontally, you're actually summing across columns (i.e. going over all columns for a specific row and summing them up). 00:36:20 The way he showed us that how an untrained model and an trained model differs by their output, oh man.... Really I don't comment at all but this has to be said, you are the best in this field. Really looking forward to learn more from you sir. 00:34:00 Weirdly enough I only get different results when sampling words, at around in the video. 01:36:20 , instead of all the values? 01:20:48 at AK says that output of the model is logcounts but at which step have we converted them to logcounts? I thought all we did was created an array of indices of bigrams, separated them to inputs/outputs, turned inputs to vectors using one-hot and multiplied them by random numbers generated by torch - at which point have we gotten logcounts? 01:22:55 I also have a question about logits in . If I understood properly, we *intentionally* interpreted (xenc @ W) to be log-counts(logits) to match the N matrix by first, exponentiating the logits (to only get positive values) and second, dividing each row by the sum of each row (so the sum is 1).My question is, is there a specific reason why we use an exponential function?I tried using other functions(y = |x|, a quadratic function, etc) but it seemed to give far worse results. 00:47:00 At , the bug you discussed - is it similar to doing P.sum(dim=0, keepdim=True), instead of dim=1?I mean, does using dim=0 and the buggy way yield the same result? 00:31:00 At , when you're doing the initial sampling, do the column letters not actually represent the second element of the bigram rather than the first? It looks to me like the rows are the first letters of the bigram. 01:55:45 Although Im not a newbie, its always nice to listen to you, especially when you explain ML concepts with such ease :). Btw results werent "exactly" the same ;) 00:34:16 Thank you so much sir for this video. i would also like to quote the famous 'law of large number' due to which we are getting very random and meaningless names at time stamp as we have very less sample i.e 1 in this case due to which it does not approximately represents the probability distribution that we give as an input to torch.multinomial. 00:20:16 Can you please explain why did you used j,i in plt.text instead of i,j ? 00:18:02 at he says "honestly, this is because everything starts at 0", but I wonder if it's not true. What that +1 is doing is incrementing the counter in the tensor , and that's what providing the stats on each letter. Just saying that sent me on a rabbit trail :) But it's really neat learning both ML & Python at the same time! 01:13:00 you explained just before that [T]ensor forced to be float, but recomended to not use it? 00:22:44 i+1 is nice trick but enumerate(chars, start=1) feels better 00:46:46 The code at actually works.Those long names can be explained. 01:09:50 ~ -- "You have to be careful out there..." (while drinkin' yo' juice and buildin' NNs) #realtalk #thuglifeSeriously though, if my university lectures back in the day offered a fraction of the clarity or context your lectures bring to the table, surely, I wouldn't have skipped so many! 01:50:00 At around timestep, do you mean to say xenc selects a column in W? xenc @ W is a matrix multiplication; rowxcolumn? 01:18:54 @ do we have 27 inputs, or do we have 5 inputs (where each input is the next character of the name 'emma' that we're training on?)? Confused 01:55:40 Just a small tidbit from in this video: when you sampled new names with the neural network at the outputs are actually NOT quite the same as the from the model that was sampling from the counts. Look at the last three examples. Almost the same, but not quite. 01:55:44 At when you sample using the gradient based model and compare to the output of the first, counts-based model, you say it's anticlimactic because the outputs are exactly the same. They're not strictly the same though! If you look closely at the third to fifth rows, you haveminaymoryles vsminaynnnyleskondlaisah vskoneraisahanchthizarie vsandhumizarieDoes that change the assumption you make about the equivalence of the two models? Clearly they are producing very similar output, anyway. 00:47:16 - why "keepdim" parameter is necessary and why keepdim=False is BUGGY here. 01:26:38 Now I'm confused even more. Where are these neurons? Do we have neuron weights but without neurons itself? 😞 01:02:39 we can calculate negative log likelihood using "nll = -(P.log().nan_to_num(neginf=0) * N).sum() / N.sum()" 00:58:01 At , may I ask why are we calculating log likelihood for all names together, shouldnt it be specific to each name? 01:55:54 Correct me if I am wrong, but at those are not the same results.Mine were different as well when I ran on my notebook.I suspect the seed. 00:53:12 @ how does this code deals with prediction of the likelihood to occur bigrams when we are just keeping a list of probabilities of bigrams in a text corpus? 00:53:10 : "We're looking at the probabilities that the model assigns to everyone of these bigrams in the dataset"How can we hope that they will all be high? We're told that in a good model, they should all be close to one, but does not that violate probs summing to 1, or what exactly are we measuring here? 00:48:05 I think it is neither normalizing the rows nor normalizing the columns. Actually, it is using the sum of columns for each row as the SUM VALUE, broadcasting it for each row as the denominator when perform P = P / P.sum(1). In this example, it happens to be true because the sum of rows equals the sum of columns (symmetric matrix). we can check it by multiply column 1 of P by a constant factor and see the result:P = N.float()P[:, 1] *= 1.2P = P / P.sum(1)then you will see P[:, 0].sum().item() != 1 01:56:05 What may also interest you: when looking at , you can see that the results are not the same. The third and the fifth are different, but since the probabilities of the models are very similar, they are likely to continue the same way as soon as they reach the same character again. So to be very precise, the loss and the sampling output are not the same - they are just very similar. And I assume that equal probabilities could be reached when optimizing it further (more iterations, smaller factor for update), and also the model smoothings might have prevented the probabilities to be exactly the same. The spelled-out intro to language modeling: building makemore We implement a bigram character-level language model, which we will further complexify in followup videos into a modern Transformer language model, like GPT. In this video, the focus is on (1) introducing torch.Tensor and its subtleties and use in efficiently evaluating neural networks and (2) the overall framework of language modeling that includes model training, sampling, and the evaluation of a loss (e.g. the negative log likelihood for classification). Links: - makemore on github: https://github.com/karpathy/makemore - jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part1_bigrams.ipynb - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - (new) Neural Networks: Zero to Hero series Discord channel: https://discord.gg/3zy8kqD9Cp , for people who'd like to chat more and go beyond youtube comments Useful links for practice: - Python + Numpy tutorial from CS231n https://cs231n.github.io/python-numpy-tutorial/ . We use torch.tensor instead of numpy.array in this video. Their design (e.g. broadcasting, data types, etc.) is so similar that practicing one is basically practicing the other, just be careful with some of the APIs - how various functions are named, what arguments they take, etc. - these details can vary. - PyTorch tutorial on Tensor https://pytorch.org/tutorials/beginner/basics/tensorqs_tutorial.html - Another PyTorch intro to Tensor https://pytorch.org/tutorials/beginner/nlp/pytorch_tutorial.html Exercises: E01: train a trigram language model, i.e. take two characters as an input to predict the 3rd one. Feel free to use either counting or a neural net. Evaluate the loss; Did it improve over a bigram model? E02: split up the dataset randomly into 80% train set, 10% dev set, 10% test set. Train the bigram and trigram models only on the training set. Evaluate them on dev and test splits. What can you see? E03: use the dev set to tune the strength of smoothing (or regularization) for the trigram model - i.e. try many possibilities and see which one works best based on the dev set loss. What patterns can you see in the train and dev set loss as you tune this strength? Take the best setting of the smoothing and evaluate on the test set once and at the end. How good of a loss do you achieve? E04: we saw that our 1-hot vectors merely select a row of W, so producing these vectors explicitly feels wasteful. Can you delete our use of F.one_hot in favor of simply indexing into rows of W? E05: look up and use F.cross_entropy instead. You should achieve the same result. Can you think of why we'd prefer to use F.cross_entropy instead? E06: meta-exercise! Think of a fun/interesting exercise and complete it. Chapters: 00:00:00 intro 00:03:03 reading and exploring the dataset 00:06:24 exploring the bigrams in the dataset 00:09:24 counting bigrams in a python dictionary 00:12:45 counting bigrams in a 2D torch tensor ("training the model") 00:18:19 visualizing the bigram tensor 00:20:54 deleting spurious (S) and (E) tokens in favor of a single . token 00:24:02 sampling from the model 00:36:17 efficiency! vectorized normalization of the rows, tensor broadcasting 00:50:14 loss function (the negative log likelihood of the data under our model) 01:00:50 model smoothing with fake counts 01:02:57 PART 2: the neural network approach: intro 01:05:26 creating the bigram dataset for the neural net 01:10:01 feeding integers into neural nets? one-hot encodings 01:13:53 the "neural net": one linear layer of neurons implemented with matrix multiplication 01:18:46 transforming neural net outputs into probabilities: the softmax 01:26:17 summary, preview to next steps, reference to micrograd 01:35:49 vectorized loss 01:38:36 backward and update, in PyTorch 01:42:55 putting everything together 01:47:49 note 1: one-hot encoding really just selects a row of the next Linear layer's weight matrix 01:50:18 note 2: model smoothing as regularization loss 01:54:31 sampling from the neural net 01:56:16 conclusion #deep learning #language model #gpt #bigram #neural network #pytorch #torch #tensor https://favtu.be/timelines/v/kVpDARqZdrQ Sat, 20 Aug 22 03:32:25 +0900 Stable diffusion dreams of psychedelic faces Prompt: "psychedelic faces" Stable diffusion takes a noise vector as input and samples an image. To create this video I smoothly (spherically) interpolate between randomly chosen noise vectors and render frames along the way. This video was produced by one A100 GPU taking about 10 tabs and dreaming about the prompt overnight (~8 hours). While I slept and dreamt about other things. Music: Stars by JVNA Links: - Stable diffusion: https://stability.ai/blog - Code used to make this video: https://gist.github.com/karpathy/00103b0037c5aaea32fe1da1af553355 - My twitter: https://twitter.com/karpathy https://favtu.be/timelines/v/2oKjtvYslMY Thu, 18 Aug 22 01:58:01 +0900 Stable diffusion dreams of steampunk brains Prompt: "ultrarealistic steam punk neural network machine in the shape of a brain, placed on a pedestal, covered with neurons made of gears. dramatic lighting. #unrealengine" Stable diffusion takes a noise vector as input and samples an image. To create this video I smoothly (spherically) interpolate between randomly chosen noise vectors and render frames along the way. This video was produced by one A100 GPU dreaming about the prompt overnight (~8 hours). While I slept and dreamt about other things. This is version 2 video of this prompt, with (I think?) a bit higher quality and trippy AGI music. Music: Wonders by JVNA Links: - Stable diffusion: https://stability.ai/blog - Code used to make this video: https://gist.github.com/karpathy/00103b0037c5aaea32fe1da1af553355 - My twitter: https://twitter.com/karpathy https://favtu.be/timelines/v/sM9bozW295Q Wed, 17 Aug 22 08:02:58 +0900 Stable diffusion dreams of tattoos Dreams of tattoos. (There are a few discrete jumps in the video because I had to erase portions that got just a little 🌶️, believe I got most of it) Links - Stable diffusion: https://stability.ai/blog - Code used to make this video: https://gist.github.com/karpathy/00103b0037c5aaea32fe1da1af553355 - My twitter: https://twitter.com/karpathy https://favtu.be/timelines/v/VMj-3S1tku0 Wed, 17 Aug 22 07:44:26 +0900 00:00:00 - 🎓 Andre introduces the lecture on deep neural network training, focusing on a hands-on approach using a Jupyter notebook. 00:00:55 - ⚙ Micrograd is presented as an autograd engine that implements backpropagation, essential for training neural networks by minimizing loss functions. 00:01:37 - 🧩 Micrograd allows users to build mathematical expressions with operations like addition, multiplication, and power raising through a value object system. 00:02:34 - 🔄 Backpropagation enables the evaluation of derivatives with respect to inputs, providing insights into how changes in inputs affect the output. 00:03:59 - 📈 Understanding derivatives helps in tuning weights during neural network training, indicating how each input influences the output value. 00:06:04 - 📝 Micrograd's implementation is simple, consisting of just over 100 lines of code for the core engine and minimal lines for the neural network library. 00:08:12 - 🌐 This lecture aims to provide a clear conceptual understanding of derivatives, crucial for comprehending neural network functionality. 00:12:24 - 📊 The derivative is defined as the slope of the function, and can be approximated numerically using a small change in input (h). 00:18:56 - 🔗 The importance of maintaining expression graphs is highlighted for neural networks, which consist of interconnected mathematical expressions. 00:22:50 - 🏗 The creation of a "value" class facilitates operations among values while also tracking dependencies within the expression graph. 00:23:04 - 🧩 New variable `prev` stores the set of children in the micrograd structure for efficiency. 00:24:14 - 🔄 Operations are tracked using a simple string indicating whether the value is from addition or multiplication. 00:25:00 - 📊 Visualization of expressions is done using the `drawdot` function to graphically represent mathematical relationships. 00:29:11 - ➕ Forward pass computes output from inputs, producing output value `l` based on previous values. 00:30:50 - 📈 The gradient (`grad`) represents the effect of value changes on output, initialized to zero for all nodes. 00:32:02 - 🔄 Manual backpropagation begins by calculating initial gradients starting from the output value. 00:38:16 - 🔗 Understanding gradients for each node is crucial for effective backpropagation and training of neural networks. 00:44:03 - 📚 The chain rule is crucial for determining how changes in intermediate node values affect the final output gradient. 00:48:54 - 🔄 Derivatives are calculated as part of the backpropagation process, with a focus on specific values for 'a' and 'b'. 00:50:30 - 📈 Backpropagation involves recursively applying the chain rule from the output back through the computation graph. 00:51:44 - 🧮 Nudge inputs in the direction of the gradient to optimize the outcome, demonstrating a single optimization step. 00:53:10 - 🧠 The neuron model consists of weighted inputs, biases, and an activation function that transforms the output. 00:54:34 - ⚙ The activation function, such as tanh, smoothly limits outputs to a defined range, influencing the model's predictions. 00:56:58 - 📊 Understanding that only the local derivative is essential to adjusting weights and biases during optimization. 01:00:06 - 🔍 The tanh activation function requires the implementation of exponentiation for proper calculation within the model. 01:03:54 - 📉 The local derivative of tanh is calculated using its defined mathematical formula, critical for backpropagation. 01:04:45 - 📦 Backpropagation allows gradients to flow through nodes based on operations like multiplication and addition. 01:09:17 - ⚡ Transitioning to a more automated backward pass approach helps simplify the backpropagation process across the neural network. 01:14:38 - 🎯 Gradients are initialized with a value of 1.0 to start backpropagation. 01:18:16 - 🔄 Topological sorting organizes computation graphs ensuring dependencies are processed before backpropagation. 01:21:30 - 🛠 A backward method is implemented to automate gradient calculations for backpropagation in neural networks. 01:25:28 - 🚫 Gradient overwriting occurs when a variable is reused multiple times; accumulation is necessary to fix it. 01:28:05 - ➗ Division can be implemented through exponentiation, transforming the operation into a multiplication of powers. 01:30:08 - 🧮 Implementing power functions allows for more complex operations and backpropagation through various mathematical functions. 01:36:12 - 🔗 Subtraction is handled by defining it as negation and implementing negation through multiplication by -1. 01:38:43 - 📊 The exercise demonstrated that breaking down functions like tanh into simpler components retains mathematical equivalence in both forward and backward passes. 01:38:56 - 🔄 Understanding that the essence of operations in neural networks is about inputs and outputs, regardless of their complexity. 01:39:11 - ⚙ PyTorch allows building complex operations through a tensor-based framework, enabling efficient deep learning. 01:39:40 - 📊 Scalars in Micrograd are replaced with tensors in PyTorch, which are n-dimensional arrays enhancing computational efficiency. 01:40:21 - ✅ Tensors in PyTorch must explicitly require gradients for proper training, as they default to not requiring them for efficiency. 01:41:03 - 📈 Performing arithmetic operations in PyTorch is analogous to Micrograd, maintaining a similar data access pattern. 01:44:08 - 🧠 Neural networks can be viewed as specific mathematical expressions, enabling structured implementation. 01:45:32 - ⚡ Constructing a neuron involves initializing weights and biases, followed by implementing the forward process through multiplication and addition. 01:48:02 - 🏗 A layer of neurons operates independently, with each neuron receiving inputs and producing outputs without interconnectivity. 01:49:41 - 🔗 An entire multi-layer perceptron (MLP) processes inputs through sequential layers, allowing for complex data transformations. 01:50:56 - 📉 Loss is a critical metric for measuring neural network performance and guides the adjustment of weights to improve predictions. 02:02:22 - 📉 Gradient descent is implemented by adjusting weights in the direction opposite to the gradient, minimizing the loss effectively. 02:03:16 - ⚖ Minimizing loss is crucial, requiring adjustments in the negative direction of the gradient. 02:04:44 - 📉 Regularly re-evaluating loss during the forward and backward passes helps track performance improvements. 02:05:58 - 🔄 Repeating the forward and backward passes iteratively improves neural network predictions through gradient descent. 02:06:48 - 🚫 Too large learning rates can destabilize training, potentially causing the loss to increase rather than decrease. 02:08:00 - ⚙ Tuning the learning rate is a delicate process; an inappropriate rate can lead to slow convergence or unstable training. 02:10:36 - 🔍 Identifying and fixing common bugs in neural network code is essential for successful training. 02:12:28 - 📝 Resetting gradients (zero grad) before each backward pass is crucial for ensuring correct gradient computation. 02:14:52 - 🧠 Neural networks can handle complex tasks by iteratively minimizing the loss function, driven by gradient descent principles. 02:16:32 - 📊 The principles of neural network training remain the same across various architectures, such as GPT, despite differences in complexity and detail. 02:21:13 - 🔍 Micrograd allows for a simplified approach to understanding neural network operations compared to more complex libraries like PyTorch. 00:00:42 *🧠 Micrograd is an autograd engine that implements backpropagation, essential for efficiently evaluating the gradient of a loss function with respect to neural network weights.* 00:02:34 *📊 Micrograd allows building mathematical expressions, enabling operations like addition, multiplication, exponentiation, and more, forming an expression graph.* 00:03:16 *🔍 Backpropagation in micrograd initializes at a node, recursively applying the chain rule to evaluate derivatives of internal nodes and inputs, crucial for understanding how inputs affect the output.* 00:05:50 *💡 Micrograd operates at the scalar level for pedagogical reasons, but real-world neural network training employs tensors for efficiency, maintaining the same mathematical principles.* 00:07:57 *⚙️ Micrograd's simplicity is highlighted by its small codebase: the autograd engine comprises just 100 lines of Python code, while the entire neural network library built atop it is only around 150 lines.* 00:08:12 *📈 Understanding derivatives intuitively is essential for grasping their significance in neural network training, demonstrated through numerical approximation and analysis of slope changes.* 00:19:11 *🛠️ Building data structures like the 'value' object is crucial for maintaining expressions in neural network training, laying the foundation for constructing more complex networks.* 00:19:26 *🛠️ The Value class in micrograd is a simple wrapper for a single scalar value, allowing operations like addition and multiplication.* 00:20:04 *➕ Python's special double underscore methods are used to define operators like addition for custom objects such as the Value class.* 00:22:50 *📊 To maintain expression graphs in micrograd, each value object keeps track of its children, predecessors, and the operation that created it.* 00:25:00 *📈 Graph visualization in micrograd helps to understand complex expression graphs, aiding in debugging and analysis.* 00:29:40 *⏪ Backpropagation in micrograd involves computing gradients backward from the output to the input, enabling derivative calculations for optimization algorithms like gradient descent.* 00:31:05 *🔄 The grad attribute in micrograd's Value class tracks the derivative of the output with respect to each value, facilitating gradient computation during backpropagation.* 00:38:16 *🔢 Deriving gradients during backpropagation involves understanding how changes in intermediate values affect the final output, following the chain rule of calculus.* 00:41:46 *📚 The chain rule in calculus is fundamental for differentiating through function compositions, expressing how derivatives are multiplied together correctly.* 00:43:07 *🚗 The chain rule allows for calculating the rate of change of a composite function by multiplying the rates of change of its components, akin to a car's speed being a product of its individual speed changes.* 00:45:33 *🔀 Plus nodes in neural networks effectively route gradients, distributing derivatives to all child nodes, as indicated by the chain rule.* 00:47:28 *🔄 Backpropagation involves recursively applying the chain rule backward through the computation graph, multiplying local derivatives along the way.* 00:51:14 *⏫ Adjusting inputs in the direction of the gradient during optimization can increase the output of a neural network, demonstrating the power of gradients in influencing outcomes.* 00:53:10 *🧠 Understanding backpropagation through neurons lays the groundwork for building neural networks, utilizing mathematical models to propagate gradients and optimize network performance.* 01:04:08 *🧠 Understanding the local derivative of the hyperbolic tangent function (tanh) is crucial in backpropagation, where it's expressed as \(1 - \text{tanh}^2(x)\).* 01:04:45 *🔄 Backpropagating through a plus node involves distributing the gradient equally to both inputs, as the local derivative is 1.* 01:06:13 *🔢 For a times node, the local derivative is the other term. Calculating gradients involves multiplying the local and global derivatives.* 01:07:36 *📉 Understanding why certain gradients are zero is crucial; in this case, if the input doesn't influence the output, the gradient is zero.* 01:10:11 *🔄 Defining backward functions for addition and multiplication operations in neural networks involves applying chain rule and accumulating gradients.* 01:17:46 *📊 Implementing backpropagation involves a topological sort to ensure proper ordering of gradient calculations, especially in complex networks.* 01:27:08 *🔢 Tanh function can be broken down into simpler expressions, aiding in understanding and implementation.* 01:28:32 *➕✖️ Implementing addition and multiplication operations in a neural network library involves handling different data types and ensuring compatibility for arithmetic operations.* 01:29:54 *➗ Understanding the concept of "r mul" (right multiplication) in Python helps handle arithmetic operations in neural networks efficiently.* 01:31:38 *📈 Implementing exponentiation and division operations in neural networks requires understanding their mathematical derivatives and chain rule for backpropagation.* 01:35:57 *➖ Implementing subtraction in a neural network involves expressing it in terms of addition and negation for efficient computation.* 01:38:43 *🧠 The design and implementation of neural network operations are flexible, allowing developers to choose the level of abstraction for efficient computation and backpropagation.* 01:52:20 *🧠 Loss in neural networks measures the performance, aiming to minimize it; mean squared error is a common loss function.* 01:57:30 *🔍 Gradients are crucial for adjusting weights in neural networks through techniques like gradient descent.* 02:02:08 *🔄 In gradient descent, parameters are adjusted iteratively in the opposite direction of the gradient to minimize loss.* 02:05:28 *📉 Gradient descent involves a cycle of forward pass, backward pass (backpropagation), and parameter updates to improve neural network predictions.* 02:07:45 *⚖️ Finding the right learning rate in training neural networks is crucial; too low leads to slow convergence, while too high can cause instability and loss explosion.* 02:10:23 *🐛 Forgetting to zero gradients before backward pass can lead to subtle bugs in neural network training, causing gradients to accumulate and affect optimization.* 02:11:58 *🔄 Resetting gradients to zero before backward pass prevents accumulation and ensures accurate gradient updates during optimization.* 02:13:12 *🧠 Training neural networks can be challenging due to potential bugs, but simple problems may mask issues; thorough testing and debugging are essential.* 02:14:11 *🤖 Neural networks consist of mathematical expressions processing input data through layers to minimize loss via gradient descent, enabling them to learn complex patterns.* 02:16:46 *🔧 While understanding neural network principles remains consistent, real-world implementations like PyTorch can be complex, with extensive codebases and nuances in functions like backward passes.* 02:21:39 *📊 Exploring PyTorch's codebase reveals complexities in functions like backward passes, where implementation details vary based on hardware and data types.* 00:00:00 📖 *Introduction to Neural Network Training*- Andrej introduces the lecture, focusing on neural network training and the use of the micrograd library.- Explains the objective: building a neural network from scratch using micrograd.- Emphasizes the step-by-step approach to understanding neural networks and backpropagation. 00:00:14 🌐 *Overview of Micrograd*- Introduction to micrograd, a library for understanding neural networks.- Describes the purpose of micrograd: a tool for learning the basics of neural network training. 00:00:29 🎓 *In-depth Explanation of Micrograd*- Detailed explanation of micrograd and its components.- Focuses on the autograd engine, a key element for implementing backpropagation. 00:01:09 🧠 *Core Concepts of Neural Networks*- Discusses backpropagation and its significance in neural network training.- Explains the iterative process of tuning neural network weights to minimize loss function and improve accuracy. 00:01:24 📈 *Example of Micrograd Functionality*- Demonstrates micrograd's functionality through an example.- Illustrates how micrograd builds and evaluates mathematical expressions. 00:02:07 🔢 *Supported Operations in Micrograd*- Lists the mathematical operations supported by micrograd.- Describes how these operations contribute to building an expression graph. 00:03:03 💡 *Forward and Backward Pass Explanation*- Explains the concept of the forward pass in neural networks.- Introduces the backward pass and backpropagation, detailing its process and significance. 00:03:59 📊 *Understanding the Role of Derivatives*- Discusses the importance of derivatives in neural networks.- Shows how changes in inputs affect the output, emphasizing the role of gradients in training. 00:04:54 🌟 *Neural Networks as Mathematical Expressions*- Highlights that neural networks are essentially complex mathematical expressions.- Discusses the generality of backpropagation beyond neural networks. 00:06:04 🚀 *Efficiency in Neural Network Libraries*- Talks about efficiency in neural network libraries and the role of tensors.- Explains how the math remains the same, but efficiency is achieved through parallel processing. 00:07:01 🛠️ *Micrograd: A Simple Yet Powerful Engine*- Reveals the simplicity of the micrograd engine and its power in neural network training.- Emphasizes that the core of neural network training can be understood with just a few lines of code. 00:08:12 🎲 *Intuitive Understanding of Derivatives*- Begins a new section to foster an intuitive understanding of derivatives.- Uses basic examples to illustrate the concept of derivatives in a more accessible manner. 00:10:29 🧐 *Deep Dive into the Concept of Derivatives*- Delves deeper into the definition and practical understanding of derivatives.- Provides examples to clarify the concept of derivatives in the context of functions and neural networks. 00:14:18 🤔 *Complex Case: Derivatives in Multivariable Context*- Explores a more complex scenario involving derivatives in a multivariable context.- Demonstrates how derivatives are calculated in situations with multiple inputs and outputs. 00:19:26 🏗️ *Building the Value Object in Micrograd*- Starts constructing the 'value' object in micrograd, crucial for neural network computations.- Explains the significance of this object in representing and processing neural network elements. 00:25:00 🖼️ *Visualizing Mathematical Expressions*- Introduces a method to visualize mathematical expressions in micrograd.- Uses graphical representations to make understanding of neural network components clearer. 00:29:40 🔄 *Understanding Backpropagation Through Manual Calculation*- Demonstrates backpropagation through manual calculation of gradients.- Provides a step-by-step breakdown of backpropagation to enhance conceptual clarity. 00:34:35 🧪 *Applying Backpropagation: An Example*- Applies backpropagation to a specific example for practical understanding.- Shows how to adjust inputs based on gradients to influence the output. 00:38:00 🧩 *Complex Example: Backpropagation Through a Neuron*- Presents a complex example of backpropagation through a single neuron.- Breaks down the process into understandable steps, highlighting the mechanics of neural network operations. 00:44:03 🔄 *Automating Backpropagation in Micrograd*- Discusses the automation of backpropagation in micrograd.- Implements functions to streamline the backpropagation process, enhancing efficiency and clarity. 00:51:14 ⚙️ *Implementing Key Operations in Micrograd*- Implements additional operations in micrograd, such as exponentiation and division.- Explores the mathematical underpinnings of these operations and their role in neural networks. 00:53:10 🧱 *Building a Neural Network Layer by Layer*- Starts building a neural network from the ground up, focusing on individual neurons and layers.- Highlights the incremental process of constructing a neural network in micrograd. 01:44:49 🧠 *Neural Network Basics*- Introduction to the basic structure and functioning of a neuron in a neural network,- Explanation of neuron inputs, weights, bias, and implementation of the `__call__` method in Python,- Demonstration of neuron output calculation using dot product and activation functions. 01:48:02 🧩 *Creating Layers in Neural Networks*- Building a layer of neurons and explaining its structure and functionality,- Illustration of how layers consist of multiple neurons, each fully connected to the input,- Explanation of initializing neurons within a layer and evaluating them independently. 01:49:11 🌐 *Multi-Layer Perceptron (MLP) Configuration*- Setting up an entire multi-layer perceptron (MLP),- Understanding how layers in an MLP feed into each other sequentially,- Explanation of the MLP's input-output relationship and forward pass process. 01:52:20 💡 *Understanding Loss in Neural Networks*- Introduction to the concept of loss in neural networks and its significance,- Explanation of how loss measures the neural network's performance,- Detailed walkthrough of implementing mean squared error loss. 01:55:09 🔄 *Backpropagation and Gradient Information*- Discussion of backpropagation in neural networks,- Understanding the role of gradients in tuning network parameters,- Demonstration of how gradient information is used to adjust weights for reducing loss. 01:58:11 ⚙️ *Parameter Management in Neural Networks*- Managing parameters in a neural network for efficient operation,- Techniques for gathering and adjusting neural network parameters,- Explanation of parameter storage and retrieval within neurons and layers. 02:05:28 📈 *Implementing Gradient Descent*- Detailed implementation of the gradient descent algorithm in a neural network,- Steps for iteratively adjusting parameters based on gradient information,- Emphasis on the importance of learning rate and its impact on convergence. 02:10:23 🐛 *Identifying and Fixing Bugs in Neural Networks*- Addressing common bugs in neural network implementation,- Importance of zeroing gradients before backward propagation,- Explanation of why and how to reset gradients for each training iteration. 02:14:11 🎓 *Summary and Insights on Neural Networks*- Summarizing key concepts and learnings about neural networks,- Discussion on the versatility and potential of neural networks in solving complex problems,- Reflection on the educational journey of building and understanding neural networks through the video. 00:00:00 intro 00:00:25 micrograd overview 00:08:08 derivative of a simple function with one input 00:14:12 derivative of a function with multiple inputs 00:19:09 starting the core Value object of micrograd and its visualization 00:32:10 manual backpropagation example #1: simple expression 00:51:10 preview of a single optimization step 00:52:52 manual backpropagation example #2: a neuron 01:09:02 implementing the backward function for each operation 01:17:32 implementing the backward function for a whole expression graph 01:22:28 fixing a backprop bug when one node is used multiple times 01:27:05 breaking up a tanh, exercising with more operations 01:39:31 doing the same thing but in PyTorch: comparison 01:43:55 building out a neural net library (multi-layer perceptron) in micrograd 01:51:04 creating a tiny dataset, writing the loss function 01:57:56 collecting all of the parameters of the neural net 02:01:12 doing gradient descent optimization manually, training the network 02:14:03 summary of what we learned, how to go towards modern neural nets 02:16:46 walkthrough of the full code of micrograd on github 02:21:10 real stuff: diving into PyTorch, finding their backward pass for tanh 02:24:39 conclusion 02:25:20 outtakes :) 00:28:09 Current progress : , Updating it tomorrow 00:51:15 : , Updating Tomorrow again😀 00:01:10 : updating tomorrow. 00:34:41 multiplicative derivative 00:46:21 bkmk 01:30:01 object oriented language is a cluster 01:22:10 @ bugs Solution @ (1) 01:25:29 then @ 02:10:20 @ bugs Solution @ (3) 02:01:00 I am confused atmy confusion stems from not understanding how the gradient is associated with the loss. I see how it relates to the OUTPUT, not the LOSS. For example, the output can only be somewhere between -1 and 1. To update the parameters he is doingp.data += -0.01 *p.gradSo I see 4 cases1) the grad is negative and we want the output to be positive 1, then data increases by a little bit and the output decreases (BAD; not what we want)2) if the grad is negative and we want the output to be -1, then data increases by a little bit and the output decreases (GOOD)3) if the grad is positive and we want the output to be positive 1, then data decreases by a little bit and the output decreases (BAD)4) if the grad is positive and we want the output to be -1, then data decreases by a little bit and the output decreases (GOOD)in cases 1 and 3 aren't we moving the output in the wrong direction? 02:03:00 at he says the loss would increase, but isn't it that the OUTPUT would increase? The loss depends on whether we want positive 1 or negative 1 as our output 01:55:00 I missed the step at , I am new to Python and don't really understand how loss is a value but that is okay 01:47:45 What was his random function doing? :D At Generating 0.9xx like 5 times and at 0.88 3 times. And to the topic - thank you Andrej for this. When I see good people like you transferring the knowledge to the new generation of AI developers, I'm quite a bit more optimistic about our future. I think with more people like you we can create responsible AI. 01:51:40 generating 00:21:51 * use of repr function 01:05:55 * positive influence on o/p by positive grad (0.5) 00:25:05 I couldn't get over the "draw_dot()" function of the Digraph section at . Found some of it's statement very hard to get through and few twisted objects being created. But, still following along... 01:59:55 - The first way with the `for` loop is so much more readable. Single list comprehensions can be useful, but nesting them just means a maintainer has to stop and think about it. Maybe that's fine, but I find stopping to be disruptive. Some may say it becomes second nature, but 1) I don't believe them, and they can be proven wrong with an MRI scan while they're reading through the code, and 2) using a basic for loop doesn't require second nature, because it's _already first nature_ to any programmer. 02:11:52 - I was able to understand the bug you explained - after going through the video second time 02:10:25 I think I get a sense of the reason why even though he mentions the reason at about there being a bug and still the loss to decrease so very much is that. since we do p-data += p-grad*-1*stepsize. p-grad are stacking up as well so in fact the p-data is a big number and when it passes through tanh it gets squashed to 1 or -1 depending on the sign. and since we only had 1 or -1 in the ys list, we were getting very close answers.just a speculation haven't proven it. would like to know others' view. 02:16:28 However is it really stochastic gradient descent with a loss on all the datapoints? 01:44:00 Note that at he switches back to micrograd (from PyTorch).. may be obvious but maybe not :) 00:01:53 also fyi for around where a loss function is being evaluated, you should also add r_sub in the value class also, just to make subtraction compile regardless of order. 01:12:45 the hard part is the semantics of a closure in a private method in a class when it is invoked later in an instance of that class in python; those semantics... 01:45:40 before running this code one should import random, e.g. in the beginning of the notebook. Otherwise there will be an error (I have no experience in coding, that's why I try to write each code of block from the video myself). 00:52:00 Hey everyone, I have a question at Andrej mentions that if we nudge every leaf node with a tiny amount of it's gradient, the output L should move positively. However, I feel that not all leaf nodes would move L in positive direction. E.g. in this case b will move L in a negative direction. Is there a flaw in my logic?Cumulatively, L would move positively as all other nodes are moving it in a positive direction 00:13:00 why does this equals 14 i can't understandif results 20 and now results 14? 02:16:30 Andrej you mentioned at roughly that the training loop is using stochastic GD, I believe it's a batch GD :). Or may be my gradient descent concepts are messed up lol 00:25:29 simple fix for error at , add label parameter to int 00:51:55 Sir at of the video to see positive effect on the final variable shouldn't we add or subtract the "h" from the leaf node variables after seeing the gradient of it meaning that a positive gradient means we add the "h" while a negative means we subtract the "h" but you have just added "h" to "b" as well as "c" although the they have a negative gradient ? 01:20:22 I don't get why the use for topological sort rather than just a BFS from the final node -> prevIn this context, BFS will go in the correct order right? 01:19:31 Honestly you don't even need topological ordering, I literally implemented a recursive approach where I call backpropag on the output node and it will then first set the gradient of his children and then cycle through its children to and backpropagate on those nodes recursively 00:47:40 At why is dL/da = dL/de * de/da and not dL/da = dL/dd * dd/de * de/da? 01:39:00 까지 요약하면 특정 함수에대한 값을 구하고, 마지막 값에 대한 그래디언트를 구하는 과정을파이썬에 옮기는 법에 대해 가르치고 있다. 00:51:30 Something you didn't explain - - if we want L to go up we simply need to increase the variables in the direction of the gradient? How come it is so if some gradients are negative? 01:35:00 Instead of:other * self.data**(other-1) * out.gradcan't we do:other * out.data/self.data * out.grad 00:23:45 Around Andrej refers to ._prev and _children as children, but conceptually aren't these the *parents* of the output of the operation?? 01:19:16 I might be saying something very stupid but can't we do this with a simple recursion as we already know the prev of it 01:03:43 Great video. Shouldn't at be bo/bn = 1 - o**2? 02:10:30 Okay my bad at he mentioned about it. 02:13:52 You didn’t initialise loss.grad=1 but it somehow worked. How?Edit: self.grad=1 is written in backward function so when we call loss.backward it happens 02:12:19 I have a little question around , when you fix the bug using .zero_grad().When I was trying to implement my own backward() function earlier, I did the equivalent of .zero_grad() before I backward propagate the derivatives. However, since I saw both you and pytorch decided that it should be a separate function, I'm wondering what's the reason for not doing it in the backward() function itself? 00:23:09 - I think you decided to use a tuple as it is immutable. Python sets are mutable, so it is not a good idea to provide a mutable data structure as a default parameter value. 01:09:00 youtube.com/@AndrejKarpathy Why is it that at when you did back propagation with the tanh function you calculated the local gradient for the end nodes but didn't calculate the gradient in relation to the last node as you did before? Does the tanh function or all addition before multiplication negate the need for this? 01:43:52 random library is not imported, it might give an error while writing further code. Just run:import random 00:51:41 thanks a lot. But let ma ask a question please. In should we subtract the gradients from the values of a,b,c and f instead of adding them? 01:17:35 instead of a topological sort, tree traversal method also works and is simpler to implement. Thanks for the awesome class! 01:54:01 if you got error : TypeError: unsupported operand type(s) for -: 'float' and 'Value' . Try to adddef __rsub__(self,other):return other + (-self)to Value class 02:10:24 Thank you very much for that lecture Andrej! It really helped to understand and to combine different pieces I learned separately into one structured concept. I especially appreciated that you left the part with the bug on gradient zeroing and made an explanation for it at . Making mistakes and learning from them is so effective and undervalued imo. Thanks a lot! 00:58:41 "As long as you know how to create the local derivative - then that's all you need". Ok Karpathy. Next paper title "Local derivatives are all you need". 00:51:15 Just peak and simple explanation of Backpropagation: Recursive application of chain rule going backwards through the computation graph" , seriously blown away by Uncle Andrej. 00:45:06 For those who are confused with the chain rule , another way to think of calculating dL/dc is to substitute d by (c + e) => L = (c + e) * f => L = fc + fe => dL / dc = f(1) = f 01:56:40 something I dont get it in is that when we draw our nodes, there are bunch of nodes that are not having any parent node so im guessing that they are input data nodes but why arent they are grouped up to better represent visuality or is it just how liblary works or they are not input nodes?? I would be more than grateful if someone can able to explain this. 01:47:12 To anyone else struggling with a TypeError at . Karpathy included the __radd__ definition to the Value class prior to running this box of code. I'm not sure if I missed the part of the video where that was included or if that portion of the video was accidentally edited out. 02:13:00 Micro grad implements back prop. Can create Value objects and do operations with it. In the background it creates a computational graph and keeps track of stuff. Can call backward prop of a value object with apply chain rule to do back prop. 01:49:38 I really liked how we are reusing the x as the output of a layer and then feeding it to the next layer. 01:54:50 At if some one is facing with error of: 00:01:21 : You can merge the graph drawing logic with the backward calls. The graph traversal already happens once when rendering it. For larger graphs that scales well. This can be achieved by just abstracting the rendering as well as backward calls as listeners to be invoked when the graph is traversed. 01:17:47 A little question: at , instead of sorting then iterating through nodes to call backward(), why don't we just make a recursive call, starting from the root (loss) node? In its backward function, each node sets its children's grad (it knows how to, given its own grad) then calls backward() on each of them.Might that be correct? 00:23:11 The reason you use a set at is so if you do a + a you don't end up with a in the _prev tuple twice, which would presumably screw with the backdrop later. 00:01:20 No need for a topological sort here. I implemented backprop using a simple level-order traversal. It's simpler, easier to implement, and a tad more memory efficient. 01:43:00 how did we set the grad of variable o to be 1.0 (or any other value) before we call backward()? or it's set to 1.0 by default (which is odd?) 02:15:23 "...billions, at this point (points to himself i.e. OAI at the time) almost trillions of parameters" this was probably the first public hint to what the parameter size of GPT-4 is, considering the release of this video was around the time it finished training : ) 01:18:00 This is such an awesome resource! I've been stuck trying to beat the math of backpropagation into my head for the past week, but this video explained the logic behind gradients and using the chain rule so intuitively. One quick question though, why exactly to do we need to perform a topological sort prior to calling _backward() on the children in the section starting at ? 01:54:47 Dont forget to add _radd_ at to run loss = sum((yout - ygt)**2 for ygt, yout in zip(ys, ypred)) 00:52:00 Around you were nudging the inputs in the direction of the gradient -- should b and c have been decreased rather than increased because their gradient was negative? 01:54:44 If you're following along, at , you need to have implemented an __radd__ function in the Value object to allow you to subtract a value object from an int 02:13:00 This is a goldmine for refreshing and deepening neural network knowledge and understanding, thanks for providing it for free. And accumulating the gradients at can be considered a feature not a bug I suppose 01:54:57 , I think we should be averaging gradients of all the four passes, and then use the final value to update the weights ? 01:50:32 - error is AttributeError: 'MLP' object has no attribute '_prev'Python console is>>> x = [2.0, 3.0, -1.0]>>> n = MLP(3, [4, 4, 1])>>> n(x)Value(data=-0.8833054459616237)>>>>>> draw_dot(n)Traceback (most recent call last):File "<stdin>", line 1, in <module>File "<stdin>", line 12, in draw_dotFile "<stdin>", line 10, in traceFile "<stdin>", line 7, in buildAttributeError: 'MLP' object has no attribute '_prev' 00:23:30 At roughly in the video you talk about using tuples vs sets, sets in Python do not allow duplicate items and are hash tables, should we infer that there will never be any duplicate items in the derivative set? 00:11:40 Andrej at , small thing. I believe what you mean is not the slop at the point of the bump, but it's actually the slope of the point as you decrease the value of the bump towards 0, which be the slope at f(a) vs f(a+h). 00:52:35 if you want to INCREASE the loss function by adding to the leaf nodes by some small amount, wouldn't you need to only add to the nodes that have a POSITIVE gradient (i.e. a and f)? 00:51:50 @AndrejKarpathy, I have quick question. @ you mention that if we nudge each value in the direction of gradient, we expect positive influence on L. Why is it positive? Is it because our maximum gradient value is 6 and its sign is positive?It would be really helpful if someone could clear this doubt. 00:49:00 I'm stiill at , so basically if the operation is arithmetic then the partial derivative will just be a constant right? This is also what I've learned in calculus class. 01:54:47 Just and FYI for those following at home. If you are getting an error at you should add __radd__ into your Value class similar to __rmul__. It will allow the order of addition to not matter. I don't think it was shown in the earlier sections. 02:25:45 why does the multiplication work?! he didn't type correct the other parameter in the __mul__? 02:11:00 WHY DIDN'T YOU Reset to zero loss, but you reset the n? The loss also has Value type... () The spelled-out intro to neural networks and backpropagation: building micrograd This is the most step-by-step spelled-out explanation of backpropagation and training of neural networks. It only assumes basic knowledge of Python and a vague recollection of calculus from high school. Links: - micrograd on github: https://github.com/karpathy/micrograd - jupyter notebooks I built in this video: https://github.com/karpathy/nn-zero-to-hero/tree/master/lectures/micrograd - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - "discussion forum": nvm, use youtube comments below for now :) - (new) Neural Networks: Zero to Hero series Discord channel: https://discord.gg/3zy8kqD9Cp , for people who'd like to chat more and go beyond youtube comments Exercises: you should now be able to complete the following google collab, good luck!: https://colab.research.google.com/drive/1FPTx1RXtBfc4MaTkf7viZZD4U2F9gtKN?usp=sharing Chapters: 00:00:00 intro 00:00:25 micrograd overview 00:08:08 derivative of a simple function with one input 00:14:12 derivative of a function with multiple inputs 00:19:09 starting the core Value object of micrograd and its visualization 00:32:10 manual backpropagation example #1: simple expression 00:51:10 preview of a single optimization step 00:52:52 manual backpropagation example #2: a neuron 01:09:02 implementing the backward function for each operation 01:17:32 implementing the backward function for a whole expression graph 01:22:28 fixing a backprop bug when one node is used multiple times 01:27:05 breaking up a tanh, exercising with more operations 01:39:31 doing the same thing but in PyTorch: comparison 01:43:55 building out a neural net library (multi-layer perceptron) in micrograd 01:51:04 creating a tiny dataset, writing the loss function 01:57:56 collecting all of the parameters of the neural net 02:01:12 doing gradient descent optimization manually, training the network 02:14:03 summary of what we learned, how to go towards modern neural nets 02:16:46 walkthrough of the full code of micrograd on github 02:21:10 real stuff: diving into PyTorch, finding their backward pass for tanh 02:24:39 conclusion 02:25:20 outtakes :) #neural #network #backpropagation #lecture https://favtu.be/timelines/v/vEnetcj_728 Wed, 17 Aug 22 02:07:50 +0900 Stable diffusion dreams of "blueberry spaghetti" for one night Prompt: "blueberry spaghetti" Stable diffusion takes a noise vector as input and samples an image. To create this video I simply smoothly interpolate between randomly chosen noise vectors and render frames along the way. Links - Stable diffusion: https://stability.ai/blog - Code used to make this video: https://gist.github.com/karpathy/00103b0037c5aaea32fe1da1af553355 - My twitter: https://twitter.com/karpathy https://favtu.be/timelines/v/Jv1ayv-04H4 Tue, 16 Aug 22 10:28:15 +0900 00:01:05 I see C-3PO Stable diffusion dreams of steam punk neural networks A stable diffusion dream. The prompt was "ultrarealistic steam punk neural network machine in the shape of a brain, placed on a pedestal, covered with neurons made of gears. dramatic lighting. #unrealengine" the new and improved v2 version of this video is now here: https://www.youtube.com/watch?v=2oKjtvYslMY generated with this hacky script: https://gist.github.com/karpathy/00103b0037c5aaea32fe1da1af553355 The script slowly meanders through noise space to explore the space of possible generations for the fixed prompt. Stable diffusion: https://stability.ai/blog/stable-diffusion-announcement