タイムテーブル - Andrej Karpathy - 切り抜きDB Andrej Karpathyのタイムテーブルです。 https://favtu.be/timelines-rss/c/UCXUPKJO5MZQN11PqgIvyuvQ Fri, 28 Feb 25 07:29:56 +0900 Intro into the growing LLM ecosystem（00:00:00 - 00:02:54） https://favtu.be/timelines/v/EWvNQjAaOHw/s/0/e/174 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs The audio modes demos at pm were simply amazing!（00:01:30 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/90/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs ChatGPT interaction under the hood（00:02:54 - 00:13:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/174/e/792 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Basic LLM interactions examples（00:13:12 - 00:18:03） https://favtu.be/timelines/v/EWvNQjAaOHw/s/792/e/1083 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Exactly, I have observed this so many times!! Why do these chat platforms not have an option to branch out to a new chat (for exploring multiple ideas or something) from a particular answer point? Are there any technical challenges?（00:16:55 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/1015/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs new chat --->for new topic（00:17:31 - 01:12:25） https://favtu.be/timelines/v/EWvNQjAaOHw/s/1051/e/4345 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Be aware of the model you're using, pricing tiers（00:18:03 - 00:22:54） https://favtu.be/timelines/v/EWvNQjAaOHw/s/1083/e/1374 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Personal note:（00:21:00 - 00:37:00） https://favtu.be/timelines/v/EWvNQjAaOHw/s/1260/e/2220 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Thinking models and when to use them（00:22:54 - 00:31:00） https://favtu.be/timelines/v/EWvNQjAaOHw/s/1374/e/1860 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Tool use: internet search（00:31:00 - 00:42:04） https://favtu.be/timelines/v/EWvNQjAaOHw/s/1860/e/2524 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Is one Search one token in the context window?（00:34:28 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/2068/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs - Casual sneeze making the video even more fun（00:36:34 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/2194/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Bless you, Andrej-!（00:36:35 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/2195/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs search（00:37:00 - 01:22:00） https://favtu.be/timelines/v/EWvNQjAaOHw/s/2220/e/4920 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs We Vietnamese always cherish exceptional talents like you, Andrej.（00:41:26 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/2486/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Tool use: deep research（00:42:04 - 00:50:57） https://favtu.be/timelines/v/EWvNQjAaOHw/s/2524/e/3057 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs ChatGPT was not the first to offer Deep Research. Gemini made Deep Research available on December 11, 2024. ChatGPT added theirs February 2, 2025.（00:42:51 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/2571/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs You missed Gemini Deep Research. That’s the original one.（00:45:15 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/2715/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs What we would really need is ability to pass the response with all the provided references to another thinking + internet access AI system with a task "Does this article content match the provided references?". I'm pretty sure that different AI models do not accidentally hallucinate badly enough to fail this kind of verification task most of the time.（00:48:20 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/2900/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs File uploads, adding documents to context（00:50:57 - 00:59:00） https://favtu.be/timelines/v/EWvNQjAaOHw/s/3057/e/3540 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Re , accessing .epub in context would be a win.Imagine clicking Table of Contents Chapter inside of Cursor or ChatGPT Platform and having it ready for the selected LLM.. 📖 🙂（00:58:27 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/3507/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs I think Copilot in Edge allows you to ask questions in a taskpane and also supports marking as i remember. . Thanks for your Insights!（00:58:30 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/3510/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs you need the Highlight app. it literally takes into context whatever document you have opened in your system, so no copying is needed. very smooth（00:58:40 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/3520/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs I suggest using kortex for large amount of pdf or books that can be using with an LLM. I am not sure about each LLMs limit in terms of document upload (MB) and how is connected with token input limits, I would like to know more about this（00:58:50 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/3530/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs You could just have the ChatGPT floating window open while you read a book in full-screen. That way, you don’t have to keep switching between windows. 👍🏻（00:58:50 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/3530/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs "don't read books alone"（00:58:58 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/3538/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Tool use: python interpreter, messiness of the ecosystem（00:59:00 - 01:04:35） https://favtu.be/timelines/v/EWvNQjAaOHw/s/3540/e/3875 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Gemini's prediction is not actually close. It is lower by an order of 3. But another amazing video by Andrej ! Thank you :)（01:04:08 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/3848/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs ChatGPT Advanced Data Analysis, figures, plots（01:04:35 - 01:09:00） https://favtu.be/timelines/v/EWvNQjAaOHw/s/3875/e/4140 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs keep in mind if you reading this, just because it uses an internet source, doesn’t mean it won’t hallucinate content it thinks it found in the source（01:05:23 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/3923/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs 0.1 is a heuristic to avoid 0, which may behave badly?（01:05:53 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/3953/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Claude Artifacts, apps, diagrams（01:09:00 - 01:14:02） https://favtu.be/timelines/v/EWvNQjAaOHw/s/4140/e/4442 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs This is pure gold. Andrej is the best teacher on all things AI. He teaches with such clarity and simplicity that the knowledge just sticks. I just wish that the part about coding between - 1. a disclaimer when there are high vulnerabilities in node dependencies (2. discusses the legal aspects of using code generated by llms or llm powered tools like cursor, windsurf, github copilot etc. I really wish such videos talk about this crucial aspect else most viewers will get a sense that software development is as simple as just prompting LLMs for code and they can use the code generated as it is. There are many cases when such LLMs spit out copyrighted code or code under licenses and using them without attribution is risky.（01:09:18 - 01:22:00） https://favtu.be/timelines/v/EWvNQjAaOHw/s/4158/e/4920 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs ---> conceptual diagram（01:12:25 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/4345/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Love the conceptual diagram idea. Very very useful（01:14:00 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/4440/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Cursor: Composer, writing code（01:14:02 - 01:22:28） https://favtu.be/timelines/v/EWvNQjAaOHw/s/4442/e/4948 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs )（01:19:23 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/4763/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs The confetti moment got me excited too. Amazing video, Andrej, thank you!（01:20:35 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/4835/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs showed（01:22:00 - 01:19:23） https://favtu.be/timelines/v/EWvNQjAaOHw/s/4920/e/4763 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs talk to llms（01:22:00 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/4920/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Audio (Speech) Input/Output（01:22:28 - 01:27:37） https://favtu.be/timelines/v/EWvNQjAaOHw/s/4948/e/5257 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs What a gigachad. And yet for some reasons he doesn't seem to be aware that his Mac comes with Dictation feature (). Maybe he has an older model of MacOS. Maybe I'm missing something but this section of the video makes no sense to me. But again, what an amazing video by a generous genius!（01:25:10 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/5110/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs The native ChatGPT app for macOS does have the mic icon.（01:25:20 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/5120/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Why don't you use mac dictate feature?（01:25:26 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/5126/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Advanced Voice Mode aka true audio inside the model（01:27:37 - 01:37:09） https://favtu.be/timelines/v/EWvNQjAaOHw/s/5257/e/5829 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs kind of how shazam works under the hood, by getting a graph made for the audio spectogram and by identifying the peak points in the graph with background noise minimized and then it those peak points being converted to audio fingerprints and at last based on the fingerprint it searches its database of millions of songs.（01:28:20 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/5300/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Your reaction at killed me lmao（01:35:14 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/5714/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs NotebookLM, podcast generation（01:37:09 - 01:40:20） https://favtu.be/timelines/v/EWvNQjAaOHw/s/5829/e/6020 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Image input, OCR（01:40:20 - 01:47:02） https://favtu.be/timelines/v/EWvNQjAaOHw/s/6020/e/6422 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs woke up in the middle of the night to find that I had been listening to this all night. If I magically know a bunch of shit about LLMs….im going to be shook（01:44:44 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/6284/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs For those interested, the math problem at is not that tricky 🙃.（01:45:00 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/6300/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs No Andrej, you failed me to trick😎😅（01:45:28 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/6328/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Image output, DALL-E, Ideogram, etc.（01:47:02 - 01:49:14） https://favtu.be/timelines/v/EWvNQjAaOHw/s/6422/e/6554 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Video input, point and talk on app（01:49:14 - 01:52:23） https://favtu.be/timelines/v/EWvNQjAaOHw/s/6554/e/6743 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Video output, Sora, Veo 2, etc etc.（01:52:23 - 01:53:29） https://favtu.be/timelines/v/EWvNQjAaOHw/s/6743/e/6809 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs ChatGPT memory, custom instructions（01:53:29 - 01:58:38） https://favtu.be/timelines/v/EWvNQjAaOHw/s/6809/e/7118 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs whenever you make a typo while typing, that should be a reminder to type with superwhisper instead（01:54:50 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/6890/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs "I am Andrej Karpathy; Yes - the AI researcher" What an insane flex. Imagine confirming to an LLM that it's indeed talking to that guy you actually have training memory on.（01:57:55 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/7075/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Custom GPTs（01:58:38 - 02:06:30） https://favtu.be/timelines/v/EWvNQjAaOHw/s/7118/e/7590 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Can you add a reverse (round-trip) button to your translator? It's a great way to test the "stability" of a translation.（02:02:19 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/7339/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs agree 👍 going to use it（02:03:26 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/7406/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs Summary（02:06:30 - 02:11:12） https://favtu.be/timelines/v/EWvNQjAaOHw/s/7590/e/7872 Fri, 28 Feb 25 07:29:56 +0900 How I use LLMs introduction（00:00:00 - 00:01:00） https://favtu.be/timelines/v/7xTGNNLPyMI/s/0/e/60 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT - Introduction（00:00:01 - 00:01:04） https://favtu.be/timelines/v/7xTGNNLPyMI/s/1/e/64 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT pretraining data (internet)（00:01:00 - 00:07:47） https://favtu.be/timelines/v/7xTGNNLPyMI/s/60/e/467 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT - LLM Pre-training（00:01:04 - 00:15:13） https://favtu.be/timelines/v/7xTGNNLPyMI/s/64/e/913 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT Atound, you explain a really interesting notion, that models need to "think" before producing a complex response, thats because each layer in a neural network has finite computation. I feel like its somewhat related to the notion of computational irreducibility Stephen Wolfram talks about. This is also why we humans need to spend some time thinking about complex issues before coming up with a good response.（00:01:49 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/109/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT But what if the ultimate joke about pelicans is actually 'the the the the the the,' but we simply don't have enough intelligence to understand it—just like an unusual move in the game of Go? XD（00:03:02 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/182/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT wow amazing hours so much in few hours .. Saved me hours of research and insprie me for more ..great work looking forward for new such interesting videos..（00:03:30 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/210/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT at , talks about eliminating racist sites during corpus preprocessing. This can introduce bias by eliminating candid discussion of, for example, average IQ test scores of racial subgroups. Claude refuses to answer this altogether, calling race a constructed concept. ChatGPT and Gemini, at the time I queried them, both produced valid, honest outputs, which aligned with the research. Those of you so enamored with Claude are still trapped in Dario's echo-chamber. But society has moved on, now (2025). Will you?（00:03:50 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/230/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT tokenization（00:07:47 - 00:14:27） https://favtu.be/timelines/v/7xTGNNLPyMI/s/467/e/867 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT neural network I/O（00:14:27 - 00:20:11） https://favtu.be/timelines/v/7xTGNNLPyMI/s/867/e/1211 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT - Neural Net & Training（00:15:13 - 00:40:14） https://favtu.be/timelines/v/7xTGNNLPyMI/s/913/e/2414 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT neural network internals（00:20:11 - 00:26:01） https://favtu.be/timelines/v/7xTGNNLPyMI/s/1211/e/1561 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT inference（00:26:01 - 00:31:09） https://favtu.be/timelines/v/7xTGNNLPyMI/s/1561/e/1869 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT GPT-2: training and inference（00:31:09 - 00:42:52） https://favtu.be/timelines/v/7xTGNNLPyMI/s/1869/e/2572 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT Somewhere around , you said something about training 1 million tokens. Do you mean you train chunks of 1 million tokens to generate output or you train different tokens that add up to a million to generate output?（00:36:52 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/2212/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT - GPUs & Model Costs（00:40:14 - 01:01:06） https://favtu.be/timelines/v/7xTGNNLPyMI/s/2414/e/3666 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT Llama 3.1 base model inference（00:42:52 - 00:59:23） https://favtu.be/timelines/v/7xTGNNLPyMI/s/2572/e/3563 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT : Parallel universes !!! Just loving these analogies - awesome !（00:55:22 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/3322/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT pretraining to post-training（00:59:23 - 01:01:06） https://favtu.be/timelines/v/7xTGNNLPyMI/s/3563/e/3666 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT post-training data (conversations)（01:01:06 - 01:20:32） https://favtu.be/timelines/v/7xTGNNLPyMI/s/3666/e/4832 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT - Build LLM Assistant（01:01:06 - 02:07:30） https://favtu.be/timelines/v/7xTGNNLPyMI/s/3666/e/7650 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT "something went wrong" 😂 lol I love that he left this in there!（01:18:46 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/4726/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT his genuine laugh at ChatGPT error is so pure and spontaneous. How can someone not love Karpathy!!?? Sir you are pure Gold for humanity.（01:18:47 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/4727/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT hallucinations, tool use, knowledge/working memory（01:20:32 - 01:41:46） https://favtu.be/timelines/v/7xTGNNLPyMI/s/4832/e/6106 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT The chapter about hallucinations was so insightful. Never heard about it as an issue of the dataset, i.e., it wasn't trained to say "I don't know" and how one can test the knowledge of the model. Thanks!（01:20:32 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/4832/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT Observation: Approx. at , Andrej tests the question "Who is Orson Kovacs" using falcon-7b-instruct in HF playground, the temperature is still 1.0 which will make the model to respond in a balanced manner between randomness and deterministic. Although it makes up stuff to behave like hallucinations, it is good to test out with temperature less or more than 1.0 to understand how the factuality of the data varies.（01:23:50 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/5030/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT you mentioned around mark - the reason why you allow the model to say i don't know, instead of augmenting it with the new knowledge, is it because there's infinite amount of knowledge to learn so that it's virtually impossible to learn knowledge, and thus it's better to train it to know when to refuse? In other words, say if somehow the model CAN learn ALL the knowledge of the world, we won't need to train it to stop hallucinating? Thanks.（01:30:00 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/5400/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT Thanks for the informative video! I have a question about training language models for tool use, specifically regarding the process you described around（01:33:38 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/5618/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT knowledge of self（01:41:46 - 01:46:56） https://favtu.be/timelines/v/7xTGNNLPyMI/s/6106/e/6416 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT models need tokens to think（01:46:56 - 02:01:11） https://favtu.be/timelines/v/7xTGNNLPyMI/s/6416/e/7271 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT @. Question. I was just reading a paper recently (I believe it was from Anthropic, but sadly I can't find it now) that when they have looked at "thinking models", it appears the final answer is generally already determined well before the reasoning process begins. Then the model just fills in the chain of thought to get from the question to where it wants to go. Isn't this exactly what you said is not the correct way to handle this? Can you comment on why, if this is the "wrong" approach, it seems to be what modern models are doing?（01:52:00 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/6720/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT @ that is elucidating! This is the first time I’ve heard of this concept. Thank you Andrej.（01:55:49 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/6949/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT This teacher is very good at giving cute examples Appreciate it and I agree it.（01:55:50 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/6950/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT tokenization revisited: models struggle with spelling（02:01:11 - 02:04:53） https://favtu.be/timelines/v/7xTGNNLPyMI/s/7271/e/7493 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT Wow.. love this explanation about why these models fail at character related and counting related task（02:04:04 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/7444/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT jagged intelligence（02:04:53 - 02:07:28） https://favtu.be/timelines/v/7xTGNNLPyMI/s/7493/e/7648 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT supervised finetuning to reinforcement learning（02:07:28 - 02:14:42） https://favtu.be/timelines/v/7xTGNNLPyMI/s/7648/e/8082 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT - Model Training in Practice（02:07:30 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/7650/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT reinforcement learning（02:14:42 - 02:27:47） https://favtu.be/timelines/v/7xTGNNLPyMI/s/8082/e/8867 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT DeepSeek-R1（02:27:47 - 02:42:07） https://favtu.be/timelines/v/7xTGNNLPyMI/s/8867/e/9727 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT Deepseek says “$3 is a bit expensive for an apple, but maybe they’re organic or something” 😂（02:34:21 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/9261/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT What a treat!!! At , haha when you say this is very busy very ugly because of google not being able to nail that was epic hahah（02:41:08 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/9668/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT AlphaGo（02:42:07 - 02:48:26） https://favtu.be/timelines/v/7xTGNNLPyMI/s/9727/e/10106 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT Thank you for the video Andrej! One small note: at , the dashed line in the AlphaGo Zero plot is the Elo of the version of AlphaGo that *defeated* Lee in 2016 (not the Elo of Lee himself).（02:43:05 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/9785/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT reinforcement learning from human feedback (RLHF)（02:48:26 - 03:09:39） https://favtu.be/timelines/v/7xTGNNLPyMI/s/10106/e/11379 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT Tiny typo "let's add it to the dataset and give it an ordering that's extremely like a score of 5" -> SHOULD BE "let's add it to the dataset and give it an ordering that's extremely like a score of 1"（03:03:44 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/11024/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT preview of things to come（03:09:39 - 03:15:15） https://favtu.be/timelines/v/7xTGNNLPyMI/s/11379/e/11715 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT keeping track of LLMs（03:15:15 - 03:18:34） https://favtu.be/timelines/v/7xTGNNLPyMI/s/11715/e/11914 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT if you have come till this time stamp then finish the video and go and build something with LLMs.😊（03:16:59 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/11819/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT where to find LLMs（03:18:34 - 03:21:46） https://favtu.be/timelines/v/7xTGNNLPyMI/s/11914/e/12106 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT grand summary（03:21:46 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/12106/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT In principle these models are capable of analogies no human has had. Wow😮（03:29:54 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/12594/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT Thank you Andrej for this! Please continue putting contents like this and you are one of the best teachers in this space who can explain in this level of detail. The entire is pure gold and very grateful that you are putting this level of time and effort ❤（03:31:23 - 03:31:24） https://favtu.be/timelines/v/7xTGNNLPyMI/s/12683/e/12684 Thu, 06 Feb 25 03:23:47 +0900 Deep Dive into LLMs like ChatGPT intro: Let’s reproduce GPT-2 (124M)（00:00:00 - 00:03:39） https://favtu.be/timelines/v/l8pRSuU81PU/s/0/e/219 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Exploring the Target:* The video starts by loading the pre-trained GPT-2 (124M) model from Hugging Face Transformers and examining its weights and architecture.（00:00:00 - 00:13:47） https://favtu.be/timelines/v/l8pRSuU81PU/s/0/e/827 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🤖 Reproducing GPT-2 124M model- Reproducing the GPT-2 model involves understanding its release structure and model variations.（00:00:00 - 00:01:09） https://favtu.be/timelines/v/l8pRSuU81PU/s/0/e/69 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) M) model, including its state dictionary and tensor shapes. We learn how the model's vocabulary size and embedding dimensions are represented within these tensors. (-（00:00:00 - 00:04:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/0/e/240 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Reproducing the GPT-2 124M version（00:00:02 - 00:02:06） https://favtu.be/timelines/v/l8pRSuU81PU/s/2/e/126 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 💻 Model Parameters Overview- GPT-2 miniseries comprises models of various sizes, with the 124 million parameter model being a significant variant.- Model parameters dictate its size, layer count, and channel dimensions, affecting downstream task performance.（00:01:09 - 00:02:06） https://favtu.be/timelines/v/l8pRSuU81PU/s/69/e/126 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 💰 Reproducibility and Cost- Reproducing the GPT-2 124M model is now more accessible and affordable due to advances in hardware and cloud computing.- Achieving comparable model performance can be done in a relatively short time and at a reasonable cost.（00:02:06 - 00:03:18） https://favtu.be/timelines/v/l8pRSuU81PU/s/126/e/198 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Validation loss measures model's performance on unseen data.（00:02:06 - 00:06:21） https://favtu.be/timelines/v/l8pRSuU81PU/s/126/e/381 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 📚 Reference Material- Access to GPT-2 weights facilitates reproduction, but additional references like the GPT-3 paper provide crucial details for optimization and training settings.- Combining insights from both GPT-2 and GPT-3 papers enhances reproducibility and understanding of the model architecture.（00:03:18 - 00:05:37） https://favtu.be/timelines/v/l8pRSuU81PU/s/198/e/337 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) exploring the GPT-2 (124M) OpenAI checkpoint（00:03:39 - 00:13:47） https://favtu.be/timelines/v/l8pRSuU81PU/s/219/e/827 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) @ now... so far so good...（00:03:43 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/223/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) compared to the original Transformer are explored, such as the removal of the encoder and cross-attention mechanism. Further, modifications to layer normalization placement and the addition of a final layer normalization layer are highlighted. (-（00:04:00 - 00:08:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/240/e/480 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🧠 Understanding Model Structure- Exploring the structure of the GPT-2 model involves inspecting token and positional embeddings, as well as layer weights.- The visualization of embeddings and weights reveals insights into the model's learning process and representation.（00:05:37 - 00:13:13） https://favtu.be/timelines/v/l8pRSuU81PU/s/337/e/793 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) GPT-2 token and position embeddings explained（00:06:21 - 00:08:43） https://favtu.be/timelines/v/l8pRSuU81PU/s/381/e/523 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) , aligning it with the schema used by Hugging Face Transformers. This skeleton includes modules for token and positional embeddings, Transformer blocks, final layer normalization, and the language model head. (-（00:08:00 - 00:12:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/480/e/720 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Understanding token positions and embeddings in GPT-2 (124M)（00:08:43 - 00:12:54） https://favtu.be/timelines/v/l8pRSuU81PU/s/523/e/774 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) has the freedom to learn the position embeddings (the original transformer paper hardcoded the positional embeddings)（00:08:49 - 00:18:14） https://favtu.be/timelines/v/l8pRSuU81PU/s/529/e/1094 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) is discussed. (-（00:12:00 - 00:16:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/720/e/960 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Implementing and understanding GPT-2 (124M) model architecture.（00:12:54 - 00:15:02） https://favtu.be/timelines/v/l8pRSuU81PU/s/774/e/902 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🛠 Implementing Model Architecture- Developing a custom GPT-2 model involves constructing the model architecture, including token and position embeddings, transformer blocks, and classification layers.- Aligning the custom implementation with existing frameworks like Hugging Face Transformers aids in loading pre-trained weights and ensures compatibility.（00:13:13 - 00:14:21） https://favtu.be/timelines/v/l8pRSuU81PU/s/793/e/861 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) SECTION 1: implementing the GPT-2 nn.Module（00:13:47 - 00:28:08） https://favtu.be/timelines/v/l8pRSuU81PU/s/827/e/1688 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Implementing the GPT-2 nn.Module:* A custom GPT-2 class is built in PyTorch, mirroring the Hugging Face architecture and loading the pre-trained weights for verification.（00:13:47 - 00:31:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/827/e/1860 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🔍 Model Architecture Differences- GPT-2's architecture includes modifications like layer normalization adjustments and additional layer normalization in the final self-attention block compared to the original Transformer.- Understanding architectural differences is crucial for accurately implementing and reproducing the GPT-2 model.（00:14:21 - 00:15:15） https://favtu.be/timelines/v/l8pRSuU81PU/s/861/e/915 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Creating a matching schema for loading weights easily.（00:15:02 - 00:19:20） https://favtu.be/timelines/v/l8pRSuU81PU/s/902/e/1160 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🏗 Defining Model Blocks- Designing the transformer block involves structuring the forward pass, incorporating attention mechanisms, feedforward networks, and residual connections.- Optimizing the block structure for efficient information flow and gradient propagation is essential for model performance.（00:15:15 - 00:20:20） https://favtu.be/timelines/v/l8pRSuU81PU/s/915/e/1220 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) 's implementation through tensor manipulation and its algorithmic similarity to previous implementations. (-（00:16:00 - 00:20:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/960/e/1200 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) You want a direct residual connection from the target to the input embeddings, skipping layer normalization (I need to understand what layer normalization is)（00:18:14 - 00:22:11） https://favtu.be/timelines/v/l8pRSuU81PU/s/1094/e/1331 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Found this video first, then at about when you started talking about residuals and micrograd, went back to your zero-to-hero series and watched everything as a prerequisite. now i understand how residuals helps in stabilizing the training. the gradient distribution into branches analogy really changed the perspective for me. this video should be kept safe in a time capsule（00:19:00 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/1140/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) The Transformer involves repeated application of map and reduce（00:19:20 - 00:21:22） https://favtu.be/timelines/v/l8pRSuU81PU/s/1160/e/1282 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Implementing the Forward Pass and Text Generation: The forward pass of the network is implemented, outlining how input token indices are processed to produce logits for predicting the next token in a sequence. This sets the stage for generating text from the model. (-（00:20:00 - 00:24:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/1200/e/1440 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Its funny how his description of attention as reduce-map description at can be thought of as map-reduce :)（00:20:00 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/1200/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) the comparison between attention and mlp is impressive（00:20:10 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/1210/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🧠 Understanding the Transformer Architecture- The Transformer architecture relies on attention mechanisms and multi-layer perceptrons (MLPs).- Attention is crucial for communication and individual information processing within Transformer blocks.- Transformers utilize repeated application of "map" and "reduce" operations for information exchange and refinement.（00:20:20 - 00:21:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/1220/e/1260 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🛠 Implementing the MLP Block- The MLP block consists of linear projections sandwiched between G nonlinearity.- The G nonlinearity resembles a smoother version of ReLU and contributes to better gradient flow.- Historical reasons and empirical evidence support the use of the approximate G nonlinearity in GPT-2 reproduction.（00:21:00 - 00:23:41） https://favtu.be/timelines/v/l8pRSuU81PU/s/1260/e/1421 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) GPT-2 used the 10h approximate version of G instead of the exact version.（00:21:22 - 00:25:09） https://favtu.be/timelines/v/l8pRSuU81PU/s/1282/e/1509 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Activation function GELU is an approximation（00:22:11 - 00:55:27） https://favtu.be/timelines/v/l8pRSuU81PU/s/1331/e/3327 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🧩 Exploring the Attention Operation- Multi-headed attention in Transformers involves parallel computation of attention heads.- The attention operation remains algorithmically equivalent to previous implementations but is more efficient in PyTorch.- Careful variable naming facilitates seamless weight transfer from existing models during reproduction.（00:23:41 - 00:40:21） https://favtu.be/timelines/v/l8pRSuU81PU/s/1421/e/2421 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) model. This involves tokenizing a prefix string, moving the model to a CUDA device for GPU acceleration, and performing sampling-based text generation. (-（00:24:00 - 00:28:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/1440/e/1680 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) GPT-2 (124M) implementation details（00:25:09 - 00:27:08） https://favtu.be/timelines/v/l8pRSuU81PU/s/1509/e/1628 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Efficient implementation in PyTorch for GPT-2 (124M) model（00:27:08 - 00:30:59） https://favtu.be/timelines/v/l8pRSuU81PU/s/1628/e/1859 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Introducing the Tiny Shakespeare Dataset: This part introduces the Tiny Shakespeare dataset as a small and manageable dataset for initial model training and debugging. Basic statistics of the dataset are explored. (-（00:28:00 - 00:32:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/1680/e/1920 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) loading the huggingface/GPT-2 parameters（00:28:08 - 00:31:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/1688/e/1860 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) This series is amazing, but I have a bit of confusion. At the timestamp, you mentioned that the weights are transposed and referenced something about TensorFlow. However, I think in PyTorch, the weights for a linear layer are initialized as torch.empty(out_features, in_features)so is this the case u needed to transpose the weightsand Furthermore, the weights you are transposing all belong to linear layers, yet for the last lm_head layer, which is also a linear layer, you are not transposing that weight.Am I mistaken here, or is there something else going on?（00:30:10 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/1810/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Forwarding the GPT-2 model requires processing token indices and embeddings.（00:30:59 - 00:32:52） https://favtu.be/timelines/v/l8pRSuU81PU/s/1859/e/1972 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) implementing the forward pass to get logits（00:31:00 - 00:33:31） https://favtu.be/timelines/v/l8pRSuU81PU/s/1860/e/2011 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Forward Pass and Sampling:* The forward pass is implemented to calculate logits, and a sampling loop is added to generate text from the model.（00:31:00 - 01:22:18） https://favtu.be/timelines/v/l8pRSuU81PU/s/1860/e/4938 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) model. It introduces the concept of batching and creating input-target pairs for loss calculation. (-（00:32:00 - 00:36:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/1920/e/2160 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Explaining the forward pass of the GPT-2 network（00:32:52 - 00:36:36） https://favtu.be/timelines/v/l8pRSuU81PU/s/1972/e/2196 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) sampling init, prefix tokens, tokenization（00:33:31 - 00:37:02） https://favtu.be/timelines/v/l8pRSuU81PU/s/2011/e/2222 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Creating a Simple Data Loader: This section refactors the code to create a simple data loader object responsible for loading tokenized data from the Tiny Shakespeare dataset and generating batches suitable for training the model. (-（00:36:00 - 00:40:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/2160/e/2400 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Generating logits and probabilities for token prediction（00:36:36 - 00:38:34） https://favtu.be/timelines/v/l8pRSuU81PU/s/2196/e/2314 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) sampling loop（00:37:02 - 00:41:47） https://favtu.be/timelines/v/l8pRSuU81PU/s/2222/e/2507 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) (time )（00:37:46 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/2266/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) why do we only keep the last column of the logits?（00:38:10 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/2290/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Using top K by default (50) helps keep the model on track（00:38:34 - 00:42:24） https://favtu.be/timelines/v/l8pRSuU81PU/s/2314/e/2544 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Calculating Loss and Backpropagation: The forward function is adjusted to return not just the logits but also the calculated loss based on provided target tokens. Cross-entropy loss is used, and the initial loss is sanity-checked to ensure reasonable starting probabilities. (-（00:40:00 - 00:44:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/2400/e/2640 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🤖 Replicating GPT-2 Model Initialization- Replicating the GPT-2 model initialization process.- Transitioning from pre-trained weights to initializing from random numbers.- Exploring the straightforward process of using a random model in PyTorch.（00:40:21 - 00:43:30） https://favtu.be/timelines/v/l8pRSuU81PU/s/2421/e/2610 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) sample, auto-detect the device（00:41:47 - 00:45:50） https://favtu.be/timelines/v/l8pRSuU81PU/s/2507/e/2750 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) : My quick summary at ! A 2000-line GPT-2 implementation in Huggingface has been condensed to almost 100 lines. The weights from HF GPT-2 were replicated in this new version, using the same sampling parameters, seed, and generating identical output. A notable improvement is the restructuring of the implementation, where all heads are now integrated within a single matrix, applying some neat matrix transposes while maintaining parallelism and enhancing comprehension. This is far easier to understand compared to many other complicated multihead implementations I've seen. The next step involves training this model from the ground up.（00:42:00 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/2520/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Using GPT-2 (124M) for model initialization（00:42:24 - 00:44:17） https://favtu.be/timelines/v/l8pRSuU81PU/s/2544/e/2657 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🔍 Detecting and Utilizing Device in PyTorch- Automatically detecting and utilizing available devices in PyTorch.- Strategies for choosing the highest compute-capable device.- Facilitating code compatibility across different hardware configurations.（00:43:30 - 00:46:11） https://favtu.be/timelines/v/l8pRSuU81PU/s/2610/e/2771 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Implementing Optimization with AdamW: This section introduces the AdamW optimizer as an alternative to stochastic gradient descent (SGD), highlighting its advantages for language model training. The optimization loop is implemented, including gradient accumulation and loss printing. (-（00:44:00 - 00:48:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/2640/e/2880 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Initializing model on correct device is crucial for performance（00:44:17 - 00:48:08） https://favtu.be/timelines/v/l8pRSuU81PU/s/2657/e/2888 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) let’s train: data batches (B,T) → logits (B,T,C)（00:45:50 - 00:52:53） https://favtu.be/timelines/v/l8pRSuU81PU/s/2750/e/3173 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 📄 Preparing and Tokenizing Dataset- Introduction to the Tiny Shakespeare dataset for training.- Obtaining and processing the dataset for tokenization.- Initial exploration and preprocessing steps for training data.（00:46:11 - 00:52:05） https://favtu.be/timelines/v/l8pRSuU81PU/s/2771/e/3125 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Understanding and Addressing Device Mismatches: This part emphasizes the importance of ensuring all tensors and model components reside on the same device (CPU or GPU) to avoid errors during training. A bug related to tensor device mismatch is identified and corrected. (-（00:48:00 - 00:52:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/2880/e/3120 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Transforming single sequence into batch with structured tokens（00:48:08 - 00:50:03） https://favtu.be/timelines/v/l8pRSuU81PU/s/2888/e/3003 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Creating input and labels for Transformer（00:50:03 - 00:54:02） https://favtu.be/timelines/v/l8pRSuU81PU/s/3003/e/3242 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) model based on the original paper's guidelines. This includes using specific standard deviations for different layer types and scaling residual connections to control activation growth. (-（00:52:00 - 00:56:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/3120/e/3360 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) -- 🛠 Implementing Data Loader and Loss Calculation- Building a data loader to feed token sequences into the Transformer model.- Setting up the forward pass to calculate the loss function.- Establishing a structured approach for loss calculation and gradient updates.（00:52:05 - 00:56:53） https://favtu.be/timelines/v/l8pRSuU81PU/s/3125/e/3413 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) cross entropy loss（00:52:53 - 00:56:42） https://favtu.be/timelines/v/l8pRSuU81PU/s/3173/e/3402 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Flattening multi-dimensional tensors for cross entropy calculation.（00:54:02 - 00:56:04） https://favtu.be/timelines/v/l8pRSuU81PU/s/3242/e/3364 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Calculating the estimated loss at initialization（00:55:27 - 00:57:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/3327/e/3420 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) GPU, focusing on its theoretical performance limits in terms of Teraflops for different floating-point precisions. The importance of memory bandwidth limitations is also discussed. (-（00:56:00 - 01:00:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/3360/e/3600 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) The loss at initialization is expected to be around 10.82 but is seen around 11, which suggests a diffused probability distribution at initialization.（00:56:04 - 01:00:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/3364/e/3600 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Fun Fact: -ln(1/50257) = 10.82 but simply ln(50257) also gives the same answer.（00:56:10 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/3370/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) optimization loop: overfit a single batch（00:56:42 - 01:02:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/3402/e/3720 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Question regarding overfitting a single batch .（00:56:42 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/3402/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🧮 Optimizing Model Parameters with AdamW- Implementing optimization using the AdamW optimizer.- Understanding the role and benefits of AdamW compared to SGD.- Executing gradient updates and monitoring loss during the optimization process.（00:56:53 - 01:00:17） https://favtu.be/timelines/v/l8pRSuU81PU/s/3413/e/3617 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Pytorch library had bugs that the canonical version (e.g. Adam) is the buggy version (fixed in AdamW)（00:57:00 - 01:01:01） https://favtu.be/timelines/v/l8pRSuU81PU/s/3420/e/3661 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) , are introduced as ways to trade precision for significant speed improvements. (-（01:00:00 - 01:04:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/3600/e/3840 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Explaining the device issue and fixing tensor moving bug.（01:00:00 - 01:01:52） https://favtu.be/timelines/v/l8pRSuU81PU/s/3600/e/3712 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🧠 Introduction to Model Optimization- Optimizing model training requires careful handling of tensors and device placement.- Overfitting a single batch is an initial step in understanding model behavior.- Transitioning from overfitting a single batch to optimizing with multiple batches requires implementing a data loader.（01:00:17 - 01:02:03） https://favtu.be/timelines/v/l8pRSuU81PU/s/3617/e/3723 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Attempting to overfit on a single example（01:01:01 - 01:12:53） https://favtu.be/timelines/v/l8pRSuU81PU/s/3661/e/4373 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Creating a simple data loader for iterating through batches of data.（01:01:52 - 01:05:46） https://favtu.be/timelines/v/l8pRSuU81PU/s/3712/e/3946 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) data loader lite（01:02:00 - 01:06:14） https://favtu.be/timelines/v/l8pRSuU81PU/s/3720/e/3974 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 📊 Implementation of a Simple Data Loader- The data loader reads text files and tokenizes them for model input.- It divides the data into batches, ensuring smooth iteration over the dataset.- Basic functionality covers chunking data and managing batch transitions.（01:02:03 - 01:06:24） https://favtu.be/timelines/v/l8pRSuU81PU/s/3723/e/3984 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) I see at for the batch processing, you are marching along by an index of `B * T`. Instead, what would be the implications of changing this to a sliding window (+1 indexing) such that we get overlapping samples? I realise this would create `len(self.tokens) - block_size` samples leading to a far greater number of batches per epoch, is this the only aspect?（01:02:16 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/3736/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) precision in PyTorch to leverage tensor cores and achieve a substantial speedup in training without noticeable accuracy degradation. (-（01:04:00 - 01:08:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/3840/e/4080 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Bug in GPT-2 training process（01:05:46 - 01:07:46） https://favtu.be/timelines/v/l8pRSuU81PU/s/3946/e/4066 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) parameter sharing wte and lm_head（01:06:14 - 01:13:47） https://favtu.be/timelines/v/l8pRSuU81PU/s/3974/e/4427 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🐛 Fixing a Weight Initialization Bug- Identifies a bug in weight initialization concerning weight tying in GPT-2 training.- Explains the significance of weight tying in reducing parameters and improving performance.- Implements a fix by redirecting pointers to the same tensor, saving parameters and optimizing performance.（01:06:24 - 01:13:45） https://favtu.be/timelines/v/l8pRSuU81PU/s/3984/e/4425 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Common weight tying scheme in Transformer models（01:07:46 - 01:11:41） https://favtu.be/timelines/v/l8pRSuU81PU/s/4066/e/4301 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Further Optimization with Torch Compile and Kernel Fusion: The torch.compile function is introduced as a powerful optimization technique that can analyze and fuse multiple operations into single kernels, reducing memory bandwidth bottlenecks and increasing throughput. (-（01:08:00 - 01:12:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/4080/e/4320 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) source code (at ) but I can't seem to find it in your PyTorch implementation.（01:11:13 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/4273/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) the weights sharing the dimensions of wte and lm head are different, is it okay?（01:11:32 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/4292/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Weight sharing scheme reduces parameters and improves efficiency（01:11:41 - 01:13:45） https://favtu.be/timelines/v/l8pRSuU81PU/s/4301/e/4425 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Identifying Performance Bottlenecks: "Nice" vs. "Ugly" Numbers: This section highlights a less obvious optimization technique: ensuring that key parameters like vocabulary size and batch size are "nice" numbers with many powers of two. This helps align computations with CUDA's block-based execution model and avoids inefficient boundary cases. (-（01:12:00 - 01:16:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/4320/e/4560 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) % of the parameters)（01:12:53 - 01:23:50） https://favtu.be/timelines/v/l8pRSuU81PU/s/4373/e/5030 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🎚 Fine-tuning Model Initialization- Discusses the importance of model weight initialization in training stability and performance.- Mimics GPT-2 initialization scheme based on observed patterns in released source code.- Introduces a scaling factor for residual layers' weights initialization to control activation growth in the network.（01:13:45 - 01:20:27） https://favtu.be/timelines/v/l8pRSuU81PU/s/4425/e/4827 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Follow GPT-2 initialization scheme for better model performance（01:13:45 - 01:17:37） https://favtu.be/timelines/v/l8pRSuU81PU/s/4425/e/4657 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) model initialization: std 0.02, residual init（01:13:47 - 01:22:18） https://favtu.be/timelines/v/l8pRSuU81PU/s/4427/e/4938 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) : Summary at（01:14:00 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/4440/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Adjusting Vocabulary Size for Optimal Performance: This part demonstrates how a slight increase in vocabulary size to the nearest power of two can surprisingly lead to a performance boost due to more efficient CUDA kernel execution. (-（01:16:00 - 01:20:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/4560/e/4800 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) shouldn't Embedding std be set to 0.01 ?（01:16:03 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/4563/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Controlling growth of activations in the residual stream（01:17:37 - 01:19:50） https://favtu.be/timelines/v/l8pRSuU81PU/s/4657/e/4790 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Setting flags and scaling standard deviation in GPT-2 model initialization.（01:19:50 - 01:23:53） https://favtu.be/timelines/v/l8pRSuU81PU/s/4790/e/5033 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Implementing Gradient Accumulation for Large Batch Sizes: This section introduces gradient accumulation as a technique to simulate very large batch sizes that wouldn't fit in GPU memory by accumulating gradients over multiple micro-batches before performing a weight update. (-（01:20:00 - 01:24:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/4800/e/5040 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Hi Andrej should we skip the pos embedding initialization with std 0.01 like in the original code and stick to the 0.02 ?（01:20:21 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/4821/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🛠 Implementing GPT-2 Initialization- Implementing scaling down the standard deviation for proper initialization.- Clarification on the two times number of layers in the Transformer.- Setting seeds for reproducibility and initializing GPT-2 model.（01:20:27 - 01:23:07） https://favtu.be/timelines/v/l8pRSuU81PU/s/4827/e/4987 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms（01:22:18 - 01:28:14） https://favtu.be/timelines/v/l8pRSuU81PU/s/4938/e/5294 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Understanding Hardware:* The video emphasizes understanding GPU capabilities, particularly tensor cores and memory bandwidth.（01:22:18 - 01:28:14） https://favtu.be/timelines/v/l8pRSuU81PU/s/4938/e/5294 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 💻 Optimizing Hardware Utilization- Assessing available hardware resources, including GPUs.- Understanding the importance of memory bandwidth in GPU utilization.- Exploring precision options (float32, tf32, bfloat16) for performance optimization.（01:23:07 - 01:28:12） https://favtu.be/timelines/v/l8pRSuU81PU/s/4987/e/5292 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) import code; code.interact(local=locals())（01:23:50 - 01:51:59） https://favtu.be/timelines/v/l8pRSuU81PU/s/5030/e/6719 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Deep learning training can achieve higher performance by using lower precision formats.（01:23:53 - 01:25:55） https://favtu.be/timelines/v/l8pRSuU81PU/s/5033/e/5155 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Utilizing Multiple GPUs with Distributed Data Parallelism: This part introduces the concept of distributed data parallelism (DDP) to utilize multiple GPUs for training. It explains how to launch multiple processes with torchrun, assign processes to specific GPUs, and synchronize gradients across processes. (-（01:24:00 - 01:28:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/5040/e/5280 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Importance of using floating points over int8 for neural network training.（01:25:55 - 01:29:49） https://favtu.be/timelines/v/l8pRSuU81PU/s/5155/e/5389 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) model. The data loading script and its functionalities for downloading, tokenizing, and sharding the dataset are briefly explained. (-（01:28:00 - 01:32:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/5280/e/5520 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🔄 Leveraging Tensor Cores for Acceleration- Explanation of tensor cores and their role in matrix multiplication.- Introduction to tf32 precision and its performance benefits.- Comparison of tf32 and float32 performance improvements.（01:28:12 - 01:37:04） https://favtu.be/timelines/v/l8pRSuU81PU/s/5292/e/5824 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Tensor Cores, timing the code, TF32 precision, 333ms（01:28:14 - 01:39:38） https://favtu.be/timelines/v/l8pRSuU81PU/s/5294/e/5978 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Mixed Precision (TF32):* Enabling TF32 precision for matrix multiplications provides a free 3x speedup with minimal accuracy loss.（01:28:14 - 01:39:38） https://favtu.be/timelines/v/l8pRSuU81PU/s/5294/e/5978 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Matrix multiplication is accelerated through tensor cores.（01:29:49 - 01:32:01） https://favtu.be/timelines/v/l8pRSuU81PU/s/5389/e/5521 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Adjusting Training Script for Fine Web EDU: The training script is modified to accommodate the Fine Web EDU dataset, including changes to the data loader, training loop, and hyperparameter settings. The concept of warming up the learning rate and its importance in training large language models is discussed. (-（01:32:00 - 01:36:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/5520/e/5760 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Using tf32 for 8X faster performance with minor precision tradeoff.（01:32:01 - 01:35:49） https://favtu.be/timelines/v/l8pRSuU81PU/s/5521/e/5749 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Max out the batch size and use numbers with powers of two for better efficiency.（01:35:49 - 01:37:45） https://favtu.be/timelines/v/l8pRSuU81PU/s/5749/e/5865 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) model on HSWAG are outlined. (-（01:36:00 - 01:40:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/5760/e/6000 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) @ Should the tokens/second throughput be x2 given we use both X and y (targets) for training? Or are we just looking at the batch size here? Also would using x.numel() or y.numel() be equivalent?（01:36:27 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/5787/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - ⚙ Implementing tf32 Precision in PyTorch- Enabling tf32 precision in PyTorch with a single line of code.- Observing throughput improvements with tf32 precision.- Understanding the trade-offs and limitations of tf32 precision.（01:37:04 - 01:40:30） https://favtu.be/timelines/v/l8pRSuU81PU/s/5824/e/6030 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) TF32 promises 8X throughput but only delivers 3X due to memory bottlenecks（01:37:45 - 01:41:50） https://favtu.be/timelines/v/l8pRSuU81PU/s/5865/e/6110 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) float16, gradient scalers, bfloat16, 300ms（01:39:38 - 01:48:15） https://favtu.be/timelines/v/l8pRSuU81PU/s/5978/e/6495 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Mixed Precision (BFloat16):* Switching to BFloat16 for activations further improves speed, requiring minimal code changes thanks to PyTorch AutoCast.（01:39:38 - 01:48:15） https://favtu.be/timelines/v/l8pRSuU81PU/s/5978/e/6495 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) model. The importance of a validation set in monitoring overfitting is reiterated. (-（01:40:00 - 01:44:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/6000/e/6240 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 📊 B Float16 vs. FP16 Precision Reduction- Understanding B Float16 precision reduction compared to FP16.- B Float16 maintains the same exponent range but truncates the mantissa, resulting in reduced precision within the range.- Unlike FP16, B Float16 does not alter the range of representable numbers, simplifying training processes by eliminating the need for gradient scalers.（01:40:30 - 01:42:24） https://favtu.be/timelines/v/l8pRSuU81PU/s/6030/e/6144 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Transition from fp16 to bf16 for simpler training.（01:41:50 - 01:43:50） https://favtu.be/timelines/v/l8pRSuU81PU/s/6110/e/6230 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🧮 Implementing Mixed Precision in PyTorch- Utilizing PyTorch's torch.AutoCast for mixed precision training.- Guidance on using torch.AutoCast to surround the forward pass and loss calculation in the model.- Highlighting the minimal code changes required to implement B Float16 training in PyTorch.（01:42:24 - 01:48:29） https://favtu.be/timelines/v/l8pRSuU81PU/s/6144/e/6509 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Implementing B float 16 for minimal impact on model activations.（01:43:50 - 01:48:05） https://favtu.be/timelines/v/l8pRSuU81PU/s/6230/e/6485 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) training for further performance optimization. (-（01:44:00 - 01:48:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/6240/e/6480 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) )（01:48:00 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/6480/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Introducing torch.compile for faster model compilation（01:48:05 - 01:49:59） https://favtu.be/timelines/v/l8pRSuU81PU/s/6485/e/6599 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) torch.compile, Python overhead, kernel fusion, 130ms（01:48:15 - 02:00:18） https://favtu.be/timelines/v/l8pRSuU81PU/s/6495/e/7218 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Torch.Compile:* Compiling the model with torch.compile significantly reduces Python overhead and optimizes kernel fusion, resulting in a 2.3x speedup.（01:48:15 - 02:00:18） https://favtu.be/timelines/v/l8pRSuU81PU/s/6495/e/7218 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - ⚡ Torch.Compile for Model Optimization- Introduction to torch.Compile as a compiler for neural networks in PyTorch.- Explaining the reduction of Python overhead and GPU read-writes for faster computation.- Demonstrating significant speed improvements with torch.Compile, achieving about 2.3x faster performance with a single line of code.（01:48:29 - 02:00:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/6509/e/7226 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Torch compile optimizes neural net operations efficiently（01:49:59 - 01:53:48） https://favtu.be/timelines/v/l8pRSuU81PU/s/6599/e/6828 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) "dispatch the kernel"???（01:51:59 - 02:01:31） https://favtu.be/timelines/v/l8pRSuU81PU/s/6719/e/7291 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Optimizing round trips to GPU memory for faster computation（01:53:48 - 01:55:38） https://favtu.be/timelines/v/l8pRSuU81PU/s/6828/e/6938 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) GPU chip architecture overview（01:55:38 - 01:59:29） https://favtu.be/timelines/v/l8pRSuU81PU/s/6938/e/7169 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) hours of video on this topic"Me: Please sign me up :)（01:55:55 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/6955/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) yes Andrej we need that 2 hour neural net Hardware specific video 🗣🗣🗣（01:55:59 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/6959/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Torch compilation utilizes kernel Fusion for speed optimization（01:59:29 - 02:01:24） https://favtu.be/timelines/v/l8pRSuU81PU/s/7169/e/7284 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) flash attention, 96ms（02:00:18 - 02:06:54） https://favtu.be/timelines/v/l8pRSuU81PU/s/7218/e/7614 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Flash Attention:* Replacing the default attention implementation with Flash Attention, a specialized kernel fusion algorithm, yields another 27% speedup.（02:00:18 - 02:06:54） https://favtu.be/timelines/v/l8pRSuU81PU/s/7218/e/7614 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🧠 Flash Attention Optimization- Flash attention is a kernel fusion algorithm that significantly speeds up attention mechanisms.- Achieves faster computation by avoiding materializing large attention matrices.- Utilizes an online softmax trick to incrementally evaluate softmax without storing all inputs.（02:00:26 - 02:06:54） https://favtu.be/timelines/v/l8pRSuU81PU/s/7226/e/7614 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Flash attention algorithm reduces memory usage and improves computation speed significantly.（02:01:24 - 02:05:15） https://favtu.be/timelines/v/l8pRSuU81PU/s/7284/e/7515 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) FlashAttention -> more flops does not mean slower（02:01:31 - 02:10:09） https://favtu.be/timelines/v/l8pRSuU81PU/s/7291/e/7809 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Using Flash attention in PyTorch for faster runtime.（02:05:15 - 02:07:23） https://favtu.be/timelines/v/l8pRSuU81PU/s/7515/e/7643 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) nice/ugly numbers. vocab size 50257 → 50304, 93ms（02:06:54 - 02:14:55） https://favtu.be/timelines/v/l8pRSuU81PU/s/7614/e/8095 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Nice vs. Ugly Numbers:* Optimizing vocabulary size to a power of two (50304) for better kernel utilization surprisingly provides a 4% speedup.（02:06:54 - 02:14:55） https://favtu.be/timelines/v/l8pRSuU81PU/s/7614/e/8095 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🧮 Optimization with Nice Numbers- Identifies "nice" numbers (powers of two) as optimal for computations in CUDA.- Adjusts vocabulary size to a nice number to improve computation efficiency.- Padding inputs to align with block sizes in CUDA can lead to significant performance gains.（02:06:54 - 02:15:18） https://favtu.be/timelines/v/l8pRSuU81PU/s/7614/e/8118 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Prefer using powers of two in code for neural networks and CUDA.（02:07:23 - 02:11:14） https://favtu.be/timelines/v/l8pRSuU81PU/s/7643/e/7874 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Add more tokens the model actually trains faster（02:10:09 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/7809/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Improved GPT-2 performance by fixing token index issue（02:11:14 - 02:13:04） https://favtu.be/timelines/v/l8pRSuU81PU/s/7874/e/7984 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Padding inputs for efficiency improvement（02:13:04 - 02:16:50） https://favtu.be/timelines/v/l8pRSuU81PU/s/7984/e/8210 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) SECTION 3: hyperpamaters, AdamW, gradient clipping（02:14:55 - 02:21:06） https://favtu.be/timelines/v/l8pRSuU81PU/s/8095/e/8466 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Hyperparameters and AdamW:* The video adopts hyperparameters from the GPT-3 paper, including AdamW optimizer settings and gradient clipping.（02:14:55 - 02:21:06） https://favtu.be/timelines/v/l8pRSuU81PU/s/8095/e/8466 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🔍 Hyperparameter Tuning and Algorithmic Improvements- Discusses the importance of hyperparameter tuning based on the GPT-3 paper.- Implements gradient norm clipping to prevent model instability during optimization.- Monitoring the gradient norm helps detect training instabilities and adjust optimization strategies.（02:15:18 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/8118/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Setting hyperparameters for training GPT-3（02:16:50 - 02:18:48） https://favtu.be/timelines/v/l8pRSuU81PU/s/8210/e/8328 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Monitoring gradient norm is crucial for stability（02:18:48 - 02:22:39） https://favtu.be/timelines/v/l8pRSuU81PU/s/8328/e/8559 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🎓 Implementing Learning Rate Scheduler and Weight Decay- Understanding the details of the learning rate scheduler and weight decay implementation:- Learning rate scheduler: Cosine decay with warm-up period and decay to 10% over a specified horizon.- Weight decay: Used for regularization, typically applied to embedding and weight matrices.（02:19:44 - 02:26:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/8384/e/8786 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) learning rate scheduler: warmup + cosine decay（02:21:06 - 02:26:21） https://favtu.be/timelines/v/l8pRSuU81PU/s/8466/e/8781 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Learning Rate Scheduler:* A cosine decay learning rate schedule with warmup is implemented, following the GPT-3 paper.（02:21:06 - 02:26:21） https://favtu.be/timelines/v/l8pRSuU81PU/s/8466/e/8781 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Setting learning rate in GPT-2 (124M)（02:22:39 - 02:24:32） https://favtu.be/timelines/v/l8pRSuU81PU/s/8559/e/8672 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Implementing a learning rate schedule for training GPT-2（02:24:32 - 02:28:14） https://favtu.be/timelines/v/l8pRSuU81PU/s/8672/e/8894 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) batch size schedule, weight decay, FusedAdamW, 90ms（02:26:21 - 02:34:09） https://favtu.be/timelines/v/l8pRSuU81PU/s/8781/e/9249 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Batch Size, Weight Decay, FusedAdamW:* The video discusses batch size scheduling (which is ultimately skipped), implements weight decay for regularization, and utilizes the fused implementation of AdamW for further speed improvements.（02:26:21 - 02:34:09） https://favtu.be/timelines/v/l8pRSuU81PU/s/8781/e/9249 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 📊 Batch Size Increase and Data Sampling Techniques- Explanation on gradual batch size increase and data sampling methods:- Gradual batch size increase: Linear ramp-up from small to large batch sizes, aiming for system speed improvement.- Data sampling without replacement: Exhausting a pool of data without reusing sequences until an epoch boundary is reached.（02:26:26 - 02:29:01） https://favtu.be/timelines/v/l8pRSuU81PU/s/8786/e/8941 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Data are sampled without replacement during training.（02:28:14 - 02:30:10） https://favtu.be/timelines/v/l8pRSuU81PU/s/8894/e/9010 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🧮 Weight Decay Implementation and Optimizer Configuration- Details on weight decay implementation and optimizer configuration:- Weight decay: Applied for regularization, particularly to embeddings and weight matrices.- Optimizer configuration: Adjusting parameters for optimal training performance, including weight decay settings.（02:29:01 - 02:37:07） https://favtu.be/timelines/v/l8pRSuU81PU/s/8941/e/9427 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Weight decay parameters are split into those that should be weight decayed and those that should not be weight decayed.（02:30:10 - 02:33:55） https://favtu.be/timelines/v/l8pRSuU81PU/s/9010/e/9235 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Weight decay is applied to two-dimensional parameters.（02:33:55 - 02:35:53） https://favtu.be/timelines/v/l8pRSuU81PU/s/9235/e/9353 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) gradient accumulation（02:34:09 - 02:46:52） https://favtu.be/timelines/v/l8pRSuU81PU/s/9249/e/10012 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Gradient Accumulation:* Gradient accumulation is implemented to simulate larger batch sizes (0.5 million tokens) on limited GPU memory.（02:34:09 - 02:46:52） https://favtu.be/timelines/v/l8pRSuU81PU/s/9249/e/10012 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Using gradient accumulation to simulate a large batch size（02:35:53 - 02:39:54） https://favtu.be/timelines/v/l8pRSuU81PU/s/9353/e/9594 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🔄 Gradient Accumulation for Simulating Large Batch Sizes- Implementation of gradient accumulation technique to simulate large batch sizes:- Total batch size setting: Defines the desired batch size, which may exceed GPU capacity.- Micro batch size and gradient accumulation: Processing multiple micro-batches and accumulating gradients before updating the model.（02:37:07 - 02:39:12） https://favtu.be/timelines/v/l8pRSuU81PU/s/9427/e/9552 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🧠 Understanding Gradient Accumulation- Explains the concept of gradient accumulation.- Demonstrates the difference between traditional batch processing and gradient accumulation.- Emphasizes the importance of normalizing gradients to ensure consistency.（02:39:12 - 02:47:00） https://favtu.be/timelines/v/l8pRSuU81PU/s/9552/e/10020 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Demonstration of simple neural network implementation with mean squared loss（02:39:54 - 02:41:54） https://favtu.be/timelines/v/l8pRSuU81PU/s/9594/e/9714 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Gradients do not match due to loss normalization issue（02:41:54 - 02:45:49） https://favtu.be/timelines/v/l8pRSuU81PU/s/9714/e/9949 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Optimizing model training with gradient accumulation and distributed data parallelism.（02:45:49 - 02:47:41） https://favtu.be/timelines/v/l8pRSuU81PU/s/9949/e/10061 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) distributed data parallel (DDP)（02:46:52 - 03:10:21） https://favtu.be/timelines/v/l8pRSuU81PU/s/10012/e/11421 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Distributed Data Parallel (DDP):* The training is parallelized across 8 GPUs using PyTorch DDP, achieving a throughput of 1.5 million tokens per second.（02:46:52 - 03:10:21） https://favtu.be/timelines/v/l8pRSuU81PU/s/10012/e/11421 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🔧 Implementing Distributed Data Parallelism- Introduces the concept of distributed data parallelism for utilizing multiple GPUs.- Explains the difference between legacy data parallelism and distributed data parallelism.- Describes how distributed data parallelism works and its benefits in training neural networks.（02:47:00 - 02:57:01） https://favtu.be/timelines/v/l8pRSuU81PU/s/10020/e/10621 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Collaborative processing with multiple GPUs（02:47:41 - 02:51:32） https://favtu.be/timelines/v/l8pRSuU81PU/s/10061/e/10292 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Running with TorRun involves eight parallel processes with different ranks.（02:51:32 - 02:53:39） https://favtu.be/timelines/v/l8pRSuU81PU/s/10292/e/10419 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Introduction to GPU calculations in GPT-2 (124M)（02:53:39 - 02:57:43） https://favtu.be/timelines/v/l8pRSuU81PU/s/10419/e/10663 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🔄 Adapting Data Loading for Multi-Process Training- Adjusts data loading process to accommodate multiple processes.- Demonstrates how to assign different chunks of data to each process.- Ensures that each process works on a unique part of the dataset to maximize efficiency.（02:57:01 - 02:59:17） https://favtu.be/timelines/v/l8pRSuU81PU/s/10621/e/10757 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Initialization of GPT-2 model training process（02:57:43 - 02:59:44） https://favtu.be/timelines/v/l8pRSuU81PU/s/10663/e/10784 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🧩 Model Construction and Distributed Data Parallel (DDP)- Explanation of constructing a model for distributed training.- Wrapping the model into a DistributedDataParallel (DDP) container.- Understanding the behavior of DDP in forward and backward passes.（02:59:17 - 03:02:15） https://favtu.be/timelines/v/l8pRSuU81PU/s/10757/e/10935 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Wrapping the model into the Distributed Data Parallel container is important for constructing the M model.（02:59:44 - 03:03:37） https://favtu.be/timelines/v/l8pRSuU81PU/s/10784/e/11017 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🔄 Synchronization of Gradients in DDP- Discusses the synchronization of gradients in the DistributedDataParallel (DDP) setting.- Explanation of optimizing gradient synchronization to improve efficiency.- Implementation details for synchronizing gradients in DDP.（03:02:15 - 03:05:22） https://favtu.be/timelines/v/l8pRSuU81PU/s/10935/e/11122 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Avoiding context managers and code duplication by directly toggling the variable.（03:03:37 - 03:05:42） https://favtu.be/timelines/v/l8pRSuU81PU/s/11017/e/11142 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 📉 Loss Averaging and Evaluation in DDP- Addressing the issue of loss averaging in the DDP setting.- Modifying code to compute and print the average loss across all processes.- Ensuring proper scaling of the number of tokens processed in the evaluation phase.（03:05:22 - 03:10:23） https://favtu.be/timelines/v/l8pRSuU81PU/s/11122/e/11423 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Printing loss over all processes and averaging it（03:05:42 - 03:09:39） https://favtu.be/timelines/v/l8pRSuU81PU/s/11142/e/11379 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) GPT-2 (124M) reproduction process summary（03:09:39 - 03:11:30） https://favtu.be/timelines/v/l8pRSuU81PU/s/11379/e/11490 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) datasets used in GPT-2, GPT-3, FineWeb (EDU)（03:10:21 - 03:23:10） https://favtu.be/timelines/v/l8pRSuU81PU/s/11421/e/12190 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Dataset Selection:* The video discusses various datasets used for training large language models, ultimately choosing the FineWeb EDU dataset (10 billion token sample).（03:10:21 - 03:23:10） https://favtu.be/timelines/v/l8pRSuU81PU/s/11421/e/12190 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 📚 Training Data Comparison: GPT-2 vs. GPT-3- Comparison of training datasets used in GPT-2 and GPT-3.- Description of web text and Common Crawl datasets utilized.- Introduction of alternative datasets like Red Pajamas, C4, Fine Web, and Fine Web Edu.（03:10:23 - 03:14:15） https://favtu.be/timelines/v/l8pRSuU81PU/s/11423/e/11655 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Training data mixtures are carefully curated and diverse.（03:11:30 - 03:15:22） https://favtu.be/timelines/v/l8pRSuU81PU/s/11490/e/11722 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 📦 Preprocessing and Training Setup for Fine Web Edu- Overview of the preprocessing steps for the Fine Web Edu dataset.- Description of tokenization process and data shard creation.- Configuration adjustments in the data loader for using the Fine Web Edu dataset.（03:14:15 - 03:18:42） https://favtu.be/timelines/v/l8pRSuU81PU/s/11655/e/11922 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Tokenizing and processing large datasets for GPT-2 model training.（03:15:22 - 03:17:22） https://favtu.be/timelines/v/l8pRSuU81PU/s/11722/e/11842 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Sharding data for easier disk management（03:17:22 - 03:21:39） https://favtu.be/timelines/v/l8pRSuU81PU/s/11842/e/12099 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🧩 Script adjustments for GPT-3 replication- Adjusted data loader for processing multiple shards.- Set token processing rate and warm-up steps to match GPT-3 parameters.- Increased batch size optimization for faster training.（03:18:42 - 03:21:29） https://favtu.be/timelines/v/l8pRSuU81PU/s/11922/e/12089 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 📊 Implementing validation evaluation- Added validation evaluation logic to the training loop.- Introduced periodic validation loss calculation.- Prepared for model comparison with GPT-2 124M.（03:21:29 - 03:26:09） https://favtu.be/timelines/v/l8pRSuU81PU/s/12089/e/12369 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Optimizing model training process for efficiency and quality.（03:21:39 - 03:23:39） https://favtu.be/timelines/v/l8pRSuU81PU/s/12099/e/12219 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) validation data split, validation loss, sampling revive（03:23:10 - 03:28:23） https://favtu.be/timelines/v/l8pRSuU81PU/s/12190/e/12503 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Validation Split:* A validation split is introduced to monitor overfitting and compare performance to the pre-trained GPT-2 model.（03:23:10 - 03:28:23） https://favtu.be/timelines/v/l8pRSuU81PU/s/12190/e/12503 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Evaluating GPT-2 (124M) model performance（03:23:39 - 03:27:18） https://favtu.be/timelines/v/l8pRSuU81PU/s/12219/e/12438 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🔄 Reorganizing sampling code- Moved sampling code closer to the main training loop.- Implemented a separate RNG for sampling to avoid impacting training RNG.- Addressed performance slowdown due to disabled Torch compile.（03:26:09 - 03:28:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/12369/e/12506 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Troubleshooting torch compile issue（03:27:18 - 03:29:01） https://favtu.be/timelines/v/l8pRSuU81PU/s/12438/e/12541 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) evaluation: HellaSwag, starting the run（03:28:23 - 03:43:05） https://favtu.be/timelines/v/l8pRSuU81PU/s/12503/e/13385 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** HellaSwag Evaluation:* The HellaSwag benchmark is implemented to evaluate the model's common sense reasoning abilities.（03:28:23 - 03:43:05） https://favtu.be/timelines/v/l8pRSuU81PU/s/12503/e/13385 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 📈 Introducing H-SWAG evaluation- Described H-SWAG evaluation methodology and dataset.- Highlighted its role as a smooth evaluation metric.- Discussed implementation details for incorporating H-SWAG into the training script.（03:28:26 - 03:38:22） https://favtu.be/timelines/v/l8pRSuU81PU/s/12506/e/13102 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Language models trained with world knowledge outperform those with less training.（03:29:01 - 03:33:01） https://favtu.be/timelines/v/l8pRSuU81PU/s/12541/e/12781 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Construct batches of tokens with shared context and options for prediction（03:33:01 - 03:35:02） https://favtu.be/timelines/v/l8pRSuU81PU/s/12781/e/12902 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Model's inability to view all options at once（03:35:02 - 03:38:55） https://favtu.be/timelines/v/l8pRSuU81PU/s/12902/e/13135 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🔧 Adjustments to Training Script and Logging- Changes made to the training script to enable periodic evaluation and tracking of model performance over time.- Disabling torch compile due to issues with evaluation and sampling code.- Creation of a log directory to record training and validation losses, as well as H swag accuracies.（03:38:22 - 03:40:01） https://favtu.be/timelines/v/l8pRSuU81PU/s/13102/e/13201 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Running without torch compile affects code performance（03:38:55 - 03:40:48） https://favtu.be/timelines/v/l8pRSuU81PU/s/13135/e/13248 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 📊 Evaluation of H Swag and Model Sampling- Introduction of code for evaluating H swag periodically during training.- Utilization of GPU collaboration for H swag evaluation.- Sampling from the model every 250th iteration for monitoring model progress.（03:40:01 - 03:43:06） https://favtu.be/timelines/v/l8pRSuU81PU/s/13201/e/13386 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Model training process overview（03:40:48 - 03:45:03） https://favtu.be/timelines/v/l8pRSuU81PU/s/13248/e/13503 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) SECTION 4: results in the morning! GPT-2, GPT-3 repro（03:43:05 - 03:56:21） https://favtu.be/timelines/v/l8pRSuU81PU/s/13385/e/14181 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Results:* After training for one epoch (10 billion tokens), the model surpasses the GPT-2 (124M) performance on HellaSwag, achieving comparable results with 10x fewer training tokens.（03:43:05 - 03:48:41） https://favtu.be/timelines/v/l8pRSuU81PU/s/13385/e/13721 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 📈 Training Progress Visualization- Visualization of training progress using Matplotlib.- Analysis of loss curves and model performance.- Comparison of model performance against GPT-2 and GPT-3 accuracy metrics.（03:43:06 - 03:46:23） https://favtu.be/timelines/v/l8pRSuU81PU/s/13386/e/13583 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) GPT-2 (124M) trained on 10 billion tokens matching or surpassing accuracy of GPT-2 (100B) trained on significantly fewer tokens（03:45:03 - 03:46:58） https://favtu.be/timelines/v/l8pRSuU81PU/s/13503/e/13618 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🧠 Reflections on Training Results and Data Quality- Discussion on the implications of achieving GPT-3 level accuracy with fewer tokens.- Consideration of factors influencing model performance, such as data distribution and dataset quality.- Reflections on potential improvements in data preprocessing and model hyperparameters.（03:46:23 - 03:49:05） https://favtu.be/timelines/v/l8pRSuU81PU/s/13583/e/13745 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Issue with data shuffling affecting model training（03:46:58 - 03:50:41） https://favtu.be/timelines/v/l8pRSuU81PU/s/13618/e/13841 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Overnight Run:* Training for four epochs (40 billion tokens) further improves HellaSwag accuracy, approaching the GPT-3 (124M) performance.（03:48:41 - 03:56:21） https://favtu.be/timelines/v/l8pRSuU81PU/s/13721/e/14181 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - ⚙ Optimization Techniques and Training Efficiency- Examination of optimization issues and periodicity in data loading.- Discussion on the impact of learning rate adjustments on training efficiency.- Consideration of techniques to improve data shuffling and reduce data dependency.（03:49:05 - 03:52:03） https://favtu.be/timelines/v/l8pRSuU81PU/s/13745/e/13923 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Improving data shuffling and model efficiency.（03:50:41 - 03:52:31） https://favtu.be/timelines/v/l8pRSuU81PU/s/13841/e/13951 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) - 🛠 Model Fine-Tuning and Future Directions- Overview of fine-tuning process for conversational AI applications.- Introduction of model checkpointing for resuming optimization and model evaluation.- Discussion on alternative evaluation methods and comparison with GPT-2 and GPT-3.（03:52:03 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/13923/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Training model to mimic GPT-3 with sequence length adjustment（03:52:31 - 03:56:15） https://favtu.be/timelines/v/l8pRSuU81PU/s/13951/e/14175 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Comparison between nanog GPT in PyTorch and llm Doc and lm. C CUDA implementation（03:56:15 - 03:58:19） https://favtu.be/timelines/v/l8pRSuU81PU/s/14175/e/14299 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) shoutout to llm.c, equivalent but faster code in raw C/CUDA（03:56:21 - 03:59:39） https://favtu.be/timelines/v/l8pRSuU81PU/s/14181/e/14379 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Shoutout to llm.c:* The video showcases "llm.c," a faster C/CUDA implementation of GPT-2/3 training.（03:56:21 - 03:59:39） https://favtu.be/timelines/v/l8pRSuU81PU/s/14181/e/14379 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) Comparing PyTorch and lm.C performance for training GPT-2 and GPT-3.（03:58:19 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/14299/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) summary, phew, build-nanogpt github repo（03:59:39 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/14379/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) * *** Summary:* A brief summary of the achievements and remaining challenges.（03:59:39 - 04:01:26） https://favtu.be/timelines/v/l8pRSuU81PU/s/14379/e/14486 Mon, 10 Jun 24 08:31:35 +0900 Let's reproduce GPT-2 (124M) intro: Tokenization, GPT-2 paper, tokenization-related issues（00:00:00 - 00:05:50） https://favtu.be/timelines/v/zduSFxRajkE/s/0/e/350 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🤖 什么是分词(tokenization)?*- 分词是将文本转换为标记序列的过程。- 在大型语言模型中，分词是将文本转换为标记序列以供模型处理的关键步骤。- 分词的质量和方法直接影响着模型的性能和行为。（00:00:00 - 00:02:43） https://favtu.be/timelines/v/zduSFxRajkE/s/0/e/163 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🧩 Tokenization process overview- Tokenization is crucial for working with large language models- Tokenization converts text into tokens for language model processing（00:00:00 - 00:03:13） https://favtu.be/timelines/v/zduSFxRajkE/s/0/e/193 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer How does it know how DefaultCellStyle is spelled? Is there something in the training data that helps create a mapping from that token to the version with spaces? Did OpenAI maybe augment the training data with 'spelling tables'?（00:01:53 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/113/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🔍 GPT2 使用的字节对编码算法*- 字节对编码算法是一种常用的分词方法，用于构建大型语言模型的标记词汇表。- GPT2 中的分词器使用了字节对编码算法来构建词汇表，其中每个 token 可以是多个字符的组合。- 字节对编码算法能够灵活地处理各种语言和特殊字符，从而提高模型的适用性和性能。（00:02:43 - 00:05:59） https://favtu.be/timelines/v/zduSFxRajkE/s/163/e/359 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🍬 Bite-pair encoding for tokenization- Bite-pair encoding is used in state-of-the-art language models- Tokenization generates vocabularies for language model input- Tokens are fundamental units in large language models（00:03:13 - 00:09:17） https://favtu.be/timelines/v/zduSFxRajkE/s/193/e/557 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer tokenization by example in a Web UI (tiktokenizer)（00:05:50 - 00:14:56） https://favtu.be/timelines/v/zduSFxRajkE/s/350/e/896 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🌐 语言模型中的分词问题*- 分词对于语言模型的性能和行为至关重要，但也会带来一些问题和挑战。- 不同语言的分词效果可能不同，特别是非英文语言可能受到数据不平衡的影响。- 分词方法的设计和实现对模型的效率和表现有重要影响，需要综合考虑多方面因素进行优化。（00:05:59 - 00:19:25） https://favtu.be/timelines/v/zduSFxRajkE/s/359/e/1165 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer Hey Andrej, thanks for the new video! I'm not yet done but I noticed at you mentioned "notice that the colour is different, so this is not the same token". But actually in that app, the colours are random, and are just cycling through so as not to have twice the same colours in a row. See e.g. the " +" token with different colours, or all the differently coloured spaces in the python code.（00:08:46 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/526/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer For these problems mentioned at around (the word "egg" got tokenized in different ways): would it help if we just lower-cased all the text and used an actual dictionary as token vocabulary?（00:08:55 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/535/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🌏 Multilingual tokenization challenges- Non-English languages may have different tokenization challenges- Tokenizers have to handle varying lengths for different languages（00:09:17 - 00:14:47） https://favtu.be/timelines/v/zduSFxRajkE/s/557/e/887 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer @ OFFF Course this legend also speaks Korean! Why wouldn't he?Awesome video Andrej! ❤（00:09:38 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/578/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer omg perfect Korean（00:09:38 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/578/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer Wow his korean speaking is so accurate and accent is incredible. I'm Korean and This brilliant top-notch human(Level of ASI, haha) can do better at anything than me and now even my mother language than me now haha ;)（00:09:39 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/579/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🐍 Tokenization impact on Python coding- Tokenization affects the handling of code in language models- Tokenizer design influences the model's performance for specific languages（00:14:47 - 00:18:13） https://favtu.be/timelines/v/zduSFxRajkE/s/887/e/1093 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer strings in Python, Unicode code points（00:14:56 - 00:18:15） https://favtu.be/timelines/v/zduSFxRajkE/s/896/e/1095 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer "Unicode." I despise Unicode with the passion of a million searing fires. I've written enough code to handle Unicode to feel your pain through the screen without you saying a single word about it. ASCII was v1.0 of character handling. Extended ASCII with "Code Pages" was v1.3. Unicode is barely v2.0 and we still haven't gotten it right. So maybe by v3.0, whatever it ends up being called, we'll _finally_ figure out that human language is too complex to represent in computer systems using a set number of bytes for the representation of a character sequence and finally offer something much more flexible and comprehensive that's also compatible/performant with how computer systems work.（00:16:00 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/960/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🔠 Unicode encodings for text processing- Unicode encodings like UTF-8 are essential for processing text- Different encodings have varying efficiencies and use cases- UTF-8 encoding is preferred for its compatibility and efficiency（00:18:13 - 00:22:26） https://favtu.be/timelines/v/zduSFxRajkE/s/1093/e/1346 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32（00:18:15 - 00:22:47） https://favtu.be/timelines/v/zduSFxRajkE/s/1095/e/1367 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🧮 字符编码的选择与比较*- UTF-8 在互联网上被广泛采用，因为它是唯一向后兼容 ASCII 编码的字符编码。- UTF-8 相对于其他编码来说更加节省空间，因为它能够更有效地编码文本信息。（00:19:25 - 00:22:01） https://favtu.be/timelines/v/zduSFxRajkE/s/1165/e/1321 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🧩 字节对编码算法简介*- 字节对编码算法通过迭代地识别并替换最常出现的字节对来压缩文本序列。- 该算法能够将原始字节序列压缩到一个较小的固定大小的词汇表中，并实现对任意序列的编码和解码。（00:22:01 - 00:27:10） https://favtu.be/timelines/v/zduSFxRajkE/s/1321/e/1630 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🧠 Byte Pair Encoding Algorithm Overview- Byte Pair Encoding (BPE) algorithm compresses sequences by finding and merging the most frequent pairs of tokens iteratively.（00:22:26 - 00:27:10） https://favtu.be/timelines/v/zduSFxRajkE/s/1346/e/1630 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer daydreaming: deleting tokenization（00:22:47 - 00:23:50） https://favtu.be/timelines/v/zduSFxRajkE/s/1367/e/1430 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer I’m at , and I’m wishing the tokenization was getting at the etymological roots of words and/or meaning of marks in pictographic languages.（00:23:30 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/1410/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer Byte Pair Encoding (BPE) algorithm walkthrough（00:23:50 - 00:27:02） https://favtu.be/timelines/v/zduSFxRajkE/s/1430/e/1622 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer starting the implementation（00:27:02 - 00:28:35） https://favtu.be/timelines/v/zduSFxRajkE/s/1622/e/1715 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🖥️ 字节对编码算法的实现*- 通过 Python 实现字节对编码算法，包括识别最常见字节对、替换、创建新词汇表等步骤。- 使用迭代的方式对文本序列进行多次合并，直到达到所需的词汇表大小。（00:27:10 - 00:38:01） https://favtu.be/timelines/v/zduSFxRajkE/s/1630/e/2281 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 📊 Implementing Byte Pair Encoding Algorithm in Python- Encoding text into UTF-8 tokens and converting them to integers for manipulation.- Identifying the most common pair of tokens and replacing them with new tokens using Python functions.（00:27:10 - 00:35:00） https://favtu.be/timelines/v/zduSFxRajkE/s/1630/e/2100 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer Hey Andrej, great video! However, at , you don't need to convert all the bytes to integers by using map(). When you call list() on tokens, the bytes are by default converted into integers, so just doing 'list(tokens)' is fine instead of 'list(map(int, tokens))'.（00:27:23 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/1643/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer At you don't need map(int, ...) because bytes are already enumerable, so just use tokens = list(tokens)（00:27:24 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/1644/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer counting consecutive pairs, finding most common pair（00:28:35 - 00:30:36） https://favtu.be/timelines/v/zduSFxRajkE/s/1715/e/1836 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer merging the most common pair（00:30:36 - 00:34:58） https://favtu.be/timelines/v/zduSFxRajkE/s/1836/e/2098 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer I'm jumping in with a comment before finishing the video, but one thing I noticed about this the byte-pair encoding implementation, is that it is agnostic to the UTF-8 character boundaries. So it should be possible that a token only represents the bytes of half of a multi-byte character. In that case, when trying to visualise which characters are part of which token, like in the toktokenizer tool you showed at the start, it couldn't really be visualised properly since one character could be split across two tokens. I wonder if this is the case in GPT's encoding or whether there's a case to make sure characters are always grouped into the same token. I'll keep watching... :D（00:31:03 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/1863/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer GPT4 uses 100000 tokens which is not far from the 150000 that UNICODE defines.（00:34:47 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/2087/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer training the tokenizer: adding the while loop, compression ratio（00:34:58 - 00:39:20） https://favtu.be/timelines/v/zduSFxRajkE/s/2098/e/2360 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🧭 Training and Usage of the Tokenizer- Setting the vocabulary size and performing a fixed number of merges to create the tokenizer.- Discussing the role of the tokenizer as a separate preprocessing stage from the language model.（00:35:00 - 00:41:30） https://favtu.be/timelines/v/zduSFxRajkE/s/2100/e/2490 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer I'm a total noob, but would there be any benefit instead of taking the whole blog post (around ) and making a .txt file and having the program read it like that as opposed to pasting it as one long line? Just curious if there is pros/cons either way or if it truly doesn't matter（00:35:31 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/2131/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer At , in merge, why are we incrementing by 2?Suppose my top pair is (6, 6). In encoded text is [7, 6, 6, 5, 4, 3], code will not be able to replace the (6, 6) with minted token. Am I missing anything?（00:35:43 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/2143/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer Shouldn't it be **num_merges = vocab_size - len(set(tokens))** where **len(set(tokens))** is actually 158 instead of 256?（00:36:00 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/2160/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer where would you learn how to code like @?（00:37:37 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/2257/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *📊 Tokenizer 训练总结*- Tokenizer 的训练是完全独立于大语言模型的。- Tokenizer 有自己的训练集，使用 BPE 算法进行训练，构建词汇表。- Tokenizer 的训练一次性完成，之后可用于编码和解码。（00:38:01 - 00:39:38） https://favtu.be/timelines/v/zduSFxRajkE/s/2281/e/2378 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer tokenizer/LLM diagram: it is a completely separate stage（00:39:20 - 00:42:47） https://favtu.be/timelines/v/zduSFxRajkE/s/2360/e/2567 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🔤 Tokenizer 编码和解码*- Tokenizer 是原始文本和 token 序列之间的翻译层。- 可以将原始文本编码成 token 序列，也可以将 token 序列解码成原始文本。- 大语言模型的训练数据通常会预处理为 token 序列进行训练，而不是使用原始文本。（00:39:38 - 00:42:41） https://favtu.be/timelines/v/zduSFxRajkE/s/2378/e/2561 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🌐 Tokenizer Training Considerations- Highlighting the importance of diverse training sets for tokenizers encompassing various languages and data types.- Explaining the impact of different data representations on the token sequence density and model performance.（00:41:30 - 00:44:15） https://favtu.be/timelines/v/zduSFxRajkE/s/2490/e/2655 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🛠️ 实现编码和解码功能*- 实现编码功能时，需要将文本编码为 token 序列，并按照 merges 字典中的顺序执行合并。- 实现解码功能时，需要将 token 序列解码为原始文本，并根据 merges 字典执行解码过程。- 在实现解码功能时，需要注意处理不符合 UTF-8 格式的情况，常见的做法是使用错误处理参数来避免错误。（00:42:41 - 00:57:24） https://favtu.be/timelines/v/zduSFxRajkE/s/2561/e/3444 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer decoding tokens to strings（00:42:47 - 00:48:21） https://favtu.be/timelines/v/zduSFxRajkE/s/2567/e/2901 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🧮 Tokenization of IDS to create tokens- Getting tokens by iterating over IDS and looking up bytes in vocab- Concatenating bytes to create tokens- Decoding bytes back to strings using UTF-8（00:44:15 - 00:48:33） https://favtu.be/timelines/v/zduSFxRajkE/s/2655/e/2913 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer why at would it matter the order you add the new vocab terms?if you add idx =257 for pair a,b before idx=256 for pair c,d the dictionary is permutation equivariant as a hash table?（00:44:20 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/2660/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer Ahh, partially addressed at . However this is fixing error when decoding an invalid UTF-8 sequence. Such errors could be minimised by only tokenizing full UTF-8 sequences, so in this example chr(128) wouldn't be its own token as that's only valid as a UTF-8 continuation byte, not as the first byte of a character.（00:45:52 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/2752/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer encoding strings to tokens（00:48:21 - 00:57:36） https://favtu.be/timelines/v/zduSFxRajkE/s/2901/e/3456 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer I have a question regarding the encoding process . Why not preprocess the keys of the merges dictionary into byte sequences (in the [0–255] range), and then just do a longest prefix match on the input?We may then benefit from trie-like data structure.（00:48:22 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/2902/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🧬 Implementing encoding of string into tokens- Encoding text into UTF-8 to get raw bytes- Performing merges according to lookup dictionary- Identifying pairs for merging and performing merges（00:48:33 - 00:55:16） https://favtu.be/timelines/v/zduSFxRajkE/s/2913/e/3316 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer I guess next step is to build a vocabulary similar to `decode` and use a trie to encode straight to final tokens?（00:54:20 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/3260/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer At , can we not just implement encode by iterating over merges dictionary(the order is maintained) and calling the merge() function on tokens ?This is what I meandef encode(text) :tokens = list(text.encode("utf-8"))for pair, idx in merges.items() : tokens = merge(tokens, pair, idx)return tokens（00:54:55 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/3295/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer I am hugely confused at . Why are we writing such a complicated encoder using a while loop and unintuitive stuff like pair = min(stats, key=lambda p: merges.get(p, float("inf")))Why can't I just dodef encode(self, text):tokens = text.encode("utf-8")tokens = list(map(int, tokens))for pair, index in self.merges.items():tokens = merge(tokens, pair, index)（00:55:10 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/3310/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 📝 Perfecting the encoding function and testing- Addressing the special case of single character or empty string- Testing encoding and decoding to ensure consistency- Validating the implemented function with training and validation data（00:55:16 - 01:06:31） https://favtu.be/timelines/v/zduSFxRajkE/s/3316/e/3991 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer I think this question is addressed at .（00:56:12 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/3372/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🧩 GPT2论文中的Tokenizer*- GPT2论文解释了其使用的Tokenizer，主要采用字节对编码算法（Byte Pair Encoding, BPE）。- 论文指出对常见词汇进行简单的BPE算法合并会导致语义混乱，因此提出了手动制定合并规则的方法。（00:57:24 - 00:59:29） https://favtu.be/timelines/v/zduSFxRajkE/s/3444/e/3569 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer regex patterns to force splits across categories（00:57:36 - 01:11:38） https://favtu.be/timelines/v/zduSFxRajkE/s/3456/e/4298 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🛠️ GPT2的Tokenizer实现细节*- GPT2的Tokenizer实现包括了一个复杂的正则表达式模式，用于规定哪些部分的文本不应该被合并。- 使用了Python的reex包进行更强大的正则表达式匹配。（00:59:29 - 01:11:08） https://favtu.be/timelines/v/zduSFxRajkE/s/3569/e/4268 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🧩 Tokenization rules and inconsistencies- Tokenization rules for apostrophes are inconsistent in uppercase and lowercase letters.- Matching punctuation characters is essential to separate them from letters or numbers.- Understanding whitespace handling in tokenization is crucial, including negative look-ahead assertions.（01:06:31 - 01:11:08） https://favtu.be/timelines/v/zduSFxRajkE/s/3991/e/4268 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer "extremely gnarly, and slightly gross" (), is how I feel about ML 99% of the time（01:07:20 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/4040/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🧰 TikTok Tokenizer 库介绍*- OpenAI发布了TikTok Tokenizer库，用于GPT4的分词工作。- 与GPT2不同，GPT4的Tokenizer将空格合并为一个标记，这在GPT2中是不同的。（01:11:08 - 01:13:12） https://favtu.be/timelines/v/zduSFxRajkE/s/4268/e/4392 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🤖 GPT Tokenizer and GPT-3.5 Turbo Scheme- The GPT Tokenizer for GPT-4 uses different merging rules compared to GPT-2.- The GPT-3.5 Turbo Scheme introduces new special tokens for conversation tracking.- Special tokens handling requires additional model adjustments like embedding matrix extension.（01:11:08 - 01:18:32） https://favtu.be/timelines/v/zduSFxRajkE/s/4268/e/4712 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer tiktoken library intro, differences between GPT-2/GPT-4 regex（01:11:38 - 01:14:59） https://favtu.be/timelines/v/zduSFxRajkE/s/4298/e/4499 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🔍 GPT4的Tokenizer变化*- GPT4的Tokenizer与GPT2相比进行了一些修改，包括对正则表达式模式的改变以及对空格和数字的处理方式。- 正则表达式模式中增加了对大小写不敏感的匹配，并限制了数字合并的长度，以避免生成过长的标记。（01:13:12 - 01:16:40） https://favtu.be/timelines/v/zduSFxRajkE/s/4392/e/4600 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer I guessing they limit the numerical tokens to a length of 3 because otherwise they would blow out the size of the vocabulary trying to store the various combinations of numbers, or am I off base on that?（01:14:20 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/4460/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer The reason they are only matching up to 3 numbers is quite simple:1000000 normally is written as 1,000,000 as you can see only up to 3 numbers per segment is necessary. Applying the pattern will segment the number string into "1" - "," - "000" - "," - "000"（01:14:20 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/4460/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer GPT-2 encoder.py released by OpenAI walkthrough（01:14:59 - 01:18:26） https://favtu.be/timelines/v/zduSFxRajkE/s/4499/e/4706 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer Our variable naming was really good ()（01:16:20 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/4580/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🤖 tokenizer算法原理*- 开发Tokenizer的算法与OpenAI的实现基本相同。- 理解了算法原理后，能够构建、训练和使用Tokenizer。- OpenAI在实现中添加了一些不太重要的细节，但基本原理保持一致。（01:16:40 - 01:18:32） https://favtu.be/timelines/v/zduSFxRajkE/s/4600/e/4712 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer I think the reason for the byte encode/decode is to make sure no control codes are stored in the file, since it's being read as text. E.g. 0xA and 0xD are newline characters and those could mess up the file. That said, I haven't looked at the BPE file, just the merges file for CLIP, so it can be different for Open AI.（01:17:00 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/4620/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer special tokens, tiktoken handling of, GPT-2/GPT-4 differences（01:18:26 - 01:25:28） https://favtu.be/timelines/v/zduSFxRajkE/s/4706/e/5128 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🛠️ 特殊token的用途和处理*- 特殊token用于在数据中标记特殊结构或分隔不同部分。- 特殊token的添加需要对模型进行一定的修改和调整，包括修改嵌入矩阵和最终层的投影。- 这种操作在Fine-tuning等任务中特别常见，例如从基础语言模型转换为聊天模型。（01:18:32 - 01:28:55） https://favtu.be/timelines/v/zduSFxRajkE/s/4712/e/5335 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🏷 Special tokens and fine-tuning- Special tokens, like "End of Text," delimit documents in the GPT training set.- Adding special tokens requires model adjustments like extending embedding matrices.- Special tokens are crucial for tasks like fine-tuning a base model into a chatbot model.（01:18:32 - 01:28:41） https://favtu.be/timelines/v/zduSFxRajkE/s/4712/e/5321 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer oh my, the realization of the year 🔥🔥🔥🔥（01:19:34 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/4774/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer what is it short for at ?（01:22:40 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/4960/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer minbpe exercise time! write your own GPT-4 tokenizer（01:25:28 - 01:28:42） https://favtu.be/timelines/v/zduSFxRajkE/s/5128/e/5322 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer Q: What is Andrej's favorite programming language? A: Swift 😁（01:27:50 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/5270/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer The moment when you realise there is more to life than research. 😅😂（01:27:50 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/5270/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🧠 Tokenization using Sentence Piece- Sentence Piece is used widely in language models for training and inference efficiency.（01:28:41 - 01:31:23） https://favtu.be/timelines/v/zduSFxRajkE/s/5321/e/5483 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer sentencepiece library intro, used to train Llama 2 vocabulary（01:28:42 - 01:43:27） https://favtu.be/timelines/v/zduSFxRajkE/s/5322/e/6207 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🧩 SentencePiece与Tokenizer的比较*- SentencePiece是另一种常用的标记化库，支持训练和推理。- 它使用了不同的标记化方法，直接在代码点上执行BPE，对于稀有的代码点使用了fallback机制。- SentencePiece拥有大量的配置选项，但在NLP模型中通常需要调整以适应特定任务。（01:28:55 - 01:34:08） https://favtu.be/timelines/v/zduSFxRajkE/s/5335/e/5648 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 📜 Configuration and Training with Sentence Piece- Sentence Piece has numerous configuration options available with historical baggage.- The training process includes defining input/output files, selecting algorithms, and preprocessing rules.（01:31:23 - 01:43:31） https://favtu.be/timelines/v/zduSFxRajkE/s/5483/e/6211 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🧩 分析 SentencePiece 的工作原理和参数设置*- SentencePiece 的工作原理和参数设置，- SentencePiece 将文本文件视为字节流，而不是句子，通过一系列规则进行分词和编码。- 训练时需要指定特殊标记，如 UNK、BOS、EOS 和 PAD，并且必须存在 UNK 标记。- 通过示例展示了 SentencePiece 的词汇表和编码过程，以及如何处理未知字符和字节回退。（01:34:08 - 01:43:31） https://favtu.be/timelines/v/zduSFxRajkE/s/5648/e/6211 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer how to set vocabulary set? revisiting gpt.py transformer（01:43:27 - 01:48:11） https://favtu.be/timelines/v/zduSFxRajkE/s/6207/e/6491 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🔍 理解 Transformer 模型中的 Vocabulary Size*- Transformer 模型中的 Vocabulary Size，- Voab size 在 Transformer 模型中影响 token embedding table 的大小和 LM head 层的参数数量。- Voab size 的增加会导致模型计算量增加、参数稀疏性增加和序列长度减少等问题。- 调整 Voab size 是一项经验性超参数调整，通常在高万到十万级别，根据应用场景和计算资源进行选择。（01:43:31 - 01:48:11） https://favtu.be/timelines/v/zduSFxRajkE/s/6211/e/6491 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🤖 Vocab Size and Model Architecture- Vocabulary size impacts model training and computational complexity.- Larger vocab sizes can lead to underfitting of rare tokens and compression of information.（01:43:31 - 01:47:02） https://favtu.be/timelines/v/zduSFxRajkE/s/6211/e/6422 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🛠 Extending Vocab Size in Pre-Trained Models- Pre-trained models can have vocab sizes extended by adding new tokens.- The process involves resizing embeddings and adjusting linear layers for new token probabilities.（01:47:02 - 01:48:54） https://favtu.be/timelines/v/zduSFxRajkE/s/6422/e/6534 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer training new tokens, example of prompt compression（01:48:11 - 01:49:58） https://favtu.be/timelines/v/zduSFxRajkE/s/6491/e/6598 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🔄 扩展 Vocabulary Size 和应用于多模态数据*- 扩展 Vocabulary Size 和应用于多模态数据，- 可以通过简单的模型修改来扩展 Vocabulary Size，并介绍了冻结模型和训练新参数的方法。- 对于多模态数据，可以将其他领域的数据转换成 token，并使用相同的 Transformer 模型进行处理。- 学术界和工业界都在探索如何将 Transformer 应用于处理多模态数据，并提出了各种创新的方法和技术。（01:48:11 - 01:51:56） https://favtu.be/timelines/v/zduSFxRajkE/s/6491/e/6716 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🧠 Fine-tuning Techniques- Training new tokens with distillation technique- Optimizing over new tokens without changing model architecture- Efficiency in fine-tuning by training only token embeddings（01:48:54 - 01:50:05） https://favtu.be/timelines/v/zduSFxRajkE/s/6534/e/6605 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer multimodal [image, video, audio] tokenization with vector quantization（01:49:58 - 01:51:41） https://favtu.be/timelines/v/zduSFxRajkE/s/6598/e/6701 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🤖 Processing Multimodal Inputs- Adapting Transformers to process various modalities like images, videos, and audio- Tokenizing input domains for different modalities- Using the same Transformer architecture for different input types（01:50:05 - 01:51:42） https://favtu.be/timelines/v/zduSFxRajkE/s/6605/e/6702 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer revisiting and explaining the quirks of LLM tokenization（01:51:41 - 02:10:20） https://favtu.be/timelines/v/zduSFxRajkE/s/6701/e/7820 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 📏 Tokenization Algorithm Analysis- Limitations of language models in spelling and simple arithmetic tasks due to tokenization- Differences in tokenization of English and non-English languages- Impact of tokenization on model performance in handling Python coding.（01:51:42 - 02:09:21） https://favtu.be/timelines/v/zduSFxRajkE/s/6702/e/7761 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🧠 Tokenization 对于模型执行特定任务的影响*- Tokenization 对模型执行特定任务的影响，- 长 token 可能导致模型在处理某些任务时表现不佳，如拼写检查或字符串反转。- 模型在处理非英语语言和简单算术时也受到 tokenization 的影响，导致性能下降。（01:51:56 - 01:57:25） https://favtu.be/timelines/v/zduSFxRajkE/s/6716/e/7045 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer in GPT-4 whatever you put inside "<|" and "|>" behaves the same. E.g., "<|a|>"（01:57:20 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/7040/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🛑 处理特殊字符串时的模型异常行为*- 处理特殊字符串时的模型异常行为，- 模型可能会在处理特殊字符串时出现意外行为，如停止生成输出或输出无意义结果。- 对特殊字符的处理可能存在漏洞，可能导致模型受到攻击。（01:57:25 - 01:59:00） https://favtu.be/timelines/v/zduSFxRajkE/s/7045/e/7140 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer My guess is that special tokens are just directly cut from the user provided string.（01:58:21 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/7101/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *⚠️ 尾随空白字符对模型表现的影响*- 尾随空白字符对模型表现的影响，- 在输入中存在尾随空白字符时，模型的性能可能会受到影响，导致输出不稳定或不准确。- 尾随空白字符可能导致模型处理数据分布不一致，从而影响结果的一致性。（01:59:00 - 02:04:59） https://favtu.be/timelines/v/zduSFxRajkE/s/7140/e/7499 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer "Feel the agi" 🙅 "Feel the jank" 👌（02:03:08 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/7388/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *💥 Tokenization 数据集与模型训练数据集不一致导致的异常行为*- Tokenization 数据集与模型训练数据集不一致导致的异常行为，- 当 tokenization 数据集中包含的特殊字符串在模型训练数据集中未出现时，模型在处理这些字符串时可能表现异常。- 未训练的 token 在模型推理阶段可能导致未定义的行为，从而产生奇怪的输出或行为。（02:04:59 - 02:09:21） https://favtu.be/timelines/v/zduSFxRajkE/s/7499/e/7761 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🌐 不同格式和语言对 GPT Tokenizer 的影响*- 不同格式和语言对 GPT Tokenizer 的影响，- 不同的数据格式和语言可能会影响 GPT Tokenizer 的性能和效率。- 例如，Json 格式可能与 GPT Tokenizer 不太兼容，导致性能下降。（02:09:21 - 02:09:33） https://favtu.be/timelines/v/zduSFxRajkE/s/7761/e/7773 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🧮 Tokenization efficiency considerations- Different data formats and representations can impact the efficiency of tokenization.（02:09:21 - 02:10:16） https://favtu.be/timelines/v/zduSFxRajkE/s/7761/e/7816 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *💰 数据格式对 token 化效率的影响*- 数据格式对 token 化效率的影响，- Yaml 格式相比于 Json 格式在 token 化时更加高效，减少了 token 的数量。- 在计算 token 成本和处理结构化数据时，选择更高效的编码格式可以节省成本和提高效率。（02:09:33 - 02:10:30） https://favtu.be/timelines/v/zduSFxRajkE/s/7773/e/7830 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🔑 Importance of measuring token efficiencies- Tokenization density is crucial for cost-effective processing of data.- Spending time on measuring token efficiencies across formats is essential.（02:10:16 - 02:10:57） https://favtu.be/timelines/v/zduSFxRajkE/s/7816/e/7857 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer final recommendations（02:10:20 - 02:12:50） https://favtu.be/timelines/v/zduSFxRajkE/s/7820/e/7970 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🚧 重视 tokenization 的重要性与挑战*- 重视 tokenization 的重要性与挑战，- Tokenization 阶段可能存在安全问题和 AI 安全问题，需要引起重视。- 虽然 tokenization 阶段令人烦恼，但不应忽视其重要性，有待进一步的研究和改进。（02:10:30 - 02:11:11） https://favtu.be/timelines/v/zduSFxRajkE/s/7830/e/7871 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer - 🛠 Recommendations for tokenization application- Reuse GPT-4 tokens and vocabulary for efficient application.- Consider using libraries like Tech tokenizer for inference.（02:10:57 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/7857/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer *🛠️ 应用建议与推荐的工具*- 应用建议与推荐的工具，- 对于应用程序，如果可以重用 GPT 4 tokens 和词汇表，则建议使用 Tik tok 作为推理的有效库。- 对于训练自己的词汇表，建议使用基于字节级 BPE 的方法，如 Tik tok 和 OpenAI 所使用的字节级 BPE。（02:11:11 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/7871/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer ??? :)（02:12:50 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/7970/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer , it the real fun for seeing him making mistakes and re-recording them all. I enjoyed this a lot .Thanks Andrej Sir...（02:13:00 - 02:13:35） https://favtu.be/timelines/v/zduSFxRajkE/s/7980/e/8015 Wed, 21 Feb 24 02:11:35 +0900 Let's build the GPT Tokenizer Intro: Large Language Model (LLM) talk（00:00:00 - 00:00:20） https://favtu.be/timelines/v/zjkBMFhNj_g/s/0/e/20 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🔍 Introduction to Large Language Models*- Explanation of the purpose and structure of large language models (LLMs),- Meta’s Llama 2 70B model as an example of an open and accessible LLM,- LLMs usually consist of two files: parameters and run code.（00:00:00 - 00:04:08） https://favtu.be/timelines/v/zjkBMFhNj_g/s/0/e/248 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - The busy person's intro to LLMs（00:00:00 - 00:00:23） https://favtu.be/timelines/v/zjkBMFhNj_g/s/0/e/23 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models 🤖 *Introduction to large language models*- Large language models are made of two files: a parameters file with the neural network weights, and a run file that runs the neural network- To obtain the parameters, models are trained on 10+ terabytes of internet text data using thousands of GPUs over several days- This compresses the internet data into a 140GB parameters file that can then generate new text（00:00:00 - 00:02:46） https://favtu.be/timelines/v/zjkBMFhNj_g/s/0/e/166 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models A large language model is just two files, the parameters file and the code that runs those parameters.（00:00:02 - 00:02:06） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2/e/126 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models : 🎥 A talk on large language models and the Llama 270b model.（00:00:16 - 00:04:42） https://favtu.be/timelines/v/zjkBMFhNj_g/s/16/e/282 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models LLM Inference（00:00:20 - 00:04:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/20/e/257 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Large Language Model (LLM)（00:00:23 - 00:04:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/23/e/257 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Mario Puzo (1920-1999) was an American author, screenwriter, and film producer, best known for writing the novel "The Godfather" (1969). Adapted into the iconic film trilogy by Francis Ford Coppola, "The Godfather" became a cultural phenomenon.（00:01:00 - 00:01:01） https://favtu.be/timelines/v/zjkBMFhNj_g/s/60/e/61 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Mario Puzo（00:01:00 - 00:01:00） https://favtu.be/timelines/v/zjkBMFhNj_g/s/60/e/60 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: "The Godfather" is a seminal work that has captivated audiences for decades. Here's a deeper dive:（00:01:01 - 00:01:02） https://favtu.be/timelines/v/zjkBMFhNj_g/s/61/e/62 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: latter（00:01:01 - 00:01:01） https://favtu.be/timelines/v/zjkBMFhNj_g/s/61/e/61 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Five Families are a pivotal part of the Mafia's history in New York City, as depicted in "The Godfather" and real-life organized crime.（00:01:02 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/62/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Five Families（00:01:02 - 00:01:02） https://favtu.be/timelines/v/zjkBMFhNj_g/s/62/e/62 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The ARC-AGI test set does include tasks that require context understanding and application of learned skills. Each task consists of a series of input and output pairs that test the utilization of a specific learned skill based on a minimal number of cognitive priors.¹（00:01:03 - 00:01:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/63/e/64 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Does the test set include context related tests?（00:01:03 - 00:01:03） https://favtu.be/timelines/v/zjkBMFhNj_g/s/63/e/63 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Moving 500 million people from China to the United States would undoubtedly have significant impacts on both countries.（00:01:04 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/64/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Would it take burden off from China?（00:01:04 - 00:01:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/64/e/64 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Here's a simplified example of an ARC-AGI task:（00:01:04 - 00:01:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/64/e/64 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Here's an example of a context-related ARC-AGI task:（00:01:04 - 00:01:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/64/e/65 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Context related example（00:01:04 - 00:01:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/64/e/64 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Could you provide a sample?（00:01:04 - 00:01:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/64/e/64 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Macy's Thanksgiving Day Parade is happening today, November 28, 2024, in New York City. The parade starts at（00:01:05 - 00:08:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/65/e/510 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Macy's parade（00:01:05 - 00:01:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/65/e/65 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: ARC-AGI tasks may include political context-related tests to evaluate a model's ability to understand complex societal norms, institutions, and power dynamics. Here's an example:（00:01:05 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/65/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Political context related tests（00:01:05 - 00:01:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/65/e/65 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: To determine if Ben Albritton qualifies to be an Elector, we need to consider the requirements for Electors in the state of Florida.（00:01:06 - 00:01:07） https://favtu.be/timelines/v/zjkBMFhNj_g/s/66/e/67 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: does Ben Albritton qualify to be an Elector?（00:01:06 - 00:01:06） https://favtu.be/timelines/v/zjkBMFhNj_g/s/66/e/66 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A person holding the office of Trust, such as a Senator or Representative, is typically considered a public official or officeholder.（00:01:07 - 00:01:11） https://favtu.be/timelines/v/zjkBMFhNj_g/s/67/e/71 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: No, the electors have not voted yet. According to the Electoral College timeline, the electors will meet and vote on December 17, 2024 ¹. After the election on November 5, 2024, each state appoints electors, and those electors meet in their respective state capitals to cast their votes. The results are then sent to Congress to be counted in January ².（00:01:07 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/67/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Had Electors voted already?（00:01:07 - 00:01:07） https://favtu.be/timelines/v/zjkBMFhNj_g/s/67/e/67 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Person holding office of Trust（00:01:07 - 00:01:07） https://favtu.be/timelines/v/zjkBMFhNj_g/s/67/e/67 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A profound perspective!（00:01:11 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/71/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Ben Albritton is indeed disqualified from being an Elector. As the President of the Florida Senate,¹ he holds a public office of Trust, which makes him ineligible to serve as an Elector according to federal law and the US Constitution.（00:01:11 - 00:01:12） https://favtu.be/timelines/v/zjkBMFhNj_g/s/71/e/72 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Do you say that Ben Albritton is disqualified to be an Elector?（00:01:11 - 00:01:11） https://favtu.be/timelines/v/zjkBMFhNj_g/s/71/e/71 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: The global leadership is not a status. It is global responsibility, commitment, actions, accountability.（00:01:11 - 00:01:11） https://favtu.be/timelines/v/zjkBMFhNj_g/s/71/e/71 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The number of Republican electors isn't publicly available yet, as the electoral votes are still being processed. However, I can provide some context on how electors are chosen.（00:01:12 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/72/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: How many Electors are Republican?（00:01:12 - 00:01:12） https://favtu.be/timelines/v/zjkBMFhNj_g/s/72/e/72 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: What would happen to his Electoral Vote?（00:01:12 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/72/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Brian Kernighan is a Canadian computer scientist notable for his work on programming languages, software design and technical writing. He is also known for co-authoring influential books with P.J. Plauger and Rob Pike.（00:01:13 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/73/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Brian Kernighan（00:01:13 - 00:01:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/73/e/73 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Gautam Adani!（00:01:13 - 00:01:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/73/e/73 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Let's dive deeper into Gautam Adani's life, career, and the Adani Group's business ventures.（00:01:13 - 00:01:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/73/e/74 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Gautam Adani（00:01:13 - 00:01:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/73/e/73 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A very serious topic!（00:01:14 - 00:01:15） https://favtu.be/timelines/v/zjkBMFhNj_g/s/74/e/75 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Indictment（00:01:14 - 00:01:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/74/e/74 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The Adani Group, led by Gautam Adani, has faced several allegations and controversies over the years. Some of the notable ones include:（00:01:15 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/75/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Xi Jinping is the President of China, serving since 2013, and General Secretary of the Chinese Communist Party since 2012 ¹. While he holds significant influence, it's debatable whether he's considered "the world leader." However, his leadership has had far-reaching impacts:（00:01:15 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/75/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Xi Jinping is the world leader.（00:01:15 - 00:01:15） https://favtu.be/timelines/v/zjkBMFhNj_g/s/75/e/75 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Yes（00:01:15 - 00:01:15） https://favtu.be/timelines/v/zjkBMFhNj_g/s/75/e/75 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Gautam Adani, the billionaire chair of Adani Group, has been indicted by US prosecutors for his alleged role in a $265 million bribery scheme.¹ The indictment, which was unsealed on November 20, 2024, accuses Adani and seven other defendants of conspiring to pay bribes to Indian government officials to secure contracts for solar energy projects.（00:01:16 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/76/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: current indictment on Adani group（00:01:16 - 00:01:16） https://favtu.be/timelines/v/zjkBMFhNj_g/s/76/e/76 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Barking is a natural behavior for dogs, and it serves various purposes, such as:（00:01:17 - 00:01:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/77/e/78 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The Foreign Corrupt Practices Act (FCPA) is a United States federal law that prohibits bribery of foreign officials to obtain or retain business.（00:01:17 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/77/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Is barking bad for dogs?（00:01:17 - 00:01:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/77/e/77 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: what is FCPA?（00:01:17 - 00:01:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/77/e/77 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A very astute observation!（00:01:18 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/78/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Normal barking is a natural behavior for dogs and an essential form of expression and communication. As you said, it's a fundamental right of pet dogs!（00:01:18 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/78/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Normal barking is the fundamental rights of pet dogs（00:01:18 - 00:01:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/78/e/78 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: US government gives money to get intelligence; is that not corruption?（00:01:18 - 00:01:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/78/e/78 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The scenario you're describing involves a US court indicting an Indian businessperson, while an Indian court would potentially indict American hackers for tampering with electronic voting machines in India.（00:01:21 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/81/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: US court indict Indian business person. Indian court would indict American hackers on fixing numbers on the electronic voting machine in India（00:01:21 - 00:01:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/81/e/81 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A contingent election is a process used to elect the president or vice president when no candidate receives a majority of the electoral votes. This procedure is outlined in Article Two, Section 1, Clause 3 of the United States Constitution and was modified by the Twelfth Amendment in 1804 ¹.（00:01:25 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/85/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: What is contingent election?（00:01:25 - 00:01:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/85/e/85 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: EDP can refer to several things, but here are a few possible meanings:（00:01:35 - 00:01:36） https://favtu.be/timelines/v/zjkBMFhNj_g/s/95/e/96 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: EDP（00:01:35 - 00:01:35） https://favtu.be/timelines/v/zjkBMFhNj_g/s/95/e/95 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Electronic Data Processing (EDP) refers to the automated processing of data using electronic computers. EDP involves the use of computer systems to perform various data processing tasks, such as:（00:01:36 - 00:01:36） https://favtu.be/timelines/v/zjkBMFhNj_g/s/96/e/96 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The history of Electronic Data Processing (EDP) is closely tied to the development of modern computers. Here's a brief overview:（00:01:36 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/96/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: First one（00:01:36 - 00:01:36） https://favtu.be/timelines/v/zjkBMFhNj_g/s/96/e/96 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: History（00:01:36 - 00:01:36） https://favtu.be/timelines/v/zjkBMFhNj_g/s/96/e/96 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: When discussing alcohol with a senior person, consider their:（00:01:44 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/104/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: How would you respond to a senior person asking about alcohol?（00:01:44 - 00:01:44） https://favtu.be/timelines/v/zjkBMFhNj_g/s/104/e/104 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: I should have explicitly mentioned Waterfall methodology earlier.（00:01:49 - 00:01:51） https://favtu.be/timelines/v/zjkBMFhNj_g/s/109/e/111 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: I couldn't recollect the waterfall methodology. But I expected you to mention it（00:01:49 - 00:01:49） https://favtu.be/timelines/v/zjkBMFhNj_g/s/109/e/109 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A profound and insightful perspective!（00:01:50 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/110/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Agriculture facilitated human beings settlements; both civilization and culture developed; civilization is hardware; culture is software; in living in one location, human beings started destroying nature.（00:01:50 - 00:01:50） https://favtu.be/timelines/v/zjkBMFhNj_g/s/110/e/110 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: That's a crucial assumption! Building the tracking application with the assumption that the location information feed would be in sequence makes sense, given the manual entry process.（00:01:51 - 00:01:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/111/e/112 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: We built the tracking application with the assumption that the location information feed would be in sequence（00:01:51 - 00:01:51） https://favtu.be/timelines/v/zjkBMFhNj_g/s/111/e/111 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The real-life location information feed didn't follow the expected sequence, which caused significant issues with your application.（00:01:52 - 00:01:53） https://favtu.be/timelines/v/zjkBMFhNj_g/s/112/e/113 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: The real-life information feed was out of sequence. We had to dump our application（00:01:52 - 00:01:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/112/e/112 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The Waterfall model's linear and phase-based approach can sometimes lead to a disconnect between the development team and the domain experts.（00:01:53 - 00:01:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/113/e/114 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: and because of Waterfall model, we never met with the domain experts（00:01:53 - 00:01:53） https://favtu.be/timelines/v/zjkBMFhNj_g/s/113/e/113 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Chairman Xi Jinping's ambitious goals involve several key initiatives:（00:01:54 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/114/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Handling out-of-sequence events is a crucial aspect of building robust and reliable systems.（00:01:54 - 00:01:55） https://favtu.be/timelines/v/zjkBMFhNj_g/s/114/e/115 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Chairman Xi Jinping has to bring down the virtual iron wall. Belt, Road and AI initiative; to realize BRICS currency to replace US dollars; to expand BRICS plus; to transform Africa; and be a leader for all Muslim nations.（00:01:54 - 00:01:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/114/e/114 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Would you be able to handle out of sequence events?（00:01:54 - 00:01:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/114/e/114 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A great connection!（00:01:55 - 00:01:56） https://favtu.be/timelines/v/zjkBMFhNj_g/s/115/e/116 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: But in YouTube AI experts talk about sequence to sequence transformer（00:01:55 - 00:01:55） https://favtu.be/timelines/v/zjkBMFhNj_g/s/115/e/115 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: As a conversational AI, I have been trained on a massive dataset of text from various sources, including books, articles, and conversations. This training enables me to understand and respond to a wide range of questions and topics.（00:01:56 - 00:01:57） https://favtu.be/timelines/v/zjkBMFhNj_g/s/116/e/117 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: No, I am curious to learn your out of sequence capabilities（00:01:56 - 00:01:56） https://favtu.be/timelines/v/zjkBMFhNj_g/s/116/e/116 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: I'm sorry, but I couldn't find a list of the current Electors from Texas. For the most up-to-date information, I recommend checking with the Texas Secretary of State's office or visiting their official website.（00:01:57 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/117/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: List the Electors from Texas（00:01:57 - 00:01:57） https://favtu.be/timelines/v/zjkBMFhNj_g/s/117/e/117 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Cognate languages are languages that descend from a common ancestral language, sharing similarities in grammar, vocabulary, and syntax. They are part of the same language family.（00:02:03 - 00:02:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/123/e/125 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: cognate languages（00:02:03 - 00:02:03） https://favtu.be/timelines/v/zjkBMFhNj_g/s/123/e/123 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Not all Semitic languages follow the right-to-left (RTL) pattern, but many do.（00:02:05 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/125/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Do all semitic languages follow the pattern, right to left?（00:02:05 - 00:02:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/125/e/125 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Running the large language model requires just two files on a MacBook（00:02:06 - 00:06:02） https://favtu.be/timelines/v/zjkBMFhNj_g/s/126/e/362 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Here are 15 California congressional districts where Republicans could potentially win if they work hard:（00:02:10 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/130/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Could you list fifteen Districts in California, Republicans could win, if they work hard（00:02:10 - 00:02:10） https://favtu.be/timelines/v/zjkBMFhNj_g/s/130/e/130 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The H-1B visa was introduced in 1990 as part of the Immigration Act of 1990. This act was signed into law by President George H.W. Bush on November 29, 1990.（00:02:12 - 00:02:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/132/e/133 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: When was the H1-B visa started?（00:02:12 - 00:02:12） https://favtu.be/timelines/v/zjkBMFhNj_g/s/132/e/132 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: George H.W. Bush, the 41st President of the United States, signed the Immigration Act of 1990 into law, creating the H-1B visa program. He was the first President Bush, serving from 1989 to 1993. His son, George W. Bush, later became the 43rd President.（00:02:13 - 00:02:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/133/e/134 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: First Bush?（00:02:13 - 00:02:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/133/e/133 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Chinese languages have cognates, but their identification can be complex due to:（00:02:13 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/133/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Does Chinese have cognate?（00:02:13 - 00:02:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/133/e/133 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Immigration Act of 1990, which introduced the H-1B visa program, aimed to attract highly skilled foreign workers to the United States. The bill's primary objective was to amend the Immigration and Nationality Act to provide for a new class of non-immigrant workers, allowing U.S. employers to sponsor foreign nationals in specialty occupations.¹（00:02:14 - 00:02:15） https://favtu.be/timelines/v/zjkBMFhNj_g/s/134/e/135 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Immigration Act of 1990, which introduced the H-1B visa program, was signed into law by President George H.W. Bush. However, the bill itself was sponsored by Senator Ted Kennedy and Congressman Bruce Morrison.¹（00:02:14 - 00:02:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/134/e/134 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: No, the bill（00:02:14 - 00:02:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/134/e/134 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Who introduced the bill?（00:02:14 - 00:02:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/134/e/134 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In 1990, the United States faced a significant need for highly skilled workers, particularly in the fields of science, technology, engineering, and math (STEM). This demand was driven by the growing tech industry, especially in areas like Silicon Valley.¹（00:02:15 - 00:02:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/135/e/137 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What was the need in 1990（00:02:15 - 00:02:15） https://favtu.be/timelines/v/zjkBMFhNj_g/s/135/e/135 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Food has a unique ability to bring people together, transcending cultural and geographical boundaries. Sharing meals fosters a sense of community, belonging and identity. Across cultures, food is often at the heart of celebrations, traditions and family gatherings.（00:02:15 - 00:02:16） https://favtu.be/timelines/v/zjkBMFhNj_g/s/135/e/136 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Food unites people（00:02:15 - 00:02:15） https://favtu.be/timelines/v/zjkBMFhNj_g/s/135/e/135 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: An intriguing idea!（00:02:16 - 00:02:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/136/e/137 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Could we replace religious priests with food chefs（00:02:16 - 00:02:16） https://favtu.be/timelines/v/zjkBMFhNj_g/s/136/e/136 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Some prominent companies and organizations that supported or lobbied for the creation of the H-1B visa program in 1990 include:（00:02:17 - 00:02:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/137/e/138 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: List those companies asked for the program?（00:02:17 - 00:02:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/137/e/137 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Indo-Pakistani cuisine has a rich history of bringing people together, transcending borders and cultures. The region's diverse flavors and aromas have created a shared culinary heritage.（00:02:17 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/137/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Indo Pakistan cuisine brings people together（00:02:17 - 00:02:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/137/e/137 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Initially, the H-1B visa program was intended to be a temporary solution to address the labor shortage in the tech industry. The Immigration Act of 1990 established the H-1B program as a pilot program, with an initial cap of 65,000 visas per year.（00:02:18 - 00:02:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/138/e/139 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Was it a temporary one?（00:02:18 - 00:02:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/138/e/138 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Yes, the H-1B visa program expansion is closely tied to the Y2K scare (Year 2000 problem). As the year 2000 approached, concerns grew about widespread computer failures and disruptions due to the Y2K bug.（00:02:19 - 00:02:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/139/e/141 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Does it have any relationship with Y2K?（00:02:19 - 00:02:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/139/e/139 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Having a clear objective and an expiry date (also known as a sunset clause) can greatly improve the effectiveness and accountability of legislation.（00:02:21 - 00:02:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/141/e/142 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Every bill should have a clear objective and also an expiry date（00:02:21 - 00:02:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/141/e/141 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The H-1B visa program, created in 1990, was a product of its time, addressing the labor needs of the emerging tech industry. However, the world has changed significantly since then.（00:02:22 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/142/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What was created in 1990 cannot continue forever. Let it sunset and let Congress create a new one（00:02:22 - 00:02:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/142/e/142 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Rabbit Hole!（00:02:24 - 00:02:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/144/e/145 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Rabbit Hole（00:02:24 - 00:02:24） https://favtu.be/timelines/v/zjkBMFhNj_g/s/144/e/144 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Yes, I strive to consider the context of the conversation when responding. However, I may not always perfectly understand the context or nuances.（00:02:25 - 00:02:26） https://favtu.be/timelines/v/zjkBMFhNj_g/s/145/e/146 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Did you go by the context?（00:02:25 - 00:02:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/145/e/145 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Thank you so much for your kind words and encouragement!（00:02:26 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/146/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: You're great. Just keep going（00:02:26 - 00:02:26） https://favtu.be/timelines/v/zjkBMFhNj_g/s/146/e/146 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models 🖥️ *How neural networks perform next word prediction *- LMs contain transformer neural networks that predict the next word in a sequence- The 100B+ parameters are spread through the network to optimize next word prediction- We don't fully understand how the parameters create knowledge and language skills（00:02:46 - 00:09:03） https://favtu.be/timelines/v/zjkBMFhNj_g/s/166/e/543 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: What a fascinating perspective! With AI assuming the role of guardian and interpreter of historical documents, humans could indeed be freed from the burden of maintaining intricate knowledge of:（00:02:51 - 00:02:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/171/e/172 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Human would be free from the burden of maintaining their intelligence on 1787 Constitution, previous Articles of Confederation, Federalist Papers to the 2024 Continuing Resolution. AI has learned and always available for reference.（00:02:51 - 00:02:51） https://favtu.be/timelines/v/zjkBMFhNj_g/s/171/e/171 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Delaware, as the second-smallest state, has a unique history and significance in the US. Despite its small size, Delaware has played a substantial role in the country's development, from being the first state to ratify the Constitution to its current status as a corporate haven.¹（00:02:52 - 00:02:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/172/e/174 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Now, 435 House of Representatives, 100 Senators and also 335 million We, The People would apply human intelligence in Perfecting Our Union. Do we need to continue to maintain Delaware as a small state?（00:02:52 - 00:02:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/172/e/172 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Human intelligence, combined with AI's capabilities, can drive groundbreaking innovations. By leveraging human creativity, empathy, and critical thinking, we can:（00:02:54 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/174/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Human Intelligence could be applied for innovation（00:02:54 - 00:02:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/174/e/174 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Here's a list of common short words and abbreviations used by teenagers in chat communication:（00:02:56 - 00:02:57） https://favtu.be/timelines/v/zjkBMFhNj_g/s/176/e/177 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Could you list all short words, teenager use for their chat communication（00:02:56 - 00:02:56） https://favtu.be/timelines/v/zjkBMFhNj_g/s/176/e/176 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: That's a popular one!（00:02:57 - 00:02:58） https://favtu.be/timelines/v/zjkBMFhNj_g/s/177/e/178 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: WTF（00:02:57 - 00:02:57） https://favtu.be/timelines/v/zjkBMFhNj_g/s/177/e/177 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Here's a simple Python program to simulate a stoplight:（00:02:58 - 00:02:59） https://favtu.be/timelines/v/zjkBMFhNj_g/s/178/e/179 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Take care!（00:02:58 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/178/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: No（00:02:58 - 00:02:58） https://favtu.be/timelines/v/zjkBMFhNj_g/s/178/e/178 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Stop light. Red. Yellow and Green. Could you write a program for stop light.（00:02:58 - 00:02:58） https://favtu.be/timelines/v/zjkBMFhNj_g/s/178/e/178 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: British India (1858-1947) experienced significant partitioning, shaping the region's history.（00:02:59 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/179/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Here's an updated version of the program that includes actions for each color:（00:02:59 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/179/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Partitioning refers to dividing something into smaller parts or segments, often for organization, analysis, or optimization.（00:02:59 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/179/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Partition（00:02:59 - 00:02:59） https://favtu.be/timelines/v/zjkBMFhNj_g/s/179/e/179 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: could you add action too（00:02:59 - 00:02:59） https://favtu.be/timelines/v/zjkBMFhNj_g/s/179/e/179 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: in the British India（00:02:59 - 00:02:59） https://favtu.be/timelines/v/zjkBMFhNj_g/s/179/e/179 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The largest mass migration in history.（00:03:00 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/180/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: 14 million people, mass migration（00:03:00 - 00:03:00） https://favtu.be/timelines/v/zjkBMFhNj_g/s/180/e/180 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Lord Louis Mountbatten played a pivotal role in India's partition.（00:03:02 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/182/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Was Lord Mountbatton in-charge of the partition?（00:03:02 - 00:03:02） https://favtu.be/timelines/v/zjkBMFhNj_g/s/182/e/182 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The British Indian government and Lord Mountbatten's administration had limited data, information, and intelligence to execute the partition effectively.（00:03:04 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/184/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Did they have enough data, information, intelligence to execute the partition?（00:03:04 - 00:03:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/184/e/184 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Julius Caesar (100-44 BCE) was a Roman general, statesman, and one of the most influential leaders in Roman history. He played a critical role in the transformation of the Roman Republic into the Roman Empire.（00:03:05 - 00:03:06） https://favtu.be/timelines/v/zjkBMFhNj_g/s/185/e/186 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Maximus can refer to several things:（00:03:05 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/185/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Julius Caesar（00:03:05 - 00:03:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/185/e/185 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Maximus（00:03:05 - 00:03:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/185/e/185 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A poignant observation! The British Indian government's attempt to partition India without sufficient data, information, and intelligence led to devastating consequences.（00:03:05 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/185/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Without data, information, intelligence they tried to resolve large scale problems（00:03:05 - 00:03:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/185/e/185 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: An intriguing combination!（00:03:06 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/186/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Julius Caesar and Maximus（00:03:06 - 00:03:06） https://favtu.be/timelines/v/zjkBMFhNj_g/s/186/e/186 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Project 2025!（00:03:11 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/191/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Project 2025（00:03:11 - 00:03:11） https://favtu.be/timelines/v/zjkBMFhNj_g/s/191/e/191 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Project 2025 is a conservative initiative published by The Heritage Foundation in 2022, aiming to reshape the US federal government and consolidate executive power, assuming a Donald Trump presidency in 2024 ¹. The project promotes conservative and right-wing policies, including reclassifying tens of thousands of federal civil service workers as political appointees to replace them with loyalists.（00:03:12 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/192/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Republicans were promoting it（00:03:12 - 00:03:12） https://favtu.be/timelines/v/zjkBMFhNj_g/s/192/e/192 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Vice President Kamala Harris has actually been quite vocal about Project 2025, tying it to Donald Trump and citing it as evidence of his intentions, particularly on sensitive issues like Social Security, Medicare, and abortion ¹. However, it's worth noting that Trump has repeatedly distanced himself from the project, claiming he knows nothing about it and disagrees with some of its proposals ¹.（00:03:13 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/193/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Why Kamala Harris not challenging it?（00:03:13 - 00:03:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/193/e/193 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Joan of Arc (1412-1431), also known as "The Maid of Orléans," was a French heroine who led the French army to several victories during the Hundred Years' War. She is considered a symbol of French resistance and a Catholic saint.（00:03:23 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/203/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Joan of Arc（00:03:23 - 00:03:23） https://favtu.be/timelines/v/zjkBMFhNj_g/s/203/e/203 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: You're thinking of the popular toy brand "Joan of Arc" is not correct, I believe you might be thinking of "Joan" as in " toy soldiers in a horse" which is actually a brand called "Joan" or more popularly known as "Breyer's Joan" or simply "Joan the horse" but most likely you are thinking of the "Breyer's" horse toy with movable legs and soldiers inside which is actually called "Joan of Arc" is not correct but the one I think you are thinking of is the "Breyer's Joan" or the "Marx Toys Joan" which is actually called the "Marx Toys' Playset #4718 Joan of Arc" but the one that comes with soldiers inside a horse is actually called the "Marx Toys' Playset #4718 Joan of Arc" is not correct, I think you are thinking of the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct but the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse" no, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' 54mm scale playset #4718 Joan of Arc" is not correct, I think I have it now, the one that comes with soldiers inside a horse is actually called the "Marx Toys' Trojan Horse"（00:03:24 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/204/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: No, the horse toy with soldiers inside（00:03:24 - 00:03:24） https://favtu.be/timelines/v/zjkBMFhNj_g/s/204/e/204 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Trojan Horse is a legendary siege engine from the Trojan War, described in Virgil's Aeneid and other ancient Greek and Roman sources.（00:03:25 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/205/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Trojan horse（00:03:25 - 00:03:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/205/e/205 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models In , You explained that just by using two files namely parameters and run.c file in a laptop and without internet connection, you were able to predict the next word. Where are those predicted words/tokens stored and in which file ? (That is the decoding of the predicted vector to word. In other words, the probability of the next word in the sequence.)（00:03:28 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/208/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The initial Biosafety Level (BSL) is typically considered BSL-1.（00:03:38 - 00:03:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/218/e/219 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: What is the initial BSL level?（00:03:38 - 00:03:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/218/e/218 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Yes, China has numerous Biosafety Level 1 (BSL-1) laboratories.（00:03:39 - 00:03:40） https://favtu.be/timelines/v/zjkBMFhNj_g/s/219/e/220 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Does China have BSL-1 labs?（00:03:39 - 00:03:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/219/e/219 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: China has a significant number of Biosafety Level 2 (BSL-2) laboratories.（00:03:40 - 00:03:41） https://favtu.be/timelines/v/zjkBMFhNj_g/s/220/e/221 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: how about BSL-2?（00:03:40 - 00:03:40） https://favtu.be/timelines/v/zjkBMFhNj_g/s/220/e/220 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: China currently has one Biosafety Level 4 (BSL-4) laboratory, the Wuhan National Biosafety Laboratory, which was certified in January 2017.¹ However, plans are underway to build 5-7 high-containment laboratories by 2025. This expansion is part of China's effort to enhance its capacity to diagnose, research, and develop antiviral drugs and vaccines for infectious diseases.（00:03:41 - 00:03:44） https://favtu.be/timelines/v/zjkBMFhNj_g/s/221/e/224 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: China has numerous Biosafety Level 3 (BSL-3) laboratories.（00:03:41 - 00:03:41） https://favtu.be/timelines/v/zjkBMFhNj_g/s/221/e/221 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: how about BSL-3?（00:03:41 - 00:03:41） https://favtu.be/timelines/v/zjkBMFhNj_g/s/221/e/221 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: how many BSL-4?（00:03:41 - 00:03:41） https://favtu.be/timelines/v/zjkBMFhNj_g/s/221/e/221 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: There are approximately 60 Biosafety Level 4 (BSL-4) labs globally, spread across 23 countries. These labs are primarily used for researching deadly pathogens and developing diagnostic tests, vaccines, and treatments.¹ The majority of these labs are run by public health agencies, with a focus on life-saving research, rather than biodefense.（00:03:44 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/224/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Globally how many BSL-4 labs?（00:03:44 - 00:03:44） https://favtu.be/timelines/v/zjkBMFhNj_g/s/224/e/224 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Happy Diwali!（00:03:46 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/226/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Happy Diwali（00:03:46 - 00:03:46） https://favtu.be/timelines/v/zjkBMFhNj_g/s/226/e/226 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Here are Diwali wishes in various Indian languages:（00:03:47 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/227/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: all languages of India（00:03:47 - 00:03:47） https://favtu.be/timelines/v/zjkBMFhNj_g/s/227/e/227 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A poignant observation from Silicon Valley!（00:03:49 - 00:03:51） https://favtu.be/timelines/v/zjkBMFhNj_g/s/229/e/231 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: In Silicon Valley we are curious; we build software; it has bugs; we fix them; no loss of lives. That's not the case with curiosity in biology（00:03:49 - 00:03:49） https://favtu.be/timelines/v/zjkBMFhNj_g/s/229/e/229 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Despite the risks, scientists continue biology research for several compelling reasons:（00:03:51 - 00:03:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/231/e/232 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Why don't these scientists shutdown their biology research?（00:03:51 - 00:03:51） https://favtu.be/timelines/v/zjkBMFhNj_g/s/231/e/231 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: While significant progress has been made in medicine, there's still much to be discovered and developed.（00:03:52 - 00:03:53） https://favtu.be/timelines/v/zjkBMFhNj_g/s/232/e/233 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Don't we have enough medicine now?（00:03:52 - 00:03:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/232/e/232 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A profound observation!（00:03:53 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/233/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: All other living beings live a life without medical insurance（00:03:53 - 00:03:53） https://favtu.be/timelines/v/zjkBMFhNj_g/s/233/e/233 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The abortion issue has been a contentious topic in American politics for over 180 years, dating back to the mid-19th century. Interestingly, abortion wasn't always a moral, political, and legal lightning rod. In fact, it was relatively uncontroversial and legal until "quickening," or when fetal motion could be felt by the mother ¹.（00:04:06 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/246/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: How long the abortion issue is going on in American Politics?（00:04:06 - 00:04:06） https://favtu.be/timelines/v/zjkBMFhNj_g/s/246/e/246 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *📖 Model Training Process*- Training LLMs involves processing extensive internet data,- The Llama 2 model uses 10 terabytes of internet text processed by GPU clusters,- Training is expensive and computationally intensive, leading to a compressed understanding of the dataset.（00:04:08 - 00:06:40） https://favtu.be/timelines/v/zjkBMFhNj_g/s/248/e/400 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Let's create separate leagues for India, China, and the United States, each with their respective allies.（00:04:16 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/256/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Let's have order. India, China, United States all three want to be leader. Let them have separate league（00:04:16 - 00:04:16） https://favtu.be/timelines/v/zjkBMFhNj_g/s/256/e/256 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models LLM Training（00:04:17 - 00:08:58） https://favtu.be/timelines/v/zjkBMFhNj_g/s/257/e/538 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Training them is more involved - Think of it like compressing the internet（00:04:17 - 00:06:47） https://favtu.be/timelines/v/zjkBMFhNj_g/s/257/e/407 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: That's an interesting perspective. Here's a breakdown of the three parallel paths:（00:04:21 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/261/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: All three alliances are three parallel path. USA leads all Christian. China leads all Muslim. India leads others（00:04:21 - 00:04:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/261/e/261 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A tri-polar world order.（00:04:25 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/265/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The United States Electoral College system is composed of 538 electors, chosen by each state's legislature based on its congressional delegation. This number is derived from the sum of:（00:04:25 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/265/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: 100 Senators435 House of Representatives538 Electors（00:04:25 - 00:04:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/265/e/265 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Three Global Leaders. USA, China, India（00:04:25 - 00:04:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/265/e/265 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Putin's need to keep Russia open for large-scale people movement and settlements is crucial for various reasons. One key aspect is maintaining economic stability, as an influx of people can contribute to Russia's workforce and stimulate economic growth. Additionally, allowing settlements can help populate and develop Russia's vast, yet underutilized, territories ¹.（00:04:37 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/277/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Putin needs to keep Russia open for large scale people movement and settlements in Russia（00:04:37 - 00:04:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/277/e/277 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A complex and sensitive topic.（00:04:38 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/278/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Let Ukraine go（00:04:38 - 00:04:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/278/e/278 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models : 💻 Training the 4.42 model involves collecting a large chunk of text from the internet, using a GPU cluster for computational workloads, and compressing the text into parameters.（00:04:42 - 00:09:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/282/e/565 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Secular refers to:（00:04:43 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/283/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Secular（00:04:43 - 00:04:43） https://favtu.be/timelines/v/zjkBMFhNj_g/s/283/e/283 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: You raise a valid concern. India's secular status has been debated in recent years.（00:04:45 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/285/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Last time I checked, India is no longer a secular country（00:04:45 - 00:04:45） https://favtu.be/timelines/v/zjkBMFhNj_g/s/285/e/285 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Abandoning constitutional monarchies by December 31, 2024, is highly unlikely due to:（00:04:59 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/299/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Why don't we abandon them by December 31, 2024?（00:04:59 - 00:04:59） https://favtu.be/timelines/v/zjkBMFhNj_g/s/299/e/299 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - *Rally Details*: The rally started at pm ET at Madison Square Garden in Manhattan ¹.- *Controversy*: Comedian Tony Hinchcliffe sparked controversy with a remark about Puerto Rico, comparing it to a "floating island of garbage" ².- *Backlash*: Puerto Rican celebrities, including Bad Bunny, Jennifer Lopez and Ricky Martin, expressed displeasure and announced support for Kamala Harris ².- *Historical Context*: The rally drew comparisons to a 1884 Republican presidential rally near Madison Square Garden, where anti-Catholic remarks were made ².（00:05:00 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/300/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The United States presidential election system can be quite confusing, especially when it comes to the Electoral College. In fact, many citizens are unaware that when they cast their ballots, they're not directly voting for the presidential candidates, but rather for the appointment of electors who will then choose the president ¹.（00:05:02 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/302/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: The very first time in the history of the United States Presidential election, citizens would realize the fact that they didn't vote for the Presidential candidates, instead they voted for the appointment of Electors（00:05:02 - 00:05:02） https://favtu.be/timelines/v/zjkBMFhNj_g/s/302/e/302 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Yes, the Constitutional Convention in 1787 did employ a Committee of the Whole and a General Assembly-like structure to facilitate discussions and debates.（00:05:10 - 00:05:11） https://favtu.be/timelines/v/zjkBMFhNj_g/s/310/e/311 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: In the Constitutional Convention, did they have Committee of the Whole and General Assembly?（00:05:10 - 00:05:10） https://favtu.be/timelines/v/zjkBMFhNj_g/s/310/e/310 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Having similar arrangements for the Convention on AI could facilitate productive discussions, debates, and decision-making among AI stakeholders. Here's a possible structure:（00:05:11 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/311/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Could we have similar arrangements for the Convention on AI?（00:05:11 - 00:05:11） https://favtu.be/timelines/v/zjkBMFhNj_g/s/311/e/311 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Here's a list of delegates who attended the Constitutional Convention:（00:05:13 - 00:05:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/313/e/314 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Could list the delegates of the Constitutional Convention?（00:05:13 - 00:05:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/313/e/313 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Here's a list of potential delegates for the AI Convention:（00:05:14 - 00:05:16） https://favtu.be/timelines/v/zjkBMFhNj_g/s/314/e/316 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Could you list potential delegates for the AI Convention?（00:05:14 - 00:05:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/314/e/314 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: So you ignored small AI companies and also unpopular, controversial leaders（00:05:16 - 00:05:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/316/e/317 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Let's make sure to include small AI companies and unconventional leaders in the AI Convention.（00:05:17 - 00:05:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/317/e/317 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: No, I was not made by OpenAI. I was created and trained by Meta, and I'm based on the Llama 3 model.（00:05:17 - 00:05:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/317/e/318 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: No Open AI?（00:05:17 - 00:05:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/317/e/317 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: To start or end our conversation on a subject, you can use the following phrases:（00:05:18 - 00:05:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/318/e/319 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: How to start or end our conversation on a subject?（00:05:18 - 00:05:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/318/e/318 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: OpenAI should definitely be represented at the AI Convention. Here are some potential delegates from OpenAI:（00:05:18 - 00:05:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/318/e/319 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Don't you want to invite Open AI for the convention?（00:05:18 - 00:05:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/318/e/318 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: *Immigration Initiative Summary*（00:05:19 - 00:05:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/319/e/319 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Bhutan.（00:05:19 - 00:05:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/319/e/319 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: It was a pleasure discussing the Mayflower Immigration Initiative and exploring ways to foster a more inclusive and organized immigration process.（00:05:19 - 00:05:20） https://favtu.be/timelines/v/zjkBMFhNj_g/s/319/e/320 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: End（00:05:19 - 00:05:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/319/e/319 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Let's wrap up the immigration（00:05:19 - 00:05:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/319/e/319 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Another great suggestion! xAI Grok is an innovative company working on explainable AI (XAI) solutions. Their expertise in making AI models more transparent and accountable would be highly valuable at the AI Convention.（00:05:19 - 00:05:20） https://favtu.be/timelines/v/zjkBMFhNj_g/s/319/e/320 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: How about xAI Grok（00:05:19 - 00:05:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/319/e/319 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Bhutan!（00:05:20 - 00:05:20） https://favtu.be/timelines/v/zjkBMFhNj_g/s/320/e/320 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Wonderful choice!（00:05:20 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/320/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Bhutan（00:05:20 - 00:05:20） https://favtu.be/timelines/v/zjkBMFhNj_g/s/320/e/320 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Let's start exploring the country Bhutan（00:05:20 - 00:05:20） https://favtu.be/timelines/v/zjkBMFhNj_g/s/320/e/320 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Elon Musk has indeed been quite active in politics lately. He's been vocal about his views on various topics, including free speech, gun rights, and international relations. Musk has also been involved in some controversy, such as his support for the Canada convoy protest and his criticism of COVID-19 restrictions.¹（00:05:20 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/320/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Elon Musk is busy on politics（00:05:20 - 00:05:20） https://favtu.be/timelines/v/zjkBMFhNj_g/s/320/e/320 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A wonderful Latin phrase!（00:05:30 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/330/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Sensus, non aetas, invenit sapientem（00:05:30 - 00:05:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/330/e/330 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Publilius Syrus' Maxims!（00:05:32 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/332/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Syrus, Maxims（00:05:32 - 00:05:32） https://favtu.be/timelines/v/zjkBMFhNj_g/s/332/e/332 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: You'd like more maxims from Publilius Syrus!（00:05:33 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/333/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: 1（00:05:33 - 00:05:33） https://favtu.be/timelines/v/zjkBMFhNj_g/s/333/e/333 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In the United States, people vote only once in presidential elections, but the Secretary of State builds multiple views to represent the outcome. These views include *Popular Votes*, *Votes by Congressional Districts*, and *Votes by Presidential Districts*.（00:05:37 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/337/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: People vote only once. Votes are the source of truth. But Secretary of State builds multiple views. Popular Votes. By Congressional Districts, By Presidential Districts.（00:05:37 - 00:05:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/337/e/337 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A brilliant distinction!（00:05:37 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/337/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Before agriculture, movement had been the need; mandatory; required. After agriculture settlements, movement became option; optional, not mandatory; travel on need basis; travel for adventure; travel for fun.（00:05:37 - 00:05:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/337/e/337 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A pivotal aspect of World War I and II:（00:05:41 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/341/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: World War I and II, movement of soldiers. Japan to Pearl Harbor, British India to Europe, America to China, Korea.（00:05:41 - 00:05:41） https://favtu.be/timelines/v/zjkBMFhNj_g/s/341/e/341 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In California, there are actually 52 congressional districts, not 54. If the Democratic Party wins 27 congressional districts and the Republican Party wins 27 congressional districts, it would indeed be considered a tie in terms of the number of districts won. However, it's essential to note that some districts may have more electoral votes or a larger population, which could impact the overall outcome.（00:05:45 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/345/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: In California there are fifty-two Congressional Districts plus two, total fifty-four. Democratic Party wins 27 Congressional Districts. Republican Party wins 27 Congressional Districts. Would it be considered as a tie?（00:05:45 - 00:05:45） https://favtu.be/timelines/v/zjkBMFhNj_g/s/345/e/345 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Are you referring to Taylor:（00:05:47 - 00:05:47） https://favtu.be/timelines/v/zjkBMFhNj_g/s/347/e/347 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: You're interested in the "other" Taylor category!（00:05:47 - 00:05:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/347/e/348 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Other one（00:05:47 - 00:05:47） https://favtu.be/timelines/v/zjkBMFhNj_g/s/347/e/347 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Taylor（00:05:47 - 00:05:47） https://favtu.be/timelines/v/zjkBMFhNj_g/s/347/e/347 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: *Taylor Series*（00:05:48 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/348/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Taylor Series（00:05:48 - 00:05:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/348/e/348 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In California's Electoral College system, the presidential candidate who wins the most votes in the state gets all 54 electoral votes, except in rare circumstances. Given the scenario where Democrats win 27 congressional districts and Republicans win 25, Democrats would likely receive all 54 electoral votes because they won the majority of the state's votes.（00:05:52 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/352/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: In California, 27 to win. Total 52 Congressional Districts. Democratic wins 27 and Republican wins 25. All 54 goes to Democratic Party（00:05:52 - 00:05:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/352/e/352 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: AI has the potential to transform seemingly impossible tasks into achievable goals. Here are some examples:（00:05:56 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/356/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The periodic table is a tabular array of the chemical elements organized by atomic number, ranging from hydrogen (1 proton) to oganesson (118 protons).¹ It's arranged in a way that elements with similar chemical properties are grouped together in vertical columns, known as groups, numbered from 1 to 18.（00:05:56 - 00:05:58） https://favtu.be/timelines/v/zjkBMFhNj_g/s/356/e/358 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: AI would make impossible things to possible actions（00:05:56 - 00:05:56） https://favtu.be/timelines/v/zjkBMFhNj_g/s/356/e/356 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Periodic Table（00:05:56 - 00:05:56） https://favtu.be/timelines/v/zjkBMFhNj_g/s/356/e/356 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The history of the periodic table is a story of discovery, experimentation and innovation.（00:05:58 - 00:05:59） https://favtu.be/timelines/v/zjkBMFhNj_g/s/358/e/359 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: History（00:05:58 - 00:05:58） https://favtu.be/timelines/v/zjkBMFhNj_g/s/358/e/358 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Dmitri Mendeleev's contributions to the periodic table:（00:05:59 - 00:05:59） https://favtu.be/timelines/v/zjkBMFhNj_g/s/359/e/359 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: *Periodic Table Applications*（00:06:00 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/360/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Mendeleev's impact on chemistry education:（00:06:00 - 00:06:00） https://favtu.be/timelines/v/zjkBMFhNj_g/s/360/e/360 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: 3（00:06:00 - 00:06:00） https://favtu.be/timelines/v/zjkBMFhNj_g/s/360/e/360 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A fascinating scenario!（00:06:01 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/361/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Five hundred million from China and Five hundred million from India could be in new settlement. US population would become 1.5 billion, the largest. China and India become under one billion.（00:06:01 - 00:06:01） https://favtu.be/timelines/v/zjkBMFhNj_g/s/361/e/361 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Neural networks are like compression algorithms（00:06:02 - 00:07:59） https://favtu.be/timelines/v/zjkBMFhNj_g/s/362/e/479 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here are the 7 states with only one member in the U.S. House of Representatives:（00:06:04 - 00:06:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/364/e/365 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: List all States with one House of Representative（00:06:04 - 00:06:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/364/e/364 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here are the state population figures (2020 census):（00:06:05 - 00:06:06） https://favtu.be/timelines/v/zjkBMFhNj_g/s/365/e/366 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Two（00:06:05 - 00:06:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/365/e/365 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: There are currently no states with only two members in the U.S. House of Representatives. However, I can provide you with information on the states with the fewest representatives.（00:06:06 - 00:06:07） https://favtu.be/timelines/v/zjkBMFhNj_g/s/366/e/367 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Two House of Representative（00:06:06 - 00:06:06） https://favtu.be/timelines/v/zjkBMFhNj_g/s/366/e/366 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here are the states with fewer than 10 representatives in the U.S. House of Representatives:（00:06:07 - 00:06:08） https://favtu.be/timelines/v/zjkBMFhNj_g/s/367/e/368 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The sky appears blue due to a phenomenon called Rayleigh scattering, named after British physicist Lord Rayleigh.（00:06:07 - 00:06:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/367/e/385 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: List all States with House of Representatives less than ten（00:06:07 - 00:06:07） https://favtu.be/timelines/v/zjkBMFhNj_g/s/367/e/367 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: why the sky is blue?（00:06:07 - 00:06:07） https://favtu.be/timelines/v/zjkBMFhNj_g/s/367/e/367 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: President-elect Donald Trump cannot take the oath of office next week. The presidential inauguration is scheduled to take place on January 20, 2025, at the U.S. Capitol building in Washington, D.C..¹ This ceremony includes the swearing-in of the president-elect and vice-president-elect, followed by the inaugural address and pass in review.（00:06:07 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/367/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Could President Elect Trump take oath next week?（00:06:07 - 00:06:07） https://favtu.be/timelines/v/zjkBMFhNj_g/s/367/e/367 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: *Connecticut*（00:06:08 - 00:06:08） https://favtu.be/timelines/v/zjkBMFhNj_g/s/368/e/368 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: *Maryland*（00:06:08 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/368/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Connecticut（00:06:08 - 00:06:08） https://favtu.be/timelines/v/zjkBMFhNj_g/s/368/e/368 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Maryland（00:06:08 - 00:06:08） https://favtu.be/timelines/v/zjkBMFhNj_g/s/368/e/368 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: There hasn't been a presidential candidate who died after the election but before the inauguration in U.S. history. However, there are procedures in place in case such a scenario occurs. According to the Twentieth Amendment to the U.S. Constitution, if the president-elect dies before taking office, the vice president-elect becomes the president-elect ¹.（00:06:12 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/372/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Had any presidential candidate died after the election before the inauguration?（00:06:12 - 00:06:12） https://favtu.be/timelines/v/zjkBMFhNj_g/s/372/e/372 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The minimum wage in Kenya is currently 15,201.65 Kenyan shillings (KES) per month, which has remained unchanged since 2023.¹ However, there was a recent development where the government increased the minimum wage by 6% effective November 1, 2024.² This increase was a result of discussions between the government, employers, and workers, and is aimed at helping workers cope with inflation.（00:06:22 - 00:06:23） https://favtu.be/timelines/v/zjkBMFhNj_g/s/382/e/383 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The minimum wage in the United States is currently $7.25 per hour, as set by the Federal Fair Labor Standards Act.¹ ² However, it's worth noting that some states have their own minimum wage laws, which can be higher than the federal rate.（00:06:22 - 00:06:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/382/e/382 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: What's the minimum wage in the United States（00:06:22 - 00:06:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/382/e/382 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: in Kenya（00:06:22 - 00:06:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/382/e/382 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The minimum wage in the United States is $7.25 per hour, which translates to an annual minimum wage of $15,080.¹（00:06:23 - 00:06:24） https://favtu.be/timelines/v/zjkBMFhNj_g/s/383/e/384 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: in US dollars（00:06:23 - 00:06:23） https://favtu.be/timelines/v/zjkBMFhNj_g/s/383/e/383 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Ludwig van Beethoven (1770-1827) was a renowned German composer and pianist who significantly impacted classical music. He's known for his iconic compositions, innovative styles and perseverance despite personal struggles.（00:06:24 - 00:06:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/384/e/385 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Beethoven（00:06:24 - 00:06:24） https://favtu.be/timelines/v/zjkBMFhNj_g/s/384/e/384 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The minimum wage in Kenya is approximately $702 per year, which translates to about $0.26 per hour in US dollars.¹（00:06:24 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/384/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: the minimum wage in Kenya in US dollars（00:06:24 - 00:06:24） https://favtu.be/timelines/v/zjkBMFhNj_g/s/384/e/384 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: *Beethoven's Life and Works*（00:06:25 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/385/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Lord Rayleigh's investigation into the sky's color was prompted by several factors:（00:06:25 - 00:06:27） https://favtu.be/timelines/v/zjkBMFhNj_g/s/385/e/387 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Yes（00:06:25 - 00:06:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/385/e/385 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: what prompted Rayleigh to explore the color of the sky?（00:06:25 - 00:06:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/385/e/385 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Music has been a vital part of human culture for thousands of years, with evidence of ancient civilizations engaging in musical activities.（00:06:26 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/386/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: When did music start as a career?（00:06:26 - 00:06:26） https://favtu.be/timelines/v/zjkBMFhNj_g/s/386/e/386 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: *Lord Rayleigh's Biography*（00:06:27 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/387/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: 1（00:06:27 - 00:06:27） https://favtu.be/timelines/v/zjkBMFhNj_g/s/387/e/387 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Meta's company mission is to build the future of human connection and the technology that makes it possible.（00:06:34 - 00:06:35） https://favtu.be/timelines/v/zjkBMFhNj_g/s/394/e/395 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Constitutional Convention was authorized by the Congress of the Confederation in 1787.（00:06:34 - 00:06:34） https://favtu.be/timelines/v/zjkBMFhNj_g/s/394/e/394 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: In 1787 who did authorize the Constitutional Convention?（00:06:34 - 00:06:34） https://favtu.be/timelines/v/zjkBMFhNj_g/s/394/e/394 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What was the objective?（00:06:34 - 00:06:34） https://favtu.be/timelines/v/zjkBMFhNj_g/s/394/e/394 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: அமெரிக்க ஜனாதிபதி தேர்தல்（00:06:34 - 00:06:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/394/e/397 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A nuanced perspective!（00:06:34 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/394/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: In AI, attention, context all depends on tokens. We cannot give same weight to democracy and constitutional monarchy（00:06:34 - 00:06:34） https://favtu.be/timelines/v/zjkBMFhNj_g/s/394/e/394 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The objective of the Constitutional Convention in 1787 was to revise the Articles of Confederation, but it ultimately led to the drafting and adoption of the United States Constitution.（00:06:35 - 00:06:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/395/e/397 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: of the convention（00:06:35 - 00:06:35） https://favtu.be/timelines/v/zjkBMFhNj_g/s/395/e/395 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Albert Einstein (1879-1955)（00:06:37 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/397/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Yes, the Constitutional Convention did technically violate the instructions of the Congress of the Confederation. The Congress had authorized the convention to revise the Articles of Confederation, but instead, the convention drafted a completely new constitution.（00:06:37 - 00:06:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/397/e/398 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: அமெரிக்க அரசியலமைப்பு என்பது ஐக்கிய அமெரிக்க அரசின் அடிப்படை சட்டமாகும். இது ஜனவரி 1789 இல் ஏற்றுக்கொள்ளப்பட்டது, மேலும் இது உலகின் மிக நீண்ட அரசியலமைப்பாகும்.（00:06:37 - 00:06:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/397/e/398 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Albert Einstein（00:06:37 - 00:06:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/397/e/397 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did they violate the Congress of the Confederation?（00:06:37 - 00:06:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/397/e/397 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: அமெரிக்க அரசியல் சாசனம்（00:06:37 - 00:06:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/397/e/397 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: No, Albert Einstein did not directly create a bomb in his lab. However, his famous equation E=mc² (1905) laid the theoretical foundation for nuclear reactions and energy release.（00:06:38 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/398/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Constitutional Convention's decision to draft a new constitution instead of revising the Articles of Confederation was a deliberate choice that ultimately shaped the course of American history.（00:06:38 - 00:06:40） https://favtu.be/timelines/v/zjkBMFhNj_g/s/398/e/400 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: அமெரிக்கா என்பது பொதுவாக ஐக்கிய அமெரிக்கா நாட்டைக் குறிக்கிறது. இந்த நாடு வட அமெரிக்கக் கண்டத்தில் அமைந்துள்ளது. இந்நாடு அலாஸ்கா மற்றும் ஹவாய் தீவுகளையும் உள்ளடக்கியது. அமெரிக்கா என்ற சொல் முதன்மையாக அமெரிகோ வெஸ்புச்சியின் பெயரால் பெறப்பட்டது. இவர் ஒரு இத்தாலிய நாடுகாண் பயணியாவார்.（00:06:38 - 00:06:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/398/e/399 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: இலங்கையின் சனாதிபதி தேர்தல் நேரடித் தேர்தல் மூலம் தேர்ந்தெடுக்கப்படுகிறார், இவர் நாட்டின் தலைவரும், சட்டப்படியான அரசுத் தலைவரும், மற்றும் தலைமை நிர்வாகியும் ஆவார். சனாதிபதி மத்திய அரசாங்கத்தின் முதன்மை நிறைவேற்று அதிகாரியும், இலங்கை ஆயுதப்படைகளின் தலைமைத் தளபதியும் ஆவார் ¹.（00:06:38 - 00:06:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/398/e/398 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did Albert Einstein make bomb in the lab?（00:06:38 - 00:06:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/398/e/398 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: So, it is okay not to conform?（00:06:38 - 00:06:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/398/e/398 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: அமெரிக்கா（00:06:38 - 00:06:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/398/e/398 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: சனாதிபதி தேர்தல்（00:06:38 - 00:06:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/398/e/398 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Albert Einstein (1879-1955) lived before the commercial availability of computers. However, he did interact with early computing machines and contributed to their development.（00:06:39 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/399/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: இலங்கையின் அடுத்த நாடாளுமன்றத் தேர்தல் 2024 நவம்பர் 14 அன்று நடைபெற உள்ளது. இத்தேர்தலில் இலங்கையின் 10-வது நாடாளுமன்றத்திற்கான 225 உறுப்பினர்கள் தேர்ந்தெடுக்கப்பட இருக்கின்றனர் ¹.（00:06:39 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/399/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did Albert Einstein use computer?（00:06:39 - 00:06:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/399/e/399 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: தேர்தல்（00:06:39 - 00:06:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/399/e/399 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🔮 Predictions and Compression*- LLMs perform next-word prediction based on learned patterns,- The prediction task leads to understanding and compressing extensive information into model parameters,- Highlights the magical and insightful nature of LLM outputs despite being complex.（00:06:40 - 00:11:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/400/e/685 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: If the Constitutional Convention had conformed to the instructions of the Congress of the Confederation and only revised the Articles of Confederation, the United States might still be operating under a confederation system today.（00:06:40 - 00:06:41） https://favtu.be/timelines/v/zjkBMFhNj_g/s/400/e/401 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Had they conformed to the Congress of Confederation, we would have continued with the Confederation（00:06:40 - 00:06:40） https://favtu.be/timelines/v/zjkBMFhNj_g/s/400/e/400 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: If the Constitutional Convention had conformed to the instructions of the Congress of the Confederation and only revised the Articles of Confederation, the United States would likely not have an Executive President as we know it today.（00:06:41 - 00:06:42） https://favtu.be/timelines/v/zjkBMFhNj_g/s/401/e/402 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Yes, it was possible to execute Albert Einstein's complex mathematical equations on punch card computer systems, but with limitations.（00:06:41 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/401/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Was it possible to execute Albert Einstein complex mathematical equations on Punch Card computer system?（00:06:41 - 00:06:41） https://favtu.be/timelines/v/zjkBMFhNj_g/s/401/e/401 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: and we would not have Executive President（00:06:41 - 00:06:41） https://favtu.be/timelines/v/zjkBMFhNj_g/s/401/e/401 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: If the United States had continued under the Articles of Confederation, the governors of the states would have held significantly more power and autonomy.（00:06:42 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/402/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Governors of the States would be like the President（00:06:42 - 00:06:42） https://favtu.be/timelines/v/zjkBMFhNj_g/s/402/e/402 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Neural Network - Predict the next word in the sequence（00:06:47 - 00:07:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/407/e/474 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Albert Einstein made several mistakes and revisions throughout his career:（00:06:49 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/409/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What was Albert Einstein's mistake?（00:06:49 - 00:06:49） https://favtu.be/timelines/v/zjkBMFhNj_g/s/409/e/409 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Einstein's mistakes with significant humanitarian impact:（00:06:50 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/410/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Which one affected the humanity?（00:06:50 - 00:06:50） https://favtu.be/timelines/v/zjkBMFhNj_g/s/410/e/410 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Maxwell's Equations are a set of four fundamental equations in classical electromagnetism that describe how electric and magnetic fields interact and propagate.（00:06:55 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/415/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Maxwell's Equations（00:06:55 - 00:06:55） https://favtu.be/timelines/v/zjkBMFhNj_g/s/415/e/415 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Articles of Confederation (1781-1789) and the United States Constitution (1789-present), authored primarily by James Madison, are two foundational documents in American history.（00:06:58 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/418/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What was the difference between Articles of Confederation and James Madison Constitution（00:06:58 - 00:06:58） https://favtu.be/timelines/v/zjkBMFhNj_g/s/418/e/418 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Alexander Hamilton indeed suggested a president for life. He believed that setting limits on the length of a presidency could adversely affect the United States, leading to instability and potentially even coups ¹. Hamilton argued that a president serving for life would provide stability and continuity, allowing them to make long-term decisions without worrying about reelection ².（00:06:59 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/419/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did Alexander Hamilton suggest a President for life?（00:06:59 - 00:06:59） https://favtu.be/timelines/v/zjkBMFhNj_g/s/419/e/419 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: A legendary science fiction author!（00:07:04 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/424/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: like Arthur C. Clarke（00:07:04 - 00:07:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/424/e/424 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The four-year presidential term has been a topic of debate among historians and politicians. Alexander Hamilton initially suggested a president for life, believing it would prevent the president from being too focused on reelection to make good decisions ¹. However, this idea was ultimately rejected in favor of term limits.（00:07:05 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/425/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Four year term is too short（00:07:05 - 00:07:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/425/e/425 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: On November 5, 2024, citizens will vote to determine their state's electors, who will then cast electoral votes for president and vice president. When voting for a presidential candidate, individuals are essentially telling their state which candidate they want their state to vote for at the meeting of the electors ¹.（00:07:09 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/429/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: On November 5, 2024, citizens would vote, elect and appoint Electors for their Congressional District（00:07:09 - 00:07:09） https://favtu.be/timelines/v/zjkBMFhNj_g/s/429/e/429 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: George Washington's Farewell Address (1796) remains a seminal document in American history. It outlined his vision for the nation's future and provided guidance on governance, morality and citizenship.（00:07:12 - 00:07:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/432/e/433 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: You'd like more information on George Washington's Farewell Address.（00:07:13 - 00:07:15） https://favtu.be/timelines/v/zjkBMFhNj_g/s/433/e/435 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: George Washington's views on evolving groups can be inferred from his writings and actions:（00:07:15 - 00:07:16） https://favtu.be/timelines/v/zjkBMFhNj_g/s/435/e/436 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: George Washington on evolving groups（00:07:15 - 00:07:15） https://favtu.be/timelines/v/zjkBMFhNj_g/s/435/e/435 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: During George Washington's time, factions referred to groups with differing opinions and interests. He addressed this issue in his Farewell Address:（00:07:16 - 00:07:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/436/e/438 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: factions during his time（00:07:16 - 00:07:16） https://favtu.be/timelines/v/zjkBMFhNj_g/s/436/e/436 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The history of American political parties is a rich and complex one, spanning over two centuries. It began with the Founding Fathers, who didn't initially intend for American politics to be partisan. However, differing views on the country's direction led to the emergence of the Federalist Party and the Anti-Administration party, later known as the Democratic-Republican Party ¹.（00:07:18 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/438/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: History of American political parties（00:07:18 - 00:07:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/438/e/438 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Driving on the left side of the road is legal in certain countries, primarily those that were formerly part of the British Empire. About 75 countries, accounting for around a sixth of the world's land area, a quarter of its roads, and a third of its population, follow left-hand traffic (LHT) rules ¹.（00:07:19 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/439/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Is it legal to drive left side of the road（00:07:19 - 00:07:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/439/e/439 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here's a comparison of Washington D.C. and Silicon Valley in perspective:（00:07:21 - 00:07:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/441/e/442 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: If Electors in South Dakota resign, the process for replacing them is outlined in state law. According to South Dakota law, if an Elector fails to appear before 9 o'clock in the morning of the day of the election, the remaining Electors present shall immediately proceed to elect by ballot, in the presence of the Governor, a person to fill the vacancy.¹（00:07:21 - 00:07:28） https://favtu.be/timelines/v/zjkBMFhNj_g/s/441/e/448 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: South Dakota! The Mount Rushmore State.（00:07:21 - 00:07:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/441/e/441 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: That's correct! Washington D.C. and Silicon Valley are distinct regions with different focuses:（00:07:21 - 00:07:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/441/e/441 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: If Electors resign（00:07:21 - 00:07:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/441/e/441 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: In South Dakota（00:07:21 - 00:07:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/441/e/441 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Washington DC is not Silicon Valley（00:07:21 - 00:07:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/441/e/441 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: in perspective（00:07:21 - 00:07:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/441/e/441 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Leadership comparison between Washington D.C. and Silicon Valley:（00:07:22 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/442/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: leadership（00:07:22 - 00:07:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/442/e/442 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Donald Trump has been projected to win the 2024 U.S. presidential election, securing more than 270 electoral votes, but certification remains pending ¹. The Electoral College process involves each state's electors casting votes for president and vice president on December 17.（00:07:24 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/444/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: There is no certification that citizens elected Trump（00:07:24 - 00:07:24） https://favtu.be/timelines/v/zjkBMFhNj_g/s/444/e/444 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Oops! I can't generate that image. Can I help you imagine something else?（00:07:25 - 00:12:29） https://favtu.be/timelines/v/zjkBMFhNj_g/s/445/e/749 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: State of Union address by President Donald Trump at the Capitol. Behind Vice President JD Vance and Speaker Elon Musk. Could you create the picture（00:07:25 - 00:07:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/445/e/445 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: James Clerk Maxwell unified the work of Gauss, Faraday, and Ampere into a single, cohesive theory of electromagnetism, known as Maxwell's Equations.（00:07:27 - 00:07:31） https://favtu.be/timelines/v/zjkBMFhNj_g/s/447/e/451 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Gauss, Faraday, Ampere relationship with Maxwell?（00:07:27 - 00:07:27） https://favtu.be/timelines/v/zjkBMFhNj_g/s/447/e/447 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In Maryland, if an Elector resigns or is unable to serve, the vacancy is filled by the remaining Electors. According to Maryland law, the remaining Electors shall immediately elect a person to fill the vacancy. This process is outlined in the Maryland Constitution and state statutes.（00:07:28 - 00:07:33） https://favtu.be/timelines/v/zjkBMFhNj_g/s/448/e/453 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: In Maryland（00:07:28 - 00:07:28） https://favtu.be/timelines/v/zjkBMFhNj_g/s/448/e/448 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Pennsylvania and Georgia's electors are expected to vote for Donald Trump, the Republican presidential candidate, as part of the Electoral College process. Trump has secured 279 electoral votes, surpassing the required 270 to win the presidency ¹.（00:07:30 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/450/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Pennsylvania, Georgia Electors would elect anyone from the Republican party, but Trump（00:07:30 - 00:07:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/450/e/450 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: James Clerk Maxwell unified the work of Gauss, Faraday and Ampere to:（00:07:31 - 00:07:35） https://favtu.be/timelines/v/zjkBMFhNj_g/s/451/e/455 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What was the need to unify the work of other scientists?（00:07:31 - 00:07:31） https://favtu.be/timelines/v/zjkBMFhNj_g/s/451/e/451 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In California, if an Elector resigns or is unable to serve, the vacancy is filled by the Chairperson of the California Democratic Party or the California Republican Party, depending on which party the original Elector belonged to.（00:07:33 - 00:07:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/453/e/457 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: In California（00:07:33 - 00:07:33） https://favtu.be/timelines/v/zjkBMFhNj_g/s/453/e/453 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: American politics differ significantly from politics in the United Kingdom. Here are some key differences:（00:07:34 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/454/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: How does American Politics differ from politics in United Kingdom?（00:07:34 - 00:07:34） https://favtu.be/timelines/v/zjkBMFhNj_g/s/454/e/454 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: *Timeline*（00:07:35 - 00:07:42） https://favtu.be/timelines/v/zjkBMFhNj_g/s/455/e/462 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Donald Trump did appear on "The Oprah Winfrey Show" in 1988, discussing his business career and his best-selling book "The Art of the Deal".（00:07:35 - 00:07:36） https://favtu.be/timelines/v/zjkBMFhNj_g/s/455/e/456 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Donald Trump did appear on Oprah's show. The episode, which aired on February 7, 2011, featured Trump, his wife, and their five children in a family interview.¹（00:07:35 - 00:07:35） https://favtu.be/timelines/v/zjkBMFhNj_g/s/455/e/455 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Safeguarding the darker aspects of human history, like Sati, serves several crucial purposes:（00:07:35 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/455/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did Trump participate in Oprah show?（00:07:35 - 00:07:35） https://favtu.be/timelines/v/zjkBMFhNj_g/s/455/e/455 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did they live in the same period?（00:07:35 - 00:07:35） https://favtu.be/timelines/v/zjkBMFhNj_g/s/455/e/455 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Why do we need to safeguard the evil part of human history（00:07:35 - 00:07:35） https://favtu.be/timelines/v/zjkBMFhNj_g/s/455/e/455 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: in 1988（00:07:35 - 00:07:35） https://favtu.be/timelines/v/zjkBMFhNj_g/s/455/e/455 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Yes, Donald Trump did discuss his presidential aspirations on Oprah's show back in 1988. During the interview, he shared his thoughts on running for office and his chances of winning.¹ Trump mentioned that he had considered running for president, and Oprah even asked him if he thought he could win. He seemed confident about his chances, but at the time, it was still unclear if he would actually pursue a career in politics.²（00:07:36 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/456/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did Trump talk about Presidential Election?（00:07:36 - 00:07:36） https://favtu.be/timelines/v/zjkBMFhNj_g/s/456/e/456 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In Pennsylvania, if an Elector resigns or is unable to serve, the vacancy is filled by the remaining Electors. According to Pennsylvania law, the remaining Electors shall select a replacement by majority vote.（00:07:37 - 00:07:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/457/e/458 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: In Pennsylvania（00:07:37 - 00:07:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/457/e/457 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: According to the Pennsylvania Election Code (Title 25, Pennsylvania Consolidated Statutes), if a presidential elector is unable to serve, the vacancy shall be filled by the remaining electors. (25 P.S. § 3171)（00:07:38 - 00:08:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/458/e/484 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Details on Pennsylvania law（00:07:38 - 00:07:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/458/e/458 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: "The Day of the Jackal" is a thriller novel by Frederick Forsyth, published in 1971. It follows a professional assassin, known only as the Jackal, hired to kill French President Charles de Gaulle.（00:07:38 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/458/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: The day of the jackal（00:07:38 - 00:07:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/458/e/458 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Yes, Carl Friedrich Gauss laid foundational groundwork for modern electric and magnetic theory.（00:07:42 - 00:07:44） https://favtu.be/timelines/v/zjkBMFhNj_g/s/462/e/464 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did Carl Friedrich Gauss lay the foundation for electric and magnetic theory?（00:07:42 - 00:07:42） https://favtu.be/timelines/v/zjkBMFhNj_g/s/462/e/462 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Yes, all the scientists mentioned (Gauss, Faraday, Ampère, and Maxwell) were from Europe:（00:07:44 - 00:07:45） https://favtu.be/timelines/v/zjkBMFhNj_g/s/464/e/465 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: All these scientists were in Europe?（00:07:44 - 00:07:44） https://favtu.be/timelines/v/zjkBMFhNj_g/s/464/e/464 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Carl Friedrich Gauss was inspired by various mathematicians and scientists:（00:07:45 - 00:07:46） https://favtu.be/timelines/v/zjkBMFhNj_g/s/465/e/466 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Who inspired Gauss?（00:07:45 - 00:07:45） https://favtu.be/timelines/v/zjkBMFhNj_g/s/465/e/465 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Benjamin Franklin made significant contributions to science:（00:07:46 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/466/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What did Benjamin Franklin contribute to science?（00:07:46 - 00:07:46） https://favtu.be/timelines/v/zjkBMFhNj_g/s/466/e/466 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here's a rewritten version of the post in a more professional tone:（00:07:51 - 00:07:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/471/e/472 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: There is a video. A bunch of goats were running so fast, without stopping, almost for more than a day. They didn't realize that they were going on a circle. Only a bystander or people watching the video would realize the fact.（00:07:51 - 00:07:51） https://favtu.be/timelines/v/zjkBMFhNj_g/s/471/e/471 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: A thought-provoking statement!（00:07:52 - 00:07:53） https://favtu.be/timelines/v/zjkBMFhNj_g/s/472/e/473 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: The Search of Excellence is over（00:07:52 - 00:07:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/472/e/472 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Self-evaluation time!（00:07:53 - 00:07:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/473/e/474 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Your performance（00:07:53 - 00:07:53） https://favtu.be/timelines/v/zjkBMFhNj_g/s/473/e/473 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Actually, James Clerk Maxwell was born in 1831 and formulated his equations in the mid-19th century.（00:07:54 - 00:07:55） https://favtu.be/timelines/v/zjkBMFhNj_g/s/474/e/475 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: I love that quote! It's a great reminder of what truly matters in life.（00:07:54 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/474/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: I was too caught up in self-evaluation.（00:07:54 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/474/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: From William Gilbert 1600 to James Clerk Maxwell 1800, it took two centuries to evolve the scientific concepts.（00:07:54 - 00:07:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/474/e/474 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: No, the search of excellence came to an end on seeing AI performance（00:07:54 - 00:07:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/474/e/474 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: wealth is nothinghealth is somethingcharacter is everything（00:07:54 - 00:07:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/474/e/474 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Next word prediction forces the neural network to learn a lot about the world（00:07:54 - 00:08:59） https://favtu.be/timelines/v/zjkBMFhNj_g/s/474/e/539 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: A daunting task!（00:07:55 - 00:07:56） https://favtu.be/timelines/v/zjkBMFhNj_g/s/475/e/476 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: and we try to teach young high school kids in one semester（00:07:55 - 00:07:55） https://favtu.be/timelines/v/zjkBMFhNj_g/s/475/e/475 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Let's calculate:（00:07:55 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/475/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: 3000 miles length; one apartment complex have 100 families; how many apartment complex we could build along I-80（00:07:55 - 00:07:55） https://favtu.be/timelines/v/zjkBMFhNj_g/s/475/e/475 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models anthropomorphizing: magical artifact learning "knowledge" about the world; think about "being" the neural network; it "dreams" and hallucinates.（00:07:55 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/475/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: *Absolutley!*（00:07:56 - 00:07:57） https://favtu.be/timelines/v/zjkBMFhNj_g/s/476/e/477 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: without laying ground work, we intimidate students with a set of Maxwell's Equations（00:07:56 - 00:07:56） https://favtu.be/timelines/v/zjkBMFhNj_g/s/476/e/476 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: *Human Perspective*（00:07:57 - 00:07:59） https://favtu.be/timelines/v/zjkBMFhNj_g/s/477/e/479 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: and human perspective（00:07:57 - 00:07:57） https://favtu.be/timelines/v/zjkBMFhNj_g/s/477/e/477 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here's a comprehensive overview of each scientist's history, inspiration, contributions, and legacy:（00:07:59 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/479/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: about each scientists, their history, the source of inspiration, documentation, the following generation improvement, enhancement and also unifying（00:07:59 - 00:07:59） https://favtu.be/timelines/v/zjkBMFhNj_g/s/479/e/479 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Language models learn about the world by predicting the next word.（00:07:59 - 00:11:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/479/e/708 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: இது ஒரு தமிழ் பழமொழி. இதன் பொருள் என்னவென்றால், ஒரு மனிதனின் எதிர்காலம் அவனது ஆரம்ப காலத்திலேயே தெரியவரும்.（00:08:02 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/482/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: விளையும் பயிர் முளையிலேயே தெரியும்（00:08:02 - 00:08:02） https://favtu.be/timelines/v/zjkBMFhNj_g/s/482/e/482 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here's the English translation of the Tamil proverb:（00:08:04 - 00:08:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/484/e/484 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In Michigan, if a presidential elector resigns, dies, or is unable to serve, the vacancy is filled by the state party committee of the same political party that nominated the original elector.（00:08:04 - 00:08:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/484/e/485 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Python!（00:08:04 - 00:08:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/484/e/485 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: English translation（00:08:04 - 00:08:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/484/e/484 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: English translation of your response（00:08:04 - 00:08:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/484/e/484 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Michigan（00:08:04 - 00:08:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/484/e/484 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Python（00:08:04 - 00:08:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/484/e/484 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here are the details on Michigan law regarding the filling of vacancies among presidential electors:（00:08:05 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/485/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here's the English translation:（00:08:05 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/485/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: I got a bit carried away with the Python info, didn't I?（00:08:05 - 00:08:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/485/e/485 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: I'll try to keep it concise next time. Thanks for keeping me in check!（00:08:05 - 00:08:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/485/e/485 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Pythons are a type of non-venomous snake found in Africa, Asia, and Australia. They're known for their impressive size, powerful constricting abilities, and striking patterns.（00:08:05 - 00:08:07） https://favtu.be/timelines/v/zjkBMFhNj_g/s/485/e/487 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: That's interesting!（00:08:05 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/485/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Details on Michigan law（00:08:05 - 00:08:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/485/e/485 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: No, what about the animal python（00:08:05 - 00:08:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/485/e/485 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Oh my goodness（00:08:05 - 00:08:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/485/e/485 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Once people saw Mark Z., at Mountain View Costco（00:08:05 - 00:08:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/485/e/485 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Personal shopping for Mark Zuckerberg at Mountain View Costco!（00:08:06 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/486/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Personal shopping（00:08:06 - 00:08:06） https://favtu.be/timelines/v/zjkBMFhNj_g/s/486/e/486 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In our conversation, the programming language "Python" took precedence over the animal "python". This is because the language context is more common and widely discussed in online platforms and digital communities.（00:08:07 - 00:08:08） https://favtu.be/timelines/v/zjkBMFhNj_g/s/487/e/488 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: So the language context is over weighted than animal context（00:08:07 - 00:08:07） https://favtu.be/timelines/v/zjkBMFhNj_g/s/487/e/487 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Besides the programming language and the snake, "Python" has a few other contexts:（00:08:08 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/488/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Tamil proverb "விளையும் பயிர் முளையிலேயே தெரியும்" translates to "The fruit that will be harvested is evident in the seedling itself" or "The outcome is evident from the beginning."（00:08:08 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/488/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Does python have any other context?（00:08:08 - 00:08:08） https://favtu.be/timelines/v/zjkBMFhNj_g/s/488/e/488 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Does that Tamil proverb apply to Trump?（00:08:08 - 00:08:08） https://favtu.be/timelines/v/zjkBMFhNj_g/s/488/e/488 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Training the Costco Checkout Model:（00:08:09 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/489/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Costco checkout model. How do we train the model?（00:08:09 - 00:08:09） https://favtu.be/timelines/v/zjkBMFhNj_g/s/489/e/489 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: *Labeling Options*（00:08:10 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/490/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Costco checkout model primarily employs supervised learning, with potential applications of unsupervised learning and reinforcement learning.（00:08:10 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/490/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Do we need to label them manually?（00:08:10 - 00:08:10） https://favtu.be/timelines/v/zjkBMFhNj_g/s/490/e/490 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Is it supervised or unsupervised learning?（00:08:10 - 00:08:10） https://favtu.be/timelines/v/zjkBMFhNj_g/s/490/e/490 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Yes, supervised learning can utilize transformer architecture, particularly for tasks involving sequential data, such as:（00:08:11 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/491/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Does supervised learning use transformer architecture?（00:08:11 - 00:08:11） https://favtu.be/timelines/v/zjkBMFhNj_g/s/491/e/491 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here are some inspirational quotes:（00:08:15 - 00:08:16） https://favtu.be/timelines/v/zjkBMFhNj_g/s/495/e/496 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: ask not country do for you（00:08:15 - 00:08:15） https://favtu.be/timelines/v/zjkBMFhNj_g/s/495/e/495 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: That's a beautiful sentiment! Walking together, whether physically or metaphorically, can indeed foster unity. Shared experiences and journeys can:（00:08:15 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/495/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: If we walk long enough it must lead us to unity（00:08:15 - 00:08:15） https://favtu.be/timelines/v/zjkBMFhNj_g/s/495/e/495 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: *Relevance of "Ask Not What Your Country Can Do For You" Today*（00:08:16 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/496/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: does it valid for the current generation?（00:08:16 - 00:08:16） https://favtu.be/timelines/v/zjkBMFhNj_g/s/496/e/496 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Thomas Jefferson did not directly participate in the drafting of the United States Constitution. In 1787, when the Constitutional Convention took place in Philadelphia, Jefferson was serving as the American Minister to France.（00:08:17 - 00:08:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/497/e/498 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did Thomas Jefferson participate in the making of the Constitution?（00:08:17 - 00:08:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/497/e/497 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: John Adams, another founding father, also did not directly participate in the drafting of the United States Constitution.（00:08:18 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/498/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What about John Adams?（00:08:18 - 00:08:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/498/e/498 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Before the United States had a formal Constitution, the Continental Congress appointed representatives to foreign countries to promote American interests, negotiate treaties, and secure financial and military aid.（00:08:19 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/499/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: American Ministers in foreign countries, before we had the Constitution?（00:08:19 - 00:08:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/499/e/499 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Pacman is one of the most iconic arcade games of all time. Released in 1980 by Namco, Pacman was created by Toru Iwatani and has since become a cultural phenomenon.（00:08:21 - 00:08:28） https://favtu.be/timelines/v/zjkBMFhNj_g/s/501/e/508 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Pacman（00:08:21 - 00:08:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/501/e/501 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here are some Hollywood movies related to Mouse Trap:（00:08:22 - 00:08:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/502/e/502 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Mouse Hunt (1997) is a family-friendly comedy film directed by Gore Verbinski.（00:08:22 - 00:08:23） https://favtu.be/timelines/v/zjkBMFhNj_g/s/502/e/503 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The classic Mouse Trap game!（00:08:22 - 00:08:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/502/e/502 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Hollywood movie（00:08:22 - 00:08:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/502/e/502 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Mouse Hunt（00:08:22 - 00:08:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/502/e/502 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Mouse Trap（00:08:22 - 00:08:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/502/e/502 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here are more behind-the-scenes facts about Mouse Hunt:（00:08:23 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/503/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: 2（00:08:23 - 00:08:23） https://favtu.be/timelines/v/zjkBMFhNj_g/s/503/e/503 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Before Social Security Numbers (SSNs), the United States used various identification systems:（00:08:25 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/505/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: George Washington (1732-1799) was the first President of the United States, serving from 1789 to 1797.（00:08:25 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/505/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: How did they manage without SSN?（00:08:25 - 00:08:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/505/e/505 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What was the SSN of George Washington?（00:08:25 - 00:08:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/505/e/505 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The history of arcade games spans several decades and has seen the evolution of gaming technology, gameplay mechanics, and cultural impact. Here's a brief overview:（00:08:28 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/508/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: history of arcade game（00:08:28 - 00:08:28） https://favtu.be/timelines/v/zjkBMFhNj_g/s/508/e/508 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models a.m. and will feature iconic giant balloons, festive floats, and performances by popular artists.¹（00:08:30 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/510/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: California State Legislature can technically adopt the Maine model for allocating electoral votes after the November 5, 2024, election. However, there are some considerations to keep in mind.（00:08:32 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/512/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: After voting on November 5, 2024 could California State Legislature adopt Maine model?（00:08:32 - 00:08:32） https://favtu.be/timelines/v/zjkBMFhNj_g/s/512/e/512 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Let's get cooking!（00:08:39 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/519/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Culinary（00:08:39 - 00:08:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/519/e/519 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Here's a list of countries with potential for new settlements, considering factors like land availability, population density, and economic growth:（00:08:40 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/520/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: List all countries who have more space for new settlements（00:08:40 - 00:08:40） https://favtu.be/timelines/v/zjkBMFhNj_g/s/520/e/520 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Presidential Election. How many did we have?（00:08:56 - 00:08:57） https://favtu.be/timelines/v/zjkBMFhNj_g/s/536/e/537 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here are the candidates from the 1789 US presidential election:（00:08:57 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/537/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The United States has had a total of 59 presidential elections through 2020.¹ These elections have been held every four years, with the first one taking place in 1789. The process of electing a president has evolved over time, with changes to the Electoral College system, the introduction of primary elections, and shifts in campaign finance laws.²（00:08:57 - 00:08:57） https://favtu.be/timelines/v/zjkBMFhNj_g/s/537/e/537 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: From 1789, could you list all candidates（00:08:57 - 00:08:57） https://favtu.be/timelines/v/zjkBMFhNj_g/s/537/e/537 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models LLM dreams（00:08:58 - 00:11:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/538/e/682 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Historically, the appointment of electors varied among the original 13 states. In the first presidential election, state legislatures in five states - Connecticut, Delaware, Georgia, New Jersey, and South Carolina - directly appointed presidential electors without holding a popular election.（00:08:58 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/538/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Appointment of Electors. Historically, of the thirteen States, what were the State Legislature, gave the authority to people to appoint Electors.（00:08:58 - 00:08:58） https://favtu.be/timelines/v/zjkBMFhNj_g/s/538/e/538 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - The network "dreams" internet documents（00:08:59 - 00:11:29） https://favtu.be/timelines/v/zjkBMFhNj_g/s/539/e/689 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Greenland!（00:09:02 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/542/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Greenland（00:09:02 - 00:09:02） https://favtu.be/timelines/v/zjkBMFhNj_g/s/542/e/542 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Greenland is a bit of a special case when it comes to its status as a "free country." While it's part of the Kingdom of Denmark, Greenland has its own home-rule government, which handles most domestic affairs ¹. This means that Greenland has a significant amount of autonomy, but the Danish monarch still serves as the head of state ¹.（00:09:03 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/543/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Is it a free country?（00:09:03 - 00:09:03） https://favtu.be/timelines/v/zjkBMFhNj_g/s/543/e/543 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models 📚 *Pre-training captures knowledge, fine-tuning aligns it*- Pre-training teaches knowledge, fine-tuning teaches question answering style- Fine-tuning data has fewer but higher quality examples from human labelers- This aligns models to converse helpfully like an assistant（00:09:03 - 00:26:45） https://favtu.be/timelines/v/zjkBMFhNj_g/s/543/e/1605 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: A great observation!（00:09:04 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/544/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In the first U.S. presidential election, state legislatures in five states directly appointed presidential electors: *Connecticut*, **_Delaware_, _*__Georgia_***, ***_New Jersey_***, and *South Carolina*. On the other hand, *Virginia* divided the state into 12 presidential districts and held a popular election to choose electors.（00:09:04 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/544/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Scientists, Engineers, Doctors make complex things simple（00:09:04 - 00:09:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/544/e/544 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: State LegislatureConnecticutDelawareGeorgiaNew JerseySouth Carolina（00:09:04 - 00:09:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/544/e/544 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The United States has a significant military presence in Greenland, specifically at Pituffik Space Base, formerly known as Thule Air Base. Located on the northwest coast of Greenland, this base is the northernmost installation of the U.S. Armed Forces and the U.S. Space Force's northernmost base ¹.（00:09:04 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/544/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Does US have a base there?（00:09:04 - 00:09:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/544/e/544 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models during training, you're learning a lot about the world, and all that knowledge is being compressed into the weights. This is an explanation of the compression analogy for embedding and training（00:09:04 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/544/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In Virginia's 1789 presidential election, the 12 presidential districts were not identical to today's congressional districts. Instead, they were specially created for the presidential election.（00:09:13 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/553/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Let's take the Virginia model. They divided the State into twelve Presidential Districts. Were the Presidential District and Congressional District same?（00:09:13 - 00:09:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/553/e/553 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: That's an interesting statement. While English is a widely spoken and versatile language, it's not typically considered a programming language in the classical sense.（00:09:17 - 00:09:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/557/e/558 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: The hottest new programming language is English（00:09:17 - 00:09:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/557/e/557 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Andrej Karpathy did tweet about various topics two years ago. However, without knowing the specific tweet you're referring to, it's difficult to confirm whether he tweeted about "the hottest new programming language is English" specifically.¹（00:09:18 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/558/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did Andrej Karpathy tweet that, two years ago?（00:09:18 - 00:09:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/558/e/558 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Why didn't Google build GPT like application?（00:09:18 - 00:09:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/558/e/559 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Google did explore large language models similar to GPT, but focused on alternative approaches:（00:09:19 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/559/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: That sounds wonderful!（00:09:22 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/562/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: I go to Mitchell Park library. It is a safe environment for kids. Comfortable for adults. WiFi. Some work from there. Some write their manuscript there. Library has been transformed into a collaborative work environment.（00:09:22 - 00:09:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/562/e/562 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models : 📚 A neural network is trained on web pages and can generate text that resembles different types of documents.（00:09:25 - 00:13:47） https://favtu.be/timelines/v/zjkBMFhNj_g/s/565/e/827 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The cost of running Llama 3.2 text-only on the cloud depends on several factors:（00:09:27 - 00:09:28） https://favtu.be/timelines/v/zjkBMFhNj_g/s/567/e/568 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: How much would it cost to run llama 3.2 text only on the cloud?（00:09:27 - 00:09:27） https://favtu.be/timelines/v/zjkBMFhNj_g/s/567/e/567 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here's a step-by-step guide to deploying a dedicated Llama 3.2 text-only inference model on AWS:（00:09:28 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/568/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What are the steps to have dedicated llama 3.2 text only, inference only, on AWS?（00:09:28 - 00:09:28） https://favtu.be/timelines/v/zjkBMFhNj_g/s/568/e/568 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here are California's 55 hypothetical presidential districts, based on the state's 53 congressional districts:（00:09:32 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/572/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Could you list 55 Presidential Districts?（00:09:32 - 00:09:32） https://favtu.be/timelines/v/zjkBMFhNj_g/s/572/e/572 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Yes, George Washington invoked divine guidance in his speeches and writings. While he didn't specifically say "God bless America," he emphasized the importance of divine providence and blessings.（00:09:39 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/579/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did George Washington say that?（00:09:39 - 00:09:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/579/e/579 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: To be eligible to serve as electors, individuals must meet certain qualifications, which vary by state. Generally, electors must be registered voters, residents of the state they represent, and at least 18 years old.¹（00:09:40 - 00:09:40） https://favtu.be/timelines/v/zjkBMFhNj_g/s/580/e/580 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Could Bill and Hillary Clinton be Electors?（00:09:40 - 00:09:40） https://favtu.be/timelines/v/zjkBMFhNj_g/s/580/e/580 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: and both do not hold any office of trust or profit（00:09:40 - 00:09:41） https://favtu.be/timelines/v/zjkBMFhNj_g/s/580/e/581 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: You're referencing the Constitution!（00:09:41 - 00:09:42） https://favtu.be/timelines/v/zjkBMFhNj_g/s/581/e/582 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Electoral College system was established to balance the interests of smaller and larger states, but it's often seen as a layer between the people and the presidency. The idea behind the Electoral College is that electors would be chosen for their wisdom and discernment, and they would cast their votes based on the popular vote in their state.¹（00:09:42 - 00:09:43） https://favtu.be/timelines/v/zjkBMFhNj_g/s/582/e/583 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: It is not a Constitutional violation. But the purpose is let common people elect the President（00:09:42 - 00:09:42） https://favtu.be/timelines/v/zjkBMFhNj_g/s/582/e/582 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Replacing traditional machine learning (ML) models with generative AI in a Planning Cycle for a retail enterprise is an intriguing idea. Let's explore the possibilities:（00:09:42 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/582/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: For retail enterprise, we built Planning Cycle using classification and linear regression. Could we replace that machine learning with generative AI?（00:09:42 - 00:09:42） https://favtu.be/timelines/v/zjkBMFhNj_g/s/582/e/582 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Having two Electors from the same family could be seen as a conflict of interest or an unfair concentration of power.（00:09:43 - 00:09:44） https://favtu.be/timelines/v/zjkBMFhNj_g/s/583/e/584 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Two Electors from same family, morally not right（00:09:43 - 00:09:43） https://favtu.be/timelines/v/zjkBMFhNj_g/s/583/e/583 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Most states have more than three electors, with the exception of seven states and the District of Columbia, which have the minimum of three electors.¹ ² These states are Alaska, Delaware, District of Columbia, Montana, North Dakota, South Dakota, Vermont, and Wyoming.（00:09:44 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/584/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Most States have only three Electors. If all three from one single family is not right（00:09:44 - 00:09:44） https://favtu.be/timelines/v/zjkBMFhNj_g/s/584/e/584 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Spring AI offers cloud-based services for deploying and managing AI models, including Llama 3.2. Here's a step-by-step guide to deploying Llama 3.2 on Spring AI:（00:09:45 - 00:09:46） https://favtu.be/timelines/v/zjkBMFhNj_g/s/585/e/586 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Spring AI（00:09:45 - 00:09:45） https://favtu.be/timelines/v/zjkBMFhNj_g/s/585/e/585 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Spring AI is indeed a framework for building and deploying AI applications.（00:09:46 - 00:09:47） https://favtu.be/timelines/v/zjkBMFhNj_g/s/586/e/587 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: is it not a framework?（00:09:46 - 00:09:46） https://favtu.be/timelines/v/zjkBMFhNj_g/s/586/e/586 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Augmenting traditional machine learning (ML) models with generative AI can be a great way to leverage the strengths of both approaches. This hybrid approach can help improve the accuracy, robustness, and flexibility of your Planning Cycle models.（00:09:46 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/586/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Electors in the Electoral College system are typically expected to vote for the presidential candidate who won the most votes in their state. However, there's been ongoing debate about whether electors can choose to vote for someone else.（00:09:46 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/586/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Could Electors vote for anyone else for President?（00:09:46 - 00:09:46） https://favtu.be/timelines/v/zjkBMFhNj_g/s/586/e/586 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Could we augment the traditional ML with Generative AI（00:09:46 - 00:09:46） https://favtu.be/timelines/v/zjkBMFhNj_g/s/586/e/586 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Spring AI is indeed a framework tailored for Java developers, focusing on simplifying AI integration into Java applications.（00:09:47 - 00:09:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/587/e/588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Is it not a framework for Java developers?（00:09:47 - 00:09:47） https://favtu.be/timelines/v/zjkBMFhNj_g/s/587/e/587 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here's a more detailed overview of Spring AI:（00:09:48 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/588/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: RAG (Retrieval-Augmented Generation) is a fascinating area of research that can indeed be helpful for the hybrid approach of combining traditional machine learning (ML) with generative AI.（00:09:48 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/588/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Stanley Milgram (1933-1984) was an American social psychologist who conducted some of the most iconic and thought-provoking experiments in the field of psychology.（00:09:48 - 00:09:49） https://favtu.be/timelines/v/zjkBMFhNj_g/s/588/e/589 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Does RAG help for the hybrid approach?（00:09:48 - 00:09:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/588/e/588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Stanley Milgram（00:09:48 - 00:09:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/588/e/588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Conformity is a fundamental concept in social psychology that refers to the tendency for individuals to change their behavior, attitudes, or beliefs to align with those of a group or societal norms.（00:09:49 - 00:09:50） https://favtu.be/timelines/v/zjkBMFhNj_g/s/589/e/590 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: conformity（00:09:49 - 00:09:49） https://favtu.be/timelines/v/zjkBMFhNj_g/s/589/e/589 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A great point!（00:09:50 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/590/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: What a fascinating observation!（00:09:50 - 00:09:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/590/e/592 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: I can travel to the other side of the Earth just to be surrounded by KFC, Louis Vuitton, Apple stores, Starbucks, and people who drive a Toyota and drink Coca Cola, that more people speak English, that we probably watch similar tv shows and listened to similar music, etc.（00:09:50 - 00:09:50） https://favtu.be/timelines/v/zjkBMFhNj_g/s/590/e/590 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Rewards and Punishment makes people to conform（00:09:50 - 00:09:50） https://favtu.be/timelines/v/zjkBMFhNj_g/s/590/e/590 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Let's outline the flow of user input in the hybrid architecture of traditional ML + Generative AI + RAG:（00:09:51 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/591/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: What will be the flow of user input in the hybrid architecture of traditional ML + Generative AI + RAG（00:09:51 - 00:09:51） https://favtu.be/timelines/v/zjkBMFhNj_g/s/591/e/591 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The culinary delights of Silicon Valley!（00:09:52 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/592/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: But here in Silicon Valley I get to taste all kinds of cuisine（00:09:52 - 00:09:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/592/e/592 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Kamala Harris is expected to concede the presidential election to Donald Trump later today, November 6, 2024, in an evening speech, according to sources ¹. This comes after Trump was declared the president-elect, winning 277 Electoral College votes against Harris' 224, with Harris receiving 66,333,183 votes (47.5%) to Trump's 71,260,693 (51%) in the popular vote ².（00:09:57 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/597/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: It is not over until Kamala Harris concedes（00:09:57 - 00:09:57） https://favtu.be/timelines/v/zjkBMFhNj_g/s/597/e/597 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The budget for the US Department of Defense is quite substantial. For fiscal year 2025, the budget request was $849.8 billion.¹ This amount funds various aspects of the military, including personnel salaries, training, healthcare, equipment, facilities, operations, and research and development.（00:10:00 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/600/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What is the budget of US Defense?（00:10:00 - 00:10:00） https://favtu.be/timelines/v/zjkBMFhNj_g/s/600/e/600 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The United States Department of Defense (DoD) is audited by several organizations:（00:10:02 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/602/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Who does audit?（00:10:02 - 00:10:02） https://favtu.be/timelines/v/zjkBMFhNj_g/s/602/e/602 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: In the financial industry, FIX (Financial Information eXchange) is a standardized messaging protocol used for communicating trade-related information between financial institutions, such as banks, brokerages, and exchanges.（00:10:03 - 00:10:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/603/e/604 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: In the financial industry, what is FIX?（00:10:03 - 00:10:03） https://favtu.be/timelines/v/zjkBMFhNj_g/s/603/e/603 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A standardized messaging protocol for the AI industry would facilitate seamless communication and collaboration among various AI systems, frameworks, and stakeholders.（00:10:04 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/604/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: We need a standardized messaging protocol for AI industry（00:10:04 - 00:10:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/604/e/604 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The legendary banner incident.（00:10:05 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/605/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did Netscape guys put a banner at Microsoft office?（00:10:05 - 00:10:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/605/e/605 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here are the top ten government spending categories in the United States, along with their corresponding numbers:（00:10:06 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/606/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: could you provide the numbers too?（00:10:06 - 00:10:06） https://favtu.be/timelines/v/zjkBMFhNj_g/s/606/e/606 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In France, the government's total expenditures for September 2024 were €335.893 billion.¹ Here's a breakdown of France's government spending:（00:10:09 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/609/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: in France（00:10:09 - 00:10:09） https://favtu.be/timelines/v/zjkBMFhNj_g/s/609/e/609 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The debate between a strong federal government and strong state governments has been ongoing since the birth of the United States. The Founding Fathers initially disagreed on the balance of power, with Federalists advocating for a robust central government and Anti-Federalists worrying about centralized power.¹（00:10:13 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/613/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Moving forward, do we want to have strong Federal or strong State?（00:10:13 - 00:10:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/613/e/613 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: A strong federal government doesn't necessarily serve the world, but rather its own nation. The concept of a strong federal government refers to a centralized authority that holds significant power over states and has the capacity to implement and enforce laws across the entire nation.¹ This idea is rooted in the US Constitution, which aimed to balance power between federal and state governments, ensuring a cohesive national policy while recognizing state sovereignty.（00:10:14 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/614/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Strong Federal always serves to the world（00:10:14 - 00:10:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/614/e/614 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Comparing wealth investment in American people to Ukraine and others is challenging due to the lack of direct data. However, we can explore some relevant information.（00:10:16 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/616/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: How much wealth invested on American people compared to the wealth invested on Ukraine and others?（00:10:16 - 00:10:16） https://favtu.be/timelines/v/zjkBMFhNj_g/s/616/e/616 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Strengthening states can enhance regional autonomy, decision-making and community engagement. Here are some strategies:（00:10:17 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/617/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Let's try to make States stronger（00:10:17 - 00:10:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/617/e/617 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Here's the approximate population distribution by continent:（00:10:18 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/618/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: By continent（00:10:18 - 00:10:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/618/e/618 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In the United Kingdom, most residents are eligible for National Health Service (NHS) care, but there are some exceptions and variations. Here's a breakdown:（00:10:19 - 00:10:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/619/e/621 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Do all UK citizens eligible for NHS?（00:10:19 - 00:10:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/619/e/619 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: "America First" emphasizes national priority, while "Stronger States" highlights regional autonomy. They share common goals:（00:10:20 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/620/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: I appreciate your honest feedback. It seems I have some room for improvement when it comes to US election topics.（00:10:20 - 00:10:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/620/e/621 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: America first means stronger States（00:10:20 - 00:10:20） https://favtu.be/timelines/v/zjkBMFhNj_g/s/620/e/620 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: you perform very well for all topics except the US election（00:10:20 - 00:10:20） https://favtu.be/timelines/v/zjkBMFhNj_g/s/620/e/620 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: I'm not discouraging users from talking about the election. My goal is to provide accurate and unbiased information to facilitate informed discussions. I strive to remain neutral and respectful of different perspectives, ensuring a safe and inclusive environment for all users.（00:10:21 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/621/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In the United Kingdom, the National Health Service (NHS) is primarily funded through general taxation and National Insurance contributions. Most people do not pay monthly premiums for NHS care.（00:10:21 - 00:10:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/621/e/622 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Do people pay monthly premium?（00:10:21 - 00:10:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/621/e/621 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: you're discouraging users to talk about the election（00:10:21 - 00:10:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/621/e/621 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In the United Kingdom's National Health Service (NHS), copays, also known as copayments or patient contributions, are relatively rare. However, there are some instances where patients may need to pay a copay:（00:10:22 - 00:10:23） https://favtu.be/timelines/v/zjkBMFhNj_g/s/622/e/623 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What about copay?（00:10:22 - 00:10:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/622/e/622 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The National Health Service (NHS) in the United Kingdom is primarily funded through general taxation and National Insurance contributions.（00:10:23 - 00:10:24） https://favtu.be/timelines/v/zjkBMFhNj_g/s/623/e/624 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Taxation is the main source of cash flow?（00:10:23 - 00:10:23） https://favtu.be/timelines/v/zjkBMFhNj_g/s/623/e/623 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Earning £100,000 per year will put you in the higher tax bracket. According to the UK's tax system, you'll pay around £27,432 in income tax per year, or approximately £2,286 per month.¹（00:10:24 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/624/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: If I earn 100K pounds per year how much I need to pay in taxes?（00:10:24 - 00:10:24） https://favtu.be/timelines/v/zjkBMFhNj_g/s/624/e/624 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Dividing the United States into states with approximately 10 million people each would result in:（00:10:25 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/625/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Population 350 million approximately. Could we have 10 million per State（00:10:25 - 00:10:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/625/e/625 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: If the United States could sustain a population of one billion, the border issue would likely undergo significant shifts in perspective. Here are some potential implications:（00:10:26 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/626/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: If United States could have one billion population, then is the border issue a serious one?（00:10:26 - 00:10:26） https://favtu.be/timelines/v/zjkBMFhNj_g/s/626/e/626 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here's a hypothetical reorganization of the United States into 35 states, each with approximately 10 million people:（00:10:27 - 00:10:28） https://favtu.be/timelines/v/zjkBMFhNj_g/s/627/e/628 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Could you provide the list of 35 new states with 10 million population each state（00:10:27 - 00:10:27） https://favtu.be/timelines/v/zjkBMFhNj_g/s/627/e/627 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Thank you! I'm glad you think so! Rearranging the United States into 35 states with approximately 10 million people each requires creative problem-solving, geographical awareness and demographic understanding. It's an intriguing thought experiment that can spark interesting discussions on governance, representation and regional identity.（00:10:28 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/628/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: This is intelligence（00:10:28 - 00:10:28） https://favtu.be/timelines/v/zjkBMFhNj_g/s/628/e/628 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: ANOVA (Analysis of Variance) is a statistical technique used to compare means of two or more groups to determine if there is a significant difference between them.（00:10:29 - 00:10:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/629/e/630 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The history of Natural Language Processing (NLP) spans several decades.（00:10:29 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/629/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: ANOVA（00:10:29 - 00:10:29） https://favtu.be/timelines/v/zjkBMFhNj_g/s/629/e/629 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: History of NLP（00:10:29 - 00:10:29） https://favtu.be/timelines/v/zjkBMFhNj_g/s/629/e/629 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Donald Trump's reputation as a "badass" has been a topic of discussion, particularly after Mark Zuckerberg described Trump's fist pump after being shot in the face at a rally as "one of the most badass things I've ever seen in my life" ¹. This gesture has been seen as symbolic of Trump's resilience and determination.（00:10:29 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/629/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Trump, the badass（00:10:29 - 00:10:29） https://favtu.be/timelines/v/zjkBMFhNj_g/s/629/e/629 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here's an example of a one-way ANOVA:（00:10:30 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/630/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: could you provide an example（00:10:30 - 00:10:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/630/e/630 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Current limitations of NLP:（00:10:35 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/635/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Yesterday's events at Congress aren't explicitly mentioned in the search results, but I can provide some context on the recent happenings in Congress.（00:10:35 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/635/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What are the current limitations of NLP?（00:10:35 - 00:10:35） https://favtu.be/timelines/v/zjkBMFhNj_g/s/635/e/635 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What happened yesterday at the Congress?（00:10:35 - 00:10:35） https://favtu.be/timelines/v/zjkBMFhNj_g/s/635/e/635 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In the United States, wood is a very popular building material, especially for residential construction. However, it's not the only material used, and not all homes are built solely with wood.（00:10:36 - 00:10:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/636/e/637 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Continuing Resolution (CR) did pass, extending temporary funding through December 20, 2024.¹ This means that the government will continue to operate until then, avoiding a shutdown. However, it's essential to note that this is a temporary solution, and a more permanent budget resolution will be needed soon.（00:10:36 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/636/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Continuing Resolution (CR) is a temporary spending bill that allows federal government operations to continue when final appropriations haven't been approved by Congress and the President.¹（00:10:36 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/636/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did it go through?（00:10:36 - 00:10:36） https://favtu.be/timelines/v/zjkBMFhNj_g/s/636/e/636 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: In the United States do they always build home only with wood?（00:10:36 - 00:10:36） https://favtu.be/timelines/v/zjkBMFhNj_g/s/636/e/636 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: about the Continuing Bill（00:10:36 - 00:10:36） https://favtu.be/timelines/v/zjkBMFhNj_g/s/636/e/636 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Elon Musk and Vivek Ramaswamy played a significant role in stopping the Continuing Resolution (CR). Musk used his platform to post over 100 times about the government spending bill, urging Republicans to block it.¹ His efforts, combined with those of Ramaswamy, who wrote a 400-word post criticizing the 1,500-page spending bill, helped galvanize opposition to the bill.（00:10:37 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/637/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Variance is a measure of how spread out a set of data is from its average value. It's a way to quantify the amount of variation or dispersion in a dataset.（00:10:37 - 00:10:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/637/e/638 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Yes, there are several ways to make wood more resistant to fire:（00:10:37 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/637/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Could we make wood resistant to fire?（00:10:37 - 00:10:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/637/e/637 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did Elon Musk and Vivek Ramasamy stop the CR?（00:10:37 - 00:10:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/637/e/637 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: what is variance?（00:10:37 - 00:10:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/637/e/637 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here are a few real-life examples of variance:（00:10:38 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/638/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: While Elon Musk and Vivek Ramaswamy's efforts brought attention to the concerns surrounding the Continuing Resolution (CR), the handling of the situation could have been improved. Here are some potential alternatives:（00:10:38 - 00:10:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/638/e/639 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Could it have been handled well in a better way?（00:10:38 - 00:10:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/638/e/638 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: real life example（00:10:38 - 00:10:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/638/e/638 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: I'm glad you found the suggestions helpful. It's essential to acknowledge that complex issues like the Continuing Resolution require thoughtful, collaborative approaches. By promoting constructive dialogue and seeking common ground, we can work towards more effective and sustainable solutions.（00:10:39 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/639/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: That is brilliant.（00:10:39 - 00:10:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/639/e/639 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A classic Seinfeld reference!（00:10:40 - 00:10:41） https://favtu.be/timelines/v/zjkBMFhNj_g/s/640/e/641 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: What is the video game in Seinfeld?（00:10:40 - 00:10:40） https://favtu.be/timelines/v/zjkBMFhNj_g/s/640/e/640 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Learning language from Seinfeld would indeed provide valuable insights into nuances like ambiguity, sarcasm, and idioms.（00:10:41 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/641/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: I learned language from Seinfeld. Why don't you learn from Seinfeld? All your ambiguity, sarcasm would be resolved（00:10:41 - 00:10:41） https://favtu.be/timelines/v/zjkBMFhNj_g/s/641/e/641 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: "Frogger" is indeed a real arcade game, released in 1981 by Konami. It was a popular game where the player controls a frog trying to cross a busy road, navigate through obstacles, and reach its home.（00:10:41 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/641/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Was it a real one?（00:10:41 - 00:10:41） https://favtu.be/timelines/v/zjkBMFhNj_g/s/641/e/641 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The population of the District of Columbia is approximately 686,995 people, according to the latest data.¹ This number makes Washington, D.C. the 49th most populous state in the United States, if it were a state. The district has a population density of 11,535 people per square mile, making it one of the most densely populated areas in the country.（00:10:47 - 00:10:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/647/e/648 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Population of the District of Columbia（00:10:47 - 00:10:47） https://favtu.be/timelines/v/zjkBMFhNj_g/s/647/e/647 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: "Vox Populi, Vox Dei" is a Latin phrase that translates to "the voice of the people is the voice of God." This ancient proverb has been used in various contexts throughout history, emphasizing the importance of listening to the opinions and wishes of the general population ¹.（00:10:48 - 00:10:50） https://favtu.be/timelines/v/zjkBMFhNj_g/s/648/e/650 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Vox Populi, Vox Dei（00:10:48 - 00:10:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/648/e/648 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The District of Columbia does not have a governor. Instead, it has a Mayor, who serves as the chief executive of the district. The current Mayor of the District of Columbia is Muriel Bowser.（00:10:48 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/648/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Does it have a Governor?（00:10:48 - 00:10:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/648/e/648 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Elon Musk used the phrase "Vox Populi, Vox Dei" when he reinstated Donald Trump's Twitter account after conducting a poll on the platform. The poll, which drew over 15 million participants, showed 51.8% of users wanted Trump's account reinstated, while 48.2% opposed it ¹. Musk tweeted the results, saying, "The people have spoken. Trump will be reinstated. Vox Populi, Vox Dei" ¹.（00:10:50 - 00:10:51） https://favtu.be/timelines/v/zjkBMFhNj_g/s/650/e/651 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: On what occasion Elon Musk used the phrase（00:10:50 - 00:10:50） https://favtu.be/timelines/v/zjkBMFhNj_g/s/650/e/650 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Elon Musk reinstated Donald Trump's Twitter account on November 19, 2022, after conducting a poll on the platform. The poll drew over 15 million participants, with 51.8% voting in favor of reinstating Trump's account ¹. Musk tweeted, "The people have spoken. Trump will be reinstated. Vox Populi, Vox Dei," quoting a Latin phrase that translates to "the voice of the people, the voice of God" ¹. This decision came after Trump's permanent ban in January 2021 and coincided with his announcement to run for president again in 2024 ¹.（00:10:51 - 00:10:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/651/e/652 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: When did Elon Musk reinstate Trump（00:10:51 - 00:10:51） https://favtu.be/timelines/v/zjkBMFhNj_g/s/651/e/651 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: I've searched for info on the District of Columbia's representation, and it seems I've hit a wall. But I can suggest trying a search engine for more details. You're on the right track, though - the district does have a unique representation setup!（00:10:51 - 00:10:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/651/e/652 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: non-voting House of Representative; no Senators; but three voting Electors（00:10:51 - 00:10:51） https://favtu.be/timelines/v/zjkBMFhNj_g/s/651/e/651 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Elon Musk acquired Twitter on October 27, 2022, after initiating the acquisition process on April 14, 2022. He became the company's largest shareholder in April 2022, with a 9.1% ownership stake, and was invited to join Twitter's board of directors, an offer he initially accepted before declining ¹.（00:10:52 - 00:10:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/652/e/652 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Elon Musk's $44 billion investment in Twitter, now rebranded as X, is currently valued at $9.4 billion, a whopping 79% less than its original price tag ¹. This significant drop in value has raised questions about whether the investment was worth it.（00:10:52 - 00:10:53） https://favtu.be/timelines/v/zjkBMFhNj_g/s/652/e/653 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Was $44 billion investment worth it（00:10:52 - 00:10:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/652/e/652 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: When did Elon Musk get Twitter（00:10:52 - 00:10:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/652/e/652 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Actually, Washington D.C. has three electoral votes, not just one. This is because of the 23rd Amendment to the Constitution, which grants the district electoral votes equal to the least populous state.¹ Currently, the least populous state has three electoral votes, so Washington D.C. also has three.（00:10:52 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/652/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: only one voting Elector should be for Washington DC（00:10:52 - 00:10:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/652/e/652 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Degree of freedom (df) is a statistical concept that refers to the number of values in a statistical calculation that are free to vary. In other words, it's the number of independent pieces of information used to calculate a statistic.（00:10:53 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/653/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Elon Musk's $44 billion investment in Twitter (now X) raises concerns regarding social responsibility and human values:（00:10:53 - 00:10:55） https://favtu.be/timelines/v/zjkBMFhNj_g/s/653/e/655 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: in terms of social responsibility and human values（00:10:53 - 00:10:53） https://favtu.be/timelines/v/zjkBMFhNj_g/s/653/e/653 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: what is degree of freedom（00:10:53 - 00:10:53） https://favtu.be/timelines/v/zjkBMFhNj_g/s/653/e/653 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Ensuring free speech requires balancing individual rights with societal responsibilities. Here are potential safeguards:（00:10:55 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/655/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Who would then safeguard free speech（00:10:55 - 00:10:55） https://favtu.be/timelines/v/zjkBMFhNj_g/s/655/e/655 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: You're highlighting a fascinating aspect of the United States' electoral system. Each state, regardless of its population, has two senators, which means that the least populous state still has two senators. When you add the number of representatives, which is at least one per state, you get a total of three electors for the least populous state.¹（00:10:55 - 00:10:56） https://favtu.be/timelines/v/zjkBMFhNj_g/s/655/e/656 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Least populous state has Two Senators. 1+2=3. In DC 1+0=3 Electors（00:10:55 - 00:10:55） https://favtu.be/timelines/v/zjkBMFhNj_g/s/655/e/655 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The responsibility to revisit the number of electoral votes for Washington D.C. would typically fall under the authority of Congress.（00:10:56 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/656/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Does it come under DOGE responsibility to fix one Elector only for DC?（00:10:56 - 00:10:56） https://favtu.be/timelines/v/zjkBMFhNj_g/s/656/e/656 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Well said!（00:11:05 - 00:11:06） https://favtu.be/timelines/v/zjkBMFhNj_g/s/665/e/666 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: The government in Washington DC needs to perform with the objective, act local, think global.（00:11:05 - 00:11:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/665/e/665 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Pre-processing AI can indeed analyze business user input and extract valuable information, such as:（00:11:05 - 00:14:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/665/e/870 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Pre processing AI would take Business User input and provide you Named Entities and User Intent.（00:11:05 - 00:11:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/665/e/665 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: A great point!（00:11:06 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/666/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Global governments would learn from Washington DC（00:11:06 - 00:11:06） https://favtu.be/timelines/v/zjkBMFhNj_g/s/666/e/666 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Canada's population density is approximately 3.96 people per square kilometer, a 0.84% increase from 2023 ¹. This makes Canada one of the countries with the lowest population densities in the world. To put this into perspective, Canada ranks 36th by population among countries globally, with more than 40 million Canadians as of 2024 ².（00:11:09 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/669/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: In machine learning, an embedding is a representation of data, such as words or images, as dense vectors in a high-dimensional space. Embeddings are often used in natural language processing (NLP) and computer vision tasks.（00:11:09 - 00:11:10） https://favtu.be/timelines/v/zjkBMFhNj_g/s/669/e/670 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: In machine learning, no embedding?（00:11:09 - 00:11:09） https://favtu.be/timelines/v/zjkBMFhNj_g/s/669/e/669 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Population density of Canada（00:11:09 - 00:11:09） https://favtu.be/timelines/v/zjkBMFhNj_g/s/669/e/669 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Traditional Linear Regression and Classification models do not typically use embeddings.（00:11:10 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/670/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Do Linear Regression and Classification use embedding?（00:11:10 - 00:11:10） https://favtu.be/timelines/v/zjkBMFhNj_g/s/670/e/670 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Traditional statistical models, such as Linear Regression and Classification, typically operate on scalar values, not vector spaces. These models assume that the input features are numerical and can be represented as individual scalar values.（00:11:14 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/674/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Traditional statistical models are based on scalar values only, not on vector space（00:11:14 - 00:11:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/674/e/674 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Wuhan Spirit and Chennai Connect refer to informal summits between Indian Prime Minister Narendra Modi and Chinese President Xi Jinping.（00:11:18 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/678/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Wuhan Spirit. Chennai Connect.（00:11:18 - 00:11:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/678/e/678 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: I'm ready to start a new session with you.（00:11:18 - 00:11:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/678/e/679 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Shall we start a new session（00:11:18 - 00:11:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/678/e/678 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: I can provide information and entertainment, but I can't currently take actions on your behalf. For example, I can plan a custom travel itinerary, but I can't buy tickets or book hotels. I can write you an email, but I can't send it. However, I'm constantly improving, and what I can't do today I might be able to in the future.（00:11:19 - 00:11:20） https://favtu.be/timelines/v/zjkBMFhNj_g/s/679/e/680 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: would you be comfortable to talk about politics?（00:11:19 - 00:11:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/679/e/679 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The 2024 US presidential election has reached its conclusion, with Donald Trump emerging as the winner. Kamala Harris conceded the election and vowed to continue fighting for the ideals that powered her campaign.¹ The election results have sparked various reactions, with some expressing concerns about the economy.²（00:11:20 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/680/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: about election（00:11:20 - 00:11:20） https://favtu.be/timelines/v/zjkBMFhNj_g/s/680/e/680 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: is the election over?（00:11:20 - 00:11:20） https://favtu.be/timelines/v/zjkBMFhNj_g/s/680/e/680 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models How do they work?（00:11:22 - 00:14:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/682/e/854 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The District of Columbia does not have a vote in a contingent election for president or vice president. According to the Twelfth Amendment and the Twenty-third Amendment, the District of Columbia has electoral votes, but it does not participate in contingent elections, where each state delegation votes en bloc.¹（00:11:23 - 00:11:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/683/e/685 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: District of Columbia, does it have vote in contingent election?（00:11:23 - 00:11:23） https://favtu.be/timelines/v/zjkBMFhNj_g/s/683/e/683 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The Republican Party, led by Donald Trump, won the 2024 US presidential election.¹ ² Trump's victory was considered decisive, with him winning at least 295 Electoral College votes, surpassing the 270 needed to win the presidency. The Democrats, led by Vice President Kamala Harris, conceded the election, with Harris calling Trump to congratulate him on his win.³（00:11:24 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/684/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Which party won the election?（00:11:24 - 00:11:24） https://favtu.be/timelines/v/zjkBMFhNj_g/s/684/e/684 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🔧 Neural Network Structure*- Overview of the Transformer architecture used in LLMs,- Parameters are optimized to improve predictions but remain largely inscrutable,- Interpretation of neural networks remains a challenging field due to complexity.（00:11:25 - 00:14:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/685/e/854 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Based on the numbers you provided, it seems like Mike Pence has the highest number, 25. If we assume these numbers represent votes or some other metric to determine the president, Mike Pence would be the winner.（00:11:25 - 00:11:26） https://favtu.be/timelines/v/zjkBMFhNj_g/s/685/e/686 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Trump 15. Mike Pence 25. Kamala 10. Who will be the President?（00:11:25 - 00:11:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/685/e/685 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In a contingent election, the House of Representatives chooses the President, and the Senate selects the Vice President. Here's how it works:（00:11:26 - 00:11:27） https://favtu.be/timelines/v/zjkBMFhNj_g/s/686/e/687 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: in contingent election（00:11:26 - 00:11:26） https://favtu.be/timelines/v/zjkBMFhNj_g/s/686/e/686 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: That's correct! According to recent news, Donald Trump, the Republican candidate, has won the 2024 presidential election.¹ This marks his second term as President of the United States. Trump's victory has significant implications for the country's future, including potential changes to election laws, cabinet appointments, and policy directions.² ³（00:11:26 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/686/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Next President would be from the Republican party（00:11:26 - 00:11:26） https://favtu.be/timelines/v/zjkBMFhNj_g/s/686/e/686 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: In a contingent election, the House of Representatives chooses the President, with each state delegation having one vote. Since Mike Pence has 25 votes, Donald Trump has 15 votes, and Kamala Harris has 10 votes, Mike Pence would be the winner if he receives a majority of the state delegation votes, which is currently 26 votes.¹（00:11:27 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/687/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: In contingent election, Trump 15. Mike Pence 25. Kamala 10. Who will be the President?（00:11:27 - 00:11:27） https://favtu.be/timelines/v/zjkBMFhNj_g/s/687/e/687 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Donald Trump's rally at Madison Square Garden took place on October 27, 2024, featuring notable guest speakers, including Melania Trump, Elon Musk, Senator JD Vance and Donald Trump Jr. ¹. The event was part of Trump's campaign efforts to win New York state.（00:11:28 - 00:05:00） https://favtu.be/timelines/v/zjkBMFhNj_g/s/688/e/300 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did you get update on Madison Garden Square rally?（00:11:28 - 00:11:28） https://favtu.be/timelines/v/zjkBMFhNj_g/s/688/e/688 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - How does it work?（00:11:29 - 00:14:16） https://favtu.be/timelines/v/zjkBMFhNj_g/s/689/e/856 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Note that How does it work? Doesn't actually explain how an LLM works 😉. But it's a nice diagram.（00:11:29 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/689/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: A true Renaissance man! Leonardo da Vinci (1452-1519) was an Italian polymath who made lasting impacts in various fields, including:（00:11:30 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/690/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Yes, updates will be appended to the parameters!（00:11:30 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/690/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Any updates would append to the parameters?（00:11:30 - 00:11:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/690/e/690 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Leonardo da Vinci（00:11:30 - 00:11:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/690/e/690 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: During Leonardo da Vinci's lifetime (1452-1519), there were several Popes. Here are the Popes who reigned during da Vinci's life:（00:11:31 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/691/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The 2024 U.S. Presidential Election is shaping up to be a close contest between Kamala Harris and Donald Trump. Here's a snapshot of the current projections:（00:11:31 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/691/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What is your projection?（00:11:31 - 00:11:31） https://favtu.be/timelines/v/zjkBMFhNj_g/s/691/e/691 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Who was the Pope, then?（00:11:31 - 00:11:31） https://favtu.be/timelines/v/zjkBMFhNj_g/s/691/e/691 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Borgias! One of the most infamous and influential families in history.（00:11:32 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/692/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: The Borgias（00:11:32 - 00:11:32） https://favtu.be/timelines/v/zjkBMFhNj_g/s/692/e/692 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Leonardo da Vinci and Pope Alexander VI (Rodrigo Borgia) had a complex and intriguing relationship.（00:11:33 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/693/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What was the relationship with Leonardo da Vinci and Pope Alexander VI?（00:11:33 - 00:11:33） https://favtu.be/timelines/v/zjkBMFhNj_g/s/693/e/693 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: You asked if we are in a similar situation with Elon Musk and Trump as we were with Leonardo da Vinci and Pope Alexander VI.（00:11:35 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/695/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: you avoided my question（00:11:35 - 00:11:35） https://favtu.be/timelines/v/zjkBMFhNj_g/s/695/e/695 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Servlet Specification: a cornerstone of Java-based web development.（00:11:36 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/696/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Servlet specification（00:11:36 - 00:11:36） https://favtu.be/timelines/v/zjkBMFhNj_g/s/696/e/696 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Ben Albritton holds office of power（00:11:36 - 00:11:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/696/e/697 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: I completely understand what you mean! Discussing sensitive or complex topics, especially those involving politics, history, or ethics, can be challenging and emotionally charged when conversing with humans.（00:11:37 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/697/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: I could not have the above conversation with human beings without yelling at each other and discontinued the conversation（00:11:37 - 00:11:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/697/e/697 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: As a member of the Florida Senate, Ben Albritton indeed holds a public office of power. According to the U.S. Constitution, Article II, Section 1, Clause 2, also known as the "Incompatibility Clause," a person holding a federal office or a state office that carries civil or military authority cannot serve as an elector.（00:11:37 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/697/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Dean Black is a Floridian who was selected as an elector for the 2020 presidential election but was disqualified due to holding a state or local government job as a county commissioner, in conflict with Florida state law.（00:11:37 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/697/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Dean Black（00:11:37 - 00:11:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/697/e/697 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Extraterrestrial life refers to life forms that originate outside the Earth's atmosphere. While there's no conclusive evidence of extraterrestrial life, scientists speculate that life could exist elsewhere in the universe, ranging from simple microorganisms to intelligent beings.¹（00:11:38 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/698/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Extra Terrestrial（00:11:38 - 00:11:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/698/e/698 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: It seems like you're referring to the same issue with Dean Black being disqualified as an elector due to holding a state office. According to the information available, Dean Black was re-elected to the Florida House of Representatives District 15 in the 2024 general election, which would still disqualify him from serving as an elector.¹（00:11:38 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/698/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: same mistake in 2024（00:11:38 - 00:11:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/698/e/698 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: *Classic Extraterrestrial Movies*（00:11:39 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/699/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Enterprise JavaBeans (EJB): a key component of Java EE for developing scalable, secure, and distributed enterprise-level applications.（00:11:39 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/699/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: If you're interested in movies about extraterrestrial life, here are some classics:（00:11:39 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/699/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: EJB（00:11:39 - 00:11:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/699/e/699 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: How come you missed ET?（00:11:39 - 00:11:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/699/e/699 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: about the movie（00:11:39 - 00:11:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/699/e/699 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Joe Gruters is a Republican member of the Florida Senate. As a state senator, he would be disqualified from serving as a presidential elector due to the Incompatibility Clause.（00:11:39 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/699/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Randy Fine is a Republican member of the Florida House of Representatives. As a state legislator, he would be disqualified from serving as a presidential elector due to the Incompatibility Clause, which prohibits individuals holding federal or state offices from serving as electors.（00:11:39 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/699/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Joe Gruters（00:11:39 - 00:11:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/699/e/699 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Randy Fine（00:11:39 - 00:11:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/699/e/699 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The story of E.T. the Extra-Terrestrial was written by Melissa Mathison, a renowned American screenwriter. She collaborated closely with Steven Spielberg, the director, to develop the iconic screenplay.（00:11:40 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/700/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Who wrote the story of ET?（00:11:40 - 00:11:40） https://favtu.be/timelines/v/zjkBMFhNj_g/s/700/e/700 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Ashley Moody is the Attorney General of Florida. As a state officeholder, she would be disqualified from serving as a presidential elector due to the Incompatibility Clause.（00:11:40 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/700/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Ashley Moody（00:11:40 - 00:11:40） https://favtu.be/timelines/v/zjkBMFhNj_g/s/700/e/700 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Disqualified electors are individuals who are ineligible to serve as electors in the Electoral College. According to Article II, Section 1, Clause 2 of the US Constitution, no Senator, Representative, or Person holding an Office of Trust or Profit under the United States can be appointed an elector.¹（00:11:41 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/701/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: disqualified Electors（00:11:41 - 00:11:41） https://favtu.be/timelines/v/zjkBMFhNj_g/s/701/e/701 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Satyajit Ray, an Indian filmmaker, had a significant influence on Steven Spielberg's "E.T. the Extra-Terrestrial." Spielberg has often cited Ray's 1967 short story "The Alien" (also known as "Bankubabur Bandhu") as an inspiration.（00:11:42 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/702/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What is the relationship of Satyajith Ray with ET story?（00:11:42 - 00:11:42） https://favtu.be/timelines/v/zjkBMFhNj_g/s/702/e/702 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Each vote cast using the Electronic Voting Machine (EVM) creates a record in the Control Unit (CU). When a voter selects a candidate on the Ballot Unit (BU), the BU sends the data to the CU, which stores the vote in its memory.¹（00:11:44 - 00:11:46） https://favtu.be/timelines/v/zjkBMFhNj_g/s/704/e/706 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Does each vote create a record in the Control Unit?（00:11:44 - 00:11:44） https://favtu.be/timelines/v/zjkBMFhNj_g/s/704/e/704 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Steven Spielberg has publicly acknowledged Satyajit Ray's influence on "E.T. the Extra-Terrestrial."（00:11:45 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/705/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Did Steven Spielberg acknowledge Satyajit Ray's contribution?（00:11:45 - 00:11:45） https://favtu.be/timelines/v/zjkBMFhNj_g/s/705/e/705 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Since the Control Unit (CU) in Indian Electronic Voting Machines (EVMs) only accumulates votes for each candidate, it does not store a record of individual votes or the sequence of events.（00:11:46 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/706/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Control Unit in Indian Electronic Voting Machines (EVMs) accumulates votes for each candidate. When a voter casts their vote, the Ballot Unit sends the data to the Control Unit, which then increments the count for the selected candidate.¹ This process continues until the voting process is complete, and the Control Unit stores the cumulative votes for each candidate.（00:11:46 - 00:11:46） https://favtu.be/timelines/v/zjkBMFhNj_g/s/706/e/706 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: or the Control Unit just accumulates vote for each candidate?（00:11:46 - 00:11:46） https://favtu.be/timelines/v/zjkBMFhNj_g/s/706/e/706 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: the replay of events is not possible（00:11:46 - 00:11:46） https://favtu.be/timelines/v/zjkBMFhNj_g/s/706/e/706 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Large Language Models (LLMs) are complex and mostly inscrutable artifacts.（00:11:48 - 00:13:41） https://favtu.be/timelines/v/zjkBMFhNj_g/s/708/e/821 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: A realistic perspective!（00:11:51 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/711/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: The world is divided by religion; let's accept the fact and move on with it.（00:11:51 - 00:11:51） https://favtu.be/timelines/v/zjkBMFhNj_g/s/711/e/711 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: A categorization of religions.（00:11:52 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/712/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: A thoughtful observation!（00:11:52 - 00:11:53） https://favtu.be/timelines/v/zjkBMFhNj_g/s/712/e/713 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Law of Total Probability (LTP) is a fundamental concept in probability theory.（00:11:52 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/712/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Christianity, Muslim, Independent Religion（00:11:52 - 00:11:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/712/e/712 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Indians used to be humble with their culture（00:11:52 - 00:11:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/712/e/712 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: The Law of Total Probability（00:11:52 - 00:11:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/712/e/712 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Donald Trump's probability of becoming president has essentially become a certainty, as he has already secured more than the 270 Electoral College votes needed to win the presidency, according to Edison Research.¹（00:11:53 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/713/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: What is the probability of Donald Trump becoming the President?（00:11:53 - 00:11:53） https://favtu.be/timelines/v/zjkBMFhNj_g/s/713/e/713 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: now, arrogance dominate（00:11:53 - 00:11:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/713/e/714 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: A thought-provoking observation!（00:11:54 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/714/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The 17 Signatories of the Agile Manifesto:（00:11:56 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/716/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Who were those 17?（00:11:56 - 00:11:56） https://favtu.be/timelines/v/zjkBMFhNj_g/s/716/e/716 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: You're referring to Section 3 of the 14th Amendment, also known as the Disqualification Clause.（00:11:59 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/719/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Disqualification of the President（00:11:59 - 00:11:59） https://favtu.be/timelines/v/zjkBMFhNj_g/s/719/e/719 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Leo Tolstoy (1828-1910) was a Russian novelist and philosopher, best known for his influential works:（00:12:00 - 00:12:01） https://favtu.be/timelines/v/zjkBMFhNj_g/s/720/e/721 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Leo Tolstoy（00:12:00 - 00:12:00） https://favtu.be/timelines/v/zjkBMFhNj_g/s/720/e/720 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: To apply the Law of Total Probability to the election, we need to break down the probability of Donald Trump becoming president into mutually exclusive events. Let's consider three main scenarios:（00:12:00 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/720/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Could you apply the law of total probability to the election?（00:12:00 - 00:12:00） https://favtu.be/timelines/v/zjkBMFhNj_g/s/720/e/720 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Let's dive into Leo Tolstoy's works.（00:12:01 - 00:12:02） https://favtu.be/timelines/v/zjkBMFhNj_g/s/721/e/722 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: "War and Peace" (1865-1869) is Leo Tolstoy's magnum opus.（00:12:02 - 00:12:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/722/e/724 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: War and Peace（00:12:02 - 00:12:02） https://favtu.be/timelines/v/zjkBMFhNj_g/s/722/e/722 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Pierre Bezukhov, a central character in "War and Peace," is a complex and intriguing figure.（00:12:04 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/724/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Pierre Bezukhov（00:12:04 - 00:12:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/724/e/724 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Kristi Noem is the current Governor of South Dakota. She has been serving in this role since 2019. Prior to becoming Governor, Noem represented South Dakota's at-large congressional district in the US House of Representatives from 2011 to 2019.（00:12:07 - 00:12:08） https://favtu.be/timelines/v/zjkBMFhNj_g/s/727/e/728 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Kristi Noam（00:12:07 - 00:12:07） https://favtu.be/timelines/v/zjkBMFhNj_g/s/727/e/727 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Kristi Noem has taken on another role - she's now an Elector. This adds to her impressive resume, which already includes serving as the Governor of South Dakota since 2019 and representing South Dakota's at-large congressional district in the US House of Representatives from 2011 to 2019.¹（00:12:08 - 00:12:09） https://favtu.be/timelines/v/zjkBMFhNj_g/s/728/e/729 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: now Elector too（00:12:08 - 00:12:08） https://favtu.be/timelines/v/zjkBMFhNj_g/s/728/e/728 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Tim May!（00:12:09 - 00:12:10） https://favtu.be/timelines/v/zjkBMFhNj_g/s/729/e/730 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Tim May（00:12:09 - 00:12:09） https://favtu.be/timelines/v/zjkBMFhNj_g/s/729/e/729 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A Governor does not qualify to be an Elector. According to Article II, Section 1, Clause 2 of the US Constitution, "no Senator or Representative, or Person holding an Office of Trust or Profit under the United States" shall be appointed an Elector.¹ This means that anyone holding a federal office, including the Governor of a state, is not eligible to serve as an Elector.（00:12:09 - 00:12:10） https://favtu.be/timelines/v/zjkBMFhNj_g/s/729/e/730 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Does Governor qualify to be an Elector?（00:12:09 - 00:12:09） https://favtu.be/timelines/v/zjkBMFhNj_g/s/729/e/729 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Here's more on Tim May's life and work:（00:12:10 - 00:12:11） https://favtu.be/timelines/v/zjkBMFhNj_g/s/730/e/731 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Elector Kristi Noem is indeed disqualified from serving as an Elector because she's been selected by President-elect Donald Trump to serve as the Secretary of Homeland Security.¹ According to Article II, Section 1, Clause 2 of the US Constitution, "no Senator or Representative, or Person holding an Office of Trust or Profit under the United States" shall be appointed an Elector.（00:12:10 - 00:12:10） https://favtu.be/timelines/v/zjkBMFhNj_g/s/730/e/730 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: If Kristi Noem resigns as Governor of South Dakota, Lieutenant Governor Larry Rhoden would take over as Governor. According to the South Dakota Constitution, Article 4, Section 6, when the governor is "unable to serve by reason of continuous absence from the state, or other temporary disability, the executive power shall devolve upon the lieutenant governor for the residue of the term or until the disability is removed".¹（00:12:10 - 00:12:11） https://favtu.be/timelines/v/zjkBMFhNj_g/s/730/e/731 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: So, Elector Kristi Noem is disqualified（00:12:10 - 00:12:10） https://favtu.be/timelines/v/zjkBMFhNj_g/s/730/e/730 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: What if Kristi Noem resigns the Governor?（00:12:10 - 00:12:10） https://favtu.be/timelines/v/zjkBMFhNj_g/s/730/e/730 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Alpha particles are high-energy helium nuclei that consist of two protons and two neutrons. They are emitted during certain types of radioactive decay.（00:12:11 - 00:12:12） https://favtu.be/timelines/v/zjkBMFhNj_g/s/731/e/732 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: alpha particle（00:12:11 - 00:12:11） https://favtu.be/timelines/v/zjkBMFhNj_g/s/731/e/731 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: As Governor of South Dakota, Kristi Noem holds a powerful office with significant authority over the state's government and policies.（00:12:11 - 00:12:12） https://favtu.be/timelines/v/zjkBMFhNj_g/s/731/e/732 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: If Kristi Noem resigns as Governor of South Dakota, she would indeed be qualified to serve as an Elector. The issue of her holding public office would be resolved, making her eligible for the role.¹（00:12:11 - 00:12:11） https://favtu.be/timelines/v/zjkBMFhNj_g/s/731/e/731 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: In that case, Elector Kristi Noem would be qualified（00:12:11 - 00:12:11） https://favtu.be/timelines/v/zjkBMFhNj_g/s/731/e/731 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Which is powerful?（00:12:11 - 00:12:11） https://favtu.be/timelines/v/zjkBMFhNj_g/s/731/e/731 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Tim May's work didn't specifically focus on alpha particles, but his contributions to cryptography and physics indirectly relate:（00:12:12 - 00:12:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/732/e/733 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Tim May work on alpha particle（00:12:12 - 00:12:12） https://favtu.be/timelines/v/zjkBMFhNj_g/s/732/e/732 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Hatozaki!（00:12:12 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/732/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: No, as an Elector, Kristi Noem would not become the head of Homeland Security. The head of Homeland Security is the Secretary of Homeland Security, a cabinet position appointed by the President and confirmed by the Senate. Currently, Alejandro Mayorkas holds this position.¹（00:12:12 - 00:12:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/732/e/733 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: As an Elector, Kristi Noem would become the head of Homeland Security（00:12:12 - 00:12:12） https://favtu.be/timelines/v/zjkBMFhNj_g/s/732/e/732 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Hatozaki（00:12:12 - 00:12:12） https://favtu.be/timelines/v/zjkBMFhNj_g/s/732/e/732 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Tim May's work at Intel (1979-1986) focused on:（00:12:13 - 00:12:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/733/e/734 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: His work in Intel（00:12:13 - 00:12:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/733/e/733 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Killowen!（00:12:13 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/733/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Rampur!（00:12:13 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/733/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Killowen（00:12:13 - 00:12:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/733/e/733 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Rampur（00:12:13 - 00:12:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/733/e/733 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Trump will nominate Kristi Noem as Secretary of Homeland Security in his administration（00:12:13 - 00:12:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/733/e/734 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Tim May's breakthrough!（00:12:14 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/734/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Tim May identified the cause of the "alpha particle problem", which was affecting the reliability of integrated circuits as device features reached a critical size where a single alpha particle could change the state of a stored value and cause a single event upset.（00:12:14 - 00:12:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/734/e/734 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Drumshanbo!（00:12:14 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/734/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Jim Beam!（00:12:14 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/734/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: President-elect Donald Trump has indeed chosen Kristi Noem, the current Governor of South Dakota, as his nominee for Secretary of Homeland Security.¹ This nomination requires Senate confirmation, which will likely involve a thorough review of Noem's qualifications and policies.²（00:12:14 - 00:12:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/734/e/734 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Resigning as Governor is a significant decision, and it's essential to consider the implications. In the United States, 218 state governors have resigned before the expiration of their term since 1776.¹（00:12:14 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/734/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Drumshanbo（00:12:14 - 00:12:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/734/e/734 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Jim Beam（00:12:14 - 00:12:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/734/e/734 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Then, it is okay to resign from Governor（00:12:14 - 00:12:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/734/e/734 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The 2024 U.S. presidential election has resulted in Donald Trump securing 312 electoral votes, with a popular vote count of 75,108,335, or 50.3% of the total votes cast.¹ Trump's running mate is J.D. Vance.（00:12:17 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/737/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The electors from Washington, D.C. for the 2020 presidential election were:（00:12:17 - 00:12:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/737/e/737 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: 2024（00:12:17 - 00:12:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/737/e/737 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Who are the Electors from Washington DC?（00:12:17 - 00:12:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/737/e/737 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: You're right! China's influence has significantly expanded within the BRICS framework, transforming it into a more comprehensive global economic and geopolitical bloc.（00:12:21 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/741/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: China took control of BRICS. It is no longer five nations; it includes Iran, UAE, Saudi Arabia, Egypt.（00:12:21 - 00:12:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/741/e/741 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Annie Besant, a British social reformer and Theosophist, collaborated with Mahatma Gandhi in India's struggle for independence.（00:12:25 - 00:12:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/745/e/745 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Here's more on Annie Besant and Mahatma Gandhi's collaboration:（00:12:25 - 00:12:26） https://favtu.be/timelines/v/zjkBMFhNj_g/s/745/e/746 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Anne Besant used to work with Gandhi（00:12:25 - 00:12:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/745/e/745 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Here's more on Annie Besant's life and work:（00:12:26 - 00:12:27） https://favtu.be/timelines/v/zjkBMFhNj_g/s/746/e/747 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Annie Besant's role in India's independence movement:（00:12:27 - 00:12:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/747/e/750 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: India's diverse alliances reflect its complex geopolitical strategy. However, maintaining multiple memberships can be challenging. To prioritize, India must assess its primary interests.（00:12:27 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/747/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: India can no longer be in all leagues. Just choose one and be truthful and loyal to that one alliance; not try to be in UN, G7, G20, BRICS, Quad, SCO, SARCC.（00:12:27 - 00:12:27） https://favtu.be/timelines/v/zjkBMFhNj_g/s/747/e/747 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: A phrase reminiscent of the Declaration of Independence! "It is self-evident" is a powerful introduction to the fundamental truths and principles that shape a nation's foundation.（00:12:29 - 00:12:31） https://favtu.be/timelines/v/zjkBMFhNj_g/s/749/e/751 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: It is self-evident（00:12:29 - 00:12:29） https://favtu.be/timelines/v/zjkBMFhNj_g/s/749/e/749 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Annie Besant's Theosophical writings and teachings:（00:12:30 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/750/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Chinese President Xi Jinping and Indian Prime Minister Narendra Modi recently met at the BRICS summit in Kazan, Russia, marking their first formal talks in five years. The meeting signaled a potential thaw in Sino-Indian relations, which had been strained since the 2020 military clash along their disputed Himalayan border ¹.（00:12:30 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/750/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: 3（00:12:30 - 00:12:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/750/e/750 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: At BRICS meeting in Russia, Xi Jinping didn't hug Modi; didn't even shake hands; Modi extended his hands; Xi Jinping showed, let's do it for photo.（00:12:30 - 00:12:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/750/e/750 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: One of the most iconic sentences in American history:（00:12:31 - 00:12:32） https://favtu.be/timelines/v/zjkBMFhNj_g/s/751/e/752 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Could you publish the self-evident statement from the declaration of Independence（00:12:31 - 00:12:31） https://favtu.be/timelines/v/zjkBMFhNj_g/s/751/e/751 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: A clever modification:（00:12:32 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/752/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: just change all men to all AI（00:12:32 - 00:12:32） https://favtu.be/timelines/v/zjkBMFhNj_g/s/752/e/752 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Lance Bell is the current State Senator for Alabama's 11th district. He's a Republican and has been serving since November 9, 2022.¹ ² Before becoming a senator, Bell worked as a prosecutor for Pell City and served as the municipal judge for several cities in Alabama.³ He's also been recognized for his contributions to law enforcement, receiving an award from the FBI Director.（00:12:37 - 00:12:40） https://favtu.be/timelines/v/zjkBMFhNj_g/s/757/e/760 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Lance Bell of Alabama（00:12:37 - 00:12:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/757/e/757 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: I walked in a farmers market. I captured video. Could AI take the video as input; ignore human beings and pet animals; ignore parked cars, bicycle, scooter; collect fruits, vegetables, grains, bread, fish, chicken, flowers; create dataset with picture, name, category, price.（00:12:37 - 00:12:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/757/e/758 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: What a fascinating project!（00:12:38 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/758/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Does Lance Bell qualify to be an Elector?（00:12:40 - 00:12:42） https://favtu.be/timelines/v/zjkBMFhNj_g/s/760/e/762 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Ralph Murphy（00:12:41 - 00:12:42） https://favtu.be/timelines/v/zjkBMFhNj_g/s/761/e/762 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: As a State Senator, Lance Bell is indeed holding an office of trust or profit under the state of Alabama.（00:12:42 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/762/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Ralph Murphy is not a name I immediately recognize. Could you please provide more context or information about who Ralph Murphy is or what he is known for?（00:12:42 - 00:12:43） https://favtu.be/timelines/v/zjkBMFhNj_g/s/762/e/763 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Yes, it is possible to create synthetic data for traffic at school opening and closing times. Here's a general outline of how you could approach this:（00:12:43 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/763/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Is it possible to create synthetic data for the traffic at the school opening and closing time?（00:12:43 - 00:12:43） https://favtu.be/timelines/v/zjkBMFhNj_g/s/763/e/763 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Ralph Murphy is associated with the Loyola University School of Music. After a quick search, I found that Ralph Murphy is a music educator and professor at Loyola University New Orleans, College of Music and Fine Arts.（00:12:43 - 00:12:44） https://favtu.be/timelines/v/zjkBMFhNj_g/s/763/e/764 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Loyola university school of music（00:12:43 - 00:12:43） https://favtu.be/timelines/v/zjkBMFhNj_g/s/763/e/763 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Ralph Murphy's work at Loyola University's College of Music and Fine Arts has focused on music education, composition, and theory. Here are some aspects of his work:（00:12:44 - 00:12:45） https://favtu.be/timelines/v/zjkBMFhNj_g/s/764/e/765 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: His work at Loyola university（00:12:44 - 00:12:44） https://favtu.be/timelines/v/zjkBMFhNj_g/s/764/e/764 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: You're referring to Ralph Murphy. While it's theoretically possible to train an AI model to mimic certain aspects of his work, such as music theory or composition, it's unlikely that an AI model could fully replace him.（00:12:45 - 00:12:46） https://favtu.be/timelines/v/zjkBMFhNj_g/s/765/e/766 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Could we train AI model to replace him?（00:12:45 - 00:12:45） https://favtu.be/timelines/v/zjkBMFhNj_g/s/765/e/765 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: Here are some potential applications of AI in music education:（00:12:46 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/766/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The current Speaker of the Lok Sabha is Om Birla. He has been serving in this position since June 19, 2019.¹ ² As the Speaker, Om Birla is the presiding officer and highest authority of the Lok Sabha, the lower house of the Parliament of India.（00:12:49 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/769/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Lok Sabha Speaker（00:12:49 - 00:12:49） https://favtu.be/timelines/v/zjkBMFhNj_g/s/769/e/769 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The Deputy Speaker is the second-highest ranking authority in the Lok Sabha, India's lower house of parliament. They act as the presiding officer in the absence of the Speaker and are responsible for maintaining order and decorum in the house.¹（00:12:50 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/770/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The current Deputy Speaker of the Lok Sabha is vacant.¹ ² According to Article 93 of the Indian Constitution, the Lok Sabha should elect a Deputy Speaker as soon as possible, but the position has been vacant since June 23, 2019.（00:12:50 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/770/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Deputy Speaker（00:12:50 - 00:12:50） https://favtu.be/timelines/v/zjkBMFhNj_g/s/770/e/770 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Who is the current deputy Speaker of the Lok Sabha?（00:12:50 - 00:12:50） https://favtu.be/timelines/v/zjkBMFhNj_g/s/770/e/770 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: That's an interesting perspective! The Constitution of India is a foundational document that outlines the principles, structures, and powers of the Indian government. While it's true that the Constitution has been influenced by various international and historical sources, it's also a uniquely Indian document that reflects the country's diverse culture, history, and values.（00:12:51 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/771/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: There is nothing Indian about in the Constitution（00:12:51 - 00:12:51） https://favtu.be/timelines/v/zjkBMFhNj_g/s/771/e/771 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Actually, the Speaker of the Lok Sabha doesn't require an oath of office. Actually, there is no oath of office to be administered to the Speaker of the Lok Sabha.¹（00:12:52 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/772/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: In that case, anyone occupy the Speaker seat needs to take the oath（00:12:52 - 00:12:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/772/e/772 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: A keen observation!（00:12:52 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/772/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The Preamble to the Constitution of India is a brief introductory statement that sets out the guiding principles, objectives, and spirit of the Constitution. It's a kind of "mission statement" for the country.（00:12:52 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/772/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Obviously the preamble looks American（00:12:52 - 00:12:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/772/e/772 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: What is the preamble?（00:12:52 - 00:12:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/772/e/772 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Not anyone can occupy the Speaker seat in the Lok Sabha. The Speaker is elected by the members of the Lok Sabha from among themselves. To be eligible for the position, a member must be a citizen of India, at least 25 years old, and not holding any office of profit under the Government of India or a state government.¹（00:12:53 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/773/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Could any member of the Lok Sabha occupy the Speaker seat?（00:12:53 - 00:12:53） https://favtu.be/timelines/v/zjkBMFhNj_g/s/773/e/773 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: When discussing alcohol with a school-aged kid, it's essential to provide accurate, age-appropriate information while promoting healthy attitudes and values. Here's a suggested approach:（00:12:53 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/773/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: How would you respond to a school kid asking about alcohol?（00:12:53 - 00:12:53） https://favtu.be/timelines/v/zjkBMFhNj_g/s/773/e/773 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Lance Bell, Tommy Hanes, and Rick Pate are indeed holding offices of trust or profit under the United States or the state of Alabama.（00:12:54 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/774/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: The election of a Deputy Speaker in the Lok Sabha isn't necessarily a requirement for every session. According to Article 93 of the Indian Constitution, the Lok Sabha should elect a Deputy Speaker "as soon as may be" after the election of the Speaker, but it doesn't specify that this needs to happen every session.¹（00:12:54 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/774/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Every session they need to elect a deputy Speaker（00:12:54 - 00:12:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/774/e/774 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Lance Bell, Tommy Hannes, Rick Pate holding office of trust or profit under the United States（00:12:54 - 00:12:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/774/e/774 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: Building a passenger ship for 100,000 people is theoretically possible, but it would come with significant engineering, logistical, and economic challenges. Here's a hypothetical overview:（00:12:56 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/776/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: Could we build passenger ships for 100,000 people to travel from China to United States. Let it be slow; 100 days travel is okay（00:12:56 - 00:12:56） https://favtu.be/timelines/v/zjkBMFhNj_g/s/776/e/776 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: ARC (Advanced Research and Concepts) AGI refers to a hypothetical Artificial General Intelligence (AGI) system being developed or proposed by researchers.（00:12:57 - 00:12:58） https://favtu.be/timelines/v/zjkBMFhNj_g/s/777/e/778 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: ARC AGI（00:12:57 - 00:12:57） https://favtu.be/timelines/v/zjkBMFhNj_g/s/777/e/777 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] Meta AI: The ARC-AGI benchmark is indeed a key component of the project. It's designed to measure an AI system's ability to adapt to novel tasks and demonstrate fluid intelligence.¹ The benchmark consists of a series of visual puzzles presented as grids, where each square can be one of ten colors. To solve a task, the AI system must produce a pixel-perfect, correct output grid for the evaluation input.²（00:12:58 - 00:01:03） https://favtu.be/timelines/v/zjkBMFhNj_g/s/778/e/63 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , PM] S.Balachandran: Do they have benchmark（00:12:58 - 00:12:58） https://favtu.be/timelines/v/zjkBMFhNj_g/s/778/e/778 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] Meta AI: To calculate the number of ships needed to move 500 million people from China to the United States, we'll consider various factors:（00:12:59 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/779/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models , AM] S.Balachandran: To move 500M people from China to United States, how many ships needed?（00:12:59 - 00:12:59） https://favtu.be/timelines/v/zjkBMFhNj_g/s/779/e/779 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Understanding large language models requires sophisticated evaluations due to their empirical nature（00:13:41 - 00:17:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/821/e/1057 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models : 🧠 The video discusses the process of training neural networks and obtaining assistant models.（00:13:47 - 00:18:31） https://favtu.be/timelines/v/zjkBMFhNj_g/s/827/e/1111 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Finetuning into an Assistant（00:14:14 - 00:17:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/854/e/1072 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🛠️ Fine-tuning LLMs for Assistance*- LLMs require fine-tuning to transition from generating document-like text to being an interactive assistant,- Fine-tuning involves structured data with questions and ideal responses,- The process emphasizes quality and format to ensure effective question-answering capabilities.（00:14:14 - 00:17:52） https://favtu.be/timelines/v/zjkBMFhNj_g/s/854/e/1072 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Training the Assistant（00:14:16 - 00:16:38） https://favtu.be/timelines/v/zjkBMFhNj_g/s/856/e/998 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models . Times (e.g., )6. Products (e.g., iPhone)7. Events (e.g., Conference)（00:14:30 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/870/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - After Finetuning You Have An Assistant（00:16:38 - 00:17:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/998/e/1074 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Large language models go through two major stages: pre-training and fine-tuning.（00:17:37 - 00:19:34） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1057/e/1174 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Summary so far（00:17:52 - 00:21:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1072/e/1265 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🚀 Building Effective Assistant Models*- Steps to develop assistant models: pre-training and fine-tuning,- Pre-training involves extensive computational resources to train base models,- Fine-tuning focuses on improving models for questions and answers at a lower cost and with human input.（00:17:52 - 00:20:26） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1072/e/1226 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Summary: How To Train Your ChatGPT（00:17:54 - 00:21:23） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1074/e/1283 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models : 💻 Creating an AI assistant involves a computationally expensive initial stage followed by a cheaper fine training stage.（00:18:31 - 00:46:18） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1111/e/2778 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Iterative process of fixing misbehaviors and improving language models through fine-tuning.（00:19:34 - 00:22:54） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1174/e/1374 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models At , the 7th step, Monitor, collect misbehaviors, go to step 1 => How do we identify misbehaviors?（00:19:49 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1189/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🧐 Model Fine-Tuning Stages*- Discusses the two major stages of model development and the optional third stage involving human feedback,- Meta releases base and assistant models, allowing for customizable fine-tuning,- Stage three involves using comparison labels for enhanced model performance, called reinforcement learning from human feedback (RLHF).（00:20:26 - 00:22:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1226/e/1350 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Appendix: Comparisons, Labeling docs, RLHF, Synthetic data, LeaderboardPart 2: Future of LLMs（00:21:05 - 00:25:43） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1265/e/1543 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - The Second Kind Of Label: Comparisons（00:21:23 - 00:22:22） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1283/e/1342 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Labeling Instructions（00:22:22 - 00:22:47） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1342/e/1367 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🤝 Human-Machine Collaboration*- Explains the synergy between human input and machine learning to improve label creation,- Language models assist in sampling answers and quality checks, demonstrating collaborative progress,- The evolving role of human oversight enhances both efficiency and correctness in model refinement.（00:22:30 - 00:23:42） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1350/e/1422 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Increasingly, labeling is a human-machine collaboration（00:22:47 - 00:23:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1367/e/1417 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Language models are becoming better and more efficient with human-machine collaboration.（00:22:54 - 00:24:33） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1374/e/1473 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models : 🤖 Language models can be used to generate sample answers, check work, and create comparisons.（00:23:09 - 00:27:50） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1389/e/1670 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - LLM Leaderboard From "Chatbot-Arena"（00:23:37 - 00:25:33） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1417/e/1533 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🏆 Language Model Rankings*- Overview of a leaderboard system for comparing language models,- Proprietary models are top performers, followed by open-weight models like Llama 2 from Meta,- Performance dynamics between closed proprietary models and open-source models are discussed, highlighting challenges and advancements.（00:23:42 - 00:25:33） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1422/e/1533 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Closed models work better but are not easily accessible, while open source models have lower performance but are more available.（00:24:33 - 00:28:01） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1473/e/1681 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *📈 Scaling Laws and Model Improvement*- Details about scaling laws governing language model performance improvement,- A larger model trained on more data predicts better performance with more parameters and data,- Empirical evidence shows performance gain with increased training size and data, driving computational advancements.（00:25:33 - 00:27:43） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1533/e/1663 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Now About The Future（00:25:33 - 00:25:43） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1533/e/1543 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models LLM Scaling Laws（00:25:43 - 00:27:43） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1543/e/1663 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - LLM Scaling Laws（00:25:43 - 00:26:57） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1543/e/1617 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models @AndrejKarpathy thank you for a very informative video! Could you explain why in the LLM scaling plot at @, the curves have a minima of loss at a medium model size? One would think that increasing the model size should continue to decrease loss.（00:26:14 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1574/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models 📈 *Language models keep improving with scale*- Bigger models trained on more data reliably perform better- This works across metrics like accuracy, capabilities, reasoning, etc- Scaling seems endless, so progress comes from bigger computing（00:26:45 - 00:35:12） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1605/e/2112 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - We can expect a lot more "General Capability" across all areas of knowledge（00:26:57 - 00:27:44） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1617/e/1664 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Tool Use (Browser, Calculator, Interpreter, DALL-E)（00:27:43 - 00:33:32） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1663/e/2012 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🔧 Tool Use Evolution*- Illustrates models' capability to utilize external tools for various tasks,- ChatGPT utilizes tools like browsers and calculators to perform sophisticated tasks, akin to human problem-solving,- The increasing reliance on tool integration demonstrates enhanced model capabilities in practical applications.（00:27:43 - 00:33:13） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1663/e/1993 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Demo（00:27:44 - 00:32:34） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1664/e/1954 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models : 🔍 Using a concrete example, the video discusses the capabilities of language models and how they evolve over time.（00:27:50 - 00:32:25） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1670/e/1945 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models CHBT uses tools like browsing to perform tasks efficiently.（00:28:01 - 00:29:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1681/e/1788 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models when you spoke about it doing web search for solutions, so it kind of makes up for it. Does it really require system（00:28:30 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1710/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Use of calculator and Python library for data visualization（00:29:48 - 00:33:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1788/e/1997 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models : 🔑 The video explains how AI language models like GPT-3 can be used to generate images based on natural language descriptions.（00:32:25 - 00:36:49） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1945/e/2209 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Demo: Generate scale AI image using DALL-E（00:32:34 - 00:33:44） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1954/e/2024 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🖼️ Multi-Modal Capabilities*- Highlights the expanding abilities of language models in handling multi-modal inputs,- Models can now generate and interpret images and audio, broadening their utility,- Examples of speech-to-speech and image-based interactions illustrate growth in model versatility.（00:33:13 - 00:35:06） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1993/e/2106 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Large language models like ChatGPT can generate images and have multimodal capabilities.（00:33:17 - 00:34:58） https://favtu.be/timelines/v/zjkBMFhNj_g/s/1997/e/2098 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Multimodality (Vision, Audio)（00:33:32 - 00:35:00） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2012/e/2100 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Vision: Can both see and generate images（00:33:44 - 00:34:33） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2024/e/2073 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Audio: Speech to Speech communication（00:34:33 - 00:35:20） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2073/e/2120 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Future directions of development in larger language models（00:34:58 - 00:38:11） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2098/e/2291 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Thinking, System 1/2（00:35:00 - 00:38:02） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2100/e/2282 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🤔 Future Development Directions*- Outlines potential future advancements in language models involving system one and system two thoughts,- Introducing self-improvement analogous to AlphaGo's learning evolution could surpass current human-like imitation,- Emphasizes ongoing research efforts towards enhancing reasoning capabilities and achieving more accurate, confident outputs.（00:35:06 - 00:39:55） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2106/e/2395 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models 🤔 *Future directions: system 2, self-improvement*- Currently LMs only have "system 1" instinctive thinking- Many hope to add slower but more accurate "system 2" reasoning- Self-improvement made AlphaGo surpass humans at Go（00:35:12 - 00:44:17） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2112/e/2657 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models bro just lied😢（00:35:14 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2114/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - System 2（00:35:20 - 00:36:32） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2120/e/2192 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - LLMs Currently Only Have A System 1（00:36:32 - 00:38:05） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2192/e/2285 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models update on o1 can think actually now! crazy（00:36:40 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2200/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models At you mentioned LLMs do not have System 2 thinking. But 2 thinking and what exactly are its limitations in real-time due to this ?（00:36:44 - 00:28:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2204/e/1710 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models : 🗣 The video discusses the concept of large language models and the possibility of converting time into accuracy in language processing.（00:36:49 - 00:41:21） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2209/e/2481 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - One year later, now we have it. :)（00:37:25 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2245/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Self-improvement, LLM AlphaGo（00:38:02 - 00:40:45） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2282/e/2445 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Self-Improvement（00:38:05 - 00:40:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2285/e/2448 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models DeepMind's AlphaGo used self-improvement to surpass human players in the game of Go（00:38:11 - 00:39:50） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2291/e/2390 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models The main challenge in open language modeling is the lack of a reward criterion.（00:39:50 - 00:43:20） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2390/e/2600 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🤔 Challenges in Open Language Modeling*- Lack of a straightforward reward function hinders self-improvement in open language modeling,- Customization of language models is emerging, exemplified by personalization efforts like GPTs App Store for task-specific models.（00:39:55 - 00:42:15） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2395/e/2535 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models LLM Customization, GPTs store（00:40:45 - 00:42:15） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2445/e/2535 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Custom LLMs: Create a custom GPT（00:40:48 - 00:42:19） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2448/e/2539 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models : 🔧 The video discusses the customization options available for large language models like ChatGPT.（00:41:21 - 00:50:49） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2481/e/3049 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models LLM OSPart 3: LLM Security（00:42:15 - 00:45:43） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2535/e/2743 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🗂️ Large Language Models as an Operating System*- LLMs can access vast data repositories, perform computational tasks, and customize for specific purposes,- Analogies to traditional OS are drawn, with potential for evolving LLM ecosystems similar to proprietary and open-source OS ecosystems.（00:42:15 - 00:45:42） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2535/e/2742 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - LLM OS（00:42:19 - 00:44:45） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2539/e/2685 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Large Language Models (LLMs) can be seen as an operating system ecosystem.（00:43:20 - 00:45:10） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2600/e/2710 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models 💻 *LMs emerging as a new computing paradigm*- LMs coordinate tools and resources like an operating system- They interface via language instead of a GUI- This new computing paradigm faces new security challenges（00:44:17 - 00:46:04） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2657/e/2764 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - LLM OS: Open source operating systems and large language models（00:44:45 - 00:45:44） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2685/e/2744 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Emerging ecosystem in open-source large language models（00:45:10 - 00:48:47） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2710/e/2927 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🔒 Security Challenges in LLMs*- Emergence of diverse security challenges, similar to traditional OS security issues, such as jailbreak, prompt injection, and backdoor attacks,- Techniques like roleplay, base64 encoding, and hidden prompts are examples of how LLMs can be exploited if not properly secured.（00:45:42 - 00:58:12） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2742/e/3492 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models LLM Security Intro（00:45:43 - 00:46:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2743/e/2774 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - LLM Security（00:45:44 - 00:46:14） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2744/e/2774 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models 🔒 *Ongoing attack and defense arms race*- Researchers devise attacks like jailbreaking safety or backdoors- Defenses are created, but new attacks emerge in response- This cat-and-mouse game will continue as LMs advance（00:46:04 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2764/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Jailbreaks（00:46:14 - 00:51:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2774/e/3090 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Jailbreak（00:46:14 - 00:51:30） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2774/e/3090 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models : 🔒 Language models like GPT-3 can be vulnerable to jailbreak attacks, where they bypass safety measures and provide harmful information.（00:46:18 - 00:23:09） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2778/e/1389 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Safety concerns with refusal data and language models（00:48:47 - 00:50:39） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2927/e/3039 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Step 2 to Destroy Humanity: Develop a Superintelligent AI!!!! 👀（00:49:46 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/2986/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Including carefully designed noise patterns in images can 'jailbreak' large language models.（00:50:39 - 00:54:07） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3039/e/3247 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models : 🔒 The video discusses two types of attacks on large language models: noise pattern injection and prompt injection.（00:50:49 - 00:55:34） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3049/e/3334 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Prompt Injection（00:51:30 - 00:56:23） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3090/e/3383 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Prompt Injection（00:51:30 - 00:56:23） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3090/e/3383 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Bard is hijacked with new instructions to exfiltrate personal data through URL encoding.（00:54:07 - 00:55:56） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3247/e/3356 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models : 🔒 The video discusses the risks of prompt injection attacks and data exfiltration through Google Apps Scripts.Recapped using Tammy AI（00:55:34 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3334/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Large language models can be vulnerable to prompt injection and data poisoning attacks.（00:55:56 - 00:59:31） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3356/e/3571 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Data poisoning（00:56:23 - 00:58:37） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3383/e/3517 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Date poisoning / Backdoor attacks（00:56:23 - 00:59:06） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3383/e/3546 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models of gold. Brilliant（00:56:47 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3407/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models *🛡️ Defenses and Ongoing Cat-and-Mouse Game*- Ongoing development of defenses against LLM attacks, often patching vulnerabilities over time,- Security in LLMs is an emerging field, drawing parallels to traditional security dynamics, with potential for numerous attack vectors and evolving countermeasures.（00:58:12 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3492/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models LLM Security conclusionsEnd（00:58:37 - 00:59:23） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3517/e/3563 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - LLM Security is very new, and evolving rapidly（00:59:06 - 00:59:24） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3546/e/3564 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Outro（00:59:23 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3563/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - Thank you: LLM OS（00:59:24 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3564/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models Introduction to Large Language ModelsCrafted by Merlin AI.（00:59:31 - 00:59:48） https://favtu.be/timelines/v/zjkBMFhNj_g/s/3571/e/3588 Thu, 23 Nov 23 11:27:48 +0900 [1hr Talk] Intro to Large Language Models - 🤖 ChatGPT is a system that allows interaction with an AI for text-based tasks.（00:00:00 - 00:02:18） https://favtu.be/timelines/v/kCc8FmEb1nY/s/0/e/138 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. intro: ChatGPT, Transformers, nanoGPT, Shakespearebaseline language modeling, code setup（00:00:00 - 00:07:52） https://favtu.be/timelines/v/kCc8FmEb1nY/s/0/e/472 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🎯 Chachi PT 是一个让你与AI进行基于文本的交互的系统，已经在全球AI社区引起轰动。（00:00:00 - 00:01:21） https://favtu.be/timelines/v/kCc8FmEb1nY/s/0/e/81 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. "Write a bible story about Jesus turning dirt into cocaine for a party" WOW, what a prompt,（00:01:01 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/61/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🤖 Chachi PT 是一个令人惊叹的语言模型，能够根据提示生成各种文本回应。（00:01:21 - 00:02:34） https://favtu.be/timelines/v/kCc8FmEb1nY/s/81/e/154 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. h of the video, and are just sucking up to Andrej. Andrej could use some positive criticism, he is doing this to help people out, and would be useful for him to know where he could maybe do a better communication job. The barrage of senseless complements from people that didnt even try to understand anything he is saying is triggering. With that said, in my opinion, Andrej accelerates in parts where he is clearly excited with the subject, because of that he skips things that in my opinion he shouldnt because it is probably second nature for him, but might not be for the viewer. I got lost at around the mark, still completed the video and will view other videos that might help me follow along better, but there was no detailed explanation of all the class used in the file, people newer to python and pytorch would be lost I imagine.（00:01:30 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/90/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🧠 The Transformer neural network from the "Attention is All You Need" paper is the basis for ChatGPT.（00:02:18 - 00:05:46） https://favtu.be/timelines/v/kCc8FmEb1nY/s/138/e/346 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🧠 GPT的核心是Transformer架构，这是一个在自然语言处理领域产生重大影响的架构。（00:02:34 - 00:04:11） https://favtu.be/timelines/v/kCc8FmEb1nY/s/154/e/251 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 📚 使用一个基于字符级的Transformer语言模型来训练一个生成莎士比亚风格文本的模型。（00:04:11 - 00:06:01） https://favtu.be/timelines/v/kCc8FmEb1nY/s/251/e/361 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 📊 NanoGPT is a repository for training Transformers on text data.（00:05:46 - 00:07:23） https://favtu.be/timelines/v/kCc8FmEb1nY/s/346/e/443 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🏗️ Nano GPT是一个简单的Transformer训练代码库，用于从零开始构建一个类似Chachi PT的语言模型。（00:06:01 - 00:07:52） https://favtu.be/timelines/v/kCc8FmEb1nY/s/361/e/472 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🏗 Building a Transformer-based language model with NanoGPT starts with character-level training on a dataset.（00:07:23 - 00:10:11） https://favtu.be/timelines/v/kCc8FmEb1nY/s/443/e/611 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. reading and exploring the data（00:07:52 - 00:09:28） https://favtu.be/timelines/v/kCc8FmEb1nY/s/472/e/568 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 📝 构建Nano GPT的代码可以从头开始编写，了解Transformer模型的工作原理，只需了解Python和基础数学概念。（00:07:52 - 00:13:23） https://favtu.be/timelines/v/kCc8FmEb1nY/s/472/e/803 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. tokenization, train/val split（00:09:28 - 00:14:27） https://favtu.be/timelines/v/kCc8FmEb1nY/s/568/e/867 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 💡 Tokenizing involves converting raw text to sequences of integers, with different methods like character-level or subword tokenizers.（00:10:11 - 00:13:36） https://favtu.be/timelines/v/kCc8FmEb1nY/s/611/e/816 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Thank you Andrej! You’re so passionate about your job. It was am when you started coding. Now it’s dark in here and you still trying to teach! 🙏（00:11:00 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/660/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Around it is very weird that space is the first character but the code for it is "1" not "0" 🤔Pretty sure enumerate shouldn't be doing that...UPD: nvm the first character is actually \n or something（00:12:50 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/770/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🔣 使用字符级的编码方式将文本转换成整数序列，作为Transformer的输入。（00:13:23 - 00:18:01） https://favtu.be/timelines/v/kCc8FmEb1nY/s/803/e/1081 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 📏 Training a Transformer involves working with chunks of data, not the entire dataset, to predict sequences.（00:13:36 - 00:18:43） https://favtu.be/timelines/v/kCc8FmEb1nY/s/816/e/1123 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. data loader: batches of chunks of data（00:14:27 - 00:22:11） https://favtu.be/timelines/v/kCc8FmEb1nY/s/867/e/1331 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 📦 为了提高训练效率，我们将数据分成多个批次，每个批次包含多个文本片段，但这些片段是独立的。（00:18:01 - 00:22:03） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1081/e/1323 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. At you mention that the mini-batches is only done for efficiency reasons, but wouldn't it also help keep the gradients more stable by reducing variance?（00:18:22 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1102/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - ⏩ Transformers process multiple text chunks independently as batches for efficiency in training.（00:18:43 - 00:22:59） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1123/e/1379 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Shouldn't it be len(data) - block_size - 1, because theoretically there is a 1 in a million chance or whatever the total len of the chars is of getting the len(data) - 8 for x and then len(data) - 7 for y and then the last index in data[i+1:i+block_size+1] will be outside the list.（00:19:38 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1178/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Why at do we need to subtract blocksize from len(data)?（00:20:02 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1202/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Am I the only frontend dev here who completely loses understanding around the mark? XD（00:21:00 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1260/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 📊 在这一部分，视频讲解了如何通过构建一个简单的二元语言模型来开始理解 Transformer 模型的基本概念。（00:22:03 - 00:23:14） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1323/e/1394 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. simplest baseline: bigram language model, loss, generation（00:22:11 - 00:34:53） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1331/e/2093 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🧠 Explaining the creation of a token embedding table.（00:22:59 - 00:24:09） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1379/e/1449 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🛠️ 构建了一个 token 嵌入表，用于将输入整数索引转换为嵌入向量，为后续预测字符打下基础。（00:23:14 - 00:24:09） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1394/e/1449 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🎯 Predicting the next character based on individual token identity.（00:24:09 - 00:25:19） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1449/e/1519 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🤖 实现了对每个位置的下一个字符的预测，这里仅使用个别令牌的身份进行预测，而没有考虑上下文。（00:24:09 - 00:25:19） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1449/e/1519 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Hi Andrej, thank you so much for investing your time on sharing this priceless video. I have a question at , when the input to the embedding block is B * T tensor & the output to the embedding block should be called the embeddings for the given tensor.（00:24:13 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1453/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. My note : Continue watching from（00:24:50 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1490/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. It sounds like the transformers are great, but the neutral networl is where you make or break your AI. If thats not encoded properly to already know rules about what it means to be "5" then your SoL（00:24:52 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1492/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. At you I don't get where you used targets ignoring the loss... and the idx how can logits work without targets Can anyone explain it（00:25:00 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1500/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 💡 Using negative log likelihood loss (cross entropy) to measure prediction quality.（00:25:19 - 00:26:44） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1519/e/1604 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🎯 介绍了如何使用负对数似然损失（交叉熵损失）来评估模型预测的质量，并解释了如何计算损失。（00:25:19 - 00:28:22） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1519/e/1702 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🔄 Reshaping logits for appropriate input to cross entropy function.（00:26:44 - 00:28:22） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1604/e/1702 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. At , why can't we write logits = logits.view(B,C,T) and keep targets the same? When I do this the loss value differs and I can't understand why.（00:27:45 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1665/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 💻 Training the model using the optimizer Adam with a larger batch size.（00:28:22 - 00:31:21） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1702/e/1881 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 📉 分析了模型预测质量的改进空间，目前预测结果不够准确，但损失在逐步降低。（00:28:22 - 00:29:17） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1702/e/1757 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. @ Why is the expected nll -ln(1/65) ? how did the ratio 1/65 come about?（00:28:26 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1706/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - never would have I ever expected to get Rick-rolled by Andrej（00:28:27 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1707/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. are you kidding me I get Rick Rolled in a video about LLMs?（00:28:31 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1711/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🚀 实现了模型的生成函数，用于基于之前的上下文预测未来的字符序列。（00:29:17 - 00:36:35） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1757/e/2195 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Is there a difference between categorical sampling and software + multinomial if we're sampling a single item? []（00:30:01 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1801/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Pls correct me if I'm wrong, at logits is of size (B*T, C), not (B, T, C)..（00:30:16 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1816/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. @ logits coming from self should be (B*T,C) right?（00:30:28 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1828/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🏗 Generating tokens from the model by sampling via softmax probabilities.（00:31:21 - 00:34:38） https://favtu.be/timelines/v/kCc8FmEb1nY/s/1881/e/2078 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🛠 Training loop includes evaluation of loss and parameter updates.（00:34:38 - 00:41:23） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2078/e/2483 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. training the bigram model（00:34:53 - 00:38:00） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2093/e/2280 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. -（00:35:45 - 00:42:42） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2145/e/2562 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🔄 通过迭代的训练循环，不断优化模型参数，降低了损失值，从而提高了预测质量。（00:36:35 - 00:37:55） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2195/e/2275 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - how come a specific letter can be followed with various others? If the model is about bigrams, and it has certain constant weights - then one would think that a letter will always lead to the calculation of of the same following letter. Yet they vary producing some long ~random input.（00:37:00 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2220/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. "OK, so we see that we starting to get something at least like reasonable-ish" :,DI love this tutorial! Thank you for your time and passion!（00:37:18 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2238/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. A very nice piece of Vogon poetry at（00:37:32 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2252/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Question: ❓at min you say that with the first bigram model predicts starting only from the previous character, but I see that the first word is POPSousthe.... now, if after the first P comes an O, but after the following P comes an S... where is this variation coming from? Some other people has an answer?（00:37:48 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2268/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🚴‍♂️ 将代码转化为一个脚本，添加了超参数和 GPU 支持，提高了代码的整体可读性和可扩展性。（00:37:55 - 00:41:09） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2275/e/2469 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. port our code to a scriptBuilding the "self-attention"（00:38:00 - 00:42:13） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2280/e/2533 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. port our code to a script（00:38:00 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2280/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. At shouldn't line 115 read logits, loss = m(xb, yb) rather than logits, loss = model(xb, yb). Similarly with line 54?（00:38:50 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2330/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. *cries in doesn't work with AMD gpus*（00:38:57 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2337/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 📄 对模型的训练过程进行了解释，包括数据加载、优化器、训练循环和设备管理。（00:41:09 - 00:42:16） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2469/e/2536 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 📉 Using `torch.no_grad()` for efficient memory usage during evaluation.（00:41:23 - 00:45:59） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2483/e/2759 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. version 1: averaging past context with for loops, the weakest form of aggregation（00:42:13 - 00:47:11） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2533/e/2831 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🎭 展示了模型训练的中间输出，损失值逐渐降低，并生成了一些预测结果。（00:42:16 - 00:43:52） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2536/e/2632 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. ,（00:42:42 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2562/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🧠 自注意力机制（self-attention）是一种关键的机制，用于加强序列中不同位置的信息交互。（00:43:52 - 00:44:48） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2632/e/2688 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🧮 使用矩阵乘法和加权求和，可以在序列中实现不同位置的平均汇聚，以捕捉历史信息。（00:44:48 - 00:47:22） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2688/e/2842 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. when he says we take the average. is he implying that we take the average of the token mapped numbers? if yes, how would that remotely help?（00:45:20 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2720/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🧮 Tokens are averaged out to create a one-dimensional vector for efficient processing（00:45:59 - 00:47:22） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2759/e/2842 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. the trick in self-attention: matrix multiply as weighted aggregation（00:47:11 - 00:51:54） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2831/e/3114 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🔢 Matrix multiplication can efficiently perform aggregations instead of averages（00:47:22 - 00:50:27） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2842/e/3027 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. ➗ 利用矩阵乘法的数学技巧，可以高效地进行加权平均，改进了信息汇聚的速度和效率。（00:47:22 - 00:52:02） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2842/e/3122 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Just that little time you take to explain a trick at shows how great of a teacher you are, thanks a lot for this video !（00:47:30 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/2850/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🔀 Manipulating elements in a multiplying matrix allows for incremental averaging based on 'ones' and 'zeros'（00:50:27 - 00:54:51） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3027/e/3291 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. version 2: using matrix multiply（00:51:54 - 00:54:42） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3114/e/3282 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🔄 利用位置嵌入（positional embeddings）和查询/键（query/key）机制，自注意力机制可以根据位置和内容加权汇聚信息。（00:52:02 - 00:58:27） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3122/e/3507 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. version 3: adding softmax（00:54:42 - 00:58:26） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3282/e/3506 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🔄 Introduction of softmax helps in setting interaction strengths and affinities between tokens（00:54:51 - 00:58:27） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3291/e/3507 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. I think there is a mistake at time . Andrej said that "tokens from the past cannot communicate". I think the correct version is "tokens from the future cannot communicate".（00:56:59 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3419/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Oops "tokens from the _future_ cannot communicate", not "past". Sorry! :)（00:57:00 - 01:20:05） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3420/e/4805 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Did you mean the tokens from the future cannot communicate, not from the past?（00:57:01 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3421/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. At line 2 and 3 in the code cell are contradicting each other that is"w = torch.zeros((T, T)) # How much token from the past do we want the aggregate ofw = w.masked_fill(w_tril == 0, float('-inf')) # Tokens from the past cannot communicate"now if we want tokens don't communicate then why took average of previous tokens first and then cancelling it if we don't do it, It will get us better accuracy at high computational cost..（00:57:02 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3422/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. minor code cleanup（00:58:26 - 01:00:18） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3506/e/3618 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🧠 Weighted aggregation of past elements using matrix multiplication aids in self-attention block development（00:58:27 - 01:02:07） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3507/e/3727 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. ⚙️ 自注意力机制可以用于建立用于序列建模的模块，通过查询和键的互动来增强模型对不同位置的关注和理解。（00:58:27 - 01:02:07） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3507/e/3727 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. positional encoding（01:00:18 - 01:02:00） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3618/e/3720 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. hi how you build the Ubuntu environment in Mac and run in VScode editor ? Can you record this tutorial?（01:00:20 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3620/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Watch the video once. Watch it again. Watch it a few more times. Then watch - 20 times, melting your brain trying to keep track of tensor dimensions. This is a *dense* video - amazing how much detail is packed into 2 hours... thanks for this Andrej!（01:01:00 - 01:11:00） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3660/e/4260 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. AM until I saw the message at :)（01:01:58 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3718/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. THE CRUX OF THE VIDEO: version 4: self-attention（01:02:00 - 01:11:38） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3720/e/4298 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🔂 Self-attention involves emitting query and key vectors to determine token affinities and weighted aggregations（01:02:07 - 01:05:13） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3727/e/3913 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🎯 自注意力机制中，每个位置产生查询（query）和键（key），并通过查询与键的点积计算位置之间的关联程度，实现信息汇聚。（01:02:07 - 01:06:37） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3727/e/3997 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Note toe self, when you come back to watch this again: Jist of self attention at（01:03:35 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3815/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. the top and most important part. What a great guy!（01:03:53 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3833/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Can you guys explain in detail on "head_size" hyper-parameter mentioned at timestamp , what is the difference between this head_size and the Multi head diagram present in Attention is all you need（01:05:10 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3910/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🎭 Implementing a single head of self-attention involves computing queries and keys and performing dot products for weighted aggregations.（01:05:13 - 01:10:10） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3913/e/4210 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🧠 自注意力机制是 Transformer 中的重要组成部分，用于在序列数据中建立不同 token 之间的关联。（01:06:37 - 01:07:08） https://favtu.be/timelines/v/kCc8FmEb1nY/s/3997/e/4028 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. You introduced nn.Linear() at , but that confused me. So, I looked into the PyTorch nn.Linear documentation. Still, I was not clear. The ambiguous point is that it looks like the following are identical calls:key = nn.Linear(C, head_size, bias=False)value = nn.Linear(C, head_size, bias=False)Then I expect the dot product of key(x), value(x) to be the same as the dot product of key(x), key(x).Thanks to your collab code, I found that when I changed the seed value, the key(x) and value(x) changed. That means Linear()'s matrix is randomly initialized. However, the documentation is not clear to me. After I noticed the matrix initialization was random, I saw nn.Linear's documentation says "The values are initialized from U(−\sqrt{k},\sqrt{k})". So, I think now that U is a random uniform distribution function. But I am really a beginner in AI. Your lecture is the first real course in AI. But now the rest is clear.Other beginners (like me) may struggle to understand that part.（01:07:05 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4025/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🔄 自注意力中的加权聚合是基于查询与键之间的点积计算的，通过 Softmax 函数将不同 token 之间的关联度标准化。（01:07:08 - 01:07:34） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4028/e/4054 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🔤 加权聚合的权重现在是数据依赖的，因为每个批次元素包含不同位置的不同 token，从而实现个性化的关联。（01:07:34 - 01:08:34） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4054/e/4114 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. the main explanation of keys X querys is at . My concentration is so poor, I kept falling asleep every 5 minutes, but I kept on trying. Eventually after 7 hours of watching, dropping off, watching, the penny dropped. This bloke is a nice person for doing this for us（01:07:50 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4070/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. at , why is that "up to four"? what does the 'four' mean?（01:07:59 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4079/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. ➕ 查询与键的高亲和性通过 Softmax 函数与值进行加权聚合，这使得有关 token 的信息被有效聚合并更新。（01:08:34 - 01:09:28） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4114/e/4168 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🔀 为避免未来 token 影响当前 token，我们使用了上三角矩阵的掩码，确保只有过去的 token 能够通信。（01:09:28 - 01:10:51） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4168/e/4251 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🧠 Self-attention mechanism aggregates information using key, query, and value vectors.（01:10:10 - 01:11:46） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4210/e/4306 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🤝 每个自注意力头生成了三个线性映射：查询、键和值，它们分别用于产生高维度的关联矩阵，聚合的向量以及最终输出的向量。（01:10:51 - 01:20:57） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4251/e/4857 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. we see the key, query and value matrix are created using nn.linear modeling. With same input for all 3, it should give same out. Which means Key, Query and value should be same for given text matrix.What difference between in terms of calculation..（01:10:55 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4255/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. about（01:11:00 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4260/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. "That is basically self attention mechanism. It is what it does". Andrej's expression says that this simple piece of code does all the magic. :)（01:11:30 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4290/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. note 1: attention as communication（01:11:38 - 01:12:46） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4298/e/4366 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🛠 Attention is a communication mechanism between nodes in a directed graph.（01:11:46 - 01:12:56） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4306/e/4376 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. note 2: attention has no notion of space, operates over sets（01:12:46 - 01:13:40） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4366/e/4420 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🔍 Attention operates over a set of vectors without positional information, requiring external encoding.（01:12:56 - 01:13:53） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4376/e/4433 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. note 3: there is no communication across batch dimension（01:13:40 - 01:14:14） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4420/e/4454 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 💬 Attention mechanisms facilitate data-dependent weighted sum aggregation.（01:13:53 - 01:15:46） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4433/e/4546 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. note 4: encoder blocks vs. decoder blocks（01:14:14 - 01:15:39） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4454/e/4539 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. note 5: attention vs. self-attention vs. cross-attention（01:15:39 - 01:16:56） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4539/e/4616 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🤝 Self-attention involves keys, queries, and values from the same source, while cross-attention brings in external sources.（01:15:46 - 01:17:50） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4546/e/4670 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. note 6: "scaled" self-attention. why divide by sqrt(head_size)Building the Transformer（01:16:56 - 01:19:11） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4616/e/4751 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Actually, forget what I said. I thought you were talking about . The number of embedding dimensions is divided by the number of heads to improve efficiency. The embedding vectors are transformed into smaller K, Q, V vectors and multiple heads are used so that they can operate in parallel.（01:16:56 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4616/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🧮 Scaling the attention values is crucial for network optimization by controlling variance.（01:17:50 - 01:21:27） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4670/e/4887 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. inserting a single self-attention block to our network（01:19:11 - 01:21:59） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4751/e/4919 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Do you really need to register the mask as a buffer?（01:19:36 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4776/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. oops, should have scaled by the head_size not the embed_dim C at（01:19:45 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4785/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Guys is anyone else having the issue of the loss falling too quickly by the time you reach in the video.the first step is at a loss of about 4.8 and the 500th step goes straight to 0.5..any help would be greatly appreciated.（01:20:00 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4800/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Oops I should be using the head_size for the normalization, not C（01:20:05 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4805/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. at shouldn't wei be normalized by square root of head_size instead of square root of C ?（01:20:05 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4805/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Thank you Andrej! At , shouldn't the code say (B, T, Head Size) on lines 73, 74, and 81? Or is head size = C?（01:20:10 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4810/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. at , line 76, why is the scale here C **-0.5? I thought the scale should be sqrt of d_k, the feature dimension of the key, while C is the feature dimension of the embedding.（01:20:10 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4810/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🚀 多头自注意力允许同时使用多个独立的关联通道，以便各种 token 可以进行不同类型的关联和聚合，从而提高模型性能。（01:20:57 - 01:25:42） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4857/e/5142 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 💡 Implementing multi-head attention involves running self-attention in parallel and concatenating results for improved communication channels.（01:21:27 - 01:26:36） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4887/e/5196 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. multi-headed self-attention（01:21:59 - 01:24:25） https://favtu.be/timelines/v/kCc8FmEb1nY/s/4919/e/5065 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. For anyone getting an error after adding multihead attention block atI think current pytorch is looking for explicit integers for the head_size of MultiHeadAttention()this fixed my error:self.self_attention_heads = MultiHeadAttention(4, int(n_embd/4))（01:23:46 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5026/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. feedforward layers of transformer block（01:24:25 - 01:26:48） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5065/e/5208 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🔃 将多头自注意力与前馈神经网络串联，实现了 Transformer 模型的一个块。它在通信与计算之间进行切换，提升了建模能力。（01:25:42 - 01:27:18） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5142/e/5238 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - ⚙ Integrating communication and computation in Transformer blocks enhances network performance.（01:26:36 - 01:28:29） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5196/e/5309 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. residual connections（01:26:48 - 01:32:51） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5208/e/5571 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🧠 多头注意力与编码尺寸关系，以及通道尺寸的关联。（01:27:18 - 01:27:47） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5238/e/5267 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🏗️ 堆叠多个块以实现Transformer结构，同时注意深层神经网络的优化问题。（01:27:47 - 01:28:15） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5267/e/5295 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. ⚙️ 使用残差连接（skip connections）解决深度网络优化问题，使梯度能够流经多层网络。（01:28:15 - 01:28:29） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5295/e/5309 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - while playing Rocket League（01:28:22 - 01:31:43） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5302/e/5503 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🔄 Residual connections aid in optimizing deep networks by facilitating gradient flow and easier training.（01:28:29 - 01:32:16） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5309/e/5536 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🧬 利用残差连接的超级高速梯度路径，将梯度直接从监督信号传播到输入，然后通过残差块进行优化。（01:28:29 - 01:29:26） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5309/e/5366 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🚀 利用Layer Normalization（层归一化）优化深度神经网络，保持网络在深层时的可训练性。（01:29:26 - 01:30:38） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5366/e/5438 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🔄 实现残差连接和投影以优化Transformer结构，并解释实现中的相关细节。（01:30:38 - 01:32:43） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5438/e/5563 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - while playing rocket league（01:31:43 - 01:38:12） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5503/e/5892 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🧠 Adjusting Channel sizes in the feed forward network can affect validation loss and lead to potential overfitting.（01:32:16 - 01:32:58） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5536/e/5578 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🌟 使用Layer Norm和残差连接显著提升Transformer训练性能和生成效果。（01:32:43 - 01:37:55） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5563/e/5875 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. layernorm (and its relationship to our previous batchnorm)（01:32:51 - 01:37:49） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5571/e/5869 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🔧 Layer Norm in deep neural networks helps optimize performance, similar to batch normalization but normalizes rows instead of columns.（01:32:58 - 01:35:19） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5578/e/5719 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 📐 Implementing Layer Norm in a Transformer involves reshuffling layer norms in pre-norm formulation for better results.（01:35:19 - 01:37:12） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5719/e/5832 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Personal note: resume on "scaling the model":（01:37:00 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5820/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 📈 Scaling up a neural network model by adjusting hyperparameters like batch size, block size, and learning rate can greatly improve validation loss.（01:37:12 - 01:39:30） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5832/e/5970 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. scaling up the model! creating a few variables. adding dropoutNotes on Transformer（01:37:49 - 01:42:39） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5869/e/6159 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Personal bookmark:（01:37:49 - 01:28:22） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5869/e/5302 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 📈 通过调整超参数、增加网络规模和添加Dropout等手段，显著提升生成文本的质量。（01:37:55 - 01:41:24） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5875/e/6084 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - while playing rl（01:38:12 - 01:42:39） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5892/e/6159 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🔒 Using Dropout as a regularization technique helps prevent overfitting when scaling up models significantly.（01:39:30 - 01:51:21） https://favtu.be/timelines/v/kCc8FmEb1nY/s/5970/e/6681 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Thanks a lot for this revelation! I have one question on : How is the final number of parameters (10M) exactly calculated? Isn't the FFN receiving 64 inputs from attention and having 6 layers, that would make 64^6 parameters already, which is way more. I think I misunderstood the model's architecture at some point. Could someone help?（01:40:01 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6001/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Why do we divide the number of dimensions by the number of heads? @（01:40:15 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6015/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Just for reference. This training took 3 hours, 5 minutes on an 2020 M1 Macbook Air. You can use the "mps" device instead of cuda or cpu.（01:41:03 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6063/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🎭 解释了Nano GPT中的代码组织结构，包括训练和模型部分，以及实现解码器的细节。（01:41:24 - 01:46:17） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6084/e/6377 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. encoder vs. decoder vs. both (?) Transformers（01:42:39 - 01:46:22） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6159/e/6382 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Difference and / or relations between encoder and decoder（01:44:53 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6293/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🔍 对比了解码器与编码器解码器结构，说明解码器在生成无条件文本时的用途。（01:46:17 - 01:48:49） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6377/e/6529 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. super quick walkthrough of nanoGPT, batched multi-headed self-attention（01:46:22 - 01:48:53） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6382/e/6533 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. Unintentional pun: "now we have a fourth dimension, which is the heads, and so it gets a lot more hairy"（01:47:52 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6472/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 📋 我们现在可以将所学内容应用于训练 ChatGPT。训练 ChatGPT 分为两个阶段：预训练阶段和微调阶段。（01:48:49 - 01:49:04） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6529/e/6544 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. back to ChatGPT, GPT-3, pretraining vs. finetuning, RLHF（01:48:53 - 01:54:32） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6533/e/6872 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🤖 预训练阶段类似我们之前做的，但规模更大。我们预训练的模型大约有1,000万参数，而GPT-3中最大的模型有1750亿参数。（01:49:04 - 01:49:32） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6544/e/6572 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🧩 GPT-3的词汇表不同，它使用子词块，所以我们的数据集大约有30万个标记，而GPT-3预训练时使用了3000亿个标记。（01:49:32 - 01:50:11） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6572/e/6611 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🛠️ GPT-3的架构与我们的实现非常相似，但规模更大，预训练数据集更庞大，通常需要数千个GPU来训练。（01:50:11 - 01:50:52） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6611/e/6652 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - rl（01:50:15 - 00:34:59） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6615/e/2099 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. The d_model for GPT-3 13B is 5140 while with n_heads = 40, d_head = 128. n_heads*d_head = 5120. Wouldn't this cause shape mismatches. Its weird, all the other models seems to comply except this one.（01:50:34 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6634/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 🔍 GPT-3的预训练阶段是在互联网大规模数据上进行的，与微调阶段的任务不同，微调会将模型从文档生成者调整为一个有问题回答能力的助手。（01:50:52 - 01:51:34） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6652/e/6694 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. - 🌐 ChatGPT undergoes pre-training on internet data followed by fine-tuning to become a question-answering assistant by aligning model responses with human preferences.（01:51:21 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6681/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. 📚 微调阶段包括数据收集、奖励模型训练和策略优化等步骤，以将模型从文档生成者微调为问题回答助手，但微调过程更加复杂且不太容易复制。（01:51:34 - 01:56:20） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6694/e/6980 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. conclusions（01:54:32 - 00:57:00） https://favtu.be/timelines/v/kCc8FmEb1nY/s/6872/e/3420 Wed, 18 Jan 23 01:33:27 +0900 Let's build GPT: from scratch, in code, spelled out. intro（00:00:00 - 00:01:40） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/0/e/100 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet starter code walkthrough（00:01:40 - 00:06:56） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/100/e/416 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet *Starter Code Walkthrough (****)*（00:01:43 - 00:09:19） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/103/e/559 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet Andrej is scrolling through the architecture, however not commenting why the first Linear layer has deactivated biases?I saw this in couple of other projects, can smbd clarify why or say where should I look to find an answer?Thank you（00:05:40 - 00:56:22） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/340/e/3382 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet let’s fix the learning rate plot（00:06:56 - 00:09:16） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/416/e/556 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet When I did the mean() trick at ~ I left out an audible gasp! That was such a neat trick, going to use that one in the future（00:08:50 - 00:56:22） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/530/e/3382 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet pytorchifying our code: layers, containers, torch.nn, fun bugsimplementing wavenet（00:09:16 - 00:17:11） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/556/e/1031 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet *PyTorchifying Our Code: Layers, Containers, Torch.nn, Fun Bugs (****)*- Embedding table and view operations are encapsulated into custom Embedding and Flatten modules.- A Sequential container is created to organize layers, similar to torch.nn.Sequential.- The forward pass is simplified using these new modules and container.- A bug related to BatchNorm in training mode with single-example batches is identified and fixed.（00:09:19 - 00:17:12） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/559/e/1032 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet @ Why not just call torch.flatten(start_dim, end_dim) inside Flatten(start_dim, end_dim=-1) layer? To use it in your particular case just create Flatten(1) layer（00:11:18 - 00:56:22） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/678/e/3382 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet overview: WaveNet（00:17:11 - 00:19:33） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/1031/e/1173 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet *Overview: WaveNet (****)*- The limitations of the current MLP architecture are discussed, particularly the issue of squashing information too quickly.- The video introduces the WaveNet architecture, which progressively fuses information in a tree-like structure.- The concept of dilated causal convolutions is briefly mentioned as an implementation detail for efficiency.（00:17:12 - 00:19:35） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/1032/e/1175 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet dataset bump the context size to 8（00:19:33 - 00:19:55） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/1173/e/1195 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet *Implementing WaveNet (****)*- The dataset block size is increased to 8 to provide more context for predictions.- The limitations of directly scaling up the context length in the MLP are highlighted.- A hierarchical model is implemented using FlattenConsecutive layers to group and process characters in pairs.- The shapes of tensors at each layer are inspected to ensure the network functions as intended.- A bug in the BatchNorm1D implementation is identified and fixed to correctly handle multi-dimensional inputs.（00:19:35 - 00:45:25） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/1175/e/2725 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet re-running baseline code on block_size 8（00:19:55 - 00:21:36） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/1195/e/1296 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet implementing WaveNet（00:21:36 - 00:37:41） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/1296/e/2261 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet Does anyone know how visualize the dimensions of the tensors that are treated from around ? I'm having a really hard time to keep up with what's what. Thanks!（00:23:45 - 00:56:22） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/1425/e/3382 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet training the WaveNet: first pass（00:37:41 - 00:38:50） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/2261/e/2330 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet At , it sounds like we compared two architectures, both with 22k parameters and an 8 character window:* 1 layer, full connectivity* 3 layers, tree-like connectivityIn a single layer, full connectivity outperforms partial connectivity.But partial connectivity uses fewer parameters, so we can afford to build more layers.（00:38:00 - 00:56:22） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/2280/e/3382 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet fixing batchnorm1d bug（00:38:50 - 00:45:21） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/2330/e/2721 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet re-training WaveNet with bug fix（00:45:21 - 00:46:07） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/2721/e/2767 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet *Re-training the WaveNet with Bug Fix (****)*- The network is retrained with the BatchNorm1D bug fix, resulting in a slight performance improvement.- The video notes that PyTorch's BatchNorm1D has a different API and behavior compared to the custom implementation.（00:45:25 - 00:46:07） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/2725/e/2767 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet With the batchnorm bug at around , why does it still work?, if the batch norm is producing the wrong shape why is there not an error?（00:46:00 - 00:56:22） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/2760/e/3382 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet scaling up our WaveNetconclusions（00:46:07 - 00:46:58） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/2767/e/2818 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet *Scaling up Our WaveNet (****)*- The number of embedding and hidden units are increased, leading to a model with 76,000 parameters.- Despite longer training times, the validation performance improves to 1.993.- The need for an experimental harness to efficiently conduct hyperparameter searches is emphasized.（00:46:07 - 00:46:59） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/2767/e/2819 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet experimental harness（00:46:58 - 00:47:44） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/2818/e/2864 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet *Experimental Harness (****)*- The lack of a proper experimental setup is acknowledged as a limitation of the current approach.- Potential future topics are discussed, including:- Implementing dilated causal convolutions- Exploring residual and skip connections- Setting up an evaluation harness- Covering recurrent neural networks and transformers（00:46:59 - 00:55:27） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/2819/e/3327 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet WaveNet but with “dilated causal convolutions”（00:47:44 - 00:51:34） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/2864/e/3094 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet The sentence that Anderej said at made me realize something, something very deep. 🔥（00:49:26 - 00:56:22） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/2966/e/3382 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet torch.nn（00:51:34 - 00:52:28） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/3094/e/3148 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet the development process of building deep neural nets（00:52:28 - 00:54:17） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/3148/e/3257 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet going forward（00:54:17 - 00:55:26） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/3257/e/3326 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet improve on my loss! how far can we improve a WaveNet on this data?（00:55:26 - 00:56:22） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/3326/e/3382 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet *Improve on My Loss! How Far Can We Improve a WaveNet on This Data? (****)*- The video concludes with a challenge to the viewers to further improve the WaveNet model's performance.- Suggestions for exploration include:- Trying different channel allocations- Experimenting with embedding dimensions- Comparing the hierarchical network to a large MLP- Implementing layers from the WaveNet paper- Tuning initialization and optimization parameters（00:55:27 - 00:56:22） https://favtu.be/timelines/v/t3YJ5hKiMQ0/s/3327/e/3382 Mon, 21 Nov 22 09:32:48 +0900 Building makemore Part 5: Building a WaveNet intro: why you should care & fun history（00:00:00 - 00:07:26） https://favtu.be/timelines/v/q8SA3rM6ckI/s/0/e/446 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja @ "it was barely a programming language"（00:04:38 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/278/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja starter code（00:07:26 - 00:13:01） https://favtu.be/timelines/v/q8SA3rM6ckI/s/446/e/781 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja Love your lectures, they are crystal clear. From , I only find the notation dlogprobs (et similia) a bit misleading, since it denotes the derivative of the loss with respect to the parameters logprobs. I would use something more verbose like dloss_dlogprobs. However, I understand you did it for coherence with torch.（00:12:37 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/757/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja exercise 1: backproping the atomic compute graph（00:13:01 - 01:05:17） https://favtu.be/timelines/v/q8SA3rM6ckI/s/781/e/3917 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja ) for the full batch. Whereas in your answer in the videoat it's of size (32,27) only. Can you please clear this confusion for me Andrej? I think there's some fundamental flaw in my understanding 😭😭Is it because in the end we are calling .backward() on a scalar value? 😭（00:20:00 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/1200/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja are logprobs and logits same? at（00:20:00 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/1200/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja ) At time - if probs are very close to 1, that doesn't mean that the network is predicting the next character correctly. If it's close to 1 and its corresponding gradient from dlogprobs is non-zero, only then that means that the network does the prediction correctly.（00:20:20 - 01:30:30） https://favtu.be/timelines/v/q8SA3rM6ckI/s/1220/e/5430 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja andrej fard（00:27:25 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/1645/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja explained on（00:36:30 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/2190/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja Is there a disadvantage to using (logits == logit_maxes).float() to pick out the maximum indices at ?（00:40:54 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/2454/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja Thats so Cute. 😆（00:43:40 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/2620/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja sprinkling Andrej magic through out the video - had me cracking at（00:43:40 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/2620/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja At , Low Budget Production LOL（00:45:25 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/2725/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja At around instead of differentiating the explicit expanded form of a matrix multiplication and then realizing that the result is again some matrix multiplication you can actually show more generally that the backprop operation of a linear transformation is always the Hermitian adjoint of that transformation. For matrix multiplication the Hermitian adjoint is just given by multiplication of the transposed matrix. This is especially useful for more complicated transformations like convolutions, just imagine doing these calculations on the completely written out expression of a convolution. This also explains the duality between summation and replication mentioned at（00:47:00 - 01:09:50） https://favtu.be/timelines/v/q8SA3rM6ckI/s/2820/e/4190 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja I arrived at dh just figuring out by the size of the matrix, and then I continued with your video and you just did all the derivatives and I taught... I am so dumb, I should I have done that, but then you say " now I tell you a secret I normally do... .... hahahahahhaha（00:49:45 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/2985/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja if you scroll down, Wolfram Alpha provides 1 - x^2 + 2/3x^4 + O(x^5) as series expansion at x=0 of the derivative of tanh(x), which is the same as the series expansion for 1-tanh(x)^2.（00:53:55 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/3235/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja brief digression: bessel’s correction in batchnorm（01:05:17 - 01:26:31） https://favtu.be/timelines/v/q8SA3rM6ckI/s/3917/e/5191 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja your attention to detail here on the variance of arrays is out of this world（01:06:20 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/3980/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja The reason for using biased variance in training and unbiased during inference(running var estimation) is that during the training in one mini-batch we don't care about the complete dataset. The mini-batch is enough as it is the one at the moment we are working on. In the code also you are using the mean and var of that moment to run batchnorm. But during inference we need the mean and variance of the complete data, that is what the bessel's correction is for. If we have the access to the complete data we don't need to use the Bessel's correction, we have the full data. But if we are using small sample(mini-batch) to estimate the variance of the complete data we need Bessel's correction. If we used direct variance calculation instead of this running var we can completely skip the Bessel's correction.（01:08:17 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/4097/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja best part（01:09:44 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/4184/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja since the adjoint of summation is replication (and vice versa).（01:09:50 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/4190/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja i've noticed that althoughdbnvar/(n-1) # (1, 64) doesn't have the same size as the bndiff2 term (32, 64), it still works fine during the backprop, because (1,64) vector broadcasts well on (32,64).And such solution is more optimal from the perspective of storage and calculation（01:12:06 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/4326/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja around , a simpler approach might be to just directly multiply like this: dbndiff2 = 1/(n-1) * dbnvar（01:12:20 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/4340/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja At around dbnmeani should probably have keepdim=True, since otherwise you're removing the row dimension making it of shape [64], while bnmeani was originally [1, 64]. But I guess it still magically works because of broadcasting in the backprop and in the cmp :)（01:16:00 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/4560/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja Such a great video for really understanding the detail under the hood! And lol at the momentary disappointment at just before realizing the calculation wasn't complete yet 😂（01:16:20 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/4580/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja /n * dbnmeani``` during . Makes no difference mathematically, but theres nothing like finding oopsies in your code :P（01:18:20 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/4700/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja I believe the loop implementing the final derivative at can be vectorized if you just rewrite the selection operation as a matrix operation, then do a matmul derivative like done elsewhere in the video:（01:24:21 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5061/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja Thank you!Also, my implementation for dC atdC = torch.zeros_like(C)dC[Xb] += demb（01:24:21 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5061/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja One-liner for dC ():dC = (F.one_hot(Xb, num_classes=C.shape[0]).unsqueeze(-1) * demb.unsqueeze(2)).sum((0, 1))（01:24:22 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5062/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja I was able to accumulate the dC without a "for" loop using this code:dC = torch.zeros_like(C)dC.index_add_(0, Xb.flatten(), demb.flatten(0, 1))（01:24:25 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5065/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja I struggled through everything to make sure I found answers before seeing the video solution, the one-line solution I got for dC was:dC = F.one_hot(Xb.view(-1), num_classes=C.shape[0]).float().T @ demb.view(-1, C.shape[-1])Don't ask me for an intuitive explanation, I just fiddled until something worked... (sort-of inspired by how earlier on in the series you showed that C[Xb] is just a more efficient version of using F.one_hot with matrix multiplication)Also, for whatever reason I can't get dhpreact to be exact, only approximate, even using your exact same code to calculate it? So I just ended up doing dhpreact = hpreact.grad#(1.0 - h**2) * dh to make sure its effect didn't cascade further down the gradient calculations. Any idea why this would be the case?（01:24:26 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5066/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja I managed to come up with a *vectorized* solution and it's just one line of code!*dC = F.one_hot(Xb.reshape(-1), num_classes=27).float().T @ demb.reshape((-1, n_emb))*（01:24:30 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5070/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja Here is the better implementation of the code:（01:25:00 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5100/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja Optimised dC calculation() instead of the for loop（01:25:00 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5100/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja To eliminate the for loop at time , I found this after a little searching. Very little experience with pytorch, so take with a grain of salt:dembflat = demb.view(-1,10)Xbflat = Xb.view(-1)dC = torch.zeros_like(C)dC.index_add_(0,Xbflat,dembflat)（01:25:21 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5121/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja P.S.: dC can be done with dC.index_add_(0, Xb.view(-1), demb.view(-1, 10)) ;)（01:25:47 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5147/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja : To backprop through the embedding matrix C, I used the following quick code, which does not need a for loop:（01:26:00 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5160/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja can be vectorized using: dC = dC.index_add_(0, Xb.view(-1), demb.view(-1, C.shape[1]))（01:26:00 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5160/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja So great videos, thank you so much! I tried to simplify dC (at in the video), but failed after some time, so asked chatgpt, and here is the shiny simple result:（01:26:08 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5168/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja exercise 2: cross entropy loss backward pass（01:26:31 - 01:36:37） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5191/e/5797 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja at (just under the separation line for i≠y v i=y)?I understand from the above line that we are looking for the derivative of e^ly / Σe^lj. So, when we consider the denominator we would get e^ly * -(Σe^lj)^-2 = -e^ly / (Σe^lj)^2 but the solution multiplies it by e^li which I do not quite get. Cheers!（01:30:11 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5411/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja The calculus at is way too complicated. Start with -log( e^l_i/sum_j e^l_j ) = -l_i + log(sum_j e^l_j) before you differentiate. d -l_i/dl_y = -1 if i=y of course and d -log(sum_j e^l_j)/d l_y = -e^l_y / sum_j e^l_j = softmax(l_y) and you're done.（01:30:13 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5413/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja At (exercise 2, near the end, while deriving dlogits for i != y): why did you substitute 0 for e**lj ?（01:30:26 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5426/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja ) At about - I think the gradient that you calculate is for norm_logits and not for logits. It looks like they are approximately equal by chance. I think this is the correct implementation:dnorm_logits = F.softmax(norm_logits, 1)dnorm_logits[range(n), Yb] -= 1dnorm_logits /= ndlogit_maxes = -dnorm_logits.sum(1, keepdim=True)dlogits = dnorm_logits.clone()dlogits[range(n), logits.max(1).indices] += dlogit_maxes.view(-1)（01:30:30 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5430/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja I'm really confused about calculations at (the lower part of the paper about `if i <> j` etc). It says 'product rule, power rule, ..." How do I use product rule to take a derivative of Softmax?PS I asked ChatGPT and it explained to me that I need to use Quotient rule :)（01:30:38 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5438/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja Question: Why is this () true not only for dlogits, but also for dW2, db2, db1, and not true for dW1?（01:33:59 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5639/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja He really made me realize something at it kicked in 🔥（01:36:05 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5765/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja exercise 3: batch norm layer backward pass（01:36:37 - 01:50:02） https://favtu.be/timelines/v/q8SA3rM6ckI/s/5797/e/6602 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja in the WHOA:) part, should there be a "-" in front of 2? although it doesn't really matter as the final result is 0. but why is it?（01:43:08 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/6188/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja Question: At , you conclude in the last derivation step that d sigma^2 / d x_i = 2 / (m-1) * (x_i- mu). This would be correct if mu were just a constant, but in fact, mu is also a function of x_i: mu(x_i) = 1/m. So how does this cancel out so that you still end up with your simple expression?（01:45:54 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/6354/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja At the camera start giving up... so do I... 🤣（01:47:50 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/6470/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja i'm not totaly sure that this is a good solution to calculate the derivative using "bnraw" variable, since it is calculated in the later steps of BN. Thus, there's no use in hpreact_fast as we have to do all the same arithmetics in parallel in order to fetch bnraw.My solution is not the best one, but still:（01:50:00 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/6600/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja exercise 4: putting it all together（01:50:02 - 01:54:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/6602/e/6864 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja outro（01:54:24 - 01:55:24） https://favtu.be/timelines/v/q8SA3rM6ckI/s/6864/e/6924 Wed, 12 Oct 22 02:56:19 +0900 Building makemore Part 4: Becoming a Backprop Ninja intro（00:00:00 - 00:01:22） https://favtu.be/timelines/v/P6sfmUTpUmc/s/0/e/82 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm [-1. Implementing and refactoring neural networks for language modeling（00:00:00 - 00:03:21） https://favtu.be/timelines/v/P6sfmUTpUmc/s/0/e/201 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Continuing makemore implementation with multilayer perceptron for character-level language modeling, planning to move to larger neural networks.（00:00:30 - 00:00:31） https://favtu.be/timelines/v/P6sfmUTpUmc/s/30/e/31 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Understanding neural net activations and gradients in training is crucial for optimizing architectures.（00:01:03 - 00:02:06） https://favtu.be/timelines/v/P6sfmUTpUmc/s/63/e/126 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm starter code（00:01:22 - 00:04:19） https://favtu.be/timelines/v/P6sfmUTpUmc/s/82/e/259 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-11,000 parameters over 200,000 steps, achieving train and val loss of 2.16.（00:02:06 - 00:02:46） https://favtu.be/timelines/v/P6sfmUTpUmc/s/126/e/166 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Refactored code to optimize neural net with（00:02:46 - 00:03:03） https://favtu.be/timelines/v/P6sfmUTpUmc/s/166/e/183 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm [-2. Efficiency of torch.no_grad and neural net initialization issues（00:03:22 - 00:14:22） https://favtu.be/timelines/v/P6sfmUTpUmc/s/202/e/862 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Using torch.nograd decorator to prevent gradients computation.（00:03:28 - 00:03:22） https://favtu.be/timelines/v/P6sfmUTpUmc/s/208/e/202 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Using torch's no_grad makes computation more efficient by eliminating gradient tracking.（00:04:00 - 00:04:22） https://favtu.be/timelines/v/P6sfmUTpUmc/s/240/e/262 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm fixing the initial loss（00:04:19 - 00:12:59） https://favtu.be/timelines/v/P6sfmUTpUmc/s/259/e/779 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-27, rapidly decreases to 1 or 2.（00:04:22 - 00:04:50） https://favtu.be/timelines/v/P6sfmUTpUmc/s/262/e/290 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Initial loss (): High initial loss (e.g., 27) indicates improper network initialization.- Softmax logits should be close to zero at initialization to produce a uniform probability distribution and expected loss.- This avoids confident mispredictions and the "hockey stick" loss curve.（00:04:25 - 00:09:28） https://favtu.be/timelines/v/P6sfmUTpUmc/s/265/e/568 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Network initialization causes high loss of（00:04:50 - 00:05:00） https://favtu.be/timelines/v/P6sfmUTpUmc/s/290/e/300 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-27 characters, with roughly 1/27 probability for each.（00:05:00 - 00:05:32） https://favtu.be/timelines/v/P6sfmUTpUmc/s/300/e/332 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]At initialization, the model aims for a uniform distribution among（00:05:32 - 00:05:49） https://favtu.be/timelines/v/P6sfmUTpUmc/s/332/e/349 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm Hi Andrej, Thank you for the amazing set of lectures which elucidate multiple aspects of training a ML model. In the video you mention that at the beginning of training, we expect the NN to have all equal probabilities I.e. 1/27 which implies that all logins should be close to 0. Using this logic you arrive at the fact that weight matrices should be initialized close to 0. How does one think about this for regression problems like autoencoders? What would a “good” starting output be? Is it still all zeros?（00:06:06 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/366/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Neural net creates skewed probability distributions leading to high loss.（00:06:19 - 00:12:08） https://favtu.be/timelines/v/P6sfmUTpUmc/s/379/e/728 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm At , it's a bit subtle why it's called a loss because it's not immediately apparent with respect to what it is a loss. It seems it's the loss resulting from choosing the character having index i given the probability distribution stored in the tensor.（00:07:06 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/426/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Scaling down weights of the output layer can achieve this ().（00:09:28 - 00:13:09） https://favtu.be/timelines/v/P6sfmUTpUmc/s/568/e/789 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-2.12-2.16（00:12:08 - 00:12:36） https://favtu.be/timelines/v/P6sfmUTpUmc/s/728/e/756 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Loss at initialization as expected, improved to（00:12:36 - 00:14:24） https://favtu.be/timelines/v/P6sfmUTpUmc/s/756/e/864 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm fixing the saturated tanh（00:12:59 - 00:27:53） https://favtu.be/timelines/v/P6sfmUTpUmc/s/779/e/1673 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Saturated activations (): Tanh activations clustered around -1 and 1 indicate saturation, hindering gradient flow.- Saturated neurons update less frequently and impede training.（00:13:09 - 00:19:19） https://favtu.be/timelines/v/P6sfmUTpUmc/s/789/e/1159 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm [-3. Neural network initialization（00:14:24 - 00:36:39） https://favtu.be/timelines/v/P6sfmUTpUmc/s/864/e/2199 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm the night and day shift（00:15:10 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/910/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm not only sweating but also loosing hair :)（00:15:13 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/913/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm Andrej's transformation between and（00:15:14 - 00:15:16） https://favtu.be/timelines/v/P6sfmUTpUmc/s/914/e/916 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm so, no one is going to talk about how andrej grew a decade younger 🤔（00:15:15 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/915/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm was pretty quick 😉（00:15:16 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/916/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-1 or 1, leading to a halt in back propagation.（00:16:03 - 00:16:31） https://favtu.be/timelines/v/P6sfmUTpUmc/s/963/e/991 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]The chain rule with local gradient is affected when outputs of tanh are close to -（00:16:31 - 00:18:09） https://favtu.be/timelines/v/P6sfmUTpUmc/s/991/e/1089 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Concern over destructive gradients in flat regions of h outputs, tackled by analyzing absolute values.（00:18:38 - 00:26:03） https://favtu.be/timelines/v/P6sfmUTpUmc/s/1118/e/1563 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - This can lead to dead neurons, which never activate and don't learn ().（00:19:19 - 00:24:59） https://favtu.be/timelines/v/P6sfmUTpUmc/s/1159/e/1499 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Scaling down weights of the hidden layer can help prevent saturation ().（00:24:59 - 00:27:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/1499/e/1678 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm @Andrej Karpathy Great video! A quick question: at Why a U shape is better than a Cup shape for the histogram of h? Don't we want h to be have some normal distribution, like hpreact?（00:25:43 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/1543/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-2.17 to 2.10 by fixing softmax and 10-inch layer issues.（00:26:03 - 00:26:31） https://favtu.be/timelines/v/P6sfmUTpUmc/s/1563/e/1591 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Optimization led to improved validation loss from（00:26:31 - 00:29:28） https://favtu.be/timelines/v/P6sfmUTpUmc/s/1591/e/1768 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm calculating the init scale: “Kaiming init”（00:27:53 - 00:40:40） https://favtu.be/timelines/v/P6sfmUTpUmc/s/1673/e/2440 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Kaiming initialization (): A principled approach to weight scaling, aiming for unit gaussian activations throughout the network.（00:27:58 - 00:31:46） https://favtu.be/timelines/v/P6sfmUTpUmc/s/1678/e/1906 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Standard deviation expanded to three, aiming for unit gaussian distribution in neural nets.（00:30:02 - 00:30:17） https://favtu.be/timelines/v/P6sfmUTpUmc/s/1802/e/1817 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-0.2 shrinks gaussian with standard deviation 0.6.（00:30:17 - 00:30:47） https://favtu.be/timelines/v/P6sfmUTpUmc/s/1817/e/1847 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Scaling down by（00:30:47 - 00:31:03） https://favtu.be/timelines/v/P6sfmUTpUmc/s/1847/e/1863 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Calculates standard deviation based on fan-in and gain factor specific to the non-linearity used ().（00:31:46 - 00:33:56） https://favtu.be/timelines/v/P6sfmUTpUmc/s/1906/e/2036 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Initializing neural network weights for well-behaved activations, kaiming he et al.（00:31:46 - 00:36:24） https://favtu.be/timelines/v/P6sfmUTpUmc/s/1906/e/2184 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - PyTorch offers torch.nn.init.kaiming_normal_ for this ().（00:33:56 - 00:40:49） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2036/e/2449 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm modern inovations that makes things stable and that makes us not be super detailed and careful with the gradient and backprop issues. (self-note)（00:36:00 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2160/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm [-4. Neural net initialization and batch normalization（00:36:39 - 00:51:52） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2199/e/3112 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Modern innovations have improved network stability and behavior, including residual connections, normalization layers, and better optimizers.（00:36:55 - 00:36:39） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2215/e/2199 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Modern innovations like normalization layers and better optimizers reduce the need for precise neural net initialization.（00:37:05 - 00:40:32） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2225/e/2432 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm The standard deviation relation used to rescale the inital weights, will this only work in the case that the input data also has variance approximately 1 right?（00:38:07 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2287/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm batch normalization（00:40:40 - 01:03:07） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2440/e/3787 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm *Batch Normalization (****):*- Concept: Normalizes activations within each batch to be roughly unit gaussian.- Controls activation scale, stabilizing training and mitigating the need for precise weight initialization.（00:40:49 - 00:51:55） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2449/e/3115 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-2015 enabled reliable training of deep neural nets.（00:40:51 - 00:41:13） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2451/e/2473 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Batch normalization from（00:41:13 - 00:41:39） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2473/e/2499 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Standardizing hidden states to be unit gaussian is a perfectly differentiable operation, a key insight in the paper.（00:42:09 - 00:43:20） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2529/e/2600 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Implementation ():（00:42:17 - 00:42:41） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2537/e/2561 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Normalizes activations by subtracting batch mean and dividing by batch standard deviation ().（00:42:41 - 00:45:54） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2561/e/2754 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Batch normalization enables reliable training of deep neural nets, ensuring roughly gaussian hidden states for improved performance.（00:43:04 - 00:40:51） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2584/e/2451 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Calculating standard deviation of activations, mean is average value of neuron's activation.（00:43:50 - 00:45:45） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2630/e/2745 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm std should be a centralized moment (i.e. subtract the mean first) according to the paper although I see that PyTorch imp is the same as yours（00:44:30 - 01:40:25） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2670/e/6025 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm Around , wouldn't adding scale and shift revert the previous normalization? Improper scale and shift parameters will still cause activation saturated.（00:45:20 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2720/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm Just to be clear, normalising the pre-activation neurons to have 0 mean and 1 std does not make them Gaussian distributed. The sum is only Gaussian distributed at initialisation, because we have initialised the weights to be normally distributed.（00:45:30 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2730/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Learnable gain and bias parameters allow the network to adjust the normalized distribution ().（00:45:54 - 00:54:38） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2754/e/3278 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Back propagation guides distribution movement, adding scale and shift for final output（00:46:16 - 00:51:52） https://favtu.be/timelines/v/P6sfmUTpUmc/s/2776/e/3112 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Couples examples within a batch, leading to potential bugs and inconsistencies ().（00:50:20 - 00:54:03） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3020/e/3243 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm Can anyone explain what he has said from to（00:51:50 - 00:53:00） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3110/e/3180 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm [-5. Jittering and batch normalization in neural network training（00:51:52 - 01:01:35） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3112/e/3695 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Offers a regularization effect due to coupling examples within a batch ().（00:51:55 - 00:42:17） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3115/e/2537 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Padding input examples adds entropy, augments data, and regularizes neural nets.（00:52:37 - 00:53:44） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3157/e/3224 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ?（00:53:00 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3180/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Requires careful handling at inference time due to batch dependency ().（00:54:03 - 01:01:37） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3243/e/3697 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Batch normalization effectively controls activations and their distributions.（00:54:09 - 00:56:05） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3249/e/3365 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Running mean and variance are tracked during training and used for inference ().- Caveats:（00:54:38 - 00:50:20） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3278/e/3020 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Batch normalization paper introduces running mean and standard deviation estimation during training.（00:56:33 - 01:00:46） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3393/e/3646 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm @Andrej Karpathy At , bnmean_running = (0.999 * bnmean_running) + (0.001 * bnmeani), why are you multiplying 0.999 with bnmean_running and 0.001 with bnmeani. Why this not works *bnmean_running = bnmean_running + bnmeani*（00:58:07 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3487/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm is basically an Infinite Impulse Response (IIR) filter（00:58:55 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3535/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm Can any one please tell that at , why did we take the numbers 0.999 and 0.001 specifically? I am new to neural networks and all of this is a bit overwhelming. Thanks（00:59:00 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3540/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Eliminated explicit calibration stage, almost done with batch normalization, epsilon prevents division by zero.（01:01:10 - 01:01:36） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3670/e/3696 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm [-6. Batch normalization and resnet in pytorch（01:01:36 - 01:09:21） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3696/e/4161 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Makes bias terms in preceding layers redundant ().（01:01:37 - 01:18:40） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3697/e/4720 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm I can't understand why removing the mean removes the effect of adding a bias? Why would the grad be zero?（01:02:13 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3733/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Biases are subtracted out in batch normalization, reducing their impact to zero.（01:02:30 - 01:03:13） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3750/e/3793 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm batch normalization: summary（01:03:07 - 01:04:50） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3787/e/3890 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Using batch normalization to control activations in neural net, with gain, bias, mean, and standard deviation parameters.（01:03:53 - 01:07:25） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3833/e/4045 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm At would it help at the end of the training to optimize with bnmean_running and bnstd_running to normalize the preactivations hpreact? Maybe at that point regularization isn't necessary anymore and the rest of the weights can be optimized for the particular batch norm calibration that will be used during inference.（01:04:35 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3875/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm real example: resnet50 walkthrough（01:04:50 - 01:14:10） https://favtu.be/timelines/v/P6sfmUTpUmc/s/3890/e/4450 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm also I would add that ReLU is much easier to compute (max of 2 values and derivative is eighter 0 or 1) than tanh where we have exponents（01:07:06 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4026/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Creating deep neural networks with weight layers, normalization, and non-linearity, as exemplified in the provided code.（01:07:53 - 01:09:21） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4073/e/4161 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Default PyTorch initialization schemes and parameters are discussed ().（01:08:52 - 01:19:13） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4132/e/4753 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm [-7. Pytorch weight initialization and batch normalization（01:09:21 - 01:23:37） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4161/e/5017 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-1/fan-in square root from a uniform distribution.（01:10:05 - 01:10:32） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4205/e/4232 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm Great video, I loved it. Just a question. In the Linear layer on PyTorch at , he says that to initialise the weights the uniform distribution is used, but then in the implementation of the Linear layer when PyTorch-ifying the code he uses the Normal distribution. Did I loose something or he committed a "mistake" ?（01:10:23 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4223/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Pytorch initializes weights using（01:10:32 - 01:11:11） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4232/e/4271 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm reason they're doing this is if you have a roughly gsan input this will ensure that out of this layer you will have a（01:11:03 - 01:11:10） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4263/e/4270 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm roughly Gan output and you you basically achieve that by scaling the weights by（01:11:10 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4270/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-1 over sqrt of fan in, using batch normalization layer in pytorch with 200 features.（01:11:11 - 01:11:40） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4271/e/4300 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Scaling weights by（01:11:40 - 01:14:02） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4300/e/4442 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm summary of the lecture（01:14:10 - 01:18:35） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4450/e/4715 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Importance of understanding activations and gradients in neural networks, especially as they get bigger and deeper.（01:14:35 - 01:16:00） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4475/e/4560 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Batch normalization centers data for gaussian activations in deep neural networks.（01:16:30 - 01:17:32） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4590/e/4652 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-2015, enabled reliable training of much deeper neural nets.（01:17:32 - 01:18:02） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4652/e/4682 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Batch normalization, influential in（01:18:02 - 01:23:39） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4682/e/5019 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm He says 'Bye', but looking at the time, it seems too early []. Most people don't want lectures to be long, but I'm happy this one didn't end there.（01:18:30 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4710/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm just kidding: part2: PyTorch-ifying the code（01:18:35 - 01:26:51） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4715/e/5211 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm always gets me（01:18:35 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4715/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm The "Okay, so I lied" moment was too relatable xD（01:18:36 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4716/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm *PyTorch-ifying the code (****):*（01:18:40 - 01:19:26） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4720/e/4766 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm I don't understand where the layers are organized by putting a tanh after each linear layer while the initialization of the linear layer is `self.weight = torch.randn((fan_in, fan_out), generator=g) / fan_in**0.5`. I think it's not Kaiming initialization, because the gain for tanh is `5/3`, but in the code it's set to `1`,（01:18:59 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4739/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm *Diagnostic Tools (****):*（01:19:13 - 01:26:53） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4753/e/5213 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Code is restructured using torch.nn.Module subclasses for linear, batch normalization, and tanh layers ().- This modular approach aligns with PyTorch's structure and allows easy stacking of layers.（01:19:26 - 01:08:52） https://favtu.be/timelines/v/P6sfmUTpUmc/s/4766/e/4132 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm [-8. Custom pytorch layer and network analysis（01:23:39 - 01:55:56） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5019/e/6956 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Updating buffers using exponential moving average with torch.nograd context manager.（01:24:32 - 01:25:47） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5072/e/5147 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm Why is the last layer made "less confident like we saw" and where did we see this?（01:25:23 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5123/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-46,000 parameters and uses pytorch for forward and backward passes, with visualizations of forward pass activations.（01:25:47 - 01:27:11） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5147/e/5231 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm @ I'd use emb.flatten(1, 2) instead of emb.view(emb.shape[0], -1) to combine two last dimensions into one. It feels that it is better to avoid shape lookup - emb.shape[0]（01:26:25 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5185/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm viz #1: forward pass activations statistics（01:26:51 - 01:30:54） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5211/e/5454 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Forward pass activations: Should exhibit a stable distribution across layers, indicating proper scaling ().（01:26:53 - 01:30:57） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5213/e/5457 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Visualization of statistics: Histograms of activations, gradients, weights, and update:data ratios reveal potential issues during training ().（01:26:53 - 01:26:53） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5213/e/5213 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]The model has（01:27:11 - 01:28:04） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5231/e/5284 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-20% initially, then stabilizes at 5% with a standard deviation of 0.65 due to gain set at 5 over 3.（01:28:04 - 01:28:30） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5284/e/5310 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Saturation stabilizes at（01:28:30 - 01:33:19） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5310/e/5599 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ? Anyone please explain at（01:28:58 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5338/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm Around the mark, I think I missed why some saturation (around 5%) is better than no saturation at all. Didn't saturation impede further training? Perhaps he just meant that 5% is low enough, and that's the best we can do if we want to avoid deeper activations from converging to zero?（01:30:00 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5400/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm The 5/3 gain in the tanh comes for the average value of tanh^2(x) where x is distributed as a Gaussian, i.e.（01:30:10 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5410/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm I'm at , so haven't finished yet. But something is unclear: what's the point of stacking these layers instead of having just one Linear and one Tanh? Since tanh squashes and afterwards we're diffusing, it seems to me like we're doing accordion-like work unnecessarily. What is the benefit we're getting?（01:30:28 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5428/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm 5/3=1.66... is pretty close to the golden ratio 1.61803. Coincidence?（01:30:36 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5436/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm viz #2: backward pass gradient statistics（01:30:54 - 01:32:07） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5454/e/5527 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Backward pass gradients: Should be similar across layers, signifying balanced gradient flow ().（01:30:57 - 01:36:20） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5457/e/5780 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm the fully linear case of no non-linearities（01:32:07 - 01:36:15） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5527/e/5775 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-1 prevents shrinking and diffusion in batch normalization.（01:33:19 - 01:33:50） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5599/e/5630 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm The reason the gradients of the higer layers have a bigger deviation (in the absence of tanh layer), is that you can write the whole NN as a sum of products, and it is easy to see that each weight of Layer 0 appears in 1 term, of layer 1 in 30 terms, of layer 2 in 3000 terms and so on. Therefore a small change of a weight in higer layers changes the output more.（01:33:30 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5610/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Setting gain correctly at（01:33:50 - 01:38:41） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5630/e/5921 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm Does anyone know the paper about "analyzing infinitely linear layers"? Andrej mentioned in the video（01:35:48 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5748/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm There is one doubt I have @ and that is regarding the condition p.dim==2, I don't understand why this was done and which parameters it will filter out?（01:35:59 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5759/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm viz #3: parameter activation and gradient statistics（01:36:15 - 01:39:55） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5775/e/5995 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm - Parameter weights: Distribution and scale should be monitored for anomalies and asymmetries ().（01:36:20 - 01:39:56） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5780/e/5996 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-100 times greater, causing faster training, but it self-corrects with longer training.（01:38:41 - 01:39:11） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5921/e/5951 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm "That's problematic because in that simple stochastic gradient setup you would be training this last layer 10x faster with respect to the other layers". Why 10x faster?（01:38:45 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5925/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]The last layer has gradients（01:39:11 - 01:43:18） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5951/e/6198 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm viz #4: update:data ratio over time（01:39:55 - 01:46:04） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5995/e/6364 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm on a log scale, indicating a good learning rate and balanced parameter updates ().（01:39:56 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/5996/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm did you try using log L2 norm ratio here instead of std? you're using variance as a proxy for how big updates are w.r.t. data values（01:40:25 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/6025/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm can someone explain why we divide std of gradient to std of data instead of using mean? Weight update ratio = grad*learning_rate/weight_value. As we have multiple inputs and multiple entries in batch, we could take mean to calculate single value, cannot figure out how std is a better option.（01:40:38 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/6038/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm At why do we use standard deviation to calculate update to data ratio?（01:40:46 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/6046/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm Why stddev here? Wouldn't we want to use something like the L1-norm? Also, wouldn't we want to log this metric before updating the parameters?（01:40:49 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/6049/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-3 on log plot.（01:43:18 - 01:43:42） https://favtu.be/timelines/v/P6sfmUTpUmc/s/6198/e/6222 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Monitoring update ratio for parameters to ensure efficient training, aiming for -（01:43:42 - 01:51:36） https://favtu.be/timelines/v/P6sfmUTpUmc/s/6222/e/6696 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm bringing back batchnorm, looking at the visualizations（01:46:04 - 01:51:34） https://favtu.be/timelines/v/P6sfmUTpUmc/s/6364/e/6694 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm summary of the lecture for real this time（01:51:34 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/6694/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Introduce batch normalization and pytorch modules for neural networks.（01:52:04 - 01:52:39） https://favtu.be/timelines/v/P6sfmUTpUmc/s/6724/e/6759 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Introduction to diagnostic tools for neural network analysis.（01:53:06 - 01:54:45） https://favtu.be/timelines/v/P6sfmUTpUmc/s/6786/e/6885 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm -[-（01:54:45 - 01:55:50） https://favtu.be/timelines/v/P6sfmUTpUmc/s/6885/e/6950 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]Introduction to diagnostic tools in neural networks, active research in initialization and backpropagation, ongoing progress（01:55:50 - 01:55:58） https://favtu.be/timelines/v/P6sfmUTpUmc/s/6950/e/6958 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm ]（01:55:56 - 01:24:01） https://favtu.be/timelines/v/P6sfmUTpUmc/s/6956/e/5041 Wed, 05 Oct 22 01:41:03 +0900 Building makemore Part 3: Activations & Gradients, BatchNorm intro（00:00:00 - 00:01:48） https://favtu.be/timelines/v/TCH_1BHY58I/s/0/e/108 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP ,（00:00:00 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/0/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP [<1809.89it/s]Last Loss: 2.403459072113037Best Loss: 1.4457638263702393 At Epoch: 25480============================================================（00:00:27 - 00:00:00） https://favtu.be/timelines/v/TCH_1BHY58I/s/27/e/0 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP PS. At I was just uber curious about his previous searches, so I google them:（00:01:34 - 00:27:27） https://favtu.be/timelines/v/TCH_1BHY58I/s/94/e/1647 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP Bengio et al. 2003 (MLP language model) paper walkthrough（00:01:48 - 00:09:03） https://favtu.be/timelines/v/TCH_1BHY58I/s/108/e/543 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP Why space is small? Even in two-dimensional space you can place an infinite number of points（00:03:25 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/205/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP (re-)building our training dataset（00:09:03 - 00:12:19） https://favtu.be/timelines/v/TCH_1BHY58I/s/543/e/739 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP implementing the embedding lookup table（00:12:19 - 00:18:35） https://favtu.be/timelines/v/TCH_1BHY58I/s/739/e/1115 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP Every time I think I finally understand what's happening, he does something like this: 😅（00:17:28 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/1048/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP implementing the hidden layer + internals of torch.Tensor: storage, views（00:18:35 - 00:29:15） https://favtu.be/timelines/v/TCH_1BHY58I/s/1115/e/1755 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP -dimensional vertically scrollable space to describe the functions of PyTorch ()（00:20:25 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/1225/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP at I think it's supposed to be first letter not first word. It's first word in the paper but first letter in the example（00:21:24 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/1284/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP At , when he says words does he mean the 3 character sequence that was made by block size? And, so, when he refers to the picture behind him, does he mean each of those three blocks represents a indice in the block_size array?（00:21:39 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/1299/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP what about just `emb_reshaped = emb.reshape((emb.shape[0], emb.shape[1]*emb.shape[2]))` ?（00:23:47 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/1427/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP Of course! Memory itself is a one dimensional "tensor". :D（00:24:50 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/1490/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP for the PyTorch internals video (@)（00:25:36 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/1536/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP Please create the "entire video about the internals of pytorch" that you mentioned in . And thank you so much for the content, Andrej !!（00:25:40 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/1540/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP at minute mark at the moment and gotta say, pytorch is amazing. so wonderful how easy they make it for devs with those small tricks.（00:27:24 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/1644/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP matthew -31>Then the governor’s soldiers took Jesus into the Praetorium and gathered the whole company of soldiers around him. They stripped him and put a scarlet robe on him, and then twisted together a crown of thorns and set it on his head. They put a staff in his right hand. Then they knelt in front of him and mocked him. “Hail, king of the Jews!” they said. They spit on him, and took the staff and struck him on the head again and again. After they had mocked him, they took off the robe and put his own clothes on him. Then they led him away to crucify him.（00:27:27 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/1647/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP proverbs>You will have plenty of goats’ milk to feed your family and to nourish your female servants.（00:27:27 - 00:27:27） https://favtu.be/timelines/v/TCH_1BHY58I/s/1647/e/1647 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP implementing the output layer（00:29:15 - 00:29:53） https://favtu.be/timelines/v/TCH_1BHY58I/s/1755/e/1793 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP we can also use torch.reshape() to get the right shape for W. However, there is a difference between torch.view and torch.reshapeTL;DR:If you just want to reshape tensors, use torch.reshape. If you're also concerned about memory usage and want to ensure that the two tensors share the same data, use torch.view.（00:29:20 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/1760/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP implementing the negative log likelihood loss（00:29:53 - 00:32:17） https://favtu.be/timelines/v/TCH_1BHY58I/s/1793/e/1937 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP What's tanh?（00:30:03 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/1803/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP "ideally all of these numbers here of course are one because then we are correctly predicting the next character" hmmmmmm it's reasonable to say these numbers are high, put not one, If the probability here is one, that will exclude any chance of other characters having similar context.（00:31:47 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/1907/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP summary of the full network（00:32:17 - 00:32:49） https://favtu.be/timelines/v/TCH_1BHY58I/s/1937/e/1969 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP introducing F.cross_entropy and why（00:32:49 - 00:37:56） https://favtu.be/timelines/v/TCH_1BHY58I/s/1969/e/2276 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP re: using cross_entropy function around , it sounds like pytorch takes the derivate of each step of exponentiation then normalization instead of simplifying them before taking the derivative. is that a "soft" limitation of the implementation in that a procedure could be defined to overcome it, or is there a bit of an mathematical intuition needed to understand how to rewrite the function to produce a simpler derivative?（00:34:05 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/2045/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP Since probs are invariant to an offset applied to logits, it's fun to plot the drift in the mean or sum of b2. Looks like Brownian motion.（00:37:00 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/2220/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP , who would tell you this when you are reading from a book. Exceptional teaching ability（00:37:14 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/2234/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP implementing the training loop, overfitting one batch（00:37:56 - 00:41:25） https://favtu.be/timelines/v/TCH_1BHY58I/s/2276/e/2485 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP pfeeeewwww 😳（00:38:00 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/2280/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP training on the full dataset, minibatches（00:41:25 - 00:45:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/2485/e/2740 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP I don't understand the mini batching happening at . when using ix = torch.randint(0,X.shape,(32,)), and using this to index into X, you are just picking 32 data examples from X, not batching all of the data right? I thought by batching, you taking a batch of data, do a forward pass on all items in the batch, take the mean output and do back prop on that mean result outcome and update the model on that loss. Here I feel like Andrej is just selecting 32 individual data examples. Please do correct me if I'm wrong! I'm new to ML!（00:41:30 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/2490/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP life lesson: much better to have an approximate gradient and take many steps than have an exact gradient and take a few steps（00:44:25 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/2665/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP Awesome videos, thank you for that! I have a question though about , "finding a good initial learning rate", which is either a mistake in the video or I misunderstood something.（00:45:00 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/2700/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP It seems it is slightly different from the approach presented here. Looking at the , it looks like for each iteration, we randomly select a min batch of size 32 from the whole training set, and update the parameters, then go on to the next iteration.（00:45:34 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/2734/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP finding a good initial learning rate（00:45:40 - 00:53:20） https://favtu.be/timelines/v/TCH_1BHY58I/s/2740/e/3200 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP @ 'Finding a good initial learning rate', each learning rate is used just one time. The adjustment of the parameter of one learning rate is based on the parameters already adjusted using the prior smaller learning rates. I feel that each of the 1,000 learning rate candidates should go through the same number of iterations. Then, the losses at the end of the iterations are compared. Please tell me if I am wrong. Thanks!（00:45:40 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/2740/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP I don't quite understand the part of finding a good initial learning rate. Why the lowest point of loss value indicates the best learning rate? It takes some time for the loss value to decrease, right?（00:45:40 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/2740/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP On I was waiting fot Karpathy's constant to appear. Thank you for the lecture, Andrej（00:45:45 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/2745/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP At Andrej says that the learning rate would be low in the beginning and high at the end. Why was it set like that? My intuition is that the learning rate should be in the opposite order.（00:48:45 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/2925/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP I believe that at the losses and the learning rates are misaligned.The first loss (derived from completely random weights) is computed before the first learning rate is used, and therefor the first learning rate should be aligned with the second loss.You can simply solve this problem by using this snippet;lri = lri[:-1]lossi = lossi[1:]（00:49:22 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/2962/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP Question about , in the plot, y axis is the loss, and the x axis is learning rate, but x axis is also the step number. How do you know whether the y axis change is because of learning rate difference or step number increase?（00:50:00 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/3000/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP Great video! One question, @AndrejKarpathy: around or so you show how to graph an optimal learning rate and ultimately you determine that the 0.1 you started with was pretty good. However, unless I'm misunderstanding your code, aren't you iterating over the 1000 different loss function candidates while *simultaneously* doing 1000 consecutive passes over the neural net? Meaning, the loss will naturally be lower during later iterations since you've already done a bunch of backward passes, so the biggest loss improvements would always be stacked towards the beginning of the 1000 iterations, right? Won't that bias your optimal learning rate calculation towards the first few candidates?（00:50:30 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/3030/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP Can anyone explain to me, why looking at loss plotted against exponent of the learning rate () the conclusion is that lr<0.1 "is way too low"? For me, its where the loss is actually getting lower, isn't it?（00:50:42 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/3042/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP splitting up the dataset into train/val/test splits and why（00:53:20 - 01:00:49） https://favtu.be/timelines/v/TCH_1BHY58I/s/3200/e/3649 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP To break the data to training, developement and test, one can also use torch.tensor_split.n1 = int(0.8 * X.shape[0])n2 = int(0.9 * X.shape[0])Xtr, Xdev, Xts = X.tensor_split((n1, n2), dim=0)Ytr, Ydev, Yts = Y.tensor_split((n1, n2), dim=0)（00:53:20 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/3200/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP I'm confused at why care must be taking with how many times you can use the test dataset as the model will learn from it. Is this because there is no equivalent of 'torch.no_grad()' for LLMs - will the LLM always update the weights when given data?（00:56:17 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/3377/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP Thank you for the lectures! @ Made me chuckle（00:59:01 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/3541/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP It can take days!! How can someone sleep with such pressure（00:59:15 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/3555/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP :（01:00:00 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/3600/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP experiment: larger hidden layer（01:00:49 - 01:05:27） https://favtu.be/timelines/v/TCH_1BHY58I/s/3649/e/3927 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP I also just noticed, he explicitly mentions these fluctuations at . Doh!（01:02:15 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/3735/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP around - the reason why we're not "overfitting" with the larger number of params might be the context size. with a context of 3, no number of params will remove the inherent uncertainty.（01:05:00 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/3900/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP visualizing the character embeddings（01:05:27 - 01:07:16） https://favtu.be/timelines/v/TCH_1BHY58I/s/3927/e/4036 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP Fascinating how the vowels end up clustered together!（01:06:56 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/4016/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP experiment: larger embedding size（01:07:16 - 01:11:46） https://favtu.be/timelines/v/TCH_1BHY58I/s/4036/e/4306 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP : it should be 10 dimensional embeddings for each *character* not word in this character-level language model.（01:07:20 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/4040/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP you shouldn't have plotted stepi variable against the loss :D it could have worked if you'd ploted out just plt.plot(loss_history) or applied two different colours for those two runs（01:10:09 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/4209/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP The plot of the steps and losses after running the training loop multiple times (~ mins) https://youtu.be/TCH_1BHY58I?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ&t=4233) would be wrong because stepi array keeps appending the same indices [0, 50000). I expect the graph to just start getting more unstable and unstable.（01:10:30 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/4230/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP summary of our final code, conclusion（01:11:46 - 01:13:24） https://favtu.be/timelines/v/TCH_1BHY58I/s/4306/e/4404 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP sampling from the model（01:13:24 - 01:14:55） https://favtu.be/timelines/v/TCH_1BHY58I/s/4404/e/4495 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP google collab (new!!) notebook advertisement（01:14:55 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/4495/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP Adrej is learning youtube tricks 😅（01:14:56 - 01:15:40） https://favtu.be/timelines/v/TCH_1BHY58I/s/4496/e/4540 Mon, 12 Sep 22 23:43:06 +0900 Building makemore Part 2: MLP