Gold Penguin Logo with Text

Who is Andrej Karpathy — OpenAI to Tesla & Back

Andrej Karpathy is another noteworthy figure in the AI industry who’s played a pivotal role in OpenAI and Tesla. But how much do we actually know about this guy? Particularly, the story of how he helped steer modern technologies toward innovation with these two giants in the AI field?
Updated April 1, 2024
Andrej Karpathy, made with Midjourney

The rise of GPT-4 is rooted in the earlier models of artificial intelligence. Moreover, the advent of autonomous vehicles wouldn’t be a thing nowadays if not for this one person who’s also worked behind the scenes: Andrej Karpathy. He is a Slovak-Canadian computer scientist who is passionate about “training deep neural nets on large datasets”. He has worked with OpenAI and Tesla to join them in their AI missions.

In this article, we’ll dive into the story of how he started traversing the world of AI from theoretical research to practical application. We’ll also get into how he became part of the leading tech giants, OpenAI and Tesla, and changed our world forever. Now, let’s get to know Andrej Karpathy.

A Glimpse into Andrej’s Childhood and Education

On the 23rd of November 1986 in Slovakia, Andrej, the man who would later help shape the future world, was born. He was born and raised there, developing an early fascination with computer science. When he turned fifteen, his family moved to Toronto.

Andrej Karpathy attended three universities:

  • University of Toronto (2005-2009): He took a double major in computer science and physics, on top of taking math as his minor. More than that, he got to attend Geoffrey Hinton’s class, which ignited his interest in deep learning.
  • University of British Columbia (2009-2011): With his professor Michiel van de Panne, he worked on learning controllers for physically simulated figures. This is where he earned his Master of Science.
  • Stanford University (2011-2015/16): Under the guidance and supervision of his adviser, Fei-Fei Li, he completed his thesis. It centered around the use of convolutional neural networks (CNN) in computer vision, NLP, and their intersection. He received his Ph.D. in Computer Science the following year in 2016.

Andrej’s stay at Stanford University didn’t only revolve around his studies, though. So what else was he doing? More on that as we move forward to how he started his career in the field of artificial intelligence.

Before Karpathy’s Affair with OpenAI and Tesla

Andrej first found his place in academia, then into the divisions of Google. Here are the things that he’s done prior to his OpenAI-Tesla ventures while in his Stanford and Google era, and everything in between.

His Role in Google Brain, Research, and DeepMind

Andrej Karpathy did three internships at Google while still earning his Doctor of Philosophy at Stanford University. Here’s a short timeline of what he’d been up to as an intern at Google:

  • Google Brain (2011): This was an AI research division at Google. It was actually still a “baby” Google Brain—as Andrej likes to call it— at the time he worked on it, since it was in its early development stage. His role in this project focused on large-scale unsupervised learning from videos.
  • Google Research (2013): Andrej returned later to Google to work on large-scale supervised learning on YouTube videos. Oh, and speaking of YouTube, he has a YouTube channel where he talks about large language models, neural networks, backpropagation, and Stable Diffusion.
  • DeepMind (2015): Finally, Andrej worked on the Deep Reinforcement Learning team at DeepMind. This is a subsidiary of Google’s parent company, Alphabet. 

Besides being associated with Google as an intern during his stay at the university, he’s done some research as well. Next up are his contributions to the field of AI while still in the academy.

His AI Work and Research in the Academy

Andrej’s work and research on CNN with Fei-Fei Li has had a massive impact on helping computers “see” the world and understand it the way we do, perhaps even more. Right after completing his studies, he went on to share what he knows about AI along with its complex concepts in simplified ways. In fact, he makes the knowledge accessible to the wider community through his blogs posted on GitHub and Medium.

Below are some of his academic contributions to the field of artificial intelligence:

  • Stanford’s First Deep Learning Class: In 2015, Andrej became the primary instructor of the first-ever deep learning course offered by Stanford University. It was titled Convolutional Neural Networks for Visual Recognition. It eventually became one of the fastest-growing classes at Stanford. Starting from 150 enrollees in 2015, it expanded to 330 in 2016, and even grew to 750 in 2017.
  • Image Captioning: His research titled Deep Visual-Semantic Alignments for Generating Image Descriptions that he conducted with Fei-Fei Li in 2014/15 bridged the gap between NLP and computer vision. AI models were taught to generate descriptive texts for images and their regions using natural language. This allows AI systems to understand and relate texts to visual objects better.

Andrej Karpathy is not just an AI expert, but a true educator as well. And even though he’s no longer in academia, the legacy he’s left lives on.

His Publications and Pet Projects

Aside from teaching in classes and through online tutorials, he’s authored and published papers with a focus on AI. These have contributed even more to the growing AI field, informing and guiding fellow researchers. Here are some of his publications:

  • World of Bits: An Open-Domain Platform for Web-Based Agents (2017)
  • Connecting Images and Natural Language (2016)
  • ImageNet Large-Scale Visual Recognition Challenge (2015)
  • Object Discovery in 3D Scenes via Shape Analysis (2013)
  • Emergence of Object-Selective Features in Unsupervised Feature Learning (2012)

He also has other cool and interesting pet projects but there’s just so many! So below are just some of them:

  • Arxiv Sanity - making papers on machine learning more searchable and sortable
  • Neuraltalk2 - an image captioning project
  • Pretty Papers Accepted - making pages and forms look nicer and more functional
  • Research Lei - an academic papers management and discovery system
  • ulogme - a privacy-first productivity tool tracking active windows, notes, etc.

Given his background, it’s evident that Andrej Karpathy is the true combination of a hobbyist and a pro in the AI space. Now, on to the next part, we’ll go into the next big chapters of his career.

Andrej Karpathy’s First Journey with OpenAI

It was 2015. Before the year ended, a groundbreaking AI research organization was founded by a group of visionary pioneers in AI. They were set to change the course of technology forever. Andrej Karpathy was one of them. And from there, he worked alongside the team as one of its founding research scientists.

Andrej started off working on deep learning, computer vision, generative modeling, and reinforcement learning. This experience further deepened his perspectives on machine learning and other facets of AI. It also gave him exposure and firsthand experience to the early stages of generative pre-trained transformers (GPT), This has now evolved into what we know as GPT-4.

Besides his main duty, he’s also helped with recruiting and structuring the company in its earlier days. Later on, he worked on training a computer to control and use a keyboard and mouse like we do. Andrej taught the computer to accomplish various tasks such as filling out a form, writing, and more.

In the span of 18 months he was with OpenAI, he became a key contributor to its early success. But why did he join OpenAI in the first place? Well, it was their fascinating model approach of being an academia and start-up hybrid that got his attention. As we know, OpenAI is one of the first few tech companies that fostered AI. Andrej wanted to be part of it and embrace the innovation it would bring to the world.

Leaving OpenAI for Tesla

Leaving OpenAI wasn’t his idea, though. One of the co-founders, Elon Musk, poached Andrej that resulted in him leaving in 2017. With his indispensable talent for deep learning, especially in image recognition and understanding, Elon believed he’d be a good addition and asset to Tesla.

As he joined Tesla in 2017, he was appointed as the Senior Director of AI. He was tasked in leading the Tesla Autopilot’s computer vision team that builds the company’s so-called “self-driving cars”. This role puts a greater emphasis on the transition of his daring career from research to application. He worked behind the in-house data labeling, neural networks training, and deployment of algorithms that fuel Tesla’s Autopilot and Full-Self Driving (FSD) ability.

Under Andrej’s leadership, the team bolstered the safety and reliability of Tesla’s autonomous vehicles by applying computer vision. This ensures that the models are able to do the following:

  • Accurately perceive and understand the real world, 
  • Detect physical objects, 
  • Adjust to the surroundings accordingly, 
  • Achieve optimal performance. 

More than that, he furthered the development of data-centric AI focused on the quality of training data.

Karpathy’s role has truly made a difference in the progress of Tesla. In 2020, he was listed among the Innovators Under 35 by MIT Technology Review. And while still working with Elon Musk, he helped develop Tesla’s humanoid robots called “Optimus”. They can walk, pick up or hold objects, and more.

Sometime in March 2022, Andrej announced his 4-month sabbatical leave from Tesla on his Twitter. After his sabbatical, he announced that he was departing from the company in July 2022, ending his career at Tesla.

Reuniting with OpenAI

Andrej then decided to return to OpenAI, sharing the news on his Twitter on February 9, 2023. On a more personal note, it was also reported that OpenAI’s mission aligns well with his focus on open-source and education. This also served as another reason for him to find his way back to the company. He also earned the WTF Innovators Award in June 2023 during his second time with OpenAI.

Reportedly Leaving OpenAI Once Again

Everything was going well between Andrej Karpathy and OpenAI ever since the company welcomed him back. Which is shocking to find out that on the day before Valentine's Day (on February 13, 2024), Andrej broke up with OpenAI (again). 

In his tweet, Andrej Karpathy announced that he left not because of some issue or drama, but simply because it was his personal choice. He even mentioned that his final year with OpenAI had been wonderful, and that it’s been a worthwhile ride. 

So, what’s his plan after that? Where does he really want to go? Well, as he said, it’s wherever his personal projects would take him. And without a doubt, he’d continue to reach far places even if he’d most likely go solo from now on.

So, Where’s He Now?

Geographically, he’s currently in San Francisco (upon checking his LinkedIn), the same place on earth where OpenAI is. But even though he’s no longer working for them, one thing’s for sure—he never lost his place in the AI industry.

As he mentioned, he’s going to work on his personal projects. So most probably, he’s sitting in his room or home office now, focusing on his new endeavor. While he’s at it, what we can expect in the meantime is that there’s something to look forward to. Why? Simply because Andrej Karpathy is not going to falter, for an AI expert like him is bound to continue innovating—with a company or alone.

Want To Learn Even More?
If you enjoyed this article, subscribe to our free monthly newsletter
where we share tips & tricks on how to use tech & AI to grow and optimize your business, career, and life.
Written by Justin Gluska
Justin is the founder of Gold Penguin, a business technology blog that helps people start, grow, and scale their business using AI. The world is changing and he believes it's best to make use of the new technology that is starting to change the world. If it can help you make more money or save you time, he'll write about it!
Notify of

Inline Feedbacks
View all comments
Join Our Newsletter!
If you enjoyed this article, subscribe to our newsletter where we share tips & tricks on how to make use of some incredible AI tools that you can use to grow and optimize a business