About

From fresh PhD grad to research director over the past 6 years, I’ve always followed the same MO: try to anticipate the field’s next biggest challenges, build a team from scratch, solve fundamental research challenges, then do whatever it takes to ship it (product design, partnerships, legal, the works).

Led multiple research efforts that became foundational to how LLMs are built today.

Pre-Gemini days:

• Lead author of Google’s first major paper on RAG (REALM: coined the term “retrieval-augmented”, cited by GPT-3, Meta’s first RAG paper, Chinchilla, etc.)

• Co-invented the concept of post-training (co-lead author of FLAN, Google’s first paper on post-training, cited by InstructGPT, Llama etc.)

• Trained the first Transformers ever launched in Google Ads and Assistant, adding billions in annual revenue.

Research lead in Gemini:

• Built Gemini’s first fact-checking agents: a cornerstone of Gemini’s factuality evals, reward models (RL) and its citations capability.

• Developed one of the key methodologies behind Gemini’s pretraining data strategy, advancing the science of data ablations and LLM scaling laws. One of a handful of strategic initiatives directly sponsored by CEO Demis Hassabis.

News

Media coverage of my team’s 2023 work on Bard (double-check Bard’s responses with the “Google it” button):

• Google Keyword Blog: “Bard can now connect to your Google apps and services: Use Bard alongside Google apps and services, easily double-check its responses and access features in more places.”
• New York Times: “Google Connects A.I. Chatbot Bard to YouTube, Gmail and More Facts”
• CNN: “Google rolls out a major expansion of its Bard AI chatbot”
• Ars Technica: “Google’s AI assistant can now read your emails, plan trips, ‘double-check’ answers”
• Vox: “Google’s free AI isn’t just for search anymore”

Some semi-recent appearances:

• [Spring 2023] No Priors Podcast with investors Sarah Guo and Elad Gil
• [Spring 2022] Stanford CS224N NLP with Deep Learning (Spring 2022) Guest Lecture: Building Knowledge Representation
• [Winter 2020] Generally Intelligent Podcast with Kanjun Qiu and Josh Albrecht, founders of Imbue

Experience

Work

Education

Selected publications

Sorry, I’m not great at keeping this section updated. For a more comprehensive list of recent work, please check out my Google Scholar profile.

• Simfluence: Modeling the influence of individual training examples by simulating training runs
  Kelvin Guu*, Albert Webson*, Ellie Pavlick, Lucas Dixon, Ian Tenney, Tolga Bolukbasi*
  arXiv preprint, 2023
  (* lead contributors)

• RARR: Researching and revising what language models say, using language models
  Luyu Gao*, Zhuyun Dai*, Panupong Pasupat*, Anthony Chen*, Arun Tejasvi Chaganty*, Yicheng Fan*, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, Kelvin Guu*
  Association for Computational Linguistics (ACL), 2023
  (* lead contributors)

• Finetuned language models are zero-shot learners
  Jason Wei*, Maarten Bosma*, Vincent Y Zhao*, Kelvin Guu*, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, Quoc V Le
  International Conference on Learning Representations (ICLR), 2022
  (* lead contributors)

• REALM: Retrieval-augmented language model pre-training
  Kelvin Guu*, Kenton Lee*, Zora Tung, Panupong Pasupat, Ming-Wei Chang
  International Conference on Machine Learning (ICML), 2020
  (* equal contribution)

• Generating Sentences by Editing Prototypes
  Kelvin Guu*, Tatsunori B. Hashimoto*, Yonatan Oren, Percy Liang
  Transactions of the Association for Computational Linguistics (TACL), 2018
  (* equal contribution)

• Traversing Knowledge Graphs in Vector Space
  Kelvin Guu, John Miller, Percy Liang
  Empirical Methods in Natural Language Processing (EMNLP), 2015

Other projects

Presentations and tutorials

• Why naive cross-validation can fail for feature selection.
• Presentation on selection and multiple hypothesis testing, based on several great lectures by Emmanuel Candes.
• Presentation on random matrices, prepared for Percy Liang’s reading group.

Collaborators

I have had the privilege of working with and learning from great mentors and collaborators, including but not limited to:

• Mentors
  • Percy Liang, Associate Professor of Computer Science & Statistics at Stanford
  • Jakob Uszkoreit, CEO and Co-Founder of Inceptive
  • Richard Socher, CEO and Founder of you.com
  • David B. Dunson, Professor of Statistics at Duke
  • Debdeep Pati, Professor of Statistics at Texas A & M
  • Alex Hartemink, Professor of Computer Science & Biology at Duke
  • Ronald Parr, Professor of Computer Science at Duke
• Collaborators
  • Tatsunori B. Hashimoto, Assistant Professor of Computer Science at Stanford
  • Panupong (Ice) Pasupat, Senior Research Scientist at Google
  • Dan Iter, Senior Research Scientist at Microsoft
  • Evan Z. Liu, Researcher @ Imbue
  • Dorottya (Dora) Demszky, Assistant Professor in Education Data Science at Stanford
  • Justin Fu, Research Scientist at Waymo

Other interests

Finally, I think the following entities are pretty swell:

• Doug Engelbart. We still need to finish this guy’s mission.
• Wikipedia and Freebase.
• Jon Kabat-Zinn. For bringing scientific attention to the study of mindfulness.
• The Bill & Melinda Gates Foundation. I think they make a huge impact by bringing an investor’s mindset to important and good causes.

Contact

• Email: kelvin 🐌 kelvinguu.com
• Twitter: @kelvin_guu
• Github: @kelvinguu
• Google Scholar: Kelvin Guu
• LinkedIn: kelvinguu