Progress in AI

💡 JC's Newsletter #160 - 2023-02-28

Dear friends,

In JC’s Newsletter, I share the articles, documentaries, and books I enjoyed the most in the last week, with some comments on how we relate to them at Alan. I do not endorse all the articles I share, they are up for debate. 

I’m doing it because a) I love reading, it is the way that I get most of my ideas, b) I’m already sharing those ideas with my team, and c) I would love to get your perspective on those. 

If you are not subscribed yet, it's right here! 

If you like it, please share it on social networks!

---

🔎 Some topics we will cover this week

• OpenAI was founded to keep AI away from centralized control and to "freely share" their research and technology with other organizations.

• Microsoft and OpenAI have been working closely together for about three and a half years.

• MedPaLM is an open-sourced large language model for medical purposes by Google Research and DeepMind.

---

👉 OpenAI history: Thread by Houck (Ping Thread)

❓Why am I sharing this article?

• Very good analysis of the story of OpenAI and how it can have an impact on what kind of Lab we want to build

• The initial team was very small, and people accept to take lower salaries for the right job

• OpenAI is worth $20 billion

• It was founded by Elon Musk, Sam Altman, and others and has raised over $2 billion

• Years before launching ChatGPT it was started as a nonprofit

• OpenAI was founded by Elon Musk, Sam Altman, and nearly 20 others in 2015

• Originally structured as a non-profit, the organization didn't accept investments

• Instead they took $1 billion in donations from Peter Thiel, Reid Hoffman, Jessica Livingston, and more notable names

• The initial team included 9 top AI researchers. Many took lower salaries to join OpenAI.

• To keep AI away from centralized control (i.e. DeepMind being owned by Google)

• They stated they'd "freely share" their research and technology with other organizations. Sam Altman, as CEO, has repeatedly stated they had no idea how they'd make money

• In 2018, Elon stepped down from OpenAI's board of directors

• The next year, they raised $1 billion from Microsoft Altman said they'd likely spend it "within 5 years and possibly much faster"

• As a result, they launched OpenAI LP as a subsidiary company focused on generating revenue from their tech

• Employees were also given equity in OpenAI LP

• However the parent company remains a non-profit

• Interestingly, they "capped" the return investors could make at 100x the size of their investment And restricted board members from investing This type of "capped" for-profit structure was put in place due to OpenAI's mission to not allow anyone to control a future AGI

---

👉 An Interview with Daniel Gross and Nat Friedman about ChatGPT and the Near-Term Future of AI (Stratechery)

❓Why am I sharing this article?

• Overview of some of the recent developments.

• Built a kind of prompt-topuppet pipeline. So, you can type a text prompt describing a character, and then it will create a 3D controllable puppet out of them using Stable Diffusion and a pipeline of other models. 

• One of the big things that happened since we last talked was OpenAI released this Whisper model, which is a speech recognition model. The performance is really surprisingly good. If you run it over your podcasts, the large model, it’s probably near flawless in terms of its transcription. And you can run it locally. 

• Forum seeding, right? How do you get a forum off the ground? You need to get a critical mass of people. We already talked about forums are not necessarily accurate. What if there was forum participants that are always there and always replying?

---

👉 New Bing, and an Interview with Kevin Scott and Sam Altman About the Microsoft-OpenAI Partnership (Stratechery)

❓Why am I sharing this article?

• How big partnerships work and how you need commitment at the highest level, make it a priority for both companies, and even in those cases it is very hard. 

• Collaboration between structures is messy, you need to put in a lot of time

• You never get the org, the agreement, the partnership right. It is about the highest level of trust. 

• The cost of ML is going to go down. 

• We have been in the current instantiation of our partnership with OpenAI for about three and a half years now. I think very early in OpenAI’s history, they were on Azure briefly and then moved to Google for a bit.

• In 2018 we decided that it would actually be a very good idea for Microsoft and OpenAI to partner, and we’ve been working super closely together ever since.

• The way it worked is that Satya and I had a few-minute conversation at a conference together in the summer of 2018 and said, “Hey, maybe we should figure something out more.

• The level of dependence that companies have taken on each other is a little bit scary. It probably makes us both a little bit uncomfortable in different ways, and no contract in the world can protect you for that.

• We’re building a platform of powerful computers and powerful AI models that lots and lots of other people are going to build on top of.

• The way it has worked is not what I would’ve thought, which is the clean, bright line delineation, but actually just very close collaboration at each step.

• If you think about GitHub CoPilot, it was just a lot of work to figure out how to turn that into a product and it was work across a whole bunch of different teams. So multiple parts of OpenAI, multiple parts of GitHub, and multiple parts of Microsoft — and that one was especially complicated because it was three organizations that are working together.

• With these things, you never get the org design right. You never get the partnership right. You never get the agreement right. It’s either you trust each other and you like each other and you work together in good faith and eventually you work it all out or you don’t and you never work it all out and none of the rest of it can save you.

• One of the things that we’ve gotten a lot of confidence in over the past handful of years is our ability to performance-optimize all of this stuff, both on the training and the inference side of things. The cost is just going to go down over time.

• We will continue to open source stuff. We won’t open source Microsoft’s — their stuff is their stuff — but we will continue the open source stuff and yeah, we’re going to collaborate a lot on features.

---

👉 Free Rent: OpenAI Deal Shows One Way Microsoft Seeds Its Cloud (The Information)

❓Why am I sharing this article?

• How many of your engineers use co-pilot? S

• Interesting to see how much OpenAI paid.

• Microsoft says it exclusively sells the automatic code generator to tens of thousands of individual developers who pay $10 per month or $100 per year. The company plans to launch corporate subscriptions over the next year

• Before its exclusive partnership with Microsoft began, OpenAI paid Google more than $120 million for cloud computing in 2019 and 2020 combined

---

👉 Microsoft and OpenAI working on ChatGPT - Powered bing in challenge to Google (The Information)

❓Why am I sharing this article?

• Microsoft's move is extremely smart, and puts Europe in a tough position. 

• Microsoft has also incorporated OpenAI’s software into some of its apps for enterprise customers, including Power apps that help them make their own business apps. 

• The company also resells GPT-3 and Dall-E 2 to enterprise customers alongside its Azure server- and compute-power rental business. Eric Boyd, corporate vice president of the AI platform, leads that business, and one of his direct reports—Mikhail “Misha” Bilenko, vice president of AI platform and data science—runs a team focused on building new enterprise services that reuse OpenAI models.

---

👉 You can now join the waitlist for ChatGPT Professional, a paid ‘experimental’ version of the chatbox that is faster and includes double the searches (Business Insider)

• A Google Form to sign up for the waitlist for ChatGPT Professional, that is "always available," has faster responses, and includes at least double the amount of searches each day compared to the free version.

---

👉 Sunday Reads #170: Lemon markets, dark forests, and a firehouse of malicious garbage (Jitha Thathachari)

❓Why am I sharing this article?

• Good tip to write your reviews or issues for example if you prefer talking about something than writing 

• You can then review the first draft and go a lot faster.

  • One of my friends has started using ChatGPT to write his talks. He gets 80% done in 15 minutes. He outlines what he wants to say (transcribes a voice note through Otter.ai), and pastes it into ChatGPT. Which then proceeds to write the talk for him.

---

👉 Google and Microsoft’s Events, Monetizable Panic, Paradigms and Hardware (Stratechery)

❓Why am I sharing this article?

• On the importance of voice!

This gets at perhaps the biggest issue with these chat interfaces: they’re honestly kind of annoying to use. Typing full sentences is a pain, and waiting for a response to be typed out is inefficient. What seems to make much more sense is voice, but that doesn’t work everywhere; sometimes it’s preferable to just type a couple of keywords and go from there.

---

Tiktok

👉 How CapCut Uses AI to Unlock Human Creativity (The Split)

❓Why am I sharing this article?

• I love the feature where we can copy any video!

• CapCut also has a desktop product

• Tapping "Use template" sends you to an editing screen that copies the exact editing style as the original video. Everything is pre-edited with:

  • The same clip length

  • The same sound

  • The same text

  • The same filter

• CapCut's templates allow you to copy the editing style of any video.

---

Google

👉 Google introduces ChatGPT-like ChatBot for Healthcare (Analytics India Mag)

❓Why am I sharing this article?

• Healthcare is going to be progressively transformed by ML

• MedPaLM consists of six existing open-question answering datasets along with a new one called HealthSearchQA.

• Google Research and DeepMind recently introduced MedPaLM, an open-sourced large language model for medical purposes. 

  • It is benchmarked on MultiMedQA, a newly introduced open-source medical question-answering benchmark. It combines HealthSearchQA, a new free-response dataset of medical questions sought online, with six existing open-question answering datasets covering professional medical exams, research, and consumer queries. The benchmark also incorporates methodology for evaluating human model responses along several axes, including factuality, precision, potential harm, and bias

• MedPaLM provides datasets for multiple-choice questions and for longer responses to questions posed by medical professionals and non-professionals. These comprise the clinical topics datasets for MedQA, MedMCQA, PubMedQA, LiveQA, MedicationQA, and MMLU.

• The HealthsearchQA dataset, which consists of 3375 frequently asked consumer questions, was curated using seed medical diagnoses and their related symptoms.

• The researchers developed this model on PaLM, a 540 billion parameter LLM, and its instruction-tuned variation Flan-PaLM to evaluate LLMs using MultiMedQA.

• The resulting model that addresses this issue is Med-PaLM, which claims to perform well compared to Flan-PaLM but still needs to outperform a human medical expert’s judgment.

  • For instance, a group of doctors determined that 92.6% of the Med-PaLM responses were on par with the clinician-generated answers (92.9%), whereas just 61.9% of the long-form Flan-PaLM answers were deemed to be in line with the scientific agreement.

  • Furthermore, like Flan-PaLM, 5.8% of Med-PaLM answers were assessed as potentially contributing to negative consequences, comparable to clinician-generated answers (6.5%), while 29.7% of Flan-PaLM answers were

---

👉 General Catalyst, Spark in Talks to Back OpenAI Rival (The Information)

❓Why am I sharing this article?

• A lot of hype around new projects

• Adept AI Labs and Anthropic, which develop AI that people can activate using basic language prompts. Their founders include former leaders of OpenAI and Google AI.

• Anthropic: the 80-person company had been a busy fundraiser, adding $704 million to its coffers in less than two years.

👉 The Next Generation Of Large Language Models (Forbes)

• GLaM, a sparse expert model developed last year by Google, is 7 times larger than GPT-3, requires two-thirds less energy to train, requires half as much compute for inference, and outperforms GPT-3 on a wide range of natural language tasks. Similar work on sparse models out of Meta has yielded similarly promising results.

---

It’s already over! Please share JC’s Newsletter with your friends, and subscribe 👇

Let’s talk about this together on LinkedIn or on Twitter. Have a good week!