Five trends that could change the course of generative AI models

While the potential of these models is visible in the numbers, with ChatGPT attracting over 100 million users since December, these models have alarmed many not only because they pretend to think and act like humans. but also because they can reproduce the work of the famous. Writers and artists have the ability to replace thousands of routine jobs in seconds. I’ve listed five trends to watch in this space, and it is by no means exhaustive.

1. The Rise of Smaller Open-Source LLMs

For those new to this field, a cursory reading of the history of technology reveals that big tech companies like Microsoft and Oracle used to be staunchly opposed to open-source technologies, but abandoned them after realizing they could survive without doing so. Unable to stay, hugged him. Open source language models are demonstrating this once again.

In a leaked document accessed by Semianalysis, a Google employee claimed, “Open-source models are faster, more customizable, more private and pound-for-pound more efficient. They cost $100 and with 13B parameters” working we’re fighting for $10M (million) and 540B (billion). And doing so in week, not monthThe employee believes that people will not pay for a restricted model when free, unrestricted alternatives are comparable in quality. quickly. We should be making smaller variants more later now that we know what’s possible in the <20B parameter regime".

Google may or may not subscribe to this approach, but the fact is that open-source LLMs have not only come of age but are providing developers with a lighter and more flexible option. For example, developers are flocking to LLAMA–Meta’s open-source LL.M. META’s Large Language Model Meta AI (LLAMA) “requires little computing power and resources to test new approaches, validate the work of others, and explore new use cases”. according to meta, Foundation models are trained on a large set of unlabeled data, which makes them ideal for fine-tuning a wide variety of tasks. Meta made LLaMA available in several sizes (7B, 13B, 33B, and 65B parameters) and also shared an LLaMA model card detailing how it built the model, in contrast to the lack of transparency in OpenAI.

According to Meta, smaller models trained on more tokens – fragments of words – are easier to re-train and fine-tune for specific potential product use cases. Meta says it has trained LLaMA 65B and LLaMA 33B on 1.4 trillion tokens. Its smallest model, LLaMA 7B, is trained on one trillion tokens. Like other LLMs, LLMA takes a sequence of words as input and recursively predicts the next word to generate the text. Meta says it chose texts from 20 languages ​​with the most speakers, focusing on those with the Latin and Cyrillic alphabets, to train LLaMa.

Similarly, Low-rank optimization of large language models (LoRA) is claimed to reduce the number of trainable parameters, which optimizes the storage requirement for LLMs to specific tasks and enables efficient task-switching during deployment without latency. “The LoRA adapter also outperforms many other optimization methods, including prefix-tuning and fine-tuning.” In simple terms, developers can use LoRA to fine-tune LLaMA.

Pythia (from EluetherAI, which itself is compared to an open-source version of OpenAI) consists of 16 LLMs trained on public data and sizes ranging from 70M to 12B parameters.

Databricks Inc. released its LLM in March called Dolly, which it “trained to demonstrate ChatGPT-like human interactivity for less than $30.” “And particularly fine-tuned after the dataset on a new, high-quality human-generated instruction, crowdsourced among Databricks employees”. The company has complete Dolly 2.0, which includes training code, datasets and model weights, for commercial use, allowing any organization to build powerful LLMs without paying for API access or sharing data with third parties. Enables you to create and customize your own.

Of course, we can’t ignore Hugging Face’s Big Science Large Open-Science Open-Access Multilingual Language Model (BlueOM), which has 176 billion parameters and is capable of generating text in 46 natural languages ​​and 13 programming languages. Researchers can download, run and study Bloom to examine the performance and behavior of the recently developed LLM. The open-source LLM march has only just begun.

2. Is Generative AI Really Smart?

The power of the LLM, as I have often pointed out in earlier newspapers, stems from the use of convolutional neural networks that are able to read multiple words (sentences and paragraphs too) at once, figure out how they are related, and predict the following word. Chatbots like GPT like LLM and ChatGPT are trained on billions of words on the internet, including books and sources like Common Crawl and Wikipedia, which makes them more “knowledgeable but not necessarily more intelligent” than most humans as they could be. Are. Able to connect the dots but not necessarily understand what they spark. This means that models like LLM like GPT-3 and ChatGPT can outperform humans in some tasks, but they cannot understand what they read or write as we humans do. Furthermore, these models use human observers to make them more sensible and less toxic.

A new paper by lead author Rylan Schaefer, a second-year graduate student in computer science at Stanford University, only confirms this thinking. It reads: “With larger models, you get better performance,” he says, “but we don’t have evidence to suggest that the whole is greater than the sum of its parts.” You can ask ‘Are the emerging capabilities of large language models a mirage?’ You can read the title paper. Here. The researchers conclude that “we find strong supporting evidence that contingency capabilities may not be a fundamental property of scaling AI models”.

That said, developments in the field of AI (and generative AI) are happening too quickly to stick to any one approach, so all I can say for now is that until we get more data from the opaque LL.M. Till then let us hold our horses. OpenAI and Google’s.

3. The Dark Side of Generative AI

On May 1, alarm bells started ringing loudly when one of the so-called godfathers of AI, Geoffrey Hinton, left Google. According to The New York Times, his reason was that “…he could speak freely about the risks of AI”. “Part of him, he said, now regrets his life’s work”. Hinton, who clearly understands the technology deeply, said in the NYT article cited above, “It’s hard to see how you can prevent bad actors from using it for bad things”.

According to the article, Hinton’s immediate concern is that “the Internet will be filled with false photographs, videos and text, and the average person” will no longer be able to know what is true. , over time, upend the job market.” The fear is that generative AI is getting smarter with each passing day, and researchers are unable to understand the ‘how’ of it. Simply put, since GPT- 4 As large language models (LLMs) are self-supervised or untrained, researchers cannot understand how they train themselves and arrive at their conclusions (hence, the term ‘black box’). Furthermore, for example For example, Tencent has reportedly launched ‘deepfakes-as-a-service’ for $145 – it only needs three minutes of live-action video and 100 spoken sentences to create a high-definition digital human.

You can read more about Here And Here.

4. Generative AI for Enterprises

While AI was discussed by 17% of CEOs in the January-March quarter of this calendar year, generative AI accounted for 2.7% of all earnings calls specifically, driven by the release of ChatGPT and discussions around its potential use cases. And conversational discussion was held. AI was mentioned in 0.5% of all earnings calls – up from zero mentions in the October-December quarter, according to the latest ‘What CEOs Talk About’ report by IoT Analytics – a Germany-based market insight and strategic business intelligence provider.

Generative AI multi-modal models and tools including ChatGPT, DAL-e, Mid-Journey, Stable Diffusion, Bing, Bard and LLAMA are making waves not only because of their ability to create, review, and create blogs, but also to write reviews. videos, and generate software code, but also because they can help accelerate new drug discovery, create entirely new materials, and even generate synthetic data.

That said, once companies adopt generative AI models, they will need continuous monitoring, re-train and fine-tuning to ensure that the models continue to produce accurate outputs and stay up-to-date . Furthermore, integrating application programming interfaces (APIs) with other units’ business workflows poses its own set of challenges for companies. Still, given the frantic pace at which these models are training themselves, and the ChatGPT business pending launch, business executives would benefit from being proactive.

5. Global railings are falling in place

The EU’s AI Act, for example, now proposes that AI tools should be classified According to their perceived risk level – from low to limited, high and unacceptable.

The US-based National Artificial Intelligence Advisory Committee (NAIAC), among other things, states: “We recognize that trustworthy AI is not possible without public trust, and without clear mechanisms for transparency, accountability, and mitigation of losses.” Public trust cannot be achieved. , and prevention. Governance must require an approach that avoids these risks while allowing the benefits of value-based AI services to reach the public.”

India too needs to act fast to keep the unbridled AI horse from running amok. whyYou can read more about this in my previous newsletter: ‘We must rein in precocious generative AI kids. But how?’

This article is this week’s edition of Leslie D’Monte’s Tech Talk newsletter. Subscribe here.