How Does ChatGPT Work?

By Rebecca Casale | Published April 2023

ChatGPT is an AI language model that can debate moral philosophy, write short stories, and explain how to do your taxes. This is how it works.

ChatGPT was developed by OpenAI using machine learning, a field of AI that allows a computer to imitate intelligent human behaviour. The latest version, ChatGPT-4, was trained on a breathtaking dataset of 100 trillion parameters.

What in the name of Skynet is a GPT? This acronym describes the underlying architecture of the language model. As a generative pretrained transformer, ChatGPT generates the next word in a sequence, is pretrained on vast amounts of text data, and transforms text inputs simultaneously.

GPT stands for Generative Pretrained Transformer

Key features of a Generative Pretrained Transformer (GPT) model.

Deep Learning and Neural Networks

Let's start with machine learning. This allows an AI to learn and adapt (in this case, to language) without following explicit blow-by-blow instructions. To generate text responses, ChatGPT uses an ML model that exploits algorithms and statistical analyses to infer patterns in text-based data.

The particular type of ML used by ChatGPT is called deep learning, processing language in a manner inspired by the neural networks of the human brain.

Human thoughts are generated by an interconnected network of neurons that carry electrical signals. In a similar way, artificial intelligence processes data using neural networks: complex webs of nodes divided into layers.

In a neural network, data is fed into an input layer that sorts and analyses the content. It's like when you tip out a jigsaw puzzle; you don't start building the picture immediately. First, you turn all the pieces the right way up and separate out the edges and colours.

The hidden layers do all the heavy lifting. ChatGPT's pretraining produced complex patterns of interconnectivity between the network nodes. It then draws on these learned associations to solve a new problem.

ChatGPT "talks" in the same way that you know two pieces of a jigsaw puzzle will fit together. You use pattern-finding instincts based on the unique features of each token.

Finally, the output layer presents the solution to the user. This is like taking a photo of your finished puzzle and sharing it on Instagram, you beautiful nerd.

Neural networks use interconnected nodes in a layered structure that resembles the human brain.

Large Language Models

However, ChatGPT is more than your average neural network. In fact, the deep learning architecture that drives ChatGPT is known as a multi-head transformer network. Here are key features of the model that are vital to how ChatGPT works.

Tokenization. When a new text input is received, tokenizers break the data down into the smallest meaningful bits. Tokens are defined at the word-level, character-level, or subword-level, which extends the model to all spoken languages as well as computer programming languages like Python and C++.
Positional Embedding. This assigns each token a unique address so that it can be maintained in sequence.
Multi-Head Self-Attention. This assign value weights to the tokens, allowing the model to simultaneously attend to different tokens and discover the relationships between them. Now you've got yourself a huge jigsaw puzzle and all your friends are invited to solve it in parallel. Nigel's working the edges, Barry's got the ground, and Jeff's working his magic on the sky.
Feed Forward Neural Network. The human brain has a recurrent neural network, which allows us to feed our outputs straight back into our input holes. This feedback is how we self-reflect and learn. ChatGPT, on the other hand, uses a feed forward neural network that creates a one-way flow of information. This is why, until it can mimic our feedback loops, AI struggles to reason.
Decoders. To generate text outputs, ChatGPT uses autoregression, a statistical model that predicts the next word or token in a sequence based on the context of previous tokens.

ChatGPT works using tokenizers, positional embedding, multi-head self-attention, a feed forward neural network, and decoders to generate text outputs

ChatGPT works using a decoder language model and pre-training based on vast amounts of text.

Having trained on an insane volume of reference texts like books, articles, and websites, ChatGPT not only possesses an extensive map of virtually all human knowledge, but also the many languages we use to communicate it.

The genie is out of the lamp. It appears to know everything—and yet, technically, it comprehends nothing; despite being based on the circuitry of the human brain, ChatGPT doesn't think like we do at all.

"My responses are generated based on statistical probabilities and patterns learned from data, rather than a deep understanding of language and meaning like humans possess." - ChatGPT-3

Is ChatGPT an AGI?

Microsoft has already published a paper that describes ChatGPT-4 as having "sparks" of artificial general intelligence, or AGI, driving us increasingly faster towards the singularity.

It's so fast, in fact, that many high profile experts have signed Pause Giant AI Experiments: An Open Letter. The note calls on all AI labs worldwide to immediately pause the training of AI systems for at least six months in order to put safety restrictions in place.

As Max Tegmark explains in the conversation below, this is a make-or-break moment for humanity as a species.

Arguably, ChatGPT isn't currently aware of itself, which is almost certainly a prerequisite for considering it a conscious agent. But that day may not be far off. Even without self-awareness, language models are extraordinarily powerful, especially when given a directive and read/write access to the internet.

According to The Alignment Problem, AI agents may inadvertently harm humans on the path to helping us. For instance, an AI tasked with reducing road deaths to zero could find any number of solutions, from rolling out more self-driving vehicles to activating all the nukes on Earth (indeed, the latter would be much quicker).

ChatGPT is an astonishingly powerful tool that gives gutter-dwellers like me faster access to all human knowledge. And yet, in the coming months and years, such large language models will become inconceivably potent, taking over our jobs, our societies, and maybe even our lives.

What Can ChatGPT Do?

Whew. That became rapidly existential. Let's make a U-turn and look at the capabilities of the chatbot we're dealing with today. Like the friendly paperclip with eyebrows who just wants to help us get on with our work.

Clippy: The Poor Man's ChatGPT.

As a new user, what strikes you first about ChatGPT is its ability to generate coherent, long-form responses to very specific questions. Indeed, you can have lengthy back-and-forth conversations without it forgetting what was said earlier. ChatGPT even compensates for my shitty hurried typing by intuiting the intended meaning.

ChatGPT has an enormous range of applications, from creating programming code to writing poetry. It can write in many styles, whether composing an abstract for your academic research paper, or explaining time travel in the manner of Rick Sanchez. Everything it says is tailored to your inputs, to help you work smarter.

Here are 12 things ChatGPT can already do, as suggested by ChatGPT.

ChatGPT generates text that summarises or explains any area of human knowledge.

Content generation. Generates written content such as articles, social media posts, and product descriptions.
Language translation. Assists with translating text from one language to another.
Customer service. Acts as a virtual assistant for customer service, answering FAQs and resolving issues.
Personal productivity. Helps with organisation, scheduling, task management, and reminders to improve productivity.
Tutoring and education. Tutors in various subjects, explains concepts, and helps overcome specific challenges.
Code generation. Generates code snippets, templates, or examples for coding tasks in various programming languages.
Data analysis. Assists with data analysis such as data cleaning, data visualisation, and statistical calculations.
Research assistance. Helps with information gathering, fact checking, and summarising research papers.
Creative writing. Assists with ideas, character development, plot creation, and other aspects of creative writing.
Virtual roleplay. Acts as a virtual character in roleplaying games or simulations for interactive storytelling.
Mental health support. Offers emotional support, coping strategies, and resources for dealing with common issues like stress.
Social interaction. Provides companionship, social conversations, and human-like interactions.

Here's a cool demonstration of AI creativity using a combination of ChatGPT-4 to write a poem, Eleven Labs to narrate it using text-to-speech, Midjourney to generate images, and Kaiber to animate them:

How to Write Good GPT Prompts

ChatGPT operates differently from search engines. Instead of relying on keywords and directly pulling content from human-authored sources, it compiles information from multiple sources and summarises it in a consistent style.

But ChatGPT isn't a mind reader. By default, it will give a fairly superficial overview of a topic and relies on you to dig deeper incrementally to get to the guts of it. Generally, direct and to-the-point text prompts produce more useful answers, but you should be careful not to overwhelm it by coming in at too many angles simultaneously.

YES to clarity. Clear prompts that convey your intent or question explicitly lead to more relevant responses. Example: Summarise the main features of the latest iPhone model.
NO to ambiguity. Vague prompts often miss their target, since the model doesn't know exactly what you're aiming at. Example: Tell me about the latest gadgets.
YES to structure. Disorganised prompts can generate responses that lack structure and coherence in return. Example: Quantum computing implications. Also talk about limitations and future prospects.
NO to complexity. Long or convoluted prompts can overwhelm the model by making too many demands, resulting in fragmented answers. Example: Explain the historical background, political implications, and social impact of the Industrial Revolution in Europe during the 18th and 19th centuries, focusing on key inventions, economic changes, and effects on labour and society.
YES to creativity. Imaginative prompts may pose hypothetical scenarios or request answers in the style of a specific author. This can trigger surprising and engaging interactions. Example: Write a story set in an AI research lab about the meaning of life, written in the style of Ernest Hemingway.

For more examples, see 100 ChatGPT prompts in the fields of business, education, history, marketing, art, medicine, and more.

Final Thoughts

ChatGPT is a huge deal in the field of AI language processing. Since its launch in late 2022, developers have been scrambling to channel ChatGPT's power into business tools that will eventually automate a significant number of jobs. While AI may make our work easier at first, ultimately it will supplant us completely.

In the meantime, it's clear that ChatGPT and other large language models will become rapidly more sophisticated, achieving more natural and engaging conversations, while overhauling the way we use our machines.

You can access ChatGPT v3.5 here. It's free and there's nothing to download. It's also worth saying hello to Bing Chat in a Microsoft Edge browser as it's powered by GPT-4.

Rebecca Casale is a science writer in Auckland, New Zealand. If you like her content, please share it with your friends. If you don't like it, why not punish your enemies by sharing it with them?

Subscribe Science Me | Instagram | LinkedIn