This is Part II of an ongoing series on AI content generation — how and why it works, and how to use the new tools that are coming out. Part I lays a conceptual foundation for everything discussed in the present article, so be sure and read it if you missed it.

Subscribe now

Despite how fast the machine learning space moves, you can navigate the rapidly shifting terrain if you know what kinds of problems ML researchers are racing to solve.

There’s a constellation of discrete tasks that researchers are trying to apply ML to, tasks where the success conditions are pretty well understood and where progress can be easily measured and communicated to potential users.
Large models are developed, refined, and trained to accomplish one or more of these tasks. They’re then released for public use, either in user-facing product form (Google), as an API (OpenAI), or as open-source software (Stability AI).
B2B and B2C software developers build user-facing apps on top of those models by calling APIs that give them access to the tasks the models can perform.

Not only are there many different markets for ML, but there are many different kinds of markets — and all of these markets will be powered by a much smaller number of discrete capabilities that ML researchers will enable. So if you understand the capabilities and the models, you’ll have a much easier time keeping up with the constant flow of new apps.

The bulk of this article, then, is an overview of ML tasks and models that’s aimed at orienting you to the space and giving you something to refer back to as you explore further on your own. I’ll start by breaking down some of the tasks in that figure above, then we’ll look at the model layer, and finally, I’ll end with pointers to resources for keeping up with the app churn.

Contents:

Machine learning tasks
- Training vs. inputs
- An aside about cognition
- Interpreting images
- Transforming and generating images
- Interpreting text
- Transforming and generating text
Getting to know the models
- GPT-3
- GPT-J
- DALL-E 2
- Stable Diffusion

Machine learning tasks

The sections below focus on machine learning tasks that deal with text and images. Once you’ve got these under your belt, it’s pretty easy to understand these same tasks when they’re translated to audio and video. The names and functions are recognizable across domains, so there’s not a lot of need to repeat the material.

There are some fundamental types of ML tasks we’ll encounter in both images and texts. In the previous article, I put these tasks into three main categories: generation, classification, and transformation. We can expand that task list, though, to make it a bit more granular and descriptive:

Classifying a whole thing, e.g., “this sentence expresses a negative feeling,” or “this is a photograph of a natural environment.”
Classifying part of a thing, e.g., “this word in this sentence is a noun,” or “this region of pixels contains a tree.”
Comparing two things to measure sameness/difference, e.g., “this caption does/doesn’t fit with this image.”
Extracting part of a thing, e.g., “here’s a sentence from this document that’s an answer to a question,” or “here are all the faces in this photograph.”
Transforming a thing, e.g., “here’s a version of your input text at a lower grade level,” or “here’s a version of your input photograph of Robert De Niro rendered as a My Little Pony character.”
Generating a thing, e.g., “here’s the text an informed writer might produce in response to your prompt, or “here’s the image a talented artist might paint in response to your prompt.”
Predicting a thing, e.g., “here’s the likely next word in this incomplete sentence you just typed,” or “here are the likely pixels that might fill in the gaps in your input image.

There’s plenty of slippage between the above categories, so it’s best not to get too hung up on putting different tasks into the “correct” one. For instance, text and image generation is actually a flavor of prediction; also, extraction depends on classification, and the distinction between “whole” and “part” can change with context. But generally speaking, the above is a good list of the kinds of things ML does.

We can also split the above list into two even more general categories:

Numbers 1-4 are about interpreting a piece of input to identify, label, classify, and separate things that are the same from things that are different.
Numbers 5-7 are about generating a new thing that shares desired similarities with the input thing.

It’s weird the way that you can kind of break down ML tasks into “reading vs. writing,” isn’t it? (Or, at least it’s weird to me.) And just like with reading and writing, the different task categories depend on one another in interesting ways.

Another point worth pondering — especially if you’re an AGI skeptic — is the way that for a human to demonstrate an understanding of some phenomenon, topic, text, art, vocational skill, or other object of study, we’d expect them to be able to do some amount of all of the above types of tasks in relation to that object. In other words, in the course of being formally evaluated as a student or apprentice in some area, you’re going to need to do a mix of all six of the above before you’re judged as having mastered the material.

Computers are getting very good at all of the above task types in isolation and in specific contexts. At some point soon, some system somewhere will be very good at all using all six of these task types together in general and novel contexts. Anyway, more on this topic in the “What’s Next?” post I’m working on.

jonstokes.com

AI Content Generation, Part 2: Tasks And…

Keep reading with a 7-day free trial