Designers and automation: who the robots will replace and how to avoid it

I could have written an article about the hard skills a designer must have, but that would have been frivolous of me. Something is constantly happening to the market — technologies change, and professions change with them: designer, programmer and others. For example, diffusion neural networks like Midjourney have appeared recently, and now illustrators have to be able to work with them as co-authors.

So I’m going to talk not just about designers, but about automation — about what lies ahead for all of us designers, and how the technological environment will affect it.

Why should you listen to me? My background is design — I graduated from the Architectural Academy and in 1997 became one of the first three web designers in Yekaterinburg. And now I’m the director of a digital production company and I manage designers. We have four main lines of activity: product development, VR, AR and MR, UX/UI and digital marketing — and in all these areas the designer’s role is one of the leading ones.

What a designer does and how automation affects it

Let’s take some product — football boots, for example. How do you tell whether their design is good or bad? Check whether that design affects the people who buy the boots the way the manufacturer intended. Do the buyers get the value they spent their money on — do they like them, do they want to run in these boots, do they think they play football better in them? And in the end, do a lot of people buy them?

A designer is in the business of changing people’s behavior, creating an experience for the end users — not just the appearance of a product, but exactly the experience, which they affect through the observable properties of the product.

And as a result the company gets a benefit and the company’s client gets value.

I’ve broken a designer’s work down into 10 steps — that doesn’t mean you necessarily have to go through all of them on any given project, but the sequence is roughly this:

• Take the brief from the client
• Do research
• Design the experience
• Prepare prototypes
• Test the experience
• Develop the style
• Create the layout
• Do design supervision
• Analyze effectiveness
• Ensure the product interface keeps evolving

And now let’s go through each of the steps: what designers do better, what they do worse, and how robots handle the same tasks.

I’m focusing on automation because its progress is very fast, and robots are taking more and more routine work away from people — work that used to require human hard skills.

Don’t do work that a robot does better than you. You need to look for the places where a human is hard to replace, and develop yourself there. And also look for ways to do the work together with the robot.

Disclaimer

In this article, by “designer” I don’t mean the person who draws the GUI, but the person who designs the target experience. Very often this person, or rather a whole group of people, is called something else. So why do I say these are the designer’s tasks? Because I’m convinced that if you want to influence how the interface works, you need to take part in making these decisions, or at the very least understand them well.

1. Take the brief from the client

The first thing to do with a new project is to understand what the point of the design is: which business goal has to be achieved, what benefit has to be created. Next you should determine what experience to design in order to achieve that goal. And for that — research the context of the task: find out under what circumstances the end clients act. Then you need to find the means to solve the task — the available toolkit. For this you identify the team’s strong points: machine learning, say, or frontend, or sites on website builders, or low-code. And then propose solution options — what the final product might be.

Take Midjourney, Stable Diffusion and DALL-E. Several R&D teams invented diffusion neural networks that can turn a text query into an image expressively and precisely. And they started thinking about how to make products out of their inventions — that is, how to create an experience of human-machine interaction that would attract a lot of people who would pay the company money. Based on how each of the teams saw the goal, based on the surrounding context and on the available toolkit, Stable Diffusion and DALL-E solved this task by creating a graphical interface, while Midjourney solved it by creating an experience in the form of a conversation with a bot in Discord.

Which option is better? Or, to put it differently: what experience do these solutions create? What did the teams achieve? In short, it seems to me that in practice Midjourney’s approach worked better — despite the fact that a graphical interface is easier to understand from the first steps, working with a bot teaches users faster thanks to the P2P learning effect (person-to-person). The community teaches itself. Why am I saying this? The choice of means for solving the task determines all the rest of the design. And this example shows just how much.

How designers do on this

Most often — badly. The brief is mostly taken from the client by managers and passed on to the designer. As a result, the decisions that affect design the most are mostly made without the designers’ opinion being taken into account. Although a designer who can do this is far more valuable, and their salary is higher, and they have more influence over the project.

How robots do on this

Everything that goes into formulating the task is analytical work, requiring empathy, and most importantly — agency and the right to make a decision. So nobody handles it better than a human.

But robots can help. For instance, there are assistants that turn long speech recordings into text: Noty.ai, Otter.ai. And there are others that then pull the main meanings out of that transcript — Notion AI. You can also simplify the communication process — for example, write emails using the Voilà browser extension with ChatGPT, or create questionnaires using a dialogue builder.

2. Do research

Before designing a new experience, it makes sense to find out how people behave right now. For research people usually use analytics tools and talk to people a lot.

First of all you research the end clients’ processes — how and in which situations they use the product.

Let’s say we want to create a music player like Yandex Music, VK Music or Spotify. We have to understand how people use music players, what habits they have: do they use music as background for work or for physical training, or listen to lift their mood, or put it on to dance at a party.

I often listen to music in headphones when I’m walking the dog. In cold weather I rarely take my phone out, and the only way for me to interact with the player is through the buttons on the headphones. Those are my processes.

But there are a lot of players’ users, and their processes are different. To choose who to concentrate on, designers research the clients themselves: who they are and what clusters they fall into.

It’s also important to research analogues, the best interface practice — what other players exist, what form factor they have, what features they have and how they change people’s experience. What do you need in order to find analogues? Go and google in English. And search not only for interfaces that have already been published, but also for academic papers related to the experience of using them, and early prototypes.

How designers do on this

The ability to do deep research on your own topic is extremely rare. Everyone can find published pictures of interfaces, but few can go further, even though it isn’t hard. So research is a big gap.

A candidate who knows how to do research has a gigantic advantage, because they can lead the team into an area where there is no good practice yet.

How robots do on this

There is no centralized service yet that will do comprehensive research on any process for you, but there is a mass of services that solve separate fragments of the task. They contain:

• Collections of interface patterns;
• Modern best practices;
• Research panels where users are gathered whose experience you can study in conversation;
• Methods of extracting insights from data (the same Notion AI);
• Data on how people’s behavior changed depending on various factors, and so on.

There is a lot of information about client behavior on the internet. And the bigger the company a designer works for, the more automated this area is inside it. But designers will still go on studying clients’ processes with the help of empathy, their brain, and the ability to work with analytics tools and accumulated information.

3. Design the experience

Designing the experience is creating scenarios of using something, and this is the main thing a designer should be doing themselves. For design, what matters is not so much how the technical system behaves as how the person using the technical system behaves.

For example, a programmer puts music on because they need to get into a state of flow in order to write code. They don’t want to interact with the interface of that music until they’ve finished working. And in that case the main task of the player’s designer is to design the experience so that the user isn’t yanked out of the state of flow.

Another point, often unexpected for a designer — when designing an experience you absolutely have to think about what exactly the content of the screens will be and what requirements there are for it. A very common mistake is when designers think their task is to create a container into which you can put anything at all. And what exactly to put in it — let somebody else think about that.

For example, when we’re making a news site and instead of pictures we draw squares with crosses inside them, and we paste Lorem ipsum into the text fields — that’s a monstrous mistake, because if we don’t picture what content the user will run into, we ourselves have no way of understanding whether the interface works or not. We simply won’t develop a reliable intuition about how a person will behave, because we won’t get a dense enough context. And on top of that we won’t be able to move on to testing, because it’s impossible to test a layout filled with “for example” content.

I’m not saying that a designer has to be able to write texts as well as a talented journalist, or must be able to draw by hand like an illustrator. It would be nice, of course, but it isn’t mandatory. What is definitely necessary is understanding what content will be in the interface, being able to select it, being able to brief the copywriter or the photo editor who will find that content. And using this information at the very earliest stages of working on the structure of the screens.

The next aspect of designing the experience is key metrics. A metric is a number that answers the question of whether we are really moving toward the goal we set ourselves.

For example, the design of an audio player can be analyzed by metrics like these: how many songs a person listened to in a row, how many times they came back, how long the session was, how often they hit like. And whether an online store has good design can be judged by conversion — how many of the people who tried to buy actually bought.

Obviously these numbers are affected not only by design but by a lot of other things. For instance, the number of people who will try in the first place is a function not of design but of the marketing campaign. But the ratio of the number of people who tried to the number of people who succeeded is largely a function of design.

Design affects the numbers that have to do with changes in people’s behavior after they have come into the product.

And here we move on to interface patterns — this is what human-machine interaction looks like in various environments, for example in graphical or audio interfaces, or simply in physical interaction.

A behavioral pattern is a stable habit people have of acting in a specific way in one context or another. When somebody says that an interface is convenient or intuitive — in 98 cases out of 100 that means it is familiar. Because people have no intuition about interfaces at all if they’ve never come across anything similar before. So designing an experience has to start with researching how people are used to solving tasks like this.

How designers do on this

Most designers don’t know what designing an experience is — and that’s the industry’s main hang-up. Or rather, not quite. Most of them know the fashionable word UX and have read something about it. But in practice a designer still thinks in terms of functionality descriptions and pictures of the interface, not in terms of changes in people’s states. In other words, instead of designing an experience, the designer designs screens.

That’s because our profession got a bit unlucky: designers descend from artists, that is, from people who know how to make things beautiful. They think they’re supposed to be doing visual representation — drawing pictures, screens — and they believe the main thing in design is how the product looks.

I went to the Architectural Academy because I wanted to be a book illustrator, and not at all an engineer who designs people’s behavior. And that’s typical. It would be great if designers descended from film directors, game developers or psychologists — then it would be easier for them to create behavior scenarios.

In interface design, the main thing is not how they look, but how people behave when using them. A user experience designer is called that because they design experience. Screens are only the means of organizing that experience.

How robots do on this

For standard tasks, a huge pile of practically ready-made templates has been accumulated by now. When they’re used, in most cases no designing happens — people simply take a template, launch it into life and then fix what came out, looking at the users’ reaction. An excellent approach, in which neither designers nor developers are needed. We take an existing best practice and apply it.

But when we run into a non-standard task, we have to think with our heads. I don’t know of any successful attempts to hand this part over to machines beyond standard cases. That said, designing through dialogue interfaces like ChatGPT greatly expands the palette of cases that can be considered standard. That is, best practice is discovered and automated faster and faster, and there is less and less room left for custom interface development.

It’s a different matter if you look for services that won’t replace the designer in the task of designing the experience, but will help them — take the routine off their hands. For instance, preparing the content for a mockup. Texts can be created with the same ChatGPT or with Gerwin.io, and brought into the shape the web needs in Lebedev’s typograf.

And image processing services automate various tasks: removing watermarks — WatermarkRemover.io, upscaling with quality improvement — Upscale.media, removing the background — Erase.bg, compressing — Shrink.media.

You can trust a robot with something more significant too — decomposing a task, for example. You formulate the goal, and GoalGPT puts together a clear plan of action for how to achieve it.

4. Prepare prototypes

Before special software appeared, we made paper prototypes — we drew the states of the interface on sheets of paper. That took up to several days. And although it’s a useful exercise for the mind, few people use it any more.

Prototyping today, in most cases, means working in Figma (a graphics editor where most interface designers work). There you can draw screens that are barely distinguishable from real ones: they look and behave almost like the real thing. And you can interact with them almost like with the real thing — click and move from one to another.

Interactive prototypes usually include the main scenarios — the ones the product was created for. A music player was created for listening to music, and a store for buying something. So for an e-commerce app the main screens will be the search results, the product card, the home page with promotions, the cart page and the payment page. In reality, though, the app has not 5 screens and states but, let’s say, 300 — the rest are just less important. And there are of course even more combinations of them.

How designers do on this

Prototyping is a native designer competence. But with the arrival of Figma it stopped being a specific skill that only designers have. Any team member can create a prototype, and that’s often what happens. Especially given the existence of plugins that turn a website into an editable Figma file in one click.

How robots do on this

There are already plenty of experiments where a system can offer several screens to choose from and build an app out of them. For example, the same Midjourney, or no-code services: Bravo, Softr, BuilderX. And there’s also Skybox Lab for creating 360° panoramas, Durable for generating sites from a text query, and Taplink for generating sites from an Instagram page*.

But here’s the important part: this only works for widespread interface patterns. For example, a music player is a widespread application, so the system can create something like that. But if you want to create an invention — a service in which people will behave in a new way — you’ll run into a problem.

There are more and more heuristics that help make a prototype out of best practice. And the day isn’t far off when prototypes will be recommended by a machine. But for now, designers have to create new practice themselves.

5. Test the experience

Testing is usually done by psychologists or usability specialists, but in my view every designer should be able to test their own interface, because if you can’t test an interface, you can’t understand how it works.

I can’t imagine how you can do design for a long, long time and not show it to a single user. For me the working scenario is this: I spend three or four hours drawing an interface and get a prototype that needs testing — it’s early, not very tidy, it only has the main scenarios and reasonably plausible content. And I show it to people who are as similar as possible to the users of my application.

That’s how it went, for example, at Rideró — the self-publishing platform I’m a co-founder of. Right now we’re designing the reading experience in the interface, to change how people read books inside our service. When I showed the prototype to people who have the habit of reading e-books and watched how they behaved with the interface, I discovered (and this happens practically every time you do something for the first time) that my hypotheses were wrong, that everything I’d drawn wasn’t as clear to people as I thought, and that they had different habits.

The earlier a designer starts testing the interface, the faster they’ll manage to create a quality experience that people will like.

Testing the experience of using an interface consists of several items:

• In-depth interviews — a conversation with users for the sake of discovering the details of their experience.
• Surveys — putting together questionnaires and analyzing the answers.
• Web analytics — working with Google Analytics and Yandex Metrica. It’s important to lay goals out across the interface — to define which places the user has to reach for the interface to be considered working. Goals can be laid out by a web analyst too, but the designer has to be able to brief the web analyst, has to be able to look into web analytics reports, because that’s exactly where you can see in numbers how the interface works.
• Usability testing — observing how a person gets the experience of using the product. This includes preparing test cases — selecting people for the testing itself, preparing questions, testing hypotheses.
• Analyzing the results — assessing how the interface works.

How designers do on this

It’s not that it’s totally sad: everyone has plenty of their own experience of using tools, so it isn’t hard to put yourself in the shoes of the person who is going to use the product being tested. Although a designer usually doesn’t command the full spectrum of instruments: in big companies a researcher is a separate specialization, and in small ones there’s often not enough time and money for it.

How robots do on this

In the area of surveys and usability testing there are a great many tools, but no full automation. Services will help gather the data and will try to highlight interesting places where potential insights might lie. You’d think the task of drawing conclusions from data would be quickly taken away from people by machines. But the actual practice so far is that our environment is too fragmented, and the integration effort required in each particular case is too big for robots to become radically cheaper than people.

6. Develop the style

Some people might think this item is about the ability to draw. But I’ll say it again: drawing beautifully is a core skill for an illustrator, but not for a designer. For an interface designer it’s very useful to be able to do it, but not mandatory — they’re an engineer to a greater degree.

And to develop a style you don’t have to be an artist. Unlike the situation I had in ‘97, designers now have the whole internet in front of their eyes — they can easily find good examples of style and reproduce them.

Developing a style also consists of several stages:

• Analyzing style references — going through the specimens of style that users will compare the product with.
• Looking for expressive devices — choosing fonts, color, defining the rhythm. A purely graphical story.
• Developing the concept and picking a visual metaphor. For example, the collaboration platform Miro has a very clear visual metaphor — a whiteboard. And the visual metaphor of digital players is the previous players, the mechanical ones.
• Creating a palette of patterns — this is a Lego set of design elements (the style of fonts, buttons, cards and so on) that the designer assembles in a separate place in the file and out of which they make the rest of the interface — rather than making every screen from scratch.
• Building the composition — managing a person’s perception. Any interface has to be easy to perceive, and for that it has to have a clear visual hierarchy, which designers build with the help of composition.
• Adding key images. What isn’t written in words, people perceive through image-based thinking. The designer has to determine which images need to be shown to users and add them to the interface.

How designers do on this

Designers know how to develop a style. And many of them consider it one of the main things they’re busy with. But I think it’s already clear to you that this isn’t quite so.

How robots do on this

Diffusion and GAN networks, for example DALL-E or Glide, can analyze pictures, so they have a million collections of devices. A year ago I still thought neural networks were bad at creativity and at finding a visual metaphor. But today I think neural networks are already creative enough, and that creativity is growing fast. So in a couple of years, not using a neural network in a brainstorm as one of the participants — or the main one — will be strange at the very least.

Besides that, there are a huge number of services with collections of patterns that try to help make composition more effective and expressive. For example, a Figma plugin that lets you draw inspiration from ready-made sites. But integrating this whole huge and constantly growing palette of possibilities still remains on the human.

Services like these can also be useful to a designer when developing a style: CLIP Interrogator 2.1, which produces a prompt for an uploaded image, or Fontjoy for generating font pairs.

7. Create the layout

The layout is how the buttons, tabs and icons look, that is, the appearance — not the concept of how the product works, not the general approach to the interface, not the idea of how people will use it. This is what designers are taught on courses, and what their portfolios mostly consist of — the portfolios employers use to assess candidates. And that’s a kind of industry dislocation, because in a designer’s work layouts aren’t the main thing either.

One way or another, the ability to create layouts includes:

• Working with a proportion grid;
• Typography — using type as an instrument;
• Ensuring consistency — so that the whole interface is done at one density, in one stylistic manner;
• Creating a design system — a collection of all the patterns present in the design;
• Creating illustrations or interacting with an illustrator;
• Accidence — placing accents.

How designers do on this

Designers are good at creating layouts, but I’ll say again that this ability isn’t the main one.

How robots do on this

For everything that goes into working on a layout there’s a mountain of tools, and they’re progressing fast. The most obvious example is Figma with its auto layouts, components, styles and abundance of plugins. Now you don’t have to check colors, fonts and distances by hand. There’s also Webflow and Quarkly.

On the one hand, this creates far more creative freedom for the designer — doing design becomes more pleasant. On the other hand, the routine work is being taken away from designers — and in five years routine will be paid less than it is today.

8. Do design supervision

Design supervision is checking how far the finished interface behaves the way the designer intended. It includes:

• Talking to users — to understand whether they like it or not;
• Background testing of the interface — good designers, first at the testing stage and then while the product is in use, are constantly doing pixel hunting — looking for places where the interface has gone off;
• Interacting with developers — when designers realize that the design doesn’t work as intended, or works as intended but is inconvenient, they go to the programmers and solve the problem together.

How designers do on this

Design supervision is an important activity, and designers are usually fine with it, they like doing it.

How robots do on this

To understand how a design works, there’s no getting by without analytics. Which means robots have limited authority here. But individual services can simplify this work for a human — for instance, markup testing services like PixelPerfectTestMachine and the Markup Validation Service. Or the social media analytics service LiveDune, where you can monitor user comments.

9. Analyze the effect

Analyzing effectiveness is:

• The ability to look into web analytics in order to understand whether the interface works;
• Analyzing qualitative data — this is data from conversations with people and observations of them, interpreting the experience as it is, without translating it into numbers;
• Resolving contradictions. For instance, I’m constantly arguing with our sales director at Rideró about placing a banner in the book text editor that urges you to buy a proofreader. On the one hand it raises proofreader sales, on the other hand it wildly annoys the users who have no intention of buying them. There’s the contradiction: we have two goals, and we have to decide which one is more important.

How designers do on this

A huge problem, and one of the main things you have to teach designers in their first three years — how to answer the questions: why am I doing the work I’m doing at all? how do I tell that my work has brought a benefit to the company and its clients? and what is the point of my activity?

How robots do on this

There’s a mountain of services that make data more visual. To analyze the effectiveness of an interface you’ll have to go through several steps:

• Implement an analytics system (Yandex Metrica, Google Analytics 4 for the web, AppMetrica, GA4 or Amplitude for apps);
• Determine which metrics say something about the user’s interaction with the interface;
• Learn to collect and track them.

On top of that, you can set up an alarm for metrics getting worse and receive alerts when the numbers sag below critical values.

But automating the whole process isn’t possible at the moment, and a vivid example of why is automatic goals in Yandex Metrica and built-in events in Google Analytics. They automatically track forms on a site, and if a developer has coded some input element — search, let’s say — as a form, the system will automatically count it as a target action.

For social media analytics there’s DataFan, and LiveDune again; for internet marketing there’s MindBox. And for gathering and analyzing qualitative data there’s, for example, the neural network Sense Machine — it measures a person’s mood and real emotional response through facial expressions and eye movements. But the final conclusions still remain (for now) with people.

10. Ensure the product interface keeps evolving

Evolving products is an analytical skill that includes the ability to analyze metrics and to prioritize. In most cases this isn’t something a designer has the right to decide on their own; usually the product manager does it. But design has a very strong influence on this vision. And the effectiveness of design is directly tied to what the product’s top-level vision is. So it’s important for a designer to understand product management.

Evolving a product includes:

• Holding the trunk of the product — the ability to understand and preserve the product’s intent, to leave in the interface only the instruments that are directly connected to the trunk of the product;
• Coming up with improvements within the trunk of the product;
• Making decisions about turning around — the ability to understand that the team has come to a dead end and to start redoing everything;
• Redesign — if everything done up to that point has become outdated, then it has to be done anew, and for that you go through all the steps starting from the first one.

How designers do on this

As I’ve already said, product evolution is usually handled not by the designer but by somebody higher up, and not necessarily by someone who knows how to create interfaces. But if a designer can do product evolution, then they become the one higher up.

How robots do on this

Data can help make a decision about a redesign and can highlight the current problems. And also suggest best practice. But it won’t make the decision for you and won’t choose how exactly it should be done.

To sum up everything I’ve said: as best practice emerges and is recognized, individual fragments of it will be automated. Design will be needed as the position that integrates the fragments of solutions. First and foremost, a designer’s skills will be needed in the places where the experience is changing and there is no reliable best practice yet.

I’ve given some examples of various automation services (thanks to the designers and art directors at JetStyle for helping me collect them) — but there are far more of them. There are a lot of tool roundups on the internet that you can drown in — here’s one of them. Explore and find what can help you in your work.

Forecasts for the future

In the future the interface environment will change: a heap of new input-output interfaces connected to mixed reality and VR will appear, and voice, tactile and other input-output interfaces will change.

There will be a great deal of UX design without screens. In tactile interfaces, for example, there already aren’t any, and in a mixed-reality interface, when we look through glasses that overlay a three-dimensional model of the virtual world onto the real one, there is a screen — but it behaves as if it weren’t there. There’s no rectangle you have to look through. There’s a graphically enhanced 360° view of what’s around you.

And the most important thing — machines will get much better at predicting a person’s next steps. If the previous interface paradigm was built almost entirely on the user choosing themselves what needed to be done — clicking the button themselves, typing the words on the keyboard themselves, giving a voice command to Alice themselves — then the further we go, the more interfaces will anticipate what a person needs, and the interface environment will adjust to that.

All of this will lead to the emergence of new paradigms of the graphical user interface (GUI).

Skills that don’t go out of date

In a world where machines will take a lot on themselves, there will still be human skills in demand — based on my experience, I think the list is this:

• Reflection — the ability to analyze your own experience;
• The ability to make generalizations from observations — not just to observe some facts, but to be able to put two and two together and draw a conclusion;
• The speed with which a person masters new instruments;
• The ability to give feedback to colleagues and to receive feedback from them;
• The ability to model experience, people’s behavior, with the help of user scenarios.

What’s special about the designer’s role

Everything I’m talking about concerns designing experience, and this can be done not only by designers but also by programmers, managers, marketers, finance people, data scientists, and also by film directors, screenwriters, producers, game designers and game masters.

But mostly experience is designed by the folks who are responsible for beauty, because to a very large extent we perceive reality through our eyes.

Command of visual language, of visual culture, is very important, and that’s what’s unique about the designer’s profession.

In my view, designers have three roles that make them especially valuable:

• The tuning fork — this is a person with a sharpened taste, who has a very good feel for whether one or another property of the interface sounds precisely enough, hits the stylistic key precisely. An example of such a tuning fork for me is Jonathan Ive, who was Apple’s chief designer for a long time. At one point I moved from Windows to Mac purely because of stylistic preferences, because the Windows interface simply wasn’t as pleasant for me as the Mac one was.
• The director — this is the position closest to me. I think designers do a kind of directing that’s close to directing a game: there are a lot of people freely moving around the product and achieving their goals in such a way that the company gets a benefit, and their scenarios have to be directed. The designer-as-director looks at the system as a way of managing people’s behavior.
• The stalker — in my view, the most interesting role. That’s why I too went into design on the web at the time — because designers and programmers can invent a new experience that nobody has had yet before anyone else, that is, they can be the advance party that invents new interface practice and does experiments.

To become a designer who’s in demand and not worry about your future, it’s worth developing toward these three roles — and the rest will sooner or later be done for you by robots.

* Instagram is banned in the territory of the Russian Federation.