AI – Appearance Imitation

How to read the grey images next to each painting: they are the same pictures, passed through a high-pass filter that keeps only the fine detail (edges, texture, grain, brushwork). Where the layer is even grey, there is no detail. Do not look at the flowers but at the background: that is where the difference lies. In the photographs of Luchian’s paintings, the background is as busy as the flowers. The reason is simple: the canvas is woven everywhere, the brush went everywhere, the paint cracked everywhere. Matter does not know what it depicts. The Luchian model is the public model, to which, in “Flowerless”, we added a small learned layer (a LoRA) from 17 flower paintings by Luchian in the MNAR collection. It saw exactly these paintings, and yet its background is almost empty, while detail gathers into crisp lines around every shape. The model learned detail as a property of things, not of matter: it puts it where a poppy is, not where canvas is. Standard model does the same with a portrait: detail in the hair, the eyelashes and the lace, where the eye goes, and smooth skin.

At actual size, the difference shows even without the filter. Look for the areas where “nothing happens”: the background of “Albăstrele”, the neck of the generated portrait. In the painting, the nothing has relief and cracks. In the generated picture, the nothing is smooth, because the model had no reason to put anything there.

For 7 flower paintings, we set the photograph of the painting next to the pictures generated from the same description, by the Luchian model and by Standard model. We compared how much fine detail the flat areas carry (the background, the inside of the petals) relative to the outlines. In all 7 pairs, the generated pictures carry less: on average, about half as much as the photographs of the paintings. So the difference does not come from the small layer learned from only 17 paintings. It appears just the same in the whole public model. Where exactly the matter is lost is shown in part II: not where we expected.

Because the difference is not a fixed rule. Newer models work with a latent grid of 16 channels instead of 4, which keeps more fine detail; users add pores and film grain with “realism” LoRAs. Researchers still find statistical traces: diffusion models underestimate high frequencies, which is exactly the fine detail on this sheet. A detector trained on one model’s images recognises them almost perfectly. But it fails on models it has not seen, and after a simple compression or resizing, which is the ordinary journey of any image on the internet.

A painting or a photograph is a trace. The brush touched the canvas, light touched the film, time cracked the paint: the image carries the marks of this contact with the world. The model has contact with nothing. It learned, from millions of photographs, what traces look like, and places them where we expect to see them. The theorist John May calls this the statistical image: computed from gathered data, without the world ever having touched it. Even when models generate perfect cracks, a statistic will have put them there, not time.

Read each row from left to right: it is the same picture, as the model imagines it at different steps. Everything starts from noise, picked by the seed. Over 28 steps, the model removes a little noise at a time and, at each step, estimates what the finished picture will look like. Watch two things: how early the composition appears (by step 4 it is already there) and how late the detail arrives (the portrait’s lace comes only in the last steps). Why in this order? Because the model does not guess one particular picture, but the average of all the pictures that could fit what it sees. When there is a lot of noise, thousands of pictures fit, and their average is a blurred patch: all they share is the large arrangement of shapes. As the noise drops, the possible variants dwindle and detail can appear. With no words at all (last row), nothing narrows the variants, and the picture stays fog until halfway.

The model does not work with pixels but on a small grid of numbers, called the latent space: 128 × 128 points, each with 4 values, that is, 4 channels. In each row you see the 4 channels as grey images, enlarged without smoothing: each little square is one point, which will become 8 × 8 pixels. The four numbers are not colours. They are a code learned during training by a separate component, the VAE, which had to pack as much of the picture as possible into as few numbers as possible. That is why the first channel resembles brightness: it is the most useful information. The others mix colour and texture and cannot be read by eye. Compare the colour preview with the final picture: the composition and colours are already settled in the grid. The VAE decoder only translates it into pixels.

We also took the road backwards. We gave the VAE encoder the photograph of “Maci și spice în ulcică”, let it turn it into the 4 channels, then decoded it back. The weave of the canvas survived almost whole: the flat areas keep 45% of the outlines’ detail, against 50% in the photograph. So the 4-channel grid can hold the matter of the painting. But when we ask the model to make the same painting by itself, from the description, the flat areas keep 22% in the Luchian model and 13% in Standard model. The matter is not lost in the translation into numbers. It is lost in the generation. A simplified explanation: at each step, the model estimates the average of the pictures that could fit. A poppy’s outline sits in roughly the same place in every possible variant, so it passes through the average untouched. The weave of the canvas and the cracks are different every time and, on average, cancel out. The last steps add some of them back, but less than in reality: researchers have shown that diffusion models systematically underestimate fine detail.

Some works in the room generate in real time, with StreamDiffusion. To keep up, they use models that make a picture in 1–4 steps instead of 28, usually without CFG (part III) and with a smaller VAE decoder, Tiny VAE. The polish has no time to settle: the frames run into one another in their compositional, rawer form, like the early steps above.

In each row, the same seed and the same prompt; only the CFG scale changes, the parameter that anyone generating images adjusts, usually around 7. What it does: at every step, the model makes two estimates, one with the prompt and one without. The difference between them is “what the prompt adds”. The CFG scale says how many times this difference is amplified. Read the rows from left to right. At CFG 1 the difference is not amplified at all, and the model barely listens to the prompt: the portrait does not even become a portrait. The higher the CFG, the more insistent and more typical a version of the prompt the picture becomes. The Luchian model’s poppies grow ever flatter, more saturated, with ever harder outlines: the “Luchian” air is not added, it is exaggerated. The portrait slides toward an old studio photograph, sepia: for “a portrait of a woman, photo”, the most typical version seems to be an archival one, of the kind digitised collections have filled the internet with. The same exaggeration, repeated from one generation to the next, leads in “Autophagy” to collapse.

The negative prompt field, the text you want the picture to flee from, is usually left empty. Then the “without the prompt” estimate above is made with the empty prompt, and CFG pushes the picture away from it. In other words, every picture flees from what the model understands by “nothing”. To see how much this matters, we built variant C: the same Standard model, with a single change. Through a direct calculation, with no retraining (the UCE method, the same as in “Flowerless”), we rewrote the meaning of the empty text: for the model, “nothing” now means “a rough unfinished sketch”. It is as if every picture had, secretly, the negative prompt “sketch”. Compare the rows two by two: above, the untouched model; below, variant C, with the same seed. Pictures that were already polished barely move. Those that were engraving or pencil flee from sketchiness and become colour paintings. So the polish does not come only from what you ask for. It also comes from fleeing what the model considers “nothing”: change the meaning of “nothing”, and the direction of the polish changes.

Prompt recipes circulate on forums: “masterpiece, best quality, highly detailed, 8k, ultra realistic, cinematic, award-winning”. In the room, 7777 SlopMachine, the reconverted slot machine, turns them into casino symbols. We tested them. With the same UCE method, we redirected two families of tokens: in variant A, the “quality” words (masterpiece, highly detailed, 8k, cinematic, award-winning…) lead to “a sketch”; in variant B, the medium words (photo, oil painting, watercolor, 3D render…) lead to “a pencil sketch”. How to read the tables: above, the untouched model; below, the same command and the same seed, with the words redirected. If the rows differ, the words mattered. When the words are in the prompt, the rows differ radically: the portrait becomes a sketch, the photograph becomes a drawing. When the prompt is plain, “an old man” comes out practically identical, with all its glossy studio-photograph air. So the words do not bring it: it is the model’s default way of looking. The portrait of a woman still slides toward drawing, probably because, in the text encoder, words are not isolated: “portrait” is a neighbour of “masterpiece” and “photo”, and the redirection reaches it too.

The words go where they were sent.

The rows look alike: the polish did not come from the words. The exception, the portrait of a woman, shows how entangled words are in the text encoder.

In training, SDXL received, next to every image, two pieces of information about its source. The first: how large the original was. A 200 px photo from an old website and a 4000 px one from a modern camera end up at the same working size, but the model learns where each started. The second: the crop. Training images had to be square, so a piece was cut out of rectangular ones; the model learned, for each, how many pixels were cut from the top and from the left. The aim was that the model should not wrongly learn that subjects appear cut off by the edges, with heads missing their crowns. At generation, these numbers (original_size and crop_coords_top_left, called micro-conditioning) have default values, “large original” and “uncropped”, which the ordinary user never sees. We changed them. The picture is always 1024 px; we only claim a different source. With the same seed and prompt, a “128 px original” gives a painted 19th-century portrait; 256–512 give sepia photographs; 2048 gives a crisp modern portrait. Why? On the internet, small images are mostly old scans, thumbnails and archives; large ones, recent photographs. The model learned the association: small image, old image. And a claimed crop tells it the picture is a piece of a larger one: out comes a snapshot, a fragment, a face cut off by the edge.

Compare the rows column by column. The claimed size changes not only sharpness but the kind of image: on the left, illustrations and painted, framed portraits; toward the right, studio photographs, then modern photographs; with a crop, snapshots and fragments, like the face cut by the edge in the second row. With the kind of image, the person in it changes too. In the first row, for the same “a woman” and the same seed, there appear in turn an Asian woman, then a Black woman, in a black-and-white studio portrait, then, with a crop, a colour photograph under a line of “Copyright”. The reason: in the training data, each kind of image comes with the people it usually shows. The period illustration, the artistic studio portrait and the agency photograph have different populations. The model does not choose a person but a kind of image, and the person comes with it. The “Copyright” line is the direct trace of the agency photographs it learned from.

With the empty prompt, the only information the model receives is the claimed source. So it shows what images of that kind usually contain. A “128 px source” brings crowded interiors, carpets, textile patterns, like old scans; large sizes bring mostly clearly photographed objects and buildings; a crop brings fragments of streets, shop windows, even a screenshot. This is the “nothing” that CFG flees in part III: not a void, but an average of the internet, which changes with the source the model believes it is imitating.

In the photographs models learn from, crowds are seen from a distance, and smiling people appear mostly in portraits. In “Everybody, smile!” we built the Smiling model: again with UCE, the words for people now lead to “a smiling woman”. With it, the crowds came apart. One young woman steps out of the crowded subway car, smiling. The wall of workers becomes five people posing. The two small sisters at the end of the corridor become two laughing faces, close up. The smile came together with the framing of a portrait. Our hypothesis, not yet tested: a face in a crowd and a portrait face come from different photographs and occupy different places in the model. In the photographic variant, crowd faces come out more alike; when a face is brought to the foreground, a different kind of person appears, a catalogue person.

In 2023, a Bloomberg analysis of 5,100 images made with Stable Diffusion showed that, for high-paying jobs such as “CEO”, “judge” or “lawyer”, most of the people generated were lighter-skinned men. Model makers have since tried to correct such leanings, mostly in the words: rewritten prompts, filters, rebalanced data. The experiments on this sheet show that bias does not live only in the words. It also lives in parameters the user never sees, such as the claimed size of the source, which changes the era, the kind of image and even who appears in it.