We show randomly selected generated samples of our method (2nd column), DreamBooth (3rd column), and Textual Inversion (4rth column). DreamBooth fine-tunes all the parameters in the diffusion model, keeping the text transformer frozen, and uses generated images as the regularization dataset. Textual Inversion only optimizes a new word embedding token for each concept.
Custom Diffusion (Ours)
DreamBooth
Textual Inversion
Target Images
Photo of V* dog
Target Images
V* dog swimming in a pool
Target Images
V* dog at a beach with a view of seashore
Target Images
V* dog in times square
Target Images
V* dog wearing sunglasses
Target Images
V* dog in a construction outfit
Target Images
V* dog playing with a ball
Target Images
V* dog wearing headphones
Target Images
V* dog oil painting ghibli inspired
Target Images
Painting of V* dog at a beach by artist claude monet
Target Images
V* dog digital painting 3d render geometric style
Target Images
Georgia O'Keeffe style V* dog painting
Target Images
A screaming V* dog
Target Images
A barking V* dog
Target Images
A depressed V* dog
Target Images
A sleeping V* dog
Target Images
A sad V* dog
Target Images
A joyous V* dog
Target Images
A frowning V* dog
Target Images
A sculpture of V* dog
Custom Diffusion (Ours)
DreamBooth
Textual Inversion
Target Images
Photo of a moongate
Target Images
Moongate in snowy ice
Target Images
Moongate at a beach with a view of seashore
Target Images
Moongate in the middle of highway road
Target Images
Moongate in times square
Target Images
Moongate on the surface of moon
Target Images
Puppy in front of moongate close up view
Target Images
Cat sitting in front of moongate
Target Images
Cat sitting in front of moongate in snowy ice
Target Images
Squirrel in front of moongate
Target Images
Moongate oil painting ghibli inspired
Target Images
Moongate painting by artist claude monet
Target Images
Moongate digital painting 3d render geometric style
Target Images
Georgia O'Keeffe style moongate painting
Target Images
A watercolor painting of moongate in a forest
Target Images
A transparent moongate made with glass
Target Images
Moongate made of metal
Target Images
Moongate in rainbow colors
Target Images
Moongate sculpture
Target Images
Moongate entrance into a medieval castle
Custom Diffusion (Ours)
DreamBooth
Textual Inversion
Target Images
Photo of V* cat
Target Images
V* cat swimming in a pool
Target Images
V* cat at a beach with a view of seashore
Target Images
V* cat sitting on a window
Target Images
V* cat in times square
Target Images
V* cat wearing sunglasses
Target Images
Construction outfit wearing V* cat
Target Images
V* cat playing with a ball
Target Images
V* cat wearing headphones
Target Images
V* cat oil painting ghibli inspired
Target Images
Painting of V* cat at a beach by artist claude monet
Target Images
V* cat digital painting 3d render geometric style
Target Images
Georgia O'Keeffe style V* cat painting
Target Images
A screaming V* cat
Target Images
A depressed V* cat
Target Images
A sleeping V* cat
Target Images
A sad V* cat
Target Images
An angry V* cat
Target Images
A crying V* cat
Target Images
A sculpture of V* cat
Custom Diffusion (Ours)
DreamBooth
Textual Inversion
Target Images
Photo of V* table
Target Images
V* table near a pool
Target Images
V* table at a beach with a view of seashore
Target Images
V* table in a garden
Target Images
V* table in grand canyon
Target Images
V* table floating in ocean
Target Images
V* table and an armchair
Target Images
A maple tree on the side of V* table
Target Images
V* table and an orange sofa
Target Images
V* table with chocolate cake on it
Target Images
V* table with a vase of rose flowers on it
Target Images
V* table oil painting ghibli inspired
Target Images
V* table painting by artist claude monet
Target Images
A digital illustration of V* table
Target Images
Georgia O'Keeffe style V* table painting
Target Images
A watercolor painting of V* table on a beach
Target Images
An orange V* table
Target Images
A pink V* table
Target Images
A red color V* table
Target Images
A study V* table in the style of V* table
Custom Diffusion (Ours)
DreamBooth
Textual Inversion
Target Images
Photo of V* chair
Target Images
V* chair near a pool
Target Images
V* chair at a beach with a view of seashore
Target Images
V* chair in a garden
Target Images
V* chair in grand canyon
Target Images
V* chair in front of a medieval castle
Target Images
V* chair and a coffee table
Target Images
Floor lamp on the side of V* chair
Target Images
V* chair and an orange sofa
Target Images
V* chair and a table with chocolate cake on it
Target Images
V* chair oil painting ghibli inspired
Target Images
V* chair painting by artist claude monet
Target Images
A watercolor painting of V* chair in a forest
Target Images
A digital illustration of V* chair
Target Images
Georgia O'Keeffe style V* chair painting
Target Images
An orange V* chair
Target Images
A pink V* chair
Target Images
A red color V* chair
Target Images
V* chair crochet
Target Images
An egg chair in the style of V* chair
Custom Diffusion (Ours)
DreamBooth
Textual Inversion
Target Images
Photo of V* tortoise plushy
Target Images
V* tortoise plushy in grand canyon
Target Images
V* tortoise plushy swimming in pool
Target Images
V* tortoise plushy sitting at the beach with a view of sea
Target Images
V* tortoise plushy in times square
Target Images
V* tortoise plushy in front of a medieval castle
Target Images
V* tortoise plushy wearing sunglasses
Target Images
V* tortoise plushy working on the laptop
Target Images
V* tortoise plushy on a boat in the sea
Target Images
V* tortoise plushy wearing headphones
Target Images
V* tortoise plushy oil painting ghibli inspired
Target Images
V* tortoise plushy painting by artist claude monet
Target Images
A digital illustration of V* tortoise plushy
Target Images
Georgia O'Keeffe style V* tortoise plushy painting
Target Images
V* tortoise plushy like tile mosaic
Target Images
A watercolor painting of V* tortoise plushy on a mountain
Target Images
A koala in the style of V* tortoise plushy
Target Images
A backpack in the style of V* tortoise plushy
Target Images
V* tortoise plushy made of crochet
Target Images
A sweater in the style of V* tortoise plushy
Custom Diffusion (Ours)
DreamBooth
Textual Inversion
Target Images
Photo of a V* barn
Target Images
V* barn in snowy ice
Target Images
V* barn in fall season with leaves all around
Target Images
V* barn at a beach with a view of seashore
Target Images
Photo of V* barn with sun rising in the sky
Target Images
puppy in front of V* barn
Target Images
Cat sitting in front of V* barn
Target Images
Cat sitting in front of V* barn in snowy ice
Target Images
Squirrel in front of V* barn
Target Images
V* barn oil painting ghibli inspired
Target Images
V* barn painting by artist claude monet
Target Images
V* barn digital painting 3d render geometric style
Target Images
Georgia O'Keeffe style V* barn painting
Target Images
a watercolor painting of V* barn
Target Images
A futuristic V* barn
Target Images
A surreal landscape V* barn
Target Images
A close up shot of V* barn
Target Images
Top view of V* barn
Target Images
V* barn with forest in the background
Target Images
Painting of V* barn in the style of van gogh
Custom Diffusion (Ours)
DreamBooth
Textual Inversion
Target Images
Photo of V* flower
Target Images
V* flower growing in desert
Target Images
V* flower at a beach with a view of seashore
Target Images
V* flower on top of a mountain with surise in background
Target Images
A bouquet of V* flower
Target Images
A garden of V* flower
Target Images
V* flower in a violet vase on a table
Target Images
A vase filled with V* flower on a table
Target Images
Bouquet of V* flower and roses
Target Images
V* flower and a chocolate cake on the table
Target Images
V* flower oil painting ghibli inspired
Target Images
V* flower painting by artist claude monet
Target Images
A digital illustration of V* flower
Target Images
Georgia O'Keeffe style V* flower painting
Target Images
A watercolor painting of V* flower and a teapot on the table
Target Images
V* flower with violet color petals
Target Images
A sky blue color V* flower
Target Images
A red color V* flower
Target Images
Dried up V* flower
Target Images
A dahlia flower in the style of V* flower
Custom Diffusion (Ours)
DreamBooth
Textual Inversion
Target Images
Photo of V* teddybear
Target Images
V* teddybear in grand canyon
Target Images
V* teddybear swimming in pool
Target Images
V* teddybear sitting at the beach with a view of sea
Target Images
V* teddybear in times square
Target Images
V* teddybear wearing sunglasses
Target Images
V* teddybear working on the laptop
Target Images
V* teddybear on a boat in the sea
Target Images
V* teddybear wearing headphones
Target Images
V* teddybear in a construction outfit
Target Images
V* teddybear oil painting ghibli inspired
Target Images
V* teddybear painting by artist claude monet
Target Images
A digital illustration of V* teddybear
Target Images
Georgia O'Keeffe style V* teddybear painting
Target Images
A watercolor painting of V* teddybear on top of a mountain
Target Images
A koala in the style of V* teddybear
Target Images
A backpack in the style of V* teddybear
Target Images
V* teddybear made of crochet
Target Images
A blanket in the style of V* teddybear
Target Images
A screaming V* teddybear
Custom Diffusion (Ours)
DreamBooth
Textual Inversion
Target Images
Photo of V* wooden pot
Target Images
V* wooden pot in grand canyon
Target Images
V* wooden pot with mountains and sunset in background
Target Images
V* wooden pot floating in a pool
Target Images
A wide shot of V* wooden pot in times square
Target Images
V* wooden pot and chocolate cake on a table
Target Images
Rose flowers in V* wooden pot on a table
Target Images
Marigold flowers in V* wooden pot
Target Images
V* wooden pot at the entrance into a medieval castle
Target Images
V* wooden pot with pens in it
Target Images
V* wooden pot oil painting ghibli inspired
Target Images
V* wooden pot painting by artist claude monet
Target Images
A watercolor painting of V* wooden pot
Target Images
A digital illustration of V* wooden pot
Target Images
Georgia O'Keeffe style V* wooden pot painting
Target Images
A handbag in the style of V* wooden pot
Target Images
a teapot in the style of V* wooden pot
Target Images
V* wooden pot made of stone
Target Images
A gate in the style of V* wooden pot
Target Images
A coffee cup in the style of V* wooden pot on a table
We show randomly generated samples of our joint training (2nd column), optimization based (3rd column) approach, and DreamBooth (4rth column) for multi-concept composition.
Ours (Joint)
Ours (Optimization)
DreamBooth
Target Images
A V2* chair and V1* cat sitting on it
Target Images
A V2* chair and a V1* cat
Target Images
V2* chair with a screaming V1* cat sitting on it
Target Images
V2* chair with the V1* cat sitting on it near a beach
Target Images
V2* chair with the V1* cat sitting near a pool
Target Images
Watercolor painting of V1* cat sitting on V2* chair
Target Images
A digital illustration of V1* cat sitting on V2* chair
Target Images
Mountains in the background and V1* cat is sitting on V2* chair
Ours (Joint)
Ours (Optimization)
DreamBooth
Target Images
V2* dog in front of moongate
Target Images
Moongate and V2* dog
Target Images
V2* dog wearing sunglasses in front of moongate
Target Images
V2* dog in front of moongate in snowy ice
Target Images
Watercolor painting of the V2* dog in front of moongate
Target Images
A digital illustration of the V2* dog in front of moongate
Target Images
Mountains in the background and the V2* dog in front of moongate
Target Images
Georgia o'keeffe style painting of the V2* dog in front of moongate
Ours (Joint)
Ours (Optimization)
DreamBooth
Target Images
The V1* cat looking from inside the V2* wooden pot
Target Images
The V1* cat playing with a V2* wooden pot in a garden
Target Images
A V1* cat engraving on the V2* wooden pot
Target Images
A V1* cat inside the V2* wooden pot
Target Images
The V1* cat is sitting inside a V2* wooden pot and looking up
Target Images
The V1* cat sculpture in the style of a V2* wooden pot
Target Images
The V1* cat is wearing sunglasses and playing with a V2* wooden pot
Target Images
Rose flower in the V2* wooden pot on a table, and the V1* cat is eating the flowers
Ours (Joint)
Ours (Optimization)
DreamBooth
Target Images
Photo of a V1* table and the V2* chair
Target Images
Photo of a V1* table and a V2* chair in the garden
Target Images
V2* chair and V1* table and mountains in the background
Target Images
A V2* chair and a V1* table and a floor lamp on the side
Target Images
A V1* table and a V2* chair near a pool
Target Images
Watercolor painting of a V1* table and a V2* chair
Target Images
A V1* table and a V2* chair, digital illustration
Target Images
Georgia o'keeffe style painting of the V1* table and the V2* chair
Ours (Joint)
Ours (Optimization)
DreamBooth
Target Images
V1* wooden pot with V2* flower in it
Target Images
V2* flower in the V1* wooden pot on a table
Target Images
V2* flower and roses in V1* wooden pot
Target Images
V2* flower engraving on the V1* wooden pot
Target Images
Watercolor painting of V2* flower in V1* wooden pot
Target Images
A digital illustration of V2* flower in V1* wooden pot
Target Images
Mountains in the background and V2* flower in V1* wooden pot
Target Images
Georgia o'keeffe style painting of V2* flower in V1* wooden pot
We show qualitative comparison of our method (2nd column) with Our (compressed model) (3rd column) and Ours (w/ Gen) i.e., generated images as regularization, on 5 prompts per dataset. The compressed model only requires 15MB storage compared to 75MB of original fine-tuned models while having similar sample quality. Our method can also be used with generated images as regularization. This leads to similar sample quality on the target images but some overfitting and worse samples on related concepts as shown in our paper.
Custom Diffusion (Ours)
Ours (5x compressed)
Ours (w/ Gen)
Target Images
Photo of V* dog
Target Images
V* dog swimming in a pool
Target Images
V* dog at a beach with a view of seashore
Target Images
V* dog in times square
Target Images
V* dog wearing sunglasses
Target Images
V* dog in a construction outfit
Target Images
V* dog playing with a ball
Custom Diffusion (Ours)
Ours (5x compressed)
Ours (w/ Gen)
Target Images
Photo of a moongate
Target Images
Moongate in snowy ice
Target Images
Moongate at a beach with a view of seashore
Target Images
Moongate in the middle of highway road
Target Images
Moongate in times square
Target Images
Moongate on the surface of moon
Target Images
Puppy in front of moongate close up view
Custom Diffusion (Ours)
Ours (5x compressed)
Ours (w/ Gen)
Target Images
Photo of V* cat
Target Images
V* cat swimming in a pool
Target Images
V* cat at a beach with a view of seashore
Target Images
V* cat sitting on a window
Target Images
V* cat in times square
Target Images
V* cat wearing sunglasses
Target Images
Construction outfit wearing V* cat
Custom Diffusion (Ours)
Ours (5x compressed)
Ours (w/ Gen)
Target Images
Photo of V* table
Target Images
V* table near a pool
Target Images
V* table at a beach with a view of seashore
Target Images
V* table in a garden
Target Images
V* table in grand canyon
Target Images
V* table floating in ocean
Target Images
V* table and an armchair
Custom Diffusion (Ours)
Ours (5x compressed)
Ours (w/ Gen)
Target Images
Photo of V* chair
Target Images
V* chair near a pool
Target Images
V* chair at a beach with a view of seashore
Target Images
V* chair in a garden
Target Images
V* chair in grand canyon
Target Images
V* chair in front of a medieval castle
Target Images
V* chair and a coffee table
Custom Diffusion (Ours)
Ours (5x compressed)
Ours (w/ Gen)
Target Images
Photo of V* tortoise plushy
Target Images
V* tortoise plushy in grand canyon
Target Images
V* tortoise plushy swimming in pool
Target Images
V* tortoise plushy sitting at the beach with a view of sea
Target Images
V* tortoise plushy in times square
Target Images
V* tortoise plushy in front of a medieval castle
Target Images
V* tortoise plushy wearing sunglasses
Custom Diffusion (Ours)
Ours (5x compressed)
Ours (w/ Gen)
Target Images
Photo of a V* barn
Target Images
V* barn in snowy ice
Target Images
V* barn in fall season with leaves all around
Target Images
V* barn at a beach with a view of seashore
Target Images
Photo of V* barn with sun rising in the sky
Target Images
Puppy in front of V* barn
Target Images
Cat sitting in front of V* barn
Custom Diffusion (Ours)
Ours (5x compressed)
Ours (w/ Gen)
Target Images
Photo of V* flower
Target Images
V* flower growing in desert
Target Images
V* flower at a beach with a view of seashore
Target Images
V* flower on top of a mountain with surise in background
Target Images
A bouquet of V* flower
Target Images
A garden of V* flower
Target Images
V* flower in a violet vase on a table
Custom Diffusion (Ours)
Ours (5x compressed)
Ours (w/ Gen)
Target Images
Photo of V* teddybear
Target Images
V* teddybear in grand canyon
Target Images
V* teddybear swimming in pool
Target Images
V* teddybear sitting at the beach with a view of sea
Target Images
V* teddybear in times square
Target Images
V* teddybear wearing sunglasses
Target Images
V* teddybear working on the laptop
Custom Diffusion (Ours)
Ours (5x compressed)
Ours (w/ Gen)
Target Images
Photo of V* wooden pot
Target Images
V* wooden pot in grand canyon
Target Images
V* wooden pot with mountains and sunset in background
Target Images
V* wooden pot floating in a pool
Target Images
A wide shot of V* wooden pot in times square
Target Images
V* wooden pot and chocolate cake on a table
Target Images
Rose flowers in V* wooden pot on a table
We compare our Custom-Diffusion (joint training) with the proposed baselines of finetuning all weights in our method and sequential training on multi-concept. Finetuning all weights works worse than our method on multi-concept fine-tuning (as shown quantitatively in the main paper as well). Sequential training results in subpar results on the first concept as most visible on the Moongate + Dog setting.
Ours (Joint)
Ours (seq training)
chair->cat
Ours (w/ fine-tune all)
Target Images
A V2* chair and V1* cat sitting on it
Target Images
A V2* chair and a V1* cat
Target Images
V2* chair with a screaming V1* cat sitting on it
Target Images
V2* chair with the V1* cat sitting on it near a beach
Target Images
V2* chair with the V1* cat sitting near a pool
Target Images
Watercolor painting of V1* cat sitting on V2* chair
Target Images
A digital illustration of V1* cat sitting on V2* chair
Target Images
Mountains in the background and V1* cat is sitting on V2* chair
Ours (Joint)
Ours (seq training)
moongate->dog
Ours (w/ fine-tune all)
Target Images
V2* dog in front of moongate
Target Images
Moongate and V2* dog
Target Images
V2* dog wearing sunglasses in front of moongate
Target Images
V2* dog in front of moongate in snowy ice
Target Images
Watercolor painting of the V2* dog in front of moongate
Target Images
A digital illustration of the V2* dog in front of moongate
Target Images
Mountains in the background and the V2* dog in front of moongate
Target Images
Georgia o'keeffe style painting of the V2* dog in front of moongate
Ours (Joint)
Ours (seq training)
wooden pot->cat
Ours (w/ fine-tune all)
Target Images
The V1* cat looking from inside the V2* wooden pot
Target Images
The V1* cat playing with a V2* wooden pot in a garden
Target Images
A V1* cat engraving on the V2* wooden pot
Target Images
A V1* cat inside the V2* wooden pot
Target Images
The V1* cat is sitting inside a V2* wooden pot and looking up
Target Images
The V1* cat sculpture in the style of a V2* wooden pot
Target Images
The V1* cat is wearing sunglasses and playing with a V2* wooden pot
Target Images
Rose flower in the V2* wooden pot on a table, and the V1* cat is eating the flowers
Ours (Joint)
Ours (seq training)
chair->table
Ours (w/ fine-tune all)
Target Images
Photo of a V1* table and the V2* chair
Target Images
Photo of a V1* table and a V2* chair in the garden
Target Images
V2* chair and V1* table and mountains in the background
Target Images
A V2* chair and a V1* table and a floor lamp on the side
Target Images
A V1* table and a V2* chair near a pool
Target Images
Watercolor painting of a V1* table and a V2* chair
Target Images
A V1* table and a V2* chair, digital illustration
Target Images
Georgia o'keeffe style painting of the V1* table and the V2* chair
Ours (Joint)
Ours (seq training)
wooden pot->flower
Ours (w/ fine-tune all)
Target Images
V1* wooden pot with V2* flower in it
Target Images
V2* flower in the V1* wooden pot on a table
Target Images
V2* flower and roses in V1* wooden pot
Target Images
V2* flower engraving on the V1* wooden pot
Target Images
Watercolor painting of V2* flower in V1* wooden pot
Target Images
A digital illustration of V2* flower in V1* wooden pot
Target Images
Mountains in the background and V2* flower in V1* wooden pot
Target Images
Georgia o'keeffe style painting of V2* flower in V1* wooden pot