Goal: to test the ability of a lightweight model built and trained locally to invent insects based on a custom-built dataset, and then animate them realistically.
Instead of relying on off-the-shelf generative models, I built and trained custom architectures from scratch. The pipeline was executed in two phases: first, a VAE developed with TensorFlow on Google Colab, followed by a GAN trained locally using Keras and PyTorch. The animation and upscaling stages were then arranged within ComfyUI, using a cutting edge open-source stack including Wan2.2, TripoSplat, and Real-ESRGAN.
This project is inspired by Timothy Morton’s dark ecology, positioning humans in an entangled and complex relationship with the non-human, within an ecosystem far removed from any romanticized vision of Nature. It explores a relationship that is symbiotic by nature, yet violent and destructive within the current context of human dominance over our surroundings.
My goal was to make visible the silent disappearance of a part of the living world, while questioning the utilitarian view of the biosphere and the subjugation of what is often framed as inferior: insects. Yet, they constitute the bulk of animal biomass and we depend on them entirely. Woven throughout this work is the exploration of a human–insect–machine ecosystem.
first result with a VAE
I had already taken an interest in dataset building and model fine-tuning, so it felt only natural to start exploring how to build a model from scratch. Choosing open-source tools stems from a personal conviction, as an alternative to commercial, often opaque systems.
My first pick was auto-encoders, notably VAEs: they're fast and lightweight, and allowed me to get started quickly, as they ran perfectly well on Google Colab (using TensorFlow). But while the first steps of training brought some visually interesting results, I soon realised that, by design, VAEs allow for less variation, less novelty, even after some intensive dataset tweaking.
interesting shapes obtained with a VAE
Building a GAN was the next logical step, tackled locally using Keras, PyTorch and the excellent Keras resources, which helped me keep my sanity whenever I hit a wall (overfitting, insane training times, and the dreaded but oh so common mode collapse). One central question persisted throughout: what makes an insect identifiable? At what exact moment do we recognize an insect within a still blurry form, and consequently, when should the training be stopped?
Curating the dataset required extensive work. I initially collected and selected 65,000 images, which grew to 85,000, before being trimmed down to 57,000 to achieve a proper balance. The initial sources came primarily from museum collections, desaturated specimens with a vast majority of beetles. Introducing open-source images from iNaturalist, shared by everyday contributors, brought a wider variety of poses and colours, leading to more vibrant results. However, these images introduced a new challenge: cluttered botanical backgrounds made it difficult to isolate the subject, complicating the training process. I wrote an initial script using BiRefNet to mask the insects and isolate them against a solid white background, a successful approach that yielded significantly better results when training the GAN.
Once I generated a satisfying body of images, I attempted to convert them into 3D images using TripoSplat, an open-source model I imported in ComfyUI and built a workflow around. But the generated images weren't quite suitable for this, and the results weren't really usable as they were.
To bring the insectomorphs to life, I finally opted for 2D generative animation. The open-source model of choice was WAN 2.2's "image to video" with its LoRAs, and its corresponding workflow in ComfyUI. Generative video models can have a tendency to morph or generate things to easily match the content of the prompt, so I ended up tweaking the KSamplers and the negative prompt to create animations as faithful as possible to the source image while still allowing the model room to interpret the presented forms. The addition of movement (and legs) brought the final results seen on this page.
There is still room to expand this project, notably building an XR experience to anchor these insects in our space, a project that is currently under development.
