Generating LoRAs

LoRAs are like specialized mini-models that can be used when generating images with Stable Diffusion. They’re often used for adding characters or specific concepts not known by the base model. This page contains documentation on how I generate LoRAs.

Setting up the environment

I am unable to run kohya_ss on my own machine, so I will be renting one from RunPod instead.

1. Create an account on RunPod
2. Provision a server with the SDiffusion Dreambooth ControlNet Deforum Kohya template with at least 15 GB container storage and 30 GB persistent storage (personally I use 25 GB and 100 GB)
3. Click Connect
4. Connect to HTTP Service [Port 8888] to open Jupyter Lab and upload your training data with the following folder structure
   • /workspace/<lora name>/image/<number_of_repeats>_<lora name>
     • image1.jpg
     • image1.txt (caption)
   • /workspace/<lora name>/log
   • /workspace/<lora name>/model
5. Use the machine to download the base model
   1. In Jupyter, open a Terminal
   2. Run a command like nohup wget -qc "https://civitai.com/api/download/models/28562?type=Model&format=SafeTensor&size=pruned&fp=fp16" -O /workspace/stable-diffusion-webui/models/Stable-diffusion/anyloraCheckpoint_novaeFp16Pruned.safetensors & (this downloads the AnyLORA noVae fp16 model in the background)

Training the LoRA

1. Back in RunPod, click Connect to HTTP Service [Port 3010] to open kohya_ss GUI (wait until it’s ready)
2. Click the Dreambooth LoRA tab
3. In Model Quick Pick, select Custom
4. In Pretrained model name or path, type the path to your base model (e.g. /workspace/stable-diffusion-webui/models/Stable-diffusion/anyloraCheckpoint_novaeFp16Pruned.safetensors)
5. Click the Folders tab
6. Enter the Image folder (e.g. /workspace/<lora name>/image)
7. Enter the Output folder (e.g. /workspace/<lora name>/model)
8. Enter the Log folder (e.g. /workspace/<lora name>/log)
9. Enter the Model output name
10. Click the Training parameters tab
11. Enter the training parameters according to the type of LoRA you are making
    • Reference: [Guide] Make your own Loras, easy and free
    • Reference: ULTIMATE FREE LORA Training In Stable Diffusion! Less Than 7GB VRAM!
    • Reference: THE OTHER LoRA TRAINING RENTRY
12. Enter a sample prompt (e.g. (masterpiece, best quality:1.2), character name, 1girl, solo, cowboy shot, looking at viewer --n (worse quality, lower quality:2) --w 512 --h 768)
13. In the Jupyter Terminal, run tail -f /workspace/logs/kohya_ss.log to check the logs for kohya_ss
14. Click Train Model