How to Train an SDXL LoRA
SDXL LoRA training is mature, flexible, and easy to ruin with copied presets. The core job is simple: freeze an SDXL checkpoint, attach small trainable adapter layers, teach them from image-caption pairs, and export a .safetensors file. The production work happens around that loop—choosing the correct SDXL lineage, cleaning images, deciding what captions should preserve or vary, controlling memory, saving checkpoints, and testing beyond one flattering prompt.
This guide uses current concepts shared by kohya-ss sd-scripts and Hugging Face Diffusers without pretending one parameter set fits every character, style, product, or GPU. SDXL has two text encoders and a large UNet, so text-encoder training, caching, precision, and VAE behavior deserve deliberate choices. Pony is built on SDXL, but it has its own vocabulary and should be trained as a separate branch; use the dedicated Pony guide for that workflow.
Quick Answer
Train an SDXL LoRA against the same SDXL checkpoint family you'll use for generation. Build a varied, rights-cleared dataset, pair each image with an accurate caption and consistent trigger, use sdxl_train_network.py in sd-scripts or an SDXL LoRA script in Diffusers, start with UNet-only training, save intermediate checkpoints, and compare them with fixed prompts and seeds.
Don't treat “SDXL-compatible” as proof that every derivative shares the same vocabulary. Base SDXL, photorealistic fine-tunes, anime branches, Pony, and Illustrious can load similar file formats while responding differently. A LoRA that technically loads can still be ineffective or distort output when moved to the wrong lineage.
Choose the SDXL Base Before the Dataset
Your training base defines what the adapter can reuse. Pick a checkpoint whose existing anatomy, composition, subject domain, and prompt language are close to your target.
- SDXL 1.0 base: the clean reference for broad compatibility and neutral behavior.
- Photorealistic SDXL fine-tune: useful when the final workflow will always use that lineage, but compatibility with other SDXL checkpoints must be tested.
- Anime SDXL branch: better for illustrated targets when its tagging vocabulary matches your data.
- Pony or Illustrious branch: SDXL-derived, but specialized enough to require branch-specific captions and validation.
Save the exact base filename or repository revision. Don't quietly swap checkpoints between runs and compare their losses as if only the training settings changed.
Build a Dataset That Separates Signal from Noise
A character dataset should repeat identity while varying incidental details. A style dataset should repeat treatment while varying subjects and scenes.
Keep images that add information. Different angles, framing, expression, lighting, backgrounds, poses, and distances help. Near-duplicate frames mostly increase the weight of one view.
Delete defects you don't want learned. Blur, watermarks, compression blocks, accidental text, broken anatomy, and poor crops become training signal.
Use buckets instead of destructive square crops. SDXL is commonly trained with 1024-class resolution and aspect-ratio buckets. Follow the trainer's current recommendations and inspect every processed crop.
Hold out evidence. Keep several representative images out of training. They won't provide a formal scientific benchmark, but they help you resist judging only memorized views.
Write Captions for Control
Captions tell the adapter which features belong to the target and which should stay promptable.
For a character, use a unique trigger phrase and describe clothing, pose, setting, expression, framing, and lighting when they vary. If every caption says only the trigger, the LoRA can bind the whole image distribution to that token.
For a style, describe subject content while leaving the trigger responsible for the visual treatment. If captions omit content, the adapter may confuse recurring objects with the style itself.
Tag-style and natural-language captions can both work in SDXL, but stay consistent with the chosen checkpoint's vocabulary. Auto-taggers are assistants, not judges. Correct names, remove false attributes, normalize synonyms, and keep the trigger spelling exact.
Pick the Training Path
Kohya-ss sd-scripts
sdxl_train_network.py is the dedicated SDXL LoRA script. It accepts an SDXL model repository, Diffusers directory, or compatible .safetensors checkpoint and a dataset configuration.
Current sd-scripts documentation strongly favors UNet-only training for the basic SDXL LoRA path. SDXL has two text encoders, and training them adds complexity and memory use. If you cache text-encoder outputs, caption augmentation is restricted and the text encoders can't be trained in that cached path.
Hugging Face Diffusers
Diffusers maintains separate SDXL text-to-image and DreamBooth LoRA examples. The text-to-image script suits captioned datasets and concepts; the DreamBooth script suits subject adaptation with instance prompts.
Diffusers also documents an optional alternate VAE path because SDXL's VAE can be numerically unstable in FP16. Use the current example's recommended handling rather than copying an old command.
Numbered SDXL LoRA Training Workflow
1. Freeze the run definition
Write down the goal, exact base model, trainer commit, dataset revision, caption format, output name, seed, validation prompts, and intended inference UI. Decide what success and failure look like.
A character run might prioritize identity across clothing changes. A style run might prioritize treatment while preserving prompt control over subject matter.
2. Install the trainer cleanly
Use the project's supported Python and dependency instructions. Create a dedicated environment, install the documented Accelerate configuration, and verify that the base model can load before adding training.
Record package versions. Silent dependency drift can change precision behavior, optimizer support, or checkpoint serialization.
3. Create the dataset configuration
Point the trainer to image and caption folders, set resolution and buckets, and configure repeats or sampling deliberately. Inspect filenames, caption encoding, and bucket output.
Don't use repeats to rescue weak data. Repeats control exposure; they don't create missing poses, views, or diversity.
4. Choose adapter capacity
Rank and alpha control adapter capacity and scaling, but higher isn't automatically better. Small, focused concepts often work with modest rank. Broad styles may need more capacity.
Treat rank as an experiment. Compare checkpoints at equal data exposure instead of assuming a larger file contains a better model.
5. Configure optimization and precision
Start from the trainer's current SDXL example. Select a supported mixed-precision mode for your hardware, use a documented optimizer, and keep batch size conservative.
Gradient checkpointing, cached latents, cached text-encoder outputs, and 8-bit optimizers can reduce memory use. Each has constraints. For example, cached text outputs conflict with caption shuffling and text-encoder training in sd-scripts.
There is no reliable universal VRAM requirement. The official sd-scripts notes show SDXL LoRA can be configured for lower-memory cards using UNet-only training, checkpointing, caches, an efficient optimizer, and low dimensions, but actual fit depends on the complete run.
6. Run a short smoke test
Train only long enough to prove the pipeline. Confirm captions load, images bucket correctly, loss remains finite, samples render, and checkpoints reopen.
Watch CPU RAM, GPU memory, disk writes, and speed. Fix warnings now instead of accepting a run that happens to continue.
7. Train with intermediate saves
Save by steps or epochs at intervals that give you several candidate adapters. Generate the same validation prompts at each checkpoint.
Loss isn't a quality score. A lower value can accompany worse flexibility or visible overtraining.
8. Test outside the training prompt
Load the adapter into Forge, ComfyUI, AUTOMATIC1111, or Diffusers with the correct SDXL lineage. Use fixed seeds and compare:
- close, medium, and wide framing;
- new backgrounds and lighting;
- poses absent from the dataset;
- different prompt lengths;
- multiple adapter weights;
- the base checkpoint with and without the LoRA.
The best checkpoint preserves the target while leaving the base model responsive.
9. Diagnose failures
- Identity drifts: add defining views, improve captions, or train longer in small increments.
- Everything looks like one training image: remove duplicates, add variation, reduce exposure, or choose an earlier checkpoint.
- Trigger changes clothing or background: caption those details explicitly and diversify them.
- Colors or samples break in FP16: check the documented VAE and precision path.
- LoRA has little effect on another checkpoint: return to the training base; you may have crossed SDXL lineages.
- Prompt control collapses: lower adapter weight, use an earlier checkpoint, or reduce capacity and exposure.
10. Package and document
Export .safetensors with a model card that states the exact base model, trigger, intended use, recommended range, training resolution, known weaknesses, trainer version, dataset rights, and license.
Include a small prompt grid. A model card without reproduction details turns every user into a debugger.
Text Encoder Training: Usually Skip It First
SDXL's dual text encoders make this an advanced choice. UNet-only LoRA training is easier to fit, cache, and reproduce.
Train text encoders only when your experiment has a clear reason, such as teaching vocabulary that the base conditioning doesn't map well. Run a controlled comparison and expect higher memory use plus more restrictions around cached outputs.
How to Avoid Overtraining
Overtraining appears as memorized poses, sticky backgrounds, fixed clothing, reduced variety, harsh textures, or a trigger that overwhelms the rest of the prompt.
Save checkpoints and stop choosing by the final step. Reduce data repeats or total exposure, improve dataset diversity, lower capacity, or use an earlier checkpoint. One clean restart is better than stacking random fixes onto an unclear run.
Planned Local Workflow
Read the LoRA Studio overview for LocalForge AI's planned approach to organizing image review, caption edits, run records, and checkpoint comparisons. It isn't a shipping trainer today, so use kohya-ss or Diffusers now and validate the resulting adapter in your chosen inference UI.
Next Steps
- Choose and record the exact SDXL lineage.
- Audit the dataset for duplicates, weak crops, and unwanted artifacts.
- Edit captions to separate the target from variable details.
- Run UNet-only first with intermediate checkpoints.
- Test the selected adapter on fixed prompts in your real inference UI.
Bottom Line
SDXL LoRA training rewards controlled experiments. Keep the base and dataset fixed, change one major setting at a time, and judge intermediate checkpoints with the same prompt grid. The dataset and captions decide what can be learned; the preset only decides how the optimizer sees it.