How to Train an Illustrious LoRA
Illustrious LoRA training is not just generic anime SDXL training with a different checkpoint filename. The Illustrious XL lineage is built on Stable Diffusion XL and was developed around large, annotated illustration data, including Danbooru-style vocabulary. Official model cards also distinguish releases by lineage, native resolution, license, and training behavior. Those details affect how you choose captions, buckets, prediction settings, validation prompts, and the checkpoint that should load the finished adapter.
This guide focuses on training and testing an adapter for a declared Illustrious XL release. It does not rank Illustrious checkpoints or recommend a model roundup winner. You will freeze one exact base, verify whether it uses epsilon or v-parameterization, curate illustration data for its vocabulary, start with an SDXL-compatible UNet LoRA path, save intermediate adapters, and test on the training base before attempting transfer to another Illustrious derivative.
Quick Answer
Train an Illustrious XL LoRA with an SDXL-capable trainer, but configure the run for the exact Illustrious checkpoint rather than for "anime SDXL" in general. Record the checkpoint revision and hash, confirm its prediction type, use aspect-ratio buckets appropriate to that release, caption with accurate Danbooru-style tags or clear natural language, and start with a UNet-only adapter.
Save intermediate checkpoints and test them with Illustrious-native prompts at several LoRA strengths. The selected adapter must preserve the concept while allowing changes in pose, clothing, composition, background, and style. Prove it on the training base first. A successful load on Pony, base SDXL, NoobAI, or another illustration checkpoint does not establish compatibility.
Know Which Illustrious You Are Training
"Illustrious" names a model lineage, not one immutable checkpoint.
The official Illustrious XL v0.1 model card identifies it as an illustration-focused Stable Diffusion XL model continued from Kohaku XL Beta 5 and fine-tuned on Danbooru2023. It describes v0.1 as an untuned base intended to support later variants, LoRAs, and adapters.
The official v1.0 card describes a later SDXL checkpoint trained from v0.1, with natural-language and Danbooru-style prompting and native high-resolution behavior up to 1536 by 1536. The v2.0-STABLE card describes a stabilized point from the v2 training schedule. Other releases and community derivatives can differ again.
Before creating the dataset, capture:
- repository or source page;
- exact version and revision;
- local filename and SHA-256 hash;
- declared base lineage;
- prediction type, usually epsilon or v-parameterization;
- model license and any use restrictions;
- expected prompt vocabulary;
- documented resolution range;
- VAE requirements;
- intended inference checkpoint.
Do not rely on a marketplace label alone. Two files can both say Illustrious while representing different releases, merges, or prediction modes.
Prediction Type Is a Run-Level Requirement
SDXL architecture tells the trainer which network layout to use. It does not by itself tell the trainer how the checkpoint parameterizes the denoising target.
If the selected release uses epsilon prediction, use the trainer's documented epsilon-compatible path. If it uses v-parameterization, the training configuration, scheduler assumptions, and inference workflow must explicitly support that mode. Do not apply a v-pred preset merely because a newer Illustrious release exists, and do not force an epsilon preset onto a v-pred checkpoint.
A mismatch can produce unstable loss, weak learning, broken samples, or an adapter that only behaves in a mismatched workflow by accident. Confirm the prediction type from the primary model card or configuration, then save it in the run manifest.
When documentation is incomplete, generate an untouched baseline from the checkpoint in the intended inference UI. If the model itself cannot produce a stable known-good sample, LoRA training is not the next debugging step.
Decide What the Adapter Should Learn
Write a narrow success statement:
- a character that retains identity across outfits and scenes;
- an illustration style that transfers across unrelated subjects;
- a garment that preserves construction across poses;
- an object that remains recognizable from several views;
- a visual feature that can be added without forcing one composition.
Keep the concept stable and vary everything that should remain promptable. Illustrious already has broad illustration knowledge. The LoRA should add the missing concept or treatment, not relearn generic anime anatomy from a repetitive folder.
For a character, vary framing, angle, pose, expression, clothing, lighting, and background. For a style, vary subjects, character count, settings, palettes, camera positions, and composition. For clothing, vary wearer and pose while preserving actual garment structure.
Remove near-duplicates, watermarks, signatures, text overlays, malformed anatomy, accidental crops, and inconsistent identity. An illustration model may reproduce a polished defect convincingly, so visual appeal is not a substitute for correctness.
Use Resolution Buckets Deliberately
Illustrious v1.0 documentation emphasizes native high-resolution generation and a range extending to 1536-class outputs. That does not mean every LoRA must train every image at 1536 square.
Training resolution affects memory, speed, crop behavior, and the scale of details the adapter sees. Use aspect-ratio buckets supported by the current trainer. Choose a target that:
- matches the selected checkpoint's documented behavior;
- fits the real detail in the source images;
- fits hardware without unstable offloading or constant failure;
- preserves important full-body and wide compositions;
- can be held constant across comparison runs.
If your sources are roughly 1024-class, upscaling them to 1536 does not invent trustworthy detail. A controlled 1024-class pilot can prove captions and concept learning before a higher-resolution experiment. If fine linework or small garment details genuinely require more pixels, test a higher bucket set as a separate run.
Inspect bucket assignments and processed crops. Reject a setup that cuts off hair, hands, feet, props, garment hems, or character pairs. Record bucket limits and whether images were resized, cropped, or padded.
Caption for Illustrious Vocabulary
Illustrious was trained around annotated illustration data and supports Danbooru-style tags. Later official cards also describe natural-language understanding. Both formats can work, but the selected release and concept should determine the choice.
A tag caption for a character might be:
ilxchar, 1girl, short blue hair, amber eyes, black coat, standing, city street, night, three-quarter view
A natural-language caption might be:
ilxchar stands on a city street at night, wearing a black coat, shown from a three-quarter view.
Use a unique trigger that the base does not already strongly associate with another concept. Check it in the tokenizer if your workflow exposes tokenization, but do not chase a mythical one-token rule at the expense of a stable, uncommon string.
Character captions
Caption clothing, pose, expression, view, character count, and background when they vary. Keep permanent identity features consistent. If one outfit appears in most images, either add variation or accept that the outfit may bind to the trigger.
Style captions
Describe subjects, scene, composition, palette, medium, and visible content while leaving the style trigger responsible for the shared treatment. If every caption says only the trigger, repeated characters and layouts can become part of the style.
Tag hygiene
Normalize underscores, spaces, synonyms, character counts, view tags, and trigger spelling. Remove false auto-tags. Do not copy prompts into captions when the requested content is absent from the final image.
Illustrious and Pony are separate ecosystems. Do not import Pony score_ and source_ conventions into an Illustrious dataset unless the exact Illustrious derivative explicitly documents them. Similar anime output does not make vocabularies interchangeable.
Set Up an SDXL-Compatible Trainer
Kohya-ss sd-scripts provides sdxl_train_network.py for SDXL-family LoRAs. Hugging Face Diffusers provides SDXL LoRA examples for captioned text-to-image and subject-oriented DreamBooth workflows. Use a maintained version that can load the exact checkpoint and prediction type.
For a first run, prefer UNet-only LoRA training. SDXL has two text encoders. Training text encoders adds memory use and more variables, while cached text-encoder outputs can restrict caption augmentation and text-encoder training.
Record:
- trainer commit or package version;
- network module;
- rank and alpha;
- optimizer and learning rate;
- precision;
- batch size and accumulation;
- bucket settings;
- repeats, epochs, and total steps;
- gradient checkpointing;
- latent and text-output caches;
- seed;
- save interval;
- sample prompts;
- prediction type.
Do not begin with an unexplained community preset. A preset can be a reproducible starting point only when every field is visible and appropriate to the frozen checkpoint.
Numbered Illustrious Training Workflow
1. Freeze the base and inference target
Save the exact checkpoint, revision, hash, license, and primary model card. Generate a baseline grid in the intended UI using documented prompts and resolution.
2. Freeze the dataset revision
Keep originals outside the training directory. Remove defects and duplicates, confirm rights and consent, and create a manifest linking every image to its caption.
3. Build captions and trigger checks
Select tag or natural-language format. Normalize vocabulary and inspect every pair. Search for missing, doubled, or misspelled triggers.
4. Configure buckets
Choose a documented target range, enable aspect-ratio buckets, and preview processed crops. Do not assume maximum native inference resolution is the required training resolution.
5. Start with UNet only
Disable text-encoder training for the baseline. Choose modest adapter capacity and a documented optimizer path. Higher rank is not automatically better.
6. Run a smoke test
Train only long enough to prove that:
- the Illustrious checkpoint loads;
- prediction mode is accepted;
- all images and captions are counted;
- buckets are sensible;
- loss remains finite;
- samples render;
- checkpoints save and reload.
Stop on unexplained prediction, VAE, missing-key, or NaN warnings. A run that continues is not necessarily a valid run.
7. Train with intermediate saves
Save several adapters across the run. Generate the same prompt grid at each save. Do not select the last checkpoint automatically, because later saves may strengthen likeness while reducing flexibility.
8. Validate on the training base
Test the adapter alone with no unrelated LoRAs, embeddings, prompt expanders, or controls. Compare base-only and LoRA-enabled generations at fixed seeds.
9. Test one variable at a time
Change clothing, pose, expression, background, composition, and framing independently. Test several LoRA strengths. Include prompts without the trigger and prompts using both tags and natural language when the checkpoint supports both.
10. Package the winner
Export .safetensors and document the exact Illustrious base, prediction type, trigger, caption convention, training resolution, suggested strength range, trainer version, license, known failures, and tested derivatives.
Build an Illustrious-Specific Validation Grid
A useful grid contains at least these groups.
Recognition
Use a short prompt with the trigger and concept class. This proves that the adapter learned a usable association.
Tag control
Change Danbooru-style attributes that were captioned, such as hair, clothing, pose, view, and background. Look for attributes that remain stuck to the trigger.
Natural-language control
Describe a relationship or composition in a sentence. This tests whether the adapter preserves the checkpoint's sentence response instead of only working in one tag list.
Composition and resolution
Test portrait, landscape, square, close, medium, and wide outputs at documented inference sizes. Do not judge a high-resolution lineage using only one square portrait.
Trigger absence
Generate without the trigger at the same adapter strength. Strong concept leakage may indicate overtraining or a globally acting style adapter.
Checkpoint progression
Compare at least three intermediate saves. Score target fidelity, prompt control, anatomy, line quality, color stability, and composition diversity separately.
Transfer
Only after the training base passes, test one named Illustrious derivative. Record its hash and result. Do not turn a successful derivative test into a claim that the LoRA supports every Illustrious, SDXL, Pony, or NoobAI checkpoint.
Diagnose Common Illustrious Failures
The LoRA loads but does little. Confirm the exact base, trigger, adapter strength, and whether training actually targeted the UNet. Inspect caption loading and compare an intermediate save.
Samples become unstable or washed out. Recheck prediction type, scheduler assumptions, VAE path, precision, and the untouched base output before changing rank or repeats.
The character always has one outfit. The dataset repeated that outfit, captions omitted it, or exposure is too high. Add real variation, correct captions, or choose an earlier checkpoint.
Tag prompts work but sentences fail. The dataset may use only short tag lists, or the selected derivative may favor a particular prompting convention. Add controlled sentence-caption examples only if the target workflow needs them.
Natural language works but precise attributes drift. Normalize attribute tags, correct false captions, and verify that the images actually provide distinct evidence for each attribute.
High-resolution samples repeat subjects or break composition. Inspect training crop distribution, source resolution, bucket balance, and the base checkpoint's documented inference sizes. More pixels do not fix narrow composition data.
The LoRA works on the training base but fails on another anime model. That is an ecosystem mismatch until proven otherwise. Return to the frozen Illustrious checkpoint and document transfer as unsupported.
Everything looks like one artist or character. Remove duplicates, diversify subjects and compositions, reduce exposure, and confirm that captions separate content from style.
When to Train the Text Encoders
Keep text encoders frozen for the first usable baseline. Consider a controlled text-encoder experiment only when the UNet-only adapter reproduces visual features but cannot establish a necessary new trigger association.
If you test text-encoder LoRA:
- use trainer guidance specific to SDXL's two encoders;
- disable incompatible text-output caching;
- lower the number of simultaneous changes;
- compare against the UNet-only run at matched exposure;
- watch for prompt-language damage and overbinding.
Text-encoder training is not a repair for inaccurate captions, wrong prediction mode, a mismatched checkpoint, or missing visual evidence.
Release Checklist
Before distribution, verify:
- the exact Illustrious checkpoint and hash are recorded;
- prediction type is explicit;
- license and dataset rights were reviewed;
- captions match the selected Illustrious vocabulary;
- Pony-specific conventions were not imported by assumption;
- bucket crops preserve important content;
- intermediate checkpoints were compared;
- validation includes tags, sentences, aspect ratios, and trigger absence;
- the adapter passes on the training base;
- derivative compatibility is limited to models actually tested;
- model-card examples include seeds and reproduction settings;
- known failures and suggested strengths are stated.
Bottom Line
An Illustrious LoRA is an SDXL-family adapter trained for a specific Illustrious checkpoint, vocabulary, resolution regime, and prediction mode. Freeze those facts before choosing training settings.
Build varied illustration data, caption it for the target release, start with UNet only, inspect buckets, and select an intermediate checkpoint through an Illustrious-specific prompt grid. Architecture compatibility gets the file loaded. Base lineage and validation determine whether it actually works.
What to Do Next
Lock the checkpoint family
Record the exact Illustrious release, hash, prediction type, and intended inference base.
Normalize captions
Choose tags or natural language and remove false, inconsistent, or borrowed vocabulary.
Test intermediate adapters
Compare fixed tag and sentence prompts across strengths, aspect ratios, and checkpoints.