LoRA Training Dataset Guide
Your LoRA will only learn what your dataset repeatedly shows it. A folder full of sharp images can still produce a bad adapter if every frame has the same pose, the captions hide important details, or the background appears often enough to become part of the concept.
The fix isn't collecting hundreds of random images. Build a small, deliberate set that separates the thing you want to teach from everything you want the base model to keep controlling. This guide covers selection, cleanup, composition, captions, folder structure, validation, and the final checks worth doing before you spend time on a training run. It applies to subject, character, style, object, and clothing LoRAs, while calling out where those dataset types need different treatment.
Quick Answer
A strong LoRA dataset is consistent in concept and varied everywhere else. Keep the subject or style recognizable, but vary pose, framing, lighting, background, expression, and camera angle whenever those details aren't part of the concept.
Start with a focused set of high-quality images, remove duplicates and obvious defects, caption what should remain controllable, and reserve a few prompts for checkpoint testing. Quality beats volume. Twenty deliberate images can teach more cleanly than a hundred near-duplicates.
What a LoRA Dataset Actually Teaches
Training doesn't understand your intention. It sees correlations between pixels and caption tokens. If a character always wears a red coat, the adapter may bind the coat to the character. If every product photo has a white background, white backgrounds may appear even when you request a street scene.
Think in two columns:
- Concept signal: The identity, visual style, garment, object, or feature the LoRA must reproduce.
- Incidental signal: The pose, crop, lighting, setting, accessory, expression, and camera treatment that should stay promptable.
Keep the concept signal stable. Break correlations in the incidental signal through deliberate variation and accurate captions.
Step 1: Define One Training Goal
Write a one-sentence target before collecting anything. “A LoRA that reproduces this person across different clothes and environments” is useful. “A LoRA that makes good pictures of this person” isn't.
Your goal determines what belongs:
- Person or character: Preserve identity while varying clothing, pose, expression, crop, angle, and location.
- Product or object: Preserve shape, materials, markings, and proportions while varying viewpoint, scale, and setting.
- Clothing: Preserve construction, silhouette, fabric, and details while varying wearer, pose, and background.
- Style: Preserve recurring visual treatment while varying subject matter, composition, palette, and scene.
Don't combine unrelated goals in a first run. A character-plus-style dataset can work, but it makes failures harder to diagnose because you can't tell which signal caused them.
Step 2: Collect Candidates, Then Curate
Gather more candidates than you plan to train. Curation should remove images, not rationalize weak ones.
Keep images that are:
- Readable: The concept is large enough to inspect and isn't hidden by heavy blur, overlays, or compression.
- Accurate: Anatomy, object geometry, and identity details are correct.
- Useful: The image adds a viewpoint, pose, crop, lighting setup, or context not already covered.
- Legitimate: You own the images or have permission and sufficient rights to use them for training.
Reject watermarks, text overlays, accidental borders, severe JPEG artifacts, unwanted signatures, and generated defects. A trainer doesn't know an extra finger is a mistake. Repeated errors become learned features.
Step 3: Remove Duplicates and Near-Duplicates
Exact duplicates increase the weight of one image. Near-duplicates do the same thing less obviously.
Keep one frame from a burst unless meaningful details change. For generated data, don't keep six outputs that share the same seed-like composition, face angle, and outfit. Pick the cleanest frame and use the freed space for a genuinely different example.
Duplication is especially damaging in small datasets. Five similar portraits in a 20-image set make one composition a quarter of the evidence.
Step 4: Build Deliberate Coverage
A subject dataset should usually include close, medium, and wider views. Include front, three-quarter, and side angles when they are available and accurate. Add expression and lighting changes without letting any one outfit or location dominate.
A style dataset needs a different balance. Keep the rendering language consistent, then vary content aggressively. If every style image shows a woman in a forest, the LoRA may learn “woman in a forest” as strongly as the style.
Review your set as a contact sheet. Repetition is easier to see in a grid than while opening files one by one.
Step 5: Prepare Images Without Destroying Them
Don't stretch images to a square. Use aspect-ratio bucketing when your trainer supports it; OneTrainer documents bucketing as a way to train mixed aspect ratios while resizing toward a similar pixel count.
Crop only to remove junk or improve concept visibility. Avoid cutting through important features such as the top of a hairstyle, a product handle, or garment trim. Upscaling a tiny source doesn't restore real detail, so exclude sources that are too small for the target training resolution.
Preserve normal color unless color itself is an artifact. Heavy denoising, face restoration, and aggressive sharpening can replace real texture with processed texture.
Step 6: Caption for Control
Captions tell the trainer how to divide visual information among words. Include the trigger token when your workflow needs one, then describe visible features you want to control at generation time.
For a character wearing a black jacket in one image, caption the jacket if you don't want it permanently bound to identity. For a style LoRA, caption the subject and scene so the style is the remaining shared signal.
Use one caption format consistently:
- Tag captions: Compact, easy to shuffle, and common for anime or booru-style datasets.
- Natural-language captions: Better for describing relationships, actions, and complex scenes.
- Hybrid captions: Useful only when your trainer and target model handle the structure you choose.
In sd-scripts, matching .txt sidecars are supported through caption_extension. Its dataset configuration also supports shuffle_caption and keep_tokens; keeping the first token fixed is useful when the trigger must remain first.
Step 7: Organize Files Predictably
Give each image a unique, plain filename. Pair it with a caption file using the same stem:
subject_001.png and subject_001.txt
subject_002.jpg and subject_002.txt
Some workflows use metadata instead. Hugging Face Diffusers' local ImageFolder path expects metadata.jsonl to map files to captions. Follow the format required by your trainer instead of mixing conventions.
Keep originals outside the training folder. Your training directory should contain only the final images, captions, and files the selected trainer expects.
Step 8: Choose Repeats After Curation
Repeats control exposure; they don't repair poor coverage. Doubling repeats on 12 similar portraits makes the model see those correlations more often.
Judge training by total exposure and checkpoint behavior, not image count alone. Smaller datasets often need more repeats per epoch, but the correct stopping point depends on the model, optimizer, rank, learning rate, and concept. Hugging Face warns that personalization training can overfit quickly, so save intermediate checkpoints and compare them.
Step 9: Build a Validation Prompt Set
Use fixed prompts and seeds to compare checkpoints fairly. Test:
- A neutral prompt that checks basic concept recognition.
- A prompt with an unseen background.
- A prompt with different clothing, color, or material.
- A difficult angle or composition.
- A prompt without the trigger, to check unwanted base-model drift.
Look beyond likeness. Check prompt obedience, composition range, background leakage, anatomy, and whether increasing LoRA weight makes training-image details take over.
Common Dataset Failures
- Same outfit everywhere: The garment becomes part of identity.
- Portrait-only coverage: Full-body generations become weak or generic.
- Uncaptioned backgrounds: Repeated rooms or scenery leak into outputs.
- Too many synthetic images: Existing model errors and one visual bias get amplified.
- Mixed identity quality: The adapter learns an average instead of a stable subject.
- No checkpoint tests: You keep training past the useful point and call the final file a failure.
A Practical Preflight Checklist
Before training, confirm:
- Every image supports the same written goal.
- Near-duplicates are gone.
- Important angles and framing ranges are represented.
- Repeated incidental details are captioned or removed.
- Image-caption filename pairs match exactly.
- Trigger spelling and capitalization are consistent.
- No caption is blank by accident.
- You have fixed validation prompts and a checkpoint schedule.
If you want a guided local workflow after the dataset is clean, LoRA Studio from LocalForge AI can handle setup without turning dataset decisions into a black box. OneTrainer, sd-scripts, and Diffusers remain solid choices when you want direct control over every setting.
Bottom Line
Build the smallest dataset that covers the concept honestly, then add images only when they contribute new evidence. Consistency teaches identity; variation protects control.
Don't train to discover obvious dataset mistakes. Review the contact sheet, captions, pairs, and validation prompts first. Those checks cost minutes and can save an entire run.