How to Caption a LoRA Dataset
Captioning a LoRA dataset is easier when you stop trying to describe everything. The job is to label visible details that should remain promptable while leaving the target concept clearly assigned to the LoRA or its trigger token. That means accurate, consistent captions—not long ones.
This walkthrough takes you from an image-only folder to a checked caption set ready for common local trainers. You'll define the concept, choose a trigger and class word, select tags or natural language, create draft captions, normalize vocabulary, pair filenames, configure shuffling correctly, and run a final audit. It also includes examples for a person, character, product, garment, and style so you can see how the same rules change with the training goal. The result is a dataset you can inspect and debug.
Quick Answer
For each image, put the trigger token first if your workflow uses one. Add a class word when useful, then describe variable visible details: clothing, pose, expression, framing, angle, background, lighting, and other objects.
Save the caption using the image's filename stem when your trainer uses sidecars. Review automatic captions manually, use one vocabulary consistently, and keep required trigger text fixed if you shuffle tags.
Before You Start
Finish image curation first. Captioning duplicate, defective, or rejected files wastes time and creates pairing errors later.
Make a working copy of the final image folder. Decide which trainer you'll use because dataset formats differ. sd-scripts commonly reads matching text sidecars, OneTrainer can load prompts from sidecars and other sources, and Hugging Face Diffusers documents local ImageFolder caption mapping through metadata.jsonl.
Step 1: Write the Training Goal
Use one sentence:
Teach the identity of this person while keeping clothing, pose, framing, and background controllable.
Or:
Teach this watercolor-and-ink style across many subjects and compositions.
Highlight the concept the LoRA should own. Everything else becomes a caption candidate.
Step 2: Make an Attribute Policy
Create three lists before writing captions.
Always identify
- trigger token, when required;
- class word, when part of the plan;
- dataset-specific fields your workflow expects.
Caption when visible
- clothing and accessories;
- pose, action, expression, and gaze;
- crop, view, and camera angle;
- background, setting, and lighting;
- other subjects or objects;
- colors that vary.
Usually leave to the concept
- fixed identity features;
- fixed product construction;
- the target style itself;
- defining character details that never change.
This policy prevents you from changing strategy halfway through the folder.
Step 3: Choose the Trigger and Class Word
Pick a stable, low-conflict trigger such as ohwxperson or zvpack. Avoid real names, common descriptive words, artist names, and existing brands.
Add a broad class when it helps the model place the concept:
ohwxperson woman
zvpack backpack
nvycharacter character
Keep spelling and capitalization identical. Don't alternate between ZVPack, zvpack, and zv_pack.
Step 4: Choose One Caption Format
Use comma-separated tags when your base model and workflow are tag-oriented:
ohwxperson, woman, red sweater, sitting, indoors, medium shot
Use factual sentences when the model benefits from natural language:
A medium photo of ohwxperson wearing a red sweater and sitting indoors.
Don't shuffle sentence fragments. Don't mix dense booru tags, prose, and filename fragments randomly across one dataset.
Step 5: Build a Small Vocabulary
Standardize common terms before captioning:
close-up,medium shot,full body;front view,three-quarter view,side view,rear view;looking at viewer,looking away;indoors,outdoors, or more specific locations;- consistent garment and product-part names.
A short controlled vocabulary makes the set easier to search and correct.
Step 6: Draft Captions Per Image
Inspect one image at a time. Write only what is visible and relevant.
Use this order for tags:
- trigger and class;
- subject count or other subjects;
- variable clothing or state;
- pose, action, and expression;
- framing and view;
- setting and lighting.
Order isn't sacred, but a template reduces omissions.
Person example
Image: a close portrait in a blue hoodie, smiling, outside.
Caption:
ohwxperson, woman, blue hoodie, smiling, close-up, outdoors
Don't list fixed facial geometry if identity is the learned concept.
Character example
Image: the character in alternate armor, viewed from the side.
Caption:
nvycharacter, character, silver armor, standing, side view, stone hallway
Caption alternate clothing. Leave defining identity features to the character concept unless they vary.
Product example
Image: an open backpack viewed from above on a desk.
Caption:
zvpack backpack, open, top view, on a desk, indoor lighting
Garment example
Image: a green jacket worn by a seated man.
Caption:
qjacket garment, green jacket, man, seated, three-quarter view, studio background
Here the garment is the concept, so captions should separate wearer, pose, view, and background.
Style example
Image: a fox crossing snow beneath dark trees.
Caption:
a red fox walking through snow beneath dark pine trees, wide composition
Describe content well so repeated rendering treatment remains the style signal.
Step 7: Use Auto-Captioning Without Trusting It
Run an image captioner or tagger to create drafts when the dataset is large. OneTrainer includes caption tooling, and other local utilities can generate sidecars.
Review every draft for:
- wrong colors, counts, clothing, or objects;
- guessed demographic or identity claims;
- missed product details and accessories;
- inconsistent synonyms;
- irrelevant score or quality tags;
- text describing watermarks or defects;
- missing trigger and class prefix.
Delete false tags before adding more detail. A short accurate caption is better than a comprehensive wrong one.
Step 8: Save Captions in the Required Format
For sidecar workflows, filenames must share an exact stem:
dataset_001.png
dataset_001.txt
Use plain UTF-8 text. Avoid accidental .txt.txt extensions on Windows by showing file extensions in Explorer.
For Diffusers ImageFolder, follow the current metadata.jsonl mapping documented by Hugging Face instead of assuming sidecars will be discovered. The training script expects the configured caption column and dataset structure.
Step 9: Configure Shuffling Correctly
For comma-separated tags in sd-scripts, shuffle_caption = true can reduce fixed-order dependence. Set keep_tokens to preserve leading caption items when needed. With one trigger item first, keep_tokens = 1 is a common setup.
Check how your caption is split. If ohwxperson, woman is meant to remain a combined prefix, preserving only one comma-separated item won't preserve both.
Don't enable shuffling for normal prose. A shuffled sentence stops being a reliable sentence.
Step 10: Decide Whether to Use Caption Dropout
Start without clever dropout settings. Build a baseline and observe the output.
Dropout can reduce dependence on individual tags, but it can also cause dropped visual details to become associated with the concept. If you use it, preserve required trigger items and compare checkpoints with fixed prompts.
Trainer names and behavior differ, so use its current documentation rather than copying a screenshot from an old tutorial.
Step 11: Run a Mechanical QA Pass
Check:
- one expected caption per image;
- no caption without an image;
- no blank caption files;
- no duplicate image stems;
- trigger appears with exact spelling;
- file encoding loads correctly;
- caption extension matches trainer configuration.
Sort both image and caption lists by name. Pairing gaps become obvious.
Step 12: Run a Meaning QA Pass
Read captions beside a contact sheet.
Ask:
- Are repeated outfits always named?
- Are backgrounds missing from one part of the set?
- Did the tagger assign two words to the same garment?
- Is a fixed identity feature over-described?
- Are style images described broadly enough to separate content from style?
- Does every caption tell the truth?
Search and replace can normalize vocabulary, but inspect replacements. Replacing coat with jacket blindly can make correct captions wrong.
Step 13: Test Caption Strategy With Checkpoints
Captions are hypotheses. Validate them.
Use prompts that change:
- clothing;
- background;
- pose and view;
- color;
- subject matter for a style LoRA.
If an uncaptioned detail keeps leaking into outputs, correct the dataset and retrain. If a fixed feature becomes hard to reproduce, you may have overcaptioned the concept or trained too little.
For a guided local path, LoRA Studio from LocalForge AI can help you move from paired captions into training. OneTrainer, sd-scripts, and Diffusers remain the better choice when you want to configure each caption-processing option directly.
Bottom Line
Define responsibility first, caption second. Put variable visual facts in plain, consistent text and let the target concept carry what is fixed.
Finish with both mechanical and meaning checks. A perfect filename pair with a false caption is still bad training data.