Automatic Captioning for LoRA Training
Automatic captioning can turn an image folder into a first-pass LoRA dataset quickly, but speed is the only part it guarantees. A caption model does not know which visual concept your LoRA should own, which details you want to control later, or whether a repeated object is intentional. It can confidently name the wrong color, miss a small accessory, invent a relationship, or apply several words to the same garment. Those errors become training labels if you do not remove them.
This guide focuses on running automation as a controlled drafting stage. It explains when to choose sentence captioning or tag prediction, how confidence thresholds change WD tagger output, where to place a trigger, and how to review generated files in passes that catch both factual and dataset-wide errors. It is not a general captioning theory page or a manual file-by-file tutorial. The goal is a repeatable automatic pipeline that saves labor without handing editorial judgment to a vision model.
Quick Answer
Choose an automatic captioner that matches the caption style expected by your base model. BLIP-style models generate sentences. WD taggers predict comma-separated tags and are especially aligned with Danbooru-style vocabularies. Generate drafts into a working copy, never over your only captions.
Add the exact trigger with a controlled prefix or a separate batch edit. Then review every image-caption pair. Remove false claims first, normalize synonyms second, add important missing variable details third, and run a folder-wide consistency check last. Automation is useful when review takes less time than writing every caption from zero.
What Automatic Captioning Actually Does
An automatic captioner maps pixels to likely text based on its training data. It does not discover your training objective. Two tools can inspect the same portrait and produce very different drafts:
a woman wearing a red jacket standing outside
1girl, red jacket, outdoors, looking at viewer, upper body
The first is a generated sentence. The second is a set of predicted labels. Both may be factually reasonable, but neither knows whether the person, jacket, photographic treatment, or location is the LoRA concept.
Before generating anything, write one sentence describing what the adapter should learn. Then list the variable details that prompts should retain. That policy becomes the review standard. Without it, automatic captions tend to become either noisy inventories or generic sentences that fail to separate repeated correlations.
Choose Sentence Generation or Tag Prediction
Use a BLIP-style captioner for factual sentences
BLIP is a vision-language framework with an image captioning model that uses a vision encoder and text decoder. Hugging Face documents conditional and unconditional caption generation. OneTrainer includes BLIP and BLIP2 options in its caption tooling.
Sentence output is useful when:
- the target model responds well to natural language;
- relationships matter, such as who holds an object;
- the dataset contains products or scenes outside a booru vocabulary;
- you need readable drafts for human editors.
A sentence generator often omits framing, subtle accessories, exact product parts, or uncommon visual terms. It may also guess gender, age, identity, or actions. Keep sentences factual and compact during review.
Use a WD tagger for comma-separated labels
WD taggers produce ratings, character labels, and general tags from a Danbooru-derived vocabulary. The WD ViT Tagger v3 model card states that it was trained on Danbooru images and supports those tag groups.
Tag output is useful when:
- the base model is accustomed to booru-like tags;
- the dataset is anime, illustration, or character focused;
- you want separable items that a trainer can shuffle;
- you need searchable vocabulary across many files.
A WD tagger's vocabulary and training distribution matter. It can perform poorly on specialist products, unusual materials, real-world brand details, or relationships that require sentence structure. Character tags can also be actively harmful when training an original identity, so exclude categories you do not want.
Understand Confidence Thresholds
A tagger predicts scores, then emits tags above a threshold. Lowering the threshold increases recall and usually adds more weak or wrong tags. Raising it increases precision but can omit real details.
Do not treat one published validation threshold as the correct production setting for every dataset. The WD ViT Tagger v3 model card reports validation statistics for its own data, not a universal LoRA caption recipe.
Calibrate on a small sample:
- Choose ten to twenty varied images.
- Run the tagger at the proposed threshold.
- Count useful tags, false tags, and important omissions.
- Adjust the threshold once.
- Recheck the same sample before processing the folder.
If each image receives dozens of low-value labels, raise the threshold or filter categories. If obvious clothing, view, and pose labels disappear, lower it cautiously. Threshold tuning reduces review burden, but it never replaces review.
Protect the Source Dataset
Work from a copy or commit a clean dataset state before batch generation. Confirm how the tool handles existing sidecars:
- overwrite replaces current captions;
- append can duplicate triggers and stale tags;
- prepend can change the intended fixed token count;
- skip preserves old files and may leave an incomplete mixed set.
Test the behavior on two disposable images. Open the resulting files and inspect encoding, extension, separators, and filename stems.
OneTrainer's caption interface supports automatic generation and prefix or postfix text. Its documentation warns that prefix and postfix values are inserted directly, so punctuation and spacing must be included deliberately. A prefix of zvpack, is different from zvpack.
Add the Trigger Deliberately
Caption models cannot choose a safe trigger for your concept. Select it before generation and insert it consistently.
For tags:
zvpack backpack, open, top view, on table
For natural language:
A top-view photo of zvpack backpack open on a table.
Use a tool prefix only when it creates the exact syntax you intend. Otherwise, generate the visual draft first and apply a controlled batch edit. Search the finished folder and confirm the trigger count matches the intended image-caption count.
Do not ask an image captioner to infer the trigger from filenames. The trigger is an experimental label, not a visible property.
Configure Output for the Trainer
Common sd-scripts workflows use a caption sidecar with the same stem as the image. The current dataset configuration supports caption_extension, so .txt and .caption are choices rather than assumptions.
Example:
product_014.webp
product_014.txt
Hugging Face Datasets ImageFolder uses a different mapping pattern. Its documentation shows a metadata.jsonl file with one JSON object per line, a file_name field, and a text field for the caption. A folder of correct .txt sidecars will not automatically satisfy every Diffusers script.
Verify:
- expected extension;
- exact image-caption stem pairing;
- UTF-8 text;
- whether subdirectories are included;
- whether existing captions are overwritten;
- the caption or text column selected by the script.
Format mistakes can look like weak training because the trainer may fall back to class text or empty captions.
Review Pass 1: Remove False Claims
Open each image beside its caption. Delete anything that is not visibly true.
High-risk errors include:
- wrong object, clothing, hair, or eye color;
- wrong subject count;
- invented text, logo, or brand;
- incorrect character identity;
- guessed age or relationship;
- accessories copied from a likely visual association;
- labels for watermarks, borders, or defects you will crop out.
False labels are worse than missing labels. A missing descriptor leaves a correlation unexplained. A false descriptor explicitly pairs the image with text that does not match.
Review Pass 2: Remove Noise
Taggers can output ratings, character names, quality labels, and long-tail descriptors that do not serve the training goal. Sentence models can add filler such as generic praise or uncertain scene interpretations.
Remove:
- rating tags unless the workflow has a specific reason for them;
- named characters that are not the target;
- quality or score tags added mechanically;
- redundant synonyms;
- invisible details;
- words that describe the target concept when the trigger should carry it.
Do not delete every fixed feature by formula. Decide based on what the LoRA should own and what should remain promptable.
Review Pass 3: Add Missing Control Details
Automatic output often misses the details that matter most for disentanglement. Add relevant variable facts:
- changing clothes or accessories;
- pose, expression, and gaze;
- front, side, rear, or top view;
- close-up, medium shot, or full body;
- background and lighting when they vary;
- open, closed, folded, worn, or in-use product state;
- other people or objects.
For a style LoRA, describe image content thoroughly enough that the shared rendering treatment remains the unexplained signal. For a product LoRA, name orientation and state because those are common sources of unwanted entanglement.
Review Pass 4: Normalize the Folder
A caption can be true alone and still be inconsistent with the set. Search all captions for competing words:
side viewversusprofile;close upversusclose-up;hoodieversushooded sweatshirt;- trigger capitalization variants;
- singular and plural class words.
Choose one accurate term for one recurring fact. Use batch replacement only after checking that each replacement remains true. coat and jacket are not interchangeable in every image.
Use Shuffling Only for Separable Tags
sd-scripts documents shuffle_caption for comma-separated caption items and keep_tokens for preserving leading items. If one trigger item must stay first, keep_tokens = 1 is a common configuration.
Count comma-separated items, not visual concepts. In:
zvpack, backpack, open, top view
preserving one item keeps only zvpack. If zvpack backpack must remain together, make it one item or configure the exact preservation behavior supported by the trainer.
Do not shuffle ordinary sentences. Random clause order damages grammar and relationships.
Decide Whether Automation Saved Time
Measure a sample instead of assuming.
Record:
- generation time;
- average review time per image;
- false tags per caption;
- missing important details per caption;
- percentage needing a near-total rewrite.
Automation is a good fit when drafts are mostly accurate and normalization is repetitive. Manual-first work can be faster for ten specialist product images, a dataset with subtle identity distinctions, or any set where the captioner repeatedly misunderstands the subject.
A useful hybrid is automatic generation followed by manual review in contact-sheet order. Another is manual captions for a representative subset, then automation for the rest using the manual vocabulary as the editing standard.
Final Automatic Caption QA
Before training, verify:
- Every intended image has one readable caption.
- No rejected image or orphan caption remains.
- Every required trigger uses exact spelling.
- No caption contains a known false statement.
- Recurring visual facts use consistent terms.
- Variable details are not systematically omitted.
- Caption format matches the trainer and base model.
- Shuffling and fixed-prefix settings match the syntax.
- A backup exists before any final batch edit.
LoRA Studio can guide a local caption workflow, while OneTrainer provides integrated automatic caption tools and sd-scripts exposes detailed caption processing controls. Use whichever path lets you inspect the actual files.
Bottom Line
Automatic captioning is a drafting accelerator. Pick sentence generation or tag prediction based on the base model and dataset, calibrate on a small sample, and protect the source files.
The quality gate is still human: remove false labels, normalize vocabulary, add missing control details, and verify format. If that review costs more than writing accurate captions manually, automation is the wrong tool for that dataset.
What to Do Next
Test one captioner on a representative image sample.
Follow understand lora captioning principles for the detailed workflow and checks.
Calibrate output and add the exact trigger.
Follow compare manual and automatic captioning for the detailed workflow and checks.
Review truth, control details, and folder consistency.
Follow choose the correct caption format for the detailed workflow and checks.