How to Fix LoRA Captions That Are Not Working

Fix Sidecar Pairing

For sidecar workflows, the image and caption need the same stem:

subject_007.png

subject_007.txt

Common failures:

• subject_007.png.txt;
• subject-007.txt with a different separator;
• case mismatch on a case-sensitive platform;
• captions use .txt while config expects .caption;
• files are one directory above the scanned image folder;
• image names changed after captions were generated;
• captions exist for rejected images while included images are missing them.

sd-scripts documents caption_extension in dataset configuration, including more-specific scopes that can override broader settings. Inspect the exact subset, not only the top section.

List image stems and caption stems, then compare the sets. Count nonblank files separately.

Fix Metadata Mapping

Hugging Face Datasets ImageFolder documents a metadata.jsonl pattern with one JSON object per line and a file_name key linking the image. Caption text is stored in another field, commonly text.

Check:

• the file is named exactly as the loader expects;
• each line parses as a JSON object;
• it is JSON Lines, not one array;
• file_name is relative to the correct directory;
• filenames match case and extension;
• the training script selects the real text column;
• no duplicate filename rows exist;
• no image lacks a row.

A dataset can load images successfully while selecting the wrong caption column. Inspect column names printed by the script or dataset object.

Check Empty and Invisible Text

A file can exist but contain no usable caption.

Look for:

• zero-byte files;
• whitespace-only files;
• a blank first line;
• unsupported encoding;
• replacement characters;
• invisible control characters;
• captions stored on a line the loader does not read.

Use plain UTF-8 and one logical caption per sidecar unless documentation says otherwise. Remove accidental byte-order or control characters if the loader exposes them as prompt text.

Layer 2: Inspect Caption Processing

The stored caption may not be the caption used for a step. Trainers can apply prefixes, suffixes, shuffling, fixed-item rules, complete-caption dropout, tag dropout, wildcard expansion, or cached text encoding.

Record every enabled caption option. Temporarily disable transformations and confirm the raw caption appears correctly. Then re-enable one feature at a time.

Do not toggle many options and retrain immediately. You need to know which processor changed the text.

Fix Broken Sentence Shuffling

shuffle_caption in sd-scripts is intended for separated caption items, commonly comma-delimited tags.

Tag caption:

ohwxperson, red coat, sitting, indoors

Sentence:

A photo of ohwxperson wearing a red coat while sitting indoors.

Shuffling the tag items can be reasonable. Shuffling sentence fragments can damage grammar and relationships.

If natural-language captions are scrambled, disable caption shuffling. If a hybrid format is used, define separators and fixed sections with options supported by the exact script version.

Fix Trigger Movement or Dropout

A trigger may be present in files but absent from some processed steps.

Causes include:

• tag shuffling moves it away from the intended prefix;
• complete-caption dropout removes the caption;
• tag dropout removes the trigger item;
• prefix logic duplicates or malformed it;
• the fixed-item count preserves too few items;
• a keep-token separator is misused.

sd-scripts documents keep_tokens for preserving leading separated items during shuffling. In:

ohwxperson, person, red coat

preserving one item keeps ohwxperson, not both trigger and class.

For diagnosis, disable dropout and shuffling. Train a short controlled run with one exact trigger form. Add processing only after the baseline works.

Fix Prefix and Suffix Syntax

OneTrainer's caption-tool documentation notes that prefix and postfix text are added directly. Missing punctuation or spaces can produce:

zvpackbackpack, open

instead of:

zvpack backpack, open

Other failures include doubled triggers, doubled commas, and suffixes attached to the final word.

Inspect a generated file and the training preview. Decide whether the trigger is physically stored or dynamically added, then use one source of truth.

Clear Stale Caption and Text Caches

Some workflows cache metadata, latents, or text-encoder outputs. Latent caches represent images and usually are not the caption source. Text-encoder output caches represent processed text and can make later caption edits ineffective.

If you changed:

• trigger spelling;
• caption text;
• shuffling;
• dropout;
• prefix or suffix;
• tokenizer or text encoder;
• caption column;

rebuild any caption-derived or text-encoder cache according to the trainer's documentation.

Current sd-scripts documentation describes text-encoder output caching with restrictions on caption shuffling and dropout in relevant workflows. Do not expect dynamic caption augmentation after static text outputs have already been cached.

Preserve logs showing which cache was reused or regenerated.

Check Truncation and Caption Length

Text encoders have finite sequence lengths. Long captions full of quality tags, repeated synonyms, camera jargon, and irrelevant detail can push important text beyond the usable range.

Put the trigger and critical class phrase early. Remove low-value filler. Inspect tokenized length with the exact tokenizer where possible.

Do not confuse word count with token count. Punctuation, uncommon strings, and model-specific tokenization change the number of pieces.

Symptoms of truncation include:

• late attributes never affect learning;
• moving a term earlier changes behavior;
• logs warn about truncated input;
• long automatic captions perform worse than concise edited ones.

Shorten captions based on training intent, not an arbitrary character limit.

Layer 3: Verify Caption Meaning

If files load and process correctly, inspect the labels.

False captions

A tagger says blue eyes on brown-eyed images or identifies the wrong garment. Remove false labels first.

Inconsistent vocabulary

The same fact appears as side view, profile, and from the side. Normalize terms unless the base-model language justifies deliberate variation.

Missing variable details

Clothing, background, view, or pose changes but captions do not name them. Those correlations can leak into the trigger.

Overcaptioned concept

Every fixed identity feature is described with ordinary words, spreading the concept across generic labels rather than the trigger.

Underdescribed style content

Style images show repeated castles and sunsets, but captions say only the style trigger. The adapter may bind those subjects to the style.

Run a per-image truth pass and a folder-wide correlation pass.

Layer 4: Separate Caption Failure From Data Correlation

Captions label evidence; they do not erase the evidence.

If every person image uses the same red coat, adding red coat helps identify the correlation but does not show the model that the person can wear something else. If every product image uses a white studio background, a caption cannot manufacture background diversity.

Fix the dataset when:

• one outfit dominates;
• views are missing;
• backgrounds barely vary;
• one crop appears in most images;
• watermarks or defects repeat;
• duplicates multiply a correlation.

Add real visual variation, remove duplicates, and rebalance the set. Captions and image diversity work together.

Layer 5: Check Training Strength

Correct captions can appear ineffective when the adapter underfits or overfits.

Underfitting signs:

• concept barely appears at reasonable LoRA weights;
• trigger has weak effect;
• later checkpoints improve steadily;
• training examples are not reproduced even broadly.

Overfitting signs:

• trigger forces training compositions;
• prompts cannot change clothes or background;
• concept appears too strongly without trigger;
• earlier checkpoints are more flexible;
• high adapter weight overwhelms prompt text.

Compare intermediate checkpoints with fixed prompts. Do not rewrite captions solely because the final checkpoint was trained too long.

Layer 6: Check Inference

The training captions can be fine while the test setup is wrong.

Verify:

• the LoRA matches the base-model family;
• the adapter is actually loaded;
• the trigger uses exact spelling;
• LoRA weight is reasonable;
• prompt syntax is correct for the UI;
• another LoRA, embedding, or style preset is not competing;
• negative prompts do not suppress the concept;
• the sampler and seed are held constant during comparisons.

Test:

A. base model, no LoRA, no trigger;

B. LoRA loaded, no trigger;

C. LoRA loaded, trigger;

D. LoRA loaded, trigger plus class;

E. the same setup at several reasonable adapter weights.

If A and B are identical, confirm loading. If B already shows the concept strongly, the adapter may have a global effect. If C changes nothing but D works, document the class phrase.

Symptom-to-Cause Map

Trainer says every caption is missing

Check extension, stem pairing, folder scope, and metadata source.

Some captions load and others do not

Compare case, image extensions, subdirectories, blank files, and renamed stems.

Edited captions have no effect

Rebuild text caches, confirm the correct dataset copy is used, and inspect whether a metadata column overrides sidecars.

Trigger disappears during training

Disable dropout and shuffling, inspect fixed-item count, and search dynamic prefix settings.

Captions look scrambled

Disable tag shuffling for sentences and validate separators.

Trigger always produces one outfit

Audit outfit frequency, missing clothing labels, duplicate images, training duration, and adapter weight.

Prompt attributes are ignored

Check overtraining, dataset imbalance, caption omissions, adapter weight, prompt conflict, and truncation.

Automatic captions make results worse

Remove false tags, filter unwanted categories, normalize vocabulary, and compare with a manually captioned subset.

Minimal Reproduction

Build a copy with four to eight varied images. Give each a short, visibly correct caption and one exact trigger. Disable shuffling, dropout, wildcard processing, and text-output caching.

Confirm the dataset preview, then run a short training test with saved checkpoints. Use fixed validation prompts.

If the minimal set loads correctly, add the full dataset while preserving the simple processing configuration. If failure returns, compare file pairing and captions. If loading works but behavior changes, inspect correlations and training balance.

A minimal reproduction is faster than guessing across hundreds of files.

Final Fix Checklist

Before retraining:

1. expected image and caption counts match;
2. every caption is nonblank and correctly paired;
3. metadata parses and the right text column is selected;
4. exact trigger count is correct;
5. processed caption preview looks intentional;
6. sentences are not shuffled;
7. fixed tag items and dropout are controlled;
8. text caches reflect current captions;
9. important text is not truncated;
10. false labels and synonym drift are corrected;
11. dataset correlations were audited;
12. inference uses the correct base, adapter, trigger, and weight.

LoRA Studio can help organize local caption setup. OneTrainer, sd-scripts, and Diffusers workflows remain useful because they expose different evidence about files, processing, and dataset mappings.

Bottom Line

Diagnose captions in order: loaded, processed, learned, then prompted. File and cache problems require configuration fixes. Entanglement requires caption or dataset fixes. Underfitting and overfitting require training changes.

Prove the failed layer with counts, logs, previews, and fixed-seed tests before retraining. That prevents a correct caption set from being rewritten to solve the wrong problem.