Kohya LoRA Settings Explained

Network Rank and Alpha

network_dim, often labeled rank or dimension, controls LoRA capacity. A higher rank creates more trainable parameters and a larger adapter. It may help a complex concept, but it can also increase memory use, file size, and the ability to memorize a narrow dataset.

network_alpha scales the LoRA update. Alpha must be interpreted relative to rank, not as an independent quality slider. Current sd-scripts guidance describes values around half the rank or equal to the rank as common configurations, while the basic training document notes that alpha affects learning-rate scaling.

If you change rank while leaving alpha fixed, you have changed capacity and effective scaling. If you want to isolate capacity, keep the alpha-to-rank relationship deliberate and document it. Do not compare rank 16 with alpha 16 against rank 64 with alpha 1 and conclude that rank alone caused the result.

Convolution dimensions and alphas add another capacity path when the selected network supports them. Treat conv_dim and conv_alpha as part of the same experiment record. They are not free detail switches.

Trained Modules and Separate Learning Rates

A LoRA run can target the denoiser, text encoder, or both, depending on architecture and script support. The choice changes memory, token association, overfitting risk, and which learning-rate fields matter.

For general Stable Diffusion training, sd-scripts exposes an overall learning rate plus separate U-Net and text encoder rates. SDXL has separate rates for its two text encoders. The current documentation recommends text encoder rates lower than the U-Net rate as a baseline.

Check precedence:

• Overall learning rate: Supplies the default when a module-specific value is absent.
• U-Net or denoiser rate: Overrides the overall rate for that adapter group.
• Text encoder rate or rates: Override the overall rate for text encoder adapter groups.
• Train U-Net only: Removes text encoder adapters from optimization.
• Train text encoder only: Removes denoiser adapters from optimization.

More module training is not automatically better. If the trigger association is already strong, text encoder updates can reduce prompt flexibility. Establish a denoiser-only baseline before adding another trained component unless the architecture documentation or a controlled test supports the change.

Optimizer, Learning Rate, and Scheduler

These three fields form one decision. AdamW, AdamW8bit, Adafactor, Prodigy, and D-Adaptation do not interpret learning-rate setups identically.

For AdamW variants, the configured learning rate directly sets the update scale before scheduler effects. AdamW8bit reduces optimizer-state memory through bitsandbytes, but that does not make a rate copied from another model family safe.

Current sd-scripts advanced documentation recommends a learning rate around 1.0 for Prodigy and D-Adaptation because they adapt an internal scale. It also says to use a constant scheduler rather than imposing a conventional decay schedule. The current optimizer implementation warns when their configured rate is 0.1 or lower and warns that, when multiple module learning rates are supplied, only the first takes effect.

Adafactor can use relative-step behavior and the matching adafactor scheduler, or it can be configured for an explicit external rate. Those are different modes. Copying an Adafactor name without its optimizer arguments and scheduler is not a reproducible choice.

Use the dedicated LoRA optimizer comparison before changing this group. Never change optimizer while claiming to test learning rate alone.

Batch Size, Accumulation, Epochs, and Steps

Batch size is the number of examples processed in one forward and backward pass per process. Gradient accumulation delays the optimizer update while gradients are gathered across multiple passes. Together they determine effective batch behavior, but the exact distributed formula also depends on process count.

Larger accumulation can fit a larger effective batch without holding all examples in VRAM at once. It costs more time per optimizer update and can change how many updates occur within an epoch if other settings are adjusted. The official advanced guide also states that experimental fused backward pass cannot be combined with gradient accumulation above one.

Repeats, epochs, batch size, accumulation, number of processes, and dataset size determine exposure and optimizer steps. Review the trainer's resolved step count rather than reasoning from epochs alone. If you double repeats and halve epochs, verify the resulting update count before calling the runs equivalent.

Use either maximum epochs or maximum steps as the controlling budget and confirm precedence in the current help output. Current sd-scripts documentation says epoch specification takes precedence when both are provided.

Precision, Attention, and Gradient Checkpointing

Mixed precision controls numeric representation during training. fp16 and bf16 can reduce memory and improve speed, but hardware support and model stability matter. Full half-precision options can save more memory while increasing stability risk, so they should not be enabled only because a preset includes them.

Attention implementations such as PyTorch scaled dot-product attention affect speed and memory, not the conceptual learning target. Gradient checkpointing saves activation memory by recomputing values during backward passes. It commonly slows training. Neither setting repairs an oversized dataset resolution or an optimizer state that exceeds available VRAM.

Change these controls to solve measured resource constraints. Then rerun a short smoke test and compare throughput, peak memory, loss finiteness, and sample behavior.

Latent and Text Encoder Caching

Caching fields have some of the most important hidden interactions in Kohya.

Cache latents stores VAE outputs so the VAE does not run for every training example. Current sd-scripts documentation states that image augmentations such as color augmentation, flip augmentation, and random crop are disabled when latents are cached. Disk caching reduces repeated preprocessing at the cost of disk space and input/output activity.

Cache text encoder outputs stores encoded captions. Current documentation states that caption shuffle and caption dropout are disabled. It also requires U-Net-only network training because text encoder LoRA modules cannot be trained when their outputs are cached.

Therefore, a cache toggle can change the data seen during training or remove an entire trained component. Compare cached and uncached runs only after accounting for those changes. The LoRA captioning guide explains caption controls that may be affected.

Caption Shuffle, Keep Tokens, and Dropout

Caption shuffle changes tag order, which can reduce dependence on a fixed sequence in comma-separated tag captions. Keep-token settings protect a leading trigger or class phrase from being shuffled. Caption dropout removes some caption conditioning on selected steps or examples, depending on the selected control.

These fields interact with caption format. Shuffling a natural-language sentence can destroy grammar. Keeping too many tokens fixed can preserve unwanted correlations. Dropping the unique trigger too often can weaken the association you need.

Inspect captions after the trainer applies its rules. Do not infer behavior from source text files alone. If text encoder outputs are cached, verify which caption transformations remain available before assuming that shuffle or dropout is active.

Regularization, Noise, and Loss Controls

Network dropout, noise offset, multi-resolution noise, input perturbation, Min-SNR weighting, and alternate loss functions change different parts of training. They should not be bundled into a generic "quality" preset.

• Network dropout: Regularizes adapter modules and may suppress memorization, but can also weaken learning.
• Noise offset: Changes the noise distribution and can affect brightness or contrast behavior.
• Min-SNR weighting: Reweights loss across timesteps.
• Huber or smooth L1 loss: Changes sensitivity to outliers compared with the default L2 behavior.
• Weight norm scaling: Constrains adapter weight norms and may help control overfitting.

Add one only when you have a specific failure hypothesis and fixed validation prompts. More regularization is not automatically safer.

Saving, Sampling, and Logging Fields

Save cadence determines whether you can recover the best point before the final checkpoint. Sample cadence determines whether you can see the learning curve. Logging determines whether the resolved rate and optimizer behavior can be audited.

Use fixed sample prompts and seeds. Save often enough to compare early, middle, and late behavior without flooding the disk. When resuming, distinguish full state resume from loading LoRA weights. The advanced documentation says resume restores optimizer state and step count, while loading network weights alone does not.

At minimum, log the optimizer name, resolved module rates, scheduler, global step, loss finiteness, dataset size, and save timing. Use the trained LoRA testing guide to evaluate checkpoint grids outside the training interface.

A Controlled Change Order

When a run is weak or unstable, use this order:

1. Confirm architecture, script, base model, and adapter targets.
2. Confirm captions and dataset resolution are actually loaded.
3. Confirm optimizer, optimizer arguments, learning rates, and scheduler.
4. Calculate exposure and resolved optimizer steps.
5. Check caching constraints and disabled augmentations.
6. Review precision, loss finiteness, and memory behavior.
7. Compare fixed samples across saved checkpoints.
8. Change one justified field group and repeat the comparison.

The LoRA Studio overview describes the broader project workflow for keeping these artifacts together.

Bottom Line

Kohya settings are coupled controls. Rank must be read with alpha, optimizer with learning rate and scheduler, batch size with accumulation and steps, and caching with the features it disables. Architecture and trained modules come first because they determine which fields are valid.

Save the resolved configuration, release or commit, command, logs, and sample grid for every run. A preset is useful only when you can explain what it changes and reproduce the environment that interpreted it.