LoRA Optimizer Comparison for Kohya Training
A LoRA optimizer should be chosen with its learning-rate model, state memory, scheduler, trained modules, and available implementation, not from a leaderboard. AdamW is a clear baseline with familiar explicit learning rates. AdamW8bit and paged AdamW variants change optimizer-state storage or memory handling, but they do not remove the need to tune the rate. Adafactor can reduce state memory further and offers relative-step behavior, although that mode changes how the configured learning rate is interpreted. Prodigy and D-Adaptation estimate an internal scale and therefore use a very different configured-rate convention in current sd-scripts.
This comparison covers the optimizer families that current official Kohya sd-scripts documentation supports well enough to discuss responsibly: AdamW variants, Adafactor, Prodigy, and D-Adaptation. It does not declare one universal winner. Image quality still depends on the dataset, architecture, adapter targets, rank and alpha, effective batch, exposure, and stopping point. Use one known run as the control, preserve everything except the optimizer group, and compare checkpoints with fixed prompts and seeds.
The Short Decision
Use AdamW when you want the simplest auditable baseline and have enough memory for full optimizer states. Use AdamW8bit when optimizer-state memory is a meaningful constraint and bitsandbytes works reliably on the target system. Consider paged AdamW when the supported bitsandbytes implementation helps manage memory pressure, but do not treat paging as free VRAM.
Use Adafactor when optimizer-state memory is especially important or when you need an sd-scripts feature that currently supports only Adafactor, such as experimental fused backward pass. Decide explicitly between relative-step mode and a fixed external learning rate.
Consider Prodigy or a D-Adaptation variant when you want an optimizer that adapts its effective scale and you can follow its different learning-rate convention. Current sd-scripts guidance calls for a configured learning rate around 1.0, a constant scheduler, and no LoRA+ combination.
None of these choices repairs bad captions, incompatible architecture, poor data coverage, or excessive exposure. The Kohya LoRA training guide covers those upstream controls.
Compare the Whole Optimizer Group
The optimizer name is only one field. A fair comparison records:
- optimizer type and package version;
- optimizer arguments, including betas, weight decay, epsilon, or optimizer-specific controls;
- overall and module-specific learning rates;
- scheduler and warmup;
- gradient clipping;
- batch size, gradient accumulation, and process count;
- maximum steps or epochs;
- trained modules, rank, and alpha;
- mixed precision and memory controls;
- sd-scripts release or commit.
If you switch from AdamW8bit at a conventional explicit rate to Prodigy but leave that small rate unchanged, you are not testing Prodigy as documented. If you change optimizer, rate, scheduler, rank, and total steps together, the result cannot identify an optimizer effect.
Why Optimizer State Matters for LoRA
LoRA freezes most base-model weights and trains smaller adapter matrices. That greatly reduces trainable parameters, but training still holds the base model, activations, gradients, adapter weights, and optimizer state. The optimizer state is only one part of total memory, yet it can decide whether a high-rank adapter or additional text encoder modules fit.
Adam-style optimizers maintain moving statistics for trainable parameters. Full-precision state costs more memory than quantized or factored alternatives. AdamW8bit reduces state storage through bitsandbytes. Adafactor reduces state requirements for suitable parameter shapes by factoring second-moment estimates. Prodigy and D-Adaptation add their own adaptive state and behavior, so "automatic learning rate" should not be read as "no state" or "no configuration."
Measure peak allocated and reserved memory on the actual run. Architecture, rank, target modules, batch, resolution, checkpointing, and caching can outweigh optimizer savings.
AdamW: The Explicit Baseline
AdamW is the default optimizer in the current sd-scripts optimizer dispatcher when no optimizer type is specified. It uses PyTorch's AdamW implementation and applies decoupled weight decay. Its main advantage is legibility: the configured learning rate, scheduler, betas, epsilon, and weight decay define a familiar experiment.
Strengths:
- available without a bitsandbytes optimizer dependency;
- predictable explicit-rate workflow;
- useful as a baseline when comparing other optimizer families;
- supports separate parameter groups and module learning rates in ordinary configurations.
Tradeoffs:
- full optimizer states use more memory than 8-bit AdamW or factored Adafactor;
- explicit rates still require tuning for architecture, trained modules, rank and alpha, data, and batch geometry;
- a scheduler copied from another step budget can decay too early.
AdamW is not inherently higher quality than lower-memory variants. It is valuable because it gives you a clear control run.
AdamW8bit: Lower State Memory, Same Tuning Duty
AdamW8bit uses bitsandbytes in current sd-scripts. It stores optimizer state in a lower-bit representation to reduce memory. Official sd-scripts documentation describes it as common and memory efficient.
The important boundary is that AdamW8bit remains an AdamW-style explicit-rate optimizer. Do not set the learning rate to 1.0 merely because that convention appears in a Prodigy example. Start from an architecture-appropriate AdamW baseline and preserve the scheduler while testing the implementation change.
Strengths:
- lower optimizer-state memory than standard AdamW;
- familiar explicit learning-rate and scheduler workflow;
- often a practical baseline for local LoRA training.
Tradeoffs:
- requires a working bitsandbytes installation and supported environment;
- memory savings do not guarantee more speed;
- numeric and package behavior should be verified with a smoke test;
- a lower state footprint does not solve activation-memory pressure.
Compare a short AdamW and AdamW8bit run with the same seed, batches, rate, and saves. Confirm finite loss, similar early learning direction, and clean checkpoint reloads before committing to a longer run.
Paged AdamW Variants: Memory Pressure Management
The current sd-scripts optimizer implementation supports PagedAdamW, PagedAdamW8bit, and PagedAdamW32bit through bitsandbytes. Paged variants are designed to manage optimizer memory using bitsandbytes paging behavior when memory pressure occurs.
PagedAdamW8bit combines quantized state with paging. PagedAdamW32bit uses 32-bit state with paging. Their names describe state format and memory handling, not expected image quality.
Consider a paged variant when:
- the non-paged counterpart nearly fits but suffers memory spikes;
- bitsandbytes is already validated on the machine;
- you can measure whether paging changes throughput or stability.
Do not assume:
- system memory is as fast as GPU memory;
- paging makes an oversized configuration efficient;
- paged and non-paged runs are identical in wall-clock behavior;
- the optimizer can compensate for excessive resolution or batch size.
Lower the batch or adapter capacity when that is the cleaner solution. Paging is an implementation option, not a quality upgrade.
Adafactor: Factored State With Two Rate Modes
Adafactor is documented by sd-scripts as more memory efficient than AdamW8bit in relevant configurations and proven in SDXL full-model training. Its factored second-moment representation can substantially reduce state memory for matrix-shaped parameters. The exact saving depends on parameter shapes and what is trained.
The critical configuration choice is rate behavior. In current sd-scripts, Adafactor defaults relative_step to true when the argument is absent. When relative-step mode is active, the optimizer derives a changing step size and the configured learning rate is treated as an initial rate by the wrapper. The advanced guide recommends the matching adafactor scheduler for this mode.
Adafactor can also be configured with relative_step=False and an explicit external learning rate, but that is a distinct experiment. Record scale_parameter and warmup_init as well because copied examples may set them differently.
Strengths:
- low optimizer-state memory for suitable trainable matrices;
- relative-step mode can provide an internally scheduled rate;
- current sd-scripts experimental fused backward pass supports only Adafactor.
Tradeoffs:
- more configuration branches than AdamW;
- relative-step and fixed-rate runs are not directly comparable without documenting the actual schedule;
- experimental fused backward pass cannot be used with gradient accumulation above one in current sd-scripts;
- a memory win does not prove a quality win.
Log the resolved learning rate at every save. An Adafactor run labeled only with learning_rate=... is incomplete if the relative-step arguments are unknown.
Prodigy: Adaptive Scale With a Different Convention
Prodigy is available through the prodigyopt package in current sd-scripts. It adapts an internal estimate used to scale updates. This is why the configured learning rate uses a different convention from an AdamW baseline.
Current official sd-scripts guidance says:
- specify a learning rate around
1.0; - use a constant scheduler because an external decay schedule is not needed;
- do not combine Prodigy with LoRA+;
- when separate module rates are supplied, only the first rate takes effect in the current optimizer path.
The optimizer implementation warns when the configured rate is 0.1 or lower. It also logs the product of the internal d estimate and learning rate, which is more informative than reading the configured rate alone.
Strengths:
- reduces reliance on selecting a small conventional base rate;
- exposes an adaptive effective scale that can be logged;
- supported directly by the current optimizer dispatcher.
Tradeoffs:
- does not remove other hyperparameters or stopping decisions;
- separate U-Net and text encoder rates do not behave like an ordinary AdamW parameter-group setup;
- requires an additional package;
- incompatible with LoRA+ in current sd-scripts guidance;
- a constant configured rate does not mean constant effective updates.
Prodigy can be a reasonable controlled experiment, especially when explicit-rate tuning is the tested bottleneck. It is not evidence that dataset and exposure decisions no longer matter.
D-Adaptation: A Family, Not One Single Optimizer
Current sd-scripts supports multiple D-Adaptation variants, including Adam, AdaGrad, Adan, Lion, and SGD based forms. Selecting "D-Adaptation" without the exact optimizer type is therefore insufficient for reproduction.
Like Prodigy, these variants adapt an internal scale. Current sd-scripts applies the same main usage warnings:
- configure a learning rate around
1.0; - use a constant scheduler;
- do not combine with LoRA+;
- do not expect multiple module-specific rates to work like standard AdamW groups.
Strengths:
- adaptive-scale approach across several underlying optimizer families;
- useful for a controlled test when manual explicit-rate choice is uncertain;
- effective scale can be inspected through sd-scripts logging.
Tradeoffs:
- "D-Adaptation" alone is ambiguous;
- behavior and state differ by the selected variant;
- dependency and optimizer arguments must be preserved;
- automatic scale estimation does not choose rank, exposure, scheduler exceptions, or the best checkpoint.
Start with one documented variant, not a sweep of every name. Compare the resulting learning curve against a known AdamW or AdamW8bit control.
Learning-Rate Coupling by Family
The most damaging optimizer mistake is carrying one numeric rate across incompatible conventions.
AdamW, AdamW8bit, and paged AdamW: Use an explicit learning rate. The scheduler transforms that configured rate over the run. Separate module rates can be meaningful in supported parameter groups.
Adafactor with relative steps: The optimizer derives a schedule-like step size. Use the documented Adafactor scheduler path and log the resolved rate. A supplied rate may act as an initial value rather than a fixed rate.
Adafactor with relative steps disabled: Use an explicit learning rate and document the optimizer arguments that disabled internal relative-step behavior.
Prodigy and D-Adaptation: Use the current sd-scripts convention of a configured rate around 1.0 and a constant scheduler. Inspect the adaptive effective scale. Avoid assumptions about separate module rates.
The Kohya settings reference explains how these optimizer choices interact with rank, alpha, modules, caching, and batch geometry.
How to Run a Fair Optimizer Comparison
Use one dataset, base model, architecture script, adapter target, rank, alpha, batch configuration, total steps, precision, seed, and save cadence.
- Run a short control with AdamW or AdamW8bit and an explicit documented rate.
- Save early, middle, and late checkpoints.
- Change only the optimizer group: type, required arguments, rate convention, and scheduler.
- Match the same optimizer-step budget.
- Log peak memory, time per step, resolved learning rate, finite loss, and any internal adaptive scale.
- Test all checkpoints with fixed prompts, seeds, sampler, dimensions, and adapter weights.
- Compare concept fidelity, prompt control, variation, leakage, artifacts, and the point where overfitting begins.
Do not select the run with the lowest training loss automatically. Diffusion loss is noisy and does not fully measure visual usefulness. Use the guide to testing a trained LoRA for a repeatable checkpoint grid.
Memory Problems the Optimizer Cannot Fix
If changing optimizer saves little memory, inspect the larger allocations:
- base-model architecture and precision;
- training resolution and bucket sizes;
- batch size;
- denoiser and text encoder activations;
- adapter rank and targeted layers;
- gradient checkpointing;
- latent and text encoder caching;
- other GPU processes.
Optimizer choice affects trainable-state memory. It does not shrink every activation or base-model tensor. The LoRA out-of-memory guide helps separate those pressure sources.
Bottom Line
There is no universal best LoRA optimizer. AdamW is the clearest full-state baseline. AdamW8bit often reduces state memory while preserving an explicit-rate workflow. Paged variants can help manage memory pressure. Adafactor offers lower state memory and distinct relative-step or fixed-rate modes. Prodigy and D-Adaptation adapt an internal scale but require the documented rate and scheduler convention.
Choose based on the constraint you can measure, preserve the complete optimizer group, and compare fixed checkpoint grids. The planned LoRA Studio workflow can organize these run records, but it cannot replace controlled evidence.
What to Do Next
Review coupled Kohya settings
Check how optimizer choice interacts with rank, alpha, module rates, batches, caching, and total steps.
Build a checkpoint test grid
Compare optimizer runs with fixed prompts, seeds, inference settings, and adapter weights.
Diagnose memory pressure
Separate optimizer-state memory from activations, resolution, rank, caching, and other GPU allocations.