How to Fix LoRA Training Out of Memory

The Quick Fix

Apply these changes in order and restart after each major change:

1. Close image generators, browsers using GPU acceleration, games, and stale Python processes.
2. Set the training batch size to 1.
3. Enable supported mixed precision: bf16 on compatible hardware or fp16 where appropriate.
4. Enable gradient checkpointing.
5. Cache image latents, preferably to disk when system RAM is limited.
6. For supported SDXL workflows, cache text-encoder outputs and train the denoiser only.
7. Lower image resolution or maximum bucket size.
8. Use gradient accumulation if you wanted a larger effective batch.
9. Select a memory-efficient optimizer only after confirming trainer and platform support.
10. Reduce network rank last unless the adapter itself is unusually large.

Don't toggle everything at once. The error location may reveal a simpler fix, and several memory options disable augmentations or text-encoder training.

First: Identify When Memory Runs Out

Copy the complete console error and note the last successful action. “CUDA out of memory” is the result, not the diagnosis.

It crashes while loading the model

The base model, text encoders, VAE, or precision choice doesn't fit before training begins.

• Confirm you selected the correct model family and training script.
• Use a supported lower-precision model load.
• Remove duplicate models or inference pipelines loaded by notebooks.
• Restart the Python process to release retained allocations.
• Avoid loading an inference UI and trainer on the same GPU.

If a Flux or other large transformer pipeline can't finish loading, batch-size changes won't help because no batch exists yet.

It crashes while caching latents

The VAE is encoding training images, often in groups. Lower the VAE or cache batch size if the trainer exposes it, cache to disk, reduce maximum image dimensions, or disable in-memory caching.

Caching latents is intended to reduce repeated VAE work during training, but the caching pass has its own peak. A setting that saves VRAM later can still crash while building the cache.

It crashes on the first training step

This points to activations, gradients, optimizer state, or all three.

Set batch size to one, enable mixed precision and gradient checkpointing, and confirm the optimizer isn't storing full-size state you don't need. Lower resolution if the allocation still fails.

It crashes after several steps

Watch for a memory leak, changing bucket size, periodic logging, or another process taking VRAM. A later, larger-resolution bucket can require more memory than the first batch.

Compare the current image bucket with earlier steps. Disable optional previews and restart with a clean process. If memory grows each step, update or isolate the trainer before treating it as a normal capacity limit.

It crashes during validation or sample generation

Training fits, but the trainer loads or retains an inference pipeline for previews. Lower validation batch size and sample resolution, generate fewer samples, or run samples in a separate process after saving.

This is a distinct failure. Reducing training rank may do nothing if the preview pipeline is the peak.

Step 1: Free Memory Outside the Trainer

Use your operating system or NVIDIA tools to see which processes own GPU memory. Close ComfyUI, Forge, video editors, games, and old training terminals. A closed browser tab may not release the browser's GPU process immediately.

Restarting the trainer clears cached and fragmented allocations held by its Python process. PyTorch's reserved memory isn't always available to a second process even when a simple dashboard makes the GPU look partly free.

Don't kill unknown system processes. Identify the executable and close the application normally where possible.

Step 2: Reduce the Real Batch

Set training batch size to one. This directly reduces the number of examples whose activations must exist at the same time.

If you wanted the optimization behavior of a larger batch, use gradient accumulation. Accumulation processes smaller microbatches before applying an optimizer update. It can increase run time and doesn't eliminate every fixed memory cost, but it avoids holding the full effective batch in one forward pass.

Verify that the trainer's displayed effective batch matches your intent. Multi-GPU process count and accumulation can change the calculation.

Step 3: Enable Mixed Precision Correctly

Mixed precision stores and computes many tensors in a lower-precision format, reducing memory use. Choose only a mode supported by your GPU, framework, and trainer.

• BF16: generally preferred on compatible hardware because its wider exponent range can be more stable.
• FP16: broadly used on supported NVIDIA setups, but some operations may need full precision.
• Full-precision fallbacks: can fix numerical failures but increase memory.

Kohya's SDXL documentation recommends bf16 or fp16 and notes that an SDXL VAE can produce NaNs in half precision. Its --no_half_vae option addresses that numerical problem by running the VAE in full precision, but the tradeoff is higher memory during VAE work.

Don't use --no_half_vae as an OOM fix. Use it when logs or cached latents show NaNs and your memory budget can support it.

Step 4: Turn On Gradient Checkpointing

Gradient checkpointing saves memory by retaining fewer intermediate activations and recomputing some during the backward pass. The tradeoff is slower training.

This is one of the first options to enable for SDXL LoRA training. Current sd-scripts documentation recommends it because SDXL consumes substantial memory.

Confirm the setting appears in the launched command or resolved configuration. A UI checkbox that isn't passed to the underlying script changes nothing.

Step 5: Cache Latents

Latent caching runs training images through the VAE in advance and reuses those encoded representations. Kohya documents --cache_latents and --cache_latents_to_disk for this purpose.

• Cache in RAM: faster access, but consumes system memory.
• Cache to disk: lowers RAM pressure and preserves the cache between supported runs, but needs storage and adds disk I/O.

There is a real tradeoff: cached latents disable image augmentations that require changing the source image, including options such as color augmentation, random crop, or flip in documented sd-scripts workflows. Decide whether memory savings or those augmentations matter more for your dataset.

Delete stale caches after changing image crops, resolution, bucketing, or augmentation assumptions. A cache built from old preprocessing isn't valid evidence for the new run.

Step 6: Cache Text-Encoder Outputs Where Supported

SDXL has two text encoders, which add memory pressure. sd-scripts supports caching their outputs to memory or disk in compatible configurations.

This option usually means:

• the text encoders don't need to remain active for every step;
• text-encoder LoRA training is disabled;
• caption shuffling or caption dropout can't be applied after outputs are cached;
• denoiser-only training is required in the documented workflow.

Use this when denoiser-only training fits your goal. Don't enable it if your experiment specifically needs text-encoder adaptation or dynamic caption augmentation.

Flux and other architectures have different encoder stacks and trainer implementations. Use only the caching flags documented by that trainer.

Step 7: Lower Resolution and Bucket Limits

Image activations grow quickly with spatial dimensions. Lowering resolution or the largest bucket often saves more memory than reducing LoRA rank.

Inspect the actual files, not only the nominal training resolution. Bucketing may create a tall or wide batch with a larger pixel area than expected. Cap maximum bucket dimensions or area according to your trainer's options.

Model-family caveat:

• SD 1.5: typically uses lower native training dimensions than SDXL. An SDXL-sized dataset can waste memory and may not improve the result.
• SDXL and Pony derivatives: are commonly trained near SDXL's native scale, but you can use smaller buckets when hardware is the hard limit. Expect some detail tradeoff.
• Flux: memory behavior depends heavily on the transformer, text encoders, quantization support, and trainer. Follow that trainer's recommended resolution path.

Don't upscale weak source images just to hit a large number. Upscaling consumes memory without creating real detail.

Step 8: Review Optimizer Memory

Optimizers maintain state in addition to model weights and gradients. Memory-efficient or lower-bit optimizers can reduce that state, but availability and behavior vary by operating system, GPU, package version, and trainer.

Current sd-scripts guidance lists eight-bit optimizers or Adafactor among memory-saving options for constrained SDXL LoRA training. Treat that as a supported direction, not a promise that every package build works on your machine.

Verify the optimizer initialized successfully in the log. A silent fallback or import error can leave you running a different configuration than the UI shows.

Step 9: Reduce Rank Only When It Helps

LoRA rank, often called network dimension, controls adapter capacity and affects trainable parameters plus optimizer state. Lower rank can save memory, especially when many modules are targeted.

But rank usually isn't the largest source of peak memory. The frozen base, activations, text encoders, VAE, and optimizer can dominate.

Reduce rank after batch size, checkpointing, caching, precision, and resolution. Then compare quality using the same validation prompts; a smaller adapter may underfit a complex style or identity.

Step 10: Separate Training From Previews

If samples or validation trigger the OOM, save a checkpoint and test it in a fresh inference process. Don't keep both trainer and full generation UI loaded.

Lower preview resolution and count. Use fixed prompts and seeds so fewer samples still provide useful evidence.

The planned LoRA Studio aims to organize these runs; LocalForge AI is a separate local generation product. Kohya sd-scripts, OneTrainer, and other trainers remain valid current choices. Whichever tool you use, inspect the launched settings and peak phase rather than relying on a preset name.

Error Messages That Point Elsewhere

“Tried to allocate” with almost no free memory: another process or the current model already consumed the budget.

Reserved memory is much larger than allocated memory: fragmentation may contribute. Restart first. Use allocator configuration only when current PyTorch guidance and your trainer support it.

NaN loss or black samples: that's a numerical-stability problem, not necessarily OOM. Review precision, VAE, learning rate, and data.

Missing or unexpected keys: likely model-family or adapter-format incompatibility. Memory settings won't fix it.

System RAM or page-file crash: disk caching, dataloader worker count, preloading, and latent-cache location matter. CUDA VRAM fixes don't address host-memory exhaustion.

Minimal Safe Troubleshooting Order

Use this sequence:

1. Save the error and last log lines.
2. Restart with no other GPU applications.
3. Confirm the correct family-specific script.
4. Set batch size one.
5. Enable supported mixed precision and gradient checkpointing.
6. Cache latents, then text outputs only if the workflow permits.
7. Lower maximum image area.
8. Change optimizer.
9. Lower rank.
10. Move validation and samples to a separate process.

Bottom Line

Fix the phase that actually runs out of memory. Loading, caching, training, and validation have different peaks.

Batch size one, mixed precision, gradient checkpointing, and family-appropriate caching solve the common training-step case. Lower resolution before gutting adapter capacity, and document the augmentation or text-encoder features you give up for each memory optimization.