How to Fix LoRA Training That Keeps Crashing
When LoRA training keeps crashing, the last visible line is often not the real cause. GPU operations can report errors asynchronously, data workers can fail outside the main process, a corrupt image may appear only in one bucket, and validation can load a second pipeline after hours of stable training. Treating every crash as an out-of-memory problem wastes time and can hide damaged data or an unstable environment.
This guide diagnoses crashes by phase, recurrence, and exception class. It deliberately does not duplicate the dedicated CUDA OOM page. If the log explicitly says CUDA out of memory or Tried to allocate, use that memory guide. Stay here for access violations, process exits, CUDA initialization and kernel errors, DataLoader worker failures, NaNs, disk errors, dependency mismatches, crashes on one sample, and runs that die during saving or previews. Preserve the full log and a working configuration before changing anything.
First: Capture Evidence Before Restarting
Save the complete console output, not only the final sentence. Record:
- trainer and script version or commit;
- Python, PyTorch, CUDA runtime, driver, and GPU;
- launched command or resolved configuration;
- base model and architecture;
- dataset manifest and image count;
- last completed step and current bucket;
- system RAM, GPU memory, free disk, and page-file state;
- whether a checkpoint or preview was being written;
- Windows Event Viewer or Linux kernel messages when the process vanished.
Copy the last known working config. Do not update packages, replace the model, and rewrite the dataset at once. A crash investigation needs a stable baseline.
For the surrounding workflow, use the LoRA Studio overview, then continue with fix LoRA training out-of-memory errors when that decision becomes relevant.
Route Explicit OOM Errors Elsewhere
CUDA out of memory, Tried to allocate, and a process killed by confirmed host-memory exhaustion are capacity problems. Use the LoRA training out-of-memory guide for batch size, precision, gradient checkpointing, caches, resolution, optimizer state, and validation peaks.
This page handles the broader case where training terminates despite apparently fitting in memory. Memory can still contribute, but the evidence and fixes differ.
Identify the Crash Phase
Before the model loads
Import errors, missing libraries, unsupported operators, authentication failures, and incompatible package versions happen before useful training begins. Recreate the documented environment for the trainer version. Confirm the base repository and local files are complete.
A GUI opening successfully does not prove its training subprocess has the same Python environment.
During model or adapter initialization
Missing keys, unexpected shapes, invalid headers, and component lookup errors point to architecture or file compatibility. Confirm SD 1.5, SDXL, Flux, and other families are not mixed. Re-download a file only after checking its expected hash or source.
During dataset scanning or caching
A corrupt image, unreadable caption, extreme dimensions, unsupported color mode, or full cache disk can crash before step one. Validate every file with the same image library used by the trainer. Check paths, permissions, caption encoding, and available storage.
On the first training step
This phase initializes activations, gradients, optimizer state, and compiled kernels. Explicit OOM belongs on the memory route. Illegal memory access, unsupported dtype, missing kernel, or operator errors suggest a PyTorch, CUDA, extension, precision, or hardware mismatch.
At a repeatable later step
A crash at the same step often maps to the same sample, bucket, save interval, or validation interval. Log sample paths or isolate dataset partitions. Compare the step with checkpoint and preview schedules.
At random later steps
Random exits can come from unstable GPU clocks, driver resets, overheating, failing RAM, storage errors, multiprocessing races, or another process changing resources. Return hardware to stock settings and run a minimal controlled job.
During saving or preview generation
Check free disk, output permissions, filename length, antivirus interference, and whether the destination is a network or synchronized folder. Preview code may load a separate inference pipeline and fail independently of training.
Step 1: Reproduce With a Minimal Run
Make a copy of the configuration and create a small diagnostic dataset from known-good files. Disable optional previews, external loggers, uploads, compilation, custom extensions, and nonessential augmentations. Save locally to a short path.
Keep the same model family and trainer. Run enough steps to cross the previous failure point if practical.
- If the minimal run fails the same way, focus on environment, model, hardware, or core settings.
- If it succeeds, reintroduce dataset partitions and optional features one at a time.
- If only the full run fails, inspect resources and interval-triggered work.
A minimal reproduction is more useful than repeatedly launching the full job with random toggles.
Step 2: Validate Images and Captions
Decode every image fully rather than trusting file extensions. Flag zero-byte files, truncated streams, giant pixel dimensions, unsupported formats, and inconsistent modes. Apply EXIF orientation safely and write corrected copies instead of modifying the only originals.
Check that caption filenames and extensions match trainer rules. Detect invalid text encoding and unexpectedly long captions. If the trainer logs sample paths, preserve that option until the failure is resolved.
To isolate a deterministic data crash, split the dataset in halves and run the same smoke test. Continue narrowing the failing half. Keep seed, worker count, and ordering fixed so the test remains meaningful.
Step 3: Reduce DataLoader Complexity
Worker subprocesses can hide the original exception behind a message such as DataLoader worker exited unexpectedly. Set the worker count to zero for diagnosis so loading happens in the main process and the traceback is clearer.
If that fixes the run, increase workers gradually. Review persistent workers, prefetching, pinned memory, custom transforms, and multiprocessing behavior. On platforms using CUDA subprocesses, follow PyTorch's multiprocessing guidance. PyTorch documents that CUDA requires spawn or forkserver rather than the fork start method.
Worker zero is a diagnostic setting, not always the fastest final configuration.
Step 4: Make CUDA Errors Synchronous
CUDA work is normally asynchronous, so a stack trace may point to the operation that noticed an earlier failure. PyTorch documents CUDA_LAUNCH_BLOCKING=1 as a debugging aid that makes CUDA calls synchronous.
Set it only for a diagnostic run because it slows execution. Capture the new traceback, then remove it after locating the failing operation. Do not publish it as a performance fix.
For specialized memory checking, PyTorch also documents disabling its caching allocator with PYTORCH_NO_CUDA_MEMORY_CACHING=1. Use that only with an appropriate debugging tool; it changes behavior and performance.
Step 5: Align the Software Stack
Verify the trainer's supported Python and PyTorch versions, the installed CUDA build, GPU driver compatibility, and compiled extensions such as attention or low-bit optimizer packages.
Recreate a clean environment from the trainer's current instructions. Do not solve one import error by installing arbitrary latest versions into a previously working environment. Save the working package list before experiments.
If disabling an optional extension fixes the crash, confirm that extension provides a build for your exact PyTorch and CUDA combination. A successful import does not prove every kernel path is valid.
Step 6: Separate Numerical Failure From Process Failure
NaN loss, infinite gradients, black samples, and invalid latent values are numerical problems. Check learning rate, precision, VAE behavior, optimizer support, gradient scaling, corrupt inputs, and architecture-specific recommendations.
A NaN does not necessarily crash immediately. It may poison later validation or saving. Stop the run when loss first becomes nonfinite and inspect the preceding steps.
Try a supported safer precision mode and conservative learning rate in a separate diagnostic run. Full precision may use substantially more memory, so do not apply it without checking capacity.
Step 7: Inspect Driver and Hardware Stability
Reset GPU overclocks and undervolts to stock, including factory tuning if a diagnostic baseline is available. Watch temperatures and power. Close games, inference tools, browsers using GPU acceleration, and monitoring overlays that inject into GPU applications.
A driver reset may appear as a disappearing Python process, device loss, or operating-system display recovery. Check system logs around the timestamp. Update or roll back the GPU driver only with evidence and a recorded current version.
If unrelated GPU workloads also fail, stop blaming the trainer. Test the hardware and driver with a known stable workload.
Step 8: Check Storage and Output Paths
Check free space on the output, cache, temporary, and system drives. Atomic checkpoint saves can require temporary space beyond the final file size. A synchronized folder may lock or upload a partially written file.
Use a local SSD path with ordinary characters and reasonable length for diagnosis. Confirm write permission by creating and replacing a test file through the same user account. Exclude only the specific project directory from security scanning if your policy permits it; do not disable protection system-wide.
Never resume from a checkpoint created during a failed write until it loads and passes an integrity check.
Step 9: Isolate Checkpointing and Validation
If the crash lands exactly on a save interval, reduce output complexity and test a manual save. If it lands on validation, disable previews temporarily or run them in a clean inference process from a completed checkpoint.
Lower preview batch and resolution only when the error is memory related. For other exceptions, inspect the validation pipeline, prompt file, sampler, VAE, and adapter reload path.
Training succeeding while validation crashes proves the optimizer path can run. It does not prove the exported adapter or preview configuration is valid.
Step 10: Resume Safely
Resume only from a checkpoint written before the failure and verified by the trainer. Confirm global step, optimizer, scheduler, and random state restoration. Keep the damaged or suspect output untouched for evidence.
If dataset ordering caused a deterministic crash, repair the input and start a clearly labeled new run unless the trainer documents safe continuation. Changing data under a checkpoint can invalidate comparisons.
Symptom Map
Worker exited unexpectedly: set workers to zero, validate data, then rebuild multiprocessing settings.
Illegal memory access or device-side assert: use synchronous CUDA debugging, inspect dtype and kernels, then align packages.
Process vanished with no Python traceback: inspect OS logs, driver resets, RAM, disk, security software, and hardware stability.
Crash at every save: check storage, permissions, path, serializer, and checkpoint integrity.
Crash at every preview: isolate the inference pipeline and its model, VAE, prompt, and resource settings.
Explicit CUDA OOM: follow the dedicated OOM guide rather than this general checklist.
Bottom Line
Diagnose the phase and preserve evidence. Repeatable step failures usually point to data or scheduled work. Random exits point more strongly toward environment, driver, hardware, or resource instability.
Start with a minimal run, worker count zero, validated files, local output, and optional features disabled. Use PyTorch's synchronous CUDA debugging only to improve the traceback. Once the cause is proven, restore features one at a time and keep the repaired environment locked.
What to Do Next
Route explicit OOM errors
Follow fix lora training out-of-memory errors for the detailed workflow and checks.
Check model compatibility
Follow check lora model compatibility for the detailed workflow and checks.
Continue with audit the LoRA dataset
Use audit the lora dataset to complete the next focused check.