How to Train a Pony LoRA
Pony LoRA training uses familiar SDXL tools, but a generic SDXL tutorial misses the part that matters most: Pony's own learned vocabulary. Pony Diffusion V6 XL is an SDXL fine-tune trained on a mix of natural-language captions and tags. Its model card documents score, source, and rating tags that shape prompting. A dataset captioned for a photorealistic SDXL checkpoint can therefore teach the wrong relationships even when the training script runs perfectly.
This guide focuses on Pony Diffusion V6 XL and compatible V6-derived checkpoints. It covers model selection, image cleaning, booru-style and natural-language captions, score-tag handling, current SDXL LoRA tools, memory controls, checkpoints, and validation. It also draws a hard line between Pony V6's SDXL architecture and Pony V7's AuraFlow lineage. Their shared name doesn't make their LoRAs interchangeable.
Quick Answer
Train a Pony V6 LoRA with an SDXL LoRA trainer, but use a Pony V6 or compatible V6-derived checkpoint as the base and caption for Pony's vocabulary. Keep a consistent trigger, use accurate content tags or clear natural language, handle score_ and source_ tags deliberately, save intermediate checkpoints, and test with Pony prompts at multiple LoRA weights.
Don't train against base SDXL and assume the adapter becomes “Pony” at inference. Don't move a Pony V6 LoRA to Pony V7: V6 is SDXL-based, while V7 is based on AuraFlow. Architecture mismatch isn't fixed by renaming the file or changing the strength.
Understand the Pony Families
Pony Diffusion V6 XL is an SDXL fine-tune. It uses SDXL-compatible training scripts and loaders, but its data distribution and prompt vocabulary are specialized.
The V6 model card documents:
- quality score tags such as
score_9,score_8_up, and lower score tiers; - source tags including
source_pony,source_furry,source_cartoon, andsource_anime; - rating tags such as
rating_safe,rating_questionable, andrating_explicit; - support for both tags and natural-language descriptions;
- an opinionated prompt style that doesn't require generic quality phrases used by other anime checkpoints.
Pony V7 is not an SDXL checkpoint. It belongs to an AuraFlow-based branch. Use a trainer and adapter format that explicitly support that family. This page's sd-scripts and SDXL advice targets V6.
Community models labeled “Pony” can be merges or derivatives with different recommended tags. Read the exact checkpoint card and keep its revision in your run notes.
Choose the Training Base
Train on the checkpoint you expect to use most often. The original V6 XL model is the cleanest compatibility reference. A derivative may perform better in a narrow style but can make the LoRA less portable.
Ask three questions:
- Does the checkpoint explicitly identify Pony V6 or SDXL ancestry?
- Does its model card preserve V6 score, source, rating, and CLIP behavior?
- Does its license permit your training and distribution plan?
Don't infer compatibility from sample art. Two illustrated models can look similar and use different architectures or vocabularies.
Build a Pony Dataset
Pony is strong at stylized characters, anthro subjects, animals, and anime or cartoon aesthetics. That strength doesn't rescue a repetitive dataset.
For a character, vary angle, expression, pose, outfit, background, framing, and lighting while keeping identity features visible.
For a style, vary characters, species, scenes, poses, and compositions while preserving the treatment you want learned.
For an object or feature, include clear views and negative variation. If the feature appears only in one pose, the LoRA can bind feature and pose together.
Delete blurry images, watermarks, accidental text, severe compression, bad crops, and near-duplicates. Review hands, limbs, tails, ears, wings, and other anatomy you expect the LoRA to reproduce.
Use SDXL-style resolution buckets rather than forcing every image to a square. Follow the current trainer's 1024-class workflow and inspect crops before training.
Caption for Pony Instead of Generic SDXL
Pony V6 understands both natural language and tags. Booru-style tags are useful when they accurately name character traits, species, clothing, pose, expression, camera framing, and scene details.
A practical character caption might follow this order:
score_9, source_anime, myponychar, solo, blue hair, green eyes, black jacket, standing, city street, night
The order isn't magic, but consistency helps auditing. Use one unique trigger and spell it the same way in every relevant caption.
Handle quality tags deliberately
If every training caption starts with score_9, the adapter can learn an association between your trigger and that tag distribution. That may be acceptable, but it isn't neutral.
Decide whether quality tags describe your source data, act as conditioning you want available, or should be omitted from the training captions. Document the choice and test prompts with and without score tags.
Use source tags accurately
Apply source_anime, source_cartoon, source_furry, or source_pony only when the image matches. Wrong source tags teach noise and make prompt diagnosis harder.
Describe variable attributes
If outfits change, caption them. If backgrounds change, caption them. If a tail, horn, marking, or accessory defines the character, name it consistently.
Auto-taggers often produce long lists with false positives and synonym drift. Use them to draft, then remove wrong tags, merge synonyms, normalize ordering, and check the trigger.
Numbered Pony V6 Training Workflow
1. Freeze the goal and checkpoint
State whether you're training identity, style, anatomy, clothing, pose, or another concept. Save the exact Pony V6 checkpoint filename, source URL, revision, and license.
Define success in prompt-control terms. “Looks good” isn't enough. A character LoRA should retain identity across outfits and settings; a style LoRA should affect treatment without inserting the same subject.
2. Audit and version the dataset
Remove defects and duplicates, check consent and rights, and freeze a dataset copy. Keep a short manifest with image source, caption status, and any exclusions.
Hold back a few representative references. Don't alter the dataset halfway through a run you plan to compare.
3. Create captions
Add the trigger, relevant source and rating tags, visible attributes, pose, framing, and environment. Choose a stable convention for underscores, commas, and tag order.
Inspect every caption beside its image. A tagger that confuses ears, tails, clothing, or character count creates direct training errors.
4. Install an SDXL LoRA trainer
Kohya-ss sd-scripts uses sdxl_train_network.py for SDXL-family LoRAs. A maintained Kohya GUI can expose the same settings, while Diffusers provides code-first SDXL LoRA scripts.
Use a clean environment and pin the trainer version. Confirm the Pony checkpoint loads and generates a baseline image before training.
5. Start with UNet-only training
Current sd-scripts guidance recommends UNet-only training for basic SDXL LoRAs because SDXL has two text encoders. This is the sensible first Pony run too.
Text-encoder training can help specialized vocabulary experiments, but it increases memory use and interacts with text-output caching. Don't add it until a controlled UNet-only run proves the need.
6. Set capacity and optimization
Start from a current SDXL LoRA example rather than a random Pony preset. Choose rank, alpha, optimizer, precision, learning rate, buckets, and batch size as one documented configuration.
Gradient checkpointing, cached latents, cached text-encoder outputs, and efficient optimizers can reduce GPU memory use. Caching text outputs disables caption augmentation and prevents text-encoder LoRA training in the documented sd-scripts path.
Don't claim that Pony “needs” a fixed amount of VRAM. Fit changes with rank, precision, optimizer, batch, resolution, caches, and whether text encoders train. Measure the exact run and use the trainer's current low-memory guidance.
7. Run a smoke test
Prove that the trainer sees all captions, image buckets look right, loss stays finite, samples render, and checkpoints save. Check that the sample prompt uses Pony vocabulary.
A generic prompt can hide a broken setup. Include the trigger, source tag, score tag, character or concept details, and a known Pony sampler configuration in validation.
8. Train with checkpoints
Save several intermediate adapters and generate a fixed sample grid. Keep seed, checkpoint, sampler, steps, resolution, prompt, and negative prompt constant.
An earlier checkpoint often has better flexibility. Don't select the final adapter by default.
9. Test Pony-specific control
Test:
- multiple
score_levels or no score tag; - relevant
source_tags; - safe or other rating conditioning appropriate to your lawful use;
- different outfits, backgrounds, poses, and expressions;
- close, medium, and wide framing;
- several LoRA strengths;
- the training checkpoint and one compatible V6 derivative.
Check whether the trigger overrides source or rating tags. If it does, captions or exposure may have bound too much to the concept.
10. Package the adapter
Export .safetensors and state “Pony V6 / SDXL” clearly. Include the exact base checkpoint, trigger, score and source tag expectations, suggested strength range, resolution, trainer version, known failures, and license.
Add a warning that Pony V7/AuraFlow isn't compatible. This prevents a predictable support problem.
Common Pony LoRA Problems
- Plastic or muddy output: confirm the exact checkpoint's recommended CLIP and VAE behavior, then compare an untouched base sample.
- Character only works in one outfit: captions omitted clothing variation or the dataset repeated one look.
- Wrong species or features: source and anatomy tags are inaccurate, inconsistent, or underrepresented.
- Generic anime quality words do little: use the checkpoint's documented Pony score and source vocabulary.
- LoRA loads but barely works: the inference checkpoint may be a different SDXL lineage or incompatible Pony derivative.
- Every prompt becomes the training composition: use an earlier checkpoint, remove duplicates, add variation, or reduce exposure.
- V7 produces errors or nonsense: a V6 SDXL LoRA isn't an AuraFlow adapter.
Ethical and Legal Dataset Rules
Train on material you created, licensed, or have permission to use. Don't train non-consensual intimate content or deceptive identity models. Keep source records and honor base-model and dataset licenses.
Rating tags are conditioning vocabulary, not permission. Your use still has to follow law, consent, platform rules, and the model license.
Planned Local Workflow
Read the LoRA Studio overview for LocalForge AI's planned approach to organizing private-image review, tag cleanup, run records, and checkpoint grids. It isn't a currently available trainer, so use Kohya or Diffusers now and verify the Pony checkpoint family, license, and exported adapter compatibility yourself.
Next Steps
- Choose an exact Pony V6 checkpoint and save its model card.
- Clean and version the training images.
- Normalize trigger, score, source, rating, and content tags.
- Run a short UNet-only smoke test with Pony validation prompts.
- Compare intermediate adapters before packaging the winner.
Bottom Line
Pony V6 LoRA training is SDXL training with a specialized language. The script is the easy part. Correct model lineage, honest tags, varied data, and Pony-specific validation decide whether the adapter works. Keep V6 and V7 separate, and document the compatibility boundary in the model card.