AC3S: Adaptive Conditioning for 3D-Aware Synthetic Data Generation

Abstract

Synthetic data generation has emerged as a powerful tool for improving data scalability in computer vision. Recent diffusion-based pipelines have demonstrated strong photorealism. However, how to enforce precise 3D structure and pose consistency in generated images remains challenging. Existing methods leverage visual prompts such as edge maps to guide diffusion models, but often suffer from over-conditioning artifacts that degrade image realism and limit dataset quality. In this paper, we present a diffusion-based image generation framework that enforces 3D structural alignment while preserving photorealism through adaptive conditioning. Our framework, Adaptive Conditioning for 3D-Aware Synthetic Data Generation (AC3S), introduces a self-supervised visual prompt modulator that dynamically adjusts the strength of ControlNet conditioning, preventing over-conditioning and enabling the diffusion model to retain its generative expressiveness. To further enhance diversity and semantic consistency, we develop a multi-agent vision language model framework that composes detailed and 3D-aware prompts aligned with the underlying geometric structure. Together, these components enable the scalable generation of high-quality synthetic datasets with accurate 2D and 3D annotations. Extensive experiments demonstrate that our method significantly improves image quality and downstream utility.

Contributions

Method

AC3S adaptively controls both the visual and the textual conditioning of a pre-trained diffusion model. The pipeline is composed of four interconnected modules.

Visual prompts from rendering

Following 3D-DST, 3D models are sourced from ShapeNet, Objaverse, and OmniObject3D (about 30 CAD models per category). For each model we sample a camera viewpoint by varying distance, azimuth, and elevation under constraints that avoid implausible perspectives, render it with Blender, and extract a Canny edge map as the visual prompt. Renders need no materials, lighting, or backgrounds: they are used purely as a source of geometric structure. We also explored depth prompts, but edge maps consistently gave the highest quality samples.

Adaptive visual prompt modulator

Edge maps computed from material-less renders are sparse, and injecting unmodulated ControlNet outputs over-conditions the diffusion model: textures are oversimplified and scenes become uninteresting. We find that text conditioning lowers the denoising objective, whereas visual conditioning raises it, suggesting that ControlNet introduces harmful noise. Attenuating ControlNet residuals with a conditioning scale λ directly mitigates the damage:

A single hand-tuned λ does not generalize across categories, and even varies across random seeds for the same prompt. However, λ correlates with the magnitudes of the U-Net features ‖f‖ and the ControlNet outputs ‖𝓕‖. Our modulator is therefore a three-layer MLP that takes the per-block feature norms of ControlNet and the U-Net at the first denoising step and predicts λ in [0.3, 1.0].

Self-supervised pseudo-labels via just-noticeable difference

Training the modulator with a denoising loss is degenerate, since the loss is trivially minimized at λ = 0. Instead we use self-supervision with pseudo-labels. Sweeping λ from 0.3 to 1.0 in steps of 0.1 with a fixed seed, the desired 3D structure is suddenly enforced once λ crosses a threshold, while image quality degrades gradually as λ grows. The optimal λ is therefore the smallest value above that threshold, analogous to a just-noticeable difference (JND) in pose. We detect it by running k-means (k = 2) on the pixel-level features of the generated sequence, taking the cluster containing the λ = 1.0 image as the aligned cluster, and selecting its smallest λ. Repeating this over all categories yields pseudo-labels for training the modulator with no human annotation.

Multi-agent VLM text prompt generator

Template prompts such as “a high-quality image of a [CLASS]” lack detail, while captions extracted from real images often conflict with the visual prompt (for example, describing a close-up with multiple instances when the render shows a single holistic object), producing hallucinations. We orchestrate a team of specialized VLM agents, each profiled with its role, input modalities, output schema, and behavioral constraints, to compose prompts that are semantically consistent with the visual prompt:

• A1Subject Identifier. Cross-references the render I_render and a real ImageNet reference I_ref to commit to a 1–3 word subject label, richer than the bare ImageNet class.
• A2Genre Selector. Picks a photography genre (wildlife, macro, product, …) from a predefined set, encoding a prior on style and conventions.
• A3Pose Extractor. Reads the subject and camera pose from the render, so the prompt agrees in perspective with the edge map extracted from the same render.
• A4Scene Composer. Extracts the subject's visual attributes (color, texture) and background scenery from the reference image, capturing the diversity present in ImageNet.
• A5Positive Prompt Synthesizer. Assembles the outputs of A2–A4 into a coherent positive prompt within a CLIP-friendly word budget.
• A6Protected Terms Identifier. Lists synonyms of the positive prompt that must never appear in the negative prompt, so the subject is not inadvertently suppressed.
• A7Negative Prompt Synthesizer. Assembles tiered negative terms (floating artifacts, CG-render looks, color casts, plain backgrounds, …) while excluding protected terms.

Each agent follows a structured prompt template with seven sections (Role, Goal, Context, Constraints, Contrastive Examples, Output Format, Criteria) built on chain-of-thought and structured-output principles. The workflow is implemented with LangGraph's StateGraph: A1 runs first, A2–A4 run in parallel, and A5–A7 run sequentially, with agents communicating through a shared typed state following the message-pool pattern of MetaGPT. We use Qwen3-VL-Instruct with 4-bit quantization.

Dataset filtering with DreamSim

Rather than training a per-class pose estimator with k-fold cross-validation to score samples as in 3D-DST, which is expensive at scale and unreliable for small datasets, we rank samples with DreamSim, a perceptual metric that primarily captures object pose and semantics. Each synthetic image is scored against its render after converting both to grayscale to stay invariant to color, and the strongest samples are kept.

Results

Image quality ablation

We compare 1,000 synthetic images per method against 1,000 ImageNet images over 50 classes with fixed seeds. Starting from the 3D-DST baseline, the modulator and the text prompt generator improve FID by 9.66 and 8.26 points respectively, and together by 15.95 points, with the largest gain (23.8 points) on animal classes.

Synthetic pre-training for classification

We generate AC3S-50, a 50-class dataset with 1,000 images per class (Stable Diffusion v1.5, 20 denoising steps), and compare against Text2Img and 3D-DST datasets of the same size. ResNet-18, Swin-S, and ConvNeXt-S are trained from scratch on synthetic data and then fine-tuned on ImageNet. AC3S pre-training gives the largest overall gain for every backbone: +2.6% for ConvNeXt-S and Swin-S and +0.99% for ResNet-18 over the real-only baseline.

Top-1 accuracy (%) on a held-out ImageNet test split. All methods except the baseline are pre-trained on synthetic data before fine-tuning on ImageNet.

Synthetic pre-training for pose estimation

To validate the 3D annotations, we pre-train ResNet-18 pose estimators on synthetic data and fine-tune on PASCAL3D+ for the three categories shared with AC3S-50. Accuracy is the fraction of predictions whose geodesic rotation error is below π/6 or π/18. AC3S improves over 3D-DST on every category, by 1.23% at π/6 and 2.17% at π/18 on average, with a 7.39% jump over the real-only baseline on sofa at π/18.

Pose accuracy (%) on PASCAL3D+ after synthetic pre-training and fine-tuning on real data.

Without any fine-tuning on real data, the gap widens dramatically, indicating that AC3S images resemble the PASCAL3D+ distribution far more closely than 3D-DST images.

Pose accuracy (%) on PASCAL3D+ when training on synthetic data only.

Code

The official implementation is available on GitHub under the MIT license. It contains the visual prompt modulator (src/ac3s), the LangGraph multi-agent prompt generator (src/vlm_agents), the Stable Diffusion + ControlNet generator (src/stable_diffusion), and end-to-end scripts.

conda create -n ac3s python=3.10 -y
conda activate ac3s
pip install -e .
pip install -r requirements.txt

CAD edge renders are produced with DST3D. Given the renders and ImageNet reference photos, the pipeline runs in five steps, each writing the default input of the next:

1. Generate text prompts. The multi-agent VLM writes a caption and negative prompt for every render. scripts/generate_text_prompts.py
2. Generate pseudo-labels. Sweep λ ∈ {0.3, …, 1.0} per render, cluster the sequence with k-means, and record the JND scale together with ControlNet and U-Net feature norms. scripts/generate_pseudo_labels.py
3. Train the modulator. Fit the three-layer MLP on the pseudo-labels with an MSE loss. scripts/train_modulator.py
4. Generate synthetic images. Predict λ from a single denoising pass, add a small safety padding, clamp to [0.3, 1.0], and sample with Stable Diffusion v1.5 + ControlNet. scripts/synthetic_generation.py
5. Filter with DreamSim. Score every render/synthetic pair in grayscale to rank samples. scripts/dreamsim_filtering.py

# 1. VLM captions + negative prompts
python scripts/generate_text_prompts.py \
    --data_path /path/to/cad_renders --imagenet_path /path/to/imagenet --output_path prompts/
# 2. Pseudo-labels for the modulator
python scripts/generate_pseudo_labels.py --data_dir /path/to/cad_renders --prompts_dir prompts/
# 3. Train the visual prompt modulator
python scripts/train_modulator.py
# 4. Generate synthetic images
python scripts/synthetic_generation.py \
    --data_directory /path/to/cad_renders --prompts_directory prompts/
# 5. DreamSim filtering
python scripts/dreamsim_filtering.py --data_directory /path/to/cad_renders

Citation

Acknowledgements

This research is supported by the National Artificial Intelligence Research Resource Pilot Awards NAIRR250199, NAIRR260019, and NAIRR260077, the AMD University Program's AI & HPC Cluster, NVIDIA Academic Grant Program, and Lambda's Research Grant. It is also supported in part by the Center for Networked Intelligent Components and Environments, a partnership between the University of Illinois Urbana-Champaign and Foxconn Interconnect Technology. Computational resources are also provided by Delta and DeltaAI at the National Center for Supercomputing Applications through ACCESS allocations CIS250012, CIS250816, and CIS251188.