Work / Photo to Minecraft building

Photo to Minecraft building

An on-device pipeline that turns one picture of a house, whether a photo, a sketch or a painting, into a playable Minecraft build.

Sheet 1 of 5
Type	Personal project
Timeline	Sep 2026, ongoing
Status	In progress
Stack	Python, PyTorch, DINOv2, ONNX Runtime, Java 21, TypeScript, WebGL2, Fabric and NeoForge (Minecraft 1.21.1)
Code	Private repository, walkthrough on request

Drag to scrub

Fig. 1. How one picture becomes a build, step by step. The description fields are real output for this test image; the overlay and the block model are simplified drawings of what each stage does. Finished builds are further down.

The problem

Players love recreating buildings in Minecraft, but doing it well takes hours and real building craft. Blocks are one-metre cubes from a limited palette, so detail has to come from technique (stairs for a roof pitch, slabs, trapdoors for shutters), not from resolution. A single picture also hides three sides and the whole interior, and pictures come in every style: photos, anime art, pencil sketches, pixel art.

The goal: give it one picture of a house, in any style, and get back a complete Minecraft house with Minecraft proportions, real building techniques and finished back sides. It has to run offline on a player’s PC while the game is running, so a large general vision-language model is out of the question.

How it works

The principle is simple: the AI only reads; the engine builds.

1Read the picture
- Building type, style, storeys, roof
- Wall, window and door pixels
- Coarse 3D voxels of the shape
- Style and materials
Small DINOv2-based models, ONNX Runtime in a separate Java process
2Describe it
- Volumes and floors
- Roofs and facade grids
- Materials and palette
An editable building description. Edits rebuild in milliseconds.
3Build it
- Parametric component library
- Real building techniques
- Invents the unseen sides
Deterministic Java engine, then checks that rooms are reachable, lit and supported
4Place it
- Desktop preview studio
- Fabric and NeoForge mod
Ghost preview in the world: move, rotate, place, undo

Fig. 2. The system. Everything to the right of the picture runs on the player's machine, on the CPU.

Read. Small models read the picture: what kind of building it is, its style, storeys and roof; which pixels are wall, window and door; and a coarse 3D voxel estimate of its shape.
Describe. Everything goes into an editable building description. Changing a value rebuilds the house without running the models again.
Build. A deterministic Java engine turns the description into blocks with real Minecraft techniques, completes the sides the picture doesn’t show, and checks the result is walkable and safe to play in.
Place. The build opens in a desktop preview, or in the game through the mod, where you can position it and place it block by block.

Key decisions

A description plus an engine, not end-to-end voxels. A model that outputs blocks directly would produce invalid, uneditable builds. Splitting the job means perception only has to describe what it sees, while the engine guarantees that any valid description becomes a legal house, and that the same description always gives the same build.
Small models on the device instead of a general VLM. Players can’t download tens of gigabytes, and general models are slow on a CPU and poor at counting floors. So the pipeline uses small task models exported to ONNX and runs them in a separate Java process, where a crash can’t take the game down. A large LLM is used only offline, as a teacher that labels training pictures.
A GPU synthetic-data factory, and an honest negative result. No dataset labels doors, roofs or facade grids the way the engine needs, so I built a WebGL2 renderer over the engine that outputs every image with pixel-exact labels. Training the facade model on those synthetic images alone made it worse on real photos (63.8 vs 75.0 mIoU), so the model trained on real facades stayed. The factory proved its worth later, as the label source for the 3D model and for restyled training pictures.
A 3D model for the shape. Flat attributes weren’t enough to recover a building’s massing, so the pipeline gained a model that predicts semantic voxels (wall, roof, window, door and more) from the picture; the Java side fits them back into an editable description. This lifted the end-to-end building IoU from 0.515 to 0.588.
An “aesthetic floor”. Recognition will sometimes be wrong, so the engine has to produce a good-looking house from any description: curated style kits, a step that snaps descriptions onto them, and checks that catch ugly combinations.

Results so far

Measure	Value	Context
Facade segmentation	75.0 mIoU	hznu facade benchmark, 5 classes
Global attribute model	97.2%, 96.4%, 88.5%	Medium, style family and view on real photos; matches its LLM teacher at about 50 MB and 48 ms on a CPU
Synthetic data factory	385.8 images/min	Each image with 9 pixel-exact label channels
End-to-end building IoU	0.515 → 0.588	+0.073 (90% CI 0.042 to 0.107) over 21 reference houses
Engine rebuild	about 85 ms	Median time to rebuild a house exterior, so edits feel instant
Mod	44/44 GameTests	On both Fabric and NeoForge

Inputs shown on this page are AI-generated test pictures. Renders use Minecraft's own block textures from a local copy of the game.

What it makes

Pencil sketch of a two-storey Tudor house with a jettied upper floor, a steep roof and a tall chimney — **Fig. 3.** Six test pictures in different styles and the Minecraft builds made from them. The pictures are AI-generated test images.

Minecraft build of the Tudor house with a steep brown roof, dark timber framing on white walls and a chimney — **Fig. 3.** Six test pictures in different styles and the Minecraft builds made from them. The pictures are AI-generated test images.

Early engine render of a timber-framed house with a plain plank roof inside a wireframe bounding box — **Fig. 4.** The same building description through the engine on day 2 and on day 20. The craft passes added a layered roof with eaves, a chimney, window boxes, stone paths and a garden.

Later engine render of the same house with a stepped roof, a chimney, window boxes, stone paths and a tree — **Fig. 4.** The same building description through the engine on day 2 and on day 20. The craft passes added a layered roof with eaves, a chimney, window boxes, stone paths and a garden.

Rows showing an engine render of a house, a depth map, and the same view restyled as a photo, cartoon, watercolour and concept art — **Fig. 5.** Synthetic training data: an engine render, its depth, then restyled as a photo, a cartoon, a watercolour and concept art.

Four panels of the same rendered house: colour image, colour-coded facade classes, depth map and building mask — **Fig. 6.** One synthetic view with its pixel-exact labels: colour, facade classes, depth and building mask.

A dozen small Minecraft builds of chimney, roof edge and ridge cap variants — **Fig. 7.** Parametric components: chimneys, roof edges and ridge caps the engine assembles from parameters.

Pairs of Minecraft houses: plain boxy versions on the left, finished versions with styled roofs and gardens on the right — **Fig. 8.** The aesthetic floor: random descriptions before and after being snapped onto a style kit.

Two versions of the same Minecraft houses, the second with steeper roofs, dormers and chimneys — **Fig. 9.** Roof craft before and after: steeper backed roofs, dormers and chimneys.

A building facade photo with predicted wall, window and door regions beside the ground-truth labels — **Fig. 10.** Facade segmentation on a real facade: prediction next to ground truth (hznu facade data, CC BY 4.0).

What I built

Perception models: five small PyTorch models on DINOv2 backbones (ViT-S and ViT-B): a global attribute head, a facade segmenter, a description student distilled from an LLM teacher, a style and material classifier, and a triplane decoder on a LoRA-tuned ViT-B that predicts a 64³ grid of semantic voxels.
On-device runtime: ONNX export with int8 and fp16 quantisation, parity gates against the PyTorch models, and ONNX Runtime in a separate Java process that loads a hash-verified model pack.
Data: a GPU synthetic-data factory (WebGL2 multiple render targets, 385.8 images a minute with 9 label channels), diffusion restyling with a depth ControlNet, and LLM-teacher labels with strict JSON schemas for about 3,000 licensed photos.
Engine and tools: a deterministic Java 21 engine (Gradle multi-module) with a parametric component library, a block-state solver and validity checks; .schem, .litematic and .nbt exporters; and a desktop studio in TypeScript, Vite and WebGL, packaged as a single Windows executable.
Mod: Fabric and NeoForge mods for Minecraft 1.21.1 with an in-world ghost preview, tick-budgeted placement and undo, tested with 44 GameTests on each loader.
Testing: about 37,700 automated tests (JUnit, Vitest, Playwright and GameTests).

The code was written with AI coding assistants (Claude Code, GPT and Codex).

Not an official Minecraft product. Not approved by or associated with Mojang or Microsoft. Facade segmentation trained and evaluated on the hznu facade dataset (CC BY 4.0).