Adding artificial intelligence to 3D design

Bram
Backend developer
data-engineering
Adding artificial intelligence to 3D design
In 2014 we won an innovation grant from the province of Gelderland based on our proposal to provide 'intelligent' gardening advice to users of Draw Your Garden (Dutch: Teken Je Tuin). We created this web application for one of our clients and we have been gradually expanding it since its release. In the app users can both design their garden and view it in 3D as well as order products and contact gardeners, who in turn can submit proposals based on the users' design.

In this article we will give you an overview of how we created methods that allow users to design their gardens intelligently. We think these methods could translate nicely to domains outside gardening. We are curious if you feel the same after reading this article.

Draw Your Garden Draw Your Garden / Teken je Tuin

Partnered with our client we proposed to develop a number of prototypes aimed at assisting users in several areas which hitherto required extensive expert knowledge. Appropriately, we named this project 'Your GardenAssistant'.

For the project we focused on three areas of garden design:

  • Advice on watering a garden; where to place different kinds of sprinklers, what kind of plants need more water, which types of sprinklers are suited to different parts of the garden and how sprinklers should be connected to what type of water source.
  • The presence and influence of sunlight and shadow in the garden; to show which areas of the garden receive most light in different seasons and parts of the day, where to locate a terrace and where to plant different sorts of plant.
  • Advice on the properties of plants and trees; which plants would prosper in different parts of the garden, how many plants flower this month, what plants are edible or poisonous.

Domain knowledge and user feedback

The first thing we concluded was that it would be crucial to involve different kinds of experts, gardeners and users in the project. We realized we needed to include them from the very start of the project, and to check back with them regularly to decide how to proceed.

The Plant Advice Prototype

3D design

The preliminary task was to create a suitable 3D design & drawing application based on our 3D Framework. Our approach is to use a full 3D environment with an top-down view for drawing. This view, a kind of orthographic projection, provides the advantages of 'flat', easy-to-use interaction during drawing & design, while the full 3D experience is just one camera shift away.

Design tool Drawing in full 3D with a top-down perspective.

Design tool The same garden, but rotated slightly

Watering

To be able to give advice about watering a garden we created a 'provided graph' and a 'moisture graph', to store how much water the sprinklers provided, and how much water different parts of the garden required. We used these to create a wizard designed to help the user choose both the right type of sprinker and a water source, and to connect these two, all in a few easy steps.

The Plant Advice Prototype Provided graph: how much water is provided. Note that the sprinkler on the bottom-right is not connected to a water source.

The Plant Advice Prototype Moisture graph: green areas receive enough water, red areas too little or possibly too much (if they do not require water, like the terrace).

Sunlight and shadows

The first thing here was to create a 'heat map' to represent the areas of the garden with the most sun exposure. To do this, we created a realistic simulation of the sun in the 3D representation of the garden. The result of the 3D simulation, the 'heat map', is used later on when the user adds plants to the garden design: we can quickly look up the amount of sunlight at the precise spot the plant is placed. Apart from the heat map we also created a nice user-friendly interface which allows users to instantly see which parts of the garden receive the most sun in a selected season and hour of the day.

3D sunlight simulation

Side note: during the project we also looked into artificial light sources. If you are a developer you might be interested in our findings about Point light shadows in Three.js. Three.js is the javascript 3D library most widely used to create 3D web apps with WebGL.

Plant advice

Giving intelligent advice about which plants to use in your garden, and where best to place them, was arguably the most challenging part of the project. Especially since unwanted advice can very easily irritate a user a great deal.

Based on feedback from our focus groups and surveys we decided to adopt an approach inspired by knowledge-based systems. We created a modular system of multiple 'Advisors', all of which provide advice or warnings based on a simple rule. For instance, we created a GrowthAdvisor based on the simple rule that fast-growing plants should not be placed too close together. This way we could easily create many more advisors based on all sorts of simple rules.

Advice An early mockup of the advice UI

In addition to our conceptual and technical efforts we also put a lot of thought into UI approaches, i.e. how to best present the most relevant advice. We hope to come back to this in a future blog.

Conclusions

Creating a design application is not an easy task. Creating a design application in which users receive meaningful advice during the design process is, well, a major challenge. The grant gave us the opportunity to seriously engage these challenges. Looking back we are very happy with the results. The translation from data (i.e. plant properties as well as data generated from simulations) to guidance, by means of advisors, turned out to be both feasible and elegant.
Looking at other domains, we feel this approach is applicable to design apps and product configurators in a great variety of fields. Don't hesitate to contact us if you want to explore the possibilities. We look forward to working on a challenging project like this one in the future.

Thanks

We would like to thank all the participants of this project, in particular:

  • The province of Gelderland, for making the project possible
  • Teken je tuin, our partner for this project
  • All participants from the focus groups and test groups
  • The users of Draw Your Garden for providing us with useful feedback
Bram
Backend developer
data-engineering

Stay up-to-date

Stay up-to-date with our work and blog posts?

Related articles

While working on the Roc compiler, we regularly dive deep on computer science topics. A recurring theme is speed, both the runtime performance of the code that we generate, as well as the performance of our compiler itself.

One extremely useful technique that we have been playing with is data-oriented design: the idea that the actual data you have should guide how code is structured.

toolsdata-engineeringopen-sourcerust
Read article
December 9, 2022

Sorting with SIMD

Google recently published a blog article and paper introducing their SIMD-accelerated sorting algorithm.

SIMD stands for single instruction, multiple data. A single instruction is used to apply the same operation to multiple pieces of data. The prototypical example is addition, where one instruction can do e.g. 4 32-bit additions. A single SIMD addition should be roughly 4 times faster than performing 4 individual additions.

This kind of instruction-level parallelism has many applications in areas with a lot of number crunching, e.g. machine learning, physics simulations, and game engines. But how can this be used for sorting? Sorting does not involve arithmetic, and the whole idea of sorting is that each element moves to its unique correct place in the output. In other words, we don't want to perform the same work for each element, so at first sight it's hard to see where SIMD can help.

To understand the basic concepts, I played around with the ideas from the paper Fast Quicksort Implementation Using AVX Instructions by Shay Gueron and Vlad Krasnov. They provide an implementation in (surprisingly readable) assembly on their github. Let's see how we can make SIMD sort.

algorithmsdata-engineeringrust
Read article
Over the past months, we have worked with Scailable to optimize their neural network evaluation. Scailable runs neural networks on edge devices, taking a neural network specification and turning it into executable machine code.
algorithmsdata-engineeringembedded
Read article