Lately we have been dipping our toes into the world of Photogrammetry. In particular, the Penshurst Place and Gardens project involved scanning sculptures and artefacts for inclusion in the app. More on this in the case study here.
Let's take a look at the process of scanning objects using Photogrammetry, what it is, how it works, and what to be aware of when planning to use this technique with your project.
In simple terms, Photogrammetry is the process of analysing photos of an object from many angles and then using that data to construct the object as a 3D model. The photos are then used as source material for the texture of the object.
There are many tools available for converting photos into 3D models. A very popular desktop solution is Reality Capture (now owned by Epic Games). Apple also have a cloud-based solution called Object Capture. The release of this API caused a flood of apps offering a convenient portable tool for anyone to create 3D models using their iPhone. Our favourite is Polycam which has a feature for using LiDAR as well as Photogrammetry. They also have an Android version. Polycam has a feature that allows you to take photos automatically as you move around the subject. This is convenient as it means you don't have to keep tapping the button.
Photogrammetry is not perfect and there are certainly things it is not suitable for. Organic objects are great subjects as long as they are still. If it moves around and you can't take all of your photos simultaneously then it will not be able to calculate the structure of the model and you'll end up with something that looks very wrong. I have tried to create a 3D model of myself and no matter how still I thought I was, the results told a very different story.
The other big issue with scanning is lighting. If it is bright and there are lots of shadows, they will be baked into the textures. Ideally you want to scan the subject towards dusk when the sun is low in the sky and the light is equally distributed across the object. Inconsistent light around an object will confuse the algorithm gods and in some cases result in an object that has completely lost its structure.
As well as lighting, you need to be aware of reflections, and transparency. I have found that very shiny or metallic objects (if they scan at all) can look dirty as the reflective surface just gets baked into the texture. Photogrammetry cannot make decisions about which part of a model is metallic, or which surfaces as emissive; it just creates a single mesh and wraps it with an image.
You should also choose subjects that are solid and without too many holes and crevices. If there's an object with a deep hole it will usually just fill it and you'll need to do some cleaning up afterwards. Thin structures like a wire fence or cables may end up being split or not as uniform as you'd like. Again, you may need to go in and manually create some meshes to add to the model. In the Penshurst Place app there was a Porcupine sculpture that didn't come out that well as many of the spines were disconnected and floating. We had to clean it up manually but wasn't great.
The Archer sculpture was also tricky as it is made up of twisted thick wires. Most of the gaps between the wires are filled in and this would have taken days to clean up properly.
When you have the environment right, and you have chosen the right subject you then need to start taking the photos. Polycam recommend that you make sure you leave an overlap between the last photo taken and the current one. Generally, you should find a spot (I like to try and start at the top of the subject) and then work in a spiral around the object. If this was a sculpture of a person for example, you'd start at the top of their head and work around and down to the bottom of their feet. You would have to take extra photos and adjust the path slightly to make sure you get under the chin, or under the armpits. If you can't get under the object you may end up with a hole, or it may just fill it in as best it can. Take as many photos as you can (this is why Polycam is great as it doesn't fill your Photo library up) and take as much time as you need to get all the details. There is a photo limit on Polycam though.
Here's an example of great results when you take your time and be very careful about capturing all of the details. This was captured at the Tate Modern and I spent about 20 minutes taking the photos, much to the bewilderment of the other visitors.
Your scan is complete and looks great ... but wait, there are a few areas that aren't quite right. You may need to get rid of the floor, or fill a hole. Blender is a great free 3D editing tool and, although this post couldn't even begin to explain how it works, there are plenty of free resources to teach you the basics. Most of the time, to clean up a model you just need to learn how to edit the vertices of the model. You will likely need to remove a lot of them, and others may need to be repositioned and sewn together to fill some gaps. Blender will do as best a job as it can to stitch the textures together for you, but in some cases you may also need to go and paint some of it in with a clone tool.
In the case of this Penshurst Place model, I needed to use it for Augmented Reality, which meant it had to be optimised and as low-poly as possible. This means the mesh needs to be reduced in complexity, and as a consequence, the detail of the object lost a bit. This is an unfortunate compromise you have to make if you want users to be able to view and interact with your model on a mobile device. Performance is critical. The best way to deal with this is to do it manually and choose which areas of the model should be simplified and which areas need to stay complex to keep the detail intact. But this requires expert knowledge of 3D editing software and in most cases it is just quicker and easier to let Blender decimate the model for you. The Decimate modifier in Blender will reduce the complexity of the model globally and doesn't care which areas should be left intact. Here's an example of decimation in action.
After cleaning up, your model is ready to be shared with the world. There are many services online for hosting your model such as Sketchfab and MeshMorph, and you can also use your model in games, or for Augmented Reality. Feel free to try the clock model in AR from the MeshMorph embed below.
Photogrammetry is a great tool for quickly capturing objects to use as props for a game or (as in the case for Penshurst Place) to give users a virtual object they can view in Augmented Reality. It might not be the best choice if you need to have a model with moving parts or interactive elements, although you could use it to scan the base static model and then add features to it with 3D modelling software. if you are interested in using Photogrammetry for your project and would like some advice, please get in touch with us.
