In part 1 we discussed how we sort and store our LEGO collection at a high level. Now we’ll get into the nitty gritty of why we built an app to help and the thought process to land on the solution. It’s a combo of lessons learnt, AI and Machine Learning info and much more.
The Challenge
Even after organising into the 4 main types of storage containers and printing the brick labels for each container, it can be challenging to find the right brick. You can remember the location of the commonly used bricks. The others though can be time consuming.
Solution #1 – document on the label sheet
Our first solution was to print out the LEGO Brick Labels Contact Sheet. This easily allowed us to note down which container kept each brick on the ~30 pages of parts. We can quickly shuffle through the sheets, find the reference and then immediately know where the brick is. Each brick has a reference with two parts – The Shelf and the Container.
Let’s review the 1×2 Grille Brick to explain. The first number (18) is the shelf number. The second number shows which container (W31). With this info, we can immediately look for shelf 18 and then for container W31 (W is for Wall main category). Some of the other bricks shown in the sample above don’t have a shelf number as I know the shelf is always the same for the same container. E.g. the W31 container for the Grille is the same W31 container on Shelf 18 shown in the example above.
Initially we utilised a fairly simple naming convention that sped up finding the right container. The first letter of the Brick Architect main Label was the first letter (e.g. W for Wall) followed by a number relative to the size of the container.
1 – 30: Bunnings small 12 compartment container
31 – 50: Bunnings medium 8 compartment container
51 – 80: Ikea 5L Samla Tub
81 – 110: Ikea 11L Samla Tub.
Clear as mud right?
A Faster way to search?
The approach above works pretty well. But there are a few challenges:
A) What happens if the brick is not included in the Brick Architects LEGO Brick Labels?
B) What do you do if your eyes have glazed over after looking at 30 pages of parts?
C) What can you do if you need to change the container number or the shelf?
The next logical step was to build a basic summary of the Brick Labels in excel. It’s rough and ready but allows several additional notes to be added. Then instead of just using your eyes, you can quickly search through the excel to find the brick and thus the shelf and container location.
The sample above shows the approach taken. The nature of the Brick Architect Labels was such that there was no particularly easy way to transcribe from the sheet to excel. For that, I had to revert to the old fashioned method. Outsourcing it to my daughter. The $10/hour was money very well spent.
Once it was in excel, we’d stepped up the process a notch. It was much faster to search if you knew the brick number or name or even category. We could also add new bricks that weren’t in the Brick Architects list. Getting the brick part number is quite easy if you’ve got great eyes and/or lots of light. Fortunately LEGO prints the ID on every brick. It’s often very small though so can be hard when building at night.
Surely AI can help?
With all the buzz of AI, surely our computer overlords can help with the process of searching? I’m glad you asked, because they certainly can. Over the past decade, computer vision tasks have come forward in leaps and bounds. The detection of everyday objects has improved dramatically. LEGO bricks present quite a unique challenge though.
There are so many LEGO bricks that are quite similar to each other and depending on the angle, can be identical. Imagine a 2 x 4 brick and a 2 x 2 brick. Viewed from the front they are exactly identical (from a computers P.O.V.). As a write this, I can see I’m going to delve into a 3000 word essay on machine learning and computer vision, so I’ll cut to the chase. But if you want to read more, then this article provides excellent technical info.
In 2022, Piotr Rybak released the Brickognize AI model which had been trained on synthetic data. It was a revolution as without synthetic data, the process to get a model working effectively with all LEGO bricks is extremely difficult. You would need to capture real images of LEGO bricks from many angles. This might be 200+ images for each brick type. The more images, the better the training (generally speaking). As you can imagine, this would be very challenging. However with the use of synthetic data you can use “virtual” bricks to train the model. This approach lets you dramatically speed up the creation of a comprehensive data set. Think a 1000x or more speed reduction. The model accuracy is also significantly improved. This Youtube presentation by Piotr provides excellent insight.
Not only did he create this utterly fantastic system, he released it for free for people to use. You can head there right now and test it out.
Can we combine the two systems?
We now had two systems in play.
1) The Excel sheet which had all our information, and;
2) a system to work out what sort of brick it is if we were unsure.
Using these two systems manually allowed us to work out the location of most bricks. It was a little clunky however as you’d need to scan the brick, then open excel and search for the piece in the sheet.
With a couple of days off work I thought I’d see how hard it is to build an app. There’s quite a few out there already, but none that really worked the way I wanted. We had tried Instabrick in the past which is great if you want to run a bricklink store. Or if you care about each individual colours. But manually adding in the shelf and container location for 36 different coloured 2 x 4 Bricks was not feasible.
Building the app – LEGO SEARCHER 3000
After a few hours of searching, I quickly realised that building an app entirely from scratch is probably beyond my capability given the time I’m prepared to invest. However, there are a lot of online tools that allow you to code with many pre-built modules. It’s like coding but on steroids. After a bit of trial and error I landed on Retool.com. It has a great free version that meets my needs.
Using this solution, you can drag and drop many elements and start pulling together the basics of your app. Like any “good” software project, I introduced feature creep quite consistently from the initial idea!
In our case, I wanted three types of search functions.
1) The ability to enter the brick ID or description and search on that basis.
2) The ability to search via category and show all bricks
3) The ability take a photo of an image and have Brickognize suggest what pieces it might be.
In each of the options, you can click the brick and the app will show you the shelf, container and what the tub looks like. As we developed the app, I quickly realised it would be great to update the reference sheet with new bricks found by Brickognize that aren’t already in the database. So this was added to to the development queue.
The coding process
I’ve a little experience coding with Python but not much Javascript. The basics of Retool are pretty self explanatory but once you want a bit more functionality, you need to get into Javascript. Retool have a great support community. But there comes a time when you need to admit your coding skills aren’t up to scratch. At this fork in the road, you can either buckle down to start learning, or you can call in the big guns.
In this case, the big guns came to my rescue in the form of outsourcing the complex coding. I used the UpWork platform. It was utterly amazing. With a basic scope of work, I hired Haseeb to help with my Retool requirements. Over the course of a week we progressed the scope and created an app that is perfect for our needs. Ultimately it was around $100 AUD with me doing ~70% of the basic work and Haseeb doing the Javascript, API and coding.
I’ve copied his details below as he was fantastic to work with. Communicated well, understood the intent and delivered expertly. Upwork doesn’t seem to let me link directly to his profile, but if you’re interested, definitely search for him on the site.
The finished app
I’m ecstatic with how the app turned out in the end. The short video below shows it in action. It works really well for our workflow and how we sort and store our LEGO.