Due to the current circumstances, I had to record the lectures on augmented reality, which I am typically holding live. This was much more work than anticipated.. On the other hand, this means that I can make them available via Youtube now.
So, if you ever wanted to learn about the basic algorithms behind Augmented Reality, now is your chance.
The lecture is structured in two parts
Camera Calibration, and
Object Pose Estimation.
You can click on the TOC links below to directly jump to a specific topic.
I am quite frustrated with corona graphs in the news, since most reporters seem to have skipped math classes back then. For instance, just plotting the number of confirmed infections at the respective dates does not tell you anything due to the different time point of outbreak. So lets see whether I can do better:
Typically you would need to feed it into OpenGL to get an image out of it. However, there are occasions when setting up OpenGL would be too much hassle or when you deliberately want to render on the CPU.
In this case we can use the OpenCV to do the rendering in two function calls as:
See the documentation of cv2.projectPoints for the meaning of the parameters.
Note how we only project each vertex once and only apply the mesh topology afterwards. Here, we just use the numpy advanced indexing as pts2d[indices] to perform the expansion.
This is pretty fast too. The code above only takes about 9ms on my machine.
In case you want filled polygons, this is pretty easy as well
for face in indices:
cv2.fillConvexPoly(img, pts2d[face], (64, 64, 192))
Filled polygon rendering
However, as we need to a python loop in this case and also have quite some overdraw, it is considerable slower at 20ms.
Of course you can also combine both to get an image like in the post title.
From here on you can continue to go crazy and compute face normals to do culling and shading.
But you can only get that far without a depth buffer
So after having made fun of people for “wearing toothbrushes”, I finally came to buy such headphones for myself.
Having used non-true wireless Bluetooth headphones before I was curious what the usability advantage would feel like.
Here I went for the Xiaomi AirDots Pro 2 aka True Wireless Earphones 2S which I could grab for 399 Yuan which is about 51€, which seems like the right price-point for this kind of accessoire.
Keep in mind that the built-in battery only survives so many charging cycles and once it dies you can throw them away.
The Airdots (right) compared to the Airpods (middle) and the Oral-B Precision Clean
The initial feeling of using true wireless headphones is surprisingly relieving – there is simply no cord to untangle or to be aware of while wearing. This is especially true during phone calls, where one needs to keep the microphone aligned.
The downside is that the headphones are too small to accommodate any buttons for volume and playback control.
The Air 2 kind of make up for it by automatically connecting to your phone once you put them on and by automatically pausing the music when you put one out of the ear. This is achieved by a built-in brightness sensor.
Furthermore you have double-tap actions, which default to play/ pause on the right headphone and launching the voice assistant (e.g. Google Assistant) on the left headphone.
The battery life is stated with 4 hours per-charge with 2 extra charges in the case. I could confirm those on a long distance flight.
Compared to the Airpods 2
Looking at the feature-list above or simply at the images, the similarity to the Apple Airpods is apparent.
Out of curiosity I borrowed some from friend for comparison. The most important point is probably sound quality. Here we found the two virtually in-distinguishable. But keep in mind that we only did a quick test and did not use them extensively.
The second point is likely the form. Here, both earphones have the same ear-part and only differ by the shaft. So if one fits your ear, so should the other.
The shaft however is considerably wider on the Airdots. This is less apparent when viewed from the side as the thickness is similar.
For me, the more important difference is being able to control the headphones from my Android smartphone. This is currently not possible with the Airpods, while there is some way for the Airdots;
Companion app & Software integration
To control the earphones, you have to sideload the Xiao Ai Lite App. The main purpose of it is to provide the Xiaomi voice assistant and the Air2 options likely just ended up there instead of Xiaomi Home as they offer an always-on assistant integration just like the Airpods.
It handles firmware updates and allows you to configure the douple-tap action per earphone as well as displaying the charging status of the earphones and the case. By default android will only display the charging status of the least charged earphone.
Furthermore, you can use the fast-paring if the app is running. Here, it is sufficient to hold the earphone case close to the phone and just open it. The app will ask for confirmation. This is only slightly more convenient then holding the pairing button and using the normal bluetooth pairing procedure.
The downside is that the app is currently only available in Chinese and consequently the voice assistant only works with Chinese.
I tried out some voice commands via google translate and everything works as it should. However if you are not fluent in chinese it is far from practical. Most people should disable the assistant in the settings to avoid accidentally triggering it.
A serious advantage of Xiomi/ Huawei phones is the availability of the LHDC Bluetooth Codec which offers a superior bandwidth and latency. While I am fine with the bandwidth provided by AAC when listening to music, there is still a noticeable and annoying delay when watching videos and playing games.
Noise Shielding
The firmware upgrade to v2.6.9.0 significantly improved the acoustic pattern of the headphones by tuning up the low bands (base) and thus general sound quality.
This results in a very noticeable noise reduction compared to v2.6.2.0 – especially in the lower frequencies; things like your footsteps get filtered out. Higher frequencies like car motor sounds are still perceivable though. This is however a good compromise for me.
Addendum: previously I stated that the headphones provide active noise cancellation. They do not. I was mistaken by the significantly improved noise shielding through the tuned up bases.
Compared to True Wireless Earphones 2 Basic / Air 2 SE (Addendum)
Ultimately my left earphone died (see below) – likely due to a deep discharge. The second pair of the Air 2 still works though. I will update this review when I find out whether I just had bad luck or whether there is a systematic issue.
Anyway.. as a replacement I ordered myself the Air2 SE earphones, given my phone does not support LHDC and other online reviews described them as “the Air2 without LHDC”.
Well, I guess some sites try to sell an unboxing story as a review..
Air2 SE – Air 2
As you can see on the image above the Air 2 SE are considerably longer than the normal Air 2 and also have different vents.
While they are slightly too long now for my taste, my largest complaint is the sound quality. Basically, the bases which improved sound quality and shielding on the Air2 after the firmware update are missing here. So we have no shielding and a noticeably worse sound quality here. Next, the microphones on the Air 2 SE are noticeable less sensitive and I had to raise my voice from normal level so my callees could here me. Finally, the Air2 SE are not supported by the XiaoAI app, so you get no firmware updates and no configurable tap-actions.
Given the small pricing difference between the Air 2 and the Air 2 SE, I therefore recommend to always go for the former.
Also, if you encounter some sites that tell you the Air2 and Air2 SE are the same, please enable your adblocker. Especially if they jabber about different touch zones, even though the marketing material makes it quite clear that up/ middle/ down means you can touch the single zone at any position.
Charging issues
Over time I noticed that the left earphone consistently runs out of battery before the right one. The issue seems to be that it discharges while stored inside the case. Putting it in and out resolves the issue – but only until it is fully charged. This makes keeping the earphones pre-charged and ready quite an issue.
Hopefully this can be addressed with a future firmware update. (reproduced with firmwares up to v2.7.1.0)
Addendum: the replacement pair (see above) do not have any charging issues. Therefore I guess the issue is just a singular manufacturing defect.
This means that calibration is no longer restricted to a Linux PC – you can also calibrate cameras attached to Windows/ OSX and even mobile phones. Furthermore you will not have to calibrate at all if your device is already known to the service. The underlying algorithm ensures that the obtained calibrations are reliable and thus can be shared between devices of the same series.
Aggregating calibrations while providing on-the-fly calibrations for unknown devices form the calibDB web-service.
In the future we will make our REST API public so you can transparently retrieve calibrations for use with your computer vision algorithms. This will make them accessible to a variety of devices – without you having to worry about the calibration data.
When using Nextcloud it makes some sense to host it yourself at home to get the maximum benefit of having your own cloud.
If you would use a virtual private server or shared hosting, your data would still be exposed to a third party and the storage would be limited as you would have to rent it.
When setting up a server at home one is tempted to use a Raspberry Pi or similar ARM based device. Those are quite cheap and only consume little power. Especially the latter property is important as the machine will run 24/7.
I was as well tempted and started my self-hosting experience with an ARM based boards, so here are my experiences.
Do not use a Raspberry Pi for hosting
Actually this is true for any ARM based board. As for the Pi itself, only the most recent Pi 4B has a decent enough CPU and enough RAM to handle multiple PHP request (WebCAL, Contacts, WebDAV) from different clients without slowdown. Also only with the Pi 4B you can properly attach storage over USB3.0 – previously your transfer rates would be limited by the USB2.0 bus.
One might argue that other ARM based computers are better suited. Indeed you could get the decently equipped Odroid U3, long before the Pi 4B was available. However, non-pi boards have their own set of problems. Typically, they are based on an Smartphone design (e.g. the Odroid U3 essentialy is a Galaxy Note 2).
This makes them plagued by the Android update issues, as these boards require a custom kernel, that includes some of the board specific patches which means you cannot just grab an Ubuntu ARM build. Instead you have to wait for a special image from the vendor – and just as with Android, at some point, there will be no more updates.
Furthermore ARM boards are actually not that cheap. While the Pi board itself is indeed not expensive at ~60€, you have to add power-supply housing and storage.
Intel NUC devices are a great choice
While everyone was looking at cheap and efficient ARM based boards, Intel has released some great NUC competitors. Those went largely unnoticed as typically only the high-end NUCs get news coverage. It is more impressive to report how much power one can cram into a small form-factor.
However one can obviously also put only little power in there. More precisely, Intels tablet celeron chips that range around 4-6W TDP and thus compete with ARM boards power-wise. (Still they are an order of magnitude faster then a Raspberry Pi)
Device
Power (Idle)
Power (load)
Odroid U3
3.7 W
9 W
GB-BPCE-3350C
4.5 W
9.6 W
Here, you get the advantages of the mature x86 platform, namely interchangeable RAM, interchangeable WiFi modules, SATA & m2 SSD ports and notably upstream Linux compatibilty (and Windows for that matter).
As you might have guessed by the hardware choice above, I made the switch already some time ago. On the one hand you only get reports for the by now outdated N3350 CPU – but on on the other hand it makes this a long term evaluation.
Regarding the specific NUC model, I went with the Gigabyte GB-BPCE-3350C, which are less expensive (currently priced around 90€) than the Intel models.
Consequently the C probably stands for “cheap” as it lacks a second SO-DIMM slot and a SD-card reader. However it is fan-less and thus perfectly fine for hosting.
So after 2 Years of usage and a successful upgrade between two Ubuntu LTS releases, I can report that switching to the x86 platform was worth it.
If anything I would probably choose a NUC model that also supports M.2/ M-Key in addition to SATA to build a software RAID-1.
The Lenovo D330 2-in-1 convertible (or netbook as we used to say) is a quite interesting device. It is based on Intels current low-power core platform, Gemini Lake (GLK), and thus offers great battery-life and a fan-less design.
This similar to what you would from an ARM based tablet. However being x86 based and Windows focused we can expect to get Ubuntu Linux running – without requiring any out-of-tree drivers or custom kernels that never get updated as we are used-to from the ARM world. This post will be about my experiences on doing so.
For this I will use the most recent Ubuntu 19.04 release as it contains fractional scaling support, which is essential for a 10″ 1920x1200px device. Also the orientation sensor (mostly) works out of the box, when compared to the 18.04 LTS release.
Camera calibration is an essential step for many computer vision applications as well as computational photography.
If you want to know how far something is away from your camera or even get an undistorted image out of it, it must be calibrated.
Unfortunately existing calibration utilities are cumbersome to use and often not available on linux.
Therefore I am happy to announce that the app from my publication
Efficient Pose Selection for Interactive Camera Calibration
is now available in the snap store:
It will guide you through the calibration process by displaying a series of target overlays like you see in the cover image. For details on the pose selection, see the above paper.
Additionally the source code is available for you to inspect on github:
In case you want to employ Blender for Computer Vision like e.g. for generating synthetic data, you will need to map the parameters of a calibrated camera to Blender as well as mapping the blender camera parameters to the ones of a calibrated camera.
But if we look at the Blender Camera, we find lots non-standard and duplicate parameters with random or without any units, like
unitless shift_x
duplicate angle, angle_x, angle_y, lens
Doing someresearch on their meaning and fixing various bugs in the proposed conversion formula, I could however come up with the following python code to do the conversion from blender to OpenCV
# get the relevant data
cam = bpy.data.objects["cameraName"].data
scene = bpy.context.scene
# assume image is not scaled
assert scene.render.resolution_percentage == 100
# assume angles describe the horizontal field of view
assert cam.sensor_fit != 'VERTICAL'
f_in_mm = cam.lens
sensor_width_in_mm = cam.sensor_width
w = scene.render.resolution_x
h = scene.render.resolution_y
pixel_aspect = scene.render.pixel_aspect_y / scene.render.pixel_aspect_x
f_x = f_in_mm / sensor_width_in_mm * w
f_y = f_x * pixel_aspect
# yes, shift_x is inverted. WTF blender?
c_x = w * (0.5 - cam.shift_x)
# and shift_y is still a percentage of width..
c_y = h * 0.5 + w * cam.shift_y
K = [[f_x, 0, c_x],
[0, f_y, c_y],
[0, 0, 1]]
So to summarize the above code
Note that f_x/ f_y encodes the pixel aspect ratio and not the image aspect ratio w/ h.
Blender enforces identical sensor and image aspect ratio. Therefore we do not have to consider it explicitly. Non square pixels are instead handled via pixel_aspect_x/ pixel_aspect_y.
We left out the skew factor s (non rectangular pixels) because neither OpenCV nor Blender support it.
Blender allows us to scale the output, resulting in a different resolution, but this can be easily handled post-projection. So we explicitly do not handle that.
Blender has the peculiarity of converting the focal length to either horizontal or vertical field of view (sensor_fit). Going the vertical branch is left as an exercise to the reader.
The reverse transform can now be derived trivially as
One major grief for me when surfing on Android are ads. They not only increase page size and loading time, but also take away precious screen estate.
Unfortunately the native Android browser, which nowadays is Chrome, does not support extensions and hence there is no ad-blocker.
Therefore I was quite optimistic when Google announced they will be enforcing the betterads standards with Chrome – aka ad-blocking light.
However after having used Chrome only showing “betterads”, I must say that they are far away from what is tolerable to me. I am more in line with the Acceptable Ads criteria. (My site also keeps to them – if you choose to disable ad-blocking here)
As someone who has to pay for hosting I fully understand that Ads are part of the game – but lets face it; as long as annoying ads get you more money, there will be annoying ads. Ad-blockers are a very effective way to let money speak here..
So I needed an adblock-capable browser on Android. Fortunately Mozilla greatly improved Firefox performance with their Quantum incentive. Or maybe modern Smartphones just got a lot faster. Anyway.. a recent Firefox virtually performs the same as Chrome on Android and thus is a viable alternative.
As of recently there is also Microsoft Edge for Android, but actually it does not gain an edge over anything. So lets stick with open source software.
With switching to Firefox on Android one should switch to Firefox on Desktop as well, so you get sync across devices.
On Linux
Unfortunately Firefox has bad default settings on Linux. For one – unlike Chrome – it does not use client side decorations by default, and thus wastes space in the title bar. But this is easy to fix.
Then it still uses the slow software rendering path. To make it use the GPU, visit about:config and set the following properties to true
layers.acceleration.force-enabled enable OpenGL based compositing which for smooth scrolling. (enabled by default on OSX, Windows)
layers.omtp.enabled (OMTP) further improve performance when scrolling. (enabled by default on OSX, Windows)
gfx.webrender.all enable the new Servo based compositing which for smooth scrolling. (enabled by default on OSX, Windows)
Note that, there is also gfx.webrender.all to enable the new Servo based compositor which is significantly faster (250%). However, it also does have significant issues with Nvidia proprietary drivers, so better leave it off for now.
Additionally, if you use a touch-pad or touch-screen, you should add the following environment variable:
echo "export MOZ_USE_XINPUT2=1" >> ~/.profile
this will make Firefox correctly handle touch events instead of translating them to mouse wheel scrolling. This way you get pixel perfect scrolling on touch-pads and it is a prerequisite for drag to scroll on touch-screens.
On Android
On Android Firefox generally has sane defaults. The only setting missing here to bring it on par with Chrome are the Encrypted Media Extensions. For this again visit about:config and create the following property and set it to true
media.eme.enabled
Still you will need some time to adapt to Firefox; e.g. there is no pull to refresh. However there are other bonus points besides adblocking; for me the synchronized tabs sidebar (on desktop) has proven to be an invaluable usability improvement.
