Beyond the Raspberry Pi for Nextcloud hosting

When using Nextcloud it makes some sense to host it yourself at home to get the maximum benefit of having your own cloud.

If you would use a virtual private server or shared hosting, your data would still be exposed to a third party and the storage would be limited as you would have to rent it.

When setting up a server at home one is tempted to use a Raspberry Pi or similar ARM based device. Those are quite cheap and only consume little power. Especially the latter property is important as the machine will run 24/7.

I was as well tempted and started my self-hosting experience with an ARM based boards, so here are my experiences.

Do not use a Raspberry Pi for hosting

Actually this is true for any ARM based board. As for the Pi itself, only the most recent Pi 4B has a decent enough CPU and enough RAM to handle multiple PHP request (WebCAL, Contacts, WebDAV) from different clients without slowdown.
Also only with the Pi 4B you can properly attach storage over USB3.0 – previously your transfer rates would be limited by the USB2.0 bus.

One might argue that other ARM based computers are better suited. Indeed you could get the decently equipped Odroid U3, long before the Pi 4B was available.
However, non-pi boards have their own set of problems. Typically, they are based on an Smartphone design (e.g. the Odroid U3 essentialy is a Galaxy Note 2).

This makes them plagued by the Android update issues, as these boards require a custom kernel, that includes some of the board specific patches which means you cannot just grab an Ubuntu ARM build.
Instead you have to wait for a special image from the vendor – and just as with Android, at some point, there will be no more updates.

Furthermore ARM boards are actually not that cheap. While the Pi board itself is indeed not expensive at ~60€, you have to add power-supply housing and storage.

Intel NUC devices are a great choice

While everyone was looking at cheap and efficient ARM based boards, Intel has released some great NUC competitors.
Those went largely unnoticed as typically only the high-end NUCs get news coverage. It is more impressive to report how much power one can cram into a small form-factor.

However one can obviously also put only little power in there. More precisely, Intels tablet celeron chips that range around 4-6W TDP and thus compete with ARM boards power-wise. (Still they are an order of magnitude faster then a Raspberry Pi)

DevicePower (Idle)Power (load)
Odroid U33.7 W9 W
GB-BPCE-3350C4.5 W9.6 W

Here, you get the advantages of the mature x86 platform, namely interchangeable RAM, interchangeable WiFi modules, SATA & m2 SSD ports and notably upstream Linux compatibilty (and Windows for that matter).

As you might have guessed by the hardware choice above, I made the switch already some time ago. On the one hand you only get reports for the by now outdated N3350 CPU – but on on the other hand it makes this a long term evaluation.

Regarding the specific NUC model, I went with the Gigabyte GB-BPCE-3350C, which are less expensive (currently priced around 90€) than the Intel models.

Consequently the C probably stands for “cheap” as it lacks a second SO-DIMM slot and a SD-card reader. However it is fan-less and thus perfectly fine for hosting.

So after 2 Years of usage and a successful upgrade between two Ubuntu LTS releases, I can report that switching to the x86 platform was worth it.

If anything I would probably choose a NUC model that also supports M.2/ M-Key in addition to SATA to build a software RAID-1.

Ubuntu on the Lenovo D330

The Lenovo D330 2-in-1 convertible (or netbook as we used to say) is a quite interesting device. It is based on Intels current low-power core platform, Gemini Lake (GLK), and thus offers great battery-life and a fan-less design.

This similar to what you would from an ARM based tablet. However being x86 based and Windows focused we can expect to get Ubuntu Linux running – without requiring any out-of-tree drivers or custom kernels that never get updated as we are used-to from the ARM world.
This post will be about my experiences on doing so.

For this I will use the most recent Ubuntu 19.04 release as it contains fractional scaling support, which is essential for a 10″ 1920x1200px device. Also the orientation sensor (mostly) works out of the box, when compared to the 18.04 LTS release.

Continue reading Ubuntu on the Lenovo D330

Do not use Meson

Recently the Meson Build System gained some momentum. It is time to stop that.
Not that Meson is a bad piece of software – on the contrary, it is quite well designed.
Still it makes building C/C++ applications worse, by (quoting xkcd) basically creating this:

It sets out to create a cross-platform, more readable and faster alternative to autotools. But there is already CMake that solves this.

You might say that CMake is ugly, but note that the CMake 2.x you might have tried is not the same CMake 3.x that is available today. Many patterns have improved and are now both more logical and more readable.

Nowadays the difference between Meson and CMake is just a matter of syntactic preference. The Meson authors seem to agree here.

The actual criterion for selecting a build system however should be tooling support and community spread. CMake easily wins here:

After the introduction of the server mode it got native support by QtCreator, CLion, Android Studio (NDK) and even Microsofts Visual Studio. Native means that you do not have to generate any intermediate project files, but the CMakeLists.txt is used directly by the IDE.

On the community spread side we got e.g. KDE, OpenCV, zlib, libpng, freetype and as of recently Boost. These projects using CMake not only guarantees that you can easily use them, but that you can also include them in your build via add_subdirectory such that they become part of your project. This is especially useful if you are cross-compiling – for instance to a Raspberry Pi.

On the other hand, reinventing a wheel that is tailored to the needs of a specific community (Gnome), means that it will fall behind and eventually die. This is what is currently happening to the Vala language that had a similar birth to Meson.

The meson devs might object that Meson generates build files that run faster on a Raspberry Pi. However if your cross compiling is working you do not need that. And honestly, that particular improvement could have been also achieved by providing a patch to the CMake Ninja generator..

Addendum 27-11-2018
Stumbled over another great guide to modern CMake.

Addendum 15-06-2018
A new guide for CMake called CGold can be found here, which is of comparable quality to the Meson docs.

Addendum 4-1-2018
Some comments (rightfully) note that Meson has generally a better documentation and avoids some of its pitfalls. However this is mostly due to Meson not being around long enough such that the way you do things in Meson changed. Neither did it see such a widespread use like CMake yet. (think of corner-cases)

But even if you argue that this is precisely the point why you should use Meson, I would argue that improving the existing documentation in CMake and adding more educational warnings is easier then writing something from scratch.

Addendum 29-02-2019
Part of the perceived superiority of Meson, was that it just was not in use for long enough to notice its flaws – contrary to long lasting legacy of CMake.
With its adaptation, things like only one global namespace start to get attention – things that are already solved in CMake..

Creating PyGTK app snaps with snapcraft

Snap is a new packaging format introduced by Ubuntu as an successor to dpkg aka debian package. It offers sandboxing and transactional updates and thus is a competitor to the flatpak format and resembles docker images.

As with every new technology the weakest point of working with snaps is the documentation. Your best bet so far is the snappy-playpen repository.

There are also some rough edges regarding desktop integration and python interoperability, so this is what the post will be about.

I will introduce some quircks that were needed to get teatime running, which is written in Python3 and uses Unity and GTK3 via GObject introspection.

The most important thing to be aware of is that snaps are similar to containers in that each snap has its own rootfs and only restricted access outside of it. This is basically what the sandboxing is about.
However a typical desktop application needs to know quite a lot about the outside world:

  • It must know which theme the user currently uses, and after that it also needs to access the theme files.
  • For saving anything it needs access to /home
  • If it should access the internet it needs system level access as well; like querying whether there actually is an active internet connection

To declare that we want to write to home, play back sound and use unity features we use the plugs keyword like

apps:
    teatime:
         # ...
         plugs: [unity7, home, pulseaudio]

However we must also tell our app to look for the according libraries inside its snap instead of the system paths. For this one must change quite a few environment variables manually. Fortunately Ubuntu provides wrapper scripts that take care of this for us. They are called desktop-launchers.

To use the launcher the configures the GTK3 environment we have to extend the teatime part like this:

apps:
    teatime:
        command: desktop-launch $SNAP/usr/share/teatime/teatime.py
        # ...
parts:
    teatime:
         # ...
         after: [desktop/gtk3]

The desktop-launch script takes care of telling PyGTK where the GI repository files are.

You can see the full snapcraft.yml here.

Update:

Before my fix, one had to use this rather lengthy startup command

env GI_TYPELIB_PATH=$SNAP/usr/lib/girepository-1.0:$SNAP/usr/lib/x86_64-linux-gnu/girepository-1.0 desktop-launch $SNAP/usr/share/teatime/teatime.py

which hard-coded the architecture.

End Update

After this teatime will start, but the paths still have to be fixed. Inside a snap “/” still refers to the system root, so all absolute paths must be prefixed with $SNAP.

Actually I think the design of flatpak is more elegant in this regard where “/” points to the local rootfs and one does not have to change absolute paths. To bring in the system parts flatpak uses bind mounts.

Conclusions

Once you get the hang of how snaps work, packaging becomes quite straightforward, however currently there are still some drawbacks

  • the snap package results in 120MB compared to a 12KB deb. This is actually a feature as shipping all the dependencies makes the snap installable on every linux distribution. However I hope that we can get this down by introducing shared frameworks (like GTK3, Python) that do not have to be included in the app package.
  • [Update: fixed] Due to another issue, your snap has only the C locale available and thus will not be localized.
  • [Update: fixed] Unity desktop notifications do not work. You will get a DBus exception at the corresponding call.
  • [Update: fixed] The shipped .desktop file is not hooked up with the system, so you can only launch the app via the command line.

Converting a Ubuntu and Windows dual-boot installation to UEFI

UEFI is the successor to BIOS for communicating with the Firmware on your Mainboard.
While the first BIOS was released with the IBM-PC in 1981, the first UEFI version (EFI 2.0) was released 25 years later in 2006 building upon the lessons learned in that timespan. So UEFI is without any doubt the more modern solution.

The user-visible advantages of using UEFI instead of BIOS are

You could reinstall both Windows and Ubuntu to get UEFI. However it is also possible to convert existing installations of both on the fly – without the backup/ restore cycle. You should still do a backup in case something goes wrong though.

Prerequisites

Only the 64bit Versions of Windows support UEFI. Therefore this guide assumes that you run the 64bit versions of both Windows and Ubuntu.

Furthermore verify the following items before you continue – otherwise you will not be able to finish the conversion. Use GParted in case you have not enough space before the first or after the last partition.

  • 250MB space in front of first partition
    Typically Windows 8 creates a 350MB System Partition upon installation. This space can be reclaimed for a 100MB EFI partiton and a new 100MB Windows System partition.
  • 1-2MB behind last partiton (for the GPT backup)
  • UEFI bootable Ubuntu USB drive.
    You can use the startup disk creator on ubuntu with an Ubuntu 14.04+ ISO.
  • UEFI bootable Windows USB drive.
    You can use the Microsoft Media Creation tool for Windows 10 to get one.

to test that the sticks are indeed UEFI compatible, try booting them with CSM Mode disabled in your BIOS.

Convert the drive to GPT

UEFI requires a GUID Partition Table (GPT), so first we need to convert from MBR to GPT.

After this step you will not be able to boot your system any more. So make sure you have the Ubuntu USB drive ready.

We will use gdisk to perform the conversion as following:

sudo gdisk /dev/sdX
Command (? for help): w

where sdX is your system drive (e.g. sda)

Convert Windows to UEFI

Now boot your Windows USB drive and enter the command prompt as described in this Microsoft Technet article at step 6.

Continue with the following steps from the Article. Note that we have skipped steps 1-4 as we used Ubuntu to convert the disk to GPT.

We have now created a EFI partition, a new EFI compatible Windows System Partition and we have installed the Windows Bootloader to the EFI partition. Your Windows installation should now start again.
At this point you could also perform an upgrade to Windows 10, as the upgrade would erase grub from the EFI partition anyway.

Next we are going to install grub to the EFI partition and make it manage the boot.

Enter a Ubuntu chroot

As we can not directly boot our Ubuntu installation, we will instead boot from the Ubuntu USB drive and the switch to the installed Ubuntu.
To do the switch we have to setup and enter a chroot as following

sudo mount /dev/sdXY /mnt 
sudo mount /dev/sdX1 /mnt/boot/efi 
sudo mount -o bind /dev /mnt/dev
sudo mount -o bind /sys /mnt/sys
sudo mount -t proc /proc /mnt/proc
sudo cp /proc/mounts /mnt/etc/mtab
sudo cp /etc/resolv.conf /mnt/etc/resolv.conf
sudo chroot /mnt

where sdXY is the partition where your Ubuntu system is installed (e.g. sda5)

Convert Ubuntu to UEFI

Inside your Ubuntu Installation we have to replace grub for BIOS (aka grub-pc) with grub for UEFI (aka grub-efi) as:

sudo apt-get --reinstall install grub-common grub-efi-amd64 os-prober

this would be enough to get the system booting again, however we also aim for secure boot so we also need to install the following:

sudo apt-get install shim-signed grub-efi-amd64-signed linux-signed-generic

This installs signatures for grub and the kernel which are used to verify the integrity of these at boot. Furthermore we install shim, which is a passthrough bootloader that translates from the Microsoft signatures on you mainboard to the signatures by Canonical used to sign grub and the kernel (see this for details).

Next we finally install grub to the EFI partition by:

sudo grub-install --uefi-secure-boot /dev/sdX
sudo update-grub

where sdX is again your system drive (e.g. sda).

Now you can enable secure boot in your BIOS and benefit. Note that some BIOS implementations additionaly require you to select the trusted signatures. Look out for an option called “Install Default Secure Boot keys” or similar to select the Microsoft signatures.

Updating Crucial MX100 Firmware with Ubuntu

There has been a Firmware update for the Crucial MX100 to MU02. In case you are running Ubuntu there is an easy way to perform the update without using a CD or USB Stick.

As the firmware comes in form of an iso image containing Tiny Core Linux, we can instruct grub2 to directly boot from it. Here is how:

  1. append the following to /etc/grub.d/40_custom:
    menuentry "MX100 FW Update" {
     set isofile="/home/<USERNAME>/Downloads/MX100_MU02_BOOTABLE_ALL_CAP.iso"
     # assuming your home is on /dev/sda3 ATTENTION: change this so it matches your setup
     loopback loop (hd0,msdos3)$isofile
     linux (loop)/boot/vmlinuz libata.allow_tpm=1 quiet base loglevel=3 waitusb=10 superuser rssd-fw-update rssd-fwdir=/opt/firmware rssd-model=MX100
     initrd (loop)/boot/core.gz
    }

    read this for details of the file format.

  2. run sudo update-grub
  3. reboot and select “MX100 FW Update”
  4. Now you can delete the menuentry created in step1

Note that this actually much “cleaner” than using windows where you have to download 150MB of the Crucial Store Executive Software which actually is a local webserver written in Java (urgh!). But all it can do is display some SMART monitoring information and automatically perform the above steps on windows.

Header Image CC-by MiNe

Introducing Sensors Unity

Sensors-Unity is a new lm-sensors GUI for the Unity Desktop. It allows monitoring the output of the sensors CLI utility while integrating with the Unity desktop. This means there is no GPU/ HDD support and no plotting.
If you need those you are probably better suited with psensor. However if you just need a overview of the sensor readings and if you appreciate a clean UI you should give it a shot.

It is written in Python3 / GTK3 and uses sensors.py. You can contribute code or help translating via launchpad.

Overview

In contrast to other applications the interface is designed around being a application. Instead of getting another indicator in the top-right, you get an icon in the launcher:

The user interface
The user interface

The idea is that you do not need the sensor information all the time. Instead you launch the app when you do. If you want to passively monitor some value you can minimize the app while selecting the value to display in the launcher icon.

To get the data libsensors is used which means that you need to get lm-sensors running before you will see anything.

However once the sensors command line utility works you will see the same results in Sensors-Unity as it shares the configuration in /etc/sensors3.conf.

Configuration

Unfortunately configuring lm-sensors via /etc/sensors3.conf is quite poorly documented, so lets quickly recap the usage.

  • /etc/sensors3.conf contains the configuration for all sensors known by lm-sensors
  • however every mainboard can use each chip in a slightly different way
  • therefore you can override /etc/sensors3.conf by placing your specific configuration in /etc/sensors.d/ (see this for details)
  • you can find a list of these board specific configurations in the lm-sensors repository
  • to disable a sensor use the ignore statement
    #ignore everything from this chip
    chip "acpitz-virtual-0"
       ignore temp1
       ignore temp2 
    
  • to change the label use the label statement
    chip "coretemp-*"
       label temp1 "CPU Package"
    

Sensors-Unity Specific Configuration

Sensors-Unity allows using the Pango Markup Language for sensor labels. For instance if you want “VAXG” instead of “CPU Graphics” to be displayed, you would write:

label in4 "V<sub>AXG</sub>"

In order not to interfere with other utilities and to allow per-user configuration of the labels/ sensors Sensors-Unity first tries to read ~/.config/sensors3.conf before continuing with the lm-sensors config lookup described above.

Replacing your desktop laptop with a ITX workstation

If you use your laptop as a desktop replacement, you will at some point get an external display and a mouse/ keyboard for more convenient usage.
At this point the laptop becomes only a small case of non-upgradable components.

Now you could as well replace your laptop by a real case of comparable size.  This will make your PC not only easily upgradable, but allow higher-end components while being more silent at the same time.

Continue reading Replacing your desktop laptop with a ITX workstation

Secure Nextcloud Server

This article is about how to securely configure the machine where your Nextcloud/ Owncloud instance will be running.
Even if you set-up your connection with Owncloud in a secure way,  your data still can be compromised by exploiting security flaws in the underlying architecture.

In the following we specifically will cover the underlying software stack and brute-force password hacking attempts.

Continue reading Secure Nextcloud Server

How to manually update a deb package from source

Probably everyone has encountered a package in Ubuntu which was not the newest released version while one for some reason needed the newest one. The first step is to search for a PPA with the desired version. But what if there is no such PPA or you want to build the version yourself? This is where this guide comes in. Note however that this is not aimed at ordinary users – you need some experience with programming/ compiling to successfully build a package.

Before you start

Before you start make sure that you have source packages enabled in your software sources.
Next you obviously need the upstream source tar-ball of the new program which should look something like <packagename><version>.tar.gz.
Download this tar-ball to a new directory <somedir> and extract it there.

Updating Package info

For the following commands I assume you are in the previously created directory <somedir>.

First we need to get the old version of the source package

apt-get source <packagename>

This will download and extract the old source package into <packagename><oldversion>.

Now we need some helper scripts to perform the upgrading as well as the build-time dependencies of the package

sudo apt-get install dpkg-dev devscripts fakeroot
sudo apt-get build-dep <packagename>

Next change into the extracted sources of the old package and update the packaging

cd <packagename>-<oldversion>
uupdate -v <newversion> ../<packagename>-<newversion>.tar.gz

# change into the extracted new package
cd ../<packagename>-<newversion>

# update version info
dch -l ~ppa -D $(lsb_release -sc)

For more information see the Debian New Maintainers Guide.

Building the program

To trigger a rebuild of the program simply execute

dpkg-buildpackage

Uploading your version to a PPA

To upload a package to a PPA you first need to sign it to prove that you are the author. To do this you have to execute the following in the <packagename><newversion> directory

debuild -S

Furthermore you need the upload tool dput to actually perform the uploading

sudo apt-get install dput

Now change to <somedir> and execute

dput ppa:<your_username>/<repository> <source.changes>

You can find more information at Launchpad.