Build your own NAS/HTPC from scratch – part 1

A NAS/HTPC ?

In this article I’ll walk you through building your own NAS/HTPC, from hardware to software. Since it should be quite long, it should be broken down in several parts.

Let’s see first what I want as a NAS/HTPC, how much would it cost, what was chosen !

Wish list

Here goes my wish list:

  • Store a lot of data (movies, series, musics, ebooks, backups, …)
  • Store some data safely (at least RAID 1) such as family data (pictures, videos)
  • Export that data to different devices (Android tablet, other computers, …), for example ebooks
  • Watch all of the movies, series easily in HD on the NAS/HTPC but also on others devices within my LAN
  • Control most of it from WEB GUI and/or Android applications
  • Be cheap, in comparison to turnkey solutions
  • Low consumption / Low noise

Choices made

I’ve tried out several solutions, if I had been brighter, perhaps I would have listed the solutions and their drawbacks, but well I didn’t think about publishing an article. Let’s go through both hardware and software solutions, these are my choices, don’t follow blindly do your researches according to your wish list.

Hardware

On the hardware side, choices were clearly ruled by the price, I’ve tried to reduce the cost as much as possible without having to give up on quality.

  • Motherboard: ASUSTeK Computer INC. P5K-E I got it for free from a friend. It’s enough for my needs.
  • CPU: Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz, free too it really costs nothing nowdays.
  • CPU FAN: Noctua NH-U9B SE2 (~50€)
  • RAM: 4GB DDR2 800MHZ, also free, it’s consuming right now 1.3GB
  • DISKS: 4TB SATA3 WG40EZRZ (~ 80€ on special sales) // 500GB HITACHI // 500GB TOSHIBA, both I had bought years before
  • CASE: Cooler Master N300 (KKN1) (~50€)
  • FANS: 2 x 120mm front Noctua NF-S12B redux (~30€)+ 1 x 120mm side + Rheobus Lian Li PT-FN01 (10€)
  • GPU: GeForce GT 630, I got it back from an old computer, you’ll see in the hardware part how I fixed its well known noise problem, bit hackish. It doesn’t cost much and can do 1080p HD.
  • POWER SUPPLY: LDLC QS-460 FLP Quality Select 80PLUS Platinum (~50€ on special sales)
  • EXTRAS: anti noise screws for hard drive (~10€), HDMI cables, ethernet cables …

The overall cost is ~300€. So yes most of the hardware is old but my main goal is only to do a decent NAS/HTPC without putting much money on it. Conclusion is that it’s cheaper than turnkey solutions and is more flexible. I can go up to seven hard drives with my current case (6 only with my motherboard but well PCI SATA exists), 8GB RAM (or more), the GPU is decent enough for my needs. I strongly recommand to reuse components from older computers as I did, not only to save the bill but also the environment.

Software

As the base OS, openmediavault is my choice, it’s the perfect “NAS software” solution to me, it’s easy to use/install and quite light. You can manage your disks, RAID arrays, shares without effort (WEB GUI but also SSH hopefully). There are also plugins like Let’s Encrypt or Domoticz that come handy. It has no x server but we will see how to configure one with nodm.

For the HTPC part, kodi is clearly the right choice. An entire article is dedicated to kodi so I won’t say much about it now, but you’ll be able to start a video on your HTPC and continue it on your tablet at the timestamp you stopped, theming, control from your Android devices with Yatse.

Everyone downloads; so after many tests with several torrent clients, rtorrent is my choice ! What weight in the balance is the fact that you can use a WEB GUI, Deluge too, but that last one caused me many problems.

No cost at all except a long amount of time, but these articles should save you some time.

What’s next ?

In the next articles I’ll show you how the hardware takes place, it won’t be a step by step tutorial but it will show you the final product.

Duplicity to backup safely your data

A good backup tool ? Duplicity

I’ve tried many tools to backup, from bare rsync to tools such as restic or borg. Although they’re all good solutions, my requirements were never fully met. It’s why I chose to test and use duplicity. In this post I’ll focus on full server backup.

Here are my requirements for a good backup tool:

  • compression: I don’t have much to backup but I don’t want to use much either.
  • encryption: I only have a basic FTP that I don’t own, there it’s using GnuPG.
  • incremental backup: Also to save disk space.
  • on a FTP: with restic or borg directly I had lock problems over a curlftpfs mount, any backup would fail, it’s not related to these tools but to my usage.
  • easy to use and restore, it keeps ownership and rights.

Install duplicity

It’s pretty straightforward, install the duplicity package through your package manager on your server. You may also install ncftp if you want to backup over FTP.

# apt-get install duplicity ncftp
Install duplicity and ncftp

Generate your keys

Let’s generate your GNuPG key pair so we can encrypt our backup. This time, on your own computer, and not on your server, do the following.

# gpg --gen-key
Generate your GnuPG key pair

Take a 4096 bits length key, most of the default values should be enough.

# gpg --gen-key
gpg (GnuPG) 1.4.20; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: keyring `/home/floreo/.gnupg/secring.gpg' created
gpg: keyring `/home/floreo/.gnupg/pubring.gpg' created
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: Floreo Backup
Email address: xxx@floreo.info
Comment: Floreo Backup
You selected this USER-ID:
    "Floreo Backup (Floreo Backup) <xxx@floreo.info>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
........+++++
.............+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.+++++
.......+++++
gpg: /home/floreo/.gnupg/trustdb.gpg: trustdb created
gpg: key 1A6F2256 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   4096R/1A6F2256 2017-11-11
      Key fingerprint = CC2D 8FF5 9BB2 8D88 0E30  0056 FC84 663E 1A6F 2256
uid                  Floreo Backup (Floreo Backup) <xxx@floreo.info>
sub   4096R/874F94F5 2017-11-11
Generate your GNuPG key pair

Now export both the private and public keys, the first one you will keep it safely on a USB drive or anywhere safe, it would be used to decrypt your data, the other one will be required on your server. First find out your key UID.

# gpg --list-key
/home/floreo/.gnupg/pubring.gpg
-------------------------------
pub   4096R/1A6F2256 2017-11-11
uid                  Floreo Backup (Floreo Backup) <xxx@floreo.info>
sub   4096R/874F94F5 2017-11-11
List your GnuPG keys

There the UID is 1A6F2256, you can export the keys.

# gpg --output backup_pub.gpg --armor --export 1A6F2256
# gpg --output backup.gpg --armor --export-secret-key 1A6F2256
Export both the pub and private keys

Import your public key

SSH to your server so you can import your key only, first upload the public key through scp for example.

# scp backup_pub.gpg xxx.xxx.xxx.xxx:/home/toto/.
SCP your public key to your server

Your public key being on your server you need to import it to gpg.

# gpg --import backup_pub.gpg
Import your public key.

You would think it’s over but you have to change the trust of the key, do the following command.

# gpg --edit 1A6F2256
Change the trust of your key

One you have the GNuPG prompt, type trust, select optimal, say yes and you’re done !

First server backup

Duplicity allows you to do full backups or incremental ones. Of course the first one is a full backup. One cool thing is that you don’t have to tell it explicitly to do an incremental one once the first one is made.

I suggest you use that very simple script to do your backup, you can edit it to your needs, mostly the exclude part. It backups fully on Monday, following days of the week are incremental. On Monday it also removes old backups, it’s done this dirty way since you cannot use duplicity’s cleanup function without the private key and the passphrase which I refuse to use online. It keeps two weeks of backup which is enough for me. You need to set an autofs mount to erase old backups since I didn’t script the FTP deletion yet.

Now run it, it may take a while so I recommend that you do that in a screen or tmux.

Rather than doing it manually, don’t forget to set a cron task to backup every night your data.

0 2 * * * root bash /root/scripts/backup.sh &>/dev/null
/etc/cron.d/backup

Restore your data

Obviously if you backup anything, you need to restore it some day soon. This time this operation requires the private key so you can decrypt the data ! Do what’s best for you, but it’s safer to keep your private key on your computer, you have to import your private key in gpg first.

# gpg --import backup.gpg 
gpg: key 1A6F2256: secret key imported
gpg: key 1A6F2256: public key "Floreo Backup (Floreo Backup) <xxx@floreo.info>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
gpg:       secret keys read: 1
gpg:   secret keys imported: 1
Import your private key to your local computer

Next step is really simple you have to revert the duplicity command to restore.

# export PASSPHRASE=<PASSPHRASE>
# export FTP_PASSWORD=<FTP PASSWORD>
# duplicity  ftp://<USER>@<HOST>/<REMOTE_DIRECTORY>/ <LOCAL_DIRECTORY>
# unset PASSPHRASE FTP_PASSWORD
Restore your encrypted data locally

Finally rsync your data to your server, I know it looks complicated but it makes sure that your private key is never on your server.

Do not forget to remove your private key from your computer, it’s safer too.

gpg --delete-secret-key 1A6F2256
Delete your imported private key

A word of advice if you have several backup chains, for example I backup fully on Monday and do incremental ones for other days and keep two weeks top, duplicity will show you only the last full backup even if you restore stuff before the last full backup. It’s weird, but your restoration would work, it would just display the wrong date. To be sure I check the last mail.log, you might do it as well.

Check that your backups are good quite often. For further details, you can find some more explanations following the Ubuntu documentation.

Secure and extend you Philips Hue bridge with a reverse proxy

Philips Hue bridge

Recently I bought a Hue bridge with two bulbs, it was a specific “cheap” pack around 50€, the bridge itself usually costs that price, so I was quite interested. My idea was just to play with those bulbs and do funny things such as blinking when I get new mail, etc … After messing with it for a while using the Philips HUE app on Android, I wanted to do more so I checked the API. It’s quite well made and allows you to do a lot on your own, though, I had in mind to hide my bridge in my own local network, for security purpose and also to add more functionalities to it, let’s see how to make a hue bridge reverse proxy !

Requirements

To do that, you need:

  • a router, your own home box is enough
  • a raspberry PI or any computer with Apache and PHP
  • (optional) a USB to ethernet adapter, I bought this cheap one
  • some time to configure it all !

Configure the bridge

Follow the official instructions to install your bridge, you have to connect it to your router, it needs an IP within your local network so we can reach it with an other computer.

Once it’s all set and that it got an IP, open up your web browser and go to the following URL: http://192.168.1.xxx/debug/clip.html, obviously set the IP address to the correct one.

Create a new user following the steps on the API documentation. Keep the generated hash around, it’s important to control your bridge.

Stay in your web browser in the debug and do a GET call to http://<ip-address.of.the.bridge>/api/<username>/config, it will look like that:

Get the bridge information

Get the bridge information

Copy the mac field and save it somewhere, it’s important.

Finally, we will configure the bridge to stop DHCP and also to take the IP we want it to get out of the local range.

Still in your browser, do a PUT request to http://<ip-address.of.the.bridge>/api/<username>/config with the following content:

{"ipaddress":"10.50.0.2", "dhcp":false, "netmask": "255.255.255.0", "gateway": "10.50.0.1" }
Do a PUT request to http://<ip-address.of.the.bridge>/api/<username>/config

Once you run it, you should lose the control to the bridge ! No worries, we’ll get it back.

Network adapter

If you bought the network adapter I suggested, it should look like that:

Network adapter in its blister

Network adapter in its blister

SSH to your raspberry pi, and check the network configuration:

 # ifconfig -a
 eth0      Link encap:Ethernet  HWaddr XX:XX:XX:XX:XX:XX
           inet adr:192.168.1.3  Bcast:192.168.1.255  Masque:255.255.255.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:1575384 errors:0 dropped:62 overruns:0 frame:0
           TX packets:810579 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 lg file transmission:1000
           RX bytes:271762165 (259.1 MiB)  TX bytes:189260556 (180.4 MiB)
 
 lo        Link encap:Boucle locale
           inet adr:127.0.0.1  Masque:255.0.0.0
           UP LOOPBACK RUNNING  MTU:65536  Metric:1
           RX packets:4781492 errors:0 dropped:0 overruns:0 frame:0
           TX packets:4781492 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 lg file transmission:1
           RX bytes:402951806 (384.2 MiB)  TX bytes:402951806 (384.2 MiB)
ifconfig -a before plugging the adapter

Now plug the network adapter USB side to your PI, and connect the ethernet cable to the ethernet adapter.

Network adapter plugged

Network adapter plugged

We’ll check the adapter is working by doing the same command, but now we should see eth1 !

 # ifconfig -a
 eth0      Link encap:Ethernet  HWaddr XX:XX:XX:XX:XX:XX
           inet adr:192.168.1.3  Bcast:192.168.1.255  Masque:255.255.255.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:1576244 errors:0 dropped:62 overruns:0 frame:0
           TX packets:811252 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 lg file transmission:1000
           RX bytes:271834756 (259.2 MiB)  TX bytes:189382050 (180.6 MiB)
 
 lo        Link encap:Boucle locale
           inet adr:127.0.0.1  Masque:255.0.0.0
           UP LOOPBACK RUNNING  MTU:65536  Metric:1
           RX packets:4781492 errors:0 dropped:0 overruns:0 frame:0
           TX packets:4781492 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 lg file transmission:1
           RX bytes:402951806 (384.2 MiB)  TX bytes:402951806 (384.2 MiB)

 eth1      Link encap:Ethernet  HWaddr XX:XX:XX:XX:XX:XX
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 lg file transmission:1000
           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
ifconfig -a after plugging the adapter

Perfect, now let’s configure the network, edit the file /etc/network/interfaces by adding the following:

 auto eth1
 iface eth1 inet static
         address 10.50.0.1
         netmask 24
/etc/network/interfaces

Bring up eth1 end ping your bridge:

#  ifup eth1
 # ping 10.50.0.2
 PING 10.50.0.2 (10.50.0.2) 56(84) bytes of data.
 64 bytes from 10.50.0.2: icmp_seq=1 ttl=64 time=1.86 ms
 64 bytes from 10.50.0.2: icmp_seq=2 ttl=64 time=1.45 ms
 ^C
 --- 10.50.0.2 ping statistics ---
 2 packets transmitted, 2 received, 0% packet loss, time 1001ms
 rtt min/avg/max/mdev = 1.454/1.661/1.868/0.207 ms
Bring up eth1 and ping your bridge !

If it all works, you can move on to the HTTPS part, else, try using some networking tools such as tcpdump to determine what goes wrong.

HTTPS

New release of the Hue bridge uses HTTPS, of course it can’t rely on a real domain name since it’s connecting on the IP of the bridge and it can be different, after doing some reverse engineering it I found out how it works. Actually every bridge as an ID. Now that the link between your PI and the bridge is UP you can get your ID easily and prepare to create your own HTTPS certificate.

# openssl s_client -showcerts -connect 10.50.0.2:443 </dev/null
...
---
Server certificate
subject=/C=NL/O=Philips Hue/CN=00xxxxxxxx
issuer=/C=NL/O=Philips Hue/CN=00xxxxxxxx
---
...
Check what's the CN of your bridge

So there you go you have your CN which is actually what’s necessary to create your own HTTPS certificate. If you want to change your ID, you can, you’ll have to change it also in the PHP below.

First create a directory to store your key, certificate.

# mkdir -p /etc/ssl/hue
Create the /etc/ssl/hue directory

Now the key and the certificate

# openssl req -newkey rsa:4096 -nodes -keyout key_hue.pem -x509 -days 3650 -out certificate_hue.pem
Create your self signed certificate

The only parameter that matters here is the CN, put a correct bridge ID, in my case I just changed a few values.

Finally just concat both files to create a pem.

# cat certificate_hue.pem key_hue.pem > pem_hue.pem
Create the full certificate

A word of advice, as of now (version 1806051111 of the bridge), the hue app will use HTTPS to connect to the bridge, the first time you validate the connection it will stick the certificate. If you ever change the certificate, you’ll have to remove the credentials in your Android/iOS (i.e clear all data of the app) and press on the button again.

Second word of advice, if you test a lot, be careful to keep clean your whitelist user, it gets messy very fast ! You can delete some doing a DELETE request on /api/userYouControl/config/whitelist/userYouWishTodelete

Reverse proxy

Do you remember the MAC address of your bridge ? If, as I previously said you did, then we will use it pretty soon.

Stay on your PI and open up again /etc/network/interfaces, we will change the mac address of eth0 so if it will be seen as a HUE bridge for the mobile apps. You can use the mac address of the real bridge and alter it so it’s different, I suggest you use this website to make it sure it’s still seen as “Philips Lighting BV“.

auto eth0
iface eth0 inet static
        address 192.168.1.2
        netmask 24
        gateway 192.168.1.254
        dns-nameservers 127.0.0.1
        # faking MAC address to Philips HUE style
        hwaddress ether 00:17:88:78:45:12
Change the mac address

Now on your home box, set the IP of your PI as a static one for this tutorial it will be 192.168.1.3.

It’s time to install haproxy, apache and php, I won’t describe this here, do as you wish, we’ll just go through what’s really important.

There goes a valid haproxy configuration:

# Faking Hue Bridge requires HTTPS now

frontend f_http_hue
        mode http
        bind 192.168.1.3:80
        use_backend b_http_hue

frontend f_https_hue
        mode http
        bind 192.168.1.3:443 ssl crt /etc/ssl/hue/pem_hue.pem
        use_backend b_http_hue

backend b_http_hue
        mode http
        server bridge 127.0.0.1:8080
/etc/haproxy/haproxy.cfg

Install mod_proxy and mod_http_proxy, create a new virtualhost /etc/apache2/sites-available/hue.conf this way:

<VirtualHost 127.0.0.1:8080>
    DocumentRoot /var/www
    ServerName xxxxxxx
    CustomLog /var/log/apache2/hue_access.log combined env=!forwarded
    CustomLog /var/log/apache2/hue_access.log proxy env=forwarded
    ErrorLog /var/log/apache2/hue_error.log

    RewriteEngine On
    RewriteCond %{REQUEST_URI}  ^$ [OR]
    RewriteCond %{REQUEST_URI}  ^/$ [OR]
    RewriteCond %{REQUEST_URI}  \.png$ [OR]
    RewriteCond %{REQUEST_URI}  \.xml$ [OR]
    RewriteCond %{REQUEST_URI}  ^/debug/clip.html
    RewriteRule (.*)              http://10.50.0.2$1    [P,L]

    RewriteRule ^(/api.*)$ /index.php?q=$1 [L,QSA]
</VirtualHost>
/etc/apache2/sites-available/hue.conf

Activate it and edit /var/www/index.php:

<?php
/*
* Activate proxy_module and proxy_http_module
*/

ignore_user_abort(true);

$mac_address = [ '<mac address of the real bridge>',  '<your fake Philips HUE mac address>' ];
$ip_hue = [ '<ip of the real bridge>', '<ip of your PI>' ];
$bridge_id = [ '<the real bridge id>', '<faked id bridge>' ];
$gateway_hue = [ '<ip of your PI>', '<gateway of your local network>' ];
$url = 'http://10.50.0.2'.$_SERVER['REQUEST_URI'];
$useDB = false;

$options =[ 
    'http' => [
        'header'  => "Accept-Encoding: gzip, deflate\r\nAccept-language: en-US,en;q=0.8\r\nUser-Agent: ".$_SERVER["HTTP_USER_AGENT"]."\r\n",
        'method'  => $_SERVER["REQUEST_METHOD"],
    ]
];

if($_SERVER["REQUEST_METHOD"] !== 'GET' ){
    $options['http']['header'] .= "Content-type: application/x-www-form-urlencoded\r\n";
    $options['http']['content'] = file_get_contents("php://input");
}

$context  = stream_context_create($options);
$result = file_get_contents($url, false, $context);
if ($result === FALSE) {  }

// faking HUE bridge EDIT: 13/08, it needs to be faked all the time now
$result = str_replace($mac_address[0], $mac_address[1], $result);
$result = str_replace($ip_hue[0], $ip_hue[1], $result);
$result = str_replace($gateway_hue[0], $gateway_hue[1], $result);
$result = str_replace($bridge_id[0], $bridge_id[1], $result);

ob_start();
echo $result;
$size = ob_get_length();
header("Content-Length: {$size}");
header("Connection: close");
ob_end_flush();
ob_flush();
flush();

/** DB part **/
if($useDB){

    $_user = '';
    $_password = '';
    $_database = '';
    $_host = '';

    try {
	$_db_link = new PDO('mysql:host='.$_host.';dbname='.$_database.';charset=utf8', $_user, $_password);
    } catch (Exception $e) {
        die('Erreur : ' . $e->getMessage());
    }

    $stmt = $_db_link->prepare("INSERT INTO hue_log (ip, method, url, content) VALUES (:ip, :method, :url, :content)");
    $stmt->bindParam(':ip', hash('sha256', $_SERVER['REMOTE_ADDR']));
    $stmt->bindParam(':method', $_SERVER['REQUEST_METHOD']);
    $stmt->bindParam(':url', $_SERVER['REQUEST_URI']);
    $stmt->bindParam(':content', $result);
    $stmt->execute();
}

Now restart Apache and open your browser on http://192.168.1.3, it should load ! Check the logs of Apache, you should also see some requests.

Philips’ upgrades

I didn’t mention it, because I hadn’t figure it yet, but all the Philips’ Upgrades won’t be done anymore since the bridge has no internet access. I found out one way to do it, it’s tricky but anyone can do it.

I use the Hue App on my mobile to control lights, it tells you when some upgrades are required, it’s how I know I should run them. When it happens, I simply forward packets from my bridge to my PI and force the update thought the API, let’s do it.

Allow packet forwarding and forward the bridge to the PI:

 echo 1 > /proc/sys/net/ipv4/ip_forward
 iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
 iptables -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
 iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT
Allow kernel forwarding and forward bridge to the PI

You should notice that the third light is now lit, it reaches internet, no worries we open it up only temporarily.

Connect to the API and run the following commands:

 {
         "portalservices": true
 }
Do a PUT request to http://<ip-address.of.the.bridge>/api/<username>/config
{"swupdate": {"checkforupdate":true}}
Do a PUT request to http://<ip-address.of.the.bridge>/api/<username>/config
 {
         "swupdate": {
                     "updatestate": 3
                         }
 }
Do a PUT request to http://<ip-address.of.the.bridge>/api/<username>/config

It should download the update, restart your bridge, launch the update from your mobile app, sometimes it’s quite long ! To update three bulbs it took me around 1 hour, it depends on the update.

Once everything is done, cut the bridge from the internet:

 iptables -F nat
 iptables -F FORWARD
 echo 0 > /proc/sys/net/ipv4/ip_forward
Clean up the firewall and shut down forwarding

Reboot it once again so remaining connections will be stopped. Only two lights should remain.

You might check that your bridge is really off the internet by doing a GET request to http://<ip-address.of.the.bridge>/api/<username>/config, you should have that:

     "portalstate": {
         "signedon": false,
         "incoming": false,
         "outgoing": false,
         "communication": "disconnected"
     },
     "internetservices": {
         "internet": "disconnected",
         "remoteaccess": "disconnected",
         "time": "disconnected",
         "swupdate": "disconnected"
     },
Content of the GET to http://<ip-address.of.the.bridge>/api/<username>/config

Also, it’s possible you do a tcpdump to check to what your bridge tries to contact, it might amuse you.

Sources for the upgrades:

Epilogue

Your HUE bridge is hidden behind your raspberry PI, it’s secure moreover you can also edit the PHP to add new functionality !

Cache and compress your favicon !

Favicon ?!

Any “good” website comes with a favicon, that little picture near the URL bar in your web browser. I’ve noticed recently that for mine, it couldn’t be cached by web browsers and that neither was it compressed as you may notice on that webpagetest.org.

Screenshot of failed favicon caching

So let’s solve that quickly !

Set the MIME type

First of all, I have no clue why, but Apache2 does not recognize .ico correctly, so you have to set the MIME type by yourself, let’s do it:

AddType image/x-icon .ico
/etc/apache2/mods-enabled/mime.conf

If you do not specify that binding, the following steps won’t work !

Caching

Now let’s activate the expires module if it’s not already done:

a2enmod  expires
Activate expires mod

And let’s add our caching rule:

ExpiresByType image/x-icon "access plus 1 year"
/etc/apache2/mods-enabled/expires.load

I set the caching time to one year since I don’t change frequently my favicon, it’s up to you.

Compressing

The deflate module allows Apache2 to compress stuff, here we’ll first activate it:

a2enmod deflate
Activate deflate mod

We need to configure it now:

AddOutputFilterByType DEFLATE image/x-icon
/etc/apache2/mods-enabled/deflate.conf

Last but not least, restart Apache2:

service apache2 restart
Restart Apache2

Result

Let’s run the test once again at webpagetest.org !

Best grade at webpagetest for compressing images

And the favicon is no more within the “not compressed” nor “not cached” section:

Favicon cached and compressed !

 

I do agree that it’s just the favicon who cares, but you can use that configuration for other kind of assets, such as pictures, scripts, stylesheets, …

Use Radicale to get your own shared calendar !

Heyo,

I was looking for a shared calendar, my prerequisites were:

  • self hosted
  • open source and free
  • “cross-platform” (I want it on my laptop, PC and my mobile)

After some research, I ended up on Radicale it’s a small CalDAV and CardDAV python server, and it fulfills all my requirements. Also you can:

  • secure the connection (I don’t mind I use HAProxy on top with Let’s Encrypt certificate, but still it’s cool)
  • support authentication (I use .htpasswd)
  • rights access, you can set some pretty interesting rights to your calendars, like read only for some users, and write for some others to others’ calendar …
  • storage hook, you can add a hook for example git, so each time you modify a calendar it’s committed and pushed to git !
  • other cool stuff, I didn’t completely review it all

So if it’s still appealing to you, follow me !

Install Radicale

Open up your terminal and do the following (still only showing Debian version):

aptitude install python3-pip
python3 -m pip install --upgrade radicale
Install requirements and Radicale

Configure it

For security reasons and convenience, I run my Radicale server behind HAProxy, so my configuration file maybe won’t be useful to you, you just have to change the hosts though.

It’s possible you don’t want the same choice I’ve made within the configuration below either (storage, git hook, authentication, file rights) so I suggest you take a look at the wonderful documentation.

If you want to use git as a hook, and also to have an authentication mechanism which I highly recommand, do the following:

aptitude install git libffi-dev apache2-utils
python3 -m pip install --upgrade passlib bcrypt
Requirements for authentication and git hook

Do not forget to create the storage folder:

mkdir -p /var/lib/radicale/collections
Create storage folder

If you choose an “.htpasswd authentication”, you can create each access like that:

htpasswd -B /etc/radicale/users <username>
Create a user access

Now edit /etc/radicale/config:

[server]
hosts               = 127.0.0.1:5232
max_connections     = 5
timeout             = 2

[storage]
filesystem_folder   = /var/lib/radicale/collections
hook                = git add -A && (git diff --cached --quiet || git commit -m "Changes by "%(user)s)

[auth]
type                = htpasswd
htpasswd_filename   = /etc/radicale/users
htpasswd_encryption = bcrypt

[web]
#type                = none

[logging]
debug               = true

[rights]
type                = from_file
file                = /etc/radicale/rights
/etc/radicale/config

Some quick explanations:

  • server section is quite self explanatory
  • auth section, you can define several fields such as the type of auth, where is your file and the encryption
  • web needs some explanation, if you set type to none, then you won’t have the interface though the web (it’s cool when you are in production), there I comment it so I have access to the interface so I can create calendars
  • logging is easy to understand
  • rights, in my case I choose to load the rights from a file but there are other options

You have to init the git repository if you chose to use it:

cd /var/lib/radicale/collections
git init
Init the repository

Now create a .gitignore like that and you’re done:

.Radicale.cache
.Radicale.lock
.Radicale.tmp.*
.gitignore

Let’s move on to the configuration of rights in /etc/radicale/rights:

## GLOBAL ##

# any authenticated user can reach root collection
[read]
user = .+
collection =
permission = r

# specific to login to web panel, if not you can't login because only your data is allowed and not your root
[rw_own_root]
user = .+
collection = %(login)s
permission = rw

# any authenticated user can rw its data
[rw_own]
user = .+
collection = %(login)s/.*
permission = rw
/etc/radicale/rights

Again it’s quite easy to understand, a few notes:

  • the name within the bracket can be what you want
  • you cannot put several user in one section, let’s say you want to give rw to several user, you’ll have to repeat each section
  • careful between user = .* (matches everyone including anonymous users) and user = .+ (only matches authenticated users)

A special note, in my case I have more right with my user, it can rw an other user’s calendar and also only read an other. Here’s a preview if you want to adapt it for your own purpose:

# floreo can read blublu
[r_floreo_blublu]
user = floreo
collection = blublu/.*
permission = r

# floreo can read/write blibli
[rw_floreo_blibli]
user = floreo
collection = blibli/.*
permission = rw
Extra for /etc/radicale/rights

Check the configuration documentation for further details.

I suggest you have a look at the rights configuration since it can be very dangerous.

Supervisor

I use Supervisor to run Radicale in case it crashes. I know you can write an init script or you can use screen or tmux but that’s ugly.

aptitude install supervisor
Install Supervisor

Now let’s configure it to run Radicale in /etc/supervisor/conf.d/radicale.conf:

[program:radicale]
command=python3 -m radicale
stderr_logfile = /var/log/supervisor/radicale-stderr.log
stdout_logfile = /var/log/supervisor/radicale-stdout.log
/etc/supervisor/conf.d/radicale.conf

Time to start it:

supervisorctl reread
supervisorctl start radicale
reread and start Radicale

To be clean, let’s add a logrotate in /etc/logrotate.d/supervisor:

/var/log/supervisor/*.log {
    weekly
    rotate 52
    compress
    delaycompress
    notifempty
    missingok
    copytruncate
}
/etc/logrotate.d/supervisor

HAProxy

As I said previously I’m running HAProxy on top, here goes one possible configuration in /etc/haproxy/haproxy.cfg:

frontend https
        bind :::443 v4v6 ssl crt <path to your pem>
        http-request set-header X-Forwarded-Proto https

        use_backend radicale if { hdr(Host) -i <your domaine name> }

frontend http
        bind :::80 v4v6
        http-request redirect scheme https if { hdr(host) -i <your domaine name> } !{ ssl_fc }
        use_backend radicale if { hdr(Host) -i <your domaine name> }

backend radicale
        option forwardfor
        server radicale localhost:5232
/etc/haproxy/haproxy.cfg

Runtime !

Now if you connect to your Radicale web panel with one of your user created previously in the .htpasswd file, you should be able to create a new calendar or adressbook. So you can do it, that shouldn’t be hard, and it will give you a full link to your own collection, that’s the link you’ll have to insert in your clients !

Clients

On my computer I use Thunderbird with the lightning modules, careful because if you have several account in Radicale, Thunderbird only sticks to one, so I put the user and password in the URL (i.e http://username:password@example.com/) to avoid the login prompt …

On my mobile, I use Open Sync to deal with the sync of CalDAV and SolCalendar (it looks like it got removed from the Play Store recently …)

The end

If you followed well, you should remember that we have the Radicale web panel UP, I suggest to turn it off once you have created the calendar and addressbook for each of your users since you can manage events and contacts from your clients. To do so, open up /etc/radicale/config and uncomment the lines:

[server]
hosts               = 127.0.0.1:5232
max_connections     = 5
timeout             = 2

[storage]
filesystem_folder   = /var/lib/radicale/collections
hook                = git add -A && (git diff --cached --quiet || git commit -m "Changes by "%(user)s)

[auth]
type                = htpasswd
htpasswd_filename   = /etc/radicale/users
htpasswd_encryption = bcrypt

[web]
type                = none

[logging]
debug               = true

[rights]
type                = from_file
file                = /etc/radicale/rights
/etc/radicale/rights

And don’t forget to restart Radicale through Supervisor:

supervisorctl restart radicale
Restart Radicale through Supervisor

Done, it should all work well 🙂

Addendum : how to get your calendar back in case you fucked up

So it happened to me few minutes ago to completely destroy my calendar using a script trough WebDAV. Hopefully I use git hook to track all modifications to my calendar so here it goes to get your calendar back !

First, don’t panic.

Secondly, go to your Radicale root lib directory, it should be /var/lib/radicale/collections and list your modifications in git:

# git log
commit d78ec10d0b80ccae7fb25ecda26d5b5f05e2f69f
Author: root <root@xxx>
Date:   Thu Aug 17 18:55:30 2017 +0200

    Changes by fuckedup_script

commit 7fe7c14099834ef843175c06e6bd2bfa1212a68c
Author: root <root@xxx>
Date:   Thu Aug 17 18:50:20 2017 +0200

    Changes by floreo
git log

So in this example, we can see two commits, the last one was made by a script, and the other one by me directly. We need to go back to the previous commit made by me, and not by the script, so just write down the commit number, here it’s 7fe7c14099834ef843175c06e6bd2bfa1212a68c.

Let’s go back to our previous commit:

git reset --hard 7fe7c14099834ef843175c06e6bd2bfa1212a68c
git reset

And voilà, you got your calendar back, fiuuu !

Addendum 2: keep your Radicale up to date

It’s quite easy to do so, just run an upgrade once in a while, or read that page

python3 -m pip install --upgrade radicale
Upgrade Radicale

You now have to restart it, if you use Supervisor as I explained above, just kill nicely the pid and it will run automagically:

kill -3 $(ps aux | grep -i radicale | grep -v grep | awk '{ print $2 }')
Kill Radicale!

Check the version is alright:

radicale --version
Check Radicale's version