Bastien Bourdon

Application-level encryption and Personally Identifiable Information (PII)

Tue, 12 Nov 2019 00:00:00 +0000

More privacy regulations and laws such as GDPR in Europe (with worldwide implications) require businesses to be very careful and responsible in the way they process, store and communicate user data.

Extra care must be taken when processing what is often called Personally Identifiable Data, also referred to as PII. In most web application, PII means a user’s email address, phone number, full name, etc. Basically it is any piece of information that, taken individually, can identify a person.

As a side note, IP addresses are sometimes considered personally identifiable, although it is a debated topic. The website https://eugdprcompliant.com/personal-data/ mentions: “A much discussed topic is the IP address. The GDPR states that IP addresses should be considered personal data as it enters the scope of ‘online identifiers’.”

As a web application developer, you should take every reasonable step to protect your user PII data from leakage. There are many steps to this, at various levels of your stack. One of the ways to protect this data is to use application-level encryption.

Application Level Encryption and its limitations

Premise

Web applications usually store data in a database on a database server, either managed by a cloud provider (Microsoft Azure SQL Database, AWS RDS, etc.) or on a self-managed VM, or even on the same machine as the application.

At the end of day, the database data is saved on disk somewhere. Most likely, the machine uses disk encryption so the data is encrypted at rest: if someone walks into your cloud provider’s datacenter (good luck) and pulls out the drive storing your database, they would only find gibberish. Also your application (hopefully) connects to your database server over SSL/TLS so the data is encrypted in transit.

However consider if an attacker gains access to the database server (via SSH or any other way) or gets their hand on a database dump: the data is accessible in clear and the user PII is leaked.

Solution

Application Level Encryption remedies this by having the application encrypt the data before sending it and saving it in the database. When reading the data, the application decrypts it on the fly before serving it or using it.

This means the data is never stored in clear anywhere, and the clear version only lives shortly in memory when it is used.

Drawbacks and Limitations

Application Level Encryption comes with a lot of overhead. Encryption and decryption are expensive operations (slow and requires a lot of resource relatively to your application). Implementing it requires additional depencies to manage and maintain carefully and periodically, and developers are usually not very familiar with cryptography at this level. And finally it requires the usage of strong encryption keys which you need to handle very securely: if someone gets their hand on the key, they can access all the encrypted data.

So application level encryption requires a good and careful key management process. Using a Key Management System is strongly advised. I found that Azure Key Vault is a great solution, but there are other good ones such as Hashicorp Vault. A KMS will allow you to safely store your keys, and manage access to the keys granularly to each application or developer, giving them the minimum required privileges using role-based access control (RBAC). It will also facilitate key rotation: meaning changing the keys periodically (typically every year).

All of this is additional maintenance, cost, and complexity, and application level encryption can often be dismissed as overkill.

Another drawback: your application cannot lookup a database row by an encrypted field. In the case of PII, this means you cannot lookup a user by email for example, if the email field is stored encrypted (see last section of this article for a practical problem with this and a solution). This is because modern encryption algorithms use some smart random padding before encrypting a message: as a result, encrypting the same message several times does not produce the same encrypted value (but all these encrypted values decrypt to the same message: some serious mathematical black magic happening here). This also makes it impossible to sort by or filter by encrypted fields (you could select a subset of rows and sort/filter the whole set in code in your application, but that requires memory and is much less efficient than database-side). You can pretty much just read/decrypt these fields from rows you’ve already selected.

Personally, the main drawback is that I don’t find application level encryption much more secure than just using encryption in transit and at rest on disk. Most likely, if an attacker manages to gain access to the database, it would be through a phishing attack on one of your users, a vulnerability in your application, or gaining access to your application server. In this case, the attacker would also likely use your application to just decrypt the database data before dumping it…

Why you might still need to use Application Level Encryption

You might have noticed that the “drawbacks” section above is much larger than the “Premise” and “Solutions sections”. So in what cases might you still need to use it?

I have found myself being forced to implement application level encryption for some large enterprise clients, whose Security Architecture teams would make that a mandatory requirement. In this case, we just priced that extra overhead in the commercial agreement and implemented it.

In general, application level encryption is useful when you cannot completely trust the database server or provider, or if this is a mandatory requirement from a third party regulator or compliance team.

One last problem: encrypting application usernames

Many web applications use email address as the username for its users. Unfortunately the email address is personally identifiable information, and you might be required to store it only encrypted.

As we saw earlier, it is impossible to lookup a database row by an encrypted field. So how do you go about querying your database for a particular user by email address when they attempt to login for example?

I’ll describe a solution to this problem in my next article.

Setting up a data backup system at home

Sat, 07 Oct 2017 00:00:00 +0000

Data is more and more important, and grouping and backing up your personal data such as important documents and photos can be daunting and time consuming.

It is the most important task you most likely don’t do.

I find it easy to backup my data online on the fly: for example most of my documents (paperwork type) are in a Google Drive folder automaticcaly synced from my laptop(s) to my Drive account (using the Google Backup and Sync software), and my photos are all automatically backed up on Google Photos, as I take them from my phone. The first thing I do after coming back from travelling is to upload my non-connected camera’s (DLSR and GoPro) photos to my Google Photos account.

This is a good solution, as long a you trust the online storage providers involved. But who knows how it will be in a few years, in a decade, in a few decades… And what if you lose your access to your account somehow (a hack or simply by forgetting)?

In this article I describe how to have a small cheap machine (Raspberry Pi) constantly running at home, backing up the content of your different online storage accounts such as Google Photos and Google Drive, Dropbox, etc. into a single hard drive, safely stored at home.

It is cheap both because of the Raspberry Pi is inexpensive (you can even use an older model), and you won’t even notice its electricity consumption on your home bill.

The process involves installing an open source software called rclone on your Raspberry Pi, in charge of periodically copying your data from your various accounts online, into an external hard drive.

Install rclone on the Raspberry Pi

rclone is an open source software that replicates the infamous rsync unix command, but allows you to easily use online storage accounts such as Dropbox and Google Drive as remote servers.

rclone has a quite a few binary distributions available, including a Linux 32bit ARM compatible with the Raspberry Pi 3.

Install it by SSHing into your Pi and executing the following commands from :

curl -O https://downloads.rclone.org/rclone-v1.38-linux-arm.zip
unzip rclone-v1.38-linux-arm.zip 
rm rclone-v1.38-linux-arm.zip 
cd rclone-v1.38-linux-arm/
sudo cp rclone /usr/bin/
sudo chown root:root /usr/bin/rclone 
sudo chmod 755 /usr/bin/rclone 
sudo mkdir -p /usr/local/share/man/man1
sudo cp rclone.1 /usr/local/share/man/man1/
sudo mandb 

Note: These steps can be made into a Docker image for easily running rclone on any other Raspberry Pi.

Create a SSH tunnel during rclone config

Run rclone config and pick n to register a new remote storage account and choose a name for it. In my example, I register my Google Drive account calling it drive_bastien.

When registering a remote storage into rclone, you will need to authenticate to that storage account most likely. This is the case for Google Drive for example, with which you need to authenticate to your Google account in order to authorize your rclone setup to access and modify your drive. rclone config makes this authentication super easy, by temporarily creating a local webserver, asking you to open your browser at http://127.0.0.1:53682/auth and authenticate to your Google account there. However most likely you will be setting up rclone on a headless server where you cannot do that. My solution is to create a SSH tunnel from your local machine to your headless server, forwarding the server’s port 53682 to a port on your machine.

In order to open the temporary browser window for authenticating with Google Drive account on your Raspberry Pi:

ssh -nNT -L 9001:127.0.0.1:53682 pi@raspberrypi.local

(the -nNT flags makes the command not create a tty session and open any shell)

On your machine (not the Raspberry Pi), open your browser at http://localhost:9001/auth, and sign in to your Google Account. Once the authentication succeeded, you will have to manually change the domain in the URL in your browser back to http://localhost:9001 to complete the setup.

Kill the tunnel once the config is completed.

Try to sync a folder from remote into a local folder

That’s it, you can now easily copy and synchronise any folders and files. Try running rclone lsd drive_bastien: and see the list of top level folders listed. For my setup, I test backing up one particular folder called “Papers” from my Google Drive into a folder called “backup” on the Raspberry Pi:

rclone copy drive_bastien:Papers/ ./backup/Papers

and notice that all the folders and files under Papers have copied locally, preserving the file timestamps and meta-data etc.

Create a CRON job to run every day to backup the data you want from your Google Drive

crontab -e

I recommend explicitely adding one line per folder you want, with appropriate schedule:

0 0 * * * rclone copy drive_bastien:Google\ Photos /backup/Photos
0 0 * * * rclone copy drive_bastien:Papers /backup/Papers

Quickly setup a USB sound card on a Raspberry Pi 3

Fri, 16 Sep 2016 00:00:00 +0000

The raspberry-pi is a marvellous little device! So versatile, so powerful (relative to its price and size), it allows an infinite number of applications.

As a music fan and guitar playing aficionado, the first ideas that came to my mind were mostly about combining my interests in computers and music. In my personal pipeline of potential projects: a guitar tuner, a whole plethora of guitar effects (all-in-one effect pedal), a loop station, training a Recurrent Neural Network to correct my mistakes as I play, etc…

Unfortunately, the raspberry-pi does not come with a built in microphone or any audio input. And setting one up is not straight forward: to my surprise, there is a high number of posts online asking about how to setup audio input on the pi, and very little satisfying answers.

My investigation led me to using a USB sound card instead of trying to use the GPIO pins (way too manual and raw). Yes it sounds a little bit less maker-y, but hey, I want to get stuff done. As far as I know, there is only one GPIO sound card availbale on the market, but there is close to no documentation and support for it.

I settled for a Sound Blaster E1 (30 GBP as I write this article): it was within my price range and I trusted Sound Blaster would take care of writing proper drivers compatible with most OS including linux (I run Debian Jessie). I am sure many other and possibly cheaper options are available out there. However the trouble with USB cards is that you have to trust the maker provide sane drivers, ideally plug-and-play for any linux machines. This is the case with the Sound Blaster E1. Also: it looks pretty cool.

Enough talk, let’s get that bad boy working.

Start by plugging the USB sound card into any of your raspberry-pi’s USB ports, and ssh onto its command line. If your card has sane drivers, you can make sure the sound card is available as sound output device by typing:

$ aplay -l

which should return at least 2 cards’ descriptions (one for the built in audio output and one for your sound card)

**** List of PLAYBACK Hardware Devices ****
card 0: ALSA [bcm2835 ALSA], device 0: bcm2835 ALSA [bcm2835 ALSA]
  Subdevices: 8/8
  Subdevice #0: subdevice #0
  Subdevice #1: subdevice #1
  Subdevice #2: subdevice #2
  Subdevice #3: subdevice #3
  Subdevice #4: subdevice #4
  Subdevice #5: subdevice #5
  Subdevice #6: subdevice #6
  Subdevice #7: subdevice #7
card 0: ALSA [bcm2835 ALSA], device 1: bcm2835 ALSA [bcm2835 IEC958/HDMI]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 1: E1 [Sound Blaster E1], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

And similarily check the audio input devices:

$ arecord -l

Check the order in which your cards have been loaded:

$ cat /proc/asound/modules

In my case and I think in the case of most USB sound cards, we get the following, meaning the USB card comes second:

0 snd_bcm2835
1 snd_usb_audio

The first thing to do is to make everything a lot easier by switching this default order, so that your USB sound card always gets defined as the default card 0. You do this by editing or creating the following file:

$ sudo nano /etc/modprobe.d/alsa-base.conf

and giving it the following content

# This sets the index value of the cards but doesn't reorder.
options snd_usb_audio index=0
options snd_bcm2835 index=1

# Does the reordering.
options snd slots=snd_usb_audio,snd_bcm2835

Then we need to make sure the audio configuration shipped with the USB card is used: this is done by editing the alsa configuration file /usr/share/alsa/alsa.conf by only uncommenting one line (remove the # sign):

load card-specific configuration files (on request)

Then simply reboot your raspberry-pi. You can now check again aplay -l and arecord -l for the order in which devices come. A quick way to test the default setup is to run the speaker-test commandin the shell. You should hear some white noise coming out of the speaker/headphone plugged to your USB sound card.

You can record a few seconds of audio using the arecord test.wav command, and CTRL+C to stop recording. Listen to your test with aplay test.wav.

You should now be setup to use your Raspberry Pi with a USB soundcard and get cracking on some cool project involving audio input and output. Personally, my first project is making a guitar tuner using different pitch detection technics. To get started withthis project, I published a post in which I explain one way to get started with Digital Signal Processing on the Raspberry Pi 3.

Work in progress: Experimenting with DSP on Raspberry Pi

Mon, 12 Sep 2016 00:00:00 +0000

In this article I describe how I got started experimenting with Digital Sound Processing (DSP) on Raspberry Pi.

My first objective was to configure the Pi the play and record audio to and from my USB sound card. You can read how to set this up in my previous article.

The initial project I want to achieve is creating a guitar tuner; it would consist of:

the Raspberry Pi listening to audio input as stream (coming from the USB sound card)
taking samples of 1 or 2 seconds
approximating the pitch of the string played (implementing my own pitch detection algorithm)
determining what is the closest expected note and how far from that note the current pitch is
and using two 8-segments led displays (wired to the Pi’s GPIOs) to give a visual indication of the target note (A, B, C#, etc.) and of how to get there (up or down)

I start with experimenting with Python Sounddevice library which allows to play and record audio represented as Numpy arrays. This library binds with the PortAudio library. I choose python-sounddevice because I am comfortable dealing with Numpy and the scipy stack in general. Moreover, I know whatever pitch detection algorithm I implement, it will have to be fast, and pure python would likely be too slow.

First let’s install PortAudio on the Pi. It requires Alsa (which of course is already installed on Debian Jessie) but also its developer library:

sudo apt-get install libasound-dev libportaudio2

Also CFFI (C Foreign Function Interface for Python) needs to be installed, you can get it via pip nut you need to install libffi first:

sudo apt-get install libffi-dev
pip install cffi

We can finally install sounddevice:

pip install sounddevice

Now we can test recording and playing some audio in our python3 interpreter:

>>> import sounddevice as sd
>>> sd.query_devices()
  0 Sound Blaster E1: USB Audio (hw:0,0), ALSA (1 in, 2 out)
  1 bcm2835 ALSA: - (hw:1,0), ALSA (0 in, 2 out)
  2 bcm2835 ALSA: IEC958/HDMI (hw:1,1), ALSA (0 in, 2 out)
  3 sysdefault, ALSA (128 in, 128 out)
  4 front, ALSA (0 in, 2 out)
  5 surround40, ALSA (0 in, 2 out)
  6 iec958, ALSA (0 in, 2 out)
  7 spdif, ALSA (1 in, 2 out)
  8 dmix, ALSA (0 in, 2 out)
* 9 default, ALSA (128 in, 128 out)

>>> device = 0    # we use my USB sound card device
>>> duration = 5  # seconds
>>> fs = 44100    # samples by second
>>> myrecording = sd.rec(duration * fs, samplerate=fs, channels=2, device=device)
>>> myrecording.shape
(220500, 2)

We have recorded 5 seconds of audio from the default input device (my sound blaster card, listed as device 0) at a sample rate of 44100 Hz, on 2 channels. To simplify things in the future, I will always use one single channel, since I don’t care about stereo input for pitch detection.

Hello world!

Mon, 12 Sep 2016 00:00:00 +0000

Hello and welcome!

My name is Bastien Bourdon and this is the first post on my personal website. Its main purpose will be mostly to write for myself: I find writing an article about a subject one is working on helps a lot with completing a specific goal or milestone within that project, and forces good structure and communication.

I will write mostly about tech (web development, dev ops, data science), my experience as a startup co-founder and entrepreneur in general, and possibly any other subject or idea that piques my interest.

Web technologies are just a tiny subset of my original background, but as I spent time building online solutions, I have been astonished by the openness of its amazing community of developers and everything they are willing to share. Giving back and contributing is the primary motivation for this website.

Throughout my posts you will encounter some code examples presented in some beautiful highlighting such as:

def say_hello(greetings="Bonjour!"):
  print(greetings)
  return "User has been greeted."

While you wait for more progress on my writing, check out my startup’s website: edgefolio.com. Do not hesitate contacting me if you think any of it is interesting to you.

Cheers!