ProofOfPizza

Bye G***le, Hi Plausible - Analytics with a clear conscience

2022-04-11T00:00:00+00:00

So you have a web page, or maybe even a bunch of pages, and you are interested to optimize them, either for better user experience, more sales revenue, or you just want to know if anyone reads your blog that you spend all these hours on. Well technology is here to save you! If you dive in this topic the first second third and fourth thing (do I keep counting?) you will find is G***gle Analytics. It is free, and installation is easy and it will give you all the data you might ever need. But what does “free” mean? And why not look at what you need first, and then see if you can get just that? In this blog we explore an alternative, Plausibe analytics, why and how we at InQuisitive transitioned.

If you are not paying for it, you’re not the customer; you’re the product being sold. blue_beetle

This is a famous quote by now, but in this context it is actually worse. How much worse? Well that depends on your success. Because your site is free to visit for anyone, and you have G***le Analytics for free, but who is the product here? Not just you, all the people who visit your site in good faith. Every friend, customer, family member you show it to, your colleagues that you send a link etc. They are all products, you give away their data, so that all of us, including all those people who never even went to your site are being targeted with ever growing precision: manipulating our buying decisions, voting decisions, deciding on the information that reaches us so we’ll form our opinions aligned with the Big Buyers interests… Is it that bad ? Yes it is!

You should understand that this incredible power lies not in specific data you provide, it lies in the immense amounts of data that organisations such as G***le or F***book (or Meta!) have of us. Their algorithms find patterns, and using those patterns they’re able to predict so many things about us that in many ways, they know us better then we know ourselves or our loved ones. Our freedoms are at stake, and even though we can not fix this problem by ourselves, we can choose to not be an accomplice in giving them away… for free!

There is, simply, no way, to ignore privacy. Because a citizenry’s freedoms are interdependent, to surrender your own privacy is really to surrender everyone’s. You might choose to give it up out of convenience, or under the popular pretext that privacy is only required by those who have something to hide. But saying that you don’t need or want privacy because you have nothing to hide is to assume that no one should have, or could have to hide anything – including their immigration status, unemployment history, financial history, and health records. You’re assuming that no one, including yourself, might object to revealing to anyone information about their religious beliefs, political affiliations and sexual activities, as casually as some choose to reveal their movie and music tastes and reading preferences. Ultimately, saying that you don’t care about privacy because you have nothing to hide is no different from saying you don’t care about freedom of speech because you have nothing to say. Or that you don’t care about freedom of the press because you don’t like to read. Or that you don’t care about freedom of religion because you don’t believe in God. Or that you don’t care about the freedom to peaceably assemble because you’re a lazy, antisocial agoraphobe. Just because this or that freedom might not have meaning to you today doesn’t mean that that it doesn’t or won’t have meaning tomorrow, to you, or to your neighbor – or to the crowds of principled dissidents I was following on my phone who were protesting halfway across the planet, hoping to gain just a fraction of the freedom that my country was busily dismantling. Edward Snowden

How does that work ?

You installed G***gle Analytics on your website. Now every time someone comes to your site, navigates, loads a widget, clicks on a link etc a call is made to G***gle to inform them. This is how they get their data, and how they get you to work for them… for free! Well not entirely: they give you some insights into that data of course. Obviously, as said before, data is much more valuable if it exists in greater quantities so the same data is probably more worth to them than to you.

A simple search on the internet shows us that G***le is retrieving so much data it is hard to understand, there are countless plugins and other software to link it to just to make sense of it! But why ? Well, because it was never designed for you or your purpose of it.

Are there alternatives ?

Yes there are! There are lots of them actually, just none as famous and big as G***le Analytics, and therefore you have to seek them out. To name but a few, we have:

This is just a small list, there are many more. These five are all open source. So you see there is plenty to choose from! For a comparison you can for instance see sites like rigorous themes.

Where to go from here ?

I started looking into this, because I was interested to know if anyone reads my blogs. As simple as that. If there were only people from the Netherlands looking I might consider changing the language to Dutch. And as a privacy geek I wanted to know what possibilities exist that might enable me to get some insights into my visitors, without selling it all out to the big guys, and without me feeling guilty for becoming just like them! After some research into ease of use, easy of install, data provided, price, documentation and type of project/company, I settled on Plausible.

“Plausible Analytics is an open-source project dedicated to making web analytics more privacy-friendly.” it says on their about page. I like that! And just reading through the site and the documentation was a fresh breeze. So I signed up for the 30 days trial, to see if I could indeed get it working, and to see if I would find it worth while. It turns out that configuration for my jekyll site was almost to easy to mention. I had to add two lines to my _config.yaml:

analytics:
  provider: "custom"

And then include a small, one line script that Plausible provided for me in the folder _includes. That. Was. It.

I looked into the statistics and immediately understood what I was looking and found directly what it was I was looking for. I could see how many people visit my pages. If I want I can send a url with a small query parameter so that I can easily recognise from where the people came, and which actions I take, such as sharing it on LinkedIn, or directly with my colleagues, or referencing it in StackOverflow answers, result in people actually reading my work.

I was very satisfied, and decided that this would be a good step, not just for me and my blog but also for the company I work with, InQuisitive.

Please note that while I am very enthusiastic about their product I am in no way affiliated to Plausible.

Price and possibilities

I choose to go for the easiest way, the hosted version where I do not have to manage any servers and they promise to keep my data in Europe, and all in accordance with the GDPR. This is important because we see countries moving towards stricter interpretations and enforcements when it comes to data protection. Cases in Austria and France had led to the EU declaring G***le Analytics illegal.

The software itself is free and open source, so if you want, or need to you can host it all yourself and keep the data in house. They made script for you make it easy to install and run everything, and if you see a possible improvement, just make a PR! But I really think for the money you do not need to, a basic subscription is as low as 9 Euros per month. But if you do and you use it heavily, by all means please consider their suggestion:

If you’re self-hosting Plausible, sponsoring us is a great way to give back to the community and to contribute to the long-term sustainability of the project. Thank you for supporting independent creators of Free Open Source Software!

Introducing it in our company’s websites

Next I called my colleague Maxime who works as the chief marketeer in our office. I said “Hey Maxime, remember those ideas we all wrote down about wanting to do some good with this company? Well I may have an idea to put our money where our mouth is, and I’d like to discuss it with you”. So we set up a meeting and I asked her a bunch of questions like, what are we using now for our analytics? What are the data points you actually look at and use? Do you get all the info you need? Is it cluttered with stuff you do not actually want to collect?

Then I proceeded to show my screen and show her what Plausible looked like for my site, and asked if that might work for her. She liked the idea (duh, nobody wants to be part of …), and said she would look into it. We agreed she’d read up on it, and if she wanted to proceed with a trial next to the existing analytics, we’d meet up and I’d help her set it up. Except, she did not.

I waited, and waited a bit more. And then I saw a message “Hey, just wanted to let you know that I managed to get it working for both our sites with the Plausible plugin [for Wordpress]. Looking good!” Now that was nice, no need for any techie to assist, just fiddle around a bit and you have it working!

Conclusions

We at InQuisitive now have all our sites set up with Plausible, and G***le is out. It may not seem much but the important thing is that it is something. A step in the right direction, in the direction we want to go. Where we are autonomous to choose what we do, instead of walking on some big data bully’s leash. Also remember that just “your little bit of data” is not the problem, but all those little bits make up a really big problem. So take control, keep yourself, your company and the people around you safe. Or at least: refuse to be part of the threat. It’s fun, it’s really not that hard, so spend a day on it!

A well spent day brings happy sleep. Leonardo da Vinci

#AutomaticAlley #3 - Safe and easy AWS with 2FA and scripted login

2022-03-13T00:00:00+00:00

And here is the third in the series “AutomaticAlley” where I share some tips, tricks, bits and bites to make day to day life easier. Especially for us nerds on the command line. In this episode we’ll have a look at enforcing AWS IAM users to setup 2FA and then how to make that bearable by scripting the login. This writing is sort an extension of my earlier blog post.

As I have said before 2FA (or MFA in general) is of crucial importance when working with cloud providers such as AWS, GCP, or Azure. A simple risk analysis says that:

Risk = probability * impact
Risk = really quite possible though not immediately likely * selling your house and living in debt for the rest of your days = worth the trouble of using 2FA

See? Math is easy!

Now then how do we make sure our IAM users (being your colleagues) to use 2FA ? We setup a policy for that! IAM let’s us create users or roles, and attach policies to them which describe exactly:

What the effect is: allow or deny
What action we are talking about
What the relevant resources are for that action
Which version of the language syntax rules.

There are more options, such as conditions to make even more fine grained rules for you users. Specifying Sid can help you document and make for easier understanding. All of this is just in plan json structure so for most of us reading this, and even remotely interested in setting this up, quite readable. If you find it hard, there is always the option to specify it in the AWS Console. (In fact in that case I would recommend selecting json, the copy paste this document, click on Policy Summary.)

The policy to deny everything if no MFA is present

In our case the policy would like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowViewAccountInfo",
            "Effect": "Allow",
            "Action": [
                "iam:ListUsers",
                "iam:ListMFADevices",
                "iam:GetAccountPasswordPolicy",
                "iam:GetAccountSummary"
            ],
            "Resource": "*"
        },
        {
            "Sid": "AllowChangeOwnPasswordsOnFirstLogin",
            "Effect": "Allow",
            "Action": [
                "iam:ChangePassword",
                "iam:GetUser"
            ],
            "Resource": "arn:aws:iam::*:user/${aws:username}"
        },
        {
            "Sid": "AllowChangeOwnPasswordsAfterMFAEnabled",
            "Effect": "Allow",
            "Action": [
                "iam:GetLoginProfile",
                "iam:UpdateLoginProfile"
            ],
            "Resource": "arn:aws:iam::*:user/${aws:username}"
        },
        {
            "Sid": "AllowManageOwnVirtualMFADevice",
            "Effect": "Allow",
            "Action": [
                "iam:CreateVirtualMFADevice",
                "iam:DeleteVirtualMFADevice"
            ],
            "Resource": "arn:aws:iam::*:mfa/${aws:username}"
        },
        {
            "Sid": "AllowManageOwnUserMFA",
            "Effect": "Allow",
            "Action": [
                "iam:DeactivateMFADevice",
                "iam:EnableMFADevice",
                "iam:ListMFADevices",
                "iam:ResyncMFADevice"
            ],
            "Resource": "arn:aws:iam::*:user/${aws:username}"
        },
        {
            "Sid": "DenyAllExceptListedIfNoMFA",
            "Effect": "Deny",
            "NotAction": [
                "iam:ListUsers",
                "iam:ListMFADevices",
                "iam:ChangePassword",
                "iam:GetUser",
                "iam:CreateVirtualMFADevice",
                "iam:DeleteVirtualMFADevice",
                "iam:DeactivateMFADevice",
                "iam:EnableMFADevice",
                "iam:ListMFADevices",
                "iam:ResyncMFADevice"
            ],
            "Resource": "*",
            "Condition": {
                "BoolIfExists": {
                    "aws:MultiFactorAuthPresent": "false"
                }
            }
        }
    ],
    "Id": "MFA Required IAM Policy"
}

Shall we do quick run through of this document? First we describe rights to see account info: we allow some list and read access to users, devices, and policies. We specify this for "Resource": "arn:aws:iam::*:user/${aws:username}". This is an ARN or Amazon Resource Name, a way to identify well, Amazon resources. It is made up of several components, usually something like arn:aws:<the service>:<the account number>:<resource-type>. In this case it is simply within service iam, for any (*) account you have access to, the current user. In short you can do all this, as long as it is about your current user.

Then the following blocks are somewhat similar: We allow the user to change their password on first login, after setting up MFA, and of course we allow the user to setup MFA and use it. If you understood the structure of the first block these blocks are sort of self-evident.

Now the spice in the salsa here is found in the last block DenyAllExceptListedIfNoMFA. Contrary to the other statements this one has effect Deny. Then it has another negation: NotAction, which means that we deny not the following actions or we deny all but the following actions or in even more plain english: we allow ONLY the following actions. Then we list those actions: all basic actions that will allow our user to to nothing important except setup their MFA.

The last part to point your attention to is the Condition block. It basically says: “All the above, is only valid under this condition: ‘If the user has no MFA present’”

Pretty neat right ? If you want to play along: just create a user in IAM, and attach this policy to them. Then either login into the console, or make sure you create access keys and log in on the cli. You will find it nice and boring! Nothing to do, except set up you MFA! But once you do, well… then who knows? Meaning: You should also specify more policies for the user. Normally we would do this in a separate document, because you can attach multiple policies to a user. This way you can reuse this policy for all your users, or those in certain groups. And at the same time, specify specific access to users based on what they need to be doing in AWS.

The AWS cli is a very useful and powerful tool. It gives you direct access to the AWS api: you can list and look at resources, create update and delete them. As such it should be clear why it is important to be careful with your credentials!

Running a simple command such as aws ec2 describe-instances will be successful only if you can be authenticated (you are who you say you are) and you have sufficient authorization (you are allowed to do what you want to do). How does this work on the cli ?

There are multiple options:

Provide your credentials in the command line with the command every time. (Who even does this ?)
Make use of the environment or shell variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and if using MFA AWS_SESSION_TOKEN
Use the credentials stored in the default credential file: ~/.aws/credentials.

This is also the order in which the possibilities are evaluated. The last option is easiest to setup using the aws configure command.

Lets have a look at that file. If you open it in your favorite editor you should see something like:

[default]
aws_access_key_id=AKIAXXXXXXXXXXXXXXXX
aws_secret_access_key=someunreadablemumbojumboxxxxxxxxxxxxxxx

Almost nothing to explain there: it shows the credentials and [default] tells us this is the default profile. We can set up different profiles and name them as we want. For instance if we have a different user that has read only rights. To use that user we would add a part to ~/.aws/credentials:

[default]
aws_access_key_id=AKIAXXXXXXXXXXXXXXXX
aws_secret_access_key=someunreadablemumbojumboxxxxxxxxxxxxxxx

[readonly]
aws_access_key_id=AKIAYYYYYYYYYYYYYYYY
aws_secret_access_key=someotherunreadablemumbojumboxxxxxxxxxx

But what about MFA ?

But if you have been impatient, curious or both and you set up the policy as described above and you tried to run the command aws ec2 describe-instances you probably found:

An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation.

Note: This assumes you have your credentials set up in the credentials file. Otherwise you will not even be authenticated!

This tells us we were in fact authenticated, but we were not allowed to perform this operation. Makes sense, we explicitly denied all that! But maybe you were smart and actually set up MFA. For instance using your phone and an authenticator app. Now you want to login but how ??

Not to worry, this is how: From the above options we ignore option 1. simply because that is insane. Options 2 and 3 are somewhat more doable.

To login we normally need our username, password and MFA token. That temporary token you get from your authenticator app. To login using the cli we use the access_key, the secrets_access_key and the session_token. And we can get the session_token using the command aws sts get-session-token. It takes a number of arguments:

The ARN of your MFA device. You can find it by running aws iam list-mfa-devices
Token code. You find this in your authenticator app.
Duration in seconds. You should set this to a sane value. We’ll get to this later on.

So the final command could be:

aws sts get-session-token --serial-number arn:aws:iam::111222333444:mfa/My_User --token-code 192952 --duration-seconds 72000

The answer will look something like:

Credentials:
  AccessKeyId: ASIAXXXXXXXXXXXXXXXX
  Expiration: '2022-03-14T07:07:36+00:00'
  SecretAccessKey: /19dlJorSomeothing+like_this4tFdicOkBJ1I
  SessionToken: IQoJb3JpZ2luX2VjEMv//////////wEaDGV1LWNlbnRyYWwtMSJIMEYCIQDUiP18GiJROFi19/TSRJBVG1ZYmpBCNEcZOmEUPEyXEQIhAPtHSOWW27fpk5fGyttOoHx1KKshS4LjbAEKb62hWWK1Ku8BCEQQARoMNjQwNzUzMjQ0NDk4IgzYyM9Lhvtb097v57oqzAHKmj784kousxlPGZIHt6Rkn5fN+FYMhQUquk+g7dauTngCIskvOxvgUjTUSIit6Fg8r2EcGMIKD+vdKNwdkchZkfvgxypioVmb1t70NTMrqgPoMjexBqiGVq9SxUKofxnhBKe5lphfseqpXUKk/QEXHqUVqrZ2P9XCeMyf/clT+Q4npL1NhMLsXXeS5fvoYMlCfZcXkTvfau3WEwlDVGEZQQiUP+yF9j4wdhtZX3saGMVUrqwUO6mEJphxnKavVoNAxljmtrM/cj2bBcYw+KC3kQY6lwEUWU3FDrG+R0IIjrV4lNP9E9Fx6JODMGktCoIgpfTewoHhli0IBEhrHU7U7900ACNAtMIsjT2ezFK2kNmhyH2p+OUORC56eeLVEv53+QHgjLNdbpUITHobFTrwjjmH+LAxzTdZh89BrNRyWsiOaFLRsJuCqpKliAJyuHldrRoA1DXRAojB37ihrZTz1h0ShML8sYE1OjaX

Great! … But how do we now use that ?

Environment Variables

As we saw, the second option was to use environment variables. Setting them on the command line is easy! Just run:

export AWS_ACCESS_KEY_ID=ASIAXXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=/19dlJorSomeothing+like_this4tFdicOkBJ1I
export AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEMv//////////wEaDGV1LWNlbnRyYWwtMSJIMEYCIQDUiP18GiJROFi19/TSRJBVG1ZYmpBCNEcZOmEUPEyXEQIhAPtHSOWW27fpk5fGyttOoHx1KKshS4LjbAEKb62hWWK1Ku8BCEQQARoMNjQwNzUzMjQ0NDk4IgzYyM9Lhvtb097v57oqzAHKmj784kousxlPGZIHt6Rkn5fN+FYMhQUquk+g7dauTngCIskvOxvgUjTUSIit6Fg8r2EcGMIKD+vdKNwdkchZkfvgxypioVmb1t70NTMrqgPoMjexBqiGVq9SxUKofxnhBKe5lphfseqpXUKk/QEXHqUVqrZ2P9XCeMyf/clT+Q4npL1NhMLsXXeS5fvoYMlCfZcXkTvfau3WEwlDVGEZQQiUP+yF9j4wdhtZX3saGMVUrqwUO6mEJphxnKavVoNAxljmtrM/cj2bBcYw+KC3kQY6lwEUWU3FDrG+R0IIjrV4lNP9E9Fx6JODMGktCoIgpfTewoHhli0IBEhrHU7U7900ACNAtMIsjT2ezFK2kNmhyH2p+OUORC56eeLVEv53+QHgjLNdbpUITHobFTrwjjmH+LAxzTdZh89BrNRyWsiOaFLRsJuCqpKliAJyuHldrRoA1DXRAojB37ihrZTz1h0ShML8sYE1OjaX

And that is it! Now if you run aws ec2 describe-instances you should get a pretty answer that shows you all your ec2 machines and their properties.

The credentials file

The other option is to tweak the credentials file. We can add a profile [MFA] there and list the data we got from the aws sts get-session-token:

[default]
aws_access_key_id=AKIAXXXXXXXXXXXXXXXX
aws_secret_access_key=someunreadablemumbojumboxxxxxxxxxxxxxxx

[MFA]
aws_access_key_id=ASIAXXXXXXXXXXXXXXXX
aws_ecret_access_key=/19dlJorSomeothing+like_this4tFdicOkBJ1I
aws_session_token=IQoJb3JpZ2luX2VjEMv//////////wEaDGV1LWNlbnRyYWwtMSJIMEYCIQDUiP18GiJROFi19/TSRJBVG1ZYmpBCNEcZOmEUPEyXEQIhAPtHSOWW27fpk5fGyttOoHx1KKshS4LjbAEKb62hWWK1Ku8BCEQQARoMNjQwNzUzMjQ0NDk4IgzYyM9Lhvtb097v57oqzAHKmj784kousxlPGZIHt6Rkn5fN+FYMhQUquk+g7dauTngCIskvOxvgUjTUSIit6Fg8r2EcGMIKD+vdKNwdkchZkfvgxypioVmb1t70NTMrqgPoMjexBqiGVq9SxUKofxnhBKe5lphfseqpXUKk/QEXHqUVqrZ2P9XCeMyf/clT+Q4npL1NhMLsXXeS5fvoYMlCfZcXkTvfau3WEwlDVGEZQQiUP+yF9j4wdhtZX3saGMVUrqwUO6mEJphxnKavVoNAxljmtrM/cj2bBcYw+KC3kQY6lwEUWU3FDrG+R0IIjrV4lNP9E9Fx6JODMGktCoIgpfTewoHhli0IBEhrHU7U7900ACNAtMIsjT2ezFK2kNmhyH2p+OUORC56eeLVEv53+QHgjLNdbpUITHobFTrwjjmH+LAxzTdZh89BrNRyWsiOaFLRsJuCqpKliAJyuHldrRoA1DXRAojB37ihrZTz1h0ShML8sYE1OjaX

To use this profile we have to specify it in the commands we use. So now it becomes:

aws ec2 describe-instances --profile MFA

And again: works like a charm! We can use this with any aws cli command, and we will have to if we want to use this profile.

The problem

Now then, what is the problem with this ? Well… Do you know anyone who wants to do this all the time ? Each time you open up a new terminal get the credentials and set all the environment variables? Or time and time again editing the ~/.aws/credentials file copy pasting stuff ? Well I don’t. So the results will be:

People start complaining. And with people I mean us too!
People will try to minimize the effort by putting the duration as long as possible, effectively reducing security.
If the above happens enough and pressure gets high enough you may be forced to undo this whole exercise and return to a situation with no enforced MFA. A very dangerous situation indeed!

The solution

Of course the solution is what it always is for these type of manual nuisances: automate them into oblivion!

So without further ado let get our hands dirty and bash away (Sorry for that, could not resist!). Find a good location to store bash scripts, and create a new file aws-set-auth.sh. In it we write:

#/bin/bash

TOKEN=${1}
if [[ -z $TOKEN ]]
then
  echo "please provide a session token"
  exit 1
fi
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset AWS_SESSION_TOKEN
CREDS=$(aws sts get-session-token --serial-number arn:aws:iam::640753244498:mfa/Chai_inQuisitive --token-code $TOKEN --duration-seconds 14400 --output json | jq .Credentials )
if [[ -z $CREDS ]]
then
  echo "Invalid token. Access Denied"
  exit 1
fi
AWS_ACCESS_KEY_ID=$(echo $CREDS | jq -r .AccessKeyId)
AWS_SECRET_ACCESS_KEY=$(echo $CREDS | jq -r .SecretAccessKey)
AWS_SESSION_TOKEN=$(echo $CREDS | jq -r .SessionToken)
echo "export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID" >> tmp.env
echo "export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" >> tmp.env
echo "export AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN" >> tmp.env
echo "creds are set in ENVIRONMENT VARS"

CRED_FILE_NAME=~/.aws/credentials
OUTPUT_FILE=()
readarray -t CRED_FILE < "${CRED_FILE_NAME}"
for LINE in "${CRED_FILE[@]}"
do
  OUTPUT_FILE+=( "${LINE}" )
  if [[ "${#OUTPUT_FILE[@]}" >1 && "${OUTPUT_FILE[-2]}" = "[MFA]" ]]
  then
    unset OUTPUT_FILE[-1]
  fi
  if [[ "${OUTPUT_FILE[-1]}" = "[MFA]" && -z $LINE ]]
  then
    OUTPUT_FILE+=( "aws_access_key_id=${AWS_ACCESS_KEY_ID}" )
    OUTPUT_FILE+=( "aws_secret_access_key=${AWS_SECRET_ACCESS_KEY}" )
    OUTPUT_FILE+=( "aws_session_token=${AWS_SESSION_TOKEN}" )
    OUTPUT_FILE+=( "${LINE}" )
  fi
done

printf "%s\n" "${OUTPUT_FILE[@]}" > "${CRED_FILE_NAME}"
echo "profile [MFA] updated ~./aws/credentials as profile"

Okay, so let’s have a look at what we have here. The first line just tells us we want this file to be interpreted as a bash file. TOKEN=${1} means store the first command line argument (after the script name!) in a variable TOKEN.

Then we check if that argument was given and if not we tell the user to provide it and we exit. If all is good we proceed to the next part where we unset the environment variables.

The next line is what we have seen before: Getting the credentials. In this case we pipe them into jq to parse them and store the result in a variable CREDS. This means that you need to have jq installed on your machine. If you don’t it will not work. But you probably should anyway because it is one of the most useful and crucial tools we have on the command line.

Then we do another check to see if all went well, if not, we tell the user and exit. Just like before.

After this we will use the credentials in the ways we read about before: exporting them as environment variables and editing the ~/.aws/credentials file. There is a bit of weird magic here. We first simply export the environment variables as we did before. Nothing strange about that, we know how that works. But then we also write these export instructions themselves to a temporary file. We will use these later! Just read on for now!

The last bit is probably the most difficult to read if you are not very familiar with bash scripting. In general what it does is this:

Create an empty output file
Read the credentials file
Go over it line by line and
- Add the line to the output file
- Check if the previous line was [MFA]. So we identify the part that we need to change.
- If so add the credentials.
Overwrite the credentials file with the output file.

Note: If you carefully followed this part you see why this script can handle any number of profiles in your credentials file but also why the [MFA] profile MUST be the last one there! There are probably ways to improve. There always are. But to balance complexity with usability brought me to this point if you want to improve on it: go right ahead!

Also please note that I have set the duration to 14400 seconds which is 60 sec * 60 minutes * 4 hours. For me 4 hours sounded reasonable. If you need it to be stricter, just turn it down to 1 hour or whatever meets your needs.

Using the script

So we are now ready to use the script. We run:

bash path-to-script-directory/aws-set-auth.sh <token>

Where we replace <token> with the token we get from the authenticator app. It will print out:

creds are set in ENVIRONMENT VARS
profile [MFA] updated ~./aws/credentials as profile

So nice! Right? But can we make it even easier? Sure! Because first of all this command is slightly longer then a lazy person likes, and more importantly if we have to look for that script and the path every time… well… Eeeuw!

Wrap it all in one simple command

Bash, zsh or fish whatever our shell of choice is, will allow us to call this script. This enables us to use a single, easy command wherever we are on the command line (provided that in that location we have access to the aws cli and jq of course, but most people would have those installed globally).

The normal and quick way to do such things is: Create an alias instruction in the .bashrc (or .zshrc or .fishrc) file. Something like: alias authaws="bash ~/script-directory/aws-set-auth.sh"

Doing this will give you no problems or errors. The only downside is that the script actually does not really work. Sad! The reason is somewhat technical in that bash will start a new session, execute the script there, set all the environment variables, and then finish the script and kill the session. Then returning to where you were it will happily say all was well. But you are left without proper environment variables set of course!

To get around this we adapt this solution:

In our aws-set-auth.sh script we write export instruction to a temporary file tmp.env
In .bashrc we write a function that calls the script, we use the command source to execute the tmp.env file and then remove that file

In our .bashrc this looks like:

alias authaws='_authaws(){bash ~/.config/nixpkgs/scripts/aws-set-auth.sh "$1" && . ./tmp.env && rm ./tmp.env;}; _authaws'

This looks a bit like tricky magic but it isn’t. Inside the alias authaws we define a function _authaws() which executes our script and uses the . instruction to source the file (see the mentioned answer on stack exchange). Then in the end it actually calls that function.

So now if you followed along you can just run:

authaws <token>

And we can do this in any directory, wherever we are. If it runs out, after the specified time, just rerun it and boom, that is it! Easy as eating chocolate and safe as milk!

Final thoughts

There is always a conflict of interests when it comes to easy of use or comfort on the one hand and security and or privacy on the other. This can lead to very ugly situations.

Some of you may recall a famous incident with a company named digiNotar that was doing great and went bankrupt in a month. This was NOT because they had not thought about security, or because they had no security measures in place. In fact they had very tight security in place. But it was just… so annoying! So they decided to make life easier with an extra cable and some settings…

This was not all that was wrong but the bottom line of the story is: Security measures are only useful when people follow them, and the chances of people following them are increased greatly if it is made easy for them. Therefore automation in some cases is not just to speed up things, make them easier or less error-prone, in some cases it actually increases security.

If you have any suggestions, comments or improvements reach out! Have a nice week!

How to learn new skills, without magic promises

2022-03-07T00:00:00+00:00

“Professor”, my colleague asks me jokingly, “Why don’t you write a blog about how I can learn Javascript.” That got me thinking, since I am eternally learning new things, always studying something: Music theory, playing an instrument, Spanish, Turkish, Math, Programming, and the list goes on. Have I learned anything about the process of learning? Did I distill any rules that I now apply either consciously or unconsciously?

The question of how to learn is asked by many, but only few of us actually listen to the answers, and even fewer of us put them into practice. That is because even if you do all the right things, the right ways (assuming we could clearly define those) it still costs us. It takes effort, time, energy, focus. And for us working people with a life that means it takes up space in a life that is already full and something will need to give. You can learn effectively, sure, but here is the first consideration: There are no shortcuts.

1. There are no shortcuts

Or maybe…

While there are things to avoid, and things to strive for it is good to be as realistic as possible with your expectations. You want to learn javascript in three weeks? Well of course you can spend three weeks learning javascript, but chances of you mastering it are very slim, and even that might still be an exaggeration. You will have to put in the effort, and understanding this before you start gives you a much better chance at achieving your goals.

2. Set yourself a goal

This is an obvious one, that you probably have seen everywhere. Goal setting. Now before you do this, let’s think a bit about why that would be useful and what kind of goals are helpful. In fact is it true that any goal is better than none? Any goal is a good goal ? I beg to differ. Goals are points at the horizon. You define them, so you have something to look at, a point on which to focus so that your energy and efforts are getting aligned with achieving that goal. And that means there are do’s and don’ts!

A goal that is too big, or in terms of our image of the “horizon”: too far.. is a goal that will not help you. It is a dream to keep dreaming about, but if you put it that far, it is past the horizon, you can not see it so it will not help you focus and thinking about the work necessary to achieve it will only paralyze and / or demotivate you.

A goal that is too small, or “too close”, will not be interesting enough, ambitious enough, to inspire you to do the work… and therefore the unintuitive result is that goals that like this are also hard to achieve!

A good goal should be in the middle, ambitious enough to scare you a bit, to make you feel excited and a bit anxious. At the same time it should be attainable, you should be able to imagine the necessary work and effort it will cost you. You should see it at the horizon, so you can direct yourself towards it.

And since setting goals, as well as achieving is a habit that is trained, the goals naturally become more audacious as you continue. There is no fun in staying safe folk !

3. Get support

As said in the beginning, it takes time, effort, and focus. So unless your life is empty and boring now, that means some other parts of your life will have to give that space. Trying to learn something new, without changing anything in your life is doomed to fail. Talk to your partner, talk to your colleagues, talk to your employer, talk to your friends. If people support you, and help you that is a great advantage, a super power you have got there. If they don’t then it is up to you to decide: either you still go for it, and others, while not supportive will at least know what you’re working for and why that changes the way you spend your time and energy. Or decide to just not do it.

I’ll say it again, trying to learn something without creating this space, without a way to even put in the necessary work is a really good way to cheat yourself and then be disappointed.

4. Get in the game

Live as if you were to die tomorrow. Learn as if you were to live forever. Mahatma Gandhi

You still did not get to do all before mentioned points, so you’re waiting for the right moment? Waiting until work gets calmer? Waiting until new year ? Be careful: you will do what you practice, always. So be afraid to practice procrastination! Just stop pretending you will get to it. And when you are serious about it, when you feel enthusiastic about learning something new: do it. Sort out a way to get started and get in the game. Tweak the other points as you go. Preparation is great, but it does not have to be perfect!

5. Immerse yourself

Learning something by doing it an hour a time, two times a week is very, very hard. You will find that you are wasting a lot of time and energy bringing back up what you already learned, what you were doing. If you want to learn effectively it is much better to take a period in which you spend as much time on it as possible. Don’t try to learn five things at the same time. Choose one, and immerse yourself as much as possible. This is why #3 is important!.

Immersion will also speed up learning, and that will do all kinds of wonderful things: You notice you’re improving which fuels enthusiasm. It can bring you to a point where the subject really becomes central in your thoughts. And while that may make you somewhat less attentive to other things, it fortifies your learning massively. Learning something is all about associations. The more associations you create in your brain, the stronger. As a simple example: Say you’re learning coding, and you already know how to play some guitar. At some point, if you may get to the point that you go to rehearsal with your friends and suddenly it occurs to you that the guitar solo is just:

const base_sequence = "Am Am G"
for (let i = 1, i<=8, i++) {
  if i%2 == 1 { // i the uneven rounds
    const play(base_sequence + "E7")
  else {
    const play(base_sequence + "Am")
    }
  }
}

Or what about the days:

public abstract class Day {
  boolean breakfast = true;

  boolean kidsGoToSchool;

  String todaysCook = "Mom";

  public void brushYourTeeth() {
    System.out.println("Kids come brush your teeeeeeeth!");
  }
}

public class Monday extends Day {
  boolean kidsGoToSchool = true;
  String todaysCook = "Dad";

  public static void main(String... args) {
    Day today = new Monday();
    System.out.println(today.breakfast);
    System.out.println(today.todaysCook == "Dad"); // is it ?
    System.out.println(today.kidsGoToSchool);
    today.brushYourTeeth();
  }
}

These type of associations and abstractions will come as a result of immersing yourself. It is then that your learning continues throughout the day, also when doing completely different things. It is a very powerful tool to accelerate and solidify your learning.

We all marvel at the way kids seem to learn things effortlessly and quick. Well, did you ever try to observe how many hours of their waking day are spend going back and forth through the alphabet? How many times the repeat their multiplication tables? Riding a bike? Right: Now imagine sitting down working through your lessons in Java for as long as you see your kids doing any of these. Would that make you learn faster? You betcha! (But yes, please observe #3 or I will get angry comments from abandoned loved ones!)

6. Fake it ‘till you make it

We generally know this is important, recommend it to others, but fail to apply it to ourselves. Did you ever meet anyone from another country that is now learning your language? “Just go out and practice. Talk to people, that is a great way to learn”. This is often the first advice given.

So what about when you learn something new, Docker for example, then what ? Here is one of the most scary and at the same time most effective suggestions I have for you: Get involved in Stack Overflow. And I mean involved. Create an account. (You would be surprised how many of us techies use Stack Overflow all day every day, but never bothered to make an account!). Before anything else: take 5 minutes to familiarize yourself with the community guidelines. You’ll be disappointed if you don’t! Then there are three things to do:

Search for questions on the subject you are learning (use the filters for that). You can filter on highest score and you will without exception find questions and answers that will teach you something new, dive just that bit deeper into the subject you are busy with!
Ask questions: Asking good questions is a valuable skill, so train it! Also in accordance with Stack Overflow’s “How to ask a good question” setting up a minimal reproducible example solves half of your problems without asking, and for the other half you will learn to view your issue from another side, and when you get answers they will not only fix the problem you had, but teach you something more general about the topic as well.
This is the most important: Filter on newest and find questions that you know the answer to. Or think you know. Or think you should know. And then go ahead and answer them. This again costs you some time and effort but I assure you it is worth it: You will research just a bit more, and force yourself to organise your thoughts, reaffirming what you’re learning again. And also: you will see that quite quickly you are able to actually help others and you find you know more than you thought! If you’re like me, then you’ll find it very motivating to continue.

On that same note: I recently upped this game for myself. Browse around on this blog and you will see a bunch of topics I wrote about even though most likely there are others who are more knowledgeable. Why? Because for one, I do not want to forget what I just spend days figuring out, and two after writing it all down I invariably learn more about it: To get all the steps and lines in place I want to make sure it is correct.. and why.

On many occasions we are satisfied with knowing something works. “Fake it ‘till you make it” makes sure we also dig into why it works. An added bonus is that I get to learn because of other people’s tips and suggestions!

Pro tip: There are many ways to put this principle into practice: Take up relevant tasks at work, create a project for yourself or with friends/colleagues, give a presentation … .. .

7. Have fun

Learning something only so that you can enjoy it, after you have mastered it makes no sense. I repeat: No sense, at all. Why would you enjoy it when you are good at it… when you do not even enjoy it when it is a shiny new adventure ? That does not happen. So allow yourself to have fun, and when it weighs on you: take a break. Don’t stop, just take a break and relax, or shift your attention on something on the side. You’re frustrated because the RxJs exercise just does not seem to work no matter what you do ? Relax, take a coffee read a it on Reddit, maybe someone writes something interesting about Reactive Programming in Java (Wow is that a thing?, interesting…!). This way, you keep immersed, you keep busy… and you can because it keeps being fun!

Conclusion

If you hoped I had some super trick to learn anything you want in a breeze you might be a bit demoralized now. If so: I do not apologize. I did not make it that way, I am just telling it the way I see it. If you’re serious about learning stuff then hopefully you found something here to inspire you to take the next step. It may not be easy, but at least it is simple; After all, if you want to get to that spot on the horizon, you can do it:

While not at goal:
  Take the step in front of me.

Terminal file managers and my vifm setup

2022-02-27T00:00:00+00:00

Another blog about some nice tools and tricks to make life easier. This time let’s look at terminal managers, why would we use them ? And I’ll dive into my setup of choice with Vifm. It was supposed to go in the AutomaticAlley series but it became a bit more elaborate so I’ll just put it out on it’s own.

This is another very opionated blog, and I feel that that is what it should be when it comes to these type of tools. If you use your laptop for anything more than just plain browsing and reading emails, I think you should have a terminal file manager. That is, somehow there exist people who do things like programming, sys admin or devops stuff and they do not touch the command line / terminal. This is crazy to me, I would not know how, and even less why … There are many discussions on this topic such as this one on reddit. Some comments include:

In my experience, people who rely on Git plugins in their IDE will often never truly understand what the plugin is doing under the hood. It provides an abstraction for you so you don’t have the understand the details. That is, until you run into some situation where the plugin can’t do what you want, or it’s not obvious how to do it via the plugin. This will happen eventually. /u/bears-repeating

This. The CLI changes far less often than GUI tools, is the same across any system and also works over ssh /u/CharacterUse

Some people comment that they prefer the UI / plugins, mostly for comfort or out out laziness:

UI because laziness. I know how to git on command line but for must use cases a UI is much more convenient. /u/YMK1234

And this sums up the basic distrust of UI’s that I absolutely feel. As said, it is opinionated:

I use the CLI. It’s the way I learned git and the one that feels most comfortable to me. I am always wary of a UI doing something other than what I intended. /u/MrSloppyPants

And obviously these discussions are repeated for many other things we use as techs like docker, editing configs, checking logs etc. For all of these we know that once we need to be on remote servers through SSH for instance, we’re thrown back on the command line.

Another very strong argument is that whatever you do on the cli can be automated. You’re writing scripts? Pipelines ? Just automate the commands you already know and love.

So I say: Why not make the command line the default, and in order to do that, take time to learn and configure it so it does exactly what we want. Because it seems to me that those are the two points that stand in the way for most people.

The time and energy to configure stuff
The time to learn things

And yes it is a matter of taste, but I personally enjoy figuring out things and then making sure I can navigate and use my system quickly and with easy (please no mousing and clicking… eeeeuw!).

One of the most important tools for that is the file manager. If you use a terminal file manager it works seamlessly with the terminal. Jump in, navigate, jump out. And most decent file managers do much much more, quickly create files, move or copy files, split into multiple panes to make that even easier, navigate using short cuts, opening files with predefined specific tools etc etc.

Also, most of us use keyboard shortcuts, right ? Why not leverage that idea into everything we can find? Apparently it helps us be more productive in our favorite tools, why would it not do the same everywhere else ?

I have so far used two of them: Ranger and Vifm. Both of them use Vi key bindings, which as a Vim user I like. I used Ranger first, and it was great, but then I found Vifm which is even more Vim-like, and I tried it, and stayed.

‘Nuff with the banter, show us the setup!

Okay okay, I have tortured you enough with my opinions. If you were not interested you probably never made it here, if you were interested in Vifm, you probably did not need any convincing. So thanks for indulging me. Now let’s get to the nerdy stuff!

If you’re spending your time on the command line, I strongly suggest looking into a terminal emulator you want like URXVT, Konsole or my current favorite Alacritty. Then choose a shell you like: good ol’ bash, my current love zsh, or the new fish on the block. And while you are at it: Setup a decent font. You’ll spend hours looking to find tiny mistakes in files etc, so make it easy on the eyes!

Good, that out of the way let’s get to Vifm. Installing it is easy on mac or linux. On windows ? Who knows ? You probably need a few more things first, and ten it should not be too hard.

Once you have it, just run vifm and tadaa, it opens up. And it is probably quite ugly. So time to tweak your theme! Choose a nice one here and let’s get going. I currently use zenburn_1. We locate the file .vifmrc. It is most likely in ~/vifm/.vifmrc. As with most of these terminal things, all important settings are in the rc-file. It has a section about color schemes:

" Selected color scheme
colorscheme zenburn_1

Reopen Vifm and check it out! Beautiful! Alright up next let’s navigate a bit. Use whatever you want: Arrow keys, hjkl-vi keys, or even, and we’ll see this more: hit : and just type cd some-directory. So easy! So now quit vifm, annnnd so sad! We are back at the home directory!

That is useless right ? No worries we can fix it. Unfortunately I did not find any native way to do that (let me know if you do!) so my solution was to write a simple function and an alias in my .zshrc. So open up your .zshrc or .bashrc , depending on your choice and add:

alias r = "vicd ./"

vicd()
{
  local dst="$(command ~/.vifm/scripts/vifm -- choose-dir -"$@")"
  if [ -z "$dst" ]; then
    echo 'Directory picking cancelled/failed'
    return 1
  fi
  cd "$dst"
}

This function stores the last directory you where at and on closing vifm directs you right there. The alias, r, well yeah, that is because I got used to it from my time using Ranger, and I did not want to change it, but you might use another alias obviously. Save it and reload your shell, either by closing and reopening the terminal or by running zsh. I will just use zsh from now on, if you use another shell, just replace that.

Now let’s type in our alias r and enter. Navigate to your prettiest directory and quit. And tadaa, you exit to the terminal exactly where you were. Great improvement right ? Another way to achieve this is by using the command :shell, which would make it easier to automate, but then you startup a lot of shells inside of each other and that gets messy!

By the way, if you are lazy like me: check out the bottom of the vifmrc:

" Sample mappings
" quit on q
nnoremap q :q<cr>

Nice! Just a q and as you see, the nnoremap is just like vim, key mappings, so we can leverage that for some quick navigation as well! In my case for instance:

nnoremap nx  :cd ~/.config/nixpkgs<cr>
nnoremap gd  :cd ~/Downloads<cr>

And by the way, to make life quick ‘n’ easy I have similar mappings in .zshrc:

alias gd = "cd ~/Downloads && r"
alias gh = "cd ~ && r"
alias nx = "~/.config/nixpkgs"
alias nxr = "nx && r"

This way navigating in Vifm, and on the command line itself, jumping in and out of Vifm happens fast and intuitively. You would have to make some aliases that make sense for you of course, keys you hit quickly and things that require no effort to remember.

Pro tip: remember the blog about sops? Well then nnoremap s :!sops %f<cr> should make your again a lot easier! But check if you do not have other mappings for s to avoid collisions.

Copying, renaming, creating files and other such fun stuff!

Now let’s have a look at some other basic things you might want to do. To copy a file: navigate, hit yy to copy to clipboard, navigate and p to paste. As expected. To delete ? Use dd (and p somewhere else for moving it of course) Rename? Hit cc, or A and see! And for making selections to perform these things? V.

In general on the Vifm site under “What users are saying…” it is said best:

“Thank you for this great Vifm tool whose slogan could be: “If you don’t know how to do it, don’t look at the docs, just think how you would do it in vi.”” Carlos Pita

Multiple panes

Something that I had to get used to for a bit, but resulted to be incredibly useful were the dual panes. Just hit ctrl-w v to open the second pane, and crtl-w o to close it. (Just like…). Vifm will remember your last choice in this. If you have two panes switch between them with space. Now imagine if I want to copy a few things from Downloads to ~/.config/nixpkgs/scripts: I hit r anywhere. Then gd, some ups or downs to get to my file, yy, space, nx, navigate, p, space … rinse and repeat. Now to read this, is cumbersome. But to do this, once you have a feel for these shortcuts (and if you are a Vim user this costs you exactly zero effort) is really really quick and intuitive. No menus, no mouse, no clicking, chosing options from secondary menus nothing. And then you want to edit the file? Just hit enter and go.

That is… if you set up the correct programs to open files…

Setting up default programs to open your files

Again we open .vifmrc. Find the section on opening files, and adjust to your liking. I for instance have:

filetype *.wav,*.mp3,*.flac,*.m4a,*.wma,*.ape,*.ac3,*.og[agx],*.spx,*.opus,*.aiff
        \ vlc %f

filextype *.pdf
        \ evince %c

Everything that is not matched against any definitions will be opened with your default Vim. If you would like to use something else, like VI or even something else completely like codium you can edit this line in .vifmrc:

set vicmd=vi

File Preview

Another nice thing to have in a file manager are file previews. This is also possible, and configurable to suit your wishes in Vifm. To start: Open Vifm, and make sure you have two panes. Then hitting the w key will toggle the preview. Nice right ? I agree! So anything that is previewable (if only that were a word!) in Vim should be immediately working now.

But for some files it is a bit more involved to get the previews working nicely. And luckily others have done some work for us, and we need only extend it a bit. For images we find here in the docs a suggestion to use Ueberzug. The link to the reddit shows some scripts that the author of Ueberzug, /u/seebye wrote. It also has a link to youtube to a video by DistroTube that give a little more explanation, and where in the comments of that video, we find a variation of those scripts to support videos as well. I then took that and extended it to be able to preview pdfs as well. Here is how to do it:

Create two files in .vifm/scripts (by the way, finding things in Vifm is just like finding them in Vim right, so use the / as you would!):

vifmimg
vifmrun

Both will contain bash scripts: vifmimg

#!/usr/bin/env bash
readonly ID_PREVIEW="preview"

if [ -e "$FIFO_UEBERZUG" ]; then
    if [[ "$1" == "draw" ]]; then
        declare -p -A cmd=([action]=add [identifier]="$ID_PREVIEW"
                           [x]="$2" [y]="$3" [width]="$4" [height]="$5" \
                           [path]="${PWD}/$6") \
            > "$FIFO_UEBERZUG"
    elif [[ "$1" == "videopreview" ]]; then
        [[ ! -f "/tmp/$6.png" ]] && ffmpegthumbnailer -i "${PWD}/$6" -o "/tmp/$6.png" -s 0 -q 10
        declare -p -A cmd=([action]=add [identifier]="$ID_PREVIEW"
                           [x]="$2" [y]="$3" [width]="$4" [height]="$5" \
                           [path]="/tmp/$6.png") \
            > "$FIFO_UEBERZUG"
    elif [[ "$1" == "pdfpreview" ]]; then
        [[ ! -f "/tmp/$6.jpg" ]] && pdftoppm -singlefile -jpeg  "${PWD}/$6" "/tmp/$6"
        declare -p -A cmd=([action]=add [identifier]="$ID_PREVIEW"
                           [x]="$2" [y]="$3" [width]="$4" [height]="$5" \
                           [path]="/tmp/$6.jpg") \
            > "$FIFO_UEBERZUG"
    elif [[ "$1" == "clear" ]]; then
        declare -p -A cmd=([action]=remove [identifier]="$ID_PREVIEW") \
            > "$FIFO_UEBERZUG"
    fi
fi

This basically says, if you have Ueberzug ready and you call this script with an argument of draw then draw the preview of a image file ($6) with the dimensions given in the rest of the arguments ($2 … $5). If the argument is pdfpreview or videopreview then make an image of that, and draw that. If the argument is clear then clear up the screen.

vifmrun

#!/usr/bin/env bash
export FIFO_UEBERZUG="/tmp/vifm-ueberzug-${PPID}"

function cleanup {
    rm "$FIFO_UEBERZUG" 2>/dev/null
    pkill -P $$ 2>/dev/null
}

rm "$FIFO_UEBERZUG" 2>/dev/null
mkfifo "$FIFO_UEBERZUG"
trap cleanup EXIT
tail --follow "$FIFO_UEBERZUG" | ueberzug layer --silent --parser bash &

vifm --choose-dir - "$@"
cleanup

This file basically runs Vifm with Ueberzug prepared. I adapted it in this line: vifm --choose-dir - "$@" to be able to work with our vicd() function. So that means we need to make two more small changes. But before that: Save these files and make them executable by running:

chmod +x vifmrun
chmod +x vifmimg

Now let’s edit .zshrc and make sure we have:

vicd()
{
  local dst="$(command ~/.vifm/scripts/vifmrun "$@")"
  if [ -z "$dst" ]; then
    echo 'Directory picking cancelled/failed'
    return 1
  fi
  cd "$dst"
}

This way we call the vimrun script with that same argument, whenever we callvicd() which again is whenever we call r or whatever your personal alias was.

Finally we adapt the sections in vifmrc:

fileviewer *.pdf
           \ vifmimg pdfpreview %px %py %pw %ph %c
           \ %pc
           \ vifmimg clear

fileviewer *.bmp,*.jpg,*.jpeg,*.png,*.gif,*.xpm
           \ vifmimg draw %px %py %pw %ph %c
           \ %pc
           \ ~/.vifm/scripts/vifmimg clear

           fileviewer *.avi,*.mp4,*.wmv,*.dat,*.3gp,*.ogv,*.mkv,*.mpg,*.mpeg,*.vob,
        \*.fl[icv],*.m2v,*.mov,*.webm,*.ts,*.mts,*.m4v,*.r[am],*.qt,*.divx
        \ vifmimg videopreview %px %py %pw %ph %c
        \ %pc
        \ vifmimg clear

The fileviewer sections are for the previews, where the filetype and filextype are for actually opening the files. Lets try it out! And boom there you go, previews of PDF, images and videos all working nicely!

Bonus: add Vifm to Vim!

Sounds like inception right ? Well kind of, but then in an awesome way. I have tried things like nerd tree and never really got the hang of it, but now we can use Vifm, exactly as we configured it with all it’s magic directly in Vim. For this we use a plugin called Floaterm. It spins up a floating terminal window, and we can tell it what we want to do with it. And what do we want? V.I.F.M ! Vifm! (Ah imagine the dance that goes with that!). In our vimrc (mind you not vifmrc this time of course!). We add configuration for it:

"Floaterm
let g:floaterm_opener = 'edit'
vnoremap <leader>m :FloatermNew --autoclose=2 vifm<CR>
nnoremap <leader>m :FloatermNew --autoclose=2 vifm<CR>

I use space+m to open the vifm window and set it to edit meaning if I choose to open a file it just opens it in a new buffer in Vim with out splitting into two panes which is the default. Obviously you might have to add something to load the plugin and maybe use a different configuration, but all of that can be found on Floatterm’s page.

Now if we put all of that together we can get something like this. Pretty sweet right ? Let me know what you think, and if you have suggestions I am always ready to ear them!

#AutomaticAlley #2 - FZF and Ripgrep: fuzzy find files infuriatingly fast!

2022-02-21T00:00:00+00:00

E voila: the second in the series, doubtfully dubbed #AutomaticAlley. It’s where I share small bits and bites of scripts and other tiny tricks to automate away boring and error prone stuff. The things we keep doing all the time, the things that annoy us that we need to to automate especially if we want to keep working on the command line a sane practice. Today let’s have a look at FZF and Ripgrep to fuzzily find files!

“If you just remembered where you put it, you wouldn’t have to look for it!” Every mom ever

Now that would be nice, but it is simply not feasible, and we have accepted this a long time ago. What can we do ? Well we’ll just find our file, and then open it! Let’s try this the good ol’ bash way: To find something with a lot of matches: open the terminal in your home directory ~/ and run:

find . | grep bash

Or something with few or no matches:

find . | grep bashIsGreatButNotTooFast

It takes a while, and if you have bad luck you’re trying to find something in “one of those sytem directory thingies” … was it /sys ? /run? /etc ? Yah one of those right? So you try:

cd /
find . | grep bash

Now you really need to be patient!

Enter FZF the Fuzzy finder and Ripgrep!

Some of us are not that patient, so they did something about it and wrote FZF. It is a command line fuzzy finder. That means it will find what you are looking for, even if you do not exactly know what you’re looking for!

To install it there are a bunch of options, so choose whatever fits you best! If you use Vim I definitely recommend installing the Vim plugin as well. Because as they in the README:

Why you should use fzf on Vim

Because you can and you love fzf.

Then let us also install Ripgrep. This will enable us to search content inside files. Again an easy process, just use whatever package manager you use for your system.

Now Ripgrep and FZF work awesomely together so lets tweak some settings here. There are a lot of options, and it has many things you might want. What I wanted was:

A sensible default to search and find files
A way to find files by searching for content
The search results displayed in a nice way
A way to quickly find some file and open it in Vim
Ability to use FZF in Vim to find files
Ability to find stuff inside files in Vim

Number 2 I almost do not use, because I generally have an idea of where to look, and I generally look for stuff when I am working on a project. For instance I will have a code file open in Vim and then think, “Wait, where did I set that variable?” And then go for option 6.

For this I added to my .zshrc (if you use bash or fish use the appropriate rc file):

export FZF_DEFAULT_OPTS="-m"
FZF_DEFAULT_OPTS+=" --color='light'"
FZF_DEFAULT_OPTS+=" --bind 'ctrl-/:toggle-preview'"
FZF_DEFAULT_OPTS+=" --preview 'head -500 {}' --height 80%"
FZF_DEFAULT_OPTS+=" --preview-window=up:40%:hidden"
FZF_DEFAULT_OPTS+=" --height=80%"
FZF_DEFAULT_OPTS+=" --layout=reverse"
FZF_DEFAULT_OPTS+=" --border"
export FZF_DEFAULT_COMMAND='rg --files --hidden --follow --no-ignore-vcs'
export FZF_CTRL_T_COMMAND="$FZF_DEFAULT_COMMAND"

So a bunch of default options for the preview screen, actually those are for the use in Vim. And a sensible default. Use ripgrep, look for hidden files as well, follow symbolic links and do not ignore vcs files.

I also add a few custom aliases:

fzfi = "rg --files --hidden --follow --no-ignore-vcs -g '!{node_modules,.git}' | fzf"
o = "x=$(fzfi); if [[ ! -z $x ]]; then vim $x; fi"
nx = "~/.config/nixpkgs"
nxo = "nx && o"

An alias fzfi to use the default FZF command except that it does not search in node_modules or .git directories. Generally I am not looking for files there, because they are generated or managed for me, so I do not not need to edit those. Then o to look for a file using fzfi, and if I find it, open it with Vim. Finally an example nxo (there are many more examples, for each directory I commonly use and want to go quicly to edit files) it refers to nx which changes to my directory of nix configurations, and then runs o.

Configs in Vim

Now for the ones interested in using Vim, I added these configs in .vimrc:

    "search
    nnoremap <C-g> :GFiles?<CR>
    nnoremap <C-h> :History<CR>
    nnoremap <C-l> :Rg<CR>
    nnoremap <C-b> :BLines<CR>
    nnoremap <C-p> :All<CR>
    nnoremap <leader>b :Buffers<CR>
    "set grepprg=rg\ --vimgrep\ --smart-case\ --hidden\ --follow
    let g:fzf_preview_window = ['up:50%:hidden', 'ctrl-/']
    command! -bang -nargs=*  All
      \ call fzf#run(fzf#wrap({'source': 'rg --files --hidden --no-ignore-vcs --glob "!{node_modules/*,.git/*}"', 'options': '--expect=ctrl-t,ctrl-x,ctrl-v --multi --reverse' }))

    "============ copied from source: https://github.com/junegunn/fzf.vim/blob/master/plugin/fzf.vim =========
    "============ can possibly be removed  after update ======================================================
    command! -bang -nargs=* Rg
      \ call fzf#vim#grep("rg --column --line-number --no-heading --color=always --smart-case -- ".shellescape(<q-args>),
      \ 1, s:p(), <bang>0)

    command! -bang -nargs=* History
      \ call s:history(<q-args>, s:p(), <bang>0)'])

    command! -bar -bang -nargs=? -complete=buffer Buffers
      \ call fzf#vim#buffers(<q-args>, s:p({ "placeholder": "{1}" }), <bang>0)

    function! s:p(...)
      let preview_args = get(g:, 'fzf_preview_window', ['right', 'ctrl-/'])
      if empty(preview_args)
        return { 'options': ['--preview-window', 'hidden'] }
      endif

      " For backward-compatiblity
      if type(preview_args) == type("")
        let preview_args = [preview_args]
      endif
      return call('fzf#vim#with_preview', extend(copy(a:000), preview_args))
    endfunction

    function! s:history(arg, extra, bang)
      let bang = a:bang || a:arg[len(a:arg)-1] == '!'
      if a:arg[0] == ':'
        call fzf#vim#command_history(bang)
      elseif a:arg[0] == '/'
        call fzf#vim#search_history(bang)
      else
        call fzf#vim#history(a:extra, bang)
      endif
    endfunction

    "=========================================================================================================
    "=========================================================================================================

As you can see most of this is copied from the docs. I also found some useful ideas in this blog. Basically in here I chose some key shortcuts that made sense to me, like ctrl-p to search files, ctrl-l to search contents (lines) etc. Also if looking for contents (contentctrl-l) then we can toggle the preview with ctrl-/

All in all with this you can now do all of those five things summed up above, and you get number three for free! Have a look and let me know what you think!

Angular - A minimalistic approach to runtime configurations in docker containers

2022-02-13T00:00:00+00:00

Build your Angular app, run it inside a container and move it across environments. How do we manage our runtime configurations?

Build once, deploy everywhere! This idea relies on the ability to inject configuration settings at runtime. In backend applications there is a way of working for this, and most modern frameworks such as spring support these out of the box. They rely on reading environment variables. But what about the frontend, where code is executed in your mom’s browser at home and such environment variables are therefore not available? This was my weeks exploration, and is the topic of this blog.

Tip: If you like to look at the code while reading along you can clone the repo.

Angular has a nice system of environments and settings when it comes to build time. You specify the environments you want, and all their settings, and then you build using a flag to build for that environment:

ng build --configuration production

This allows you to create a build for local development, and a different build for the production environment. Things like application urls or database connection strings are almost certainly going to be different for these environments. Now we can also run a different build for our test, acceptance and staging environments. Problem solved ?

ng build --configuration test

Not quite!

Build once, deploy everywhere and runtime configs

There are many opinions, and strategies for building applications, making them available on different environments for various types of testing, or production use. Running a new build for each environment is definitely an option, and I know people who are in favor of that. I disagree. In my opinion and experience it has a few important drawbacks:

It makes testing results less reliable. “Only config changes, it should work exactly the same”. The fact that it should work, but actually is not guaranteed is why testing is an important effort in software development. So while in a vast majority of cases this is true, once in awhile you encounter unexpected side effects of running a new build.
It makes the pipelines from local development all the way to production slower. Extra builds means extra time.

There might be more reasons, such as the need to store more artifacts etc, but these would be my most important ones. As said, feel free to disagree with my in the comments or over coffee! I like the idea of “Run once, build everywhere”, also known as build, release, run in the 12-factor app methodology.

So what we need is a way to have different settings available at runtime. For the purpose of this blog I will assume we have an Angular app, that runs inside a docker container, at least on environments other than the local machine.

Available solutions in Angular

So what are the options we have in angular? Well, surprisingly there is no simple standard way to do this. On the other hand, Angular is a very powerful framework so it is very possible to create solutions with the tools available. A quick search on the internet will return explanations on how to do that. Most involve more or less the following:

Make the configurations available in the /dist folder after building without minifying.
A javascript function (ususally one of those cute “self invoking” functions) that makes those variables available globally. For Angular, this means setting them on the window object.
An angular service to read the variables, and this service can then be injected wherever we need access to the variables
A way to provide the service and run it’s method to load the variables before/while bootstrapping Angular. Usually it involves APP_INITIALIZER and patterns like service factory and service provider.
…some have even more steps for nice things such as custom error handling etc…

Now this is definitely doable, and possibly an elegant solution. However, to me it seems like a lot of boiler plate code, and something as simple as reading some variables there should not be that hard!

A minimalistic approach

So in addition to all the previous variations on the solution, I will here offer a different, simple version. I asked myself “If I do step 1. and 2. and all variables are available globally, why do we need step 3-x?” Well, we do not. We might want to, but we don’t need to. So lets look at the first steps and make them as easy as we can.

Let’s start by making some config files. We to make these to be available to read and manipulate after compilation. Because we need them to be copied as-is and not compiled and or minified we write them as simple old javascript files. We create a folder src/config and in it we create files with some sample configurations:

config.js

__config = {
  apiUrl: 'http://localhost'
}

config.test.js

__config = {
  apiUrl: 'https://test.amazing-app.com'
}

config.prod.js

__config = {
  apiUrl: 'https://amazing-app.com'
}

Next we edit angular.json in the project’s root folder. It has a section about assets under projects architect build options, here we specify all files and folders that we want copied as-is to the build output folder (by default: /dist). So we make this part look like:

    "assets": ["src/favicon.ico", "src/assets", "src/config"],

To check if all is well so far we run npm run build (or ng build). We go into the /dist folder and next to some minified js files we should find the assets folder as well as our new config folder with containing the files we just created.

So far so good, time to make them available to us! For this we just use the oldest trick in the javascript book. We load a script in our HTML! So edit index.html in our main folder and load the script <head> section. It is important to put it there right at the top. The browser will load whatever is in that html file from top to bottom, and we want it to load our settings before doing anything with Angular because otherwise our app will not load or work correctly! So go ahead and edit index.html to look somewhat like this:

<!DOCTYPE html>
<html lang="en">
  <head>
    <script src="config/config.js"></script>
    <meta charset="utf-8" />
    <title>ExampleAngularRuntimeConfig</title>
    <base href="/" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <link rel="icon" type="image/x-icon" href="/favicon.ico" />
  </head>
  <body>
    <app-root></app-root>
  </body>
</html>

If we use Angular with typescript we will have to let it know that the variable __config exists. We do this by specifying declare let __config: any; This really should be all that is necessary to load your configs at runtime! You can access your apiUrl by using __config.apiUrl.

Note: I have seen people use variable name __env and claim that it is a special variable. It is not. It is just a variable with a funky name. Call it pancakesWithHotsause if you want and you’ll see it work as well. The usefulness of starting the variable with __ is mostly to avoid conflicts. It is unlikely, and should be unlikely, that we’ll create a variable in our normal code somewhere called __something, so in that way it helps.

I feel that it might be smart to be a bit more specific about our data structure. If you work together with more people, and or your code grows etc, it is a good practice to use stricter typings to make your code more maintainable. If you use a code editor that is a bit smart you also get type-ahead support this way. All in all just basic coding hygiene. If you also feel that way you can go ahead with me and create an interface for the configs/ environment:

src/app/environment.interface.ts

export interface Environment {
  apiUrl: string;
}

And we change the declare statement to declare let data: Environment If you have cloned the repo you will find in src/app/app.component.ts:

declare let data: Environment;

@Component({
  selector: 'app-root',
  template: '<div></div>',
  styleUrls: ['./app.component.scss'],
})
export class AppComponent {
  apiUrl: string = __config.apiUrl;
}

So for now, we do not do anything too complex with our configuration variable, in fact we don’t do anything. Just throw it on the screen!

Running in a docker container and changing the config at runtime

Now we still have a last thing to do and that is to see that we get a feel for how we can change the config at runtime. As far as we have seen now, we have not been changing anything!

For this I will assume we run our app inside of a docker container. Today I will not dive to deep into docker, but just to give a bit of info about the setup in the repo: We have a dockerfile, in our case just the default Dockerfile. This is where we specify the build of our app. It is used to build a docker image. That image is what we want to build once, move along all our environments and maybe even ship to customers.

Dockerfile

FROM node:14-alpine as build

# copy code and run build
WORKDIR /app
COPY ./*.json ./
COPY ./src ./src
RUN npm install && npm run build

# run app with nginx
FROM nginx:stable-alpine
COPY --from=build /app/dist/example-angular-runtime-config /usr/share/nginx/html
COPY ./default.conf /etc/nginx/conf.d/default.conf
WORKDIR /start
COPY ./start-app.sh .
CMD [ "sh", "start-app.sh" ]

If we take a closer look we see some instructions for a base image, copying files running a build, and then putting the result (everything in /dist) inside a nginx folder so it can be served. But where does it take our environment variables into account? In the last step. The CMD instruction is where the image receives what is the command to run by default when sinning up a container. In this case, and that is a fairly common pattern it is running a small shell script start-app.sh. Lets have a look:

#/bin/sh
if [ "${ENVIRONMENT}" = "prod" ]
then
  echo "starting app prod"
  mv /usr/share/nginx/html/config/config.prod.js /usr/share/nginx/html/config/config.js
elif [ "${ENVIRONMENT}" = "test" ]
then
  echo "starting app test"
  mv /usr/share/nginx/html/config/config.test.js /usr/share/nginx/html/config/config.js
else
  echo "starting app default / local"
fi
nginx -g "daemon off;"

Also pretty straight forward: We check for an environment variable called ENVIRONMENT and see if it is either prod, test or anything else. Then if necessary it overwrites config.js with the prod or test version. When that is done it instructs nginx to serve our app. Pretty neat, now let’s see it in action!

Let’s build once, and run …

To build the image run:

docker build -t amazing-app . # Mind the dot in the end!

We have now built the image, and gave it a tag amazing-app. You can see it by listing the images you now have: docker image ls.

Now we want to run a container based on this image, this is our runtime where our configs come into play! The plot thickens! When you’re ready for the magic, run:

docker run -p 4200:4200 -d --env ENVIRONMENT=local amazing-app

This command tells docker to run a container, bind the port 4200 in the container to 4200 on our machine, run in detached mode (so we can easily close it and it does not block our terminal), and pass it an environment variable ENVIRONMENT=local. Lastly we specified the image that we want to use to create our container: amazing-app.

Now go to your favorite browser and navigate to http://localhost:4200 and tadaa! Is it not amazing ? Well I am sure you can build even more amazing apps than these, but it does what we want!

Let us stop our container and see some more wonders:

docker stop $(docker ps -q)

This will stop all containers that we have.

Now lets imagine that we were a pipeline and wanted deploy our app to the test environment then we would just use the same image and simply run:

docker run -p 4200:4200 -d --env ENVIRONMENT=test amazing-app

Try it and check your browser! So easy and so powerful!

Tip: If you are disappointed, try a hard refresh (shift+F5) or an incognito window!

While we know that all out solutions can be useful and have their place, many times a bit more minimalistic solutions are preferable. It keeps our projects small, and easy to maintain. Also adding code when necessary is a normal task that is never forgotten (duh) while removing unnecessary code is much more difficult. So to keep technical debt to a minimum, be sure to avoid over engineering!

I hope you liked this small demo. And if you see anything that could better, or maybe you if just completely disagree with my opinions on devops and build once, deploy everywhere then please feel a warm welcome to leave a comment or reach out!

#AutomaticAlley - Kill any program anytime with ease

2022-02-07T00:00:00+00:00

This is the first in a series to come, doubtfully dubbed #AutomaticAlley. It’s where I will share small bits and bites of scripts and other tiny tricks to automate away boring and error prone stuff. The things we keep doing all the time, the things that annoy us that we need to to automate especially if we want to keep working on the command line a sane practice.

In today’s episode: How to kill any program anytime with ease. For this we create a simple shell alias that kills any program that looks like the one that is bothering us. No need to go to activity screens or whatnot. In fact for those of us using tiling window managers or who otherwise generally live on the command line we just stay in our homes!

We use the pkill command for this. It is based on the loved grep command that finds anything by regex or name. Then from the manual (man pkill) we learn:

pgrep looks through the currently running processes and lists the process IDs which match the selection criteria to stdout.
[..]
pkill will send the specified signal (by default SIGTERM) to each process instead of listing them on stdout.

So we learn that pkill is based on pgrep which is a command that will list processes that are running currently. Now we need to be sure that the one we want is easily matched, and we do not want to have to remember exact spelling of even type all of it. After all we want to be as lazy as we can!

Let’s use pgrep to get the right matches

By the way: If you are using a mac and pkill or pgrep are not available you can install proctools: brew install proctools

Let’s ask the system what we have: pkill --help. We find:

-i, --ignore-case         match case insensitively

Yes! We’ll take it. We’re not particular about case! So let’s try it! We run spotify. And then type:

pgrep -i spotify

pgrep -i SpoTify

pgrep -i spoti

pgrep -i otif

Looks good right ? It matches all! But wait… when we run some selenium tests for instance we need to run the selenium server. Usually in a repo where we use it we run npx selenium-standalone install && npx selenium-standalone start or similar,generally automated away in a nice npm <selenium-command> in package.json. Now these have a tendency to clog up the system, or hang in zombie state somehow. So lets see if we could pgrep and thus pkill it.

pgrep -i selenium

Nothing! Nothing!? Unfortunately it does not work. So lets see what else we have:

-f, --full                use full process name to match

That sounds good. Full whatever is probably more than what we had! And indeed:

pgrep -i -f selenium

And yes there we have something! If we would like to compare it, the -a option will display the full command line:

pgrep -i -f -a selenium
pgrep -i -f -a spotify

So now lets kill ‘em!

pkill -i -f selenium
pkill -i -f spotify

And if you feel like it is too silent now, and you like more feedback, pkill -h tells us:

-e, --echo                display what is killed

So now we have:

pkill -i -f -e selenium

Let’s now make it easier! Simple is good!

Who wants to remember all of that, or look it up every time? Not us smart (lazy) kids! So now, depending on your shell we have something like a .zshrc or .bashrc usually in our home directory. In it we want to make an alias, so we add a line it:

alias dieSpotify='pkill -i -f -e spotify'

Save it and reload the shell, or open a new terminal. Then just run dieSpotify. And yes! But wait… do we have to do that for each program we might ever want to kill ? That is crazy! And of course the answer is no, we automated it, now we parametrize it! We can pass an argument to the alias. So we change the line into:

die='pkill -i $1 -f -e'

The $1 part here refers to the argument we pass so now we can kill everything at our convenience. Go Rambo!

die spoti
die chrom
die KeePass
[..etc..]

One time effort, but now hanging programs, or programs that clog up the system with processes that do not get closed properly etc. Bam… Kill them with ease and comfort :)

Was it useful ? You have a better plan ? Let me know in the comments!

Setup SOPS with AWS KMS and Terraform to encrypt your secrets in git

2022-01-31T00:00:00+00:00

In this post we’ll explore Mozilla SOPS to manage configuration secrets.

There are a number of things to take into consideration, and there are different possible solutions each with their own advantages and problems. Some things however are important and should be possible in any solution:

secrets should never be stored in plain text in git (duh!).
it should be possible to specify who has access to which secrets. For instance some members may have access only to dev secrets while others have access to all secrets.
access to the secrets should be revocable quickly and easily.

Basically there are two different approaches:

Keep the secrets in an external system like Hashicorp Vault, Azure Key Vault or AWS SecretsManager and find a way to inject them at runtime. Preferably the agent running the app has some rights based on identity for this. This means we need to setup identity management as well as a vault to keep the secrets. This adds overhead and some risks because it is not always directly clear what the potential risks are. But the advantage is that we can rotate secrets easily, and also by identity management assign or revoke access as the situation demands.
Keep the secrets encrypted in git. For this we need to manage access to the encryption key(s). The obvious advantage is that we do not need an external vault, and this may drive development speed and reduces complexity somewhat on that end. The disadvantage is that the deployment secrets are now coupled to the code in git, and in case secrets need to be rotated it involves a new deployment. If you have some gitops and automated pipelines in place this might not be a big problem, but in some situations this is a more serious issue. It also means that git bisect will no longer reliably return an answer.

In this blog post we’ll look at the second scenario. Not because it is inherently better, but because there are valid usecases for it and there is a really nice solution in the form of Mozilla SOPS. SOPS supports encrypting files as binaries, but apart from that it also has the great feature of encrypting only the values of config files as long as they are in the correct formats, being json, yaml, .env or .ini.

I’ve made a demo project on git to explore how we can set up this solution using terraform to provision the AWS users and keys, and then we configure and use SOPS to encrypt/decrypt a few example configuration secrets.

Let’s set up the repo and initialize terraform!

There is a git repo here: ProofOfPizza. There are some prerequisites to be able to follow along:

You need to have a valid AWS account.
You need to have the AWS cli installed and
You need to have credentials configured (usually in ~/.aws/credentials) for a user (preferably NOT the root user but an IAM user with sufficient rights).
You need to have terraform installed.
To follow along please do not clone the repo but fork it.

To install the repo simply go to the repo in your browser and click fork. You are then taken to your own github account and will there see your fork of the repo. Then you just click on code and copy the link. In your terminal type git clone <link> where should be replaced by the link you just copied.

Then run cd terraform && terraform init. This should get you the providers and initialize the terraform project. If you run into issues here please verify your terraform version with terraform --version. In the file providers.tf the required version is specified: required_version = "~> 1.1.0". Match this with your version if necessary.

We also see this block specifying the aws provider:

provider "aws" {
  region  = "eu-central-1"
}

If you want you can specify any aws region you want. If you have MFA set up for your user then you have some additional steps to get a token and set up a profile for that user. (I should write another quick post about that). Also if you have multiple users configured in aws-cli you can specify the profile you want to use. You can do this by adding profile = my_profile

Deeper look into the terraform code to create the AWS users and keys

Now to use SOPS we need an encryption key. SOPS is compatible with many backends, but for now we will stick to one. However, SOPS supports using multiple backends simultaneously out of the box. We will create keys in AWS Key Management Service (KMS). This key can be rotated automatically, and with a policy we can specify which users have which rights to it. This way we can also easily revoke the rights if a team member leaves the project for instance. So for this example we will need three users:

A test user with rights to the test key to use sops on the test secrets
A prod users with rights to the prod key to use sops on both prod and test secrets
A KMS key admin user who has rights to update the keys if necessary. AWS will not allow us to create a key without someone attached to it that could perform these actions.

The terraform code’s layout is pretty straightforward:

providers.tf defines the AWS provider
outputs describes all the outputs that our terraform operations will provide. We can later see them in the console using the terraform output command.
main.tf contains the definitions for the resources. It uses the terraform for_each syntax to loop over value definitions to reduce code duplication.
locals.tf contains those value definitions. The values for the three users and the two keys we create are found here.
policies/*.json are policy documents for the users. Here the allowed actions and resources are specified. To avoid circular dependencies the users have rights on all keys * and then the keys have only specific users assigned to them. Otherwise we would have to first create the keys, which would require us to first create the users, which would require us to first create the keys … well you get the point… or actually, that point never gets fully made :)
key-policy.tpl leverages the .tpl terraform template format. It is basically a json file with interpolations. We use the terraform jsonencode function to interpolate more complex values than just strings. In this case lists of users:

  "Principal": {
"AWS": ${jsonencode(key_users)}
},

If you want to learn more about all the options you can set for keys, and what they mean you can read more in the AWS docs. To learn more about how to specify these settings in terraform read the docs for iam_users and kms_keys in terraform.

Terraform apply and creating the aws resources

The KMS keys have a policy that defines who has which rights on them, this also goes for the actions create, update and delete key. For this reason we should add the IAM user that we use for terraform to the KMS keys. Otherwise terraform will be unable to manage them! In locals.tf we see:

locals {
  my_aws_user = "arn:aws:iam::12345678999:user/iam_user_name"
}

You will have to replace it with your user’s ARN. You can get it from the cli by running aws sts get-caller-identity.

Once you feel ready, we can run a speculative plan by running terraform plan. If it all looks good to you you can run terraform apply and when asked type yes to confirm. After a few moments terraform will inform you of what resources have been created.

Setup profiles for AWS cli

Now we can use these users to test SOPS, but for this we need to setup the aws cli because SOPS relies on the users being present in the aws cli configuration. AWS cli credentials are kept in a file at ~/.aws/credentials. You can edit it with your preferred text editor and add the following sections:

[KMS_ADMIN]
aws_access_key_id=AKIAZKL66TFJJ3EX6UVA
aws_secret_access_key=Qg1QYQMccamGSJww48G1lrst45ziTj5/GrZBgWyc

[KMS_TEST_USER]
aws_access_key_id=AKIAZKL66TFJHY2VOR4D
aws_secret_access_key=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

[KMS_PROD_USER]
aws_access_key_id=AKIAZKL66TFJAW7AGEVG
aws_secret_access_key=JRcWCkG8CuTV4XbluAgB0aCO/WuPKMGZ72QfiHRd

These keys will need to be replaced with theones we just created obviously. Run terraform output to display our created resources. You will notice that for the access_keys we specified sensitive = true in outputs.tf. This ensures it does not get displayed in the console by default. If you want to display it anyway you have to specify the resource: terraform output access_keys. Here we can find all the necessary details in the fields id and secret for the respective users.

To test if our profiles are working we can for instance run aws sts get-caller-identity --profile KMS_ADMIN. Any command in the aws cli can be run using a profile in this manner. If this all works we are now ready to finally use SOPS and see the magic!

Deeper look into the SOPS configuration

We can easily just encrypt and decrypt files in place specifying the kms key in the command line…. but why would we ? The real use is for projects where we work together and want to store our configs in git and so on. So let’s just get directly to a more realistic configuration. SOPS looks recursively in our directory path for a file called .sops.yaml. In this file we can precisely specify which files or paths we want to encrypt, and with which keys. For this project our configuration looks like this:

creation_rules:
    - path_regex: .*config/test/.*
    kms: arn:aws:kms:eu-central-1:640753212234:key/f9024130-ba9c-458d-ba4a-ca05b06f6f2c
    aws_profile: KMS_TEST_USER

    - path_regex: .*config/prod/.*
    kms: arn:aws:kms:eu-central-1:640753212234:key/02b01f77-35ac-4f24-a23c-87684b6cc01b
    aws_profile: KMS_PROD_USER

What does this mean ? It means that any files matching the regex .*config/test/.* meaning all files inside config/test in this case, will be encrypted using the key arn:aws:kms:eu-central-1:640753212234:key/f9024130-ba9c-458d-ba4a-ca05b06f6f2c. And thus only those with access to that key can and see and edit those files.

We suppose for instance that you just joined the team, and got your keys. If you look into the files you will see that for any JSON, YAML, .env file the values are encrypted, and for instance the ssh keys are encrypted as binaries. You can decrypt any file in place using sops -d -i filename, -d for decrypt and -i for in-place. But then you would have to remember to encrypt them later, and who wants to do that ? So the developers of sops made it easy for us. We just run: sops filename and it will decrypt it in memory and open it in our default editor (the one specified in the $EDITOR environment variable). Now we see the file in cleartext, we can edit and save it, and when we exit the file it gets reencrypted and saved to the file. How awesome is that !?

Now we assumed we came in the team just fresh, but we got both the prod and test keys. In a real situation we might be developers with only access to the test env. To simulate this edit ~/.aws/credentials and change a profile name:

[KMS_PROD_USER_XXX]
aws_access_key_id=AKIAZKL66TFJAW7AGEVG
aws_secret_access_key=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

Now go into config/test and run sops default.yaml. No problem, right? Now try the same in config/prod. You will probably see something like:

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
arn:aws:kms:eu-central-1:640753212234:key/02b01d66-35ac-4f24-a23c-87684b6cc01b: FAILED
- | Error decrypting key: NoCredentialProviders: no valid
| providers in chain. Deprecated.
| 	For verbose messaging see
| aws.Config.CredentialsChainVerboseErrors

Pretty neat right ?

Git diff

Now let’s assume you were hired to actually do something. And you need to add a property, so you run sops some.env.json in config/test. You add an application url:

{
  "database": {
    "server": "important-test-sql.database.net",
    "catalog": "important-test-sqldb-application-api",
    "username": "application-test-api-username",
    "password": "application-test-api-password"
  },
  "applicationUrl": "app.test-env.awesome-stuff.com"
}

After saving and quitting would like to see what has changed in your files compared to the way things were. And so you run git diff. But yeah…. Not quite as informative as you would like it to be ?

[...]
"username": "ENC[AES256_GCM,data:XYqrySLKWxIJifFP3vB3T7XZM9vjN6KUxTItO6s=,iv:D4Qd4KvAmApEpqjaAStF9IPR7H4HoiPzdOomh1HpYCo=,tag:3ApMYz4t41ni5qeBXv6XMw==,type:str]",
"password": "ENC[AES256_GCM,data:1vGBvFRCjhTIbbnmolmmjBh1d+eCP5E5Wa8zwG8=,iv:tuoLwKlDRV5wley3CWllgwz03MU619Y9DVeyzRGOxr4=,tag:XiMK5P2/0nEbI+rMnAIaBQ==,type:str]"
},
+       "applicationUrl": "ENC[AES256_GCM,data:f7ce62NuwHVAchtTRSi8slaawvnXSiks5EtfSLg1,iv:UKMRt8XAncqF+qrQmBOxFOCM15XwEMl5jsUchH9dOAY=,tag:+lNZEhXa6+0L7TRvzYNHjQ==,type:str]",
[...]

Nope… Not very useful. But we’re in luck because even this has been thought of! We create a file .gitattributes in the root of our project and in it we write:

config/** diff=sopsdiffer

This tells git that for every file matching the path config/**, meaning any file under this directory, we use sopsdiffer when we run git diff. But what is sopsdiffer? It is just an arbitrary name that could be anything, but we can give it some meaning by running git config diff.sopsdiffer.textconv "sops -d". This command adds a bit to our git config file found in .git.config. It now has a part:

[diff "sopsdiffer"]
textconv = sops -d --config /dev/null

If we now run git diff we find something a lot more satisfying:

[...]
"catalog": "important-test-sqldb-application-api",
"username": "application-test-api-username",
"password": "application-test-api-password"
-       }
+       },
+       "applicationUrl": "app.test-env.awesome-stuff.com"
[...]

Sops gets called every time we call git diff before displaying it in the console. Didn’t I tell you? Awesome stuff!. This should work for most IDE’s because they all call git diff under the hood. Unfortunately it only partly works if you use vscode as your IDE and try diffing in there. If you want to read more about why it fails, or want to help working out a solution you can track the issue here.

Revoking access to KMS keys

Now let’s say you did an awsesome job and are now ready to leave the team on to greater things! Now we need to revoke your access to the KMS keys. With our terraform set up this is a simple two step that can be run by the key_admin user we created:

Edit locals.tf and from the line 46:

policy = templatefile("./policies/key-policy.tpl", {key_users = [aws_iam_user.user["test"].arn, aws_iam_user.user["prod"].arn], key_admins = [aws_iam_user.user["key_admin"].arn, local.my_aws_user]})

We remove the test user so it becomes:

policy = templatefile("./policies/key-policy.tpl", {key_users = [aws_iam_user.user["prod"].arn], key_admins = [aws_iam_user.user["key_admin"].arn, local.my_aws_user]})

Edit providers.tf and profile = KMS_ADMIN
We run terraform apply.

And there, our test user now no longer can decrypt our configurations! Mind you, for now, he still has a user and everything in the cli set up. But our secrets are safe. Within 2 minutes, everything is safe again. Of course you know up front someone will leave the team, but say a lost laptop could be a more sudden event in which case you need to be able to mitigate these risks quickly! Obviously if the person not just switched teams but left the company entirely, then we would also remove the user.

Conclusion and cleanup

Don’t forget to clean up properly. DO NOT JUST THROW AWAY THE FOLDER. This will bring you a headache in the AWS console. Instead just:

edit providers.tf again and remove the profile so we run the cleanup as our own IAM user.
run terraform plan -destroy and confirm. This will clean up all the resources in AWS.
Edit your ~/.aws/credentials and remove the now redundant profiles.

That’s it! This concludes this write up and example on how to use SOPS for encrypting and decrypting secrets in our configs (and other files), and how to manage all the required resources in AWS with terraform. I hope you liked it, and please leave any comments or suggestions in the comment section or reach out to me.

How to use keepass programmatically with typescript

2022-01-24T00:00:00+00:00

Even though we all know keeping user credentials in our code is bad (right ?) … sometimes it still happens. For instance when we write our automated tests, and then it “does not really matter that much because it is just the test environment”. Well it might not, if everyone else (sysadmins etc) are doing their work correctly. But why gamble? It’s 2022 folks, security is a shared responsability these days, and there are tons of tools to help us at any level.

So today let’s look at a great example of these tools: keepass. It is a tool that allows you to generate strong passwords, store them securely in an encrypted database (a .kdbx file), and retrieve them quickly and easily. You only need to remember one master password. There are many apps out there for mac, windows, linux and mobile devices that can offer even more features such as autofill passwords in your browser etc. And then sharing your passwords is easy with something like nextcloud or even just plain old emails (but this gets you in a hell of versions obviously). For teams working with or on the same app you can store them in AWS, GCS, or Azure, and then set IAM permissions so you have fine grained control over them.

All of that is great but it gets better: We can also use keepass programmatically. This way we can have our code open up the database, and extract the users credentials and run our tests for example. This means that we can store the database with test users in git (because it is safely encrypted), and all we need to do is pass a master password. The easiest way to do that is through an environment variable (or shell variable). That way if we run it locally we can just pass it in from the command line (or possible our IDE). When we run it for instance in a pipeline, we can tell the pipeline to fetch the master password from a secure place (for instance: the pipeline will have an IAM role with a policy that allows it to retrieve secrets from AWS secrets manager)

Here is an example for typescript. You can find it on github We can use a keepass.io library for this. In this small example I have an app.ts file which imports a keepass.ts file where we do all the magic with our database. Imagine the app.ts file could be anything like a module handles logging in to our app for instance.

To get started we need to make an encrypted database. For this we install our favourite keepass app, and open it. Choose new to create a new database file, and choose your database name. I chose users. Then we can fill in a few users. I filled in:

admin_user@my-app.com
normal_user1@my-app.com
normal_user2@my-app.com

We’ll let keepass generate some nice strong passwords for us and save.

Then we install keepass.io with npm install --save keepass.io. Now this might give you a screen full of errors about python missing… but then they lovingly reassure us:

DO NOT WORRY ABOUT THESE MESSAGES. KEEPASS.IO WILL FALLBACK TO SLOWER NODE.JS METHODS, SO THERE ARE NO LIMITATIONS EXCEPT SLOWER PERFORMANCE.

If we worry about performance, we should make sure to have a working installation of python available. But then again, in most real world scenarios this is not the place where your performance bottleneck is at. Ok, then to start using we import it at the top const kpio = require('keepass.io') and we’re ready to go!

We define a class Keepass and in it one method called getPassword. Obviously you can add other methods as well if needed: keepass.io also supports writing to the database etc. But for our simple purposes we just want to retrieve a password to then use it for other great things.

In getPassword we first open the database with the master password and retrieve it’s contents with getRawApi.

      let db = new kpio.Database();
      db.addCredential(new kpio.Credentials.Password(masterpass));
      db.loadFile(dbPath, function (err: any) {
        if (err) reject(err);
        const rawDb = db.getRawApi().get();
        [...]
      })

getRawApi.get() return the contents of the database as javascript Object. If we add a console.log(rawDb) we see something like:

{
  KeePassFile: {
    Meta: {
      Generator: 'KeePass',
      HeaderHash: '9ky87SMwixNRtGVl9TvG0VSzyawLmT9hpAbl8Z2ev1o=',
      DatabaseName: 'users',
      DatabaseNameChanged: '2022-01-24T06:31:05Z',
      DatabaseDescription: '',
      DatabaseDescriptionChanged: '2022-01-24T06:30:53Z',
      DefaultUserName: '',
      DefaultUserNameChanged: '2022-01-24T06:30:53Z',
      MaintenanceHistoryDays: '365',
      Color: '',
      MasterKeyChanged: '2022-01-24T06:30:53Z',
      MasterKeyChangeRec: '-1',
      MasterKeyChangeForce: '-1',
      MemoryProtection: [Object],
      RecycleBinEnabled: 'True',
      RecycleBinUUID: 'uWPZ1nyGcEyuvZyvRSLLnA==',
      RecycleBinChanged: '2022-01-24T06:30:53Z',
      EntryTemplatesGroup: 'AAAAAAAAAAAAAAAAAAAAAA==',
      EntryTemplatesGroupChanged: '2022-01-24T06:30:53Z',
      HistoryMaxItems: '10',
      HistoryMaxSize: '6291456',
      LastSelectedGroup: 'YWA8EJj7aUaHuop6TkTKGQ==',
      LastTopVisibleGroup: 'YWA8EJj7aUaHuop6TkTKGQ==',
      Binaries: '',
      CustomData: ''
    },
    Root: { Group: [Object], DeletedObjects: '' }
  }
}

As we see there is a lot of meta data about our database here, but the real magic sits inside the Root node. From there we can drill down, and we see that inside rawDb["KeePassFile"]["Root"]["Group"]["Entry"] we have an array with a bunch of things amongst which a node String that contains an array of objects. Inspecting one of those by logging rawDb["KeePassFile"]["Root"]["Group"]["Entry"][0]["String"]:

[
  { Key: 'Notes', Value: '' },
  {
    Key: 'Password',
    Value: { _: 'AO01ylqPyTycqOCOREBz', '$': [Object] }
  },
  { Key: 'Title', Value: 'admin' },
  { Key: 'URL', Value: '' },
  { Key: 'UserName', Value: 'admin_user@my-app.com' }
]
The credentials for adminUser are: admin_user@my-app.com / AO01ylqPyTycqOCOREBz
[
  { Key: 'Notes', Value: '' },
  {
    Key: 'Password',
    Value: { _: 'AO01ylqPyTycqOCOREBz', '$': [Object] }
  },
  { Key: 'Title', Value: 'admin' },
  { Key: 'URL', Value: '' },
  { Key: 'UserName', Value: 'admin_user@my-app.com' }
]
The credentials for normalUser1 are: normal_user1@my-app.com / Mv2j8EHTbSnSTphBWHey
[
  { Key: 'Notes', Value: '' },
  {
    Key: 'Password',
    Value: { _: 'AO01ylqPyTycqOCOREBz', '$': [Object] }
  },
  { Key: 'Title', Value: 'admin' },
  { Key: 'URL', Value: '' },
  { Key: 'UserName', Value: 'admin_user@my-app.com' }
]

This has all the info we actually need. If you added a title you will find it here, as well as notes or any other fields you used. So looping through this array, and matching it against the requested userName, will gives us what we need.

The total then looks something like this:

const kpio = require("keepass.io");

const dbPath = "./users.kdbx";
const masterpass = process.env["KEEPASS_PW"];

class Keepass {
  getPassword = async (userName: string): Promise<string> => {
    return new Promise((resolve, reject) => {
      let db = new kpio.Database();
      db.addCredential(new kpio.Credentials.Password(masterpass));
      db.loadFile(dbPath, function (err: any) {
        if (err) reject(err);
        const rawDb = db.getRawApi().get();
        const entries = rawDb["KeePassFile"]["Root"]["Group"]["Entry"].map(
          (x: any) => x["String"]
        );
        const secrets = entries.map((entry: any) => {
          const userName = entry.find((x: any) => x["Key"] === "UserName")[
            "Value"
          ];
          const password = entry.find((x: any) => x["Key"] === "Password")[
            "Value"
          ]["_"];
          return {
            userName,
            password,
          };
        });
        const returnValue = secrets.find((s: any) => s.userName === userName)
          ?.password;
        resolve(returnValue);
      });
    });
  };
}

If you want to see it work then run:

git clone git@github.com:ProofOfPizza/example-keepass-and-typescript.git
npm install
tsc
KEEPASS_PW=super-secret12#$ npm start

As you see, it is quite easy to use keepass in your code and increase security when it comes to user credentials. Happy coding and let me know what you think!

Is there creativity in coding?

2022-01-10T00:00:00+00:00

“But you are such a creative person, why?”, is a comment I get often (in variations of course) when I say I work in software development. I am a trained music professional, and apparently that means I am super creative, and writing code means I am stuck with ones and zeros, and surely there is nothing creative about that. I figured it might be nice to write this piece about exactly how creative musicians and coders are, and what role creativity really plays in our work.

I will first start with some general, slightly abstract or philosophical if you will, ideas about creativity. I always feel that taking a shit might be one of the best examples of creativity. Sure, it lacks some aspects of it, but it has the most important ones and it quickly breaks some of the most prevalent myths as well. For instance this idea that creativity is “creating something out of nothing”. Right? “There was no song, now there is one. There was an empty canvas now there is a painting” Something created out of nothing. Well… no. Not really, and when I say not really I mean absolutely not. Sure the toilet was empty, and now your work of art lays there smiling and smelling at you, but it was not created out of nothing. In fact: what does it contain that you did not first eat ?

Creativity is like this, a process which has inputs (whatever you eat and drink) and restructures those, breaking it up in small parts, combining things from different sources etc., and molds it into some fantastic “new” thing as output. (Does it not just smell new?). A nice byproduct of course can be that we feel good, satisfied, energized and so on. Great stuff all along! But not as magical as we sometimes deem it to be. This is true for any form of creativity, whether physical or abstract. Music, forms, colors, ideas all these are composed and recomposed by similar creative processes.

Something that I have found to be an essential ingredient in creativity is the “why?”. Creativity is (almost?) without exception a process that is applied to solve a problem. Now I use problem here as a broad concept in the sense of obstacle, constraint, etc. When we are confronted with a problem, it is almost inevitable to start thinking about solutions. (So much in fact that it can be a problem in itself! Remember those refinement meetings where we get a solution to refine instead of a problem?) Sometimes we have one, sometimes we feel immediately that we will be unable to solve it and then stop thinking about it, and sometimes we find ourselves digging into it trying to crack it.

Then now, coming back to the main subject here: Does coding involve creativity? Does making music involve it ? The answer of course is: yes, coding is first and foremost a creative process. Firstly analysing the problems at hand, destructuring and restructuring them, then when we have thought of a solution we need to code it. Again, writing code, restructuring it, adapting other people’s solutions to similar but not equal problems. Then beautifying the code. All again and again a process of creativity, different partial solutions to partial problems that together form the solution to the great problem we were confronted with.

What about musicians then ? Well yes, surely creativity is involved. But sometimes less that you would think! In my experience people miss all types of distinctions in this field, like musicians vs composers vs songwriters vs instrumentalists. In some parts, like songwriting and composition there is a lot of creativity. The problems here could be, for example: “Can I play a melody so that people recognize the original theme in it, while we are moving over a bunch of very different chords?” Or: “Can we compose something that is undoubtedly reggae but uses polytonality in the way for instance some composers from the last century did?” When I want to compose music, I generally start by setting up a few constraints/ problems like these. It triggers my brain and ideas will come. After generating a bunch I will then take those musical ideas and craft them into compositions that I like or at least make sense to me.

Instrumentalists sometimes have to be very creative, more so when they improvise (which is another thing that we could slice up into smaller things to really see what is what). And sometimes it is just repeatedly banging out the same notes that apparently make up the song, and not so much else. Nothing against Robby Williams, but when you’ve played “let me entertain you” at a few hundred parties there is not that much creativity left there.

Of course, in coding we could also add some more context and precision. In the end it seems to come down to this: Do we get real problems to solve? The fist time we get asked to implement a fuzzy search over a few thousand articles in a DB we have to put everything at work and all creative juices flowing to get it done. The third time we get asked the same it is a repetition of moves, a thing we know how to do, but not necessarily a creative challenge.

Most coders I know, don’t want to keep repeating the same tricks. We sometimes even omit certain experiences and skills from our resumes so that we do not get asked to do those things again. This is precisely because we are creative at heart, and love to stay creative, stay challenged. If you want boredom? Well try playing in an uninspired orchestra, where you know what to do, you do not really need to study, and actually no one really cares if it gets to sound better as long as everyone gets paid in time.

There is not just black and white, in fact not even just many shades of grey. There are colours everywhere, and we can choose to have our lives full of colour and glitters! And I for one, definitely intend to keep doing just that!