Hung Ngo's Blog
Between the Lines: What are the secrets in your code?

Between the Lines: What are the secrets in your code?

Hung Ngo's photo
Hung Ngo
·

5 min read

One critical aspect of security is ensuring that sensitive data, such as passwords, API keys, and encryption keys, are not accidentally exposed in your source code. These sensitive data are usually called “secrets”, and for penetration testers, they are the low-hanging fruit as they are usually one of the first things we are looking for. In this article, I’d like to talk about some of the techniques and tools that can be used for scanning the source code for secrets.

Before we start talking about how to look for secrets, let’s first take a look at what secrets usually look like. I used to be, and in fact, I am still a PHP developer, so I will use this example of PHP code. Trust me, you will see this piece of code frequently, as it is usually contained in many of the tutorials and developers tend to copy and paste code a lot (sometimes without considering the security measurements).

<?php
$username = "myusername";
$password = "mypassword";
$dbname = "mydatabase";
$host = "localhost";

// Connect to the database
$connection = mysqli_connect($host, $username, $password, $dbname);

// Check if the connection was successful
if (!$connection) {
    die("Connection failed: " . mysqli_connect_error());
}
?>

The secrets can be spotted pretty easily in this example. They are all stated at the beginning of the file in the plaintext form. It’s every attacker’s wet dream! If I get my hands on this, I can easily connect and exploit the database – leak the data, drop it, stop the production and much more. To make this code secure, we could use environment variables to store sensitive information like the database credentials. Let’s modify the above script to incorporate environment variables.

<?php
$username = getenv('DB_USERNAME');
$password = getenv('DB_PASSWORD');
$dbname = getenv('DB_NAME');
$host = getenv('DB_HOST');

// Connect to the database
$connection = mysqli_connect($host, $username, $password, $dbname);

// Check if the connection was successful
if (!$connection) {
    die("Connection failed: " . mysqli_connect_error());
}
?>

Alternatively, we could also store the sensitive information within the external (with the emphasis on the word external) configuration file that is not tracked by version control systems, and read the values from the configuration file in our code. If we opt for this approach we have to make sure that the configuration file is properly secured and accessible only by authorized users. We could also use key vaults.

Regex

Now we know what a secret is, but how should we detect it before it gets leaked? One of the ways is to use regex, which is a pattern-matching tool used to search and manipulate text. It’s like a code that tells a computer what to look for in a piece of text, and then the computer uses that code to find anything that matches the pattern. For example, you could use regex to search for keywords such as "key", "pass", or "aws" which might be the variable names, that are used to refer to the secrets.

Many API keys are also in this specific format. AWS access key IDs, for example, typically begin with the string AKIA and are followed by 16 characters that can be either numbers (0-9) or capital letters (A-Z). Armed with this knowledge, we can craft the regex AKIA[0-9A-Z]{16} to search for the IDs of the AWS access keys.

Using this strategy, we can discover most hardcoded credentials, but we are also limited to the secrets that adhere to this specific format, which is why it is time to talk about entropy.

Shannon entropy

Shannon entropy measures how random and unpredictable something is. To better understand it, let’s imagine you have two bags of marbles. One bag contains only the marbles of the same color. In the other bag, each of the marble has a different color.

If you reach into the first back and pick out a marble at random, you already know what color it has (as there is only one color present). This bag has a low entropy. However, if you reach into the second bag and pick out a marble at random, you would have no way of predicting which color you would get, because each color has an equal chance of being chosen. In other words, the second bag has a high entropy.

In our context, Shannon entropy is often used to analyze strings of characters, such as passwords to determine how secure they are. We can leverage the fact that secrets are usually pretty complex and randomized and by analyzing Shannon entropy, we can discover suspicious strings of any format.

I have put together this simple Python script to calculate the entropy of the given string. If we were to use this for real purposes, we would, however, have to make some changes to factor in the overall entropy of the whole codebase.

import math

def get_shannon_entropy(data: str) -> float:
    # Get the frequence of each character in the given string
    frequency_dict = dict.fromkeys(list(data), 0)
    for char in data:
        frequence_dict[char] += 1

    # Calculate the probability of each character by dividing frequency 
    # of each and every key by the length of input
    probability_dict = {}
    for key in frequency_dict:
         probability_dict[key] = frequency_dict[key] / float(len(data))

    # Calculate the Shannon entropy
    entropy = 0.0
    for key in probability_dict:
        probability = probability_dict[key]
        if probability > 0:
            entropy += -probability * math.log(probability, 2)

    return entropy

If you have read so far, you might want to follow me on Mastodon or LinkedIn to be notified of any of my new content.

secretscodeappsec