Secret Key Vulnerabilities

This query uses entropy in order to detect secrets. The entropy of a random variable is the average level of information, surprise, or uncertainty inherent in the variable's possible outcomes. Shannon Entropy is used to calculate this level.

Func<string, string, double> shannon_entropy = (word, charSet) =>{
        double entropy = 0.0;

        foreach (char x in charSet)
        {
                try{
                        int countOccurences = word.Where(c => c.Equals(x)).Count();
                        double p_x = (double) countOccurences / word.Length;
                
                        if (p_x > 0)
                                entropy += -p_x * Math.Log(p_x, 2.0);
                }
                catch(Exception e){};
        }
                
        return entropy;
};

The general query FindByRegexExt is used to look for strings in all files in the project that is being analyzed. It returns Comment nodes and the strings associated with them are analyzed using the Shannon Entropy algorithm. Only strings with a minimum length of 20 characters are considered possible secrets.

int MIN_STRING_LEN = 20;

Since it is difficult to know which unknown secrets it cannot detect, this query has a tendency to generate many false positives.

In order to lower the number of false positives, two thresholds are used to filter some of those results.

double BASE64_ENTROPY = 4.5;
double HEX_ENTROPY = 2.7;

This means that the query only returns the strings in which the entropy calculated by the algorithm is higher than the specified threshold.

This query uses regular expressions in order to detect secrets.

A dictionary was created containing several regular expressions. They are designed to recognize the patterns of sensitive data.

Dictionary<string, string> regexes = new Dictionary<string, string>(); 
regexes.Add("Slack Token", "(xox[p|b|o|a]-[0-9]{12}-[0-9]{12}-[0-9]{12}-[a-z0-9]{32})");
...
regexes.Add("Twitter OAuth", "[t|T][w|W][i|I][t|T][t|T][e|E][r|R][^\\s]*['|\"][0-9a-zA-Z]{35,44}['|\"]");

This query returns all the strings associated with the Comment nodes found by FindByRegexExt (which is also used in this query to look for strings in all files in the project that is being analyzed) that match any of the Regular Expressions present on the dictionary.

foreach(String regexKey in regexes.Keys)
{       
        CxList found = All.FindByRegexExt(regexes[regexKey], "*.*");  
        result.Add(Remove_FalseComments_TruffleHog(found));
}

The only caveat to this is that the Regex specifying the format needs to exist and be accurate, but the results from this query will likely be more accurate because it is searching for concrete patterns that match sensitive data.

Since the general query FindByRegexExt only returns Comment Nodes and does not evaluate the DOM, but instead evaluates the code as a set of strings, a Common query was created to filter the strings

that effectively represent the real strings that will be used by the above queries.

The following nodes are removed:

Since it is impossible to directly compare the nodes on this list with the comments returned by FindByRegexExt, the filenames and LinePragmas are used for comparison.

When a node’s filename and a LinePragma are the same as a comment’s filename and LinePragma, it is assumed that they are the same and can be removed. Thus it is possible to filter the comments, leaving only potential secrets.