Skip to main content

Scope and Recommendations

This page summarizes the scope, limitations, and recommendations for using CxArchive.

General Scope

This section summarizes the scope and known limitations of CxSAST archiving (using the CxArchive tool):

  • Data is de-normalized, therefore additional space is required when archiving it.

  • Scheduling is not possible, although it is possible to adjust the iteration intervals to control the execution times or use Windows Task scheduler.

  • Access control is not defined at this stage. Access is provided when running the service.

  • CxArchive scope is limited to exporting partial data only, importing data back into CxSAST is unsupported.

  • CxArchive does not delete data from the CxSAST database.

  • Data retention is a separate process and is therefore not a part of the CxArchive scope.

  • CxArchive access to Elasticsearch will be available using user and password authentication.

Affected Services

CxSAST: CxArchive consumes data from the CxSAST database, which might cause regular CxSAST operations to take longer while CxArchive is exporting data.

Exporting Scope

This section details the scope regarding the information that is exported by the CxArchive Service.

Preset exportation

Currently, SAST does not hold any information regarding preset versioning. This means that it is only possible for CxArchive to export the current state of the preset, not the state it had when the scan ran. The following example illustrates a use case of this limitation:

  1. Preset A contains 3 queries.

  2. Scan 1 is executed.

  3. A new query, Query 4, is added to Preset A.

  4. Scan 2 is executed.

  5. CxArchive runs to export the scans.

    Even though Query 4 was not executed in Scan 1, the Query 4 will be listed as part of Scan 1, because it was part of Preset A when the export ran.

Custom Fields

The project and scan custom fields information is not exported as well. The project information remains until the project is removed, but the scan custom fields might be lost.

Vulnerability Status

There are no calculations of data before it is exported. This means that it’s not possible to get direct information about the status of the findings, so it needs a pre-calculation of the Elasticsearch data before it is possible to discover the number of Resolved, Recurrent or New findings for each scan.

Vulnerability State and Severity

The information about the state and severity must be related to the last record of the ResultLabels table, since there is no field holding this information calculated on the exportation process.

Recommendations

This section lists requirements and recommendations for hardware and configurations

Hardware Requirements and Recommendations

The following hardware configurations are required or recommended.

Minimum Requirements

  • 32 GB RAM

  • CPU 8 cores

Recommendations

The following is recommended.

2xDisk Space on the ELK Host

Data is de-normalized in the ELK database, which requires at least twice the disk space of the database to collect the data.

2xDisk Space on Service machine

Exporting creates large amounts of data on the service file system and the ELK database, which requires at least twice the disk space of the database to collect the data.

Software Requirements and Recommendations

The following software configurations are required or recommended.

ELK Virtual Machine (VM)

It is highly recommended to run the Virtual machine under Linux (CentOS or Ubuntu) as Linux performs faster and consumes less memory. Information and inputs on using the Virtual Machine on Windows is available at the following site:

https://www.reddit.com/r/elasticsearch/comments/9i4tdh/elk_stack_on_windows/

Distributed Installations versus Centralized Installations (AIO)

The CxArchive Service should be installed on the same host as the database and/or SourceFiles to ensure faster connections. If this is not possible, the export of data takes longer.

Configuration Recommendations

Tackle small batches

Ideally, to avoid constraints during intense activity, archiving should be executed for small batches (100 000 scans) and during down time. So if there’s 3 years of data to backup, archive batches of 6 months on the first runs.

Avoid Large Documents for ELK

Elasticsearch is designed as a search engine and benefits from small data sets/documents. it is recommended to use a low number of documents per exported scan (NumberOfResultsPerDocument<5).

For additional information, refer to

https://www.elastic.co/guide/en/elasticsearch/reference/current/general-recommendations.html

Lower the Number of Concurrent Tasks

If scans/projects share the same source folder, keep NumberOfScansToProcessInParallel to a minimum as multiple concurrent access attempts may cause failure and you may have to repeat the export.

Keeping the Interval Long Enough

The service runs in a loop with intervals in which it remains idle for ExportIntervalInMinutes minutes in order to implement a backoff-retry system. If an export attempt fails due to lack of access to a source folder or too many requests on ELK, the system attempts the export again in the next iteration. This interval grants the system time to recover. It is recommended to set it to at least ExportIntervalInMinutes = 60.

Best Practices

This section describes techniques for ensuring good performance and system usage.

Recommendations for appsettings.json

Notice

  • To ensure good performance, use a small number of scan results for each document in the key NumberOfResultsPerDocument, for example between 1 and 10.

  • In case of incorrect input in the appsettings.json file when starting the service, errors appear in the logs pointing to the specific issue.

Caution

Since the SAST data model encounters changes, it is good practice to manage one named index in Elasticsearch per SAST version, for example scans_89 for SAST v8.9.0.

Windows Service Creation and Configuring Permissions

After configuring CxArchive, perform the following tasks to complete the installation process:

  • Create Windows Service

  • Give the Windows service the required permissions

  • Give the artifacts folder (“C:\Checkmarx\CxArchive”) the required permissions

  • Give the source code folders the required permissions

Creating the Windows Service

To create the Windows service, run the following commands in a CMD command line with Administrator privileges:

## Create windows service
# binPath should point to actual CxArchive.Api.dll in case you chose a different installation path for your CxArchive folder
sc create CxArchive start= delayed-auto binpath= "dotnet C:\Checkmarx\CxArchive\CxArchive.Api.dll --run-as-winservice" displayname=CxArchive

## Add the description to the win service
sc description CxArchive "Checkmarx Archiving Service"
Providing Windows Service Permissions to CxArchive

Once the CxArchive Windows service has been created, you have to grant it user/network permissions to access the CxSAST database, the binaries folder and the sources folder path.

To facilitate this, follow the instructions below:

  1. Access your Windows service in “Services“ view.

    6436186853.png
  2. Right click the service and select Properties.

  3. On the Log On tab, click This account and then Browse.

  4. Click Advanced, then Find Now and from the search results, select the same user applied to your SAST services. In this example, it was the Network Service user.

  5. Click OK and grant it full permissions by checking the Full control checkbox.

Granting Source Code Permissions to Binaries Folder and Exported Scans

You must grant source code permissions to the CxArchive folder containing the binaries and to the exported source code folder you defined as follows:

  1. Open the CxArchive folder, right-click and select Properties from the menu.

    6436186856.png
  2. Click the Security tab and then select Edit and Add.

  3. Click Advanced, then Find Now and from the search results, select the same user as applied to SAST services and CxArchive Windows Service.

    6436186859.png
  4. Click OK and grant full permissions by checking the Full control checkbox,

  5. Click Apply and then OK.

Repeat these instructions for the source folder and you can start your CxArchive Windows Service in your Services view.