Skip to main content

Archive Installation Guide

CxArchive is the SAST archiving service that allows customers to export data from their CxSAST database reducing the load on the database. For further information and instructions on installing CxArchive, refer to the instructions below.

Prerequisites

CxArchive installs as a Windows service and requires the following pre-requisites:

  • ASP.NET Core 3.1.10 hosting bundle (download link) must be installed on the CxArchive service host.

  • The host with CxArchive installed must have connectivity to the CxSAST database host.

  • Elasticsearch must be installed and connectivity to the CxArchive host must be guaranteed at all times.

Installing CxArchive as a Windows Service

Follow the instructions below to install CxArchive:

  1. Create a folder called CxArchive on your local host.

  2. Download the CxArchive zip archive and extract its content into the newly created CxArchive folder on your local host.

  3. Move the CxArchive folder with its content into the C:\Program Files\Checkmarx folder. The path is going to be C:\Program Files\Checkmarx\CxArchive.

  4. Create a folder where to host the the exported scan source code folders. This folder can be a local folder on the CxArchive host or a network folder. In this example, the folder name and path is “C:\CxSrcArch

  5. Run the SQL setup script (cxarchive-setup.sql) in the extracted CxArchive setup folder from the archive in the CxSAST database that you wish to export.

  6. Define the feature flags defined in the environment variables listed below according to your own environment.

Configuring the Feature Flags

After running the the SQL setup script, the following set of feature flags/configuration values for the CxArchive system in your SAST database table CxComponentConfiguration becomes available:

  • SAST_ENVIRONMENT_ID - an auto generated guid that identifies your SAST system

  • CXARCHIVE_IS_ENABLE - defines if the CxArchive system is active or not

  • CXARCHIVE_SOURCES_FOLDER - the path where you want your exported scans source code folders

  • CXARCHIVE_SOURCES_ENABLE - a boolean flag that defines if the source code folder of the exported scans is going to be exported or not

Danger

Do not modify the SAST_ENVIRONMENT_ID configuration value. Modifying this value may cause data inconsistency problems.

Notice

Under CXARCHIVE_SOURCES_FOLDER, enter the path that you created in step 4. In the example used, it was “C:\CxSrcArc“.

Warning

If some source code folders are missing in the system and the flag CXARCHIVE_SOURCES_ENABLE is enabled (true), the corresponding scans export will fail.

Configuring the CxArchive Settings

After configuring the CxArchive feature flags, you have to configure the settings in appsettings.json. The appsettings file contains the connections to the SAST database, the Elasticsearch connection and the export criteria that define the scans that are going to be exported from your SAST system. The file has the following structure:

{
  "Database": { //On the first execution, the DBconnection string is encrypted.
    "CxDBClient": {
      "ConnectionString": "Data Source= ;Initial Catalog=CxDB;Integrated Security=False;User ID= ;Password= ;Pooling=True;", // ConnectionString for the CxSAST DB instance to export 
      "EncryptionKey": "",
      "EncryptionVector": ""
    },
    "ElasticSearch": { //On the first execution, the DBconnection string is encrypted.
      "Host": "http://localhost:9200",  // Elasticsearch data lake machine 'http(s)://<url>:<port>' 
      "IndexName": "scans_89" // Elasticsearch index where your scan documents will be stored
      "AuthType": "None", // None, Basic or "ApiKey"
      "Identifier": "", // username in basic or id in apikey
      "Key": "" // Password in basic authentication or key in apiKey. For base64key only use Key field and leave Identifier empty.
    }
  },
  "ExportSettings": {
    "NumberOfRetries": 3, // number of retries allowed for the export of any scan in case of failure
    "NumberOfResultsPerDocument": 5,  // number of PathResults in each denormalized scan document
    "NumberOfScansToProcessInParallel": 4,  // number of scans exported in parallel
    "MaxBatchSize": 10000, // number of scans processed in each CxArchive export iteration
    "ExportIntervalInMinutes": 60,  // interval in minutes export between executions
    "ExportCriteria": { // defines the criteria conditions to export
      "Type": "Scans",  // defines the type of data to export, 'Scans' or 'Projects'
      "SelectedIds": [],  // specific ids to export. if empty list all scans between mentioned date in ScansAgeRange will be exported
      "ScansAgeRange": { // determine the date range window of scans to archive. set values as empty string or remove field if no filter is required
        "MinScanAgeInDays": 1, 
        "MaxScanAgeInDays": 100 
      }
    }
  }
}

For details about the validation rules required for the system to operate, see AppSettings customization.

Limitations and Recommendations

For details about the limitations and recommendations, see Limitations and Recommendations.

Recommended Best Practices

This section describes techniques for ensuring good performance and system usage.

Recommendations for appsettings.json

Notice

  • To ensure good performance, use a small number of scan results for each document in the key NumberOfResultsPerDocument, for example between 1 and 10.

  • In case of incorrect input in the appsettings.json file when starting the service, errors appear in the logs pointing to the specific issue.

Warning

Since the SAST data model encounters changes, it is good practice to manage one named index in Elasticsearch per SAST version, for example scans_89 for SAST v8.9.0.

Windows Service Creation and Configuring Permissions

After configuring CxArchive, perform the following tasks to complete the installation process:

  • Create Windows Service

  • Give the Windows service the required permissions

  • Give the artifacts folder ( “C:\Checkmarx\CxArchive” ) the required permissions

  • Give the source code folders the required permissions

Creating the Windows Service

To create the Windows service, run the following commands in a CMD command line with Administrator privileges:

## Create windows service
# binPath should point to actual CxArchive.Api.dll in case you chose a different installation path for your CxArchive folder
sc create CxArchive start= delayed-auto binpath= "dotnet C:\Checkmarx\CxArchive\CxArchive.Api.dll --run-as-winservice" displayname=CxArchive

## Add the description to the win service
sc description CxArchive "Checkmarx Archiving Service" 

Providing Windows Service Permissions to CxArchive

Once the CxArchive Windows service has been created, you have to grant it user/network permissions to access the CxSAST database, the binaries folder and the sources folder path.

To facilitate this, follow the instructions below:

  1. Access your Windows service in “Services“ view.

    6436186853.png
  2. Right click the service and select Properties.

  3. On the Log On tab, click This account and then Browse.

  4. Click Advanced, then Find Now and from the search results, select the same user applied to your SAST services. In this example, it was the Network Service user.

  5. Click OK and grant it full permissions by checking the Full control checkbox.

Granting Source Code Permissions to Binaries Folder and Exported Scans

You must grant source code permissions to the CxArchive folder containing the binaries and to the exported source code folder you defined as follows:

  1. Open the CxArchive folder, right-click and select Properties from the menu.

    6436186856.png
  2. Click the Security tab and then select Edit and Add.

  3. Click Advanced, then Find Now and from the search results, select the same user as applied to SAST services and CxArchive Windows Service.

    6436186859.png
  4. Click OK and grant full permissions by checking the Full control checkbox,

  5. Click Apply and then OK.

Repeat these instructions for the source folder and you can start your CxArchive Windows Service in your Services view.

Service Start Errors

The service will not be able to start if the settings are misconfigured or the service is unable to connect to the external service dependencies. To view the cause of the failure, refer to the logs in the binaries folder “/logs“ . Once the issue has been resolved, restart the service.

Installing CxArchive using the WinService Installation Script

The manual installation of CxArchive as a WinService, is described above. This section describes how to use the WinService Installation Script: cx-archive-installer.ps1

Prerequisites

In addition to the prerequisites described here, the connection string of appsettings.json (Database.CxDBClient.ConnectionString), must match the SQL connection parameters given to this script.

Given the artifact, extract it and open a PowerShell session on the Setup directory.

6436186871.png

Note: The script can be called from anywhere, but it is assumed here that your current working directory is the Setup directory of the extracted CxArchive artifact.

Syntax

.\cx-archive-installer.ps1
    [[-SQLServer] <String>]
    [[-SQLUsername] <String>]
    [[-SQLPassword] <String>]
    [[-InstallationPath] <DirectoryInfo>]
    [-EnableCxArchiveSources]
    [[-ExportDirectoryPath] <DirectoryInfo>]
    [[-ServiceLogonUser] <String>]
    [<CommonParameters>]

Parameters

-SQLServer <String>
    The address of the SQL Server. The default value is localhost\SQLEXPRESS.
    
-SQLUsername <String>
    Username for the SQL Server. Set the username if required by your SQL Server.
    
-SQLPassword <String>
    Password for the SQL Server. Set the password if required by your SQL Server.
    
-InstallationPath <DirectoryInfo>
    Path to the CxArchive installation directory. The default value is 'C:\Checkmarx'.
    
-EnableCxArchiveSources [<SwitchParameter>]
    Flag for whether CxArchive will archive sources or not. The default value is False.
    
-ExportDirectoryPath <DirectoryInfo>
    Path to the archived scans source code directory. Requires '-EnableCxArchiveSources'.
    The default value is 'C:\CxSrcArch'.

-ServiceLogonUser <String>
    CxArchive Windows Service logon target.
    If ommitted, the logon user for the running Checkmarx services,
    on the current host, will be used by default.
    
<CommonParameters>
    This cmdlet supports the common parameters: Verbose, Debug,
    ErrorAction, ErrorVariable, WarningAction, WarningVariable,
    OutBuffer, PipelineVariable, and OutVariable. For more information, see 
    about_CommonParameters (https:/go.microsoft.com/fwlink/?LinkID=113216). 

Examples

Quickstart

To install CxArchive in the host where you have your CxSAST AIO installation, using the default parameters, run the following command:

PS> cx-archive-installer.ps1

With a remote SQL server

If your SQL server is on a different host, include the following parameters to connect to it:

PS> cx-archive-installer.ps1 -SQLServer somewhere-else\SQLEXPRESS -SQLUsername MyUser -SQLPassword MyPassword

Custom installation path

To have a customized CxArchive installation path, point to the custom path, using the following command:

PS> cx-archive-installer.ps1 -InstallationPath C:\Somewhere\Else\Checkmarx

Enable exports of the scanned source code

To enable exporting scanned source code:

PS> cx-archive-installer.ps1 -EnableCxArchiveSources

To choose a different directory for its destination:

PS> cx-archive-installer.ps1 -EnableCxArchiveSources -ExportDirectoryPath C:\Somewhere\Else\Sources

Custom logon user for the Windows service

If CxSAST isn't running on the same host, specify the logon user for the Windows service, using the following command:

PS> cx-archive-installer.ps1 -ServiceLogonUser DOMAIN\username

Verbose flag

Use the Verbose flag, to show more detailed logs:

PS> cx-archive-installer.ps1 -Verbose

If you want to copy the logs to the clipboard, for example, to report an issue, you can simply pipe it to clip and Ctrl-V in any text area:

PS> cx-archive-installer.ps1 -Verbose | clip

Remarks

To see the examples, type: "get-help cx-archive-installer.ps1 -examples".

For more information, type: "get-help cx-archive-installer.ps1 -detailed".

For technical information, type: "get-help cx-archive-installer.ps1 -full".

Limitations

If the ExportDirectoryPath parameter is pointing to a remote directory, the installer won't be able to configure the proper permissions. A warning will be issued, with instructions on how to configure it manually.

CxArchive Progress Status

In order for the user to keep track of the archiving progress at the end of each scan export, the Export completion status is logged in terms of remaining tasks and percentage of the overall process.

Installation of CxArchive as a Docker Image

This section describes for the installation of CxArchive as a Docker image.

  1. Download the CxArchive docker image.

  2. Load the CxArchive Docker image into in your CxArchive host Docker.

    docker image load --input <image filename>.tar

  3. Run the SQL setup script (cxarchive-setup.sql) in the extracted Setup folder from the artifact on your SAST database.

  4. Configure the following set of configuration values for the CxArchive system, located in the CxSAST CxComponentConfiguration database table:

    • SAST_ENVIRONMENT_ID - an auto generated guid that identifies your SAST system

      Note:Do not change theSAST_ENVIRONMENT_IDconfiguration value since it could cause data inconsistency problems.

    • CXARCHIVE_IS_ENABLE - defines whether the CxArchive system is active or not

    • CXARCHIVE_SOURCES_FOLDER - the path where you want to export the scan source code folders. See Step 5c for details how the directory paths are passed using the environment variables.

    • CXARCHIVE_SOURCES_ENABLE - a Boolean feature flag that defines if the source code folder of the scans will be exported or not

      Note:If one or more source code folders or their permissions are missing in the system and the flagCXARCHIVE_SOURCES_ENABLEis active (True), the corresponding scans export process will fail.

  5. Map the source folders to the container filesystem. This is required since Docker containers have their own separate file system.

    (If you don’t intend to export the source code of the archived scans this step can be skipped, and in this case CXARCHIVE_SOURCES_ENABLE = False.)

    1. Create either one of the following types of folders where you want to archive the source code:

      • Network folder

      • Local host folder

        For example, create the following folder on the CxArchive host: “C:\CxSrc“

    2. Create volumes for your SAST source code folder and for the corresponding archived scan source folder.

      • When using a Network folder, a separate command for creating a volume must be run, before running the docker command, described in step 7. Run the following command to create the volume mapping the desired network folder:

        docker volume create --driver local --opt type=cifs --opt device=//<machine ip>/<path to folder> --opt o=user=<user name>,domain=<domain>,password=<password> <volume name>

      • When using a Local folder, bind the directory by adding the following line to the docker run command, described in step 7:

        -v <host local path>:<internal container path>

        For example, “c:/CxSrc“ is a local folder being mapped to the “/sources“ directory in the container:

        -v c:/CxSrc:/sources

    3. Map the source directory paths to the container by passing the mapped source directory path and archived source directory path using the environment variables. To do this, create an env file with the following keys:

      • SOURCE_PATH

      • CXARCHIVE_SOURCES_FOLDER

        The following is an example of a vars.env file:

        SOURCE_PATH=/sources

        CXARCHIVE_SOURCES_FOLDER=/exported_sources

        This file is then passed in the docker run command with the following option:

        --env-file="<path to file>\vars.env"

  6. Create and configure the appsettings.json settings file. For details, see Configuring the CxArchive Settings above.

    Since we are using the docker container filesystem, we must map this file to the container filesystem, by adding the following line to the docker run command, described in step 7:

    -v <host path>/appsettings.json:/app/appsettings.json

    The destination path in the container is always /app/appsettings.json, since this is the expected location for the application.

  7. Start the CxArchive container by running the docker image with the following command:

docker run -p 8080:80 `
-v <sources volume>:/<container path from vars.env SOURCE_PATH> ` 
-v <archived sources volume>:/<container path from vars.env CXARCHIVE_SOURCES_FOLDER> `
-v <host path>/appsettings.json:/app/appsettings.json `
--env-file=<vars.env file path> `
<docker image name>

The following is an example of an AIO installation with CxArchive on the same machine as CxSAST:

docker run -p 8080:80 `
-v c:/CxSrc:/sources ` 
-v c:/CxSrcArch:/exported_sources `
-v c:/appsettings.json:/app/appsettings.json `
--env-file="C:\vars.env" `
cx-archive-1.0.4:latest

The example uses the following vars.env file:

CXARCHIVE_SOURCES_FOLDER=/exported_sources
SOURCE_PATH=/sources