Source Pulling Performance Improvement - Cloud/NAS

During source pulling processing, the source repository is cloned to a temporary CxSrc folder where the JobsManager processes it for data such as LOC (lines of code), exclusions, and deletions.

When the CxSrc is configured to use the NAS (Network Attached Storage) for HA (High Availability) environments or DR (Disaster Recovery) purposes, the cloning and JobsManager processes run directly on the NAS. Running these operations over the NAS consumes a large amount of network bandwidth and disk IOPS (input/output operations per second), and reduces performance (sometimes increasing processing time by a factor of four) and can become a bottleneck for systems using NAS storage.

For example, when using the AWS FSX service (the Windows File Servers provided by Amazon Web Services), you only receive 3k IOPS per 1TB (terabyte) of SSD storage. JobsManager can typically consume all of this storage when preparing multiple simultaneous scans. One way to obtain more disk IOPS is by over provisioning the storage, since the more TBs of storage you purchase, the more IOPS you also receive. However, over provisioning storage in FSX is expensive, increasing the cost by approximately a factor of four.

To reduce these bottlenecks and expenses, the new Source Pulling Performance Improvement feature provides an optimized way to execute the source control cloning and processing steps locally, instead of over the NAS.

Cloning and processing of the repository is performed locally, in a temporary folder, named according to the ScanId. After the files have been processed, they will be copied to the NAS storage, and the local temporary files will be deleted.

The temporary directory naming convention is as follows:

<SourcePullingTemporaryPath>\<ScanId>.tmp\<Files..>

This feature is used for the following source pulling scenarios:

The following services are affected:

The new behavior is configured with the new SourcePullingTemporaryPath key, located in the dbo.CxComponentConfiguration table. By default, the feature is disabled and the key is set to ‘’ (empty).

The maximum age of unused temporary sources is configured using the SourcePullingTemporaryFoldersMaxAge key , located in the dbo.CxComponentConfiguration table. The configured value is in minutes. All temporary folders older than the configured value will be removed. Minimum value is 12 hours, to allow for abnormally long cloning processes to finish.

Since the solution is based on a temporary path, the cloned files are removed from the temporary location after processing and copying is completed.

The temporary folder is also cleaned up periodically by SystemManager, to handle scenarios of unexpected termination of the process, as well as system crashes. The cleanup job will filter out any folder names that do not have a “.tmp” suffix, or are not within the maximum age range.

In case the cloning and processing fails on the mentioned temporary path, one of the following error messages are issued explaining the error:

Logs for the temporary folder usage (per scan) are also printed, as well as the process for copying files to the CxSrc location and temporary files cleanup.