Troubleshooting Local Collection Coverage

  • Updated

This article applies to BHCE and BHE

SharpHound collects data from domain-joined systems utilizing SMB/RPC on port 445/TCP and requires the account running SharpHound (e.g., the SharpHound Enterprise service account) to have local administrator membership on each system in scope, see SharpHound Data Collection and Permissions.

This article can assist in troubleshooting why a local collection is not successful for all systems in scope.

Computer status logfile

The computer status logfile, named `compstatus.csv`, contains information about collection result for each system in scope of the collection.

  • SharpHound Enterprise: Generates one `compstatus.csv` per local collection job (Sessions and/or Local Groups) and store it within the `log_archive` directory on the SharpHound Enterprise server. The default location for this is `%APPDATA%\Roaming\BloodHound Enterprise` - that is, App Data for the service account running the SharpHound Enterprise service. However, you may override this location within `settings.json`, see SharpHound Enterprise Local Configuration.
  • SharpHound Community Edition: Will generate `compstatus.csv` when run with `DumpComputerStatus` flag.

Analyzing compstatus.csv

The first step in troubleshooting local collection issues is by identifying and understanding errors in `compstatus.csv`.

BloodHound Enterprise customers can reach out to their Technical Account Manager (TAM) for support in this analysis.

Alternatively you may do your own troubleshooting by utilizing the below example PowerShell comands and the process described below the code block.

### Import data and get uniques without sorting them
$stats_file = Import-Csv -Path 'FILE_PATH_HERE' | Group-Object ComputerName, Task, Status, IPAddress | ForEach-Object { $_.Group[0] }

### Status Pivot Table - Exclude GetMembersInAlias as it's irrelevant for troubleshooting
$stats_file | Where-Object {$_.Task -NotLike 'GetMembersInAlias -*'} | Group-Object Task, Status -NoElement | Format-Table -Autosize

### Pivot table for failures only
$stats_file |  Where-Object {$_.Status -ne "Success"} | Group-Object Task,Status -NoElement | Format-Table -Autosize

### Which systems were unreachable on 445/TCP
$stats_file | Where-Object {$_.Task -eq "ComputerAvailability" -and $_.Status -eq "PortNotOpen"}

### IPv4 /24 subnets unreachable on 445/TCP
$stats_file | Where-Object {$_.Task -eq "ComputerAvailability" -and $_.Status -eq "PortNotOpen" -and $_.IPAddress -match '^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$'} | Group-Object {$_.IPAddress.Remove($_.IPAddress.LastIndexOf('.'))+'.0/24'} -NoElement | Sort-Object -property Count | Format-Table -Autosize

### IPv4 /16 subnets unreachable on 445/TCP
$stats_file | Where-Object {$_.Task -eq "ComputerAvailability" -and $_.Status -eq "PortNotOpen" -and $_.IPAddress -match '^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$'} | Group-Object {($_.IPAddress.split(".")[0..1] -join ".") + ".0.0/16"} -NoElement | Sort-Object -property Count | Format-Table -Autosize

### Which systems are missing permissions
$stats_file | Where-Object {$_.Status -eq "ERROR_ACCESS_DENIED" -or $_.Status -eq "StatusAccessDenied"}

To understand and resolve the errors outputted by the commands, you must understand the process involved in SharpHound's local collection, this process is described below.

Domain computer enumeration

Firstly, SharpHound queries a Domain Controller to list every enabled computer object in the domain. Every enumerated system will be in some form be represented in `compstatus.csv`.

ComputerAvailability

Next, SharpHound preforms the `ComputerAvailability` check, which filters out inactive computers, so that SharpHound later in the process only connects to active computers for the actual Sessions or Local Groups enumeration. 

Each returned computer objects is validated to be a Windows OS or not. This is done, because local collection is not supported on non-Windows OS'.

If the system is not a Windows OS, SharpHound will not perform additional checks on the system. 

  • If a system fails this check, `compstatus.csv` will contain a line for the system with the result `Task = ComputerAvailability` and `Status = NonWindowsOS`.
  • If a Windows system is wrongly marked with `NonWindowsOS`; ensure that the system's AD computer object attribute `operatingSystem` is set to a string representing a Windows OS.

If the system is a Windows OS, SharpHound will perform the following checks on the system

  1. Check if the system has changed it's password within the duration set in the key `ComputerPasswordResetWindow` in SharpHound's `settings.json`.
    • If a system fails this check, `compstatus.csv` will contain a line for the system with the result `Task = ComputerAvailability` and `Status = PwdLastSetOutOfRange`.
    • If an active system is wrongly marked with `PwdLastSetOutOfRange`, try one of the following:
      • Ensure that the `ComputerPasswordResetWindow` key in SharpHound's `settings.json` has a value corresponding to the computer's security policy `Domain member: Maximum machine account password age`. By default this value is 60 days. A description of the `ComputerPasswordResetWindow` value is found the article SharpHound Enterprise Local Configuration
      • Ensure that the system's password is changing as expected: Check the system's AD computer object attribute `pwdLastSet` has changed within the period defined in the computer's security policy `Domain member: Maximum machine account password age`
  2.    Check if the system has TCP port 445 (SMB) open.
    • If a system fails this check, `compstatus.csv` will contain a line for the system with the result `Task = ComputerAvailability` and `Status = PortNotOpen`.
    • If an active system is wrongly marked with `PortNotOpen`, try one of the following:
      • Ensure that the system running SharpHound can reach the system checked on TCP port 445. From the SharpHound system, run `Test-NetConnection -ComputerName <HOST> -Port 445` (replace <HOST> with the system's DNS name seen in `compstatus.csv`).
      • Ensure that the system running SharpHound can reach the system checked on TCP port 445 within 500ms. From the SharpHound system, run `Measure-Command { Test-NetConnection -ComputerName <HOST> -Port 445 }` (replace <HOST> with the system's DNS name seen in `compstatus.csv`).
      • Ensure that the system's DNS name found in `compstatus.csv` can be resolved, and matches the system's DNS name in Active Directory. From the SharpHound system, check the name can be resolved by running `Resolve-DnsName -Name <HOST>}` (replace <HOST> with the system's DNS name found in `compstatus.csv`).
      • Ensure that a network layer above TCP (e.g., SMB) is not being blocked by a security solution, such as an IDPS.

After these steps, the system is deemed to be available or not.

If not available, further local collection steps are not taken on the sytem.

If a system is found available, `compstatus.csv` will contain a line for the system with the result `Task = ComputerAvailability` and `Status = Success`.

Thereafter SharpHound will start the actual collection of Sessions and/or Local Groups

Local Groups

This collection gathers two types of data points:

  1. Local group memberships
  2. User Rights Assignment

Local group memberships

Firstly, SharpHound connects via RPC with `SamConnect`.

  • If unsuccessful, `compstatus.csv` will contain a line for the system with the result `Task = SamConnect` and a status depending on the error type.
    • `Status = -1073610725` means SharpHound account is not in the system's local administrators group.
    • `Status = StatusRpcServerUnavailable` means SharpHound cannot access RPC or SMB on the system. Ensure that the system running SharpHound can reach the system checked on SMB.
  • If successful, SharpHound continues with the method `GetMembersInAlias` as detailed below.

Next, SharpHound connects via RPC with `GetMembersInAlias`.

  • If unsuccessful, `compstatus.csv` will contain a line for the system with the result `Task = GetMembersInAlias` and a status depending on the error type.
  • If successful `compstatus.csv` will contain one line per local group in system with the result `Task = GetMembersInAlias - <LOCAL GROUP NAME>` and "Status = Success'

User Rights Assignment

Firstly, SharpHound connects via RPC with `LSAOpenPolicy`.

  • If unsuccessful, `compstatus.csv` will contain a line for the system with the result `Task = LSAOpenPolicy` and a status depending on the error type.
    • `Status = StatusRpcServerUnavailable` means SharpHound cannot access RPC or SMB on the system. Ensure that the system running SharpHound can reach the system checked on SMB.
  • If successful, SharpHound continues with the method `LSAEnumerateAccountsWithUserRight` as detailed below.

Next, sharpHound connects via RPC with `LSAEnumerateAccountsWithUserRight`.

  • If unsuccessful, `compstatus.csv` will contain a line for the system with the result `Task = LSAEnumerateAccountsWithUserRight` and a status depending on the error type.
    • `Status = StatusAccessDenied` means SharpHound account is not in local administrators group.
  • If successful `compstatus.csv` will contain one line per local group in system with the result `Task = LSAEnumerateAccountsWithUserRight` and "Status = Success'

Sessions

This collection gathers logon sessions via RPC with `NetWkstaUserEnum`.

  • If unsuccessful, `compstatus.csv` will contain a line for the system with the result `Task = NetWkstaUserEnum` and a status depending on the error type.
    • `Status = ErrorAccessDenied` means SharpHound account is not in local administrators group.
    • `Status = 53` means SharpHound cannot access RPC or SMB on the system. Ensure that the system running SharpHound can reach the system checked on SMB.
  • If successful `compstatus.csv` will contain one line per local group in system with the result `Task = NetWkstaUserEnum` and "Status = Success'