Targets: Data Protection

Enterprise

Synopsis

A target can refuse sensitive data, such as passwords, card numbers and national IDs. The director then protects each sensitive value before it sends a record to that target: it redacts, masks, hashes or encrypts the value. Other targets still receive the record as it was.

The sensitive data rules in the Library decide which values are sensitive.

The director does this inside your network. It protects each record after the target's own post-processing pipeline and the Smart Engine, just before it sends the record. So the Smart Engine still sees the real values, and the target receives the protected ones.

The director never sends a sensitive value to a target that refuses it:

If the director can't protect a record, it doesn't send it. It drops the record, counts it as dropped and writes the reason in its log.
If the director can't use the setting at all, for example because it can't read the key, it doesn't start sending to the target. The records wait in the queue, and the target shows a configuration error until the director can read the key.

note

The Encrypt action is not the end-to-end encryption of live data captures. That one protects a capture on its way to your browser, with a key the director makes. This one protects values in the records a target stores, with a key that you keep.

Requirements

Edition - Enterprise. On other editions, the Data protection step and tab show an Enterprise plan card, and every target you create there receives records as they are.
Directors - A director release that supports data protection. A director that can't protect data doesn't receive the target (see Directors That Don't Protect Data).
Permissions - The setting uses the target and Library permissions described under Roles:

To do this	You need
Set data protection when you create a target	Target create (`TARGET_CREATE`)
Change data protection on the target's Data protection tab, or turn it off	Target edit (`TARGET_EDIT`)
See how many enabled rules each category has, and choose rules one by one	Library read (`LIBRARY_READ`)

Without Library read, you can still choose categories, without the counts.

Turning Protection On

In the console, turn on Protect sensitive data in either of two places:

the Data protection step of the target creation wizard (see Data Protection)
the target's Data protection tab, with Manage data protection (see Data Protection Tab)

Then choose what to protect, how to protect it and, for Hash and Encrypt, the key. In a configuration file, set sensitive_data in the target's properties (see Configuration).

What Is Protected

With categories, the target protects every value that an enabled rule in those categories finds, including rules added to them later.
With rules chosen one by one, it protects only what those rules find.
A value is protected when a rule finds it. A field name alone doesn't make its value sensitive. For example, the built-in us-ssn rule finds 123-45-6789 anywhere in a record, but it doesn't find 123456789 in a JSON field named "ssn".
The built-in rules also find secrets in JSON fields, such as "password":"hunter2".
Until a director has received your Library's rules, it uses the built-in rules, each switched on or off as it ships. Five built-in rules ship switched off: credit-card-separated, phone-e164, ipv4-address, ipv6-address and mac-address. They stay off until your rules arrive. A target that names a rule you created waits for your rules: its records stay in the queue until the director has them.

caution

Only enabled rules protect anything. If you switch off, hide or delete a rule that a target uses, the target stops protecting the values that rule finds. If none of a target's chosen rules is switched on, the target receives records unchanged, and the director writes a note in its log.

What the Target Receives

How to protect it	A card number becomes	What it means
Redact	`[redacted:credit-card-number]`	The director replaces the whole value with the key of the rule that found it: `[redacted:<rule key>]`.
Mask	`411111******1111`	The characters that the rule keeps readable stay, and the rest become `*`. A rule that keeps nothing, or a value too short to keep that many characters, becomes `[masked:<rule key>]`. Masked live data looks the same.
Hash	`vmh1.630dcd29.837649812a0747b9c86c85615278e8b2`	The format is `vmh1.<key id>.<32 hex characters>`: a keyed HMAC-SHA-256, cut to 128 bits. With the same key, the same value always gives the same hash, so you can still count and correlate. Nobody can turn a hash back into the value. There is no hash without a key, because anyone could reverse an unkeyed hash of an email address or a card number by guessing.
Encrypt	`vme1.630dcd29.credit-card-number.CgsMDQ4P…`	The format is `vme1.<key id>.<rule key>.<base64url>`: the value encrypted with AES-256-GCM. Anyone with the key can decrypt it. Each value is encrypted with a new random nonce, so the same value looks different every time.

<key id> is the first 8 hexadecimal characters of the SHA-256 of the key. It shows which key protected a value, without telling anyone the key. A new key gives a new key id.

For example, with the credentials and payment categories, this record:

<86>Oct 17 10:00:00 web01 app: login user=alice password=hunter2 card=4111111111111111

reaches the target as follows:

Redact: ... login user=alice password=[redacted:generic-assignment] card=[redacted:credit-card-number]
Mask:   ... login user=alice password=[masked:generic-assignment] card=411111******1111
Hash:   ... login user=alice password=vmh1.630dcd29.980880de99ff10c727a2337e37b8b574 card=vmh1.630dcd29.837649812a0747b9c86c85615278e8b2

The hashes and the encrypted value on this page come from the key in Test Values. Your key gives other values.

In plain text, each value is replaced where it stands. In a JSON record, only the values change. Field names, nesting and field order stay as they were, and the record stays valid JSON:

{"user":"alice","card":"4111111111111111","amount":129.5}
{"user":"alice","card":"[redacted:credit-card-number]","amount":129.5}

A number that holds a sensitive value becomes a string.

warning

A destination with typed columns, such as a Microsoft Sentinel table, may refuse a string where it expects a number.

A Live Data capture on the target shows each record as the target receives it, with the protected values.

The Key

Hash and Encrypt need a key: 32 random bytes, written as 64 hexadecimal characters or in base64. The base64 form can be standard or URL-safe, 44 characters with the = at the end, or 43 without it.

The key can come from three places:

Vault - a secret in the Vault, such as $secret{id=42}. Every Vault provider works: VirtualMetric Vault, Azure Key Vault, HashiCorp Vault, AWS Secrets Manager and CyberArk. Keep the key in Vault if you can.
Environment variable - a variable on the director, such as $env{VM_PROTECT_KEY}.
Manual - typed in. The platform stores it as a secret and never shows it again. Generate a key makes one in your browser.

To make a key yourself:

In PowerShell: [Convert]::ToBase64String([Security.Cryptography.RandomNumberGenerator]::GetBytes(32))
With OpenSSL: openssl rand -base64 32, or openssl rand -hex 32 for the hexadecimal form.

caution

Keep a copy of the key where the people who need to read the data can reach it. Without the key, nobody can decrypt the values or find a value by its hash. VirtualMetric can't decrypt them for you.

The director reads a Vault key when it starts sending to the target. If you change the secret's value but keep the same reference, the director uses the new key after it restarts, or after the target's setting or the Library's rules change.

After you change the key, the values protected before keep the old key id. Keep the old key for as long as you keep data protected with it. Hashes made with the new key differ from the old ones, so you can't correlate values across the change.

Decrypting an Encrypted Value

To decrypt a vme1 value:

Split the value at its last dot. The part before the dot is vme1.<key id>.<rule key>. The part after it is base64url text, without padding.
Check that the key id matches the first 8 hexadecimal characters of the SHA-256 of your key.
Derive the AES key from your key with HKDF-SHA-256, no salt, the info string vm sensitive-data aead v1 and a 32-byte output.
Decode the base64url text. It holds a 12-byte nonce, the ciphertext and a 16-byte tag, in that order.
Decrypt with AES-256-GCM. The associated data is the text vme1.<key id>.<rule key>, in UTF-8.

The examples below read a key in standard base64. For a hexadecimal key, use [Convert]::FromHexString in PowerShell and bytes.fromhex in Python.

PowerShell (PowerShell 7.4 or later):

function Unprotect-SensitiveValue {
    param(
        [Parameter(Mandatory)] [string] $Key,    # the target's key: 32 bytes in base64
        [Parameter(Mandatory)] [string] $Value   # vme1.<key id>.<rule key>.<base64url>
    )
    if (-not $Value.StartsWith('vme1.')) { throw 'The value does not start with vme1.' }
    $keyBytes = [Convert]::FromBase64String($Key)

    # vme1.<key id>.<rule key> is everything before the last dot. It is also the associated data.
    $lastDot = $Value.LastIndexOf('.')
    $prefix  = $Value.Substring(0, $lastDot)
    $payload = $Value.Substring($lastDot + 1)
    $keyId   = $prefix.Substring(5, 8)
    $ruleKey = $prefix.Substring(14)

    # The key id is the first 8 hex characters of the key's SHA-256.
    $hash = [Security.Cryptography.SHA256]::HashData($keyBytes)
    $ownId = [Convert]::ToHexString($hash, 0, 4).ToLowerInvariant()
    if ($keyId -ne $ownId) { throw "The value was encrypted with key $keyId. This key is $ownId." }

    # The AES key comes from the target's key through HKDF-SHA-256.
    $info   = [Text.Encoding]::UTF8.GetBytes('vm sensitive-data aead v1')
    $aesKey = [Security.Cryptography.HKDF]::DeriveKey(
        [Security.Cryptography.HashAlgorithmName]::SHA256, $keyBytes, 32, [byte[]]@(), $info)

    # The base64url part is the 12-byte nonce, the ciphertext and the 16-byte tag.
    $b64 = $payload.Replace('-', '+').Replace('_', '/')
    $b64 += '=' * ((4 - $b64.Length % 4) % 4)
    $sealed     = [Convert]::FromBase64String($b64)
    $nonce      = [byte[]] $sealed[0..11]
    $tag        = [byte[]] $sealed[-16..-1]
    $ciphertext = [byte[]]::new($sealed.Length - 28)
    [Array]::Copy($sealed, 12, $ciphertext, 0, $ciphertext.Length)

    $plain = [byte[]]::new($ciphertext.Length)
    $aes = [Security.Cryptography.AesGcm]::new($aesKey, 16)
    try { $aes.Decrypt($nonce, $ciphertext, $tag, $plain, [Text.Encoding]::UTF8.GetBytes($prefix)) }
    finally { $aes.Dispose() }

    [pscustomobject]@{ Rule = $ruleKey; Value = [Text.Encoding]::UTF8.GetString($plain) }
}

PS> $key = 'AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8='
PS> Unprotect-SensitiveValue -Key $key -Value 'vme1.630dcd29.credit-card-number.CgsMDQ4PEBESExQVOLt1SSibokOB6TxbUzeWrfVfscud0j8Mdc8LZFC1YB4'

Rule               Value
----               -----
credit-card-number 4111111111111111

Python (needs the cryptography package: pip install cryptography):

import base64
import hashlib
import hmac

from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from cryptography.hazmat.primitives.kdf.hkdf import HKDF


def derive(key, info):
    # The director derives its AES and HMAC keys from the target's key with HKDF-SHA-256.
    return HKDF(algorithm=hashes.SHA256(), length=32, salt=None, info=info).derive(key)


def unprotect(key_b64, value):
    """Return the rule key and the original value of a vme1 value."""
    if not value.startswith("vme1."):
        raise ValueError("The value does not start with vme1.")
    key = base64.b64decode(key_b64, validate=True)

    # vme1.<key id>.<rule key> is everything before the last dot. It is also the associated data.
    prefix, payload = value.rsplit(".", 1)
    key_id, rule_key = prefix[5:13], prefix[14:]

    # The key id is the first 8 hex characters of the key's SHA-256.
    own_id = hashlib.sha256(key).hexdigest()[:8]
    if key_id != own_id:
        raise ValueError(f"The value was encrypted with key {key_id}. This key is {own_id}.")

    # The base64url part is the 12-byte nonce, then the ciphertext and the 16-byte tag.
    sealed = base64.urlsafe_b64decode(payload + "=" * (-len(payload) % 4))
    aes = AESGCM(derive(key, b"vm sensitive-data aead v1"))
    return rule_key, aes.decrypt(sealed[:12], sealed[12:], prefix.encode()).decode()


def hash_value(key_b64, value):
    """Return the vmh1 value that a target with this key writes for a value."""
    key = base64.b64decode(key_b64, validate=True)
    mac = hmac.new(derive(key, b"vm sensitive-data hmac v1"), value.encode(), hashlib.sha256).digest()
    return f"vmh1.{hashlib.sha256(key).hexdigest()[:8]}.{mac[:16].hex()}"

>>> key = "AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8="
>>> unprotect(key, "vme1.630dcd29.credit-card-number.CgsMDQ4PEBESExQVOLt1SSibokOB6TxbUzeWrfVfscud0j8Mdc8LZFC1YB4")
('credit-card-number', '4111111111111111')

Finding a Value by Its Hash

To find the records that hold a value you know, compute that value's hash with the target's key, and search the target's data for the result:

Derive the HMAC key from your key with HKDF-SHA-256, no salt, the info string vm sensitive-data hmac v1 and a 32-byte output.
Compute HMAC-SHA-256 of the value alone, in UTF-8, with that key.
Keep the first 16 bytes, written as 32 lowercase hexadecimal characters, after vmh1.<key id>..

A value hashes the same in a JSON record and in plain text. The Python example above has hash_value(). In PowerShell:

function Get-SensitiveValueHash {
    param(
        [Parameter(Mandatory)] [string] $Key,    # the target's key: 32 bytes in base64
        [Parameter(Mandatory)] [string] $Value   # the value to look for, such as a card number
    )
    $keyBytes = [Convert]::FromBase64String($Key)
    $keyId = [Convert]::ToHexString([Security.Cryptography.SHA256]::HashData($keyBytes), 0, 4).ToLowerInvariant()
    $info = [Text.Encoding]::UTF8.GetBytes('vm sensitive-data hmac v1')
    $hmacKey = [Security.Cryptography.HKDF]::DeriveKey(
        [Security.Cryptography.HashAlgorithmName]::SHA256, $keyBytes, 32, [byte[]]@(), $info)
    $mac = [Security.Cryptography.HMACSHA256]::HashData($hmacKey, [Text.Encoding]::UTF8.GetBytes($Value))
    'vmh1.{0}.{1}' -f $keyId, [Convert]::ToHexString($mac, 0, 16).ToLowerInvariant()
}

PS> Get-SensitiveValueHash -Key $key -Value '4111111111111111'
vmh1.630dcd29.837649812a0747b9c86c85615278e8b2

Test Values

Use these values to check your own code. The key is the bytes 0 to 31. It is public, so never use it to protect real data.

Key, in base64: AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8=
Key, in hexadecimal: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f
Key id: 630dcd29
Hash of 4111111111111111: vmh1.630dcd29.837649812a0747b9c86c85615278e8b2
4111111111111111 encrypted for the credit-card-number rule: vme1.630dcd29.credit-card-number.CgsMDQ4PEBESExQVOLt1SSibokOB6TxbUzeWrfVfscud0j8Mdc8LZFC1YB4. It decrypts to the rule credit-card-number and the value 4111111111111111.

The encrypted value uses a fixed nonce, the bytes 10 to 21, so that the example can be repeated. The director never reuses a nonce.

Directors That Don't Protect Data

An older director would ignore the setting and send sensitive data unprotected. So the platform gives a protected target only to a director that has confirmed, since it last connected, that it can protect data:

A director that runs an older release sends nothing to the target until you update it.
A director that is offline can't confirm anything, so it is left out too. It receives the target again within a few minutes after it is back online.
In a cluster, every member must confirm. One member that is offline holds the target back from the whole cluster until it reconnects.

A director that doesn't receive the target doesn't receive its quick routes either, or the target as the default target of an advanced route. Depending on its release, it drops the records it would have sent to the target at once, or retries them for up to 48 hours and then drops them.

The target's Data protection tab names these directors under Some directors don't send to this target, with their versions.

The platform sends each protected target in a separate part of the director's configuration, which older releases don't read. So a director that is downgraded to a release without data protection sends nothing to a protected target, even from the configuration it saved before. When you upgrade it again, it receives the target again without a manual step.

If you write a director's configuration by hand, put a protected target in the top-level protected part, beside targets and routes, with the routes that send to it. A route in protected replaces the route of the same name in routes. A protected target written under targets still works on a release that supports data protection, but an older release sends it unprotected.

targets:
  - name: "siem"
    type: "syslog"
    properties:
      address: "10.0.0.1"
routes:
  - name: "linux-to-siem"
    devices:
      - name: "test-linux"
    targets:
      - name: "siem"
protected:
  targets:
    - name: "vault"
      type: "syslog"
      properties:
        address: "10.0.0.9"
        sensitive_data:
          allowed: false
          action: "redact"
          categories: ["credentials"]
  routes:
    - name: "linux-to-vault"
      devices:
        - name: "test-linux"
      targets:
        - name: "vault"

Editions and Downgrades

Data protection is Enterprise only. On other editions, the platform refuses to turn protection on or to change it, and the REST API answers 403 FEATURE_NOT_AVAILABLE.

A target that was protected before a downgrade stays protected. The platform never starts sending sensitive data because a licence changed:

The target's tab shows the setting read-only, under Protection stays on. If you can edit the target, you can still turn protection off from the notice, after a confirmation. On that edition, you can't turn it back on.
Turning protection off never needs the Enterprise edition. On a smaller edition, use the console, because the REST API needs the Enterprise edition.
A clone of a protected target receives records as they are. The wizard's Data protection step warns about it with This target will receive sensitive data.

Configuration

The setting is sensitive_data, in the target's properties:

targets:
  - name: elasticsearch
    type: elastic
    properties:
      hosts: ["http://elasticsearch:9200"]
      index: "logs-%{+yyyy.MM.dd}"
      sensitive_data:
        allowed: false
        action: encrypt
        categories: ["credentials", "payment", "national_id"]
        rules: []
        key: "$secret{id=42}"

Field	Required	Default	Description
`allowed`	Y		`true` sends records as they are. `false` protects them. A target with no `sensitive_data` at all also sends records as they are
`action`	Y, when `allowed` is `false`		`redact`, `mask`, `hash` or `encrypt`
`categories`	N		Any of `credentials`, `payment`, `national_id`, `contact` and `network`. Used when `rules` is empty
`rules`	N	`[]`	Rule keys, such as `credit-card-number`. When the list holds any, only these rules apply, and `categories` is not used
`key`	Y, for `hash` and `encrypt`		The key itself, a `$secret{...}` Vault reference, or a `$env{...}` environment variable

When allowed is false, categories or rules must name at least one category or rule.

The platform refuses a setting it can't use with 400 TARGET_SENSITIVE_DATA_INVALID, and the message names the field, for example sensitive_data.key is missing. The hash action needs a key.

caution

Leaving sensitive_data out of a REST API or template update of a target turns protection off, as for any object left out of properties.

Synopsis​

Requirements​

Turning Protection On​

What Is Protected​

What the Target Receives​

The Key​

Decrypting an Encrypted Value​

Finding a Value by Its Hash​

Test Values​

Directors That Don't Protect Data​

Editions and Downgrades​

Configuration​

Synopsis

Requirements

Turning Protection On

What Is Protected

What the Target Receives

The Key

Decrypting an Encrypted Value

Finding a Value by Its Hash

Test Values

Directors That Don't Protect Data

Editions and Downgrades

Configuration