Encrypting and crypto-shredding of PII data in Marten documents using HashiCorp Vault
Protecting Personally Identifiable Information (PII) has become essential as organizations handle increasing volumes of sensitive data. Regulations like the GDPR (General Data Protection Regulation) mandate strict controls over PII to safeguard individuals' privacy and prevent data breaches. One effective approach to securing PII is data masking, a method that conceals sensitive information by replacing it with altered values, rendering it useless if accessed by unauthorized parties. Masking techniques such as character substitution, encryption, and tokenization are also used to protect data.
I have written about masking by characters substitution and encryption using a standard global key across Marten documents.
This blog post is a continuation of the encryption mechanism and will focus on using Hashicorp Vault backends, with granular per-document encryption key support. Also do crypto-shredding i.e. deliberately deleting or overwriting the encryption key which renders the encrypted data irrecoverable. It is important to read encryption using a standard global key across documents article which is a prerequisite for this blog post.
As a first step, let us look at the VaultEncryptionService
implementation which uses the HashiCorp Vault for encryption.
using VaultSharp;
using VaultSharp.V1.AuthMethods.Token;
using VaultSharp.V1.SecretsEngines.Transit;
namespace marten_docs_pii;
public class VaultEncryptionService: IEncryptionService
{
private readonly VaultClient _client;
private const string DefaultKeyName = "pii-key";
public VaultEncryptionService(string vaultAddress, string token)
{ var authMethod = new TokenAuthMethodInfo(token);
var vaultClientSettings = new VaultClientSettings(vaultAddress, authMethod);
_client = new VaultClient(vaultClientSettings);
} public async Task<string> EncryptAsync(string plainText, string? key = null)
{ key ??= DefaultKeyName;
var result = await _client.V1.Secrets.Transit.EncryptAsync(
key, new EncryptRequestOptions
{
Base64EncodedPlainText = Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(plainText))
}); return result.Data.CipherText;
}
public async Task<(bool success, string plainText)> TryDecryptAsync(string cipherText, string? key = null)
{ var (success, plainText) = await TryDecryptInternalAsync(cipherText, key);
return (success, plainText);
}
private async Task<(bool success, string plainText)> TryDecryptInternalAsync(string cipherText, string? key = null)
{ try
{
key ??= DefaultKeyName;
var result = await _client.V1.Secrets.Transit.DecryptAsync(
key, new DecryptRequestOptions { CipherText = cipherText });
var decryptedBytes = Convert.FromBase64String(result.Data.Base64EncodedPlainText);
return (true, System.Text.Encoding.UTF8.GetString(decryptedBytes));
} catch
{
return (false, string.Empty);
} }
public async Task DropEncryptionKeyAsync(string key)
{ if (string.IsNullOrWhiteSpace(key))
{ return;
} await _client.V1.Secrets.Transit.UpdateEncryptionKeyConfigAsync(key, new UpdateKeyRequestOptions
{
DeletionAllowed = true
});
await _client.V1.Secrets.Transit.DeleteEncryptionKeyAsync(key);
}}
This class provides encryption and decryption capabilities using HashiCorp Vault's Transit Secrets Engine. The service implements an IEncryptionService
interface and uses the VaultSharp client library to interact with Vault.
The service is initialized with a Vault address and authentication token, and uses a default encryption key named pii-key
. It provides three main operations:
EncryptAsync
: Encryptsplaintext
by first converting it to Base64, then using Vault's Transit engine to perform the encryption, returning the resultingciphertext
.TryDecryptAsync
: Attempts to decryptciphertext
using the specified key (or default key). It handles decryption failures gracefully by returning a tuple containing a success flag and the decrypted text (or empty string if decryption fails).DropEncryptionKeyAsync
: Provides functionality to delete an encryption key from Vault, first enabling deletion permission and then performing the actual deletion.
When key
is passed, it will used for both encryption and decryption which becomes document specific key. Otherwise, it uses the default encryption key which can be used across documents. Note that we are keeping the encryption keys secure in Vault rather than in the application itself to enhance its robustness and security.
Identification of document specific encryption key
Each document will need to implement the marker interface IHasEncryptionKey
as below
public interface IHasEncryptionKey {
string EncryptionKey { get; }
}
By way of implementing the interface, EncryptionKey
will need to be set for each document instance. EncryptionKey
will get used as the key for encryption in Vault for the properties pertaining to the document.
Let us look at how to add IHasEncryptionKey
to Person
record as below:
public record Address(string Street, string City);
public record Person(Guid Id, string Name, string Phone, Address Address) : IHasEncryptionKey
{
public string EncryptionKey => Id.ToString();
}
In the above code, EncryptionKey
is set with Id
as string.
Let us understand how this EncryptionKey
is fetched in TransformDocumentAsync
in EncryptionRules
for passing it for encryption related functions.
string? key = null;
// check if document implement IHasEncryptionKey
if (documentType.GetInterfaces().Any(x => x == typeof(IHasEncryptionKey)))
{
key = ((IHasEncryptionKey)document).EncryptionKey;
}
Call await _encryptionService.EncryptAsync(currentValue, key);
or await _encryptionService.TryDecryptAsync(currentValue, key)
passing the key
.
The following is the docker-compose file to run Vault as a docker container in your development environment:
services:
vault:
image: hashicorp/vault:latest
container_name: vault
ports:
- "8300:8300"
environment:
VAULT_DEV_ROOT_TOKEN_ID: "root"
VAULT_LOCAL_CONFIG: |
{
"listener": {
"tcp": {
"address": "0.0.0.0:8200",
"tls_disable": 1
}
},
"backend": {
"file": {
"path": "/vault/data"
}
},
"default_lease_ttl": "168h",
"max_lease_ttl": "720h",
"ui": true
}
volumes:
- ./vault-data:/vault/data
cap_add:
- IPC_LOCK
restart: unless-stopped
Note that we have configured Vault to run as a HTTP service instead of HTTPS in dev.
Now let us put all of these pieces together with an example usage. As a firsts step, create an instance of `VaultEncryptionService` as below:
```csharp
var encryptionService =
new VaultEncryptionService("http://localhost:8200", "root");
We are passing the Vault URL and token as a constructor.
As a next step, create the document store including setting up the encryption rules:
await using var store = DocumentStore.For(opts =>
{
opts.Connection(
"Host=localhost;Database=marten_testing;Username=postgres;Password=postgres");
opts.UseEncryptionRulesForProtectedInformation(encryptionService);
opts.Schema.For<Person>()
.AddEncryptionRuleForProtectedInformation(x => x.Name)
.AddEncryptionRuleForProtectedInformation(x => x.Phone)
.AddEncryptionRuleForProtectedInformation(x => x.Address.Street);
});
This code snippet demonstrates the configuration of Marten's DocumentStore
with encryption capabilities for protecting sensitive data. The configuration uses a fluent API to set up both database connection and encryption rules. First, it establishes a connection to a PostgreSQL database using standard connection parameters. Then, through UseEncryptionRulesForProtectedInformation
, it integrates the encryption service (VaultEncryptionService
in this case) into Marten's pipeline.
The most interesting part is the schema configuration for the Person
class, where it explicitly defines which properties should be encrypted using AddEncryptionRuleForProtectedInformation
. In this case, it marks Name
, Phone
, and the nested Address.Street
properties for encryption, demonstrating the system's ability to handle both top-level and nested property encryption.
Let us add a document and inspect the data in database as below:
await using var session = store.LightweightSession();
// Create and store a person
var personId = Guid.NewGuid();
var person1 = new Person(
personId,
"John Doe",
"111-111",
new Address("123 Main St", "Anytown"));
session.Store(person1);
await session.SaveChangesAsync();
When you inspect the "data at rest" as stored in database, you will see the below with the right set of properties encrypted via Vault i.e Name
, Phone
and Address.City
:
{
"Id": "ecd0be32-08d9-46f6-be03-a12355b8ab8a",
"Name": "vault:v1:h+y3qfgNEJ5IlldaUpM2yH2ZPoosQwP8Ecqrim2VqkSq5cgi",
"Phone": "vault:v1:7gHToyLraxeC4bX0eFWLedjDsrBWDKZv/2lqcuvolYS8Be4=",
"Address": {
"City": "Anytown",
"Street": "vault:v1:fsfACkhaXkqrPCPlaoYY+KN6B4uHvnbjGQfbsRzz5IIMTouRAynG"
},
"EncryptionKey": "ecd0be32-08d9-46f6-be03-a12355b8ab8a"
}
If you retrieve the document using Marten session, you will see that the data is decrypted properly as below:
var person = await session.LoadAsync<Person>(personId);
Console.WriteLine($"Name: {person?.Name}"); // Will show decrypted value
Console.WriteLine($"Phone: {person?.Phone}"); // Will show decrypted value
Console.WriteLine($"Street: {person?.Address.Street}, City: {person?.Address.City}");
You will see the output as below:
Name: John Doe
Phone: 111-111
Street: 123 Main St, City: Anytown
Name: John Doe, Phone: 111-111, Street: 123 Main St
Crypto-shredding
Crypto-shredding is the practice of rendering encrypted data unusable by deliberately deleting or overwriting the encryption keys, the data should become irrecoverable, effectively permanently deleted or "shredded".
To achieve this, we will do the following by calling DropEncryptionKeyAsync
:
await encryptionService.DropEncryptionKeyAsync(personId);
var person = await session.LoadAsync<Person>(personId);
Console.WriteLine($"Name: {person?.Name}"); // Will show decrypted value
Console.WriteLine($"Phone: {person?.Phone}"); // Will show decrypted value
Console.WriteLine($"Street: {person?.Address.Street}, City: {person?.Address.City}");
Now the output is:
Name: vault:v1:6uIYFOZC8jsYRrpN3WQyGfVgRiPbW5WTPmBuTDs9lxTVnqFX
Phone: vault:v1:XtVUMw9EKwtBV655AazcbHAFs/Th9FCM/lVcr/F+ChAjmQQ=
Street: vault:v1:BnErhi9JNvRPUvKQYh0y2fNyGLj3zAUX7vIXJSrDaKF1rG5zH2I0, City: Anytown
Name: vault:v1:6uIYFOZC8jsYRrpN3WQyGfVgRiPbW5WTPmBuTDs9lxTVnqFX, Phone: vault:v1:XtVUMw9EKwtBV655AazcbHAFs/Th9FCM/lVcr/F+ChAjmQQ=, Street: vault:v1:BnErhi9JNvRPUvKQYh0y2fNyGLj3zAUX7vIXJSrDaKF1rG5zH2I0
Notice that the encrypted data is returned "as is" since the decryption logic is not able to decrypt it. For the data itself, you could choose to do characters substitution like ****
, ###-###
etc use or combine functionality as explained in blog post masking by characters substitution
In summary, this implementation demonstrates document encryption using Marten with HashiCorp Vault with encryption per document. The code showcases a complete workflow for storing and retrieving encrypted documents. The Marten DocumentStore
is configured with encryption rules that specifically target sensitive fields in the Person
record: Name
, Phone
, and the nested Address.Street
property. The program creates a single Person
instance with sample data and demonstrates the transparent encryption/decryption process by storing and then retrieving the document. After saving, it verifies the encryption by loading the document back and displaying the automatically decrypted values. The Person
class is implemented as a simple record implementing IHasEncryptionKey
interface, meaning it uses per-document keys.
The source is available here for your ready reference. Happy coding :-)