Using OpenSSL to Encrypt and Decrypt with CMS

Introduction #

This practical guide is a follow-up to the theoretical foundation laid in my article End-to-End security for sensitive data using CMS (aka PKCS#7). This one explained which encryption capabilities but more importantly which key management mechanisms are provided by the CMS encapsulation format. Learning purely based on theory is not everyone’s favorite, so this article will provide you with a comprehensive guide with complete examples showing your how to use OpenSSL CLI to take advantage of the CMS capabilities for encryption and key management.

The following examples are performed on bash and use the following OpenSSL version:

openssl version
OpenSSL 3.2.1 30 Jan 2024 (Library: OpenSSL 3.2.1 30 Jan 2024)

After a short overview, the article is structured based on the key management mechanisms. First, the mechanisms based on shared secrets are covered. Then, the mechanisms based on public key cryptography are explored.

CLI Overview #

The primary subcommand to perform CMS operations is cms. When following along with the examples, you can always refer to the cms man pages that provide a great level of detail for individual options.

man openssl-cms

Alternatively, you can also use the OpenSSL’s online documentation.

Since CMS is the basis for S/MIME, the cms subcommand has many options that relate to common e-mail terminology (e.g. -to, -from, -subject) . Even the default output format is S/MIME. For our purposes, we will not take advantage of these options.

For every OpenSSL cms command, the following top-level options are important:

• -print: Prints out the CMS structure in a human-friendly way with many details best suited for debugging purposes
• -outform <format>:
  • PEM: Outputs the CMS message in PEM format
  • DER: Outputs the CMS message in DER format
  • SMIME: Output format for S/MIME messages (default format if outform is missing)

To specify the cms service to apply, several flags are available:

• -encrypt and -decrypt: For encryption and decryption
• -sign and -verify: For signing and verifying
• -cmsout: A special mode that instructs OpenSSL to only output the CMS structure without performing any encryption or decryption. This is useful for debugging purposes in combination with -print.

Next, let’s look at the encryption capabilities and key management mechanisms that are supported.

Supported Encryption Capabilities #

As described in CMS Encryption Capabilities, CMS features in total three encryption modes (i.e. content types) of which only EnvelopedData and AuthEnvelopedData are supported via the CLI interface.

The mode used by OpenSSL CLI is implicitly decided based on the passed cipher. If an AEAD is configured, AuthEnvelopedData is used, else EnvelopedData. In addition, OpenSSL does not support all algorithms that are described by RFCs and on the other hand supports algorithms that have no formal RFC standing. The following table is a non-exhaustive list of supported ciphers and the corresponding CMS mode:

The above listed AES modes are supported with block sizes of 128, 192 and 256 bits. In addition, the CLI supports all of the RFC 3565 standardized AES key wrap algorithms: aes128-wrap, aes192-wrap, aes256-wrap.

Examples in this article will use AES-128-GCM AEAD together with AES-128-Wrap by default but you can freely change the algorthm and block sizes.

Supported Key Management Mechanisms #

CMS supports in total four key management mechanisms. All four mechanisms are supported by the OpenSSL CLI. The mechanism is chosen based on the options or the asymmetric key type. Here is the mapping between the mechanisms and CLI options:

• Symmetric Key Encryption: -secretkey and -secretkeyid
• Password-based Key Management: -pwri_password
• Key Transport & Key Agreement: -recip

Now we are getting to more complete examples.

Key management mechanisms based on shared secrets #

As described in my previous article, CMS offers two key management mechanisms based on shared secrets. Both mechanisms assume that the sender and receiver share a common secret but sharing this secret is out of CMS’ scope.

Symmetric Key Encryption #

Before getting started, we need some sample data to encrypt:

SAMPLE_FILE=sample.txt
echo '{ "accountBalance": 1000.0 }' > $SAMPLE_FILE  

Next, we are going to generate the symmetric key that is used to encrypt and is somehow shared between sender and receiver. The symmetric key MUST have a length suitable for the chosen encryption cipher. Our example uses AES-128, thus a key with 16 bytes is required. OpenSSL requires this key in hex format. We generate this key as follows:

SECRET_KEY=$(openssl rand -hex 16)

Sharing this key between senders and receivers is out CMS’ scope. However, CMS provides a mechanism to add an identifier for the secret key using the option secretkeyid. This allows an recipient to easily identify its key, though the distribution of this identifier is also out of CMS’ scope. Mind that the key identifier must be a valid hex string. With that, we can proceed to encrypt the data:

openssl cms -encrypt -in $SAMPLE_FILE \
  -secretkey $SECRET_KEY -secretkeyid "ad2024" \
  -wrap aes128-wrap -aes-128-gcm \
  -outform DER > "$SAMPLE_FILE"_enc-kek.p7m

Let’s inspect the generated CMS data structure:

openssl cms -cmsout -in "$SAMPLE_FILE"_enc-kek.p7m -inform DER -print

We see that we have a authEnvelopedData structure (line 3) with the mac field set (line 33-34) which is in line with our AEAD algorithm. The recipient information is stored in the KEKRecipientInfo (kekri) (line 7) structure as defined in RFC 5652 section 6.2.3. We see the expected key identifier (line 10-11), the key encryption algorithm (line 15) and the encryption algorithm (line 23). The key is encrypted using AES-128-Wrap (line 15) and is stored in the encryptedKey field (line 28).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

CMS_ContentInfo: 
  contentType: id-smime-ct-authEnvelopedData (1.2.840.113549.1.9.16.1.23)
  d.authEnvelopedData: 
    version: 0
    originatorInfo: <ABSENT>
    recipientInfos:
      d.kekri: 
        version: 4
        kekid: 
          keyIdentifier: 
            0000 - ad 20 24                                    . $
          date: <ABSENT>
          other: <ABSENT>
        keyEncryptionAlgorithm: 
          algorithm: id-aes128-wrap (2.16.840.1.101.3.4.1.5)
          parameter: <ABSENT>
        encryptedKey: 
          0000 - 16 78 18 4b a5 61 cf f6-2e 47 3d 82 fc c0 62   .x.K.a...G=...b
          000f - 81 7a 32 f8 db 35 0a 3d-73                     .z2..5.=s
    authEncryptedContentInfo: 
      contentType: pkcs7-data (1.2.840.113549.1.7.1)
      contentEncryptionAlgorithm: 
        algorithm: aes-128-gcm (2.16.840.1.101.3.4.1.6)
        parameter: SEQUENCE:
    0:d=0  hl=2 l=  17 cons: SEQUENCE          
    2:d=1  hl=2 l=  12 prim:  OCTET STRING      [HEX DUMP]:4294C59D723D3D8CA69C093C
   16:d=1  hl=2 l=   1 prim:  INTEGER           :10
      encryptedContent: 
        0000 - 20 2d 85 6b 39 95 25 d8-00 2d a1 de 3c d2 78    -.k9.%..-..<.x
        000f - 36 c5 31 13 b9 82 40 1b-41 c8 e0 9b 8c 75 0f   6.1...@.A....u.
    authAttrs:
      <ABSENT>
    mac: 
      0000 - b1 10 5f 9c 43 a9 54 14-96 e6 1c 6a d7 99 35 2a   .._.C.T....j..5*
    unauthAttrs:
      <ABSENT>

Using the same secret key and key identifier, the recipient can decrypt the encrypted message as follows:

openssl cms -decrypt -in "$SAMPLE_FILE"_enc-kek.p7m -inform DER \
  -secretkey $SECRET_KEY -secretkeyid "ad2024"

This should print our initial value { "accountBalance": 1000.0 }.

Password-based Key Management #

After using a symmetric key directly, we now derive a symmetric key from a password by using PBKDF2. This is the only supported KDF. Important to note is a bug tracked in issue #23634 that prevents us from using a AEAD algorithm together with password-based key management. Hence, only non-authenticated encryption is currently possible and we fall back to AES-CBC for this example.

We again generate some sample data to encrypt:

SAMPLE_FILE=sample.txt
echo '{ "accountBalance": 1000.0 }' > $SAMPLE_FILE  

Then, we can encrypt the data using some arbitrary password:

CMS_PWD=alexdippel.de2024
openssl cms -encrypt -in $SAMPLE_FILE \
  -aes-128-cbc -pwri_password $CMS_PWD \
  -outform DER > "$SAMPLE_FILE"_enc-pwri.p7m

Let’s analyze the generated CMS data structure:

openssl cms -cmsout -in "$SAMPLE_FILE"_enc-pwri.p7m -inform DER -print

There is nothing surprising. We have a PasswordRecipientInfo (pwri) data structure (line 7) as defined in RFC 5652 section 6.2.4 together with the KDF algorithm PBKDF2 (line 10). Since we changed to non-AEAD (line 28), we have a envelopedData structure (line 3).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

CMS_ContentInfo: 
  contentType: pkcs7-envelopedData (1.2.840.113549.1.7.3)
  d.envelopedData: 
    version: 3
    originatorInfo: <ABSENT>
    recipientInfos:
      d.pwri: 
        version: 0
        keyDerivationAlgorithm: 
          algorithm: PBKDF2 (1.2.840.113549.1.5.12)
          parameter: SEQUENCE:
    0:d=0  hl=2 l=  22 cons: SEQUENCE          
    2:d=1  hl=2 l=  16 prim:  OCTET STRING      [HEX DUMP]:BBCEF990415CC98389C95776EB08C5AA
   20:d=1  hl=2 l=   2 prim:  INTEGER           :0800
        keyEncryptionAlgorithm: 
          algorithm: id-alg-PWRI-KEK (1.2.840.113549.1.9.16.3.9)
          parameter: SEQUENCE:
    0:d=0  hl=2 l=  29 cons: SEQUENCE          
    2:d=1  hl=2 l=   9 prim:  OBJECT            :aes-128-cbc
   13:d=1  hl=2 l=  16 prim:  OCTET STRING      [HEX DUMP]:61C38D5EFFDAF6B095408FF72B1F6E23
        encryptedKey: 
          0000 - ef 08 2a b6 7a c4 4a 0d-7c 91 da 4d bd f7 77   ..*.z.J.|..M..w
          000f - 7e 96 96 f1 79 b8 53 c7-3c 69 70 10 0c 14 8f   ~...y.S.<ip....
          001e - 3c 0b                                          <.
    encryptedContentInfo: 
      contentType: pkcs7-data (1.2.840.113549.1.7.1)
      contentEncryptionAlgorithm: 
        algorithm: aes-128-cbc (2.16.840.1.101.3.4.1.2)
        parameter: OCTET STRING:
          0000 - 10 cc a3 15 7e 76 ad 74-7b 64 77 ea e6 1b ff   ....~v.t{dw....
          000f - 16                                             .
      encryptedContent: 
        0000 - ff ae 20 8b b6 8a da 1f-5e 2c a8 d4 6f b3 6d   .. .....^,..o.m
        000f - 7d 3a 91 3c 02 c1 92 03-ad 12 28 22 18 ce 10   }:.<......("...
        001e - 09 bf                                          ..
    unprotectedAttrs:
      <ABSENT>

Let’s see if we can decrypt the data using the previously used data:

openssl cms -decrypt -pwri_password $CMS_PWD -in "$SAMPLE_FILE"_enc-pwri.p7m -inform DER

This should print our initial value { "accountBalance": 1000.0 }.

Key Management Mechanisms based on Public Key Cryptography #

As you can read in my previous article, CMS provides two key management mechanisms based on public key cryptography. They assume that an originator has access to the public keys of the recipients. In the following examples, we will generate self-signed certificates to illustrate the usage of these mechanisms. In a real-world scenario, a more sophisticated approach for distributing certificates is required.

Using Key Transport with RSA #

Since key transport is only described for RSA, we start off by generating a self-signed certificate with an RSA key pair. Since we intend to encrypt a symmetric key with the public key, we add the keyEncipherment key usage to the certificate:

KT_RSA_RECIPIENT_CERT=rsa_recipient.crt
KT_RSA_RECIPIENT_KEY=rsa_recipient.key
openssl req -new -newkey rsa:2048 -noenc -keyout $KT_RSA_RECIPIENT_KEY \
	-out $KT_RSA_RECIPIENT_CERT -outform PEM -x509 -days 365 \
	-subj "/O=alexdippel.de/OU=CMS Test Laboratory/CN=rsa-recipient" \
	-addext "keyUsage=keyEncipherment"

Next, generate some sample data and save it to a file:

SAMPLE_FILE=sample.txt
echo '{ "accountBalance": 1000.0 }' > $SAMPLE_FILE  

Now, we can encrypt the data using our previously generated certificate. By default, OpenSSL uses the discouraged PKCS#1 v1.5 padding. We change that by setting the rsa_padding_mode to oaep thus using the preferred RSA-OAEP padding.

openssl cms -encrypt -in $SAMPLE_FILE \
    -recip $KT_RSA_RECIPIENT_CERT \
    -aes-128-gcm \
    -keyopt rsa_padding_mode:oaep \
    -outform DER > "$SAMPLE_FILE"_enc-ktri.p7m

Let’s inspect the resulting CMS structure:

openssl cms -cmsout -in "$SAMPLE_FILE"_enc-ktri.p7m -inform DER -print

We used AES-GCM (line 22) which performs AEAD and thus produces an authEnvelopedData (line 3) structure with the mac field set (line 32-33). Since we are using key transport, the recipient information is stored in a KeyTransRecipientInfo (ktri) (line 7) structure as defined in RFC 5652 section 6.2.1. RSA-OAEP (line 13) is used to encrypt the CEK (line 16) using the recipient’s public key.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

CMS_ContentInfo: 
  contentType: id-smime-ct-authEnvelopedData (1.2.840.113549.1.9.16.1.23)
  d.authEnvelopedData: 
    version: 0
    originatorInfo: <ABSENT>
    recipientInfos:
      d.ktri: 
        version: 0
        d.issuerAndSerialNumber: 
          issuer: O=alexdippel.de, OU=CMS Test Laboratory, CN=rsa-recipient
          serialNumber: 0x32DD28D7266ACA5A5EDEED0C3B5F9DCFAA1E6FD6
        keyEncryptionAlgorithm: 
          algorithm: rsaesOaep (1.2.840.113549.1.1.7)
          parameter: SEQUENCE:
    0:d=0  hl=2 l=   0 cons: SEQUENCE          
        encryptedKey: 
          0000 - 41 49 cd 77 c0 09 b8 90-1d 25 2a 0c fa 62 26   AI.w.....%*..b&
          // ...
    authEncryptedContentInfo: 
      contentType: pkcs7-data (1.2.840.113549.1.7.1)
      contentEncryptionAlgorithm: 
        algorithm: aes-128-gcm (2.16.840.1.101.3.4.1.6)
        parameter: SEQUENCE:
    0:d=0  hl=2 l=  17 cons: SEQUENCE          
    2:d=1  hl=2 l=  12 prim:  OCTET STRING      [HEX DUMP]:92023A0C62ED5EB1A8935851
   16:d=1  hl=2 l=   1 prim:  INTEGER           :10
      encryptedContent: 
        0000 - 05 45 6d b7 bf 2c 1f c2-a2 e1 7f b0 30 05 20   .Em..,......0. 
        000f - 70 f8 94 85 c5 f3 4e aa-6f b5 7a 53 f2 7c da   p.....N.o.zS.|.
    authAttrs:
      <ABSENT>
    mac: 
      0000 - c9 a3 4f fa 77 d1 9d 73-1d 32 25 a0 09 1e 43 44   ..O.w..s.2%...CD
    unauthAttrs:
      <ABSENT>

Since we can encrypt our message to several recipients leading to several KeyTransRecipientInfo (ktri) structures. A mechanism is needed helping a recipients efficiently identify the structure intended for them. For this, the certificate’s issuer line and the recipient’s serial number are copied into the structure (see line 10 and 11). The copied information match information on the certificate:

openssl x509 -in $KT_RSA_RECIPIENT_CERT -issuer -serial -noout
issuer=O=alexdippel.de, OU=CMS Test Laboratory, CN=rsa-recipient
serial=32DD28D7266ACA5A5EDEED0C3B5F9DCFAA1E6FD6

To decrypt the content again, we use the recipient’s private key that we conveniently stored in a file previously. In addition, we need the certificate again to identify the recipient structure using the issuer and serial number information:

openssl cms -decrypt -in "$SAMPLE_FILE"_enc-ktri.p7m -inform DER \
  -inkey $KT_RSA_RECIPIENT_KEY \
  -recip $KT_RSA_RECIPIENT_CERT

This should print our initial value { "accountBalance": 1000.0 }.

Using Key Agreement with Standard ECDH #

Key agreement is a key establishment mechanism that is only described for ECC in CMS. Since key transport and key agreement are mutually exclusive, OpenSSL implicitly chooses the correct key management mechanism based on the key pair used. We will generate a self-signed certificate with an ECC key pair:

KA_ECC_RECIPIENT_CERT=ecc_recipient.crt
KA_ECC_RECIPIENT_KEY=ecc_recipient.key
openssl req -new -newkey ec -pkeyopt group:prime256v1 -noenc -keyout $KA_ECC_RECIPIENT_KEY \
	-out $KA_ECC_RECIPIENT_CERT -outform PEM -x509 -days 365 \
	-subj "/O=alexdippel.de/OU=CMS Test Laboratory/CN=ecc-recipient" \
	-addext keyUsage=critical,keyAgreement

Since we perform a key agreement with the public key, we reflect that in the key usage extension of the certificate. Next, we again require our sample data in a file:

SAMPLE_FILE=sample.txt
echo '{ "accountBalance": 1000.0 }' > $SAMPLE_FILE  

Now, we can encrypt the data using the recipient’s certificate. Only the hash algorithm that is used together with the KDF is changed to sha256 since OpenSSL uses sha1 by default but SHA1 is discouraged and should be avoided.

openssl cms -encrypt -in $SAMPLE_FILE \
  -recip $KA_ECC_RECIPIENT_CERT \
  -aes-128-gcm \
  -keyopt ecdh_kdf_md:sha256 -aes128-wrap \
  -outform DER > "$SAMPLE_FILE"_enc-kari.p7m

Let’s inspect the resulting CMS structure:

openssl cms -cmsout -in "$SAMPLE_FILE"_enc-kari.p7m -inform DER -print

We used AES-GCM (line 23) which performs AEAD and thus produces an authEnvelopedData (line 3) structure with the mac field set (line 45,47). OpenSSL uses Standard ECDH (line 21) to establish a shared secret that is used to encrypt the CEK and which is then stored in the structure (line 29). As such, recipient information are stored in the KeyAgreeRecipientInfo (kari) (line 7) as defined in RFC 5652 section 6.2.2. OpenSSL only implements ECDH in an Ephemeral-Static mode (aka C(1e, 1s, ECC DH). To allow the recipient to perform ECDH on his site with the ephemeral public key, the originator key field contains the ephemeral public key (line 13). And like before, the recipients certificate information are copied into the structure (line 27, 28).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

CMS_ContentInfo: 
  contentType: id-smime-ct-authEnvelopedData (1.2.840.113549.1.9.16.1.23)
  d.authEnvelopedData: 
    version: 0
    originatorInfo: <ABSENT>
    recipientInfos:
      d.kari: 
        version: 3
        d.originatorKey: 
          algorithm: 
            algorithm: id-ecPublicKey (1.2.840.10045.2.1)
            parameter: <ABSENT>
          publicKey:  (0 unused bits)
            0000 - 04 0b 77 4b 7e 0c e8 ad-ee 00 89 d8 4c 18   ..wK~.......L.
            000e - fe 65 67 38 cb 4b c6 bc-c8 70 ff 0c 8d a3   .eg8.K...p....
            001c - 31 6c 54 55 2a 08 41 ca-b6 d4 43 19 14 f4   1lTU*.A...C...
            002a - 0f fc 7a b7 7e 5c a8 b8-41 af 32 6b cd ec   ..z.~\..A.2k..
            0038 - b2 ea 93 c7 03 3a 94 56-5f                  .....:.V_
        ukm: <ABSENT>
        keyEncryptionAlgorithm: 
          algorithm: dhSinglePass-stdDH-sha256kdf-scheme (1.3.132.1.11.1)
          parameter: SEQUENCE:
    0:d=0  hl=2 l=  11 cons: SEQUENCE          
    2:d=1  hl=2 l=   9 prim:  OBJECT            :id-aes128-wrap
        recipientEncryptedKeys:
            d.issuerAndSerialNumber: 
              issuer: O=alexdippel.de, OU=CMS Test Laboratory, CN=ecc-recipient
              serialNumber: 0x13BF15DF42D46BA585FA01DED8B7A197F59631F3
            encryptedKey: 
              0000 - c2 c9 71 87 4e e8 d2 18-a2 35 c2 47 a0 ea   ..q.N....5.G..
              000e - 31 69 da 4a 04 e7 28 37-6c 8e               1i.J..(7l.
    authEncryptedContentInfo: 
      contentType: pkcs7-data (1.2.840.113549.1.7.1)
      contentEncryptionAlgorithm: 
        algorithm: aes-128-gcm (2.16.840.1.101.3.4.1.6)
        parameter: SEQUENCE:
    0:d=0  hl=2 l=  17 cons: SEQUENCE          
    2:d=1  hl=2 l=  12 prim:  OCTET STRING      [HEX DUMP]:24CDD7A125CB10C1FB75816B
   16:d=1  hl=2 l=   1 prim:  INTEGER           :10
      encryptedContent: 
        0000 - d5 8a ce 5a 1c 9a 15 38-f6 17 96 df 47 cb 14   ...Z...8....G..
        000f - 21 5e 57 3d a5 d3 d9 7f-67 9c c5 43 fd 55 41   !^W=....g..C.UA
    authAttrs:
      <ABSENT>
    mac: 
      0000 - 85 9c ee e7 dd 39 d4 e6-22 f7 cc 07 f4 64 1a c9   .....9.."....d..
    unauthAttrs:
      <ABSENT>

To decrypt the content again, we use the recipient’s private key that we conveniently stored in a file previously as well as the certificate required to identify the recipient using the issuer and serial number information:

openssl cms -decrypt -in "$SAMPLE_FILE"_enc-kari.p7m -inform DER \
  -inkey $KA_ECC_RECIPIENT_KEY \
  -recip $KA_ECC_RECIPIENT_CERT

This should print our initial value { "accountBalance": 1000.0 }.

That’s it for now. I hope that this guide and the examples provide you with a good starting point for your journey of adopting CMS for your encryption needs. If you have any questions or feedback, feel free to reach out to me.