Convert X509v3 certificate from ASN1 DER to PEM using OpenSSL

Introduction #

When working with cryptographic assets and PKI artifacts - like X.509v3 certificates, PKCS#10 Certificate Signing Requests (CSRs), or asymmetric key pairs - you will be faced with several technologies and terms. The most prominent are:

• Abstract Syntax Notation (ASN.1)
• Distinguished Encoding Rules (DER)
• Base64 encoding
• Privacy Enhanced Mail (PEM)

All four are used together and can be converted to one another. The common basis is the representation of data structures using ASN.1. This is where this articles starts off. By the example of a X.509v3 certificate, we will dive into the ASN.1 definition, go through the different encodings and formats to finally end up with a PEM encoded certificate. Along that way, we will touch on the history of the technologies and we will use OpenSSL for practical examples.

From ASN.1 to DER #

Certificates, asymmetric key pairs, and other cryptographic and PKI artifacts are nothing fancy but data structures. When moving those data structures around between different parties, we need a common understanding of the data to ensure that it can be generated, transmitted, and used by all communication participants. This is where Abstract Syntax Notation comes in.

A quick look at ASN.1 and its uses in PKI #

Abstract syntax notation (ASN.1) is used

• to define data structures using a programming-language independent Interface Definition Language (IDL),
• to generate programming language specific data structure from the IDL using a compiler and
• to serialize and deserialize instances of the data structures in a platform agnostic way.

If these goals sound familiar, you might also think of technologies like Apache Avro, Google Protocol Buffers, or Apache Thrift which, to some varying extend, follow the same ideas. The main difference, though, is that ASN.1 has been around since 1984 and is used extensively across industries to power a lot of the protocols and communication that we use in our daily lives (5G/LTE, NFC, etc.). If you want to learn more about ASN.1, ASN.1 Complete by John Larmouth (free PDF) or the ASN.1 Survival Guide might be worth a read. If you want to play around with data structures and schemas, I recommend the ASN.1.io web tools.

Coming back to the PKI and cryptography space, ASN.1 is used all over the place. Examples include definitions of data structures for asymmetric key pairs, digital signatures or PKI artifacts like certificates and CSRs. Many of those data structures are defined and described in Request for Comments (RFCs) published by the Internet Engineering Task Force (IETF). Besides normative text, those RFCs also contain the ASN.1 schema definitions of the data structures essential for the protocol or technology described in the RFC.

Let’s consider a standard X.509v3 certificate as an example. We find the following ASN.1 schema definition in RFC 5280 Section 4.1 - Basic Certificate Fields:

Certificate  ::=  SEQUENCE  {
    tbsCertificate       TBSCertificate,
    signatureAlgorithm   AlgorithmIdentifier,
    signatureValue       BIT STRING  }  

The certificate structure is defined as an ASN.1 sequence. A sequence can be thought of as a struct in C/C++ and Go or as a class, but without methods, in object-oriented programming languages. This data structure is composed of several sub data structures like the TBSCertificate. This one carries the essential information like the issuer, the validity period, the subject, etc. If you are interest in more details, dive into RFC 5280 Section 4.1.

ASN.1 serialization and the importance of DER #

With the certificate definition at hand, we are still unable to read or write it with any tool or library. The missing piece is the serialization and deserialization of the data structure. Surprisingly, ASN.1 has several standardized serialization formats that suit different requirements. Some serialize to textual representations like XML or JSON, but in the space of PKI, binary encodings are more common. The most relevant binary encodings are:

• Basic Encoding Rules (BER)
• Distinguished Encoding Rules (DER)

BER is a flexible format allowing for several ways to serialize the same data structure which can lead to different binary representations. DER on the other hand mandates a unique encoding (read A Layman’s Guide to a Subset of ASN.1, BER, and DER for a more technical deep dive). As such, DER serializes to a deterministic binary representation of a data structure. This is an important property when dealing with hashes and signatures because they rely on a stable binary representation to work.

With that being said, BER bears the risk that during data transfers between systems the binary representation of the data structure might change. This can lead to errors when verifying signatures. So it comes to no surprise that RFC 5280 Section 5.1.1.3 explicitly mandates to calculate the signatureValue over the DER encoding of TBSCertificate to avoid those errors from happening.

Finally, we can put our knowledge to use. We use the PKI swiss army knife OpenSSL CLI to generate a RFC5280 compliant X.509v3 self-signed certificate, serialize it to DER and output it to a file:

openssl req -new -newkey ec -pkeyopt group:secp384r1 \
  -noenc -keyout cert.key \
  -out cert.cer -outform DER -x509 -days 1 \
  -subj "/CN=My very first certificate/"

We also generate a private key, which we will use at the end of the article.

When you do a cat cert.cer, you will see some binary gibberish output. This is the binary DER encoding of our generated certificate. You can dissect the binary DER encoding using the OpenSSL’s asn1parse command:

openssl asn1parse -in cert.cer -inform DER
    0:d=0  hl=4 l= 473 cons: SEQUENCE
    4:d=1  hl=4 l= 352 cons: SEQUENCE
    8:d=2  hl=2 l=   3 cons: cont [ 0 ]
   10:d=3  hl=2 l=   1 prim: INTEGER           :02
   13:d=2  hl=2 l=  20 prim: INTEGER           :4918A30996C98882373EE50CFFACFB87E8F4EB76
   35:d=2  hl=2 l=  10 cons: SEQUENCE
   37:d=3  hl=2 l=   8 prim: OBJECT            :ecdsa-with-SHA256
   47:d=2  hl=2 l=  36 cons: SEQUENCE
   49:d=3  hl=2 l=  34 cons: SET
   51:d=4  hl=2 l=  32 cons: SEQUENCE
   53:d=5  hl=2 l=   3 prim: OBJECT            :commonName
   58:d=5  hl=2 l=  25 prim: UTF8STRING        :My very first certificate
   // ...

Don’t worry if this is rather cryptic. You can also pretty print the contents using OpenSSL’s x509 command:

openssl x509 -in cert.cer -inform DER -text -noout

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            49:18:a3:09:96:c9:88:82:37:3e:e5:0c:ff:ac:fb:87:e8:f4:eb:76
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN=My very first certificate
// ...

Making the move from ASN.1 to DER was a big step, and it marked a significant point in the overall journey. Going forward, things are much more straightforward.

From DER to Base64 #

After showing your how DER is generated, you might ask yourself:

DER serialization is pretty straight forward and fine, why do we need something else?

And you are correct. Plain DER is used quite often. A notable usage of certificates is securing connections to servers using TLS. Looking at TLS 1.2 in RFC 5246, section 7.4.2, we see how a server sends its certificate (a certificate chain to be more pricse) to the client:

struct {
    ASN.1Cert certificate_list<0..2^24-1>;
} Certificate;

This shows that a TLS server sends plain ASN.1 DER encoded certificates to the client. This is fine since TLS is a binary protocol and there simply is no need for any other encoding producing unnecessary overhead. Though, other widely used formats and protocols are text-based. Think of HTTP, SMTP, and embedding a certificate into JSON, or even just wanting to send your colleague your awesome certificate via your preferred messenger.

This isn’t a new issue. The usual solution is to use Base64 encoding. This is a way of turning binary data into text. The encoding is done by taking 3 bytes of binary data and encoding them into 4 characters from a set of 64 characters ( see RFC 4648, section 4 for a more complete description of the encoding).

Reusing our previous example certificate, we can easily convert the DER encoded certificate using the command base64:

base64 < cert.cer
MIIF8jCCBNqgAwIBAgISAxA6bWzpwCCjwai1LuTf/UevMA0GCSqGSIb3DQEBCwUAMDIxCz...

And that’s it. A textual representation of our binary DER encoded certificate that we can easily send to our colleague or embed into a JSON document.

From Base64 to PEM #

Base64 encoding is a great way to encode binary data into a text-based format. But it has one big downside:

To deal with that, an simple mechanism can be a standardization as part of an application layer protocol. If you were to embed a Base64 encoded certificate into a JSON document, you could easily document its content type as part of the JSON schema or any other documentation around your protocol (e.g. an OpenAPI spec).

Another approach is to use a self-describing encoding. And this is where Privacy Enhanced Mail (PEM) enters the stage. You might wonder though how an e-mail format can be of any help here. RFC 1421 is the origin of PEM and it was developed with the goal to add security services to e-mails without requiring changes to the wider internet e-mail infrastructure. So the idea of an encapsulation format was born which carries a

• header and footer indicating start and end of the encapsulated data
• meta-data required to enable the security services
• and the actual Base64 encoded payload

RFC 1421, section 4.6 gives us a good summary of the format:

-----BEGIN PRIVACY-ENHANCED MESSAGE-----
Proc-Type: 4,ENCRYPTED
// more meta-data ...

LLrHB0eJzyhP+/fSStdW8okeEnv47jxe7SJ/iN72ohNcUk2jHEUSoH1nvNSIWL9M...
-----END PRIVACY-ENHANCED MESSAGE-----

Based on this, the PKI community has been using PEM ideas for years. In 2015, RFC 7468 was published to standardize a textual encoding for PKI artifacts based on PEM. Just a few parts of the original PEM encoding have been used. We’re basically left with a newline-terminated, fixed-line-length encoding with a header and footer. Here’s a quick summary of the format:

-----BEGIN [LABEL_NAME]-----
[Your Base64 encoded data trimmed to 64 chars long lines termina
ted with a newline character]
-----END [LABEL_NAME]-----

This easy format though solves our initial problem. We now have a self-describing encoding of our Base64 data. The RFC standardizes several Labels and assigns them to their corresponding ASN.1 data structures in section 4. For example:

• CERTIFICATE for encoding a X.509v3 Certificate
• CERTIFICATE REQUEST for encoding a PKCS#10 CertificationRequest
• PRIVATE KEY for encoding a PKCS#8 PrivateKeyInfo
• ENCRYPTED PRIVATE KEY for encoding a PKCS#8 EncryptedPrivateKeyInfo

With that said, the previous base64 encoded DER certificate can be transformed to PEM encoding by:

1. Trimming the base64 data to several 64 chars long lines terminated with a newline character
2. Adding the header and footer with the label CERTIFICATE

We can easily build this the manual way using bash to create a perfectly fine textual encoding of our certificate:

echo -n "-----BEGIN CERTIFICATE-----\n$(base64 < cert.cer)\n-----END CERTIFICATE-----\n" | fold -w 64
-----BEGIN CERTIFICATE-----
MIIB2TCCAWCgAwIBAgIUSRijCZbJiII3PuUM/6v7h+j063YwCgYIKoZIzj0EAwIw
JDEiMCAGA1UEAwwZTXkgdmVyeSBmaXJzdCBjZXJ0aWZpY2F0ZTAeFw0yNDEwMjQx
OTUzMzlaFw0yNDEwMjUxOTUzMzlaMCQxIjAgBgNVBAMMGU15IHZlcnkgZmlyc3Qg
Y2VydGlmaWNhdGUwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAATYNPdVE/72bs+lk13p
OD83nJQ5aR+oVBlG7iMWGZ6fUENlNRIa8hUlQ51NVVzevu28sm9JI2hJ+WfOAcD7
6S9H61huHmWXJRMqdoyoyNESkGHNWRQFUxdWMMmzo8BK2pWjUzBRMB0GA1UdDgQW
BBTo2xifR3WRg6LywEgNa4HeDCkRPjAfBgNVHSMEGDAWgBTo2xifR3WRg6LywEgN
a4HeDCkRPjAPBgNVHRMBAf8EBTADAQH/MAoGCCqGSM49BAMCA2cAMGQCMC6HD/9h
z/v6KaBttcdi+OdA0MkSJNXR+iwwa8CFJl1eeVS6Q6y2yW+9O8FiZBvxeAIwIeIC
tI5Dym1qDq79MHu4/fA6mjFLI7EL6MwmLQ52IhwBcEEo25n2GS1+dULekTYy
-----END CERTIFICATE-----

But we can also use our trusted OpenSSL CLI to do the encoding by simply changing the output format to PEM:

openssl x509 -in cer.der -outform PEM

-----BEGIN CERTIFICATE-----
MIIB2TCCAWCgAwIBAgIUSRijCZbJiII3PuUM/6v7h+j063YwCgYIKoZIzj0EAwIw
JDEiMCAGA1UEAwwZTXkgdmVyeSBmaXJzdCBjZXJ0aWZpY2F0ZTAeFw0yNDEwMjQx
OTUzMzlaFw0yNDEwMjUxOTUzMzlaMCQxIjAgBgNVBAMMGU15IHZlcnkgZmlyc3Qg
Y2VydGlmaWNhdGUwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAATYNPdVE/72bs+lk13p
OD83nJQ5aR+oVBlG7iMWGZ6fUENlNRIa8hUlQ51NVVzevu28sm9JI2hJ+WfOAcD7
6S9H61huHmWXJRMqdoyoyNESkGHNWRQFUxdWMMmzo8BK2pWjUzBRMB0GA1UdDgQW
BBTo2xifR3WRg6LywEgNa4HeDCkRPjAfBgNVHSMEGDAWgBTo2xifR3WRg6LywEgN
a4HeDCkRPjAPBgNVHRMBAf8EBTADAQH/MAoGCCqGSM49BAMCA2cAMGQCMC6HD/9h
z/v6KaBttcdi+OdA0MkSJNXR+iwwa8CFJl1eeVS6Q6y2yW+9O8FiZBvxeAIwIeIC
tI5Dym1qDq79MHu4/fA6mjFLI7EL6MwmLQ52IhwBcEEo25n2GS1+dULekTYy
-----END CERTIFICATE-----

Finally! We have successfully converted our X.509v3 certificate from ASN.1 DER to Base64 to PEM using only OpenSSL and a Terminal.

You can now apply your knowledge to all kinds of PKI artifacts like PKCS#10 CSRs or PKCS#8 keys. Do you remember the private key we generated at the beginning? Test your knowledge by finding out what format OpenSSL saved the key to cert.key. Can you convert it and print out any details about the key?

That’s all for today. I hope you enjoyed learning more about encoding in the PKI space. If you have any questions or feedback, feel free to reach out to me on one of my socials below.