Some time ago I was having a problem with an old software, that used digital signatures to validate its data files. The problem with the software was that it didn't fit in the new environment that was being built, and a replacement for the creation of these digital signatures was needed. This replacement had to be compatible with the old software, because the old software will still be used to validate signatures generated by the new software. Confusing, right?

Turned out this wasn't the main issue after I discovered that the old software doesn't accept new certificates. I started some digging and have learnt a thing or two about OpenSSL.

The following is an excerpt of an ASN.1 dump of a CMS digital sign message created by the old software:

 0:d=0  hl=4 l=1506 cons: SEQUENCE
 4:d=1  hl=2 l=   9 prim:  OBJECT            :pkcs7-signedData
15:d=1  hl=4 l=1491 cons:  cont [ 0 ]
19:d=2  hl=4 l=1487 cons:   SEQUENCE
23:d=3  hl=2 l=   1 prim:    INTEGER         :01
26:d=3  hl=2 l=  11 cons:    SET
28:d=4  hl=2 l=   9 cons:     SEQUENCE
30:d=5  hl=2 l=   5 prim:      OBJECT        :sha1
37:d=5  hl=2 l=   0 prim:      NULL
39:d=3  hl=2 l=  11 cons:    SEQUENCE
41:d=4  hl=2 l=   9 prim:     OBJECT         :pkcs7-data
52:d=3  hl=4 l= 929 cons:    cont [ 0 ]

The new software was trying to mimic this structure when generating the CMS sign. This is an excerpt of a signature generated by the new software:

 0:d=0  hl=4 l=1502 cons: SEQUENCE
 4:d=1  hl=2 l=   9 prim:  OBJECT            :pkcs7-signedData
15:d=1  hl=4 l=1487 cons:  cont [ 0 ]
19:d=2  hl=4 l=1483 cons:   SEQUENCE
23:d=3  hl=2 l=   1 prim:    INTEGER         :01
26:d=3  hl=2 l=   9 cons:    SET
28:d=4  hl=2 l=   7 cons:     SEQUENCE
30:d=5  hl=2 l=   5 prim:      OBJECT        :sha1
37:d=3  hl=2 l=  11 cons:    SEQUENCE
39:d=4  hl=2 l=   9 prim:     OBJECT         :pkcs7-data
50:d=3  hl=4 l= 929 cons:    cont [ 0 ]

As you can see, first dump has an additional NULL argument in offset 37. There are 3 or 4 additional places where the old tool produces additional NULL, where OpenSSL doesn't produce it. In binary form, the NULL item is implemented by 05 00 bytes.

This absence of "padding" (from the lack of other word for it) in CMS generated by the new software makes the old software unable to properly validate the digital sign. After I've inserted this "padding" into the signature directly by using the hex-editor tool (yay!), and after I've fixed the lengths of ASN.1's TLV tokens and their parents, the old software started to properly recognize the new digital sign.

After spending some time with OpenSSL and trying to force it to adhere to a different ASN.1 formatting, I've discovered the file \crypto\evp\m_sha1.c, which needed to be patched like this:

 static const EVP_MD sha1_md=
     {
     NID_sha1,
     NID_sha1WithRSAEncryption,
     SHA_DIGEST_LENGTH,
-    EVP_MD_FLAG_PKEY_METHOD_SIGNATURE|EVP_MD_FLAG_DIGALGID_ABSENT,
+    EVP_MD_FLAG_PKEY_METHOD_SIGNATURE,

(so, the patch removes the EVP_MD_FLAG_DIGALGID_ABSENT flag).

After this patch, OpenSSL started to include the NULL argument after AlgorithmIdentifier.