'How to replicate bouncycastle CMSEnvelopedData decryption using Openssl

I have working code that uses Java Bouncycastle lib that (as I understood) decrypts CMS data.

byte[] encryptedData = Base64.decode(encryptedText);
CMSEnvelopedData cmsEnvelopedData = new CMSEnvelopedData(encryptedData);
RecipientInformationStore recipients = cmsEnvelopedData.getRecipientInfos();
X509CertificateHolder decryptPublicCertificateHolder = new X509CertificateHolder(decryptPublicCertificate.getEncoded());

for (RecipientInformation recipient : recipients) {
    org.bouncycastle.cms.RecipientId recipientId = recipient.getRID();          
    Recipient decryptRecipient = new JceKeyTransEnvelopedRecipient(decryptPrivateKey);
    byte[] plainData = recipient.getContent(decryptRecipient);
    String plainText = new String(plainData, getEncoding());
    // ...      
}
// ...

Certificate and private key are stored together in .pfx file. Encrypted message comes as string in Base64

To decrypt the message using openssl I:

  1. Extracted cert and private key into 2 separate PEM files:
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Private key is decrypted

-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
  1. Added header/footer to encrypted message
-----BEGIN PKCS7-----
...
-----END PKCS7-----
  1. Tried to decrypt the message
openssl cms -decrypt -inform PEM -in encrypted_message_file -inkey private_key.pem -recip certificate.pem

As a result I get

Error decrypting CMS structure
16972:error:060CC07A:digital envelope routines:EVP_CIPHER_asn1_to_param:cipher parameter error:../openssl-1.1.1e/crypto/evp/evp_lib.c:79:
16972:error:2E078066:CMS routines:cms_EncryptedContent_init_bio:cipher parameter initialisation error:../openssl-1.1.1e/crypto/cms/cms_enc.c:80

I also tried remove header/footer for encrypted message, tried provide .pfx file as private key and many more but no result

I'm very new to that stuff. Could you please suggest what am I doing wrong?

UPD1: openssl pkcs7 -in encrypted_message_file -text produces

unable to load PKCS7 object
2496:error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../openssl-1.1.1e/crypto/asn1/tasn_dec.c:1130:
2496:error:0D07803A:asn1 encoding routines:asn1_item_embed_d2i:nested asn1 error:../openssl-1.1.1e/crypto/asn1/tasn_dec.c:290:Type=PKCS7_ISSUER_AND_SERIAL
2496:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../openssl-1.1.1e/crypto/asn1/tasn_dec.c:627:Field=issuer_and_serial, Type=PKCS7_RECIP_INFO
2496:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../openssl-1.1.1e/crypto/asn1/tasn_dec.c:595:Field=recipientinfo, Type=PKCS7_ENVELOPE
2496:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../openssl-1.1.1e/crypto/asn1/tasn_dec.c:627:       
2496:error:0D08403A:asn1 encoding routines:asn1_template_ex_d2i:nested asn1 error:../openssl-1.1.1e/crypto/asn1/tasn_dec.c:477:Field=d.enveloped, Type=PKCS7
2496:error:0906700D:PEM routines:PEM_ASN1_read_bio:ASN1 lib:../openssl-1.1.1e/crypto/pem/pem_oth.c:33:

UPD2 in response to dave_thompson_085 comment

openssl asn1parse -i -in encrypted.pem

    0:d=0  hl=4 l=2307 cons: SEQUENCE          
    4:d=1  hl=2 l=   9 prim:  OBJECT            :pkcs7-envelopedData
   15:d=1  hl=4 l=2292 cons:  cont [ 0 ]        
   19:d=2  hl=4 l=2288 cons:   SEQUENCE          
   23:d=3  hl=2 l=   1 prim:    INTEGER           :02
   26:d=3  hl=4 l= 304 cons:    SET               
   30:d=4  hl=4 l= 300 cons:     SEQUENCE          
   34:d=5  hl=2 l=   1 prim:      INTEGER           :02
   37:d=5  hl=2 l=  20 prim:      cont [ 0 ]        
   59:d=5  hl=2 l=  13 cons:      SEQUENCE          
   61:d=6  hl=2 l=   9 prim:       OBJECT            :rsaEncryption
   72:d=6  hl=2 l=   0 prim:       NULL              
   74:d=5  hl=4 l= 256 prim:      OCTET STRING      [HEX DUMP]:512_LENGTH_HEX_STRING
  334:d=3  hl=4 l=1973 cons:    SEQUENCE          
  338:d=4  hl=2 l=   9 prim:     OBJECT            :pkcs7-data
  349:d=4  hl=2 l=  12 cons:     SEQUENCE          
  351:d=5  hl=2 l=   8 prim:      OBJECT            :rc4
  361:d=5  hl=2 l=   0 prim:      NULL              
  363:d=4  hl=4 l=1944 prim:     cont [ 0 ]        


Solution 1:[1]

Okay. My initial goal was to decrypt CMS message. In my case the message contains data key encrypted by rsa and content encrypted by rc-4.

I couldn't decrypt the message using Openssl, instead I used https://github.com/lapo-luchini/asn1js npm package to build ASN1 object from the CMS message, extract encrypted data key, decrypt that data key and decrypt content using that key.

I used https://datatracker.ietf.org/doc/html/rfc5652 to read about CMS structure. Also I used UI to visualize parsed message (https://lapo.it/asn1js/).

UPD1.

  • Eventually we decided to abandon CMS rc4. So my question is irrelevant.
  • No, I didn't find the way to decrypt the message via openssl (probably it's not possible since rc4 is not supported)
  • I decrypt it using nodejs but it was so much pain

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1