'Openssl 3.0. Rsa private methods not called

Recently I have migrated my app from OpenSSL 1.0 to OpenSSL 3.0. And now my connection is not working anymore. The methods which perform private encryption and private decryption are not called. This is what I used to have:

    RSA *rsaKey = RSA_new();
    BN_hex2bn(&rsaKey->n, n.c_str());
    BN_hex2bn(&rsaKey->e, e.c_str());
    RSA_set_method(rsaKey, &cardMethod);

    SSL_CTX_use_RSAPrivateKey(m_sslCtx, rsaKey);
    SSL_CTX_set_verify_depth(m_sslCtx, 2);   

This what the definition of cardMethod looked like:

static RSA_METHOD cardMethod = {
0,0,0,
cardPrivEnc,
cardPrivDec,
0,0,0,0,0,0,
0,0,0

};

These functions cardPrivEnc and cardPrivDec were later called during connection to server by SSL_connect inner implementation through RSA_sign.

Then I had to change my code to:

    RSA *rsaKey = RSA_new();
    m_cardMethod = RSA_meth_new("cardMethod", 0);
    RSA_meth_set_priv_dec(m_cardMethod, cardPrivDec);
    RSA_meth_set_priv_enc(m_cardMethod, cardPrivEnc);
    
    BIGNUM* nBignum = NULL;
    BIGNUM* eBignum = NULL;
    
    BN_hex2bn(&nBignum, n.c_str());
    BN_hex2bn(&eBignum, e.c_str());
    
    RSA_set0_key(rsaKey, nBignum, eBignum, NULL);
    RSA_set_method(rsaKey, m_cardMethod);

    SSL_CTX_use_RSAPrivateKey(m_sslCtx, rsaKey);
    SSL_CTX_set_verify_depth(m_sslCtx, 2);

The inner implementation of SSL_connect has changed in OpenSSL 3.0 obviously and now cardPrivEnc and cardPrivDec are not called. How do I fix it?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source