''Unconvertable character' exception in HTTPRequest node when sending a HTTPRequestHeader containing a special character

I'm getting an 'Unconvertable character' exception when sending a HTTPRequestHeader to HTTPRequest Node containing a special character.

Error in the ExceptionList when debugging:

RecoverableException
    File:CHARACTER:/jenkins/slot0/product-build/WMB/src/CommonServices/ImbConverter.cpp
    Line:INTEGER:733
    Function:CHARACTER:ImbConverterCPP::    
    Type:CHARACTER:
    Name:CHARACTER:
    Label:CHARACTER:
    Catalog:CHARACTER:BIPmsgs
    Severity:INTEGER:3
    Number:INTEGER:2136
    Text:CHARACTER:Unconvertable character
    Insert
            Type:INTEGER:5
            Text:CHARACTER:1920
    Insert
            Type:INTEGER:5
            Text:CHARACTER:4c0061006c0069006100192073002000420075007200690061006c002000460075006e006400200061006e006400200043006100720065002000450078007000 ...data truncated to first 64 chars
    Insert
            Type:INTEGER:2
            Text:CHARACTER:819

Snippet of my esql code:

SET OutputRoot.HTTPRequestHeader."Content-Type" = 'application/octet-stream';
SET OutputRoot.HTTPRequestHeader."X-IntegrationServer-Resource-Name" = rInputDocData.*:DocumentName;
--Setting the content
SET OutputRoot.BLOB.BLOB = BASE64DECODE(rInputDocAttachment.*:AttachmentData64Binary);

The special character is coming in the field rInputDocData.*:DocumentName. Some of the values are:

Printing….PDF
Mark’s Agreement .pdf

Notice the … and ’ in the above two values which are not recognized as part of UTF-8.

Is there a way to convert these values to ISO 8859-1 in ESQL as doing the same conversion in Notepad++ leads to the values getting accepted?

I have tried the below steps but none have worked and I'm still getting the same error:

  1. Setting OutputRoot.Properties.CodedCharSetId to 1208 and OutputRoot.Properties.Encoding to 546 as the properties were when the request was received in Input.

  2. Setting OutputRoot.Properties.CodedCharSetId to 819.

  3. Setting Content-Type HTTPRequestHeader to 'application/octet-stream; charset=utf-8'.

  4. Setting Content-Type HTTPRequestHeader to 'application/octet-stream; charset=iso-8859-1'.

  5. Casting the HTTPRequestHeader 'X-IntegrationServer-Resource-Name' in the below ways:

    1. SET OutputRoot.HTTPRequestHeader."X-IntegrationServer-Resource-Name" = CAST(rInputDocData.*:DocumentName AS CHARACTER CCSID 1208 ENCODING 546);
    2. SET OutputRoot.HTTPRequestHeader."X-IntegrationServer-Resource-Name" = CAST(rInputDocData.*:DocumentName AS CHARACTER CCSID 819);
    3. SET OutputRoot.HTTPRequestHeader."X-IntegrationServer-Resource-Name" = CAST(CAST(rInputDocData.*:DocumentName AS CHARACTER CCSID 1208 ENCODING 546) AS CHARACTER CCSID 819);

The source vendor have refused to handle/convert the special character so it is upon us to handle this in ACE. Any help would be much appreciated.

messagebrokeribm-integration-busextended-sqlibm-app-connect


Solution 1:[1]

The below worked for me, but as the top comment had pointed out, it did go on to fail with an encoding error in the target server:

SET OutputRoot.HTTPRequestHeader."X-IntegrationServer-Resource-Name" = CAST(CAST(rInputDocData.*:DocumentName AS BLOB CCSID InputRoot.Properties.CodedCharSetId) AS CHARACTER CCSID 819);

Solution 2:[2]

HTTP headers should only use characters in the US-ASCII range. Some other characters may be tolerated, but web server implementations are not consistent in this area.

See the discussions here for more details: what characters are allowed in HTTP header values? One of the participants in that conversation is an author of the HTTP/1.1 specification, so I think we can rely on his input.

re:

Notice the … and ’ in the above two values which are not recognized as part of UTF-8.

UTF-8 can represent any Unicode character, so that statement cannot be true. If you look carefully at the error, the quoted CCSID is 819, which is the IBM CCSID for ISO-8859-1.

Solution 3:[3]

It seems that the ... character is just not in the CCSID 819 charset. It's in the Windows version of Latin 1, CCSID 1252, but as the previous answer said, that may not work with the target Web server.

You can try these, but not sure any will work:

SET OutputRoot.HTTPRequestHeader."X-IntegrationServer-Resource-Name" = CAST(ASBITSTREAM(rInputDocData.*:DocumentName CCSID InputRoot.Properties.CodedCharSetId) AS CHARACTER CCSID 819);

SET OutputRoot.HTTPRequestHeader."X-IntegrationServer-Resource-Name" = CAST(ASBITSTREAM(rInputDocData.*:DocumentName CCSID InputRoot.Properties.CodedCharSetId) AS CHARACTER CCSID 1252);

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Goonerd
Solution 2 kimbert
Solution 3 Attila Repasi

Related Questions