Finding the contentLength for response from Salesforce Connector Mule 4

Question

I have been working on a mule app(4.4.0) where it calls a salesforce query connector and converts the response from SF connector to CSV and then upload the CSV to AWS s3 via create an object connector.

I am getting errors as: "com.amazonaws.SdkClientException: Data read has a different length than the expected"

My code config looks like below:-

        <salesforce:query doc:name="sobject from salesforce" doc:id="58660352-f117-47e3-91cf-587f5000d591" config-ref="salesforceConfig" >
                    <salesforce:salesforce-query >#[vars.query]</salesforce:salesforce-query>
                    </salesforce:query>
                    
    <ee:transform doc:name="convert to csv" doc:id="ed22a2df-82a6-4162-9c3d-eab29dc92a4f">
                    <ee:message>
                        <ee:set-payload><![CDATA[%dw 2.0
        output application/csv quoteHeader=true, quoteValues=true
        ---
        payload]]></ee:set-payload>
                    </ee:message>
                </ee:transform>

<ee:transform doc:name="Transform Message" doc:id="62f61426-f0df-4640-bb27-4a980a4f26fa" >
            <ee:message >
            </ee:message>
            <ee:variables >
                <ee:set-variable variableName="contentLength" ><![CDATA[%dw 2.0
output application/java
---
sizeOf(write(payload, "application/java"))]]></ee:set-variable>
            </ee:variables>
        </ee:transform>
    
    <s3:create-object doc:name="Create object" doc:id="136d5816-846c-4e52-acb6-0caeacb25dba" config-ref="Amazon_S3_Configuration" bucketName="#[vars.s3BucketName]" key='#[(if(!isEmpty(vars.s3WriteKey)) vars.s3WriteKey ++ "/" else "") ++ vars.fileNameOnS3]' contentLength="#[vars.contentLength]"/>

Its creating exception as like below:-

com.amazonaws.SdkClientException: More data read than expected: dataLength=1048; expectedLength=7; includeSkipped=false; in.getClass()=class com.amazonaws.internal.ReleasableInputStream; markedSupported=false; marked=0; resetSinceLastMarked=false; markCount=0; resetCount=0

I could see this error is due to contentLength which i calculate based on payload size could not match . I tried various options like sizeOf(payload) , sizeOf(write(payload)) etc . all same error.

Any helps really appreciated. Many Thanks in Advance

Answer 1

Short Version:

Opt for either of the two:

1. Do not pass the contentLength Param unless it is required for some usecase, as it is handled automatically.
2. If, for some reason, you have to pass the contentLength you should write your payload as String using write(payload, 'application/csv', {quoteHeader: true, quoteValues: true}). Then you can get the size of the payload using either sizeOf(payload) or the Content Length Metadata Selector payload.^contentLength (before using later, read the 3rd point in the long version)

Long Version:

While calculating the size of your CSV you are writing the payload as application/java, which will write your CSV into a Java ArrayList and then calculate the size of that string.

Now you would think that changing application/java to application/csv will solve the problem, but no. If you check the documentation of write function, you will find that it accepts the third parameter called writerProperties, which in your case, is {quoteHeader: true, quoteValues: true} (These are the ones that you are using while creating the payload). Because these properties are not the default ones, you will get a different output from the write function from that of your payload if you do not specify these properties in your write function.

What you should do?

1. I would first see if I really need to pass the contentLength myself? As far as I know it will be handled automatically.
2. If you really need to pass the contentLength, you should first write your payload as in the desired format after you transform it as CSV, something like the following

%dw 2.0
output text/plain
---
write (payload, "application/csv", {quoteHeader: true, quoteValues: true})

Then you can use the sizeOf(payload). You should not be modifying the payload while calculating the size, because, that modification could change the length, as you just experienced.

3. Requires Testing:. Since you are using the application/csv content type you can get the contentLength directly using the Content Length Metadata Selector. i.e. payload.^contentLength. However, I am saying that it requires testing, because this metadata selector sometimes returns null, for example, when the payload output type is application/java. I am 99% sure with csv you will always get the content type metadata, but not 100% as I do not know why it is not populated sometimes in other MIME Types. If you choose to use this, your s3 connector will look like this

<s3:create-object doc:name="Create object"
                  doc:id="136d5816-846c-4e52-acb6-0caeacb25dba"
                  config-ref="Amazon_S3_Configuration"
                  bucketName="#[vars.s3BucketName]"
                  key='#[(if(!isEmpty(vars.s3WriteKey)) vars.s3WriteKey ++ "/" else "") ++ vars.fileNameOnS3]'
                  contentLength="#[payload.^contentLength]"/>