'Folding JSON strings to send in header variables - Mailgun

I'm trying to use the JS Mailgun API to send emails. Have it working fine, until I throw template variables into 'h:X-Mailgun-Variables' like so, where jsonString is very large (17000+ characters):

const mailData = {
    from: 'Insights <[email protected]>',
    to: mailAddress,
    subject: `Insights: ${DAYS_OF_WEEK[date.getDay()]}, ${MONTHS[date.getMonth()]} ${ordinal_suffix_of(date.getDate())}  ${date.getFullYear()}`,
    template: "template1",
    'h:X-Mailgun-Variables': jsonString,
  };

Looking at the documentation here states the following:

Note The value of the “X-Mailgun-Variables” header must be valid JSON string, 
otherwise Mailgun won’t be able to parse it. If your X-Mailgun-Variables 
header exceeds 998 characters, you should use folding to spread the variables 
over multiple lines.

Referenced this post, which suggested I "fold" up the JSON by inserting CRLF characters at regular intervals. This led me here, which still does not work, though logging this does show regular line breaks and is compliant JSON:

const jsonString = JSON.stringify(templateVars).split('},').join('},\r \n');

Any insight into how to properly "fold" my JSON so I can use large template variables in my MailGun emails?

UPDATE:

As requested, adding my code. This works when data only has a few companies/posts, but when I have many companies each with many posts, I get the 400 error:

function dispatchEmails(data) {
  const DOMAIN = 'test.net';
  const mg = mailgun({apiKey: API_KEY, domain: DOMAIN});
  
  const templateVars = {
    date: theDate,
    previewText: 'preview',
    subject: 'subject',
    subhead: 'subhead',
    companies: data.companies.map(company => {
      return {
        url: company.url,
        totalParts: data.totalParts,
        currentPart: data.currentPart,
        companyData: {
          name: company.name,
          website: company.website,
          description: company.description
        },
        posts: _.map(company.news, item => {
          return {
            category: item.category,
            date: new Date(item.date),
            url: item.sourceUrl,
            title: item.title,
            source: item.publisherName,
            description: item.description,
          }
        })
      }
    })
  };

  const jsonString = JSON.stringify(templateVars).split('},').join('},\r \n');

  const mailData = {
    from: '[email protected]',
    to: '[email protected]',
    subject: 'subject',
    template: 'template',
    'h:X-Mailgun-Variables': jsonString
  };

  return mg.messages().send(mailData)
  .then(body => {
    return body;
  })
  .catch(err => {
    return {error: err};
  });
}

Solution 1:^[1]

Just thinking outside the box but why pass that much data in an email header? I assume you have something on the receiving end which is going to parse the email headers. What if instead of sending them that data you send them a key that they can call back into an API on your end to get the data

Solution 2:^[2]

I think your problem may be overall payload size rather than the string folding. The folding for strings exceeding 998 characters seems to be handled by the node.js client, possibly by the form-data package.

I ran the following test:

test_big_data (template created in mailgun through the UI)

<!DOCTYPE html>
<html>
    <body>
        <h2>An HTML template for testing large data sets.</h2>
        <ul>
            <li>{{param_name_0}}</li>
                ... other lis ...
            <li>{{param_name_999}}</li>
        </ul>
    </body>
</html>

send_email.js

const API_KEY = 'MY_KEY';
const DOMAIN = 'MY_DOMAIN.COM';

const formData = require('form-data');
const Mailgun = require('mailgun.js');

const mailgun = new Mailgun(formData);
const client = mailgun.client({ username: 'api', key: API_KEY });

const bigData = {};

for (let i = 0; i < 400; i++) {
    bigData[`param_name_${i}`] = `param_value_${i}`;
}

const dataString = JSON.stringify(bigData);
console.log(dataString.length);

const messageData = {
    from: 'Mailgun Sandbox <postmaster@DOMAIN>',
    to: 'test@MY_DOMAIN.COM',
    subject: 'Big Data Test',
    template: 'test_big_data',
    'h:X-Mailgun-Variables': dataString,
};

client.messages
    .create(DOMAIN, messageData)
    .then((res) => {
        console.log(res);
    })
    .catch((err) => {
        console.error(err);
    });

The length of the dataString in this case is 13781 characters and the email queues successfully.

However, if I bump the for loop condition to i < 1000 I get the following error when queueing the email:

[Error: Bad Request] {
  status: 400,
  details: '{"message":"Send options (parameters starting with o:, h:, or v:) are limited to 16 kB total"}\n'
}

When I asked Mailgun support about the folding warning form the documentation they pointed me to RFC 2822 section "3.2.3. Folding white space and comments". But like I said, I don't think folding is the issue here.

Cheers! https://datatracker.ietf.org/doc/html/rfc2822#page-11

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	JoshBerke
Solution 2	Jeremy Moyers

'Folding JSON strings to send in header variables - Mailgun

Solution 1:[1]

Solution 2:[2]

Sources

Related Questions

Solution 1:^[1]

Solution 2:^[2]