Retrieving Page from Spotify URL in Node.js Now Results in 406 Not Acceptable Error

Question

Using the following code in an async function and with the got library:

var { body: html, url } = await got('https://open.spotify.com/track/1iFwvtvFNbQVcIXKBiBRqD')

I was previously able to get a song page from the Spotify website in order to scrape its metadata with metascraper. However, it stopped working recently, now giving an error:

HTTPError: Response code 406 (Not Acceptable)
at Request.<anonymous> (/app/node_modules/got/dist/source/as-promise/index.js:117:42)
at processTicksAndRejections (node:internal/process/task_queues:96:5) {
code: 'ERR_NON_2XX_3XX_RESPONSE',
timings: {
start: 1641245342392,
socket: 1641245342394,
lookup: 1641245342396,
connect: 1641245342398,
secureConnect: 1641245342407,
upload: 1641245342408,
response: 1641245342538,
end: 1641245342542,
error: undefined,
abort: undefined,
phases: {
wait: 2,
dns: 2,
tcp: 2,
tls: 9,
request: 1,
firstByte: 130,
download: 4,
total: 150

Since I am not aware of any changes I made that would have impacted this, what changed and how do I fix it? It seems like the code still works for other URLs I've tried, so maybe Spotify itself changed something? Thanks.

Answer 1

I think you just need to add options in which to write headers. Although your code works for me without them.

const options = {
    headers: {
      Accept: "application/json",
      "Content-Type": "application/json",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36",
    },
  };
var { body: html, url } = await got("https://open.spotify.com/track/1iFwvtvFNbQVcIXKBiBRqD", options);