Streaming API - how to send data from node JS connection (long-running HTTP GET call) to frontend (React application)

Question

We are attempting to ingest data from a streaming sports API in Node.Js, for our React app (a MERN app). According to their API docs: "The livestream read APIs are long running HTTP GET calls. They are designed to be used to receive a continuous stream of game actions." So we am attempting to ingest data from long-running HTTP GET call in Node Js, to use in our React app.

1. Are long-running HTTP GET calls the same as websockets? We have not ingested streaming data previously, so not sure about either of these.

So far, we have added the following code into the index.js of our node app:

index.js

...

// long-running HTTP request
http.get(`https://live.test.wh.geniussports.com/v2/basketball/read/1234567?ak=OUR_KEY`, res => {
   res.on('data', chunk => {
       console.log(`NEW CHUNK`);
       console.log(Buffer.from(chunk).toString('utf-8')); // simply console log for now
   });

   res.on('end', () => {
       console.log('ENDING');
   });
});

// Start Up The Server
const PORT = process.env.PORT || 8080;
app.listen(PORT, () => console.log(`Express server up and running on port: ${PORT}`));

This successfully connects and console-log's data in the terminal, and continues to console log new data as it becomes available in the long-running GET call.

2. Is it possible to send this data from our http.get() request up to React? Can we create a route for a long-running request in the same way that other routes are made? And then call that routes in React?

Answer 1

Server-sent events works like this, with a text/event-stream content-type header.

Server-sent events have a number of benefits over Websockets, so I wouldn't say that it's an outdated technique, but it looks like they rolled their own SSE, which is definitely not great.

My recommendation is to actually just use SSE for your use-case. Built-in browser support and ideal for a stream of read-only data.

Answer 2

To answer question #1:

Websockets are different from long-running HTTP GET calls. Websockets are full-duplex connections that let you send messages of any length in either direction. When a Websocket is created, it uses HTTP for the handshake, but then changes the protocol using the Upgrade: header. The Websockets protocol itself is just a thin layer over TCP, to enable discrete messages rather than a stream. It's pretty easy to design your own protocol on top of Websockets.

To answer question #2:

I find Websockets flexible and easy to use, though I haven't used the server-sent events that Evert describes in his answer. Websockets could do what you need (as could SSE according to Evert).

If you go with Websockets, note that it's possible to queue a message to be sent, and then (like any network connection) have the Websocket close before it's sent, losing any unsent messages. Thus, you probably want to build in acknowledgements for important messages. Also note that Websockets close after about a minute of being idle, at least on Android, so you need to either send heartbeat messages or be prepared to reconnect as needed.

Answer 3

If you decided to go on with websockets , I would recommend this approach using socket.io for both client and server:

Server-Side code:

const server = require('http').createServer();
const io = require('socket.io')(server);

io.on("connection", (socket) => {
  http.get(`https://live.test.wh.geniussports.com/v2/basketball/read/1234567?ak=OUR_KEY`, res => {
    res.on('data', chunk => {
      console.log(`NEW CHUNK`);
      let dataString = Buffer.from(chunk).toString('utf-8)
      socket.emit('socketData' , {data:dataString});
    });

    res.on('end', () => {
      console.log('ENDING');
    });
  });

});

server.liste(PORT);

Client-Side code:

import {
  io
} from 'socket.io-client';

const socket = io('<your-backend-url>');

socket.on('socketData', (data) => {
  //data is the same data object that we emited from server
  console.log(data)
  //use your websocket data
})