Send webpage to client, while receiving new data

So, the architecture you need here is:

• you have a backend that supports a GET request, that is, the page load
• the GET request should not wait for all your background jobs via an infinite loop, but serve the request with some response that does not include the resource you wait for yet
• you script a polling using setInterval on client-side that will send an AJAX request intermittently, so your client-side will do the sleeping, whilst your server will actively serve the request
• the AJAX request to be handled will send a POST request to the backend which will get the latest chunk of information and send it back to the client-side
• your client-side will handle the response

Example PHP part:

if ($SERVER['REQUESTMETHOD'] === 'POST') {
     // The request is using the POST method,
     // Get the new fresh data and echo that out
} else {
    // It's a GET request, serve it at once, don't make the client-side wait forever
}

So, your initial structure should be in the else branch above, whilst the if branch, that is, the "POST" part should check for new data and respond.

JS:

//function to send an AJAX request
function sendRequest(type, url, callback, async, params) {
    if (async !== false) async = true;
    var xhttp = new XMLHttpRequest();
    if (callback) {
        xhttp.onreadystatechange = function() {
            if (this.readyState === 4) {
                callback(this);
            }
        };
    }
    xhttp.open(type, url, async);
    xhttp.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
    xhttp.send("data=" + btoa(unescape(encodeURIComponent(JSON.stringify(params)))));
}
//we start polling after the page was loaded
window.addEventListener('load', function() {
    //The callback function passed as the first parameter to setInterval will be called every second
    setInterval(function() {
        sendRequest('POST', "/", function(r) {
            if(xhr.readyState === 4 && xhr.status === 200) { //success
                document.getElementById("output").innerText += r.responseText;
            }
        })
    }, 1000);
})

and of course this presumes you have an HTML element with the id of output.