How do I simulate, for testing purposes, connection loss and slowness between a client and a service?

I had a similar problem a while back with storing file uploads on a network drive that was... less than stable. In my case, I was able to isolate the part that interacts with the file system directly, and move retry/error handling logic into something else.

I opted for building a failure simulator class that implemented an interface, which was used in place of the real file system object. The interface was simple — the bare minimum necessary to move files between a temporary upload folder and permanent storage, as well as a "trash" folder. The API was small and domain-specific, so I had very little to mock.

The failure simulator implementing this interface exposed additional methods and properties allowing tests to specify their own behavior. The retry mechanism was the system under test, which was in a class representing a file transaction. Each file was represented as an "entry"; this was the interface the failure simulator implemented.

The failure simulator had a simple list of lambda expressions (this was C#, so they were Action objects, specifically). Part of the test setup was building a list of function invocations:

var entry = new FileEntryFailureSimulator()
{
    StorePermanentlyCalls = [
        () => throw new FileNotFoundException(),
        () => throw new DirectoryNotFoundException(),
        () => throw new InvalidOperationException(),
        () => { / Succeed / }
    ]
};
var fileTransaction = new FileTransaction(baseFilePath);
fileTransaction.StorePermanently(entry);
entry.Location.ShouldBe(FileLocation.PermanentStorage);

Internally, the method in FileEntryFailureSimulator that moved the file into permanent storage just looped over the list of Action objects and invoked them one-by-one. This put my tests in the driver's seat, and allowed me to verify this retry logic in exquisite detail; that's the sweet spot for tests like this.

So, that's one approach; isolate the error handling logic from the error-prone API, and build your own simulator implementing a small, domain-specific interface.

Years ago, I wrote a small JavaScript library, which allowed me to build a complex series of connect/disconnect events, and simulate various data coming back in an AJAX request. It was a general purpose library focused on network communication in JavaScript. This was a more sophisticated API, but was reusable between projects. It had a fluent interface allowing the test writer to express the use case, which made reading the test easier. A fluent API is also nice because it aids in discoverability, which is a must-have feature for any esoteric API.

var strategy = new MockingBird.XMLHttpRequest()
    .connects()
    .receivesHeaders(200, { "content-type": "text/html; charset=utf8" })
    .receivesChunk(200, "A")
    .receivesChunk(200, "B")
    .disconnects()
    .receivesChunk(200, "C")
    .done(200, "D");
callSomeFunctionThatUsesAjax(strategy.xhr);
expect(strategy.xhr.responseText).toBe("ABCD");

Again, you'll need to isolate the error handling logic from the HTTP calls, and provide a mock object instead of a real HTTP client.

Whether you decide on a single domain-specific API, or create a fluent interface depends heavily on how your application is written, and how easy it is to isolate the error handling logic from the network calls.

Setting up a dedicated environment allowing you to interfere with real network connections is doable, but complicated and time-consuming. Maintenance of this environment becomes another chore, and integration with an existing CI/CD pipeline might not be possible. As a result, these tests are not likely to be run frequently, or even properly maintained.

I prefer either of the first two approaches I outlined above because you can run these tests in any CI/CD pipeline, and they are easy to run on your development machine, not to mention easy to debug line-by-line in a good IDE (another must-have for complex logic like this).

You make a trade-off: spend time inventing your own esoteric test API, and gain reprodicible fast tests; or spend time creating an environment that does seemingly questionable things to a machine to create tests that are hard to rerun as time goes on, but can provide test scenarios closer to real life.

I fully agree to Greg Burghardts answer. A mockable test interface or library is certainly the most reliable way to have repeatable and stable tests for CI/CD.

Only if such mocking isn't possible, e.g. when it is not feasible to influence the message forwarding on the necessary level, then you could fall back to a more complex approach like a proxy or something like a man-in-the-middle.
The more individual components you need, the more difficult it gets to set the test up and it becomes more likely that tests become unstable or run into other issues.

For example, I have such a configurable man-in-the-middle approach in use for some low-level tests, with the drawback that any security related tests detect and reject this as an "evil" man-in-the-middle.