Ingest load should be balanced across ingest servers

Steps to reproduce

In a configuration with multiple ingest servers, all ingest servers are not favoured equally, with the first ingest server getting more ingests than others.

This appears to be because the REST call used by the CAs:


returns the ingest server in order of load (least loaded first), but in cases where the load is equal (e.g. load = 0 for all), the servers are returned deterministically.

To ingests allocated fairly across ingest servers (for example for an even distribution of network traffic across interfaces), the server list should be randomized within the same load value, e.g.

ingestA load 0
ingestB load 0
ingestC load 0

Each time the service list is called, these should be returned in a random order (as long as they all have load=0), e.g.

ingestC, ingestA, ingestB
ingestC, ingestB, ingestA
ingestA, ingestC, ingestB




Stephen Marquard