Encoding Profile Load Nonuniformity Causes Issues

Steps to reproduce

Steps to reproduce:
1. Have two workers, both with N cores
2. Set an encoding profile's load value to N/4 on one worker, N/2 on the other
3. Start a workflow with those workers and encoding profile

(Possible) Actual Results:
Encoding job is created on N/4 worker, but processed on N/2 worker despite N/4 worker being preferred processor

Expected Results:
Encoding job is processed on N/4 worker, regardless of where it is created

Workaround (if any):
Set load value for encoding profile to N+1 on N/2 worker.

This issue was caused by me thinking about the load values as cost metrics, rather than as actual load values. If you're advertising cheaper encoding (the load value is lower) then there's probably a preference to process there: it's likely to be faster, for example. Right now the dispatcher sorts by current load per node, but imagine a system where the load values are 1 and 4 respectively, and processing on the second system taking twice the time the first system. It would not take long for it to be faster to load all of the encoding on the first machine, even if the encodes have to queue.

The solution here is to rewrite the job dispatcher, and the job architecture. Pull requests welcome

Status

Assignee

Greg Logan

Reporter

Greg Logan

Severity

Performance

Tags (folksonomy)

None

Components

Fix versions

Affects versions

Priority

Major
Configure