Thumbnail generation from Admin UI can fail on longer videos

Steps to reproduce

Steps to reproduce:
1. Ingest a long video (for example 8 hours)
2. Edit the video in the Admin UI
3. In the Thumbnail tab, click on "Extract from left video"

Actual Results:

On the admin node, thumbnail generation fails because the remote worker node takes longer than 60s to return a new element with the thumbnail.

2019-02-06 07:10:15,544 | ERROR | qtp1578637385-71978 | (RemoteBase:234) - Exception while trying to dispatch job to http://srvubuopc402.uct.ac.za:8080/composer/ffmpeg/imagesync: {}
org.opencastproject.security.api.TrustedHttpClientException: java.net.SocketTimeoutException: Read timed out
at org.opencastproject.kernel.security.TrustedHttpClientImpl.execute(TrustedHttpClientImpl.java:416)
at org.opencastproject.kernel.security.TrustedHttpClientImpl.execute(TrustedHttpClientImpl.java:346)
at org.opencastproject.serviceregistry.api.RemoteBase.getResponse(RemoteBase.java:214)
at org.opencastproject.serviceregistry.api.RemoteBase.getResponse(RemoteBase.java:142)
at org.opencastproject.composer.remote.ComposerServiceRemoteImpl.imageSync(ComposerServiceRemoteImpl.java:282)
at org.opencastproject.adminui.impl.ThumbnailImpl.chooseThumbnail(ThumbnailImpl.java:433)
at org.opencastproject.adminui.impl.ThumbnailImpl.chooseDefaultThumbnail(ThumbnailImpl.java:467)
at org.opencastproject.adminui.endpoint.ToolsEndpoint.editVideo(ToolsEndpoint.java:660)

On the worker node, the thumbnail is generated successfully, but takes longer than 60s to do so (89s in one observed case).

The admin node starts to mark multiple composer services as WARN or ERROR state.

Expected Results:

Thumbnail generated and returned successfully (if somewhat slowly).

Workaround (if any):

Don't extract thumbnails from longer videos.

Code fix is that RemoteBase should call TrustedHttpClient's execute() with timeout parameters (although this is still a somewhat undesirable user experience, waiting > 1min for the thumbnail to be generated).

Activity

Show:
Maximiliano Lira Del Canto
November 12, 2019, 9:19 AM
Stephen Marquard
February 6, 2019, 3:32 PM

Indeed, the Admin UI endpoint invokes a process on a worker node, and waits for the result before returning from the request.

This happens interactively when you click the button to generate a new thumbnail, but it's also triggered when you select "Save & Continue" if the thumbnail still has the default selected. This latter behaviour is really odd, since that should really just be done in the workflow that starts, rather than making the user wait for it before they get a response.

Karen Dolan
February 6, 2019, 3:28 PM

@Stephen Marquard, I don't know anything about this Thumbnail feature, but it's not a workflow that acts on a chosen inpoint? It does an ffmpeg extraction job live, without a workflow?

Stephen Marquard
February 6, 2019, 2:25 PM

The best (or worst) part about this is that most of the time taken by the operation on the worker node is download the media asset to the workspace. And you'd think if you did the operation again, it would be faster, because the track is kept in the workspace, but no: every time a new thumbnail is generated, there's a new asset manager snapshot created, and so the URL of the asset has changed, and so the track will get download each and every time you generate a thumbnail.

So if you happen to have say a 25G presenter video track and you are picky and try about 5 different thumbnail positions, you've just used up 125G in your shared workspace volume.

opencast.log:2019-02-06 15:43:16,762 | INFO | qtp1450833910-60096 | (WorkspaceImpl:403) - Downloading http://media.uct.ac.za/assets/assets/fcbd38f1-8f33-486c-ab0e-d470defab81a/track-0/4/presenter.mkv to /data/opencast/work/shared/workspace/http_media.uct.ac.za/assets/assets/fcbd38f1-8f33-486c-ab0e-d470defab81a/track-0/4/presenter.mkv
opencast.log:2019-02-06 15:49:35,922 | INFO | qtp1450833910-60121 | (WorkspaceImpl:403) - Downloading http://media.uct.ac.za/assets/assets/fcbd38f1-8f33-486c-ab0e-d470defab81a/track-0/8/presenter.mkv to /data/opencast/work/shared/workspace/http_media.uct.ac.za/assets/assets/fcbd38f1-8f33-486c-ab0e-d470defab81a/track-0/8/presenter.mkv
opencast.log:2019-02-06 16:09:53,922 | INFO | qtp1450833910-60201 | (WorkspaceImpl:403) - Downloading http://media.uct.ac.za/assets/assets/fcbd38f1-8f33-486c-ab0e-d470defab81a/track-0/12/presenter.mkv to /data/opencast/work/shared/workspace/http_media.uct.ac.za/assets/assets/fcbd38f1-8f33-486c-ab0e-d470defab81a/track-0/12/presenter.mkv
opencast.log:2019-02-06 16:17:45,126 | INFO | qtp1450833910-60230 | (WorkspaceImpl:403) - Downloading http://media.uct.ac.za/assets/assets/fcbd38f1-8f33-486c-ab0e-d470defab81a/track-0/16/presenter.mkv to /data/opencast/work/shared/workspace/http_media.uct.ac.za/assets/assets/fcbd38f1-8f33-486c-ab0e-d470defab81a/track-0/16/presenter.mkv
opencast.log:2019-02-06 16:19:48,665 | INFO | qtp1450833910-60241 | (WorkspaceImpl:403) - Downloading http://media.uct.ac.za/assets/assets/fcbd38f1-8f33-486c-ab0e-d470defab81a/track-0/20/presenter.mkv to /data/opencast/work/shared/workspace/http_media.uct.ac.za/assets/assets/fcbd38f1-8f33-486c-ab0e-d470defab81a/track-0/20/presenter.mkv

Assignee

Maximiliano Lira Del Canto

Reporter

Stephen Marquard

Severity

Incorrectly Functioning With Workaround