Retract workflow fails if run when a video is being played (with nfs storage)

Steps to reproduce

Steps to reproduce:
1. Publish a recording (e.g. 30 to 60 min)
2. Start playing the recording with Theodul player
3. In the Admin UI, run the Retract workflow on the event

Actual Results:

Retract workflow operation fails.

org.opencastproject.serviceregistry.api.ServiceRegistryException: Error handling operation 'Retract'
at org.opencastproject.distribution.download.DownloadDistributionServiceImpl.process(DownloadDistributionServiceImpl.java:456)
at org.opencastproject.job.api.AbstractJobProducer$JobRunner.call(AbstractJobProducer.java:281)
at org.opencastproject.job.api.AbstractJobProducer$JobRunner.call(AbstractJobProducer.java:240)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.opencastproject.distribution.api.DistributionException: java.io.IOException: Unable to delete directory /data/opencast/distribution/downloads/mh_default_org/engage-player/df2858a9-51e7-4bce-a79d-dc40b9a8a8a0/cef65a14-f948-497f-8c7d-dbbbf8ac3791.
at org.opencastproject.distribution.download.DownloadDistributionServiceImpl.retractElement(DownloadDistributionServiceImpl.java:416)
at org.opencastproject.distribution.download.DownloadDistributionServiceImpl.retractElements(DownloadDistributionServiceImpl.java:359)
at org.opencastproject.distribution.download.DownloadDistributionServiceImpl.process(DownloadDistributionServiceImpl.java:445)
... 6 more
Caused by: java.io.IOException: Unable to delete directory /data/opencast/distribution/downloads/mh_default_org/engage-player/df2858a9-51e7-4bce-a79d-dc40b9a8a8a0/cef65a14-f948-497f-8c7d-dbbbf8ac3791.
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1337)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1910)
at org.opencastproject.distribution.download.DownloadDistributionServiceImpl.retractElement(DownloadDistributionServiceImpl.java:403)
... 8 more

Expected Results:

Retract operation should succeed.

Workaround (if any):

Run the retract operation again later.

Activity

Show:
Stephen Marquard
September 5, 2017, 3:30 PM

Here's an example of the NFS behaviour:

/data/opencast/work/shared is an NFS mount.

Create a file /data/opencast/work/shared/deltest/delfile and open it in a terminal window (e.g. view it with more).

In another session:

  1. cd /data//opencast/work/shared/

  1. lsof | grep delfile
    more 26988 root 3r REG 0,20 3749319 75155039 /data/opencast/work/shared/deltest/delfile (uctnetapp02nas.uct.ac.za:/vol_opcworkshared_dev)

  1. ls deltest/
    delfile

  1. rm -rf deltest/
    rm: cannot remove `deltest': Directory not empty

(NFS server seems to have created a new file entry for the deleted file that is still held open by another process).

  1. ls -l deltest/
    total 3680
    rw-rr- 1 root root 3749319 Sep 5 16:54 .nfs00000000047ac65f0000007d

(close file in other session)

  1. ls -l deltest/
    total 0

  1. rmdir deltest/

(succeeds)

Fixed and reviewed
Your pinned fields
Click on the next to a field label to start pinning.

Assignee

Unassigned

Reporter

Stephen Marquard

Severity

Incorrectly Functioning With Workaround