As a user, I expect metadata changes to be propagated to third-party applications

Description

When using OAI-PMH to interface with third-party applications, there are at least three copies of metadata:

  • Opencast (as displayed in the Admin UI)

  • OAI-PMH server (database table)

  • Third-party application (own representation)

Currently, when a user changes metadata in the Opencast Admin UI, most of those changes (some indeed are propagated) are not propagated to Events that have been published to OAI-PMH.

Problems:

  • It is not possible to know what metadata has been published using the Admin UI - that would require looking into the database or the third-party application

  • Metadata changes get complex and cumbersome. For example, fixing a typo requires users to re-publish content

To goal of this story is to make the following changes being propagated automatically to the OAI-PMH server:

  • Changes to dublincore/episode and dublincore/series

  • Changes to ACLs of series and events

Optional:

  • Make automatic propagation optional in case other Adopters prefer the old way

Activity

Show:
Former user
June 16, 2017, 12:38 PM

, our Metadata Synch service is an external API endpoint for 3rd party systems (and a custom admin UI) to update OC episode metadata without impacting existing OC workflows. The service doesn't listen to updates, it takes in external update requests (from a UI or a trusted 3rd party system) and iterates over its "strategy" classes that know how to update the different Opencast data areas (scheduler, archive, workflow, search, download, in OC v1.6x). Update requests are queued, if they conflict with running workflows.

The Metasynch service for OCv1.6x cannot be directly applied to OC2x because I think each update would cause a storm of ActiveMQ messages from each of the updated service areas.

I like the messaging system because it moves the responsibility of synchronizing metadata directly into each OC service area. The Metasynch service is useful because it orchestrates data synchronization, tracks progress, success, and notifications. If a consolidated metadata update tracking and notification service is still necessary in 3x, Metasynch could start listening to service update progress messages via ActiveMQ and correct, log, notify on data disparities.

Sven Stauber
June 16, 2017, 7:26 AM

Hi Karen,

The guys of Plapadoo have started to work on this yesterday. I don't think that this change will conflict with your metadata sync service.

Unfortunately, I'm not technically profound in this area, but as far as I understand, anybody interested in metadata changes can register to the respective ActiveMQ message queue to get informed about changes.

Is it possible that your metadata sync service somehow duplicates the Opencast 2.x message bus provided by ActiveMQ?

Not sure what solution Plapadoo comes up with, but likely it will be matterhorn-conductor that needs to be adapted.

Former user
June 15, 2017, 7:25 PM

are you working on a solution? Our site has a custom "Metadata Synchronize" (Metasynch) service module that I keep returning to in order to generalized it for the OC community. The OC code keeps updating before I can submit it . Its getting higher on my immediate todo list to check if it is still needed within 3x.

My impression of 3x is that any service that is interested in metadata updates subscribes to messaged update events that cover the topics of interest.

The Metasynch service (used in our v1.63) has an "EngageUpdateStrategy" class that publishes updated catalogs and metadata to the search service (or queues the updates until a running workflow has paused). Perhaps the OAI-PMH needs a metadata change strategy utility that can safely and strategically push relevant data changes to OAI-PMH.

Fixed and reviewed

Assignee

Unassigned

Reporter

Sven Stauber