Skip to content

Split Flickr data refresh into a separate DAG #5424

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Split Flickr data refresh into a separate DAG #5424

wants to merge 3 commits into from

Conversation

@dilipom13
Copy link

@dilipom13 dilipom13 commented May 25, 2025

This PR addresses Issue #3817 by separating the refresh logic for the Flickr provider into a standalone Airflow DAG.

✨ Changes Made:

  • Created a new DAG under catalog/dags/data_refresh/flickr_refresh.py for handling refresh logic only
  • Moved the refresh task out of catalog/dags/providers/flickr.py
  • Created a reusable helper function refresh_provider_data() in catalog/dags/common/refresh.py

✅ Benefits:

  • Improves modularity and separation of concerns
  • Allows independent scheduling and retrying of refresh jobs
  • Reduces the risk of refresh failures impacting ingestion jobs

This is part of a broader effort to split ingestion and refresh pipelines for all providers.

@dilipom13 dilipom13 requested review from a team as code owners May 25, 2025 14:21
@dilipom13 dilipom13 requested review from krysal and obulat and removed request for a team May 25, 2025 14:21
@openverse-bot openverse-bot added 🧱 stack: catalog Related to the catalog and Airflow DAGs 🚦 status: awaiting triage Has not been triaged & therefore, not ready for work labels May 25, 2025
@openverse-bot openverse-bot moved this to 👀 Needs Review in Openverse PRs May 25, 2025
@krysal krysal added 🟨 priority: medium Not blocking but should be addressed soon ✨ goal: improvement Improvement to an existing user-facing feature 💻 aspect: code Concerns the software code in the repository and removed 🚦 status: awaiting triage Has not been triaged & therefore, not ready for work labels May 26, 2025
@dilipom13
Copy link
Author

dilipom13 commented Jun 1, 2025

I'd like to work on this issue. Please assign it to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@krysal krysal Awaiting requested review from krysal krysal is a code owner automatically assigned from WordPress/openverse-maintainers

@obulat obulat Awaiting requested review from obulat obulat is a code owner automatically assigned from WordPress/openverse-maintainers

Assignees

No one assigned

Labels

💻 aspect: code Concerns the software code in the repository ✨ goal: improvement Improvement to an existing user-facing feature 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: catalog Related to the catalog and Airflow DAGs

Projects

Openverse PRs
Status: 👀 Needs Review

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dilipom13 @krysal @openverse-bot