4.1.7

What's Changed

• Update to Python 3.10 by @stchris in #4328

Full Changelog: 4.1.6...4.1.7

4.1.6

This release fixes some issues related to Aleph's maintenance mode.

What's Changed

• Bugfix: maintenance mode permission checks for collection and export creation by @stchris in #4279
• Bugfix: prevent constant refetching of profile information in maintenance mode by @stchris in #4280

Full Changelog: 4.1.5...4.1.6

4.1.5

What's Changed

• Updated the FAQ page in 6cfde3e

Full Changelog: 4.1.4...4.1.5

4.1.4

What's Changed

• Use servicelayer 1.25.0 which allows Workload Identity authentication when signing links for Google Cloud Storage

Full Changelog: 4.1.3...4.1.4

4.1.3

This Aleph release updates the default version of ingest-file in the Docker Compose configuration and Helm chart to ingest-file 4.1.2.

Important

Prior versions of ingest-file contain a security vulnerability due to insecure handling of 7zip archives. We recommend that you update Aleph instances you operate to use the latest patched release of ingest-file.

Please refer to the release notes of ingest-file 4.1.2 for details.

Full Changelog: 4.1.2...4.1.3

4.1.1

What's Changed

• Fix incorrect local queue size calculation (by upgrading to servicelayer 1.24.0) by @stchris in #4160

Full Changelog: 4.1.0...4.1.1

4.1.0

Upgrade instructions

Important

If you use the Aleph API, make sure to carefully review the following upgrade instructions.

Aleph 4.1 introduces changes to API keys:

• Users can now regenerate their API key from the user settings page.
• New API keys now expire automatically after 90 days. In order to continue accessing the Aleph API, users need to regenerate their API key before it expires. Users will receive email notifications 7 days before their API key expires, as well as on the day it expires.
• Users can only see their API key directly after it has been generated, but not again after that.
• New users won’t have an API key by default anymore, they need to actively generate an API key.

After upgrading to Aleph 4.1, to continue using existing API keys, you need to run the following command in an Aleph container:

aleph hash-plaintext-api-keys

Existing API keys won’t expire automatically, but you can optionally set them to expire after 90 days by running the following command in an Aleph container:

aleph reset-api-key-expiration

What's Changed

• Implement infinite scrolling for document search by @tillprochaska in #3855

• Fix infinite request loop if metadata request fails by @tillprochaska in #3948

• Do not allow duplicate role memberships by @tillprochaska in #3815

• Make document search available in entity preview by @tillprochaska in #3879

• Improved API key handling by @tillprochaska in #3094

• Update Prometheus exporter probes so they do not actually generate all metrics by @tillprochaska in #3996

• Show login popup or redirect to OAuth provider when session token has expired by @tillprochaska in #4000

• Fix edge case in metrics collector by @tillprochaska in #3875

• Update documentation links in FAQ by @tillprochaska in #3873

• Add test to cover last_login_at timestamp in OAuth workflow by @tillprochaska in #3878

• e2e test fix: Use newer locator API, make worker start up properly by @stchris in #3877

• Use pg_isready in make upgrade instead of sleep by @stchris in #3951

• Bump flask-cors from 4.0.0 to 4.0.2 by @dependabot in #3885

• Add additional validation checks by @tillprochaska in #4104

• Do not delete API key to enable rollbacks to previous version. by @tillprochaska in #4105

Full Changelog: 4.0.2...4.1.0

4.0.2

We're announcing the release of Aleph 4.0.2 (and ingest-file 4.0.2) and highly recommend users of the 4.x branches to update to this release.

What's changed

• Update to servicelayer 1.23.2 which fixes a significant performance regression noticeable especially when there are > 10000 tasks per dataset being processed

Full Changelog: 4.0.1...4.0.2

4.0.1

We're announcing the release of Aleph 4.0.1 (and ingest-file 4.0.1) and highly recommend users of the 4.x branches to update to this release.

What's changed

Bugfix

• Update to using servicelayer 1.23.1 which fixes an issue with improper clean-up when a task exhausts it's maximum number of retries

Full Changelog: 4.0.0...4.0.1

4.0.0

Hello Aleph community! We’re excited to announce Aleph 4.0.0, a release focused on powerful new features, performance improvements, and expanded options for investigation sharing and user metrics. In addition, this release includes a few other small enhancements, bug fixes and dependency upgrades.

🚀 Bigger Changes 🚀

• RabbitMQ based task queueing backend
  • Configurable AlephWorker Stages
  • Priority Buckets for Processing
  • System Status Page Enhancements
• Updated Prometheus Metrics
• Documentation Restructure and Enhancements
• Improved Error Handling in Elasticsearch Upgrades

As always, we’d love to hear your feedback to keep improving. Feel free to reach out and share your thoughts!

---

What's Changed

Features

RabbitMQ

4.0.0 introduces a change to the way background tasks are scheduled. Previously Aleph used a Redis-based task queue, which was well designed but showed its limitations with large payloads and a risk of data loss. RabbitMQ queues are persisted to disk, but the flexibility in the way messages are queued, routed and fetched allows for certain optimizations which Aleph benefits from because of the widely varying degree of task loads.

Migration notes from Redis to RabbitMQ

Due to the significant changes in terms of task status persistence, switching between Aleph versions with RabbitMQ and Redis-based task queues requires some manual steps in order to ensure data consistency.

Perform the following steps every time you are either upgrading to a version with the RabbitMQ task queue or rolling back to the Redis-based task queue:

1. Let all pending jobs run to completion (check the status page).
2. Put Aleph into maintenance mode.
3. Stop all workers (worker, ingest-file processes).
4. (optional) Save the current state of redis in case you want to roll back using the BGSAVE command.
5. Clear Redis (by issuing FLUSHDB from redis-cli from the redis container). If you get the error message "Unknown command FLUSHDB" then this command is disabled and you can resort to this shell invocation: echo 'KEYS *' | redis-cli | grep -v '^aleph:' | sed 's/^/DEL /' | redis-cli.
6. (optional, if previous versions had conflicting RabbitMQ queue settings) Delete existing queues using rabbitmqctl delete queue {ingest,pruneentity,updateentity,exportxref,analyze,flushmapping,reingest,exportsearch,index,xref,reindex,loadmapping}. NOTE: queues are named after the stages found in ALEPH_WORKER_STAGES.
7. Perform the upgrade or rollback to the desired version of Aleph.
8. Ensure that all expected processes have started correctly.

Related changes:

• Dynamically set AlephWorker stages through env vars by @catileptic in #3748
• Completely remove network diagram embeds feature by @tillprochaska in #3751
• Feature: Priority buckets by @stchris in #3784
• Separate index worker from other stages in aleph-worker by @stchris in #3817

Prometheus metrics

We have extended the Prometheus metrics exposed by Aleph to provide more information about active users and the data in your Aleph instance. For example, you can now query for the number of active users within the past 30 days or the number of investigations related to a particular language. For details about the available metrics please refer to the metrics reference in the technical documentation.

• New and updated Prometheus metrics by @tillprochaska in #3844
• Update Prometheus metrics reference by @tillprochaska in #3845
• Fix active users metric by @tillprochaska in #3852
• Fix edge cases in custom metrics by @tillprochaska in #3861

Sharing investigations

Due to the sensitive nature of dataset access we have made some changes to the way datasets are shared, no longer allowing email addresses to autocomplete. This means one needs to know the exact email address of another user if they want to share an investigation.

• Feature: Allow sharing of investigations by @tillprochaska in #3865
• Remove sharing options from create investigation screen by @stchris in #3862
• Multiple small UX enhancements related to investigation sharing/user suggestion component by @tillprochaska in #3868

Other new features

• Display start and last updated timestamp on system status page by @tillprochaska in #3788
• Display an error message for blocked users by @tillprochaska in #3560
• aleph CLI command to downgrade the postgres DB by @stchris in #3858

Bug fixes and other changes

• Use default language when Accept-Language header is '*' by @stchris in #3724
• Exit op_index early in Aleph Worker by @catileptic in #3781
• Automatically post releases to Discourse by @stchris in #3728
• Fix phone numbers used in tests by @tillprochaska in #3847
• Remove bookmarks migration by @tillprochaska in #3752
• Fix docker compose command by @tillprochaska in #3843
• Improve date formats on status page by @tillprochaska in #3841
• Aleph upgrade will throw an exception if any ES call returns a status code < 399 by @catilepticin #3859
• Fix ES index upgrades when using index aliases by @tillprochaska in #3863

Documentation updates

• Docs: Restructure tech docs by @tillprochaskain #3569
• Misc documentation enhancements by @tillprochaska in #3819
• Docs: added a small amount of text for people looking to get started but are not aware of gitflow by @Rosencrantz in #3707
• Docs: link to download raw docker compose files instead of HTML by @vsessink in #3778
• Docs: Improved order of commands for first time setup by @vsessink in #3779
• Docs: Document how to set up MinIO in a development environment by @tillprochaska in #3857
• Docs: Document how to download files using alephclient by @tillprochaska in #3848
• Update form links in docs by @tillprochaska in #3850
• Update form links on about page etc. by @tillprochaska in #3851

Dependency updates

• Bugfix/downgrade authlib in 3.16.0 by @stchris in #3574
• Bump gunicorn[eventlet] from 21.2.0 to 22.0.0 by @dependabot in #3689
• Bump flask and authlib (as required to run flask 3+) by @stchris in #3732
• Bump react-pdf from 5.7.2 to 7.7.3 in /ui by @dependabot in #3726

Full Changelog: 3.17.0...4.0.0