bug: DataError: value too long for type character varying(100) when initiating a scan

[fschledorn]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I've encountered a fatal database error when initiating a scan on certain domains. The scan fails immediately with a django.db.utils.DataError, indicating that a value discovered during the initial phase of the scan is longer than the 100-character limit of a database column.

For me, this suggests that a field in one of the database models (likely related to the Scan model) has a character varying(100) limit that is too restrictive for data found in the wild, such as long subdomains or other metadata.

Expected Behavior

The scan should start successfully and handle long domain names or other discovered data without causing a database error. The relevant database columns should probably be of type TEXT or VARCHAR with a higher limit (e.g., VARCHAR(255) or more) to prevent this truncation error.

Steps To Reproduce

1. Go to the "Add Target" page and add a long domain (at least 32 characters).
2. Start a scan on this target with a standard scan engine configuration.
3. The scan fails to start, and the Celery logs immediately show the DataError traceback.

Environment

- reNgine: 2.2.0
- OS: Debian 12
- Python: Python 3.11.2
- Docker Engine: 28.3.3, build 980b856
- Docker Compose: v2.39.1
- Browser: Firefox

Anything else?

Logs (URL’s are censored for privacy reasons):

celery-1           | initiate_scan                           | WARNING | Initiating scan for domain <URL> on celery
celery-1           | initiate_scan                           | ERROR | value too long for type character varying(100)
celery-1           | Traceback (most recent call last):
celery-1           |   File "/home/rengine/.cache/pypoetry/virtualenvs/celery-rengine-HmEJnPQT-py3.10/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
celery-1           |     return self.cursor.execute(sql, params)
celery-1           | psycopg2.errors.StringDataRightTruncation: value too long for type character varying(100)
celery-1           |
celery-1           |
celery-1           | The above exception was the direct cause of the following exception:
celery-1           |
celery-1           | Traceback (most recent call last):
celery-1           |   File "/home/rengine/rengine/reNgine/tasks/scan.py", line 126, in initiate_scan
celery-1           |     scan.save()
celery-1           |   File "/home/rengine/.cache/pypoetry/virtualenvs/celery-rengine-HmEJnPQT-py3.10/lib/python3.10/site-packages/django/db/models/base.py", line 739, in save
celery-1           |     self.save_base(using=using, force_insert=force_insert,
celery-1           |   File "/home/rengine/.cache/pypoetry/virtualenvs/celery-rengine-HmEJnPQT-py3.10/lib/python3.10/site-packages/django/db/models/base.py", line 776, in save_base
celery-1           |     updated = self._save_table(
celery-1           |   File "/home/rengine/.cache/pypoetry/virtualenvs/celery-rengine-HmEJnPQT-py3.10/lib/python3.10/site-packages/django/db/models/base.py", line 858, in _save_table
celery-1           |     updated = self._do_update(base_qs, using, pk_val, values, update_fields,
celery-1           |   File "/home/rengine/.cache/pypoetry/virtualenvs/celery-rengine-HmEJnPQT-py3.10/lib/python3.10/site-packages/django/db/models/base.py", line 912, in _do_update
celery-1           |     return filtered._update(values) > 0
celery-1           |   File "/home/rengine/.cache/pypoetry/virtualenvs/celery-rengine-HmEJnPQT-py3.10/lib/python3.10/site-packages/django/db/models/query.py", line 802, in _update
celery-1           |     return query.get_compiler(self.db).execute_sql(CURSOR)
celery-1           |   File "/home/rengine/.cache/pypoetry/virtualenvs/celery-rengine-HmEJnPQT-py3.10/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
celery-1           |     cursor = super().execute_sql(result_type)
celery-1           |   File "/home/rengine/.cache/pypoetry/virtualenvs/celery-rengine-HmEJnPQT-py3.10/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
celery-1           |     cursor.execute(sql, params)
celery-1           |   File "/home/rengine/.cache/pypoetry/virtualenvs/celery-rengine-HmEJnPQT-py3.10/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
celery-1           |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
celery-1           |   File "/home/rengine/.cache/pypoetry/virtualenvs/celery-rengine-HmEJnPQT-py3.10/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
celery-1           |     return executor(sql, params, many, context)
celery-1           |   File "/home/rengine/.cache/pypoetry/virtualenvs/celery-rengine-HmEJnPQT-py3.10/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
celery-1           |     with self.db.wrap_database_errors:
celery-1           |   File "/home/rengine/.cache/pypoetry/virtualenvs/celery-rengine-HmEJnPQT-py3.10/lib/python3.10/site-packages/django/db/utils.py", line 90, in exit
celery-1           |     raise dj_exc_value.with_traceback(traceback) from exc_value
celery-1           |   File "/home/rengine/.cache/pypoetry/virtualenvs/celery-rengine-HmEJnPQT-py3.10/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
celery-1           |     return self.cursor.execute(sql, params)
celery-1           | django.db.utils.DataError: value too long for type character varying(100)

[github-actions]

Hey @fschledorn! 👋 Thanks for flagging this bug! 🐛🔍

You're our superhero bug hunter! 🦸‍♂️🦸‍♀️ Before we suit up to squash this bug, could you please:

📚 Double-check our documentation: https://rengine.wiki
🕵️ Make sure it's not a known issue
📝 Provide all the juicy details about this sneaky bug

Once again - thanks for your vigilance! 🛠️🚀

[fschledorn]

I've implemented a fix for this issue that addresses the root cause rather than just increasing the field limit. see #1520