chore: switch to charset normalizer (#4060)

qued · ahmetmeleq · ryannikolaidis · web-flow · commit d83df422a67f · 2025-07-22T19:02:40.000Z
Closes [SPI-44](https://linear.app/unstructured/issue/SPI-44/spike-replace-chardet-with-charset-normalizer-if-possible). Removes `chardet` as a dependency, standardizing on `charset-normalizer`. This involved: - Changing `chardet` to `charset-normalizer` in our base dependency file - Updating the code (in only one place) where `chardet` was used - pip-compiling to update our published dependency tree - Updating one test... `charset-normalizer` misdiagnosed the encoding of a file used as a test fixture. My guess is that the ~10 characters in the file were not enough for `charset-normalizer` to do a proper inference, so I re-encoded another slightly longer file that's also used for encoding testing, and it got that one. - Updating an ingest test fixture. - Updating the ingest test fixture update workflow to also update the expected markdown results (this was a task I missed when adding the markdown ingest tests) --------- Co-authored-by: Ahmet Melek <39141206+ahmetmeleq@users.noreply.github.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: qued <qued@users.noreply.github.com> Co-authored-by: Maksymilian Operlejn <36171422+MaksOpp@users.noreply.github.com>
diff --git a/.github/workflows/ingest-test-fixtures-update-pr.yml b/.github/workflows/ingest-test-fixtures-update-pr.yml
@@ -115,6 +115,10 @@ jobs:
         run: |
           source .venv/bin/activate
           make html-fixtures-update
+      - name: Update markdown fixtures
+        run: |
+          source .venv/bin/activate
+          make markdown-fixtures-update
 
       - name: Save branch name to environment file
         id: branch
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,7 @@
 ## 0.18.11-dev1
 
 ### Enhancements
+- **Standardized on `charset-normalizer` library for encoding detection** Previously we had both `chardet` and `charset-normalizer` as dependencies. We are dropping `chardet` and only using `charset-normalizer`.
 
 ### Features
 - **Type-aware `<input>` mapping in HTML transformations** Bare `<input>` elements are now classified by their `type` attribute (checkbox → Checkbox, radio → RadioButton, others → FormFieldValue).
@@ -11,19 +12,11 @@
 
 ## 0.18.10
 
-### Enhancements
-
-### Features
-- **Add OCR_AGENT_CACHE_SIZE environment variable** Added configurable cache size for OCR agents to control memory usage.
-
-### Fixes
-
-## 0.18.10-dev0
-
 ### Enhancements
 - **Updated CodeQL** Updated CodeQL GHA to v3 from deprecated v2.
 
 ### Features
+- **Add OCR_AGENT_CACHE_SIZE environment variable** Added configurable cache size for OCR agents to control memory usage.
 
 ### Fixes
 
diff --git a/example-docs/fake-html-cp1252.html b/example-docs/fake-html-cp1252.html
@@ -4,15 +4,15 @@
 
 <h1>My First Heading</h1>
 <p>My first paragraph.</p>
-<p>Some CP1252-specific characters:</p>
+<p>Some text with CP1252-specific characters:</p>
 
 <pre>
-    �	�	�	�	�	�	�	�	�	�	�	�	SHY	�	�
-�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�
-�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�
-�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�
-�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�
-�	�	�	�	�	�	�	�	�	�	�	�	�	�	�	�
+Die sch�ne Frau hat einen Kaffee mit Kuchen gegessen. Sie sagte: "Das war k�stlich!" und l�chelte dabei. Der Preis betrug 15,50 �.
+L'�t� �tait tr�s chaud cette ann�e. J'ai achet� un caf� au lait pour 3,50 �. C'�tait d�licieux ! L'homme a dit : "C'est parfait !"
+El ni�o comi� paella con �oquis. La se�ora dijo: "�Qu� rico!" y pag� 25,75 �. El restaurante ten�a un men� del d�a.
+Kvinnan �t k�ttbullar med lingonsylt. Hon sa: "Det var fantastiskt!" och betalade 45,90 �. Mannen fr�gade: "Vill du ha mer?"
+O Jo�o comprou um caf� por 2,50 �. Ele disse: "Est� �timo!" e sorriu. A mulher perguntou: "Quer mais alguma coisa?"
+De vrouw dronk koffie met koekjes. Ze zei: "Het was heerlijk!" en betaalde 4,25 �. Het kind vroeg: "Mag ik ook wat?"
 </pre>
 
 </body>
diff --git a/example-docs/umlauts-non-utf8.md b/example-docs/umlauts-non-utf8.md
@@ -1 +1,5 @@
-Umlauts: ����
+## k�nnen
+
+k�nnen
+
+����
diff --git a/requirements/base.in b/requirements/base.in
@@ -1,5 +1,5 @@
 -c ./deps/constraints.txt
-chardet
+charset-normalizer
 filetype
 python-magic
 lxml
diff --git a/requirements/base.txt b/requirements/base.txt
@@ -10,18 +10,17 @@ backoff==2.2.1
     # via -r ./base.in
 beautifulsoup4==4.13.4
     # via -r ./base.in
-certifi==2025.7.9
+certifi==2025.7.14
     # via
     #   httpcore
     #   httpx
     #   requests
     #   unstructured-client
 cffi==1.17.1
     # via cryptography
-chardet==5.2.0
-    # via -r ./base.in
 charset-normalizer==3.4.2
     # via
+    #   -r ./base.in
     #   requests
     #   unstructured-client
 click==8.2.1
@@ -80,7 +79,7 @@ numpy==2.2.6
     # via -r ./base.in
 olefile==0.47
     # via python-oxmsg
-orderly-set==5.4.1
+orderly-set==5.5.0
     # via deepdiff
 packaging==25.0
     # via
@@ -90,7 +89,7 @@ psutil==7.0.0
     # via -r ./base.in
 pycparser==2.22
     # via cffi
-pypdf==5.7.0
+pypdf==5.8.0
     # via unstructured-client
 python-dateutil==2.9.0.post0
     # via unstructured-client
diff --git a/requirements/extra-paddleocr.txt b/requirements/extra-paddleocr.txt
@@ -20,7 +20,7 @@ beautifulsoup4==4.13.4
     # via
     #   -c requirements/base.txt
     #   unstructured-paddleocr
-certifi==2025.7.9
+certifi==2025.7.14
     # via
     #   -c requirements/base.txt
     #   httpcore
diff --git a/requirements/extra-pdf-image.txt b/requirements/extra-pdf-image.txt
@@ -10,7 +10,7 @@ antlr4-python3-runtime==4.9.3
     # via omegaconf
 cachetools==5.5.2
     # via google-auth
-certifi==2025.7.9
+certifi==2025.7.14
     # via
     #   -c requirements/base.txt
     #   requests
@@ -46,7 +46,7 @@ flatbuffers==25.2.10
     # via onnxruntime
 fonttools==4.58.5
     # via matplotlib
-fsspec==2025.5.1
+fsspec==2025.7.0
     # via
     #   huggingface-hub
     #   torch
@@ -64,14 +64,13 @@ googleapis-common-protos==1.70.0
     #   grpcio-status
 grpcio==1.73.1
     # via
-    #   -c requirements/deps/constraints.txt
     #   google-api-core
     #   grpcio-status
 grpcio-status==1.73.1
     # via google-api-core
 hf-xet==1.1.5
     # via huggingface-hub
-huggingface-hub==0.33.2
+huggingface-hub==0.33.4
     # via
     #   accelerate
     #   timm
@@ -121,7 +120,7 @@ onnx==1.18.0
     # via
     #   -r ./extra-pdf-image.in
     #   unstructured-inference
-onnxruntime==1.22.0
+onnxruntime==1.22.1
     # via
     #   -r ./extra-pdf-image.in
     #   unstructured-inference
@@ -148,7 +147,7 @@ pdfminer-six==20250327
     #   unstructured-inference
 pi-heif==1.0.0
     # via -r ./extra-pdf-image.in
-pikepdf==9.9.0
+pikepdf==9.10.0
     # via -r ./extra-pdf-image.in
 pillow==11.3.0
     # via
@@ -190,7 +189,7 @@ pycparser==2.22
     #   cffi
 pyparsing==3.2.3
     # via matplotlib
-pypdf==5.7.0
+pypdf==5.8.0
     # via
     #   -c requirements/base.txt
     #   -r ./extra-pdf-image.in
@@ -243,7 +242,7 @@ sympy==1.14.0
     # via
     #   onnxruntime
     #   torch
-timm==1.0.16
+timm==1.0.17
     # via
     #   effdet
     #   unstructured-inference
@@ -267,7 +266,7 @@ tqdm==4.67.1
     #   -c requirements/base.txt
     #   huggingface-hub
     #   transformers
-transformers==4.53.1
+transformers==4.53.2
     # via unstructured-inference
 typing-extensions==4.14.1
     # via
diff --git a/requirements/huggingface.txt b/requirements/huggingface.txt
@@ -4,7 +4,7 @@
 #
 #    pip-compile ./huggingface.in
 #
-certifi==2025.7.9
+certifi==2025.7.14
     # via
     #   -c requirements/base.txt
     #   requests
@@ -21,13 +21,13 @@ filelock==3.18.0
     #   huggingface-hub
     #   torch
     #   transformers
-fsspec==2025.5.1
+fsspec==2025.7.0
     # via
     #   huggingface-hub
     #   torch
 hf-xet==1.1.5
     # via huggingface-hub
-huggingface-hub==0.33.2
+huggingface-hub==0.33.4
     # via
     #   tokenizers
     #   transformers
@@ -98,7 +98,7 @@ tqdm==4.67.1
     #   huggingface-hub
     #   sacremoses
     #   transformers
-transformers==4.53.1
+transformers==4.53.2
     # via -r ./huggingface.in
 typing-extensions==4.14.1
     # via
diff --git a/requirements/test.txt b/requirements/test.txt
@@ -30,19 +30,17 @@ flake8==7.3.0
     #   flake8-print
 flake8-print==5.0.0
     # via -r ./test.in
-freezegun==1.5.2
+freezegun==1.5.3
     # via -r ./test.in
 grpcio==1.73.1
-    # via
-    #   -c requirements/deps/constraints.txt
-    #   -r ./test.in
+    # via -r ./test.in
 iniconfig==2.1.0
     # via pytest
 liccheck==0.9.2
     # via -r ./test.in
 mccabe==0.7.0
     # via flake8
-mypy==1.16.1
+mypy==1.17.0
     # via -r ./test.in
 mypy-extensions==1.1.0
     # via
@@ -93,7 +91,7 @@ python-dateutil==2.9.0.post0
     # via
     #   -c requirements/base.txt
     #   freezegun
-ruff==0.12.2
+ruff==0.12.3
     # via -r ./test.in
 semantic-version==2.10.0
     # via liccheck
diff --git a/test_unstructured_ingest/expected-structured-output-html/local-single-file-with-encoding/fake-html-cp1252.html.html b/test_unstructured_ingest/expected-structured-output-html/local-single-file-with-encoding/fake-html-cp1252.html.html
@@ -13,16 +13,16 @@ <h1 class="Title" id="a59f117741c76dca0bc8f5ee72e2010b">
   <p class="UncategorizedText" id="82eda2671c5ead903683b67b0f8e3f29">
    My first paragraph.
   </p>
-  <p class="UncategorizedText" id="9f76e487d5df3f6c4ce8ea2ece61057f">
-   Some CP1252-specific characters:
+  <p class="UncategorizedText" id="d536ba7636a9a4603a81b358d1fe2590">
+   Some text with CP1252-specific characters:
   </p>
-  <p class="NarrativeText" id="c96f2c02e05225ffa09b7b93c303c323">
-   ¡	¢	£	¤	¥	¦	§	¨	©	ª	«	¬	SHY	®	¯
-°	±	²	³	´	µ	¶	·	¸	¹	º	»	¼	½	¾	¿
-À	Á	Â	Ã	Ä	Å	Æ	Ç	È	É	Ê	Ë	Ì	Í	Î	Ï
-Ð	Ñ	Ò	Ó	Ô	Õ	Ö	×	Ø	Ù	Ú	Û	Ü	Ý	Þ	ß
-à	á	â	ã	ä	å	æ	ç	è	é	ê	ë	ì	í	î	ï
-ð	ñ	ò	ó	ô	õ	ö	÷	ø	ù	ú	û	ü	ý	þ	ÿ
+  <p class="NarrativeText" id="3b8ca5305e52587b8fbbfcd994de0667">
+   Die schöne Frau hat einen Kaffee mit Kuchen gegessen. Sie sagte: "Das war köstlich!" und lächelte dabei. Der Preis betrug 15,50 €.
+L'été était trčs chaud cette année. J'ai acheté un café au lait pour 3,50 €. C'était délicieux ! L'homme a dit : "C'est parfait !"
+El nińo comió paella con ńoquis. La seńora dijo: "ˇQué rico!" y pagó 25,75 €. El restaurante tenía un menú del día.
+Kvinnan ĺt köttbullar med lingonsylt. Hon sa: "Det var fantastiskt!" och betalade 45,90 €. Mannen frĺgade: "Vill du ha mer?"
+O Joăo comprou um café por 2,50 €. Ele disse: "Está ótimo!" e sorriu. A mulher perguntou: "Quer mais alguma coisa?"
+De vrouw dronk koffie met koekjes. Ze zei: "Het was heerlijk!" en betaalde 4,25 €. Het kind vroeg: "Mag ik ook wat?"
   </p>
  </body>
 </html>
diff --git a/test_unstructured_ingest/expected-structured-output-markdown/local-single-file-with-encoding/fake-html-cp1252.html.md b/test_unstructured_ingest/expected-structured-output-markdown/local-single-file-with-encoding/fake-html-cp1252.html.md
@@ -1,9 +1,9 @@
 # My First Heading
 My first paragraph.
-Some CP1252-specific characters:
-    ¡	¢	£	¤	¥	¦	§	¨	©	ª	«	¬	SHY	®	¯
-°	±	²	³	´	µ	¶	·	¸	¹	º	»	¼	½	¾	¿
-À	Á	Â	Ã	Ä	Å	Æ	Ç	È	É	Ê	Ë	Ì	Í	Î	Ï
-Ð	Ñ	Ò	Ó	Ô	Õ	Ö	×	Ø	Ù	Ú	Û	Ü	Ý	Þ	ß
-à	á	â	ã	ä	å	æ	ç	è	é	ê	ë	ì	í	î	ï
-ð	ñ	ò	ó	ô	õ	ö	÷	ø	ù	ú	û	ü	ý	þ	ÿ
+Some text with CP1252-specific characters:
+Die schöne Frau hat einen Kaffee mit Kuchen gegessen. Sie sagte: "Das war köstlich!" und lächelte dabei. Der Preis betrug 15,50 €.
+L'été était trčs chaud cette année. J'ai acheté un café au lait pour 3,50 €. C'était délicieux ! L'homme a dit : "C'est parfait !"
+El nińo comió paella con ńoquis. La seńora dijo: "ˇQué rico!" y pagó 25,75 €. El restaurante tenía un menú del día.
+Kvinnan ĺt köttbullar med lingonsylt. Hon sa: "Det var fantastiskt!" och betalade 45,90 €. Mannen frĺgade: "Vill du ha mer?"
+O Joăo comprou um café por 2,50 €. Ele disse: "Está ótimo!" e sorriu. A mulher perguntou: "Quer mais alguma coisa?"
+De vrouw dronk koffie met koekjes. Ze zei: "Het was heerlijk!" en betaalde 4,25 €. Het kind vroeg: "Mag ik ook wat?"
diff --git a/test_unstructured_ingest/expected-structured-output/local-single-file-with-encoding/fake-html-cp1252.html.json b/test_unstructured_ingest/expected-structured-output/local-single-file-with-encoding/fake-html-cp1252.html.json
@@ -43,8 +43,8 @@
   },
   {
     "type": "UncategorizedText",
-    "element_id": "9f76e487d5df3f6c4ce8ea2ece61057f",
-    "text": "Some CP1252-specific characters:",
+    "element_id": "d536ba7636a9a4603a81b358d1fe2590",
+    "text": "Some text with CP1252-specific characters:",
     "metadata": {
       "languages": [
         "eng"
@@ -64,11 +64,12 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "c96f2c02e05225ffa09b7b93c303c323",
-    "text": "    ¡\t¢\t£\t¤\t¥\t¦\t§\t¨\t©\tª\t«\t¬\tSHY\t®\t¯\n°\t±\t²\t³\t´\tµ\t¶\t·\t¸\t¹\tº\t»\t¼\t½\t¾\t¿\nÀ\tÁ\tÂ\tÃ\tÄ\tÅ\tÆ\tÇ\tÈ\tÉ\tÊ\tË\tÌ\tÍ\tÎ\tÏ\nÐ\tÑ\tÒ\tÓ\tÔ\tÕ\tÖ\t×\tØ\tÙ\tÚ\tÛ\tÜ\tÝ\tÞ\tß\nà\tá\tâ\tã\tä\tå\tæ\tç\tè\té\tê\të\tì\tí\tî\tï\nð\tñ\tò\tó\tô\tõ\tö\t÷\tø\tù\tú\tû\tü\tý\tþ\tÿ",
+    "element_id": "3b8ca5305e52587b8fbbfcd994de0667",
+    "text": "Die schöne Frau hat einen Kaffee mit Kuchen gegessen. Sie sagte: \"Das war köstlich!\" und lächelte dabei. Der Preis betrug 15,50 €.\nL'été était trčs chaud cette année. J'ai acheté un café au lait pour 3,50 €. C'était délicieux ! L'homme a dit : \"C'est parfait !\"\nEl nińo comió paella con ńoquis. La seńora dijo: \"ˇQué rico!\" y pagó 25,75 €. El restaurante tenía un menú del día.\nKvinnan ĺt köttbullar med lingonsylt. Hon sa: \"Det var fantastiskt!\" och betalade 45,90 €. Mannen frĺgade: \"Vill du ha mer?\"\nO Joăo comprou um café por 2,50 €. Ele disse: \"Está ótimo!\" e sorriu. A mulher perguntou: \"Quer mais alguma coisa?\"\nDe vrouw dronk koffie met koekjes. Ze zei: \"Het was heerlijk!\" en betaalde 4,25 €. Het kind vroeg: \"Mag ik ook wat?\"",
     "metadata": {
       "languages": [
-        "vie"
+        "fra",
+        "nld"
       ],
       "filetype": "text/html",
       "data_source": {
diff --git a/unstructured/file_utils/encoding.py b/unstructured/file_utils/encoding.py
@@ -1,6 +1,6 @@
 from typing import IO, Optional, Tuple, Union
 
-import chardet
+from charset_normalizer import detect
 
 from unstructured.partition.common.common import convert_to_bytes
 
@@ -70,11 +70,11 @@ def detect_file_encoding(
     else:
         raise FileNotFoundError("No filename nor file were specified")
 
-    result = chardet.detect(byte_data)
+    result = detect(byte_data)
     encoding = result["encoding"]
     confidence = result["confidence"]
 
-    if encoding is None or confidence < ENCODE_REC_THRESHOLD:
+    if encoding is None or confidence is None or confidence < ENCODE_REC_THRESHOLD:
         # Encoding detection failed, fallback to predefined encodings
         for enc in COMMON_ENCODINGS:
             try:
