Merge pull request #49 from pescheckit/feature_added-ai-tagging

Fixed ai tagging

Author: pescheck-bram

.github/workflows/ci-cd.yml

@@ -89,6 +89,11 @@ jobs:
           python-version: ${{ matrix.python-version }}
           cache: 'pip'
 
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y gettext
+
       - name: Install test dependencies
         run: |
           python -m pip install --upgrade pip

.gitignore

@@ -8,3 +8,5 @@ db.sqlite3
 .venv
 python_gpt_po/_version.py
 CLAUDE.md
+venv/
+test_venv/

README.md

@@ -4,7 +4,7 @@
 ![PyPI](https://img.shields.io/pypi/v/gpt-po-translator?label=gpt-po-translator)
 ![Downloads](https://pepy.tech/badge/gpt-po-translator)
 
-A robust tool for translating gettext (.po) files using AI models from multiple providers (OpenAI, Azure OpenAI, Anthropic / Claude, and DeepSeek). It supports both bulk and individual translations, handles fuzzy entries, and can infer target languages based on folder structures. Available as a Python package and Docker container with support for Python 3.8-3.12.
+A robust tool for translating gettext (.po) files using AI models from multiple providers (OpenAI, Azure OpenAI, Anthropic/Claude, and DeepSeek). It supports both bulk and individual translations, handles fuzzy entries, and can infer target languages based on folder structures. Available as a Python package and Docker container with support for Python 3.8-3.12.
 
 ## What is GPT-PO Translator?
 
@@ -15,6 +15,7 @@ This tool helps you translate gettext (.po) files using AI models. It's perfect
 - **Multiple AI providers** - OpenAI, Azure OpenAI, Anthropic/Claude, and DeepSeek
 - **Flexible translation modes** - Bulk or entry-by-entry processing
 - **Smart language handling** - Auto-detects target languages from folder structure
+- **AI translation tracking** - Automatically tags AI-generated translations with comments
 - **Production-ready** - Includes retry logic, validation, and detailed logging
 - **Easy deployment** - Available as a Python package or Docker container
 - **Cross-version support** - Works with Python 3.8-3.12
@@ -68,6 +69,14 @@ docker run -v $(pwd):/data \
   -e OPENAI_API_KEY="your_key" \
   ghcr.io/pescheckit/python-gpt-po:latest \
   --folder /data --lang fr,de --bulk
+
+# Or with Azure OpenAI
+docker run -v $(pwd):/data \
+  -e AZURE_OPENAI_API_KEY="your_key" \
+  -e AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/" \
+  -e AZURE_OPENAI_API_VERSION="2024-02-01" \
+  ghcr.io/pescheckit/python-gpt-po:latest \
+  --provider azure_openai --folder /data --lang fr,de --bulk
 ```
 
 ## Setting Up API Keys
@@ -79,7 +88,11 @@ export OPENAI_API_KEY='your_api_key_here'
 # Or for other providers:
 export ANTHROPIC_API_KEY='your_api_key_here'
 export DEEPSEEK_API_KEY='your_api_key_here'
+
+# For Azure OpenAI:
 export AZURE_OPENAI_API_KEY='your_api_key_here'
+export AZURE_OPENAI_ENDPOINT='https://your-resource.openai.azure.com/'
+export AZURE_OPENAI_API_VERSION='2024-02-01'
 ```
 
 ### Option 2: Command Line
@@ -113,31 +126,61 @@ gpt-po-translator --provider anthropic --folder ./locales --lang de
 # Use DeepSeek models
 gpt-po-translator --provider deepseek --folder ./locales --lang de
 
-# List available models for openai
-gpt-po-translator --provider openai --list-models
+# Use Azure OpenAI
+gpt-po-translator --provider azure_openai \
+  --azure-openai-endpoint https://your-resource.openai.azure.com/ \
+  --azure-openai-api-version 2024-02-01 \
+  --folder ./locales --lang de
 
-# List available models for azure openai
+# List available models for different providers
+gpt-po-translator --provider openai --list-models
 gpt-po-translator --provider azure_openai \
-  --azure-openai-endpoint https://<deployment>.cognitiveservices.azure.com/ \
-  --azure-openai-api-version <api_version> \
+  --azure-openai-endpoint https://your-resource.openai.azure.com/ \
+  --azure-openai-api-version 2024-02-01 \
   --list-models
 ```
 
+### AI Translation Tracking
+
+**By default, all AI-generated translations are automatically marked with comments** for easy tracking and compliance:
+
+```bash
+# Default behavior - AI translations are tagged with comments
+gpt-po-translator --folder ./locales --lang de
+
+# Result in PO file:
+#. AI-generated
+msgid "Hello"
+msgstr "Hallo"
+
+# To disable AI tagging (not recommended)
+gpt-po-translator --folder ./locales --lang de --no-ai-comment
+```
+
+This helps you:
+- Identify which translations were made by AI vs human translators  
+- Track incremental changes when new AI translations are added
+- Comply with requirements to identify AI-generated content
+
+**Note:** Django's `makemessages` removes these comments when updating PO files, but translations are preserved. Re-run the translator after `makemessages` to restore AI tagging.
+
 ## Command Reference
 
 | Option | Description |
 |--------|-------------|
 | `--folder` | Path to your .po files |
 | `--lang` | Target language codes (comma-separated, e.g., `de,fr`) |
 | `--detail-lang` | Full language names (e.g., `"German,French"`) |
-| `--fuzzy` | Remove fuzzy entries before translating |
+| `--fuzzy` | Remove fuzzy entries before translating (DEPRECATED - use `--fix-fuzzy`) |
+| `--fix-fuzzy` | Translate and fix fuzzy entries properly (recommended) |
 | `--bulk` | Enable batch translation (recommended for large files) |
 | `--bulksize` | Entries per batch (default: 50) |
 | `--model` | Specific AI model to use |
-| `--provider` | AI provider: `openai`, `anthropic`, or `deepseek` |
+| `--provider` | AI provider: `openai`, `azure_openai`, `anthropic`, or `deepseek` |
 | `--list-models` | Show available models for selected provider |
 | `--api_key` | Your API key |
 | `--folder-language` | Auto-detect languages from folder structure |
+| `--no-ai-comment` | Disable AI-generated comment tagging (enabled by default) |
 
 ## Advanced Docker Usage
 
@@ -159,17 +202,19 @@ docker pull ghcr.io/pescheckit/python-gpt-po:0.3.0
 Mount any local directory to use in the container:
 
 ```bash
-# Windows example
+# Windows example with OpenAI
 docker run -v D:/projects/website/locales:/locales \
   -e OPENAI_API_KEY="your_key" \
   ghcr.io/pescheckit/python-gpt-po:latest \
   --folder /locales --lang fr,de --bulk
 
-# Mac/Linux example
+# Mac/Linux example with Azure OpenAI
 docker run -v /Users/username/translations:/input \
-  -e OPENAI_API_KEY="your_key" \
+  -e AZURE_OPENAI_API_KEY="your_key" \
+  -e AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/" \
+  -e AZURE_OPENAI_API_VERSION="2024-02-01" \
   ghcr.io/pescheckit/python-gpt-po:latest \
-  --folder /input --lang fr,de --bulk
+  --provider azure_openai --folder /input --lang fr,de --bulk
 ```
 
 ## Requirements
@@ -194,9 +239,10 @@ docker run --rm -v $(pwd):/app -w /app --entrypoint python python-gpt-po -m pyte
 
 ## Documentation
 
-For advanced usage and detailed documentation, please see:
-- [Advanced Usage Guide](docs/usage.md)
-- [GitHub Repository](https://github.com/pescheckit/python-gpt-po)
+For more detailed information:
+- **[Advanced Usage Guide](docs/usage.md)** - Comprehensive guide with all options and internal mechanics
+- **[Development Guide](docs/development.md)** - For contributors
+- **[GitHub Repository](https://github.com/pescheckit/python-gpt-po)** - Source code and issues
 
 ## License
 

docs/usage.md

@@ -6,7 +6,7 @@ This guide provides an in-depth look at what really happens when you run `gpt-po
 
 ## Overview
 
-`gpt-po-translator` is a multi-provider tool for translating gettext (.po) files using AI models. It supports OpenAI, Anthropic, and DeepSeek. The tool offers two primary translation modes:
+`gpt-po-translator` is a multi-provider tool for translating gettext (.po) files using AI models. It supports OpenAI, Azure OpenAI, Anthropic, and DeepSeek. The tool offers two primary translation modes:
 - **Bulk Mode:** Processes a list of texts in batches to reduce the number of API calls.
 - **Individual Mode:** Translates each text entry one-by-one for more fine-grained control.
 
@@ -22,9 +22,9 @@ It also manages fuzzy translations (by disabling or removing them) and can infer
 
 - **API Key Setup:**  
   The tool collects API keys from multiple sources:
-  - Specific arguments (`--openai-key`, `--anthropic-key`, `--deepseek-key`)
+  - Specific arguments (`--openai-key`, `--azure-openai-key`, `--anthropic-key`, `--deepseek-key`)
   - A fallback argument (`--api_key`) for OpenAI if no dedicated key is provided
-  - Environment variables (e.g., `OPENAI_API_KEY`)
+  - Environment variables (e.g., `OPENAI_API_KEY`, `AZURE_OPENAI_API_KEY`)
 
   It then initializes a `ProviderClients` instance that creates API client objects for the chosen providers.
 
@@ -76,7 +76,7 @@ It also manages fuzzy translations (by disabling or removing them) and can infer
   The tool checks whether the translation is excessively verbose compared to the original text. If so, it retries the translation to ensure it remains concise.
 
 - **Updating PO Files:**  
-  After translation, each PO file entry is updated with the new translation using `polib`. The tool logs a summary of how many entries were successfully translated and warns if any remain untranslated.
+  After translation, each PO file entry is updated with the new translation using `polib`. By default, AI-generated translations are marked with a comment (`#. AI-generated`) for easy identification. The tool logs a summary of how many entries were successfully translated and warns if any remain untranslated.
 
 ---
 
@@ -114,9 +114,14 @@ Below is a detailed explanation of all command-line arguments:
   *Behind the scenes:* These names are used in the translation prompts to give the AI clearer context, potentially improving translation quality.  
   *Note:* The number of detailed names must match the number of language codes.
 
-- **`--fuzzy`**  
-  *Description:* A flag that, when set, instructs the tool to remove fuzzy entries from the PO files before translation.  
-  *Behind the scenes:* The tool calls a dedicated method to strip fuzzy markers and flags from both the file content and metadata.
+- **`--fuzzy`** *(DEPRECATED)*  
+  *Description:* A flag that, when set, instructs the tool to remove fuzzy entries from the PO files before translation. **This option is DEPRECATED due to its risky behavior of removing fuzzy markers without actually translating the content.**  
+  *Behind the scenes:* The tool calls a dedicated method to strip fuzzy markers and flags from both the file content and metadata.  
+  *Warning:* This can lead to data loss and confusion. Use `--fix-fuzzy` instead.
+
+- **`--fix-fuzzy`**  
+  *Description:* Translate and clean fuzzy entries safely (recommended over `--fuzzy`).  
+  *Behind the scenes:* The tool filters for entries with the 'fuzzy' flag and attempts to translate them, removing the flag upon successful translation. AI-generated translations are marked as usual unless `--no-ai-comment` is used.
 
 - **`--bulk`**  
   *Description:* Enables bulk translation mode, meaning multiple texts will be translated in a single API call.  
@@ -131,7 +136,7 @@ Below is a detailed explanation of all command-line arguments:
   *Behind the scenes:* This key is merged with keys provided through other command-line arguments or environment variables.
 
 - **`--provider <provider>`**  
-  *Description:* Specifies the AI provider to use for translations. Acceptable values are `openai`, `anthropic`, or `deepseek`.  
+  *Description:* Specifies the AI provider to use for translations. Acceptable values are `openai`, `azure_openai`, `anthropic`, or `deepseek`.  
   *Behind the scenes:* If not specified, the tool auto-selects the first provider for which an API key is available.
 
 - **`--model <model>`**  
@@ -150,6 +155,18 @@ Below is a detailed explanation of all command-line arguments:
   *Description:* Provides the Anthropic API key directly.  
   *Behind the scenes:* This key is used to initialize the Anthropic client.
 
+- **`--azure-openai-key`**  
+  *Description:* Provides the Azure OpenAI API key directly.  
+  *Behind the scenes:* This key is used to initialize the Azure OpenAI client.
+
+- **`--azure-openai-endpoint`**  
+  *Description:* Provides the Azure OpenAI endpoint URL (e.g., `https://your-resource.openai.azure.com/`).  
+  *Behind the scenes:* Required for Azure OpenAI connections along with the API version.
+
+- **`--azure-openai-api-version`**  
+  *Description:* Specifies the Azure OpenAI API version (e.g., `2024-02-01`).  
+  *Behind the scenes:* Different API versions support different features and models.
+
 - **`--deepseek-key`**  
   *Description:* Provides the DeepSeek API key directly.  
   *Behind the scenes:* This key is required to make API calls to DeepSeek’s translation service.
@@ -158,13 +175,89 @@ Below is a detailed explanation of all command-line arguments:
   *Description:* Enables inferring the target language from the folder structure.  
   *Behind the scenes:* The tool inspects the path components (directory names) of each PO file and matches them against the provided language codes.
 
+- **`--no-ai-comment`**  
+  *Description:* Disables the automatic addition of 'AI-generated' comments to translated entries.  
+  *Behind the scenes:* **By default (without this flag), every translation made by the AI is marked with a `#. AI-generated` comment in the PO file.** This flag prevents that marking, making AI translations indistinguishable from human translations in the file.  
+  *Note:* AI tagging is enabled by default for tracking, compliance, and quality assurance purposes.
+
+---
+
+## AI Translation Tracking
+
+### Overview
+
+**AI translation tracking is enabled by default.** The tool automatically tracks which translations were generated by AI versus human translators. This is particularly useful for:
+- Quality assurance and review processes
+- Compliance with requirements to identify AI-generated content
+- Incremental translation workflows where you need to track changes
+
+### How It Works
+
+When a translation is generated by the AI, the tool adds a translator comment to the PO entry:
+
+```po
+#. AI-generated
+msgid "Hello, world!"
+msgstr "Hola, mundo!"
+```
+
+These comments are:
+- **Persistent**: They're saved in the PO file and preserved across edits
+- **Standard-compliant**: Using the official gettext translator comment syntax (`#.`)
+- **Tool-friendly**: Visible in PO editors like Poedit, Lokalize, etc.
+- **Searchable**: Easy to find with grep or other search tools
+
+### Managing AI Comments
+
+**Finding AI translations:**
+```bash
+# Count AI-generated translations
+grep -c "^#\. AI-generated" locales/es/LC_MESSAGES/messages.po
+
+# List files with AI translations
+grep -l "^#\. AI-generated" locales/**/*.po
+```
+
+**Important: Django Workflow Consideration**
+Django's `makemessages` command removes translator comments (including AI-generated tags) when updating PO files. This means:
+
+- **After running our translator**: AI comments are preserved in PO files
+- **After running Django makemessages**: AI comments are removed, but translations remain
+- **Best practice**: Re-run the AI translator after Django makemessages to restore AI tagging on any remaining untranslated entries
+
+**Disabling AI comments:**
+If you don't want AI translations to be marked, use the `--no-ai-comment` flag:
+```bash
+gpt-po-translator --folder ./locales --lang de --no-ai-comment
+```
+
+### Programmatic Access
+
+The tool provides helper methods for working with AI-generated translations programmatically:
+
+```python
+from python_gpt_po.services.po_file_handler import POFileHandler
+import polib
+
+# Load a PO file
+po_file = polib.pofile('messages.po')
+
+# Get all AI-generated entries
+ai_entries = POFileHandler.get_ai_generated_entries(po_file)
+
+# Remove AI-generated comments if needed
+POFileHandler.remove_ai_generated_comments(po_file)
+po_file.save()
+```
+
 ---
 
 ## Behind the Scenes: API Calls and Post-Processing
 
 - **Provider-Specific API Calls:**  
   The tool constructs different API requests based on the selected provider. For example:
   - **OpenAI:** Uses the OpenAI Python client to create a chat completion.
+  - **Azure OpenAI:** Uses the OpenAI Python client configured for Azure endpoints.
   - **Anthropic:** Sends a request to Anthropic’s API using custom headers.
   - **DeepSeek:** Uses the `requests` library to post JSON data, and then cleans up responses that may be wrapped in markdown code blocks.
 

python_gpt_po/main.py

@@ -11,7 +11,7 @@
 from argparse import Namespace
 from typing import Dict, List, Optional
 
-from .models.config import TranslationConfig
+from .models.config import TranslationConfig, TranslationFlags
 from .models.enums import ModelProvider
 from .models.provider_clients import ProviderClients
 from .services.model_manager import ModelManager
@@ -180,15 +180,25 @@ def main():
             logging.error(str(e))
             sys.exit(1)
 
+        # Check for deprecated --fuzzy option
+        if args.fuzzy:
+            logging.warning(
+                "WARNING: --fuzzy is DEPRECATED and has risky behavior. "
+                "Use --fix-fuzzy instead to properly translate and clean fuzzy entries."
+            )
         # Create translation configuration
+        flags = TranslationFlags(
+            bulk_mode=args.bulk,
+            fuzzy=args.fuzzy,
+            fix_fuzzy=args.fix_fuzzy,
+            folder_language=args.folder_language,
+            mark_ai_generated=not args.no_ai_comment
+        )
         config = TranslationConfig(
             provider_clients=provider_clients,
             provider=provider,
             model=model,
-            bulk_mode=args.bulk,
-            fuzzy=args.fuzzy,
-            fix_fuzzy=args.fix_fuzzy,
-            folder_language=args.folder_language
+            flags=flags
         )
 
         # Process translations

python_gpt_po/models/config.py

@@ -8,13 +8,20 @@
 from .provider_clients import ProviderClients
 
 
+@dataclass
+class TranslationFlags:
+    """Boolean flags for translation behavior."""
+    bulk_mode: bool = False
+    fuzzy: bool = False
+    fix_fuzzy: bool = False
+    folder_language: bool = False
+    mark_ai_generated: bool = True
+
+
 @dataclass
 class TranslationConfig:
     """Class to hold configuration parameters for the translation service."""
     provider_clients: ProviderClients
     provider: ModelProvider
     model: str
-    bulk_mode: bool = False
-    fuzzy: bool = False
-    fix_fuzzy: bool = False
-    folder_language: bool = False
+    flags: TranslationFlags