[WIP] Create a Python script for preprocessing CSV dataset to TSV format

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

Original prompt

Create a Python script for preprocessing the updated CSV dataset into a TSV format suitable for training a conditional causal LM in ReInvent4. The script should validate SMILES, canonicalize them, remove invalid entries, handle duplicates, and output in the format: prompttargetec_score. Include RDKit checks and basic statistics (e.g., count per MODE). The dataset has ~8M rows, so make it efficient with batch processing or multiprocessing. Also, provide a simple tokenizer training script using SentencePiece for SMILES with special tokens for , , etc. Finally, include a sample HuggingFace training config YAML for fine-tuning a causal LM.

This pull request was created as a result of the following prompt from Copilot chat.

Create a Python script for preprocessing the updated CSV dataset into a TSV format suitable for training a conditional causal LM in ReInvent4. The script should validate SMILES, canonicalize them, remove invalid entries, handle duplicates, and output in the format: prompttargetec_score. Include RDKit checks and basic statistics (e.g., count per MODE). The dataset has ~8M rows, so make it efficient with batch processing or multiprocessing. Also, provide a simple tokenizer training script using SentencePiece for SMILES with special tokens for , , etc. Finally, include a sample HuggingFace training config YAML for fine-tuning a causal LM.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.