fix(ChatAdapter): None/True/False parsing in Dict[str, Any] fields (#8820)

[isaacbmiller]

When LLMs output Python-style dicts with None values, json_repair.loads was converting None to the string "None" instead of null. This caused high tool call failure rates for Dict[str, Any] fields.

Changes:

• Update ChatAdapter to instruct "valid JSON" instead of "valid Python dict"
• Reorder parse_value to try ast.literal_eval before json_repair.loads
• Preserve ast.literal_eval results as fallback when json_repair fails
• Add comprehensive tests for None, True, False, and null handling

This maintains backward compatibility by accepting both Python and JSON syntax while correctly parsing None/True/False/null values.

Fixes #8820

[Copilot]

Pull Request Overview

This PR fixes an issue where None values in dict[str, Any] fields were being converted to the string "None" instead of remaining as Python None objects. The fix prioritizes Python literal parsing (ast.literal_eval) over JSON parsing (json_repair.loads) to properly handle Python-style literals like None, True, and False.

Key changes:

• Modified parse_value function to attempt ast.literal_eval first before falling back to json_repair.loads
• Added comprehensive test coverage for parsing dicts with None, boolean values, and JSON null
• Updated imports to include Any type annotation

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
dspy/adapters/utils.py	Refactored parse_value to prioritize Python literal parsing over JSON repair, ensuring None values are correctly preserved
tests/adapters/test_chat_adapter.py	Added comprehensive test cases for dict fields with None, boolean values, and various edge cases including string "None"

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

[nitpick] The ast_result variable is stored but only used if both ast.literal_eval and validation fail, then later json_repair.loads also fails. This creates an implicit fallback chain that's difficult to follow. Consider restructuring to make the fallback logic more explicit - for example, storing whether ast parsing succeeded separately, or adding a comment explaining when ast_result is used (line 183).

[Copilot]

[nitpick] The comment suggests that when ast.literal_eval succeeds but validation fails, the code will fall through to try json_repair. However, this scenario seems unusual - if Python literal parsing succeeds, it's unlikely that JSON parsing would validate better. Consider adding a comment explaining a concrete example of when this fallback is beneficial, or re-evaluate if this exception handling is necessary.

Suggested change

	pass # Parsing succeeded but validation failed, will try json_repair
	pass # If literal_eval produces a value that fails validation, we attempt json_repair as a last resort.
	# This is rare, but can occur if the annotation expects a structure that is more easily represented in JSON.

[TomeHirata]

Suggested change

	except (ValueError, SyntaxError):
	pass # Parsing failed
	except pydantic.ValidationError:
	pass # Parsing succeeded but validation failed, will try json_repair
	except (ValueError, SyntaxError, pydantic.ValidationError):
	pass

[TomeHirata]

Update ChatAdapter to instruct "valid JSON" instead of "valid Python dict"

Is this included in this PR? Also it might make sense to continue to let LLM return python literals for ChatAdapter and have separate utilities between ChatAdapter and JSONAdapter.

[TomeHirata]

Can we add some test cases in tests/adapters/test_adapter_utils.py?