⚡️ Speed up function nested_to_record by 364%

[codeflash-ai]

📄 364% (3.64x) speedup for nested_to_record in pandas/io/json/_normalize.py

⏱️ Runtime : 10.1 milliseconds → 2.17 milliseconds (best of 67 runs)

📝 Explanation and details

The key optimization in this code is replacing copy.deepcopy(d) with dict(d) on line 67, which provides a dramatic performance improvement of 363% speedup.

What changed:

• copy.deepcopy(d) was replaced with dict(d) to create a shallow copy instead of a deep copy
• The copy module import was removed since it's no longer needed

Why this optimization works:
The original code used deepcopy unnecessarily because the algorithm only mutates the top-level dictionary keys during flattening. When recursing into nested dictionaries, those nested values are completely popped from the parent and replaced with flattened key-value pairs. Since nested dictionaries are never modified in-place (only removed and replaced), a shallow copy is sufficient and much faster.

deepcopy recursively copies all nested objects, which is expensive for deeply nested structures. dict(d) only copies the top-level key-value pairs, leaving nested objects as references - exactly what's needed here.

Performance impact based on test results:

• Small nested dictionaries see 150-200% speedup (simple cases)
• Large flat dictionaries see 500%+ speedup (1000 keys: 414μs → 68.9μs)
• Deeply nested structures benefit most: 577% speedup for 10-level deep nesting
• Large datasets with many records show 300-400% improvements

Hot path considerations:
Based on the function references, nested_to_record is called from json_normalize, which is a primary pandas JSON processing function. It's used both directly for simple normalization and within recursive extraction for complex record paths. This optimization significantly benefits any JSON data processing workflows in pandas, especially those dealing with nested structures or large datasets.

The optimization is particularly effective for the common pandas use case of flattening JSON data with moderate to deep nesting, where the original deepcopy overhead dominated execution time.

✅ Correctness verification report:

Test	Status
⚙️ Existing Unit Tests	✅ 41 Passed
🌀 Generated Regression Tests	✅ 55 Passed
⏪ Replay Tests	🔘 None Found
🔎 Concolic Coverage Tests	🔘 None Found
📊 Tests Coverage	100.0%

⚙️ Existing Unit Tests and Runtime

Test File::Test Function	Original ⏱️	Optimized ⏱️	Speedup
io/json/test_normalize.py::TestNestedToRecord.test_donot_drop_nonevalues	22.2μs	7.65μs	190%✅
io/json/test_normalize.py::TestNestedToRecord.test_flat_stays_flat	8.96μs	2.79μs	221%✅
io/json/test_normalize.py::TestNestedToRecord.test_nested_flattens	21.4μs	6.91μs	210%✅
io/json/test_normalize.py::TestNestedToRecord.test_nonetype_multiple_levels	37.2μs	9.41μs	295%✅
io/json/test_normalize.py::TestNestedToRecord.test_nonetype_top_level_bottom_level	38.0μs	9.60μs	296%✅
io/json/test_normalize.py::TestNestedToRecord.test_one_level_deep_flattens	10.3μs	3.90μs	164%✅
io/json/test_normalize.py::TestNestedToRecord.test_with_large_max_level	59.9μs	11.5μs	420%✅
io/json/test_normalize.py::TestNestedToRecord.test_with_max_level	57.4μs	17.4μs	229%✅

🌀 Generated Regression Tests and Runtime

from __future__ import annotations


# imports
from pandas.io.json._normalize import nested_to_record

# unit tests

# --------------------------
# Basic Test Cases
# --------------------------


def test_flat_dict():
    # Flat dict should be unchanged
    inp = {"a": 1, "b": 2}
    expected = {"a": 1, "b": 2}
    codeflash_output = nested_to_record(inp)  # 5.42μs -> 1.89μs (187% faster)


def test_simple_nesting():
    # Single level nesting
    inp = {"a": {"b": 1}, "c": 2}
    expected = {"a.b": 1, "c": 2}
    codeflash_output = nested_to_record(inp)  # 10.3μs -> 3.61μs (186% faster)


def test_double_nesting():
    # Two levels of nesting
    inp = {"a": {"b": {"c": 1}}, "d": 2}
    expected = {"a.b.c": 1, "d": 2}
    codeflash_output = nested_to_record(inp)  # 14.3μs -> 4.58μs (213% faster)


def test_list_of_dicts():
    # List of dicts input
    inp = [{"a": {"b": 1}}, {"c": 2}]
    expected = [{"a.b": 1}, {"c": 2}]
    codeflash_output = nested_to_record(inp)  # 11.1μs -> 4.07μs (173% faster)


def test_custom_separator():
    # Custom separator
    inp = {"a": {"b": {"c": 1}}, "d": 2}
    expected = {"a|b|c": 1, "d": 2}
    codeflash_output = nested_to_record(inp, sep="|")  # 14.1μs -> 4.64μs (203% faster)


def test_prefix_and_separator():
    # Prefix and separator
    inp = {"x": {"y": 1}}
    expected = {"pre.x.y": 1}
    codeflash_output = nested_to_record(
        inp, prefix="pre", sep="."
    )  # 9.51μs -> 3.49μs (172% faster)


def test_max_level():
    # max_level should limit flattening
    inp = {"a": {"b": {"c": 1}}, "d": 2}
    # Only flatten one level
    expected = {"a.b": {"c": 1}, "d": 2}
    codeflash_output = nested_to_record(
        inp, max_level=1
    )  # 12.3μs -> 4.03μs (206% faster)


def test_empty_dict():
    # Empty dict should return empty dict
    inp = {}
    expected = {}
    codeflash_output = nested_to_record(inp)  # 5.03μs -> 1.71μs (195% faster)


def test_empty_list():
    # Empty list should return empty list
    inp = []
    expected = []
    codeflash_output = nested_to_record(inp)  # 830ns -> 778ns (6.68% faster)


# --------------------------
# Edge Test Cases
# --------------------------


def test_nested_empty_dict():
    # Nested empty dict should not produce any keys
    inp = {"a": {}, "b": 2}
    expected = {"b": 2}
    codeflash_output = nested_to_record(inp)  # 9.69μs -> 3.23μs (200% faster)


def test_deeply_nested_dict():
    # Deeply nested dict
    inp = {"a": {"b": {"c": {"d": {"e": 42}}}}}
    expected = {"a.b.c.d.e": 42}
    codeflash_output = nested_to_record(inp)  # 23.5μs -> 5.71μs (312% faster)


def test_dict_with_none_value():
    # Dict with None value
    inp = {"a": None, "b": {"c": None}}
    expected = {"a": None, "b.c": None}
    codeflash_output = nested_to_record(inp)  # 10.0μs -> 3.63μs (176% faster)


def test_dict_with_list_value():
    # Dict with list value should not flatten the list
    inp = {"a": [1, 2, 3], "b": {"c": [4, 5]}}
    expected = {"a": [1, 2, 3], "b.c": [4, 5]}
    codeflash_output = nested_to_record(inp)  # 14.0μs -> 3.62μs (286% faster)


def test_dict_with_tuple_value():
    # Dict with tuple value should not flatten the tuple
    inp = {"a": (1, 2), "b": {"c": (3, 4)}}
    expected = {"a": (1, 2), "b.c": (3, 4)}
    codeflash_output = nested_to_record(inp)  # 15.6μs -> 3.79μs (312% faster)


def test_dict_with_mixed_types():
    # Dict with mixed types in values
    inp = {"a": {"b": 1, "c": [2, 3]}, "d": "foo"}
    expected = {"a.b": 1, "a.c": [2, 3], "d": "foo"}
    codeflash_output = nested_to_record(inp)  # 14.0μs -> 4.16μs (236% faster)


def test_dict_with_duplicate_keys_after_flattening():
    # Dict with keys that would collide after flattening
    inp = {"a.b": 1, "a": {"b": 2}}
    expected = {"a.b": 2}
    # The nested key should overwrite the top-level key
    codeflash_output = nested_to_record(inp)
    result = codeflash_output  # 12.4μs -> 4.38μs (182% faster)


def test_dict_with_max_level_zero():
    # max_level=0 should not flatten at all
    inp = {"a": {"b": {"c": 1}}, "d": 2}
    expected = {"a": {"b": {"c": 1}}, "d": 2}
    codeflash_output = nested_to_record(
        inp, max_level=0
    )  # 8.64μs -> 2.27μs (281% faster)


def test_dict_with_max_level_exceeding_depth():
    # max_level > actual depth should flatten fully
    inp = {"a": {"b": {"c": 1}}, "d": 2}
    expected = {"a.b.c": 1, "d": 2}
    codeflash_output = nested_to_record(
        inp, max_level=10
    )  # 14.8μs -> 5.25μs (182% faster)


def test_dict_with_empty_string_key():
    # Empty string keys should work
    inp = {"": {"a": 1}, "b": 2}
    expected = {".a": 1, "b": 2}
    codeflash_output = nested_to_record(
        inp, prefix="", sep="."
    )  # 10.6μs -> 3.83μs (177% faster)


def test_dict_with_nested_empty_string_key():
    # Nested empty string keys
    inp = {"a": {"": 1}}
    expected = {"a.": 1}
    codeflash_output = nested_to_record(inp)  # 9.57μs -> 3.29μs (191% faster)


# --------------------------
# Large Scale Test Cases
# --------------------------


def test_large_flat_dict():
    # Large flat dict
    inp = {str(i): i for i in range(1000)}
    expected = {str(i): i for i in range(1000)}
    codeflash_output = nested_to_record(inp)  # 414μs -> 68.9μs (502% faster)


def test_large_nested_dict():
    # Large nested dict: 10 levels deep
    d = 0
    for i in reversed(range(10)):
        d = {str(i): d}
    inp = d
    # Only the deepest key should have value 0
    expected = {"0.1.2.3.4.5.6.7.8.9": 0}
    codeflash_output = nested_to_record(inp)  # 60.0μs -> 8.86μs (577% faster)


def test_large_list_of_dicts():
    # Large list of dicts
    inp = [{"a": {"b": i}} for i in range(1000)]
    expected = [{"a.b": i} for i in range(1000)]
    codeflash_output = nested_to_record(inp)  # 3.47ms -> 777μs (346% faster)


def test_large_dict_with_many_nested_keys():
    # Dict with many nested keys
    inp = {f"key{i}": {"subkey": i} for i in range(500)}
    expected = {f"key{i}.subkey": i for i in range(500)}
    codeflash_output = nested_to_record(inp)  # 1.36ms -> 304μs (345% faster)


def test_large_dict_with_max_level():
    # Large dict with max_level set
    inp = {f"key{i}": {"subkey": {"deep": i}} for i in range(100)}
    expected = {f"key{i}.subkey": {"deep": i} for i in range(100)}
    codeflash_output = nested_to_record(
        inp, max_level=1
    )  # 420μs -> 66.2μs (535% faster)


def test_large_dict_with_custom_separator():
    # Large dict with custom separator
    inp = {f"a{i}": {"b": i} for i in range(100)}
    expected = {f"a{i}|b": i for i in range(100)}
    codeflash_output = nested_to_record(inp, sep="|")  # 277μs -> 64.5μs (331% faster)


def test_large_dict_with_mixed_types():
    # Large dict with mixed types
    inp = {f"key{i}": {"num": i, "lst": [i, i + 1]} for i in range(100)}


# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

from __future__ import annotations


# imports
from pandas.io.json._normalize import nested_to_record

# unit tests

# --------------------------- BASIC TEST CASES ---------------------------


def test_flat_dict():
    # Test with a flat dictionary (no nesting)
    d = {"a": 1, "b": 2}
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 5.48μs -> 1.91μs (187% faster)


def test_single_level_nesting():
    # Test with one level of nesting
    d = {"a": 1, "b": {"c": 2, "d": 3}}
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 11.1μs -> 4.33μs (157% faster)


def test_double_level_nesting():
    # Test with two levels of nesting
    d = {"a": {"b": {"c": 1}}}
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 13.6μs -> 4.47μs (203% faster)


def test_list_of_dicts():
    # Test with a list of flat dictionaries
    ds = [{"a": 1}, {"b": 2}]
    codeflash_output = nested_to_record(ds)
    result = codeflash_output  # 6.97μs -> 2.25μs (209% faster)


def test_list_of_nested_dicts():
    # Test with a list of nested dictionaries
    ds = [{"a": {"b": 1}}, {"c": {"d": 2}}]
    codeflash_output = nested_to_record(ds)
    result = codeflash_output  # 14.3μs -> 5.02μs (186% faster)


def test_example_from_docstring():
    # Test the example from the docstring
    d = dict(flat1=1, dict1=dict(c=1, d=2), nested=dict(e=dict(c=1, d=2), d=2))
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 22.0μs -> 7.31μs (200% faster)
    expected = {
        "flat1": 1,
        "dict1.c": 1,
        "dict1.d": 2,
        "nested.e.c": 1,
        "nested.e.d": 2,
        "nested.d": 2,
    }


def test_custom_separator():
    # Test with a custom separator
    d = {"a": {"b": {"c": 1}}}
    codeflash_output = nested_to_record(d, sep="_")
    result = codeflash_output  # 13.4μs -> 4.44μs (202% faster)


def test_prefix_and_separator():
    # Test with a custom prefix and separator
    d = {"x": {"y": 2}}
    codeflash_output = nested_to_record(d, prefix="pre", sep="-")
    result = codeflash_output  # 9.51μs -> 3.48μs (174% faster)


# --------------------------- EDGE TEST CASES ---------------------------


def test_empty_dict():
    # Test with an empty dictionary
    d = {}
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 3.65μs -> 1.23μs (198% faster)


def test_empty_list():
    # Test with an empty list
    ds = []
    codeflash_output = nested_to_record(ds)
    result = codeflash_output  # 755ns -> 711ns (6.19% faster)


def test_dict_with_list_value():
    # Test with a list as a value (should not be flattened)
    d = {"a": [1, 2, 3], "b": {"c": [4, 5]}}
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 16.0μs -> 4.36μs (268% faster)


def test_dict_with_tuple_value():
    # Test with a tuple as a value (should not be flattened)
    d = {"a": (1, 2), "b": {"c": (3, 4)}}
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 15.4μs -> 3.90μs (295% faster)


def test_max_level_zero():
    # Test with max_level=0 (no flattening should occur)
    d = {"a": {"b": 1}}
    codeflash_output = nested_to_record(d, max_level=0)
    result = codeflash_output  # 6.53μs -> 2.12μs (208% faster)


def test_max_level_one():
    # Test with max_level=1 (only first level flattened)
    d = {"a": {"b": {"c": 1}}, "d": 2}
    codeflash_output = nested_to_record(d, max_level=1)
    result = codeflash_output  # 12.8μs -> 4.09μs (212% faster)


def test_max_level_two():
    # Test with max_level=2 (flatten up to two levels)
    d = {"a": {"b": {"c": 1}}, "d": 2}
    codeflash_output = nested_to_record(d, max_level=2)
    result = codeflash_output  # 14.6μs -> 5.01μs (192% faster)


def test_dict_with_empty_dict_value():
    # Test with an empty dict as a value
    d = {"a": {}, "b": 1}
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 8.49μs -> 2.86μs (197% faster)


def test_dict_with_mixed_types():
    # Test with mixed types at various levels
    d = {"a": {"b": 1, "c": [2, 3]}, "d": {"e": None, "f": True}}
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 19.4μs -> 5.71μs (239% faster)


def test_dict_with_nested_empty_dicts():
    # Test with nested empty dicts
    d = {"a": {"b": {}}}
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 11.3μs -> 3.58μs (216% faster)


def test_dict_with_deeply_nested_empty_dict():
    # Test with deep nesting ending in empty dict
    d = {"a": {"b": {"c": {}}}}
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 17.4μs -> 4.91μs (254% faster)


def test_dict_with_set_value():
    # Test with a set as a value (should not be flattened)
    d = {"a": {1, 2, 3}, "b": {"c": {4, 5}}}
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 24.8μs -> 3.74μs (562% faster)


def test_dict_with_none_value():
    # Test with None as a value
    d = {"a": None, "b": {"c": None}}
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 9.84μs -> 3.67μs (168% faster)


# --------------------------- LARGE SCALE TEST CASES ---------------------------


def test_large_flat_dict():
    # Test with a large flat dictionary
    d = {f"key{i}": i for i in range(1000)}
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 416μs -> 68.5μs (507% faster)
    expected = {f"key{i}": i for i in range(1000)}


def test_large_nested_dict():
    # Test with a large nested dictionary
    d = {}
    for i in range(100):
        d[f"level1_{i}"] = {f"level2_{j}": j for j in range(10)}
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 1.18ms -> 289μs (306% faster)
    expected = {}
    for i in range(100):
        for j in range(10):
            expected[f"level1_{i}.level2_{j}"] = j


def test_large_deeply_nested_dict():
    # Test with a deeply nested dictionary (depth 10)
    d = {}
    current = d
    for i in range(10):
        current[f"level{i}"] = {}
        if i == 9:
            current[f"level{i}"]["final"] = "end"
        current = current[f"level{i}"]
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 67.6μs -> 9.65μs (600% faster)
    # The key should be 'level0.level1.level2.level3.level4.level5.level6.level7.level8.level9.final'
    expected_key = ".".join([f"level{i}" for i in range(10)]) + ".final"


def test_large_list_of_dicts():
    # Test with a large list of flat dictionaries
    ds = [{f"key{i}": i} for i in range(500)]
    codeflash_output = nested_to_record(ds)
    result = codeflash_output  # 562μs -> 91.7μs (513% faster)
    expected = [{f"key{i}": i} for i in range(500)]


def test_large_list_of_nested_dicts():
    # Test with a large list of nested dictionaries
    ds = []
    for i in range(100):
        ds.append({f"a{i}": {"b": i}})
    codeflash_output = nested_to_record(ds)
    result = codeflash_output  # 341μs -> 73.8μs (363% faster)
    expected = [{f"a{i}.b": i} for i in range(100)]


def test_large_dict_with_mixed_types():
    # Test with a large dict with mixed types
    d = {
        f"key{i}": {"sub": i, "sublist": [i, i + 1], "subdict": {"x": i}}
        for i in range(50)
    }
    codeflash_output = nested_to_record(d)
    result = codeflash_output  # 469μs -> 87.4μs (437% faster)
    expected = {}
    for i in range(50):
        expected[f"key{i}.sub"] = i
        expected[f"key{i}.sublist"] = [i, i + 1]
        expected[f"key{i}.subdict.x"] = i


def test_large_dict_with_max_level():
    # Test with a large nested dict and max_level set
    d = {f"outer{i}": {f"inner{j}": {"val": j} for j in range(5)} for i in range(20)}
    # Set max_level=1 so only first level is flattened
    codeflash_output = nested_to_record(d, max_level=1)
    result = codeflash_output  # 289μs -> 37.7μs (668% faster)
    expected = {}
    for i in range(20):
        for j in range(5):
            expected[f"outer{i}.inner{j}"] = {"val": j}


# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

To edit these changes git checkout codeflash/optimize-nested_to_record-mi3x6q78 and push.