ENH: Make DataFrame generic

[cmp0xff]

Towards #1548

• I don't know how to make setter work so that the typings can be updated when one assigns index or columns.
• The two overlapping overloads of __getitem__ in _LocIndexerFrame need to be fixed

Comments and suggestions are welcomed.

[loicdiridollou]

I am not sure I see the whole idea about this PR, also I would have expected more breakages in the code, like for transpose method it should flip the index and columns

[loicdiridollou]

Not totally sure if this is gonna break a lot of things, by default DataFrame will have RangeIndex as index and columns, isn't this gonna change that?

[cmp0xff]

This is copying another DataFrame, I suppose, so nothing is changed. Will add tests later.

[cmp0xff]

If you create a DataFrame from data without an index, RangeIndex will surely take over. That takes further overloads of __new__.

As for now I just want to show a prototype of what can happen.

[loicdiridollou]

Shouldn't the default be RangeIndex here?

[cmp0xff]

For df.columns we somehow have a preference for it being Index[str], see def columns(self) -> Index[str]: ... in the old code. That's the reason.

[cmp0xff]

transpose method it should flip the index and columns

Can certainly do, if the whole idea is viable.

More importantly, we need to figure out if _LocIndexFrame can be fixed at all, before continuing.

Last time when I was trying to add the backend type variable to Series, too many things crashed so that I could not continue. This time it seems more controllable for now.

[cmp0xff]

This is copying another DataFrame, I suppose, so nothing is changed. Will add tests later.

[cmp0xff]

For df.columns we somehow have a preference for it being Index[str], see def columns(self) -> Index[str]: ... in the old code. That's the reason.

[cmp0xff]

If you create a DataFrame from data without an index, RangeIndex will surely take over. That takes further overloads of __new__.

As for now I just want to show a prototype of what can happen.

[Dr-Irv]

I am not sure I see the whole idea about this PR

I'm in agreement. If this were to work, what would be the benefit?

Note that DatatFrame.columns returns Index[str] since most people use strings to name their columns. Technically, they could be any Hashable, or even a MultiIndex However, if we made columns be Index[Any] then for the majority of people, they'd have to do cast to get the actual string names.

[cmp0xff]

Benefits

Aside from paving the way to resolving #1548, we have the following issue:

import pandas as pd
from typing_extensions import reveal_type

df = pd.DataFrame(
    {"a": [1, 2, 1, 2], "b": [1, 1, 2, 2], "c": [1, 2, 3, 4], "d": [5, 6, 7, 8]},
)

pivoted = df.pivot_table(["c", "d"], "a", "b")

reveal_type(pivoted["c"])  # pyright gives Series[Any], runtime is DataFrame
reveal_type(pivoted.columns)  # pyright gives Index[str], runtime is MultiIndex

Default typing for .columns

I agree it should be Index[str]

Default typing for .index

I think RangeIndex might be too radical. Keeping Index[Any] means keeping the current behaviour.

We can give RangeIndex in the following (but not exclusively) cases:

• when creating a DataFrame from unindex objects without specifying index
• when ignore_index=True
• when .reset_index()