Conversion from HuggingFace dataset to Pandas DF causing dropped 'string' columns

By default, df.describe() does NOT show string columns. It only summarizes numeric columns. String (object) columns are not dropped — they are just not displayed.

You can check your code by this:

df.columns
df.dtypes
df.describe(include="all")

HuggingFace Dataset objects can have a format attached (e.g., torch, numpy). When a format is set, only the formatted columns are returned unless explicitly told otherwise.

To get consistent columns in both Huggingface and Pandas dataframe, you can use this code:

from datasets import loaddataset
prcommits = loaddataset("hao-li/AIDev", "prcommits")["train"]
prcommits.resetformat()      # IMPORTANT
commitsdf = prcommits.topandas()
print(commitsdf.columns)
print(commits_df.dtypes)

Hope it helps!