BUG: astype() unexpected mutation of values

[hajes]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

the whole df usually has got >10 rows x 49 columns

`df = pd.DataFrame.from_dict(dictionary)`
`df = df.astype(DF_DTYPES)` where this particular column is defined `{'is_inco_oper': int}`

Issue Description

original df is int64 converted to int32, and values are mutated in row 7-9

0     170914000
1      63275000
2     248049000
3     746273000
4    1067817000
5    1022510000
6    1218091000
7    3415195000
8    3669382000
9    2531232000
Name: is_inco_oper, dtype: int64
0     170914000
1      63275000
2     248049000
3     746273000
4    1067817000
5    1022510000
6    1218091000
7    -879772296
8    -625585296
9   -1763735296
Name: is_inco_oper, dtype: int32

Expected Behavior

keep the original values

Installed Versions

INSTALLED VERSIONS

commit : d9cdd2e
python : 3.12.3.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : AMD64 Family 25 Model 33 Stepping 0, AuthenticAMD
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_Switzerland.1252

pandas : 2.2.2
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0.post0
setuptools : 70.0.0
pip : 24.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : 8.24.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.8.4
numba : 0.59.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.13.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

[Aloqeely]

Thanks for the report! It would help to post the dictionary you're using for the DataFrame, but, I think the reason for the values changing is because the highest value an int32 can carry is 2,147,483,647, and the last 3 values in your DataFrame are all bigger that so they cannot be stored as an int32 and that causes them to overflow.

[hajes]

Question is why it forces int32 if number is clearly int64. According to numpy reference manual, int64 is default dtype. Why it uses 32b on 64b is a puzzle. So far the kludge is to force pandas Int64, instead of defining just int.

` [{'f_year': 2014, 'is_net_rev': 840290000, 'is_cost_rev': -585984000, 'is_gross_prof': 254306000, 'is_rd': nan, 'is_inco_oper': 170914000, 'is_inco_pretax': 83039000, 'is_tax_prov': -21493000, 'is_inco_net': 61546000, 'is_eps_b_r': 0.73, 'is_eps_d_r': 0.73, 'is_inco_comp': 61655000, 'bs_cash_eq': 43646000, 'bs_short_invest': 11704000, 'bs_inventory': 67138000, 'bs_ass_cur_total': 193818000, 'bs_ppe': 1360596000, 'bs_goodwill': 700188000, 'bs_ass_total': 2357641000, 'bs_debt_st': 1150000000, 'bs_ac_pay': 359042000, 'bs_acc_retirement': nan, 'bs_acc_other': nan, 'bs_liab_cur_total': 1509042000, 'bs_debt': nan, 'bs_acc_retirement_costs': nan, 'bs_tax_deferred': nan, 'bs_other_deferred': 2000000, 'bs_liab_total': 1511042000, 'bs_equity_stockholder': nan, 'bs_equity_total_stockholder_liab': nan, 'bs_stock_pref_issued': nan, 'bs_stock_common_issued': nan, 'cf_depre': nan, 'cf_goodwill_amort': nan, 'cf_tax_income': nan, 'cf_cash_net_operating': 372667000, 'cf_capex': -173367000, 'cf_ppe_add': 0, 'cf_dividends_paid': nan, 'cf_cashflow': -102700000, 'cf_cash_end': 43646000, 'shares_basic_reported': nan, 'shares_diluted_reported': nan, 'shares_com_outst': 8430000000, 'shares_pref_outst': nan, 'stock_price': 0.41, 'stock_ticker': '1277.HK', 'notes': 'ST-investments = pledged deposits + Restricted cash'}, {'f_year': 2015, 'is_net_rev': 1176041000, 'is_cost_rev': -1047584000, 'is_gross_prof': 128457000, 'is_rd': nan, 'is_inco_oper': 63275000, 'is_inco_pretax': -2556000, 'is_tax_prov': 454000, 'is_inco_net': -2102000, 'is_eps_b_r': -0.02, 'is_eps_d_r': -0.02, 'is_inco_comp': -1247000, 'bs_cash_eq': 92011000, 'bs_short_invest': 5102000, 'bs_inventory': 32022000, 'bs_ass_cur_total': 178387000, 'bs_ppe': 1289771000, 'bs_goodwill': 680696000, 'bs_ass_total': 2099293000, 'bs_debt_st': 1154918000, 'bs_ac_pay': 275290000, 'bs_acc_retirement': nan, 'bs_acc_other': nan, 'bs_liab_cur_total': 1430208000, 'bs_debt': nan, 'bs_acc_retirement_costs': nan, 'bs_tax_deferred': nan, 'bs_other_deferred': 2120000, 'bs_liab_total': 1432328000, 'bs_equity_stockholder': nan, 'bs_equity_total_stockholder_liab': nan, 'bs_stock_pref_issued': nan, 'bs_stock_common_issued': nan, 'cf_depre': nan, 'cf_goodwill_amort': nan, 'cf_tax_income': nan, 'cf_cash_net_operating': 206903000, 'cf_capex': -91173000, 'cf_ppe_add': 0, 'cf_dividends_paid': nan, 'cf_cashflow': 47510000, 'cf_cash_end': 92011000, 'shares_basic_reported': nan, 'shares_diluted_reported': nan, 'shares_com_outst': 8430000000, 'shares_pref_outst': nan, 'stock_price': 0.3, 'stock_ticker': '1277.HK', 'notes': 'ST-debt = Bank loan + Other borrowing'}, {'f_year': 2016, 'is_net_rev': 1051457000, 'is_cost_rev': -750342000, 'is_gross_prof': 301115000, 'is_rd': nan, 'is_inco_oper': 248049000, 'is_inco_pretax': 193088000, 'is_tax_prov': -54982000, 'is_inco_net': 138106000, 'is_eps_b_r': 1.64, 'is_eps_d_r': 1.64, 'is_inco_comp': 138439000, 'bs_cash_eq': 85742000, 'bs_short_invest': 25101000, 'bs_inventory': 50712000, 'bs_ass_cur_total': 211894000, 'bs_ppe': 1232019000, 'bs_goodwill': 667092000, 'bs_ass_total': 2220436000, 'bs_debt_st': 579540000, 'bs_ac_pay': 201795000, 'bs_acc_retirement': nan, 'bs_acc_other': nan, 'bs_liab_cur_total': 804405000, 'bs_debt': 429993000, 'bs_acc_retirement_costs': nan, 'bs_tax_deferred': nan, 'bs_other_deferred': 2247000, 'bs_liab_total': 1236645000, 'bs_equity_stockholder': nan, 'bs_equity_total_stockholder_liab': nan, 'bs_stock_pref_issued': nan, 'bs_stock_common_issued': nan, 'cf_depre': 116339000, 'cf_goodwill_amort': 13957000, 'cf_tax_income': nan, 'cf_cash_net_operating': 264448000, 'cf_capex': -57354000, 'cf_ppe_add': -8710000, 'cf_dividends_paid': nan, 'cf_cashflow': -6602000, 'cf_cash_end': 85742000, 'shares_basic_reported': nan, 'shares_diluted_reported': nan, 'shares_com_outst': 8430000000, 'shares_pref_outst': nan, 'stock_price': 0.2, 'stock_ticker': '1277.HK', 'notes': 'Additions to PPE = Additions to land lease prepayments'}, {'f_year': 2017, 'is_net_rev': 1749538000, 'is_cost_rev': -951047000, 'is_gross_prof': 798491000, 'is_rd': nan, 'is_inco_oper': 746273000, 'is_inco_pretax': 707480000, 'is_tax_prov': -167432000, 'is_inco_net': 540048000, 'is_eps_b_r': 6.41, 'is_eps_d_r': 6.41, 'is_inco_comp': 545299000, 'bs_cash_eq': 298311000, 'bs_short_invest': 155101000, 'bs_inventory': 86036000, 'bs_ass_cur_total': 676356000, 'bs_ppe': 1235051000, 'bs_goodwill': 647963000, 'bs_ass_total': 2671891000, 'bs_debt_st': 820667000, 'bs_ac_pay': 322271000, 'bs_acc_retirement': nan, 'bs_acc_other': nan, 'bs_liab_cur_total': 1235117000, 'bs_debt': 124771000, 'bs_acc_retirement_costs': nan, 'bs_tax_deferred': nan, 'bs_other_deferred': 3582000, 'bs_liab_total': 1363470000, 'bs_equity_stockholder': nan, 'bs_equity_total_stockholder_liab': nan, 'bs_stock_pref_issued': nan, 'bs_stock_common_issued': nan, 'cf_depre': 124817000, 'cf_goodwill_amort': 19564000, 'cf_tax_income': -79359000, 'cf_cash_net_operating': 833059000, 'cf_capex': -76024000, 'cf_ppe_add': -683000, 'cf_dividends_paid': -220669000, 'cf_cashflow': 213085000, 'cf_cash_end': 298311000, 'shares_basic_reported': nan, 'shares_diluted_reported': nan, 'shares_com_outst': 8430000000, 'shares_pref_outst': nan, 'stock_price': 0.31, 'stock_ticker': '1277.HK', 'notes': 'Other non-current liabilities = Accrual for reclamation costs'}, {'f_year': 2018, 'is_net_rev': 2443435000, 'is_cost_rev': -1347190000, 'is_gross_prof': 1096245000, 'is_rd': nan, 'is_inco_oper': 1067817000, 'is_inco_pretax': 1051044000, 'is_tax_prov': -244073000, 'is_inco_net': 806971000, 'is_eps_b_r': 9.57, 'is_eps_d_r': 9.57, 'is_inco_comp': 782474000, 'bs_cash_eq': 275846000, 'bs_short_invest': 255101000, 'bs_inventory': 7579000, 'bs_ass_cur_total': 717610000, 'bs_ppe': 1180022000, 'bs_goodwill': 623205000, 'bs_ass_total': 2755002000, 'bs_debt_st': 342277000, 'bs_ac_pay': 304214000, 'bs_acc_retirement': nan, 'bs_acc_other': nan, 'bs_liab_cur_total': 831991000, 'bs_debt': 130785000, 'bs_acc_retirement_costs': nan, 'bs_tax_deferred': 12250000, 'bs_other_deferred': 3976000, 'bs_liab_total': 979002000, 'bs_equity_stockholder': nan, 'bs_equity_total_stockholder_liab': nan, 'bs_stock_pref_issued': nan, 'bs_stock_common_issued': nan, 'cf_depre': 134189000, 'cf_goodwill_amort': 25196000, 'cf_tax_income': -189797000, 'cf_cash_net_operating': 1275552000, 'cf_capex': -103725000, 'cf_ppe_add': 0, 'cf_dividends_paid': -315785000, 'cf_cashflow': -23077000, 'cf_cash_end': 275846000, 'shares_basic_reported': nan, 'shares_diluted_reported': nan, 'shares_com_outst': 8430000000, 'shares_pref_outst': nan, 'stock_price': 0.48, 'stock_ticker': '1277.HK', 'notes': ''}, {'f_year': 2019, 'is_net_rev': 2736109000, 'is_cost_rev': -1668281000, 'is_gross_prof': 1067828000, 'is_rd': nan, 'is_inco_oper': 1022510000, 'is_inco_pretax': 1019795000, 'is_tax_prov': -186492000, 'is_inco_net': 833303000, 'is_eps_b_r': 9.88, 'is_eps_d_r': 9.88, 'is_inco_comp': 818009000, 'bs_cash_eq': 497192000, 'bs_short_invest': 267073000, 'bs_inventory': 83220000, 'bs_ass_cur_total': 979954000, 'bs_ppe': 1272422000, 'bs_goodwill': 594622000, 'bs_ass_total': 3105770000, 'bs_debt_st': 357652000, 'bs_ac_pay': 320126000, 'bs_acc_retirement': nan, 'bs_acc_other': nan, 'bs_liab_cur_total': 796047000, 'bs_debt': 46447000, 'bs_acc_retirement_costs': nan, 'bs_tax_deferred': nan, 'bs_other_deferred': 4413000, 'bs_liab_total': 846907000, 'bs_equity_stockholder': nan, 'bs_equity_total_stockholder_liab': nan, 'bs_stock_pref_issued': nan, 'bs_stock_common_issued': nan, 'cf_depre': 91256000, 'cf_goodwill_amort': 29021000, 'cf_tax_income': -242241000, 'cf_cash_net_operating': 1104557000, 'cf_capex': -95067000, 'cf_ppe_add': 0, 'cf_dividends_paid': -335146000, 'cf_cashflow': 224197000, 'cf_cash_end': 497192000, 'shares_basic_reported': nan, 'shares_diluted_reported': nan, 'shares_com_outst': 8430000000, 'shares_pref_outst': nan, 'stock_price': 0.41, 'stock_ticker': '1277.HK', 'notes': 'LT debt = Long-term payables'}, {'f_year': 2020, 'is_net_rev': 2961404000, 'is_cost_rev': -1679187000, 'is_gross_prof': 1282217000, 'is_rd': nan, 'is_inco_oper': 1218091000, 'is_inco_pretax': 1228156000, 'is_tax_prov': -413360000, 'is_inco_net': 814796000, 'is_eps_b_r': 9.67, 'is_eps_d_r': 9.67, 'is_inco_comp': 840253000, 'bs_cash_eq': 877745000, 'bs_short_invest': 522790000, 'bs_inventory': 73027000, 'bs_ass_cur_total': 1708038000, 'bs_ppe': 1219320000, 'bs_goodwill': 568634000, 'bs_ass_total': 3672650000, 'bs_debt_st': 210410000, 'bs_ac_pay': 263953000, 'bs_acc_retirement': nan, 'bs_acc_other': nan, 'bs_liab_cur_total': 854650000, 'bs_debt': 25001000, 'bs_acc_retirement_costs': nan, 'bs_tax_deferred': 28380000, 'bs_other_deferred': 4899000, 'bs_liab_total': 2759720000, 'bs_equity_stockholder': nan, 'bs_equity_total_stockholder_liab': nan, 'bs_stock_pref_issued': nan, 'bs_stock_common_issued': nan, 'cf_depre': 89274000, 'cf_goodwill_amort': 26426000, 'cf_tax_income': -222797000, 'cf_cash_net_operating': 1195970000, 'cf_capex': -85832000, 'cf_ppe_add': 0, 'cf_dividends_paid': -339396000, 'cf_cashflow': 379968000, 'cf_cash_end': 877745000, 'shares_basic_reported': nan, 'shares_diluted_reported': nan, 'shares_com_outst': 8430000000, 'shares_pref_outst': nan, 'stock_price': 0.35, 'stock_ticker': '1277.HK', 'notes': 'ST investment = Financial assets at fair value through profit or loss + pledged/restricted deposits'}, {'f_year': 2021, 'is_net_rev': 5580702000, 'is_cost_rev': -1976544000, 'is_gross_prof': 3604158000, 'is_rd': nan, 'is_inco_oper': 3415195000, 'is_inco_pretax': 3422296000, 'is_tax_prov': -954737000, 'is_inco_net': 2467559000, 'is_eps_b_r': 29.28, 'is_eps_d_r': 29.28, 'is_inco_comp': 2476572000, 'bs_cash_eq': 2387239000, 'bs_short_invest': 424977000, 'bs_inventory': 63442000, 'bs_ass_cur_total': 3219009000, 'bs_ppe': 1247473000, 'bs_goodwill': 545842000, 'bs_ass_total': 6042211000, 'bs_debt_st': 275695000, 'bs_ac_pay': 329560000, 'bs_acc_retirement': nan, 'bs_acc_other': nan, 'bs_liab_cur_total': 1391272000, 'bs_debt': 26391000, 'bs_acc_retirement_costs': nan, 'bs_tax_deferred': 44800000, 'bs_other_deferred': 5438000, 'bs_liab_total': 1470169000, 'bs_equity_stockholder': nan, 'bs_equity_total_stockholder_liab': nan, 'bs_stock_pref_issued': nan, 'bs_stock_common_issued': nan, 'cf_depre': 91690000, 'cf_goodwill_amort': 31392000, 'cf_tax_income': -532686000, 'cf_cash_net_operating': 3562859000, 'cf_capex': -180746000, 'cf_ppe_add': 0, 'cf_dividends_paid': -661531000, 'cf_cashflow': 1515361000, 'cf_cash_end': 2387239000, 'shares_basic_reported': nan, 'shares_diluted_reported': nan, 'shares_com_outst': 8430000000, 'shares_pref_outst': nan, 'stock_price': 0.59, 'stock_ticker': '1277.HK', 'notes': 'Intagibles = Intagible assets + Goodwill'}, {'f_year': 2022, 'is_net_rev': 6155830000, 'is_cost_rev': -2132372000, 'is_gross_prof': 4023458000, 'is_rd': nan, 'is_inco_oper': 3669382000, 'is_inco_pretax': 3634027000, 'is_tax_prov': -977712000, 'is_inco_net': 2656315000, 'is_eps_b_r': 31.61, 'is_eps_d_r': 31.61, 'is_inco_comp': 2643976000, 'bs_cash_eq': 551866000, 'bs_short_invest': 666802000, 'bs_inventory': 110213000, 'bs_ass_cur_total': 1612209000, 'bs_ppe': 1716365000, 'bs_goodwill': 3210599000, 'bs_ass_total': 9492641000, 'bs_debt_st': 300000000, 'bs_ac_pay': 518906000, 'bs_acc_retirement': nan, 'bs_acc_other': nan, 'bs_liab_cur_total': 1815415000, 'bs_debt': 1221992000, 'bs_acc_retirement_costs': nan, 'bs_tax_deferred': 52865000, 'bs_other_deferred': 6037000, 'bs_liab_total': 3175851000, 'bs_equity_stockholder': nan, 'bs_equity_total_stockholder_liab': nan, 'bs_stock_pref_issued': nan, 'bs_stock_common_issued': nan, 'cf_depre': 90406000, 'cf_goodwill_amort': 45863000, 'cf_tax_income': -872014000, 'cf_cash_net_operating': 3966187000, 'cf_capex': -816678000, 'cf_ppe_add': 0, 'cf_dividends_paid': -899228000, 'cf_cashflow': -1831215000, 'cf_cash_end': 551866000, 'shares_basic_reported': nan, 'shares_diluted_reported': nan, 'shares_com_outst': 8430000000, 'shares_pref_outst': nan, 'stock_price': 0.62, 'stock_ticker': '1277.HK', 'notes': ''}, {'f_year': 2023, 'is_net_rev': 4745069000, 'is_cost_rev': -1942031000, 'is_gross_prof': 2803038000, 'is_rd': nan, 'is_inco_oper': 2531232000, 'is_inco_pretax': 2440901000, 'is_tax_prov': -368178000, 'is_inco_net': 2072723000, 'is_eps_b_r': 24.65, 'is_eps_d_r': 24.65, 'is_inco_comp': 2115270000, 'bs_cash_eq': 734143000, 'bs_short_invest': 948376000, 'bs_inventory': 115274000, 'bs_ass_cur_total': 2157187000, 'bs_ppe': 2483678000, 'bs_goodwill': 3484321000, 'bs_ass_total': 10937914000, 'bs_debt_st': 1033000000, 'bs_ac_pay': 1066741000, 'bs_acc_retirement': nan, 'bs_acc_other': nan, 'bs_liab_cur_total': 2572076000, 'bs_debt': 853736000, 'bs_acc_retirement_costs': nan, 'bs_tax_deferred': 41841000, 'bs_other_deferred': 43073000, 'bs_liab_total': 3517715000, 'bs_equity_stockholder': nan, 'bs_equity_total_stockholder_liab': nan, 'bs_stock_pref_issued': nan, 'bs_stock_common_issued': nan, 'cf_depre': 106139000, 'cf_goodwill_amort': 33105000, 'cf_tax_income': -762461000, 'cf_cash_net_operating': 2474267000, 'cf_capex': -490004000, 'cf_ppe_add': 0, 'cf_dividends_paid': -775243000, 'cf_cashflow': 187156000, 'cf_cash_end': 734143000, 'shares_basic_reported': nan, 'shares_diluted_reported': nan, 'shares_com_outst': 8430000000, 'shares_pref_outst': nan, 'stock_price': 0.55, 'stock_ticker': '1277.HK', 'notes': ''}] `

[Aloqeely]

int maps to different values depending on the OS regardless of the data stored in the DataFrame, I believe if you're on windows it is treated as int32 which is actually behavior from numpy

In [2]: np.dtype(int)
Out[2]: dtype('int32')

There's an issue opened in numpy/numpy#9464 to change the default to int64, but yeah as you said it would be better to have {'is_inco_oper': int} be defined as {'is_inco_oper': 'int64'} to avoid any unexpected behavior

[hajes]

so it is basically issue with NumPy arrays that are part of pandas if I understand correctly. I have studied datatypes on pandas reference manual, it is written by some programmer language that only programmers understand :-P

pandas use numpy as default, and allows own extended dtypes if I understood correctly.

[asishm]

There have been similar issues raised in the past (#44925, #49003) and all of them due to the numpy issue @Aloqeely linked. The recommendation is to specify the dtype precision if you want to be sure.