Hacker News new | ask | show | jobs
by qwhelan 2330 days ago
You should upgrade your version of pandas if possible - that's been fixed for a few versions now.
1 comments

Oh interesting I'll have to check, we did an upgrade pass a while ago maybe we just didn't upgrade pandas for whatever reason.
As mentioned elsewhere in this thread, it's opt-in to avoid breaking existing behavior. But given that ingestion points are easy to identify, it's pretty straightforward to turn on (especially if you have a schema for your inputs): https://pandas.pydata.org/pandas-docs/stable/user_guide/inte...
I saw in implementation (CSV parser in Julia) were the sentinel value was randomly assigned at read time (if a value in the input was equal to the sentinel value, change randomly).after parsing, the sentinel value would be converted to the appropriate data type (Julia Missing)
That makes sense and thanks for the info and the link. It will be very useful going forward.