[mooooooooooooo]

Would you happen to have any resources on how to treat floating point values?

I noticed some odd behaviour recently when using ruby to save to Postgres where the handoff between the two systems introduced imprecision in the saved value. Didn’t get to dig into it because it wasn’t a priority but it’s definitely an annoying unanswered question.

[opportune]

In addition to what others mentioned, to start from the basics I would try to learn how floating point values are implemented and how processors evaluate floating point expressions. For example the float is separated into two parts, it’s not able to efficiently represent many decimal numbers, etc.

A simple rule of thumb is to try to avoid using floating point values at all outside of contexts like scientific simulations. For basic situations, you can almost always use either a library (for big numbers, decimals, fractions, etc.) or express your logic with ints by using established patterns (make the unit of measurement smaller/bigger, explicitly round up or down, etc.). Any time you take something that is an int 99% of the time, convert it to a float for something, then convert it back to an int, you are doing something wrong.

[cbhl]

If you want a thorough understanding: you'll want to look up "numerical computation", "numerical methods" or "computational methods"; in particular computing error bounds and error propagation. Typically covered in bachelor's level university-level Math, CS, or Engineering departments.

If you just want to fix the odd behavior: adjust the schema so that you only work with whole numbers. For example, in the database schema, you can use DECIMAL instead of REAL/DOUBLE columns, or use two columns to specify the ratio of two integers (for example num/denom in frame rates in various video containers/codecs). In the application code: work only in whole numbers (e.g. cents, satoshis) instead of fractionals, using bigint or string types as applicable instead of e.g. double.

[gowld]

Or Fixed Point numbers .

[flopriore]

Don't Fixed Point numbers add more issues? Let's assume you have a machine that accepts 8-bits fixed-point numbers, with 6 bits for integers and 2 for decimals. For simplicity, let's use decimal digits. If you must represent the number 2.013, the resulting number would be +00002.01 So you cut the third decimal digit and wasted 4 bits with useless zeros. At the same time, if you represented the same number in 8 bits floating point, you would have all the digits.

[brianwawok]

Depends what you track. Fixed point is great for money. Floating point is not. Using an integer cents is easier than both.

[aDfbrtVt]

I have had this page bookmarked for years, hopefully you find it as useful as I do. It is a formal yet digestible treatment of floating point numbers.

https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.h...

[mfashby]

Not necessarily explaining how to do it right, but lots of examples of pitfalls

https://jvns.ca/blog/2023/01/13/examples-of-floating-point-p...