[hot_gril]

proto3 added support for optional primitives sorta recently. I've always been happy without them personally, but it was enough of a shock for people used to proto2 that it made sense to add them back.

[sebastos]

Just out of curiosity, what domain were you working in where "0.0" and "no opinion" were _always_ the same thing? The lack of optionals has infuriated me for years and I just can't form a mental model of how anybody ever found this acceptable to work with.

[jakjak123]

Like nearly every time, empty string and 0 for integers can be treated the same as "no value" if you think about it. Are you sending data or sending opinions? Usually to force a choice, you would make a enum or a one-of where the zero-value means the client has forgotten to set it and it can be modelled as a api error. Whether the value was actually on the wire or not is not really that important.

[kortex]

0 as a default for "no int" is tolerable, 0.0 as a default for "no float" is an absolute nightmare in any domain remotely related to math, machine learning, or data science.

We dealt with a bug that for weeks was silently corrupting the results of trials pitting the performance of various algos against each other. Because a valid response was "no reply/opt out", combined with a bug in processing the "opt out" enum, also combined with a bug in score aggregation, functions were treated like they replied "0.0" instead of "confidence = None".

It really should have defaulted NaN for missing floats.

[hot_gril]

What about floats made this a problem? We're treating 0.0 specially in some places.

[jakjak123]

I think your anecdote is rather weak with regards to the way protobuf works, but to entertain, why would a confidence of 0.0 be so different from None? 0.0 sounds very close to None for most numerical purposes if you ask me.

Wait, are you using Python?

[sebastos]

message LatLon {

    double lat = 1; // Geodetic latitude, decimal degrees

    double lon = 2; // Geodetic longitude, decimal degrees

}

"Hmm, lat = 0, I guess they didn't fill out the message. I'll thrown an exception and handle it as an api error"

[Later, somewhere near the equator]

"?!?!??!"

------------------

"Ok, we learned our lesson from what happened to our customers in Sao Tome and Principe: 0.0 is a perfectly valid latitude to have. No more testing for 0, we'll just trust the value we parse.

[Later, in Norway, when a bug causes a client to skip filling out the LatLon message]

"Why is it flying to Africa?!?!"

------------------

Ok, after the backlash from our equatorial customers and the disaster in Norway, we've learned our lesson. We will now use a new message that lets us handle 0's, but checks that they really meant it:

message LatLonEnforced {

    optional double lat = 1;

    optional double lon = 2;

}

[At some third party dev's desk] "Oh, latitude is optional - I'll just supply longitude"

[...]

"It's throwing exceptions? But your schema says it's optional!"

------------------

Ok, it took some blood sweat and tears but we finally got this message definition licked:

message LatLonEnforced {

    optional double lat = 1; // REQUIRED. Geodetic latitude, decimal degrees

    optional double lon = 2; // REQUIRED. Geodetic longitude, decimal degrees

}

[Later, in an MR from a new hire] "Correct LatLon doc strings to reflect optional semantics"

[hot_gril]

If both lat and lon are required, you don't need to throw an exception for lat=0. If you want lat=null lon=0.0 to mean something like "latitude is unknown but longitude is known to be 0.0," yeah you need optional or wrapped primitives.

Edit: If a client doesn't fill out the LatLng message, that's different from lat and/or lon being null or 0. The whole LatLng message itself will be null. Proto3 always supported that too. But it's usually overkill to check the message being null, unless you added it later and need to specially deal with clients predating that change. If the client just has a bug preventing it from filling out LatLng, that's the client's problem.

The confusing part here is that even if the LatLng message is null, LatLng.lat will return 0 in some proto libraries, depending on the language. You have to specifically check if the message is null if you care. But there are enough cases where you have tons of nested protos and the 0-default behavior is actually way more convenient.

[jakjak123]

In the case for Lat/Lon, I guess that 0.0 could have a meaning, though it is very unlikely someone is exactly at lat/lon 0.0. An alternative is to translate to the XY coordinate system, though that is not a perfect solution either.

If you really feel like expressing that LatLon as possibly null, it should rather be:

message User {

  optional LatLon position = 1;

}

[xarope]

working with gRPC allowed me to understand how go(lang)'s use of things like sql.nullstring works (pseudo'ish code}:

  astring = sql.nullstring{isNull=true, value=""}

  if astring.isNull {don't do anything}
  else {process astring.value}

So similarly, gRPC has a method called HasField, so:

  if packet.HasField(astring) {then process packet.astring}

Is it wordy? Yes. Is it elegant? Sadly, No. But does it work? Yes.

[kortex]

Yes it was python but that has nothing to do with it. Same would happen in go, rust, R, or matlab.

Correct answers: 1.0, 0.0, 1.0

Confidence from algo: 1.0, 0.0, n/a

Confidence on the wire: 1.0, 0.0, 0.0

Score after bug: 66%

Score as it ought to be scored: 100%

It was enough to make several algorithms which were very selective in the data they would attempt to analyze (think jpg vs png images) went from "kinda crap" in the rankings to "really good"

[jakjak123]

Well, only in python is that N/A value also a float. In protobuf, go or Java for that matter, that data model must somehow be changed to communicate the difference.

If you had use 3 float values in Go or Java you would have had the same problem.

[hot_gril]

Yeah, I think it's best to first rethink of null as just, not 0 and not any other number. What that means depends on the context.

Tangent: I've seen an antipattern of using enums to describe the "type" of some polymorphic object instead of just using a oneof with type-specific fields. Which gets even worse if you later decide something fits multiple types.

[jakjak123]

I love oneofs in the language with good support, but they are woeful in Golang and the "official" grpc-web message types.

[hot_gril]

Not really one domain in particular, just internal services in general. In many cases, the field is intended to be required in the first place. If not, surprisingly 0 isn't a real answer a lot of the time, so it means none or default: any kind of numeric ID that starts at 1 (like ticket numbers), TTLs, error codes, enums (0 is always UNKNOWN). Similarly with empty strings.

I have a hard time thinking of places I really need both 0 and none. The only example that comes to mind is building room numbers, in some room search message where we wanted null to mean wildcard. In those cases, it's not hard to wrap it in a message. Probably the best argument for optionals is when you have a lot of boolean request options where null means default, but even then I prefer instead naming them such that the defaults are all false, since that's clearer anyway.

It did take some getting used to and caused some downstream changes to how we design APIs. I think we're better for it because the whole null vs 0 thing can be tedious and error-prone, but it's very opinionated.

[kortex]

Any time you have a scalar/measurement number, basically any value with physical units, counts, percentages, anything which could be in the denominator of a ratio, those are all strong indicators of a "semantic zero" and you really want to tell the difference between None and 0. They are usually floats, but could be ints (maybe you have number_of_widgets_online, 0 means 0 units, None means "idk".)

[hot_gril]

What's the difference between none inches and 0 inches? Might need a concrete example. We deal with space a fair amount and haven't needed many optionals there.

[TwentyPosts]

They just gave a concrete example: The difference between "we don't know" and "we know that it's zero".

Here's another fun one: I've seen APIs where "0.0" was treated as "no value, so take the default value". The default value happened to be 0.2.

[jakjak123]

I had the same experience. It was a bit awkward for 6 months, but down the line we learned to design better apis, and dealing with nullable values are tedious at best. Its just easier knowing that a string or integer will _never_ cause a nullpointer.

[sebastos]

Huh, yeah I see. I guess I work more on the robotics side where messages often contain physical or geometric quantities, and that colors my thinking a bit. So "distance to that thing = 0" is a very possible situation, and yet you also want to allow it to say "I didn't measure distance to that thing". And those are very distinct concepts you never want to conflate.

[hot_gril]

I can see that. Or the rare situations where I needed 0 vs null, if for some reason that situation was multiplied times 100, I'd start wanting an optional keyword.

[jakjak123]

Same, I rarely felt a need for this distinction.