[bitwize]

Alexis King's "Parse, don't validate" is pretty much the final word on using type systems to deal with "messy real world data": https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...

tl;dr when used properly, static type systems are an enormous advantage when dealing with data from the real world because you can write a total function that accepts unstructured data like a string or byte stream and returns either a successful result with a parsed data structure, a partial result, or a value that indicates failure, without having to do a separate validation step at all -- and the type system will check that all your intermediate results are correct, type-wise.

[fiddlerwoaroof]

I’ve been using the techniques in that article for years in JavaScript, CL and Clojure. While static types are a notable part of it, the more important point is just learning to design your systems to turn incoming data to domain objects as soon as possible.

[fiddlerwoaroof]

There are runtime analogs for most of the modeling techniques people use in statically typed languages.

[blacktriangle]

You're not the first, nor fourth for that matter, person to respond to dynamic typing advocation with that blog post, and it's an interesting post but it misses the whole point. The problem is not enforcing rules on data coming in and out of the system. The problem is that I have perfectly valid collections of data that I want to shove through an information processing pipeline while preserving much of the original data and static typing systems make this very powerful and legitimate task a nightmare.

[roca]

Not a nightmare at all. For example, if you're doing JSON processing in Rust, serde_json gives you great tools for this. The serde_json::Value type can represent any JSON value. You parse JSON to specific typed structures when you want to parse-and-validate and detect errors (using #[derive(Serialize, Deserialize)] to autogenerate the code), but those structures can include islands of arbitrary serde_json::Values. You can also parse JSON objects into structs that contain a set of typed fields with all other "unknown" fields collected into a dynamic structure, so those extra fields are reserialized when necessary --- see https://serde.rs/attr-flatten.html for an example.

[rewma]

> The problem is that I have perfectly valid collections of data (...) and static typing systems make this very powerful and legitimate task a nightmare.

What leads you to believe that static typing turns a task that essencially boils down to input validation "a nightmare"?

From my perspective, with static typing that task is a treat and all headaches that come with dynamic typing simply vanish.

Take for example Typescript. Between type assertion functions, type guards, optional types and union types, inferring types from any object is a trivial task with clean code enforced by the compiler itself.

[brundolf]

Presumably the GP's data is external and therefore not checkable or inferrable by typescript. This makes the task less ideal, but still perfectly doable via validation code or highly agnostic typing

[rewma]

> Presumably the GP's data is external and therefore not checkable or inferrable by typescript.

There is no such thing as external data that is not checkable or inferable by typescript. That's what type assertion functions and type guards are for.

With typescript, you can take in an instance of type any, pass it to a type assertion function or a type guard, and depending on the outcome either narrow it to a specific type or throw an error.

[brundolf]

You said:

> inferring types from any object is a trivial task

This is true for values defined in code, but TypeScript cannot directly see data that comes in from eg. an API, and so can't infer types from it. You can give the data types yourself, and you can even give it types based on validation logic that happens at runtime, and I think this is usually worth doing and not a huge burden if you use a library. But it's disingenuous to suggest that it's free.

The closest thing to "free" would be blindly asserting the data's type, which is very dangerous and IMO usually worse than not having static types at all, because it gives you a false sense of security:

  const someApiData: any = { foo: 'bar' }

  function doSomethingWith(x: ApiData) {
    return x.bar + 12
  }

  type ApiData = {
    foo: string,
    bar: number
  }

  // no typescript errors!
  doSomethingWith(someApiData as ApiData)

The better approach is to use something like io-ts to safely "parse" the data into a type at runtime. But, again, this is not without overhead.

[rewma]

> This is true for values defined in code, but TypeScript cannot directly see data that comes in from eg. an API, and so can't infer types from it.

No, that's not right at all. TypeScript allows you to determine the exact type of an object in any code path through type assertions and type guards.

With TypeScript you can get an any instance from wherever, apply your checks, and from thereon either throw an error or narrow your any object into whatever type you're interested in.

I really do not know what leads you to believe that TypeScrip cannot handle static typing or input validation.

[brundolf]

They don't, though.

For example: a technique I've used to work with arbitrary, unknown JSON values, is to type them as a union of primitives + arrays of json values + objects of json values. And then I can pick these values apart in a way that's totally safe while making no dangerous assumptions about their contents.

Of course this opens the door for lots of potential mistakes (though runtime errors at least are impossible), but it's 100% compatible with any statically-typed language that has unions.