Not only were custom types invented decades before classes, but the languages with the strongest emphasis on type safety tend to either lack them or consider them to be unidiomatic.
Which languages would you say are really good for defining your own types? Would they be a good fit for the example I provided where a function needs to accept integers larger than zero? Do they also allow you to define your own operations on those types? That is the part where classes seem to be a good fit, methods are basically operations supported by a type.
For the integers larger than zero example, I'd say the earliest archetype for a good design (that I know of) is Ada, which lets you declare a type with a limited numeric range like so:
type MyType is range min .. max
There's already a built-in for positive integers, which is defined as
subtype Positive is Integer range 1 .. Integer'Last;
Note the subtype there. Ada recognizes that a positive integer is a type of integer, but not the other way around. And it enforces that in the type checking: You can pass any Positive into a function that accepts Integer, but you can't just pass an Integer into a function that accepts Positive. This happens even though they're not classes and this isn't OOP. Ada does have object-oriented constructs, but they are a later addition to the language. I have never used Ada professionally, but my understanding (based on book learning) is that it tends to be used conservatively.
It's similar in OCaml. Despite the O standing for "object-oriented", creating classes isn't necessarily considered idiomatic. The other tools in the chest tend to be conceptually simpler, and therefore to be preferred when they will get the job done.
"define your own operations" is a requirement I'm having a hard time making sense of. To me, that is just another way of saying, "define functions", which is a feature of every language I've used except for one really ancient dialect of BASIC.
You can go with more of a Haskell/Rust style approach, where you declare a type, and then define operations using either plain functions, or using something kinda similar an interface (typeclasses in Haskell, traits in Rust).