[kevingadd]

Realistically speaking, you can't "switch" AssemblyScript to UTF-8 unless you also decide it only can run in UTF-8 host environments (i.e. not web browsers). Right now it uses UTF-16, which is what the web uses. If you move it over to UTF-8 now every operation that passes strings to web APIs has to perform encoding and decoding, and you end up with a bunch of new performance and correctness issues. It's a very complex migration.

P.S. the web will never switch to UTF-8. It would break too many web pages. Most browser vendors won't even accept breaking 0.1% of web pages, unless they're doing it to show you more ads (i.e. Chrome).

[zodiakzz]

JavaScript is called the entire "web" now? HTML and CSS work with UTF-8 just fine and the majority of the WWW uses UTF-8 to serve them.

[kevingadd]

The canonical representation of DOM content is DOMString (https://developer.mozilla.org/en-US/docs/Web/API/DOMString), which is not UTF-8. Your HTML being encoded in UTF-8 is irrelevant, it gets decoded when it's loaded into whatever the canonical representation is. Your HTML could be in Shift-JIS or ASCII or whatever and not UTF-8, same difference.

[trusktr]

This is exactly right. UTF-8 is the transmission format that your HTML gets sent in, but it is not the format of strings in JavaScript at runtime.

The problem being discussed is about runtime interoperability between JS (with WTF-16 string format) and WebAssembly.

[paulddraper]

I think you know the poster is taking about programming languages, where the bulk of complexity lies.

[trusktr]

Yes, but WebAssembly operates at the boundary with JS, and that is not UTF-8. JS uses WTF-16 at runtime, and if WebAssembly did too then this would make interop between Wasm and JS a first-class feature with maximal performance and without security and data integrity issues.

[paulddraper]

I don't understand. Can't you just abstract this?

Python has several internal string representations to reduce conversions.

Really "UTF-16 string" and "UTF-8 string" are code smells. Applications should be using character/code point sequences or byte sequences.

Using code unit sequences is....bizzare. (Yes I know Java, C#, JS has chosen to do that, but a new language has an opportunity to improve.)

[kevingadd]

You can abstract it, but AssemblyScript did not. So it's not a trivial change, it's a complex migration. Similarly, you can use UTF-8 in Java and C#, but you can't just "switch" them over to the encoding directly, it has to be exposed via new types/etc.

[trusktr]

Note that AssemblyScript rides on TypeScript language syntax. How would

``` let foo: string = "whatever" ```

be able to work in any similar sense as TS/JS if? How can that map to multiple string types? The idea is both AS and TS use the same syntax for strings, and are compatible across boundaries (TS for JS side, AS for Wasm side). Having multiple string types is possible, but this would greatly reduce developer ergonomics.

[paulddraper]

First you have to decide whether AssemblyScript is a language, or a compiler for an existing language.

If it's a language, then it gets to decide what ```let foo: string = "whatever"``` means.

If it's a compiler for an existing language, then semantics have already been decided, and the compiler has to implement it.

But none of the precludes abstraction to reduce data type conversions.

[trusktr]

AssemblyScript is a compiler that aims to compile TypeScript code (with slight differences to be able to make sense in Wasm, though trying to minimize those differences) into Wasm. To remain compatible with TypeScript (which is AssemblyScript's goal) and be an optimal language for Wasm that will communicate with TS/JS on the other side, the type `string` would need to be in the same format to avoid any perf hit or data loss while passing thing if that type across the Wasm-JS boundary.

I'm not sure what type of compiler to label that as, but that's the goal.

[paulddraper]

> I'm not sure what type of compiler to label that as, but that's the goal.

A compiler. From TypeScript to web assembly.

Certainly it has its work cut out for it to have TS semantics (that is, JS semantics) and be optimal for target web assembly.

I can't think of any scripting languages that are optimal for targeting low-level runtimes.

I wish AssemblyScript the best; seems like a hard problem.

[paulddraper]

If I understand correctly, AssemblyScript has been building ahead of specs on Interface Types?

I sympathize with pain, but the bleeding edge of tech does...bleed.