[trishume]

I specifically assumed a max tweet size based on the maximum number of UTF-8 bytes a tweet can contain (560), with a link to an analysis of that, and discussion of how you could optimize for the common case of tweets that contain way fewer UTF-8 bytes than that. Everything in my post assumes unicode.

[Tepix]

Did you consider URLs? They don't seem to count against tue size and can be very large indeed (like 4k)

[modeless]

URLs are shortened and the shortened size counts against the tweet size. The URL shortener could be a totally separate service that the core service never interacts with at all. Though I think in real twitter URLs may be partially expanded before tweets are sent to clients, so if you wanted to maintain that then the core service would need to interact with the URL shortener.

[Cyph0n]

Thanks for clarifying. I missed the max vs. average analysis because I was focused on the text. Still, as noted in the Rust code comment, the sample implementation doesn’t handle longer tweets.

[KingLancelot]

UTF-8 is 1-4 bytes per codepoint dude, not 1-2.

280 * 4 = 1120, not 560.

[Dylan16807]

Twitter doesn't allow 280 of any codepoint. The more complicated ones count double against your limit.

[KingLancelot]

Ah, so Twitter’s Unicode implementation is fucked.

[Dylan16807]

Nothing is wrong with it.

If it counted UTF-16 code units that would be dumb. It doesn't. The cutoff was deliberately set to keep the 140 character limit for CJK but increase it to 280 for the rest. And they did that based on observational data.

https://cdn.cms-twdigitalassets.com/content/dam/blog-twitter...

https://blog.twitter.com/en_us/topics/product/2017/Giving-yo...