Guidance on handling unicode isolation with URLs in plaintext messages

[SethFalco]

Depending on the answer, it may be worth converting this to an issue as a request for documentation.

I'm using Project Fluent in a very small project, I can even inline my whole messages file here:

live =
  {$username} just went live! {$emoticon}
  {$url}

offline =
  {$username} went offline. {$emoticon}

update =
  Update: {$message}

— gitlab.com/SethFalco/matrix-stream-alerts/-/blob/main/src/i18n/locales/en.ftl

This is a plaintext application using fluent.js. I have Unicode Isolation enabled. This leads to a problem as it modifies URLs.

The message I should have a clickable link like this:

TheJoeStone⁩ just went live! ⁨⸜(｡˃ ᵕ ˂ )⸝♡⁩
⁨https://www.twitch.tv/thejoestone

Whereas if you click the link in the actual message, you can see the problem:

TheJoeStone⁩ just went live! ⁨⸜(｡˃ ᵕ ˂ )⸝♡⁩
https://www.twitch.tv/thejoestone⁩

You'll find GitHub renders them the same. Firefox shows the same URL in the bottom of the browser when hovered too! But when pressed, the bottom example includes the Unicode Isolation characters at the end of the URL. This breaks the URL, making it take the user to a 404.

I can confirm that in the following applications, the top link is fine, but the bottom link is broken:

• Firefox
• Element (Matrix client)
• Fractal (Matrix client)

I believe I can workaround this issue without disabling automatic Unicode Isolation, however all fixes I have in mind are more like hacks. For example:

• I could add a (whitespace) to the end of the URL.
• I could add a ? to the end of the URL, which would push the Unicode Isolation to be treated like a query parameter.
• I did try to use a ​ (zero-width whitespace), but unfortunately this just becomes part of the URL as well, so that won't work.

Does anyone on the team have any insight on how they would go about this?

Thank you for your time!

[alerque]

I don't know how the API works in the JavaScript implementation, but in the Rust crates you can disable this feature by turning off isolation of placeholders after spinning up your bundle:

let mut bundle = FluentBundle::new(...);
bundle.set_use_isolating(false);

I assume there is something similar in the JS libraries. Hopefully this gives you something to go on searching for what you need.

[SethFalco]

The JS API does have the same option! However, I was hoping there was a way to resolve this issue for a single string without disabling automatic Unicode isolation for every message in the bundle.

I like the feature, it just seems to have a flaw that I was hoping could be discussed before I resort to disabling it.

[alerque]

Isn't the actual flaw here in the Markdown parsers taking the plain text URL and making the isolation characters be part of the URL?

Also consider that given your use case, it might be entirely correct behavior to turn this feature off. The placeholders in your strings are already white-space separated and you have no apparent use case for this feature.

Another place you might expect to just disable this feature globally is when outputting anything for consumption in a terminal/console.

[SethFalco]

Isn't the actual flaw here in the Markdown parsers taking the plain text URL and making the isolation characters be part of the URL?

I was thinking that at first, but when I realized every User-Agent/client, including Firefox, Chromium, etc. suffers from the same problem with these links, so I took to the library. Though this probably doesn't impact HTML because it can use the <bdi> tag, but my output is plaintext for an instant messaging application.

The placeholders in your strings are already white-space separated and you have no apparent use case for this feature.

Hmm, the docs lead me to believe that my use-case is exactly why I would need it.

See: https://github.com/projectfluent/fluent.js/wiki/Unicode-Isolation#what-does-it-look-like

In that demonstration, a URL ending with a trailing slash isn't shown in the right position despite being at the end of the element, which Unicode isolation resolves.

[alerque]

I was thinking that at first, but when I realized every User-Agent/client, including Firefox, Chromium, etc. suffers from the same problem with these links

Is it really the browsers having trouble with this or are the browsers being handed pre-encoded HTML tags where the plain text has been automatically converted to something and passed to the browser? I'm still inclined to blame the plaintext/markdown -> HTML parser in this case. I don't see the browser being involved in the "make this a link" stage at all.

Hmm, the docs lead me to believe that my use-case is exactly why I would need it.

My bad, you are correct there. I was thinking about a totally different use case.

[SethFalco]

I don't see the browser being involved in the "make this a link" stage at all.

Ahh, shoot! You're completely correct. 🤦

I'm so used to my URIs being converted to links without explicitly writing the HTML/Markdown syntax, I forgot it's implicitly still being parsed under the hood.

I'll concede and treat this as a Markdown parser issue. I'll raise this with whatever library the clients are using instead. I appreciate your patience as well! 🤛

Edit:

• fix: unicode isolation should terminate links kivikakk/comrak#654