mirror of
https://git.sr.ht/~edwardloveall/scribe
synced 2025-12-10 12:41:23 +00:00
Fix markup errors caused by UTF-16/8 differences
Medium uses UTF-16 character offsets (likely to make it easier to parse in JavaScript) but Crystal uses UTF-8. Converting strings to UTF-16 to do offset calculation then back to UFT-8 fixes some markup bugs. --- Medium calculates markup offsets using UTF-16 encoding. Some characters like Emoji are count as multiple bytes which affects those offsets. For example in UTF-16 💸 is worth two bytes, but Crystal strings only count it as one. This is a problem for markup generation because it can offset the markup and even cause out-of-range errors. Take the following example: 💸💸! Imagine that `!` was bold but the emoji isn't. For Crystal, this starts at char index 2, end at char index 3. Medium's markup will say markup goes from character 4 to 5. In a 3 character string like this, trying to access character range 4...5 is an error because 5 is already out of bounds. My theory is that this is meant to be compatible with JavaScript's string length calculations, as Medium is primarily a platform built for the web: ```js "a".length // 1 "💸".length // 2 "👩❤️💋👩".length // 11 ``` To get these same numbers in Crystal strings must be converted to UTF-16: ```crystal "a".to_utf16.size # 1 "💸".to_utf16.size # 2 "👩❤️💋👩".to_utf16.size # 11 ``` The MarkupConverter now converts text into UFT-16 byte arrays on initialization. Once it's figured out the range of bytes needed for each piece of markup, it converts it back into UTF-8 strings.
This commit is contained in:
@@ -153,6 +153,21 @@ describe MarkupConverter do
|
||||
Strong.new(children: [Text.new(" bold")] of Child),
|
||||
])
|
||||
end
|
||||
|
||||
it "handles offsets from unicode text" do
|
||||
markup = PostResponse::Markup.new(
|
||||
type: PostResponse::MarkupType::STRONG,
|
||||
start: 5,
|
||||
end: 6
|
||||
)
|
||||
|
||||
result = MarkupConverter.convert(text: "💸💸 <", markups: [markup])
|
||||
|
||||
result.should eq([
|
||||
Text.new("💸💸 "),
|
||||
Strong.new(children: [Text.new("<")] of Child),
|
||||
])
|
||||
end
|
||||
end
|
||||
|
||||
describe "#wrap_in_markups" do
|
||||
|
||||
Reference in New Issue
Block a user