validateUtf8Chunk :: ByteString -> (Int, Maybe Utf8State)
Validate a ByteString as UTF-8-encoded text. To be continued using validateUtf8More.
See also validateUtf8More for details on the result of this function.
'validateUtf8Chunk' = 'validateUtf8More' 'startUtf8State'
Properties
Given:
'validateUtf8Chunk' chunk = (n, ms)
'validateUtf8Chunk' ('Data.ByteString.take' n chunk) = (n, Just 'startUtf8State')
validateUtf8More :: Utf8State -> ByteString -> (Int, Maybe Utf8State)
Validate another ByteString chunk in an ongoing stream of UTF-8-encoded text.
Returns a pair:
The first component n is the end position, relative to the current
chunk, of the longest prefix of the accumulated bytestring which is valid UTF-8.
n may be negative: that happens when an incomplete code point started in
a previous chunk and is not completed by the current chunk (either
that code point is still incomplete, or it is broken by an invalid byte).
The second component ms indicates the following:
if ms = Nothing, the remainder of the chunk contains an invalid byte,
within four bytes from position n;
if ms = Just s', you can carry on validating another chunk
by calling validateUtf8More with the new state s'.
Properties
Given:
'validateUtf8More' s chunk = (n, ms)
ms = Nothing
==> 'validateUtf8More' s (chunk '<>' more) = (n, Nothing)
ms = Just s'
==> 'validateUtf8More' s (chunk '<>' more) = 'Data.Bifunctor.first' ('Data.ByteString.length' chunk '+') ('validateUtf8More' s' more)
decodeUtf8Chunk :: ByteString -> (StrictTextBuilder, ByteString, Maybe Utf8State)
Decode a chunk of UTF-8 text. To be continued with decodeUtf8More.
See decodeUtf8More for details on the result.
Properties
'decodeUtf8Chunk' = 'decodeUtf8More' 'startUtf8State'
Given:
'decodeUtf8Chunk' chunk = (builder, rest, ms)
builder is a prefix and rest is a suffix of chunk.
'Data.Text.Encoding.encodeUtf8' ('Data.Text.Encoding.strictBuilderToText' builder) '<>' rest = chunk
decodeUtf8More :: Utf8State -> ByteString -> (StrictTextBuilder, ByteString, Maybe Utf8State)
Decode another chunk in an ongoing UTF-8 stream.
Returns a triple:
A StrictBuilder for the decoded chunk of text. You can accumulate
chunks with (<>) or output them with toText.
The undecoded remainder of the given chunk, for diagnosing errors
and resuming (presumably after skipping some bytes).
Just the new state, or Nothing if an invalid byte was encountered
(it will be within the first 4 bytes of the undecoded remainder).
Properties
Given:
(pre, suf, ms) = 'decodeUtf8More' s chunk
If the output pre is nonempty (alternatively, if length chunk > length suf)
s2b pre \`'Data.ByteString.append'\` suf = p2b s \`'Data.ByteString.append'\` chunk
where
s2b = 'Data.Text.Encoding.encodeUtf8' . 'Data.Text.Encoding.toText'
p2b = 'Data.Text.Internal.Encoding.partUtf8ToByteString'
If the output pre is empty (alternatively, if length chunk = length suf)
suf = chunk
Decoding chunks separately is equivalent to decoding their concatenation.
Given:
(pre1, suf1, Just s1) = 'decodeUtf8More' s chunk1
(pre2, suf2, ms2) = 'decodeUtf8More' s1 chunk2
(pre3, suf3, ms3) = 'decodeUtf8More' s (chunk1 \`B.append\` chunk2)
we have:
s2b (pre1 '<>' pre2) = s2b pre3
ms2 = ms3
decodeUtf8With1 :: OnDecodeError -> String -> ByteString -> Text
Helper for decodeUtf8With.
decodeUtf8With2 :: OnDecodeError -> String -> Utf8State -> ByteString -> (StrictTextBuilder, ByteString, Utf8State)
Helper for decodeUtf8With,
streamDecodeUtf8With, and lazy
decodeUtf8With,
which use an OnDecodeError to process bad bytes.
See decodeUtf8Chunk for a more flexible alternative.
State of decoding a ByteString in UTF-8.
Enables incremental decoding (validateUtf8Chunk, validateUtf8More,
decodeUtf8Chunk, decodeUtf8More).
A delayed representation of strict Text.
Use StrictTextBuilder instead
A delayed representation of strict Text.
Use StrictBuilder to build Text.
Copy Text in a StrictBuilder
Internal
skipIncomplete :: OnDecodeError -> String -> Utf8State -> StrictTextBuilder
Call the error handler on each byte of the partial code point stored in
Utf8State and append the results.
Exported for use in lazy decodeUtf8With.
getPartialUtf8 :: Utf8State -> B.ByteString
Instances
4
instance Eq Utf8Stateinstance Show Utf8Stateinstance Eq PartialUtf8CodePointinstance Show PartialUtf8CodePoint
- License
- BSD-style (see LICENSE)