Community
Modules 15

Data.ByteString.Builder.Internal

unstable, private

*Warning:* this module is internal. If you find that you need it then please contact the maintainers and explain what you are trying to do and discuss what you would need in the public API. It is important that you do this as the module may not be exposed at all in future releases.

Core types and functions for the Builder monoid and its generalization, the Put monad.

The design of the Builder monoid is optimized such that

  1. buffers of arbitrary size can be filled as efficiently as possible and

  1. sequencing of Builders is as cheap as possible.

We achieve (1) by completely handing over control over writing to the buffer to the BuildStep implementing the Builder. This BuildStep is just told the start and the end of the buffer (represented as a BufferRange). Then, the BuildStep can write to as big a prefix of this BufferRange in any way it desires. If the BuildStep is done, the BufferRange is full, or a long sequence of bytes should be inserted directly, then the BuildStep signals this to its caller using a BuildSignal.

We achieve (2) by requiring that every Builder is implemented by a BuildStep that takes a continuation BuildStep, which it calls with the updated BufferRange after it is done. Therefore, only two pointers have to be passed in a function call to implement concatenation of Builders. Moreover, many Builders are completely inlined, which enables the compiler to sequence them without a function call and with no boxing at all.

This design gives the implementation of a Builder full access to the IO monad. Therefore, utmost care has to be taken to not overwrite anything outside the given BufferRanges. Moreover, further care has to be taken to ensure that Builders and Puts are referentially transparent. See the comments of the builder and put functions for further information. Note that there are no safety belts at all, when implementing a Builder using an IO action: you are writing code that might enable the next buffer-overflow attack on a Haskell server!

1 generation diagnostics
  • unattached documentation comment at 638:9

Buffer management

data Buffer

A Buffer together with the BufferRange of free bytes. The filled space starts at offset 0 and ends at the first free byte.

Buffer :: (ForeignPtr Word8) -> BufferRange -> Buffer

data BufferRange

A range of bytes in a buffer represented by the pointer to the first byte of the range and the pointer to the first byte after the range.

BufferRange :: (Ptr Word8) -> (Ptr Word8) -> BufferRange

newBuffer :: Int -> IO Buffer

Allocate a new buffer of the given size.

bufferSize :: Buffer -> Int

Combined size of the filled and free space in the buffer.

byteStringFromBuffer :: Buffer -> S.StrictByteString

Convert the filled part of a Buffer to a StrictByteString.

data ChunkIOStream a

A stream of chunks that are constructed in the IO monad.

This datatype serves as the common interface for the buffer-by-buffer execution of a BuildStep by buildStepToCIOS. Typical users of this interface are ciosToLazyByteString or iteratee-style libraries like enumerator.

Finished :: Buffer -> a -> ChunkIOStream a

a

The partially filled last buffer together with the result.

Yield1 :: S.StrictByteString -> (IO (ChunkIOStream a)) -> ChunkIOStream a

(IO (ChunkIOStream a))

Yield a non-empty StrictByteString.

buildStepToCIOS :: forall a. AllocationStrategy -> BuildStep a -> IO (ChunkIOStream a)

Convert a BuildStep to a ChunkIOStream stream by executing it on Buffers allocated according to the given AllocationStrategy.

BuildStep to execute

AllocationStrategy

Buffer allocation strategy to use

ciosUnitToLazyByteString :: AllocationStrategy -> L.LazyByteString -> ChunkIOStream () -> L.LazyByteString

Convert a ChunkIOStream () to a LazyByteString using unsafeDupablePerformIO.

ciosToLazyByteString :: AllocationStrategy -> (a -> (b, L.LazyByteString)) -> ChunkIOStream a -> (b, L.LazyByteString)

Convert a ChunkIOStream to a lazy tuple of the result and the written LazyByteString using unsafeDupablePerformIO.

Build signals and steps

data BuildSignal a

BuildSignals abstract signals to the caller of a BuildStep. There are three signals: done, bufferFull, or 'insertChunks signals

type BuildStep a = BufferRange -> IO (BuildSignal a)

BuildSteps may be called *multiple times* and they must not rise an async. exception.

finalBuildStep :: BuildStep ()

The final build step that returns the done signal.

done :: Ptr Word8 -> a -> BuildSignal a

Signal that the current BuildStep is done and has computed a value.

Next free byte in current BufferRange

a

Computed value

bufferFull :: Int -> Ptr Word8 -> BuildStep a -> BuildSignal a

Signal that the current buffer is full.

Next free byte in current BufferRange.

BuildStep to run on the next BufferRange. This BuildStep may assume that it is called with a BufferRange of at least the required minimal size; i.e., the caller of this BuildStep must guarantee this.

Int

Minimal size of next BufferRange.

insertChunk :: Ptr Word8 -> S.StrictByteString -> BuildStep a -> BuildSignal a

Signal that a StrictByteString chunk should be inserted directly.

Next free byte in current BufferRange

BuildStep to run on next BufferRange

S.StrictByteString

Chunk to insert.

fillWithBuildStep :: BuildStep a -> (Ptr Word8 -> a -> IO b) -> (Ptr Word8 -> Int -> BuildStep a -> IO b) -> (Ptr Word8 -> S.StrictByteString -> BuildStep a -> IO b) -> BufferRange -> IO b

Fill a BufferRange using a BuildStep.

Build step to use for filling the BufferRange.

Value computed while filling this BufferRange.

(Ptr Word8 -> a -> IO b)

Handling the done signal

(Ptr Word8 -> Int -> BuildStep a -> IO b)

Handling the bufferFull signal

(Ptr Word8 -> S.StrictByteString -> BuildStep a -> IO b)

Handling the insertChunk signal

BufferRange

Buffer range to fill.

The Builder monoid

newtype Builder

Builders denote sequences of bytes. They are Monoids where mempty is the zero-length sequence and mappend is concatenation, which runs in O(1).

builder :: (forall r. BuildStep r -> BuildStep r) -> Builder

Construct a Builder. In contrast to BuildSteps, Builders are referentially transparent.

(forall r. BuildStep r -> BuildStep r)

A function that fills a BufferRange, calls the continuation with the updated BufferRange once its done, and signals its caller how to proceed using done, bufferFull, or insertChunk.

This function must be referentially transparent; i.e., calling it multiple times with equally sized BufferRanges must result in the same sequence of bytes being written. If you need mutable state, then you must allocate it anew upon each call of this function. Moreover, this function must call the continuation once its done. Otherwise, concatenation of Builders does not work. Finally, this function must write to all bytes that it claims it has written. Otherwise, the resulting Builder is not guaranteed to be referentially transparent and sensitive data might leak.

runBuilder :: Builder -> BuildStep ()

Run a Builder with the finalBuildStep.

BuildStep that writes the byte stream of this Builder and signals done upon completion.

Builder

Builder to run

runBuilderWith :: Builder -> BuildStep a -> BuildStep a

Run a Builder.

Continuation BuildStep

Builder

Builder to run

Primitive combinators

empty :: Builder

The Builder denoting a zero-length sequence of bytes. This function is only exported for use in rewriting rules. Use mempty otherwise.

append :: Builder -> Builder -> Builder

Concatenate two Builders. This function is only exported for use in rewriting rules. Use mappend otherwise.

flush :: Builder

Flush the current buffer. This introduces a chunk boundary.

ensureFree :: Int -> Builder

ensureFree n ensures that there are at least n free bytes for the following Builder.

byteStringCopy :: S.StrictByteString -> Builder

Construct a Builder that copies the StrictByteString.

Use this function to create Builders from smallish (<= 4kb) StrictByteStrings or if you need to guarantee that the StrictByteString is not shared with the chunks generated by the Builder.

byteStringInsert :: S.StrictByteString -> Builder

Construct a Builder that always inserts the StrictByteString directly as a chunk.

This implies flushing the output buffer, even if it contains just a single byte. You should therefore use byteStringInsert only for large (> 8kb) StrictByteStrings. Otherwise, the generated chunks are too fragmented to be processed efficiently afterwards.

byteStringThreshold :: Int -> S.StrictByteString -> Builder

Construct a Builder that copies the StrictByteStrings, if it is smaller than the treshold, and inserts it directly otherwise.

For example, byteStringThreshold 1024 copies StrictByteStrings whose size is less or equal to 1kb, and inserts them directly otherwise. This implies that the average chunk-size of the generated LazyByteString may be as low as 513 bytes, as there could always be just a single byte between the directly inserted 1025 byte, StrictByteStrings.

lazyByteStringCopy :: L.LazyByteString -> Builder

Construct a Builder that copies the LazyByteString.

lazyByteStringInsert :: L.LazyByteString -> Builder

Construct a Builder that inserts all chunks of the LazyByteString directly.

lazyByteStringThreshold :: Int -> L.LazyByteString -> Builder

Construct a Builder that uses the thresholding strategy of byteStringThreshold for each chunk of the LazyByteString.

shortByteString :: Sh.ShortByteString -> Builder

Construct a Builder that copies the ShortByteString.

maximalCopySize :: Int

The maximal size of a StrictByteString that is copied. 2 * smallChunkSize to guarantee that on average a chunk is of smallChunkSize.

byteString :: S.StrictByteString -> Builder

Create a Builder denoting the same sequence of bytes as a StrictByteString. The Builder inserts large StrictByteStrings directly, but copies small ones to ensure that the generated chunks are large on average.

lazyByteString :: L.LazyByteString -> Builder

Create a Builder denoting the same sequence of bytes as a lazy LazyByteString. The Builder inserts large chunks of the LazyByteString directly, but copies small ones to ensure that the generated chunks are large on average.

Execution

toLazyByteString :: Builder -> L.LazyByteString

Execute a Builder and return the generated chunks as a LazyByteString. The work is performed lazy, i.e., only when a chunk of the LazyByteString is forced.

toLazyByteStringWith :: AllocationStrategy -> L.LazyByteString -> Builder -> L.LazyByteString

Heavy inlining. Execute a Builder with custom execution parameters.

This function is inlined despite its heavy code-size to allow fusing with the allocation strategy. For example, the default Builder execution function toLazyByteString is defined as follows.

{-\# NOINLINE toLazyByteString \#-}
toLazyByteString =
  toLazyByteStringWith ('safeStrategy' 'L.smallChunkSize' 'L.defaultChunkSize') L.Empty

where L.Empty is the zero-length LazyByteString.

In most cases, the parameters used by toLazyByteString give good performance. A sub-performing case of toLazyByteString is executing short (<128 bytes) Builders. In this case, the allocation overhead for the first 4kb buffer and the trimming cost dominate the cost of executing the Builder. You can avoid this problem using

toLazyByteStringWith (safeStrategy 128 smallChunkSize) L.Empty

This reduces the allocation and trimming overhead, as all generated LazyByteStrings fit into the first buffer and there is no trimming required, if more than 64 bytes and less than 128 bytes are written.

AllocationStrategy

Buffer allocation strategy to use

L.LazyByteString

LazyByteString to use as the tail of the generated lazy LazyByteString

Builder

Builder to execute

L.LazyByteString

Resulting LazyByteString

safeStrategy :: Int -> Int -> AllocationStrategy

Use this strategy for generating LazyByteStrings whose chunks are likely to survive one garbage collection. This strategy trims buffers that are filled less than half in order to avoid spilling too much memory.

Int

Size of first buffer

Int

Size of successive buffers

AllocationStrategy

An allocation strategy that guarantees that at least half of the allocated memory is used for live data

untrimmedStrategy :: Int -> Int -> AllocationStrategy

Use this strategy for generating LazyByteStrings whose chunks are discarded right after they are generated. For example, if you just generate them to write them to a network socket.

Int

Size of the first buffer

Int

Size of successive buffers

AllocationStrategy

An allocation strategy that does not trim any of the filled buffers before converting it to a chunk

customStrategy :: (Maybe (Buffer, Int) -> IO Buffer) -> Int -> (Int -> Int -> Bool) -> AllocationStrategy

Create a custom allocation strategy. See the code for safeStrategy and untrimmedStrategy for examples.

(Maybe (Buffer, Int) -> IO Buffer)

Buffer allocation function.

  • If Nothing is given, then a new first buffer should be allocated.

  • If Just (oldBuf, minSize) is given, then a buffer with minimal size minSize must be returned. The strategy may reuse oldBuf only if oldBuf is large enough and the consumer can guarantee that this will not result in a violation of referential transparency.

Warning: for multithreaded programs, it is generally unsafe to reuse buffers when using the consumers of Builder in this package. For example, if toLazyByteStringWith is called with an AllocationStrategy that reuses buffers, evaluating the result by multiple threads simultaneously may lead to corrupted output.

Int

Default buffer size.

(Int -> Int -> Bool)

A predicate trim used allocated returning True, if the buffer should be trimmed before it is returned.

smallChunkSize :: Int

The recommended chunk size. Currently set to 4k, less the memory management overhead

defaultChunkSize :: Int

The chunk size used for I/O. Currently set to 32k, less the memory management overhead

chunkOverhead :: Int

The memory management overhead. Currently this is tuned for GHC only.

The Put monad

newtype Put a

A Put action denotes a computation of a value that writes a stream of bytes as a side-effect. Puts are strict in their side-effect; i.e., the stream of bytes will always be written before the computed value is returned.

Puts are a generalization of Builders. The typical use case is the implementation of an encoding that might fail (e.g., an interface to the zlib compression library or the conversion from Base64 encoded data to 8-bit data). For a Builder, the only way to handle and report such a failure is ignore it or call error. In contrast, Put actions are expressive enough to allow reporting and handling such a failure in a pure fashion.

Put () actions are isomorphic to Builders. The functions putBuilder and fromPut convert between these two types. Where possible, you should use Builders, as sequencing them is slightly cheaper than sequencing Puts because they do not carry around a computed value.

put :: (forall r. (a -> BuildStep r) -> BuildStep r) -> Put a

Construct a Put action. In contrast to BuildSteps, Puts are referentially transparent in the sense that sequencing the same Put multiple times yields every time the same value with the same side-effect.

(forall r. (a -> BuildStep r) -> BuildStep r)

A function that fills a BufferRange, calls the continuation with the updated BufferRange and its computed value once its done, and signals its caller how to proceed using done, bufferFull, or insertChunk signals.

This function must be referentially transparent; i.e., calling it multiple times with equally sized BufferRanges must result in the same sequence of bytes being written and the same value being computed. If you need mutable state, then you must allocate it anew upon each call of this function. Moreover, this function must call the continuation once its done. Otherwise, monadic sequencing of Puts does not work. Finally, this function must write to all bytes that it claims it has written. Otherwise, the resulting Put is not guaranteed to be referentially transparent and sensitive data might leak.

runPut :: Put a -> BuildStep a

Run a Put.

Put to run

BuildStep that first writes the byte stream of this Put and then yields the computed value using the done signal.

Execution

putToLazyByteString :: Put a -> (a, L.LazyByteString)

Execute a Put and return the computed result and the bytes written during the computation as a LazyByteString.

This function is strict in the computed result and lazy in the writing of the bytes. For example, given

infinitePut = sequence_ (repeat (putBuilder (word8 1))) >> return 0

evaluating the expression

fst $ putToLazyByteString infinitePut

does not terminate, while evaluating the expression

L.head $ snd $ putToLazyByteString infinitePut

does terminate and yields the value 1 :: Word8.

An illustrative example for these strictness properties is the implementation of Base64 decoding (http://en.wikipedia.org/wiki/Base64).

type DecodingState = ...

decodeBase64 :: 'S.StrictByteString' -> DecodingState -> 'Put' (Maybe DecodingState)
decodeBase64 = ...

The above function takes a StrictByteString supposed to represent Base64 encoded data and the current decoding state. It writes the decoded bytes as the side-effect of the Put and returns the new decoding state, if the decoding of all data in the StrictByteString was successful. The checking if the StrictByteString represents Base64 encoded data and the actual decoding are fused. This makes the common case, where all data represents Base64 encoded data, more efficient. It also implies that all data must be decoded before the final decoding state can be returned. Puts are intended for implementing such fused checking and decoding/encoding, which is reflected in their strictness properties.

Put to execute

(a, L.LazyByteString)

Result and LazyByteString written as its side-effect

putToLazyByteStringWith :: AllocationStrategy -> (a -> (b, L.LazyByteString)) -> Put a -> (b, L.LazyByteString)

Execute a Put with a buffer-allocation strategy and a continuation. For example, putToLazyByteString is implemented as follows.

putToLazyByteString = 'putToLazyByteStringWith'
   ('safeStrategy' 'L.smallChunkSize' 'L.defaultChunkSize') (\x -> (x, L.empty))

Put to execute

AllocationStrategy

Buffer allocation strategy to use

(a -> (b, L.LazyByteString))

Continuation to use for computing the final result and the tail of its side-effect (the written bytes).

(b, L.LazyByteString)

Resulting LazyByteString

hPut :: forall a. Handle -> Put a -> IO a

Run a Put action redirecting the produced output to a Handle.

The output is buffered using the Handles associated buffer. If this buffer is too small to execute one step of the Put action, then it is replaced with a large enough buffer.

Conversion to and from Builders

putBuilder :: Builder -> Put ()

Run a Builder as a side-effect of a Put () action.

fromPut :: Put () -> Builder

Convert a Put () action to a Builder.

Instances

8
  • instance NFData BufferRange
  • instance NFData Buffer

    Like the NFData instance for StrictByteString, this does not force the ForeignPtrContents field of the underlying ForeignPtr.

  • instance Semigroup Builder
  • instance Monoid Builder
  • instance IsList Builder

    For long or infinite lists use fromList because it uses LazyByteString otherwise use fromListN which uses StrictByteString.

  • instance Functor Put
  • instance Applicative Put
  • instance Monad Put
Maintainer
Simon Meier <iridcode@gmail.com>
Portability
GHC
Copyright
(c) 2010 - 2011 Simon Meier
License
BSD3-style (see LICENSE)
Build record and source files

Community contribution. The service has not verified this documentation. The generator reported 11 diagnostics.

Target: Linux on x86-64. Some inferred types and resolved links are not available.

Generator: aihc-haddock-aa6debbd-web-3

Metadata snapshot: 35f1f56d3dce7f0c7af322ec05f2fba4d1a04564930537462a4ba21b64b756fa

Open the dependency plan · Download the documentation model

Import a package

Enter a package name and an exact version from Hackage. The site contains imported versions only.