MEGA: Enchanting decoders (part 1)
In this MEGA’s article, we’ll see some useful patterns to discover magic powers hidden inside Elm decoders.
Introduction: The gate to darkness
Once upon a time, in the Pure Language realm, everything was simple, bright and predictable.
That was until a despicable programmer fairy opened the gates to the realm of Impure Languages…
This could quite easily be your story as an Elm beginner. You were used to writing clean and composable code. But then you realised that everything’s sandboxed. So you still need to deal with stuff like browser, JavaScript and API calls.
And that’s when you probably met your first decoder.
A programmable parser
A decoder is a piece of code that builds a parser to turn your serialized data into your Elm types. Otherwise, it fails.
It’s needed every time you need to obtain data at runtime. For instance, events, API calls, ports and subscriptions.
Out of the box Elm ships with basic JSON parsing capabilities, that allow you to deal with JavaScript and REST API.
Some libraries empower JSON decoding. And others allows us to deal with data different to JSON (like GraphQL).
In the following examples, I’ll show some interesting (I hope) cases of JSON parsing of a rest API.
And I’ll use elm-json-decode-pipeline, a great package that we use a lot here at Prima.
Master decoders and keep your world fair
Granted, decoders can be a bit of a headache. But once you master them, you can rid your Elm code of a lot of data consistency checks.
Decoders allow you to move all that “if-then” logic in one single point. You can then build your application model not around the backend, but around the user application needs and keep your code cleaner.
Do you remember the examples from Maybe maybe is not the right choice article?
Let’s imagine the situation where Gift is provided by a REST API like the following (I’m using jsdoc to describe explicit nullable fields, so imagine a case where we have some kind of type guarantee from BE). And you’re writing a Payment application:
/**
* @typedef {{
* hasGreetingCard: !boolean,
* gCTitle: string|null,
* gCBody: string|null,
* gCSignature: string|null,
* price: number|null
* }} Gift
*/Let’s reorder messy types
Let’s consider the price key at first.
Do we need to have Maybe Float in our model?
The answer’s no. Because the API contract says that, in every legal case that value exists, there’s a number type and it’s always a price. So a more suitable data type for the frontend could be a Decimal of decimal package.
The package exposes a nice function:
fromFloat : Float -> Maybe DecimalSo now we have two options:
- The not-so smart option. Decode a
Maybe Floatand keep that in our model. Every time we’ll need it we’ll turn that into aDecimalresolving two Maybes. Since we can’t handleNothingvariant (it’s a payment page) we’ll probably resolve theNothingwith a default. - The smart option. Attempt to decode our
Maybe Floatinto aDecimaland if we can’t… well probably something is going very wrong outside there!
Since we want to be smart let’s write our decimal decoder joining together existing ones with a pinch of custom logic to make it decode a Decimal or fail:
import Json.Decode as Decode...decimalFromFloatDecoder : Decode.Decoder Decimal
decimalFromFloatDecoder =
Decode.float
|> Decode.andThen
(\decodedFloat ->
-- Float -> Maybe Decimal
Decimal.fromFloat decodedFloat
-- if Just decimal -> success
|> Maybe.map Decode.succeed
-- if Nothing -> failure
|> Maybe.withDefault (decodingFailure decodedFloat)
)
decodingFailure : Float -> Decode.Decoder Decimal
decodingFailure decodedFloat =
"Can't turn "
++ String.fromFloat decodedFloat
++ " into a decimal"
|> JsonDecode.fail
Now it’s time for Json.Decode.Pipeline ! We can use its required function to ensure that the price field exists and our new decoder to turn it into a Decimal:
import Json.Decode as Decode
import Json.Decode.Pipeline as DecodePipeline...type alias Gift =
{ price : Decimal }
giftDecoder : Decode.Decoder Gift
giftDecoder =
Decode.succeed Gift
|> DecodePipeline.required "price" decimalFromFloatDecoder
That’s all. Now your Gift model has its nice and well typed price field!
Decode union types
So now imagine that we want to decode the Gift type with its optional greeting card pieces of information.
The first naive approach is to write a very simple decoder (because you know they can be a headache) that decodes a record like this:
type alias Gift =
{ hasGreetingCard : Bool
, greetingCardTitle : Maybe String
, greetingCardBody : Maybe String
, greetingCardSignature : Maybe String
, price : Decimal
}Instead of something better shaped like:
type Gift
= Simple { price : Decimal }
| WithGreetingCard { price : Decimal, greetingCard : GreetingCard}type alias GreetingCard =
{ title : String
, body : String
, signature : String
}
As seen in the previous article, we prefer the second choice. So we’d need to convert that record into our union type.
But how?
Well, with a well-structured decoder, we can make all the “bogus” cases collapse into the decoding failure and clean everything up.
When you work with decoders, it helps to think locally. So we’ll try to break down the big problem into simpler tasks.
First of all, we’ll try to decode the raw pieces of information that we need to build a SimpleGift:
type alias SimpleRawData =
{ price : Float, hasGreetingCard : Bool }
simpleRawDataDecoder : Decode.Decoder SimpleRawData
simpleRawDataDecoder =
Decode.succeed SimpleRawData
|> DecodePipeline.required "price" decimalFromFloatDecoder
|> DecodePipeline.required "hasGreetingCard" Decode.bool
Let’s do the same with the other variant, reusing the previous decoders. We could rewrite a whole decoder. But I’m lazy and I can use Json.Decode.Pipeline.custom function:
type alias WithGreetingCardRaw =
{ simpleRaw : SimpleRaw
, body : String
, signature : String
, title : String
}
withGreetingCardRawDecoder : Decode.Decoder WithGreetingCardRaw
withGreetingCardRawDecoder =
Decode.succeed WithGreetingCardRaw
|> DecodePipeline.custom simpleRawDecoder
|> DecodePipeline.required "gCBody" Decode.string
|> DecodePipeline.required "gCSignature" Decode.string
|> DecodePipeline.required "gCTitle" Decode.string
rawToWithGreetingCardDecoder :
WithGreetingCardRaw
-> Decode.Decoder Gift
rawToWithGreetingCardDecoder rawData =
if rawData.simpleRaw.hasGreetingCard then
WithGreetingCard
{ price = rawData.simpleRawData.price
, greetingCard =
{ title = rawData.title
, body = rawData.body
, signature = rawData.signature
}
}
|> Decode.succeed
else
Decode.fail "Not has greeting card. It can't be WithGreetingCard variant"
giftWithGreetingCardDecoder : Decode.Decoder Gift
giftWithGreetingCardDecoder =
withGreetingCardRawDecoder
|> Decode.andThen rawToWithGreetingCardDecoderNow we can simply combine them with Json.Decode.oneOf . This function tries to resolve the serialized data with a list of decoders until one succeeds (so pay attention to priority, in case you don’t have a discriminating value like hasGreetingCard ) :
module MyApp.Gift exposing (Gift, fetch, decoder)decoder : Decode.Decoder Gift
decoder =
Decode.oneOf
[ simpleDecoder
, withGreetingCardDecoder
]
fetch: (Result Http.Error Gift -> msg) -> Cmd msg
fetch msg = ...
That could become (with some love and modularization that I’ll leave to you) something like:
module Gift exposing (Gift, decoder)import Json.Decode as Decode
import Gift.SimpleGift as SimpleGift exposing (SimpleGift)
import Gift.WithGreetingCardGift as WithGreetingCardGift exposing (WithGreetingCardGift)
type Gift
= Simple SimpleGift
| WithGreetingCard WithGreetingCardGift
decoder : Decode.Decoder Gift
decoder =
Decode.oneOf
[ SimpleGift.decoder
|> Decode.map Simple
, WithGreetingCardGift.decoder
|> Decode.map WithGreetingCard
]
And that’s it. We’ve moved all our data coherence checks into a single point. Plus, we can expose only a simple fetch function, the whole decoder, or both as preferred.
We’ve reached more than one goal with that:
- We wiped out a loosely typed maybish record in favour of a more fashionable union in the whole app.
- We can move the whole business logic and consistency checks about
Giftin a single point. - We can make our consistency checks more or less strict based on our BE and requirements. We can morph the raw data directly here, providing default cases instead of making everything fail. For example what happens if gift card values are truly nullable but you always provide them even if not given (with fallback values)? You can simply provide the decoder with your fallback in case of null:
greetingCardRawDataDecoder : Decode.Decoder GreetingCardRawData
greetingCardRawDataDecoder =
Decode.succeed GreetingCardRawData
|> DecodePipeline.custom simpleGiftRawDataDecoder
|> DecodePipeline.required "gCBody"
(stringOrDefaultDecoder "Your greeting message")
-- ...
stringOrDefaultDecoder : String -> Decode.Decoder String
stringOrDefaultDecoder def =
Decode.nullable Decode.string
|> Decode.map (Maybe.withDefault def)Intercept errors and keep your flows clean
Surely you’ve noticed how messy your update function becomes when you have more than one API call and a little logic (like some security checks on it):
type AppMsg
= GetGiftResult (Result Http.Error Gift)
| GetAnotherResult (Result Http.Error Another)
--...appUpdate : AppMsg -> Model -> Update
appUpdate msg model =
case msg of
GetGiftResult (Result.Ok gift) ->
-- store and then do something
GetGiftResult (Result.Err httpError) ->
case httpError of
Http.BadStatus 401 ->
-- Unauthorized handling code
_ ->
-- show generic error
GetAnotherResult (Result.Ok another) ->
-- store and then do something
GetAnotherResult (Result.Err httpError) ->
case httpError of
Http.BadStatus 401 ->
-- Unauthorized handling code
_ ->
-- show generic error
--...
You have to admit that it’s not nice or scalable. A lot of branches are probably the same. You can group the logic with functions but the update case will stay huge.
We can do better. Let me explain.
You probably noticed that your http calls always produce a Result Http.Error decodedData as come back result.
But Elm’s faces 2 possible failures.
The first is during the request process (that always comes out as a Response type):
-- Http.elmtype Response body
= BadUrl_ String
| Timeout_
| NetworkError_
| BadStatus_ Metadata body
| GoodStatus_ Metadata body
The other possible failure happens during the decoding process and results in a parsing error:
-- Decoder.elm
type Error
= Field String Error
| Index Int Error
| OneOf (List Error)
| Failure String ValueThis because you’re used to passing Http.expectJson as request expectation:
fetch :
(Result Http.Error Gift -> msg)
-> AuthToken
-> Cmd msg
fetch tag authToken =
Http.request
{ method = "GET"
, headers = [ AuthToken.header authToken ]
, url = "https://my.api.it/get-my-gift"
, body = Http.emptyBody
, expect = Http.expectJson tag decoder
, timeout = Nothing
, tracker = Nothing
}Such function takes the internal Response body type, if the response is ok, then attempts to decode the body with your decoder.
If Response indicates a failure or decoder fails then the result will be a Http.Error type that wraps the previous two errors:
type Error
= BadUrl String -- Response.BadUrl_ case
| Timeout -- Response.Timeout_ case
| NetworkError -- Response.NetworkError_
| BadStatus Int -- Response.BadStatus with metadata statusCode
| BadBody String -- Decode.Error serialized as StringIt’s fast and simple. But there are a couple of drawbacks:
- It hides some
Responseinformation. (We can’t access the body in the case of aBadStatus. We can’t see response headers. And we can’t intercept decoding failures.) - In the case of a reusable library, you’re demanding a lot of business logic of the consumer regarding the API.
- The component’s API can’t be self-explanatory.
We can fix almost all of these weak points though.
First of all, let’s define the Gift error that we’re interested to catch:
import MyApp.Unauthorized as Unauthorized exposing (Unauthorized)type Error
= UnauthorizedError Unauthorized
| NotInterestingError String
And after that, we can write our custom expect , that intercepts 401 status code and applies another decoder on that body:
expect : (Result Error Gift -> msg) -> Http.Expect msg
expect tag =
Http.expectStringResponse tag responseResolver
responseResolver : Http.Response String -> Result Error Gift
responseResolver response =
case response of
Http.BadUrl_ string ->
"Bad Url: "
++ string
|> NotInterestingError
|> Result.Err
Http.Timeout_ ->
NotInterestingError "Timeout"
|> Result.Err
Http.NetworkError_ ->
NotInterestingError "NetworkError"
|> Result.Err
Http.BadStatus_ metadata b ->
if metadata.statusCode == 401 then
case JDecode.decodeString Unauthorized.decoder b of
Result.Ok data ->
UnauthorizedError data
|> Result.Err
Result.Err decodeError ->
"Decode unauthorized response failure: "
++ JDecode.errorToString decodeError
|> NotInterestingError
|> Result.Err
else
"BadStatusError:"
++ String.fromInt metadata.statusCode
|> NotInterestingError
|> Result.Err
Http.GoodStatus_ _ body ->
JDecode.decodeString decoder body
|> Result.mapError
(\err ->
"Decode gift failure: "
++ JDecode.errorToString err
|> NotInterestingError
)We can now make our fetch function more expressive:
type alias FetchArgs =
{ authToken : AuthToken
, onGiftFetched : Gift -> msg
, onUnauthorized : UnauthorizedData -> msg
, onError : String -> msg
}
fetch : FetchArgs -> Cmd msg
fetch args =
Http.request
{ method = "GET"
, headers = [ AuthToken.header args.authToken ]
, url = "https://my.api.it/get-my-gift"
, body = Http.emptyBody
, expect = expect args
, timeout = Nothing
, tracker = Nothing
}And change our custom expect function, adding the message retagging:
expect : FetchArgs -> Http.Expect msg
expect args =
Http.expectStringResponse (resultToTag args) responseResolver
resultToTag : FetchArgs -> Result Error Gift -> msg
resultToTag args result =
case result of
Ok gift ->
args.onGiftFetched gift
Err (Unauthorized unauthorizedData) ->
args.onUnauthorized unauthorizedData
Err (NotInterestingError errAsString) ->
args.onError errAsStringAnd finally, in our update function, we can collapse every common logic in one dedicated state, like this:
type AppMsg
= GiftFetched Gift
| AnotherThingFetched AnotherThing
| Unauthorized UnauthorizedData
| FatalError String
...appUpdate : AppMsg -> Model -> Update
appUpdate msg model =
case msg of
GiftFetched gift ->
-- store and then do something ... AnotherThingFetched anotherThing ->
-- store and then do something ... Unauthorized unauthorized ->
model -- here we can start a re-auth flow
|> withCmd Unauthorized.fetchAuth unauthorized
FatalError errString ->
model -- here we simply show a error page and log all
|> Model.setFatalError errString
|> withCmd Logger.logError errString --...
Congratulations you have reached the end…
If you’ve reached this point (without a headache), you’ve gained an entry-level Decoder Master.
Wait a minute… all of this and I’m still an entry-level ??
Ehm yes… You’ve learned that decoders can be more than simple string parsers, and how to make them more effective in cleaning up a malformed server API.
You should be able to decode complex elm types and rewrite the basic “decode or fail” logic hidden inside HTTP requests.
This is great and helps make you a better Elm’s sorcerer. But there is still much to say about decoders. I’ll try to cover up the most interesting parts in future articles.
See you soon in the next castle!
