diff --git a/cabal.project b/cabal.project index 167acec4..2bb4bdd8 100644 --- a/cabal.project +++ b/cabal.project @@ -19,6 +19,7 @@ packages: servant/ -- doc/cookbook/jwt-and-basic-auth/ -- doc/cookbook/pagination doc/cookbook/sentry + doc/cookbook/testing doc/cookbook/structuring-apis doc/cookbook/using-custom-monad doc/cookbook/using-free-client diff --git a/doc/cookbook/cabal.project b/doc/cookbook/cabal.project index 190e1c66..4b12a08a 100644 --- a/doc/cookbook/cabal.project +++ b/doc/cookbook/cabal.project @@ -11,6 +11,7 @@ packages: https/ pagination/ sentry/ + testing/ ../../servant ../../servant-server ../../servant-client-core diff --git a/doc/cookbook/index.rst b/doc/cookbook/index.rst index b7e9bd1c..d276d6fa 100644 --- a/doc/cookbook/index.rst +++ b/doc/cookbook/index.rst @@ -31,3 +31,4 @@ you name it! pagination/Pagination.lhs curl-mock/CurlMock.lhs sentry/Sentry.lhs + testing/Testing.lhs diff --git a/doc/cookbook/testing/Testing.lhs b/doc/cookbook/testing/Testing.lhs new file mode 100644 index 00000000..a075a144 --- /dev/null +++ b/doc/cookbook/testing/Testing.lhs @@ -0,0 +1,505 @@ +# How To Test Servant Applications + +Even with a nicely structured API that passes Haskell's strict type checker, +it's a good idea to write some tests for your application. + +In this recipe we'll work through some common testing strategies and provide +examples of utlizing these testing strategies in order to test Servant +applications. + +## Testing strategies + +There are many testing strategies you may wish to employ when testing your +Servant application, but included below are three common testing patterns: + +- We'll use `servant-client` to derive client functions and then send valid +requests to our API, running in another thread. This is great for testing +that our **business logic** is correctly implemented with only valid HTTP +requests. + +- We'll also use `hspec-wai` to make **arbitrary HTTP requests**, in order to +test how our application may respond to invalid or otherwise unexpected +requests. + +- Finally, we can also use `servant-quickcheck` for **whole-API tests**, in order +to assert that our entire application conforms to **best practices**. + +## Useful Libraries + +The following libraries will often come in handy when we decide to test our +Servant applications: + +- [hspec](https://hspec.github.io/) +- [hspec-wai](http://hackage.haskell.org/package/hspec-wai) +- [QuickCheck](http://hackage.haskell.org/package/QuickCheck) +- [servant-quickcheck](https://hackage.haskell.org/package/servant-quickcheck) + +## Imports and Our Testing Module + +This recipe starts with the following ingredients: + +```haskell +{-# LANGUAGE OverloadedStrings, TypeFamilies, DataKinds, + DeriveGeneric, TypeOperators #-} +import qualified Control.Concurrent as C +import Control.Concurrent.MVar +import Control.Exception (bracket) +import Control.Lens hiding (Context) +import Data.Aeson +import Data.Aeson.Lens +import qualified Data.HashMap.Strict as HM +import Data.Text +import GHC.Generics +import Network.HTTP.Client hiding (Proxy) +import Network.HTTP.Types +import Network.Wai +import qualified Network.Wai.Handler.Warp as Warp + +import Servant +import Servant.Client +import Servant.Server +import Servant.QuickCheck +import Servant.QuickCheck.Internal (serverDoesntSatisfy) + +import Test.Hspec +import Test.Hspec.Wai +import Test.Hspec.Wai.Matcher +``` + +We're going to produce different `Spec`s that represent different +aspects of our application, and we'll ask `hspec` to run all of our different +`Spec`s. This is a common organizational method for testing modules: + +```haskell +spec :: Spec +spec = do + businessLogicSpec + thirdPartyResourcesSpec + servantQuickcheckSpec +``` + +Often, codebases will use `hspec`'s +[autodiscover pragma](http://hspec.github.io/hspec-discover.html) +to find all testing modules and `Spec`s inside, but we're going to +explicitly make a `main` function to run our tests because we have only one +`spec` defined above: + +```haskell +main :: IO () +main = hspec spec +``` + +## Testing Your Business Logic + +Let's say we have an API that looks something like this: + +```haskell +data User = User { + name :: Text + , user_id :: Integer + } deriving (Eq, Show, Generic) + +instance FromJSON User +instance ToJSON User + +type UserApi = + -- One endpoint: create a user + "user" :> Capture "userId" Integer :> Post '[JSON] User +``` + +A real server would likely use a database to store, retrieve, and validate +users, but we're going to do something really simple merely to have something +to test. With that said, here's a sample handler, server, and `Application` +for the endpoint described above: + +```haskell +userApp :: Application +userApp = serve (Proxy :: Proxy UserApi) userServer + +userServer :: Server UserApi +userServer = createUser + +createUser :: Integer -> Handler User +createUser userId = do + if userId > 5000 + then pure $ User { name = "some user", user_id = userId } + else throwError $ err400 { errBody = "userId is too small" } +``` + +### Strategy 1: Spin Up a Server, Create a Client, Make Some Requests + +One of the benefits of Servant's type-level DSL for describing APIs is that +once you have provided a type-level description of your API, you can create +clients, documentation, or other tools for it somewhat magically. + +In this case, we'd like to *test* our server, so we can use `servant-client` +to create a client, after which we'll run our server, and then make requests +of it and see how it responds. + +Let's write some tests: + +```haskell +withUserApp :: IO () -> IO () +withUserApp action = + -- we can spin up a server in another thread and kill that thread when done + -- in an exception-safe way + bracket (liftIO $ C.forkIO $ Warp.run 8888 userApp) + C.killThread + (const action) + + +businessLogicSpec :: Spec +businessLogicSpec = + -- `around` will start our Server before the tests and turn it off after + around_ withUserApp $ do + -- create a test client function + let createUser = client (Proxy :: Proxy UserApi) + -- create a servant-client ClientEnv + baseUrl <- runIO $ parseBaseUrl "http://localhost:8888" + manager <- runIO $ newManager defaultManagerSettings + let clientEnv = mkClientEnv manager baseUrl + + -- testing scenarios start here + describe "POST /user" $ do + it "should create a user with a high enough ID" $ do + result <- runClientM (createUser 50001) clientEnv + result `shouldBe` (Right $ User { name = "some user", user_id = 50001}) + it "will it fail with a too-small ID?" $ do + result <- runClientM (createUser 4999) clientEnv + result `shouldBe` (Right $ User { name = "some user", user_id = 50001}) +``` + +### Running These Tests + +Let's run our tests and see what happens: + +``` +$ cabal new-test all +POST /user + should create a user with a high enough ID + should fail with a too-small ID FAILED [1] + + Failures: + + Testing.lhs:129:7: + 1) POST /user should fail with a too-small ID + expected: Right (User {name = "some user", user_id = 50001}) + but got: Left (FailureResponse (Response {responseStatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, responseHeaders = fromList [("Transfer-Encoding","chunked"),("Date","Fri, 12 Oct 2018 04:36:22 GMT"),("Server","Warp/3.2.25")], responseHttpVersion = HTTP/1.1, responseBody = "userId is too small"})) + + To rerun use: --match "/POST /user/should fail with a too-small ID/" +``` + +Hmm. One passed and one failed! It looks like I *was* expecting a success +response in the second test, but I actually got a failure. We should fix that, +but first I'd like to introduce `hspec-wai`, which will give us different +mechanisms for making requests of our application and validating the responses +we get. We're also going to spin up a fake Elasticsearch server, so that our +server can think it's talking to a real database. + + +## *Mocking* 3rd Party Resources + +Often our web applications will need to make their own web +requests to other 3rd-party applications. These requests provide a lot +of opportunity for failure and so we'd like to test that the right +messages and failure values (in addition to success values) are returned +from our application. + +### Define the 3rd-Party Resource + +With Servant's type-level API definitions, assuming you've already defined the +API you want to mock, it's relatively trivial to create a simple server for +the purposes of running tests. For instance, consider an API server that needs +to get data out of Elasticsearch. Let's first define the Elasticsearch server +and client using Servant API descriptions: + +```haskell +type SearchAPI = + -- We're using Aeson's Generic JSON `Value` to make things easier on + -- ourselves. We're also representing only one Elasticsearch endpoint: + -- get item by id + "myIndex" :> "myDocType" :> Capture "docId" Integer :> Get '[JSON] Value + +-- Here's our Servant Client function +getDocument = client (Proxy :: Proxy SearchAPI) + +-- We can use these helpers when we want to make requests +-- using our client function +clientEnv :: Text -> Text -> IO ClientEnv +clientEnv esHost esPort = do + baseUrl <- parseBaseUrl $ unpack $ esHost <> ":" <> esPort + manager <- newManager defaultManagerSettings + pure $ mkClientEnv manager baseUrl + +runSearchClient :: Text -> Text -> ClientM a -> IO (Either ServantError a) +runSearchClient esHost esPort = (clientEnv esHost esPort >>=) . runClientM +``` + +### Servant Server Example Using this 3rd-Party Resource + +So we've got an Elasticsearch server and a client to talk to it. Let's now +build a simple app server that uses this client to retrieve documents. This +is somewhat contrived, but hopefully it illustrates the typical three-tier +application architecture. + +One note: we're also going to take advantage of `lens-aeson` here, which may +look a bit foreign. The gist of it is that we're going to traverse a JSON +`Value` from Elasticsearch and try to extract some kind of document to +return. + +Imagine, then, that this is our real server implementation: + +```haskell +type DocApi = + "docs" :> Capture "docId" Integer :> Get '[JSON] Value + +docsApp :: Text -> Text -> Application +docsApp esHost esPort = serve (Proxy :: Proxy DocApi) $ docServer esHost esPort + +docServer :: Text -> Text -> Server DocApi +docServer esHost esPort = getDocById esHost esPort + +-- Our Handler tries to get a doc from Elasticsearch and then tries to parse +-- it. Unfortunately, there's a lot of opportunity for failure in these +-- actions +getDocById :: Text -> Text -> Integer -> Handler Value +getDocById esHost esPort docId = do + -- Our Servant Client function returns Either ServantError Value here: + docRes <- liftIO $ runSearchClient esHost esPort (getDocument docId) + case docRes of + Left err -> throwError $ err404 { errBody = "Failed looking up content" } + Right value -> do + -- we'll eitherĀ fail to parse our document or we'll return it + case value ^? _Object . ix "_source" of + Nothing -> throwError $ err400 { errBody = "Failed parsing content" } + Just obj -> pure obj +``` + +### Testing Our Backend + +So the above represents our application and is close to a server we may +actually deploy. How then shall we test this application? + +Ideally, we'd like it to make requests of a *real* Elasticsearch server, but +we certainly don't want our tests to trigger requests to a live, production +database. In addition, we don't want to depend on our real Elasticsearch +server having specific, consistent results for us to test against, because +that would make our tests flaky (and flaky tests are sometimes described as +worse than not having tests at all). + +One solution to this is to create a trivial Elasticsearch server as part of +our testing code. We can do this relatively easily because we already have +an API definition for it above. With a *real* server, we can then let our own +application make requests of it and we'll simulate different scenarios in +order to make sure our application responds the way we expect it to. + +Let's start with some helpers which will allow us to run a testing version +of our Elasticsearch server in another thread: + +```haskell +-- | We'll run the Elasticsearch server so we can test behaviors +withElasticsearch :: IO () -> IO () +withElasticsearch action = + bracket (liftIO $ C.forkIO $ Warp.run 9999 esTestApp) + C.killThread + (const action) + +esTestApp :: Application +esTestApp = serve (Proxy :: Proxy SearchAPI) esTestServer + +esTestServer :: Server SearchAPI +esTestServer = getESDocument + +-- This is the *mock* handler we're going to use. We create it +-- here specifically to trigger different behavior in our tests. +getESDocument :: Integer -> Handler Value +getESDocument docId + -- arbitrary things we can use in our tests to simulate failure: + -- we want to trigger different code paths. + | docId > 1000 = throwError err500 + | docId > 500 = pure . Object $ HM.fromList [("bad", String "data")] + | otherwise = pure $ Object $ HM.fromList [("_source", Object $ HM.fromList [("a", String "b")])] +``` + +Now, we should be ready to write some tests. + +In this case, we're going to use `hspec-wai`, which will give us a simple way +to run our application, make requests, and make assertions against the +responses we receive. + +Hopefully, this will simplify our testing code: + +```haskell +thirdPartyResourcesSpec :: Spec +thirdPartyResourcesSpec = around_ withElasticsearch $ do + -- we call `with` from `hspec-wai` and pass *real* `Application` + with (pure $ docsApp "localhost" "9999") $ do + describe "GET /docs" $ do + it "should be able to get a document" $ + -- `get` is a function from hspec-wai`. + get "/docs/1" `shouldRespondWith` 200 + it "should be able to handle connection failures" $ + get "/docs/1001" `shouldRespondWith` 404 + it "should be able to handle parsing failures" $ + get "/docs/501" `shouldRespondWith` 400 + it "should be able to handle odd HTTP requests" $ + -- we can also make all kinds of arbitrary custom requests to see how + -- our server responds using the `request` function: + -- request :: Method -> ByteString -> [Header] + -- -> LB.ByteString -> WaiSession SResponse + request methodPost "/docs/501" [] "{" `shouldRespondWith` 405 + it "we can also do more with the Response using hspec-wai's matchers" $ + -- see also `MatchHeader` and JSON-matching tools as well... + get "/docs/1" `shouldRespondWith` 200 { matchBody = MatchBody bodyMatcher } + +bodyMatcher :: [Network.HTTP.Types.Header] -> Body -> Maybe String +bodyMatcher _ body = case (decode body :: Maybe Value) of + -- success in this case means we return `Nothing` + Just val | val == (Object $ HM.fromList [("a", String "b")]) -> Nothing + _ -> Just "This is how we represent failure: this message will be printed" +``` + +Out of the box, `hspec-wai` provides a lot of useful tools for us to run tests +against our application. What happens when we run these tests? + +``` +$ cabal new-test all +... + +GET /docs + should be able to get a document + should be able to handle connection failures + should be able to handle parsing failures + should be able to handle odd HTTP requests + we can also do more with the Response using hspec-wai's matchers +``` + +Fortunately, they all passed! Let's move to another strategy: whole-API +testing. + +## Servant Quickcheck + +[`servant-quickcheck`](https://github.com/haskell-servant/servant-quickcheck) +is a project that allows users to write tests for whole Servant APIs using +quickcheck-style property-checking mechanisms. + +`servant-quickcheck` is great for asserting API-wide rules, such as "no +endpoint throws a 500" or "all 301 status codes also come with a Location +header". The project even comes with a number of predicates that reference +the [RFCs they originate from](https://github.com/haskell-servant/servant-quickcheck/blob/master/src/Servant/QuickCheck/Internal/Predicates.hs). + +In other words, it's one way to assert that your APIs conform to specs and +best practices. + +### Quickcheckable API + +Let's make an API and a server to demonstrate how to use `servant-quickcheck`: + +```haskell +type API = ReqBody '[JSON] String :> Post '[JSON] String + :<|> Get '[JSON] Int + :<|> BasicAuth "some-realm" () :> Get '[JSON] () + +api :: Proxy API +api = Proxy + +server :: IO (Server API) +server = do + mvar <- newMVar "" + return $ (\x -> liftIO $ swapMVar mvar x) + :<|> (liftIO $ readMVar mvar >>= return . Prelude.length) + :<|> (const $ return ()) +``` + +### Using `servant-quickcheck` + +Let's build some tests for our API using `servant-quickcheck`. + +Similar to the above examples, we're going to create `Spec`s, but in this +case, we'll rely on a number of predicates available from `servant-quickcheck` +to see if our API server conforms to best practices: + +```haskell +-- Let's set some QuickCheck values +args :: Args +args = defaultArgs { maxSuccess = 500 } + +-- Here's a Servant Context object we'll use +ctx :: Context '[BasicAuthCheck ()] +ctx = BasicAuthCheck (const . return $ NoSuchUser) :. EmptyContext + + +servantQuickcheckSpec :: Spec +servantQuickcheckSpec = describe "" $ do + it "API demonstrates best practices" $ + -- `withServerServer` and `withServantServerAndContext` come from `servant-quickcheck` + withServantServerAndContext api ctx server $ \burl -> + -- `serverSatisfies` and the predicates also come from `servant-quickcheck` + serverSatisfies api burl args (unauthorizedContainsWWWAuthenticate + <%> not500 + <%> onlyJsonObjects -- this one isn't true! + <%> mempty) + + it "API doesn't have these things implemented yet" $ + withServantServerAndContext api ctx server $ \burl -> do + serverDoesntSatisfy api burl args (getsHaveCacheControlHeader + <%> notAllowedContainsAllowHeader + <%> mempty) +``` + +Let's see what happens when we run these tests: + + +``` + API demonstrates best practices FAILED [2] ++++ OK, passed 500 tests. + API doesn't have these things implemented yet + + src/Servant/QuickCheck/Internal/QuickCheck.hs:143:11: + 2) Main[339:25] API demonstrates best practices + Failed: + Just Predicate failed + Predicate: onlyJsonObjects + + Response: + Status code: 200 + Headers: "Transfer-Encoding": "chunked" + "Date": "Fri, 12 Oct 2018 04:36:22 GMT" + "Server": "Warp/3.2.25" + "Content-Type": "application/json;charset=utf-8" + Body: "" + + To rerun use: --match "/Main[339:25]/API demonstrates best practices/" + +Randomized with seed 1046277487 + +Finished in 0.4306 seconds +``` + +Hmm. It looks like we *thought* our API only returned JSON objects, which is a +best practice, but in fact, we *did* have an endpoint that returned an empty +body, which you can see in the printed response above: `Body: ""`. We should +consider revising our API to only return top-level JSON Objects in the future! + +### Other Cool Things + +`servant-quickcheck` also has a cool mechanism where you can compare two API +servers to demonstrate that they respond identically to requests. This may be +useful if you are planning to rewrite one API in another language or with +another web framework. You have to specify whether you're looking for +`jsonEquality` vs regular `ByteString` equality, though. + +## Conclusion + +There are lots of techniques for testing and we only covered a few here. + +Useful libraries such as `hspec-wai` have ways of running Wai `Application`s +and sending requests to them, while Servant's type-level DSL for defining APIs +allows us to more easily mock out servers and to derive clients, which will +only craft valid requests. + +Lastly, if you want a broad overview of where your application fits in with +regard to best practices, consider using `servant-quickcheck`. + +This program is available as a cabal project +[here](https://github.com/haskell-servant/servant/tree/master/doc/cookbook/testing). \ No newline at end of file diff --git a/doc/cookbook/testing/testing.cabal b/doc/cookbook/testing/testing.cabal new file mode 100644 index 00000000..9e923286 --- /dev/null +++ b/doc/cookbook/testing/testing.cabal @@ -0,0 +1,37 @@ +name: cookbook-testing +version: 0.0.1 +synopsis: Common testing patterns in Servant apps +description: This recipe includes various strategies for writing tests for Servant. +homepage: http://haskell-servant.readthedocs.org/ +license: BSD3 +license-file: ../../../servant/LICENSE +author: Servant Contributors +maintainer: haskell-servant-maintainers@googlegroups.com +category: Servant +build-type: Simple +cabal-version: >=1.10 +tested-with: GHC==8.0.2, GHC==8.2.2, GHC==8.4.3 + +executable cookbook-testing + main-is: Testing.lhs + build-depends: base == 4.* + , text >= 1.2 + , aeson >= 1.2 + , lens-aeson + , lens + , servant + , servant-client + , servant-server + , servant-quickcheck + , http-client + , http-types >= 0.12 + , hspec + , hspec-wai + , QuickCheck + , unordered-containers + , warp >= 3.2 + , wai >= 3.2 + , wai-extra + default-language: Haskell2010 + ghc-options: -Wall -pgmL markdown-unlit + build-tool-depends: markdown-unlit:markdown-unlit