From 43af1d0c9e144cf3bc87b3e10b3c636a76c0bb2f Mon Sep 17 00:00:00 2001 From: Erik Aker Date: Thu, 11 Oct 2018 20:51:30 -0700 Subject: [PATCH 1/7] WIP Commit: must finish servant-quickcheck tests --- cabal.project | 1 + doc/cookbook/cabal.project | 1 + doc/cookbook/index.rst | 1 + doc/cookbook/testing/Testing.lhs | 331 +++++++++++++++++++++++++++++ doc/cookbook/testing/testing.cabal | 33 +++ 5 files changed, 367 insertions(+) create mode 100644 doc/cookbook/testing/Testing.lhs create mode 100644 doc/cookbook/testing/testing.cabal diff --git a/cabal.project b/cabal.project index ae70b99d..02c2d73e 100644 --- a/cabal.project +++ b/cabal.project @@ -19,6 +19,7 @@ packages: servant/ doc/cookbook/jwt-and-basic-auth/ doc/cookbook/pagination doc/cookbook/sentry + doc/cookbook/testing doc/cookbook/structuring-apis doc/cookbook/using-custom-monad doc/cookbook/using-free-client diff --git a/doc/cookbook/cabal.project b/doc/cookbook/cabal.project index 190e1c66..4b12a08a 100644 --- a/doc/cookbook/cabal.project +++ b/doc/cookbook/cabal.project @@ -11,6 +11,7 @@ packages: https/ pagination/ sentry/ + testing/ ../../servant ../../servant-server ../../servant-client-core diff --git a/doc/cookbook/index.rst b/doc/cookbook/index.rst index b7e9bd1c..d276d6fa 100644 --- a/doc/cookbook/index.rst +++ b/doc/cookbook/index.rst @@ -31,3 +31,4 @@ you name it! pagination/Pagination.lhs curl-mock/CurlMock.lhs sentry/Sentry.lhs + testing/Testing.lhs diff --git a/doc/cookbook/testing/Testing.lhs b/doc/cookbook/testing/Testing.lhs new file mode 100644 index 00000000..241a1bb0 --- /dev/null +++ b/doc/cookbook/testing/Testing.lhs @@ -0,0 +1,331 @@ +# How To Test Servant Applications + +Even with a nicely structured API that passes Haskell's strict type checker, +it's a good idea to write some tests for your application. + +In this recipe we'll work through some common testing strategies and provide +examples of utlizing these testing strategies in order to test Servant +applications. + +This recipe uses the following ingredients: + +```haskell +{-# LANGUAGE OverloadedStrings, TypeFamilies, DataKinds, + DeriveGeneric, TypeOperators #-} +import qualified Control.Concurrent as C +import Data.Aeson +import Data.Aeson.Lens +import Data.Text +import Network.Wai +import qualified Network.Wai.Handler.Warp as Warp +import Servant +import Servant.Client +import Servant.Server + +import Test.Hspec +import Test.Hspec.Wai +import Test.Hspec.Wai.Matcher +``` + +We're going to produce different `Spec`s that represent different +aspects of our application, and we'll ask `hspec` to run all of our different +`Spec`s. This is a common organizational method for testing modules: + +```haskell +spec :: Spec +spec = + businessLogicSpec + thirdPartyResourcesSpec + servantQuickcheckSpec +``` + +Often, codebases will use `hspec`'s +[autodiscover pragma](http://hspec.github.io/hspec-discover.html) +to find all testing modules and `Spec`s inside, but we're going to +explicitly make a `main` function to run our tests because we have only one +`spec` defined above: + +```haskell +main :: IO () +main = hspec spec +``` + +## Testing Your Business Logic + +Let's say we have an API that looks something like this: + +```haskell +data User = User { + name :: Text + , user_id :: Integer + } deriving (Eq, Show, Generic) + +instance FromJSON User +instance ToJSON User + +type UserApi = + -- One endpoint: create a user + "user" :> Capture "userId" Integer :> Post '[JSON] User +``` + +A real server would likely use a database to store, retrieve, and validate +users, but we're going to do something really simple merely to have something +to test. With that said, here's a sample handler for the endpoint described +above: + +```haskell +userServer :: Server UserApi Handler +userServer = createUser + where creatUser userId + | userId > 5000 = pure $ User { name = "some user", user_id = userId } + | otherwise = throwError $ err400 { errBody = "userId is too small" } +``` + +### Strategy 1: Spin Up a Server, Create a Client, Make Some Requests + +One of the benefits of Servant's type-level DSL for describing APIs is that +once you have provided a type-level description of your API, you can create +clients, documentation, or other tools for it somewhat magically. + +In this case, we'd like to *test* our server, so we can use `servant-client` +to create a client, after which we'll run our server, and then make requests +of it and see how it responds. + +Let's write some tests: + +```haskell +businessLogicSpec :: Spec +businessLogicSpec = do + -- create a test client function + createUser = client (Proxy :: Proxy UserApi) + -- create a servant-client ClientEnv + baseUrl <- parseBaseUrl "http://localhost:8888" + manager <- newManager defaultManagerSettings + let clientEnv = mkClientEnv manager baseUrl + + -- Run the server in another thread. + liftIO $ C.forkIO $ Warp.run 8888 (server userServer) + + -- testing scenarios start here + describe "POST /user" $ do + it "should create a user with a high enough ID" $ + result <- runClientM clientEnv (createUser 50001) + result `shouldEqual` Right $ User { name = "some_user", user_id = "5001 "} + it "should fail with a too-small ID" $ + result <- runClientM clientEnv (createUser 4999) + result `shouldEqual` Right $ User { name = "some_user", user_id = "5001 "} +``` + +### Running These Tests + +Let's run our tests and see what happens: + +``` +$ cabal new-test all +``` + +Great: we passed! Servers obviously get more complex, though, and we may not +wish to create a whole suite of clients for our server every time. In our next +scenario we're going to mock out a 3rd-party resource that our server talks to +and we're going to be using `hspec-wai` to run our `Application` instance and +to make requests. + +## *Mocking* 3rd Party Resources + +Often our web applications will need to make their own web +requests to other 3rd-party applications. These requests provide a lot +of opportunity for failure and so we'd like to test that the right +messages and failure values (in addition to success values) are returned +from our application. + +### Define the 3rd-Party Resource + +With Servant's type-level API definitions, assuming you've already defined the +API you want to mock, it's relatively trivial to create a simple server for +the purposes of running tests. For instance, consider an API server that needs +to get data out of Elasticsearch. Let's first define the Elasticsearch server +and client using Servant API descriptions: + +```haskell +type SearchAPI = + -- We're using Aeson's Generic JSON `Value` to make things easier on + -- ourselves. We're also representing only one Elasticsearch endpoint: + -- get item by id + "myIndex" :> "myDocType" :> Capture "docId" Integer :> Get '[JSON] Value + +-- Here's our Servant Client function +getDocument = client (Proxy :: Proxy SearchAPI) + +-- We can use these helpers when we want to make requests +-- using our client function +clientEnv :: Text -> Text -> IO ClientEnv +clientEnv esHost esPort = do + baseUrl <- parseBaseUrl $ T.unpack $ esHost <> ":" <> esPort + manager <- newManager defaultManagerSettings + pure $ mkClientEnv manager baseUrl + +runSearchClient :: Text -> Text -> ClientM a -> IO (Either ServantError a) +runSearchClient esHost esPort = (clientEnv esHost esPort >>=) . runClientM +``` + +### Servant Server Example Using this 3rd-Party Resource + +So we've got an Elasticsearch server and a client to talk to it with one +function that retrieves a document by its id. Let's now build a simple app +server that uses this client to retrieve documents. This is somewhat +contrived, but hopefully it illustrates the typical three-tier application +architecture. + +One note: we're also going to take advantage of `aeson-lens` here, which may +look a bit foreign. The gist of it is that we're going to traverse a JSON +`Value` from Elasticsearch and try to extract some kind of document to +return. + +```haskell +type DocApi = + "docs" :> Capture "docId" Integer :> Get '[JSON] Value + +docServer :: Text -> Text -> Server DocApi Handler +docServer esHost esPort = getDocById esHost esPort + +-- Our Handler tries to get a doc from Elasticsearch and then tries to parse +-- it. Unfortunately, there's a lot of opportunity for failure in these +-- actions +getDocById :: Text -> Text -> Integer -> Handler Value +getDocById esHost esPort docId = do + -- Our Servant Client function returns Either ServantError Value here: + docRes <- liftIO $ runSearchClient esHost esPort (getDocument docId) + case docRes of + Left err -> throwError $ err404 { errBody = "Failed looking up content" } + Right value -> do + -- we'll either fail to parse our document or we'll return it + case value ^? _Object . ix "_source" of + Nothing -> throwError $ err400 { errBody = "Failed parsing content" } + Just obj -> pure obj +``` + +### Testing Our Backend + +So the above represents our application. How shall we test this application? +Ideally, we'd like it to make requests of a real Elasticsearch server, but we +certainly don't want our tests to trigger requests to a live, production +database. In addition, we don't want to depend on our real Elasticsearch +server having specific, consistent results for us to test against, because +that would make our tests flaky (and flaky tests are sometimes described as +worse than not having tests at all). + +One solution to this is to create a trivial Elasticsearch server as part of +our testing code. We can do this relatively easily because we already have +an API definition for it above. With a *real* server, we can then let our own +application make requests of it and we'll simulate different scenarios in +order to make sure our application responds the way we expect it to. + +Let's start with some helpers which will allow us to run a testing version +of our Elasticsearch server in another thread: + +```haskell +-- | We'll run the Elasticsearch server so we can test behaviors +withElasticsearch :: IO () -> IO () +withElasticsearch action = + bracket (liftIO $ C.forkIO $ Warp.run 9999 esTestApp) + C.killThread + (const action) + +esTestApp :: Application +esTestApp = serve (Proxy :: Proxy SearchAPI) esTestServer + +esTestServer :: Server SearchAPI Handler +esTestServer = getESDocument + +getESDocument :: Integer -> Handler Value +getESDocument docId + -- arbitrary things we can trigger in our tests to check for failure + -- We want to try to trigger different code paths + | docId > 1000 = pure . Left $ ConnectionError "Bad connection!" + | docId > 500 = pure . Object $ HM.fromList [("bad", String "data")] + | otherwise = pure $ Object $ HM.fromList [("_source", defaultDocument)] +``` + +Now, we should be ready to write some tests. As mentioned above we're going +to use `hspec-wai` in this example to make test requests and to run our own +application. This should simplify our testing code somewhat: + +```haskell +thirdPartyResourcesSpec :: Spec +thirdPartyResourcesSpec = around_ withElasticsearch $ do + -- we call `with` and pass our servant-server `Application` + with (pure $ serve $ docServer "localhost" "9999") $ do + describe "GET /docs" $ do + it "should be able to get a document" $ + -- `get` is a function from hspec-wai`. + get "/docs/1" `shouldRespondWith` 200 + it "should be able to handle connection failures" $ + -- We can also make custom HTTP requests with the `request` function + get "/docs/1001" `shouldRespondWith` 404 + it "should be able to handle parsing failures" $ + get "/docs/501" `shouldRespondWith` 400 + it "should be able to handle odd HTTP requests" $ + -- we can also make all kinds of arbitrary custom requests to see how + -- our server responds using the `request` function: + -- request :: Method -> ByteString -> [Header] + -- -> LB.ByteString -> WaiSession SResponse + request methodPost "/docs/501" [] "{" `shouldRespondWith` 415 +``` + + +## Servant Quickcheck + +[`servant-quickcheck`](https://github.com/haskell-servant/servant-quickcheck) +is a project that allows users to write tests for whole Servant APIs using +quickcheck-style property-checking mechanisms. + +`servant-quickcheck` is great for asserting whole-API rules, such as "no +endpoint throws a 500" or "all 301 status codes also come with a Location +header". The project even comes with a number of predicates that reference +the [RFCs they originate from](https://github.com/haskell-servant/servant-quickcheck/blob/master/src/Servant/QuickCheck/Internal/Predicates.hs). + + +### Quickcheckable API + +Let's make an API and a server to demonstrate how to use `servant-quickcheck`: + +```haskell +type API = ReqBody '[JSON] String :> Post '[JSON] String + :<|> Get '[JSON] Int + :<|> BasicAuth "some-realm" () :> Get '[JSON] () + +api :: Proxy API +api = Proxy + +server :: IO (Server API) +server = do + mvar <- newMVar "" + return $ (\x -> liftIO $ swapMVar mvar x) + :<|> (liftIO $ readMVar mvar >>= return . length) + :<|> (const $ return ()) +``` + +### Using `servant-quickcheck` + +`servant-quickcheck` also has a cool mechanism where you can compare two API +servers to demonstrate that they respond identically to requests. This may be +useful if you are planning to rewrite one API in another language or with +another web framework. You have to specify whether you're looking for +`jsonEquality` vs regular `ByteString` equality, though: + +```haskell +servantQuickcheckSpec :: Spec +servantQuickcheckSpec = describe "" $ do + it "API demonstrates best practices" $ + withServantServer api server $ \burl -> + serverSatisfies api burl args (unauthorizedContainsWWWAuthenticate + <%> not500 + <%> onlyJsonObjects + <%> mempty) + + it "API doesn't have these things implemented yet" $ + withServantServer api server $ \burl -> do + serverDoesntSatisfy api burl args (getsHaveCacheControlHeader + <%> notAllowedContainsAllowHeader + <%> mempty) +``` \ No newline at end of file diff --git a/doc/cookbook/testing/testing.cabal b/doc/cookbook/testing/testing.cabal new file mode 100644 index 00000000..a60578b2 --- /dev/null +++ b/doc/cookbook/testing/testing.cabal @@ -0,0 +1,33 @@ +name: cookbook-testing +version: 0.0.1 +synopsis: Common testing patterns in Servant apps +description: This recipe includes various strategies for writing tests for Servant. +homepage: http://haskell-servant.readthedocs.org/ +license: BSD3 +license-file: ../../../servant/LICENSE +author: Servant Contributors +maintainer: haskell-servant-maintainers@googlegroups.com +category: Servant +build-type: Simple +cabal-version: >=1.10 +tested-with: GHC==8.0.2, GHC==8.2.2, GHC==8.4.3 + +executable cookbook-testing + main-is: Testing.lhs + build-depends: base == 4.* + , text >= 1.2 + , aeson >= 1.2 + , servant + , servant-client + , servant-server + , servant-quickcheck + , http-types >= 0.12 + , hspec + , hspec-wai + , QuickCheck + , warp >= 3.2 + , wai >= 3.2 + , wai-extra + default-language: Haskell2010 + ghc-options: -Wall -pgmL markdown-unlit + build-tool-depends: markdown-unlit:markdown-unlit From 89336aee96ffc343dbe28b1e7575f95aa53c7475 Mon Sep 17 00:00:00 2001 From: Erik Aker Date: Fri, 12 Oct 2018 08:48:25 -0700 Subject: [PATCH 2/7] Make tests run and include failings for illustrative purposes --- doc/cookbook/testing/Testing.lhs | 243 +++++++++++++++++++++-------- doc/cookbook/testing/testing.cabal | 4 + 2 files changed, 186 insertions(+), 61 deletions(-) diff --git a/doc/cookbook/testing/Testing.lhs b/doc/cookbook/testing/Testing.lhs index 241a1bb0..77bc273a 100644 --- a/doc/cookbook/testing/Testing.lhs +++ b/doc/cookbook/testing/Testing.lhs @@ -1,7 +1,7 @@ # How To Test Servant Applications Even with a nicely structured API that passes Haskell's strict type checker, -it's a good idea to write some tests for your application. +it's a good idea to write some tests for your application. In this recipe we'll work through some common testing strategies and provide examples of utlizing these testing strategies in order to test Servant @@ -13,17 +13,27 @@ This recipe uses the following ingredients: {-# LANGUAGE OverloadedStrings, TypeFamilies, DataKinds, DeriveGeneric, TypeOperators #-} import qualified Control.Concurrent as C +import Control.Concurrent.MVar +import Control.Exception (bracket) +import Control.Lens hiding (Context) import Data.Aeson import Data.Aeson.Lens +import qualified Data.HashMap.Strict as HM import Data.Text +import GHC.Generics +import Network.HTTP.Client hiding (Proxy) +import Network.HTTP.Types import Network.Wai import qualified Network.Wai.Handler.Warp as Warp + import Servant import Servant.Client import Servant.Server +import Servant.QuickCheck +import Servant.QuickCheck.Internal (serverDoesntSatisfy) import Test.Hspec -import Test.Hspec.Wai +import Test.Hspec.Wai import Test.Hspec.Wai.Matcher ``` @@ -33,13 +43,13 @@ aspects of our application, and we'll ask `hspec` to run all of our different ```haskell spec :: Spec -spec = +spec = do businessLogicSpec thirdPartyResourcesSpec servantQuickcheckSpec ``` -Often, codebases will use `hspec`'s +Often, codebases will use `hspec`'s [autodiscover pragma](http://hspec.github.io/hspec-discover.html) to find all testing modules and `Spec`s inside, but we're going to explicitly make a `main` function to run our tests because we have only one @@ -55,15 +65,15 @@ main = hspec spec Let's say we have an API that looks something like this: ```haskell -data User = User { - name :: Text - , user_id :: Integer +data User = User { + name :: Text + , user_id :: Integer } deriving (Eq, Show, Generic) instance FromJSON User instance ToJSON User -type UserApi = +type UserApi = -- One endpoint: create a user "user" :> Capture "userId" Integer :> Post '[JSON] User ``` @@ -74,11 +84,14 @@ to test. With that said, here's a sample handler for the endpoint described above: ```haskell -userServer :: Server UserApi Handler +userServer :: Server UserApi userServer = createUser - where creatUser userId - | userId > 5000 = pure $ User { name = "some user", user_id = userId } - | otherwise = throwError $ err400 { errBody = "userId is too small" } + +createUser :: Integer -> Handler User +createUser userId = do + if userId > 5000 + then pure $ User { name = "some user", user_id = userId } + else throwError $ err400 { errBody = "userId is too small" } ``` ### Strategy 1: Spin Up a Server, Create a Client, Make Some Requests @@ -97,23 +110,23 @@ Let's write some tests: businessLogicSpec :: Spec businessLogicSpec = do -- create a test client function - createUser = client (Proxy :: Proxy UserApi) + let createUser = client (Proxy :: Proxy UserApi) -- create a servant-client ClientEnv - baseUrl <- parseBaseUrl "http://localhost:8888" - manager <- newManager defaultManagerSettings + baseUrl <- runIO $ parseBaseUrl "http://localhost:8888" + manager <- runIO $ newManager defaultManagerSettings let clientEnv = mkClientEnv manager baseUrl - -- Run the server in another thread. - liftIO $ C.forkIO $ Warp.run 8888 (server userServer) + -- Run the server in another thread (`runIO` is from `hspec`) + runIO $ C.forkIO $ Warp.run 8888 (serve (Proxy :: Proxy UserApi) userServer) -- testing scenarios start here describe "POST /user" $ do - it "should create a user with a high enough ID" $ - result <- runClientM clientEnv (createUser 50001) - result `shouldEqual` Right $ User { name = "some_user", user_id = "5001 "} - it "should fail with a too-small ID" $ - result <- runClientM clientEnv (createUser 4999) - result `shouldEqual` Right $ User { name = "some_user", user_id = "5001 "} + it "should create a user with a high enough ID" $ do + result <- runClientM (createUser 50001) clientEnv + result `shouldBe` (Right $ User { name = "some user", user_id = 50001}) + it "will it fail with a too-small ID?" $ do + result <- runClientM (createUser 4999) clientEnv + result `shouldBe` (Right $ User { name = "some user", user_id = 50001}) ``` ### Running These Tests @@ -122,13 +135,26 @@ Let's run our tests and see what happens: ``` $ cabal new-test all +POST /user + should create a user with a high enough ID + should fail with a too-small ID FAILED [1] + + Failures: + + Testing.lhs:129:7: + 1) POST /user should fail with a too-small ID + expected: Right (User {name = "some user", user_id = 50001}) + but got: Left (FailureResponse (Response {responseStatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, responseHeaders = fromList [("Transfer-Encoding","chunked"),("Date","Fri, 12 Oct 2018 04:36:22 GMT"),("Server","Warp/3.2.25")], responseHttpVersion = HTTP/1.1, responseBody = "userId is too small"})) + + To rerun use: --match "/POST /user/should fail with a too-small ID/" ``` -Great: we passed! Servers obviously get more complex, though, and we may not -wish to create a whole suite of clients for our server every time. In our next -scenario we're going to mock out a 3rd-party resource that our server talks to -and we're going to be using `hspec-wai` to run our `Application` instance and -to make requests. +Hmm. One passed and one failed! It looks like I *was* expecting a success +response in the second test, but I actually got a failure. We should fix that, +but first I'd like to introduce `hspec-wai`, which will give us different +mechanisms for making requests of our application and validating the responses +we get. + ## *Mocking* 3rd Party Resources @@ -147,7 +173,7 @@ to get data out of Elasticsearch. Let's first define the Elasticsearch server and client using Servant API descriptions: ```haskell -type SearchAPI = +type SearchAPI = -- We're using Aeson's Generic JSON `Value` to make things easier on -- ourselves. We're also representing only one Elasticsearch endpoint: -- get item by id @@ -156,11 +182,11 @@ type SearchAPI = -- Here's our Servant Client function getDocument = client (Proxy :: Proxy SearchAPI) --- We can use these helpers when we want to make requests +-- We can use these helpers when we want to make requests -- using our client function clientEnv :: Text -> Text -> IO ClientEnv clientEnv esHost esPort = do - baseUrl <- parseBaseUrl $ T.unpack $ esHost <> ":" <> esPort + baseUrl <- parseBaseUrl $ unpack $ esHost <> ":" <> esPort manager <- newManager defaultManagerSettings pure $ mkClientEnv manager baseUrl @@ -170,22 +196,21 @@ runSearchClient esHost esPort = (clientEnv esHost esPort >>=) . runClientM ### Servant Server Example Using this 3rd-Party Resource -So we've got an Elasticsearch server and a client to talk to it with one -function that retrieves a document by its id. Let's now build a simple app -server that uses this client to retrieve documents. This is somewhat -contrived, but hopefully it illustrates the typical three-tier application -architecture. +So we've got an Elasticsearch server and a client to talk to it. Let's now +build a simple app server that uses this client to retrieve documents. This +is somewhat contrived, but hopefully it illustrates the typical three-tier +application architecture. One note: we're also going to take advantage of `aeson-lens` here, which may -look a bit foreign. The gist of it is that we're going to traverse a JSON +look a bit foreign. The gist of it is that we're going to traverse a JSON `Value` from Elasticsearch and try to extract some kind of document to return. ```haskell -type DocApi = +type DocApi = "docs" :> Capture "docId" Integer :> Get '[JSON] Value -docServer :: Text -> Text -> Server DocApi Handler +docServer :: Text -> Text -> Server DocApi docServer esHost esPort = getDocById esHost esPort -- Our Handler tries to get a doc from Elasticsearch and then tries to parse @@ -206,9 +231,11 @@ getDocById esHost esPort docId = do ### Testing Our Backend -So the above represents our application. How shall we test this application? -Ideally, we'd like it to make requests of a real Elasticsearch server, but we -certainly don't want our tests to trigger requests to a live, production +So the above represents our application and is close to a server we may +actually deploy. How shall we test this application? + +Ideally, we'd like it to make requests of a *real* Elasticsearch server, but +we certainly don't want our tests to trigger requests to a live, production database. In addition, we don't want to depend on our real Elasticsearch server having specific, consistent results for us to test against, because that would make our tests flaky (and flaky tests are sometimes described as @@ -234,30 +261,34 @@ withElasticsearch action = esTestApp :: Application esTestApp = serve (Proxy :: Proxy SearchAPI) esTestServer -esTestServer :: Server SearchAPI Handler +esTestServer :: Server SearchAPI esTestServer = getESDocument getESDocument :: Integer -> Handler Value getESDocument docId -- arbitrary things we can trigger in our tests to check for failure -- We want to try to trigger different code paths - | docId > 1000 = pure . Left $ ConnectionError "Bad connection!" + | docId > 1000 = throwError err500 | docId > 500 = pure . Object $ HM.fromList [("bad", String "data")] - | otherwise = pure $ Object $ HM.fromList [("_source", defaultDocument)] + | otherwise = pure $ Object $ HM.fromList [("_source", Object $ HM.fromList [("a", String "b")])] ``` -Now, we should be ready to write some tests. As mentioned above we're going -to use `hspec-wai` in this example to make test requests and to run our own -application. This should simplify our testing code somewhat: +Now, we should be ready to write some tests. + +In this case, we're going to use `hspec-wai`, which will give us a simple way +to run our application, make requests, and make assertions against the +responses we receive. + +Hopefully, this will simplify our testing code: ```haskell thirdPartyResourcesSpec :: Spec thirdPartyResourcesSpec = around_ withElasticsearch $ do -- we call `with` and pass our servant-server `Application` - with (pure $ serve $ docServer "localhost" "9999") $ do + with (pure $ serve (Proxy :: Proxy DocApi) $ docServer "localhost" "9999") $ do describe "GET /docs" $ do it "should be able to get a document" $ - -- `get` is a function from hspec-wai`. + -- `get` is a function from hspec-wai`. get "/docs/1" `shouldRespondWith` 200 it "should be able to handle connection failures" $ -- We can also make custom HTTP requests with the `request` function @@ -267,11 +298,36 @@ thirdPartyResourcesSpec = around_ withElasticsearch $ do it "should be able to handle odd HTTP requests" $ -- we can also make all kinds of arbitrary custom requests to see how -- our server responds using the `request` function: - -- request :: Method -> ByteString -> [Header] + -- request :: Method -> ByteString -> [Header] -- -> LB.ByteString -> WaiSession SResponse - request methodPost "/docs/501" [] "{" `shouldRespondWith` 415 + request methodPost "/docs/501" [] "{" `shouldRespondWith` 405 + it "we can also do more with the Response using hspec-wai's matchers" $ + -- see also `MatchHeader` and JSON-matching tools as well... + get "/docs/1" `shouldRespondWith` 200 { matchBody = MatchBody bodyMatcher } + +bodyMatcher :: [Network.HTTP.Types.Header] -> Body -> Maybe String +bodyMatcher _ body = case (decode body :: Maybe Value) of + -- success in this case means we return `Nothing` + Just val | val == (Object $ HM.fromList [("a", String "b")]) -> Nothing + _ -> Just "This is how we represent failure: this message will be printed" + ``` +What happens when we run these tests? + +``` +$ cabal new-test all +... + +GET /docs + should be able to get a document + should be able to handle connection failures + should be able to handle parsing failures + should be able to handle odd HTTP requests + we can also do more with the Response using hspec-wai's matchers +``` + +Fortunately, they all passed! ## Servant Quickcheck @@ -284,6 +340,8 @@ endpoint throws a 500" or "all 301 status codes also come with a Location header". The project even comes with a number of predicates that reference the [RFCs they originate from](https://github.com/haskell-servant/servant-quickcheck/blob/master/src/Servant/QuickCheck/Internal/Predicates.hs). +Thus, it's one way to assert that your APIs conform to specs and best +practices. ### Quickcheckable API @@ -301,31 +359,94 @@ server :: IO (Server API) server = do mvar <- newMVar "" return $ (\x -> liftIO $ swapMVar mvar x) - :<|> (liftIO $ readMVar mvar >>= return . length) + :<|> (liftIO $ readMVar mvar >>= return . Prelude.length) :<|> (const $ return ()) ``` ### Using `servant-quickcheck` -`servant-quickcheck` also has a cool mechanism where you can compare two API -servers to demonstrate that they respond identically to requests. This may be -useful if you are planning to rewrite one API in another language or with -another web framework. You have to specify whether you're looking for -`jsonEquality` vs regular `ByteString` equality, though: +Let's build some tests for our API using `servant-quickcheck`. + +Similar to the above examples, we're going to create `Spec`s, but in this +case, we'll rely on a number of predicates available from `servant-quickcheck` +to see if our API server conforms to best practices: ```haskell +-- Let's set some QuickCheck values +args :: Args +args = defaultArgs { maxSuccess = 500 } + +-- Here's a Servant Context object we'll use +ctx :: Context '[BasicAuthCheck ()] +ctx = BasicAuthCheck (const . return $ NoSuchUser) :. EmptyContext + + servantQuickcheckSpec :: Spec servantQuickcheckSpec = describe "" $ do it "API demonstrates best practices" $ - withServantServer api server $ \burl -> + -- `withServerServer` and `withServantServerAndContext` come from `servant-quickcheck` + withServantServerAndContext api ctx server $ \burl -> + -- `serverSatisfies` and the predicates also come from `servant-quickcheck` serverSatisfies api burl args (unauthorizedContainsWWWAuthenticate <%> not500 <%> onlyJsonObjects <%> mempty) it "API doesn't have these things implemented yet" $ - withServantServer api server $ \burl -> do + withServantServerAndContext api ctx server $ \burl -> do serverDoesntSatisfy api burl args (getsHaveCacheControlHeader <%> notAllowedContainsAllowHeader <%> mempty) -``` \ No newline at end of file +``` + +Let's see what happens when we run these tests: + + +``` + API demonstrates best practices FAILED [2] ++++ OK, passed 500 tests. + API doesn't have these things implemented yet + + src/Servant/QuickCheck/Internal/QuickCheck.hs:143:11: + 2) Main[339:25] API demonstrates best practices + Failed: + Just Predicate failed + Predicate: onlyJsonObjects + + Response: + Status code: 200 + Headers: "Transfer-Encoding": "chunked" + "Date": "Fri, 12 Oct 2018 04:36:22 GMT" + "Server": "Warp/3.2.25" + "Content-Type": "application/json;charset=utf-8" + Body: "" + + To rerun use: --match "/Main[339:25]/API demonstrates best practices/" + +Randomized with seed 1046277487 + +Finished in 0.4306 seconds +``` + +Hmm. It looks like we *thought* our API only return JSON objects, which is a +best practice, but in fact, we *did* have an endpoint that returned an empty +body, which you can see in the printed response above: `Body: ""`. We should +consider revising our API to only return top-level JSON Objects in the future! + +### Other Cool Things + +`servant-quickcheck` also has a cool mechanism where you can compare two API +servers to demonstrate that they respond identically to requests. This may be +useful if you are planning to rewrite one API in another language or with +another web framework. You have to specify whether you're looking for +`jsonEquality` vs regular `ByteString` equality, though. + +## Conclusion + +There are lots of techniques for testing and we only covered a few here. + +Useful libraries such as `hspec-wai` have ways of testing Wai `Application`s +and sending requests to them, while Servant's type-level DSL for defining APIs +allows us to more easily mock out servers. Lastly, if you want a broad +overview of where your application fits in with regard to best practices, +consider using `servant-quickcheck`. \ No newline at end of file diff --git a/doc/cookbook/testing/testing.cabal b/doc/cookbook/testing/testing.cabal index a60578b2..9e923286 100644 --- a/doc/cookbook/testing/testing.cabal +++ b/doc/cookbook/testing/testing.cabal @@ -17,14 +17,18 @@ executable cookbook-testing build-depends: base == 4.* , text >= 1.2 , aeson >= 1.2 + , lens-aeson + , lens , servant , servant-client , servant-server , servant-quickcheck + , http-client , http-types >= 0.12 , hspec , hspec-wai , QuickCheck + , unordered-containers , warp >= 3.2 , wai >= 3.2 , wai-extra From 18456baac5f30a2959c002233e35e2c7af84140a Mon Sep 17 00:00:00 2001 From: Erik Aker Date: Fri, 12 Oct 2018 17:32:20 -0700 Subject: [PATCH 3/7] Clean up the language so it reads better --- doc/cookbook/testing/Testing.lhs | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/doc/cookbook/testing/Testing.lhs b/doc/cookbook/testing/Testing.lhs index 77bc273a..3d8d04c1 100644 --- a/doc/cookbook/testing/Testing.lhs +++ b/doc/cookbook/testing/Testing.lhs @@ -153,7 +153,8 @@ Hmm. One passed and one failed! It looks like I *was* expecting a success response in the second test, but I actually got a failure. We should fix that, but first I'd like to introduce `hspec-wai`, which will give us different mechanisms for making requests of our application and validating the responses -we get. +we get. We're also going to spin up a fake Elasticsearch server, so that our +server can think it's talking to a real database. ## *Mocking* 3rd Party Resources @@ -201,11 +202,13 @@ build a simple app server that uses this client to retrieve documents. This is somewhat contrived, but hopefully it illustrates the typical three-tier application architecture. -One note: we're also going to take advantage of `aeson-lens` here, which may +One note: we're also going to take advantage of `lens-aeson` here, which may look a bit foreign. The gist of it is that we're going to traverse a JSON `Value` from Elasticsearch and try to extract some kind of document to return. +Imagine, then, that this is our real server implementation: + ```haskell type DocApi = "docs" :> Capture "docId" Integer :> Get '[JSON] Value @@ -232,7 +235,7 @@ getDocById esHost esPort docId = do ### Testing Our Backend So the above represents our application and is close to a server we may -actually deploy. How shall we test this application? +actually deploy. How then shall we test this application? Ideally, we'd like it to make requests of a *real* Elasticsearch server, but we certainly don't want our tests to trigger requests to a live, production @@ -284,7 +287,7 @@ Hopefully, this will simplify our testing code: ```haskell thirdPartyResourcesSpec :: Spec thirdPartyResourcesSpec = around_ withElasticsearch $ do - -- we call `with` and pass our servant-server `Application` + -- we call `with` and pass our own servant-server `Application` with (pure $ serve (Proxy :: Proxy DocApi) $ docServer "localhost" "9999") $ do describe "GET /docs" $ do it "should be able to get a document" $ @@ -310,10 +313,10 @@ bodyMatcher _ body = case (decode body :: Maybe Value) of -- success in this case means we return `Nothing` Just val | val == (Object $ HM.fromList [("a", String "b")]) -> Nothing _ -> Just "This is how we represent failure: this message will be printed" - ``` -What happens when we run these tests? +Out of the box, `hspec-wai` provides a lot of useful tools for us to run tests +against our application. What happens when we run these tests? ``` $ cabal new-test all @@ -327,7 +330,8 @@ GET /docs we can also do more with the Response using hspec-wai's matchers ``` -Fortunately, they all passed! +Fortunately, they all passed! Let's move to another strategy: whole-API +testing. ## Servant Quickcheck @@ -335,13 +339,13 @@ Fortunately, they all passed! is a project that allows users to write tests for whole Servant APIs using quickcheck-style property-checking mechanisms. -`servant-quickcheck` is great for asserting whole-API rules, such as "no +`servant-quickcheck` is great for asserting API-wide rules, such as "no endpoint throws a 500" or "all 301 status codes also come with a Location header". The project even comes with a number of predicates that reference the [RFCs they originate from](https://github.com/haskell-servant/servant-quickcheck/blob/master/src/Servant/QuickCheck/Internal/Predicates.hs). -Thus, it's one way to assert that your APIs conform to specs and best -practices. +In other words, it's one way to assert that your APIs conform to specs and +best practices. ### Quickcheckable API @@ -389,7 +393,7 @@ servantQuickcheckSpec = describe "" $ do -- `serverSatisfies` and the predicates also come from `servant-quickcheck` serverSatisfies api burl args (unauthorizedContainsWWWAuthenticate <%> not500 - <%> onlyJsonObjects + <%> onlyJsonObjects -- this one isn't true! <%> mempty) it "API doesn't have these things implemented yet" $ @@ -428,7 +432,7 @@ Randomized with seed 1046277487 Finished in 0.4306 seconds ``` -Hmm. It looks like we *thought* our API only return JSON objects, which is a +Hmm. It looks like we *thought* our API only returned JSON objects, which is a best practice, but in fact, we *did* have an endpoint that returned an empty body, which you can see in the printed response above: `Body: ""`. We should consider revising our API to only return top-level JSON Objects in the future! From 0d0fd0de82d099e6f0c87c73398fb3036e38172f Mon Sep 17 00:00:00 2001 From: Erik Aker Date: Sat, 13 Oct 2018 06:35:56 -0700 Subject: [PATCH 4/7] Clean up the comments --- doc/cookbook/testing/Testing.lhs | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/doc/cookbook/testing/Testing.lhs b/doc/cookbook/testing/Testing.lhs index 3d8d04c1..352fdbbc 100644 --- a/doc/cookbook/testing/Testing.lhs +++ b/doc/cookbook/testing/Testing.lhs @@ -267,10 +267,12 @@ esTestApp = serve (Proxy :: Proxy SearchAPI) esTestServer esTestServer :: Server SearchAPI esTestServer = getESDocument +-- This is the *mock* handler we're going to use. We create it +-- here specifically to trigger different behavior in our tests. getESDocument :: Integer -> Handler Value getESDocument docId - -- arbitrary things we can trigger in our tests to check for failure - -- We want to try to trigger different code paths + -- arbitrary things we can use in our tests to simulate failure + -- We want to trigger different code paths | docId > 1000 = throwError err500 | docId > 500 = pure . Object $ HM.fromList [("bad", String "data")] | otherwise = pure $ Object $ HM.fromList [("_source", Object $ HM.fromList [("a", String "b")])] @@ -294,7 +296,6 @@ thirdPartyResourcesSpec = around_ withElasticsearch $ do -- `get` is a function from hspec-wai`. get "/docs/1" `shouldRespondWith` 200 it "should be able to handle connection failures" $ - -- We can also make custom HTTP requests with the `request` function get "/docs/1001" `shouldRespondWith` 404 it "should be able to handle parsing failures" $ get "/docs/501" `shouldRespondWith` 400 From 64f89f600ae4d529804d991031b68ec5cafefc2a Mon Sep 17 00:00:00 2001 From: Erik Aker Date: Sun, 14 Oct 2018 10:19:49 -0700 Subject: [PATCH 5/7] Use a different server-running method and add bulleted list of strategies at start --- doc/cookbook/testing/Testing.lhs | 101 ++++++++++++++++++++++--------- 1 file changed, 73 insertions(+), 28 deletions(-) diff --git a/doc/cookbook/testing/Testing.lhs b/doc/cookbook/testing/Testing.lhs index 352fdbbc..14e88ab3 100644 --- a/doc/cookbook/testing/Testing.lhs +++ b/doc/cookbook/testing/Testing.lhs @@ -7,7 +7,36 @@ In this recipe we'll work through some common testing strategies and provide examples of utlizing these testing strategies in order to test Servant applications. -This recipe uses the following ingredients: +## Testing strategies + +There are many testing strategies you may wish to employ when testing your +Servant application, but included below are three common testing patterns: + +- We'll use `servant-client` to derive client functions and then make valid +requests of our API, running in another thread. This is great for testing +that our **business logic** is correctly implemented with only valid HTTP +requests. + +- We'll also use `hspec-wai` to make **arbitrary HTTP requests**, in order to +test how our application may respond to invalid or otherwise unexpected +requests. + +- Finally, we can also use `servant-quickcheck` for **whole-API tests**, in order +to assert that our entire application conforms to **best practices**. + +## Useful Libraries + +The following libraries will often come in handy when we decide to test our +Servant applications: + +- [hspec](https://hspec.github.io/) +- [hspec-wai](http://hackage.haskell.org/package/hspec-wai) +- [QuickCheck](http://hackage.haskell.org/package/QuickCheck) +- [servant-quickcheck](https://hackage.haskell.org/package/servant-quickcheck) + +## Imports and Our Testing Module + +This recipe starts the following ingredients: ```haskell {-# LANGUAGE OverloadedStrings, TypeFamilies, DataKinds, @@ -80,10 +109,13 @@ type UserApi = A real server would likely use a database to store, retrieve, and validate users, but we're going to do something really simple merely to have something -to test. With that said, here's a sample handler for the endpoint described -above: +to test. With that said, here's a sample handler, server, and `Application` +for the endpoint described above: ```haskell +userApp :: Application +userApp = serve (Proxy :: Proxy UserApi) userServer + userServer :: Server UserApi userServer = createUser @@ -107,26 +139,34 @@ of it and see how it responds. Let's write some tests: ```haskell +withUserApp :: IO () -> IO () +withUserApp action = + -- we can spin up a server in another thread and kill that thread when done + -- in an exception-safe way + bracket (liftIO $ C.forkIO $ Warp.run 8888 userApp) + C.killThread + (const action) + + businessLogicSpec :: Spec -businessLogicSpec = do - -- create a test client function - let createUser = client (Proxy :: Proxy UserApi) - -- create a servant-client ClientEnv - baseUrl <- runIO $ parseBaseUrl "http://localhost:8888" - manager <- runIO $ newManager defaultManagerSettings - let clientEnv = mkClientEnv manager baseUrl +businessLogicSpec = + -- `around` will our Server before the tests and turn it off after + around_ withUserApp $ do + -- create a test client function + let createUser = client (Proxy :: Proxy UserApi) + -- create a servant-client ClientEnv + baseUrl <- runIO $ parseBaseUrl "http://localhost:8888" + manager <- runIO $ newManager defaultManagerSettings + let clientEnv = mkClientEnv manager baseUrl - -- Run the server in another thread (`runIO` is from `hspec`) - runIO $ C.forkIO $ Warp.run 8888 (serve (Proxy :: Proxy UserApi) userServer) - - -- testing scenarios start here - describe "POST /user" $ do - it "should create a user with a high enough ID" $ do - result <- runClientM (createUser 50001) clientEnv - result `shouldBe` (Right $ User { name = "some user", user_id = 50001}) - it "will it fail with a too-small ID?" $ do - result <- runClientM (createUser 4999) clientEnv - result `shouldBe` (Right $ User { name = "some user", user_id = 50001}) + -- testing scenarios start here + describe "POST /user" $ do + it "should create a user with a high enough ID" $ do + result <- runClientM (createUser 50001) clientEnv + result `shouldBe` (Right $ User { name = "some user", user_id = 50001}) + it "will it fail with a too-small ID?" $ do + result <- runClientM (createUser 4999) clientEnv + result `shouldBe` (Right $ User { name = "some user", user_id = 50001}) ``` ### Running These Tests @@ -271,8 +311,8 @@ esTestServer = getESDocument -- here specifically to trigger different behavior in our tests. getESDocument :: Integer -> Handler Value getESDocument docId - -- arbitrary things we can use in our tests to simulate failure - -- We want to trigger different code paths + -- arbitrary things we can use in our tests to simulate failure: + -- we want to trigger different code paths. | docId > 1000 = throwError err500 | docId > 500 = pure . Object $ HM.fromList [("bad", String "data")] | otherwise = pure $ Object $ HM.fromList [("_source", Object $ HM.fromList [("a", String "b")])] @@ -289,7 +329,7 @@ Hopefully, this will simplify our testing code: ```haskell thirdPartyResourcesSpec :: Spec thirdPartyResourcesSpec = around_ withElasticsearch $ do - -- we call `with` and pass our own servant-server `Application` + -- we call `with` from `hspec-wai` and pass *real* `Application` with (pure $ serve (Proxy :: Proxy DocApi) $ docServer "localhost" "9999") $ do describe "GET /docs" $ do it "should be able to get a document" $ @@ -450,8 +490,13 @@ another web framework. You have to specify whether you're looking for There are lots of techniques for testing and we only covered a few here. -Useful libraries such as `hspec-wai` have ways of testing Wai `Application`s +Useful libraries such as `hspec-wai` have ways of running Wai `Application`s and sending requests to them, while Servant's type-level DSL for defining APIs -allows us to more easily mock out servers. Lastly, if you want a broad -overview of where your application fits in with regard to best practices, -consider using `servant-quickcheck`. \ No newline at end of file +allows us to more easily mock out servers and to derive clients, which will +only craft valid requests. + +Lastly, if you want a broad overview of where your application fits in with +regard to best practices, consider using `servant-quickcheck`. + +This program is available as a cabal project +[here](https://github.com/haskell-servant/servant/tree/master/doc/cookbook/testing). \ No newline at end of file From 02da07f95f706e1bbbf3201ca1e49a3fd398db07 Mon Sep 17 00:00:00 2001 From: Erik Aker Date: Sun, 14 Oct 2018 11:04:37 -0700 Subject: [PATCH 6/7] small typo cleanups --- doc/cookbook/testing/Testing.lhs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/cookbook/testing/Testing.lhs b/doc/cookbook/testing/Testing.lhs index 14e88ab3..844bd90b 100644 --- a/doc/cookbook/testing/Testing.lhs +++ b/doc/cookbook/testing/Testing.lhs @@ -12,8 +12,8 @@ applications. There are many testing strategies you may wish to employ when testing your Servant application, but included below are three common testing patterns: -- We'll use `servant-client` to derive client functions and then make valid -requests of our API, running in another thread. This is great for testing +- We'll use `servant-client` to derive client functions and then send valid +requests to our API, running in another thread. This is great for testing that our **business logic** is correctly implemented with only valid HTTP requests. @@ -36,7 +36,7 @@ Servant applications: ## Imports and Our Testing Module -This recipe starts the following ingredients: +This recipe starts with the following ingredients: ```haskell {-# LANGUAGE OverloadedStrings, TypeFamilies, DataKinds, @@ -150,7 +150,7 @@ withUserApp action = businessLogicSpec :: Spec businessLogicSpec = - -- `around` will our Server before the tests and turn it off after + -- `around` will start our Server before the tests and turn it off after around_ withUserApp $ do -- create a test client function let createUser = client (Proxy :: Proxy UserApi) From bb365159f3efbc125d052b1828f87c2faa3d5a85 Mon Sep 17 00:00:00 2001 From: Erik Aker Date: Sun, 14 Oct 2018 13:27:33 -0700 Subject: [PATCH 7/7] Pull out instance for hopefully more clarity --- doc/cookbook/testing/Testing.lhs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/cookbook/testing/Testing.lhs b/doc/cookbook/testing/Testing.lhs index 844bd90b..a075a144 100644 --- a/doc/cookbook/testing/Testing.lhs +++ b/doc/cookbook/testing/Testing.lhs @@ -253,6 +253,9 @@ Imagine, then, that this is our real server implementation: type DocApi = "docs" :> Capture "docId" Integer :> Get '[JSON] Value +docsApp :: Text -> Text -> Application +docsApp esHost esPort = serve (Proxy :: Proxy DocApi) $ docServer esHost esPort + docServer :: Text -> Text -> Server DocApi docServer esHost esPort = getDocById esHost esPort @@ -330,7 +333,7 @@ Hopefully, this will simplify our testing code: thirdPartyResourcesSpec :: Spec thirdPartyResourcesSpec = around_ withElasticsearch $ do -- we call `with` from `hspec-wai` and pass *real* `Application` - with (pure $ serve (Proxy :: Proxy DocApi) $ docServer "localhost" "9999") $ do + with (pure $ docsApp "localhost" "9999") $ do describe "GET /docs" $ do it "should be able to get a document" $ -- `get` is a function from hspec-wai`.