Jekyll2018-11-27T03:40:26+00:00//feed.xmlAbhishek’s Tech DojoBuilding the Hangman Game using ScalaZ ZIO2018-11-03T12:00:00+00:002018-11-03T12:00:00+00:00//2018/11/03/Hangman-Game-Using-ZIO<p>I came across this <a href="https://www.youtube.com/watch?v=XONTFZ4afY0">fantastic talk</a> delivered by <a href="https://twitter.com/jdegoes">John De Goes</a> for <a href="https://www.meetup.com/meetup-group-kyiv-scala-group/">Scala Kyiv Meetup</a>. In this talk
he implements a small fun game called Hangman and in the process he teaches us the <a href="https://github.com/scalaz/scalaz-zio">ZIO</a> Library.</p>
<p>I found the audio of the talk to be a little choppy and so I decided to write this blog and summarize my learnings from the talk.</p>
<p>If you are reading this article, I’ll presume that you already know why we need things like ZIO. We wrap side effecting code into IO Monads to make it referentially transparent. This allows us to push all the side effects to the boundary of our application and thus the inner functional core of the program is preserved.</p>
<p>Let me explain the game a little before we start writing it. The computer will select a random word for you and will ask you to guess the word. You can guess one character at a time. Every time you guess a character which occurs in the word, all occurrences of that character will be revealed. If you guess all the characters which occur in the chosen word you win the game. If the number of guesses reaches 10 and you still haven’t guessed all characters, you lose.</p>
<p>This is a good example to learn about the IO Monad because there are lots of side effects in this game. We need to collect user input like their name, each character they guess. We need to inform them whether their guess is right or wrong and in the end we need to tell them whether they won or lost the game. All these operations are side effects. So how can we write this application in a purely functional way?</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sbt</span> <span class="k">new</span> <span class="n">scala</span><span class="o">/</span><span class="n">hello</span><span class="o">-</span><span class="n">world</span><span class="o">.</span><span class="n">g8</span>
</code></pre></div></div>
<p>It will ask you for the project name enter “hangman”. This will create a Hello World project for you. First thing to do is to replace the contents of the build.sbt with this one. The build.sbt which the hello world project generates has lots of comments and we need a simpler file.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">lazy</span> <span class="k">val</span> <span class="n">root</span> <span class="k">=</span> <span class="o">(</span><span class="n">project</span> <span class="n">in</span> <span class="n">file</span><span class="o">(</span><span class="s">"."</span><span class="o">)).</span>
<span class="n">settings</span><span class="o">(</span>
<span class="n">inThisBuild</span><span class="o">(</span><span class="nc">List</span><span class="o">(</span>
<span class="n">organization</span> <span class="o">:=</span> <span class="s">"com.abhi"</span><span class="o">,</span>
<span class="n">scalaVersion</span> <span class="o">:=</span> <span class="s">"2.12.7"</span><span class="o">,</span>
<span class="n">version</span> <span class="o">:=</span> <span class="s">"0.1.0-SNAPSHOT"</span>
<span class="o">)),</span>
<span class="n">name</span> <span class="o">:=</span> <span class="s">"hangman"</span><span class="o">,</span>
<span class="n">libraryDependencies</span> <span class="o">++=</span> <span class="nc">Seq</span><span class="o">(</span>
<span class="s">"org.scalaz"</span> <span class="o">%%</span> <span class="s">"scalaz-zio"</span> <span class="o">%</span> <span class="s">"0.3.1"</span>
<span class="o">),</span>
<span class="n">trapExit</span> <span class="o">:=</span> <span class="kc">false</span>
<span class="o">)</span>
</code></pre></div></div>
<p>Do a <code class="highlighter-rouge">sbt compile</code> here to make sure that the project compiles and all binaries are downloaded correctly. Remove the Main.scala file from the project and add Hangman.scala with the following content.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nn">scalaz.zio._</span>
<span class="k">import</span> <span class="nn">scalaz.zio.console._</span>
<span class="k">import</span> <span class="nn">java.io.IOException</span>
<span class="k">object</span> <span class="nc">Hangman</span> <span class="k">extends</span> <span class="nc">App</span> <span class="o">{</span>
<span class="k">def</span> <span class="n">run</span><span class="o">(</span><span class="n">args</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span> <span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">IOException</span>, <span class="kt">ExitStatus</span><span class="o">]</span> <span class="k">=</span> <span class="o">???</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Right off the bat we see a fundamental difference between this App and the regular App which we use in Scala. This doesn’t have the main method which returns Unit. Instead our main method is replaced by a run method which returns a value. This value represents our entire program. The runtime will lazily evaluate this program and thus our program will run.</p>
<p>Our whole program is a value and it should always return this value whether our program runs successfully or it encounters an error. We have to do this by help of the redeem function of ScalaZ ZIO. It helps us return a value from our program in case of errors. The code below returns the value of ExitStatus(1) in case of errors and ExitStatus(0) in case of success. Note that the left side of the IO is Nothing. Here we are indicating that our program never throws an exception. This program will always return an ExitStatus.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">run</span><span class="o">(</span><span class="n">args</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span> <span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">Nothing</span>, <span class="kt">ExitStatus</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="n">hangman</span><span class="o">.</span><span class="n">redeemPure</span><span class="o">(</span>
<span class="k">_</span> <span class="k">=></span> <span class="nc">ExitStatus</span><span class="o">.</span><span class="nc">ExitNow</span><span class="o">(</span><span class="mi">1</span><span class="o">),</span>
<span class="k">_</span> <span class="k">=></span> <span class="nc">ExitStatus</span><span class="o">.</span><span class="nc">ExitNow</span><span class="o">(</span><span class="mi">0</span><span class="o">)</span>
<span class="o">)</span>
<span class="o">}</span>
<span class="k">val</span> <span class="n">hangman</span> <span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">IOException</span>, <span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span> <span class="o">???</span>
</code></pre></div></div>
<p>We need a case class which holds the state of our game. Here we capture the name of the player in the name attribute. We need to capture all the guesses which the player has made in a Set and finally the word which the player has to guess. Furthermore, we have 3 functions which return us the number of failed guesses, whether the player has won or whether s/he has lost the game.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">case</span> <span class="k">class</span> <span class="nc">State</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">guesses</span><span class="k">:</span> <span class="kt">Set</span><span class="o">[</span><span class="kt">Char</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Set</span><span class="o">.</span><span class="n">empty</span><span class="o">[</span><span class="kt">Char</span><span class="o">],</span> <span class="n">word</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span> <span class="o">{</span>
<span class="k">final</span> <span class="k">def</span> <span class="n">failures</span> <span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="o">(</span><span class="n">guesses</span> <span class="o">--</span> <span class="n">word</span><span class="o">.</span><span class="n">toSet</span><span class="o">).</span><span class="n">size</span>
<span class="k">final</span> <span class="k">def</span> <span class="n">playerLost</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="n">failures</span> <span class="o">></span> <span class="mi">10</span>
<span class="k">final</span> <span class="k">def</span> <span class="n">playerWon</span> <span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="o">(</span><span class="n">word</span><span class="o">.</span><span class="n">toSet</span> <span class="o">--</span> <span class="n">guesses</span><span class="o">).</span><span class="n">size</span> <span class="o">==</span> <span class="mi">0</span>
<span class="o">}</span>
</code></pre></div></div>
<p>We need a long list of words from which we will randomly pick a word which our player has to guess. For this I have put a <a href="https://github.com/abhsrivastava/hangman/blob/master/src/main/resources/words.txt">words.txt</a> file in my github repo which has 999 words. You can add/remove words as you like.
You have to put this file in the src/resources folder. We need to load the contents of this file in a List and then we’ll randomly pick items from this list</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">Hangman</span> <span class="k">extends</span> <span class="nc">App</span> <span class="o">{</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">State</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">guesses</span><span class="k">:</span> <span class="kt">Set</span><span class="o">[</span><span class="kt">Char</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Set</span><span class="o">.</span><span class="n">empty</span><span class="o">[</span><span class="kt">Char</span><span class="o">],</span> <span class="n">word</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span> <span class="o">{</span>
<span class="k">final</span> <span class="k">def</span> <span class="n">failures</span> <span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="o">(</span><span class="n">guesses</span> <span class="o">--</span> <span class="n">word</span><span class="o">.</span><span class="n">toSet</span><span class="o">).</span><span class="n">size</span>
<span class="k">final</span> <span class="k">def</span> <span class="n">playerLost</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="n">failures</span> <span class="o">>=</span> <span class="mi">10</span>
<span class="k">final</span> <span class="k">def</span> <span class="n">playerWon</span> <span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="o">(</span><span class="n">word</span><span class="o">.</span><span class="n">toSet</span> <span class="o">--</span> <span class="n">guesses</span><span class="o">).</span><span class="n">size</span> <span class="o">==</span> <span class="mi">0</span>
<span class="o">}</span>
<span class="k">lazy</span> <span class="k">val</span> <span class="nc">Dictionary</span> <span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="n">scala</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="nc">Source</span><span class="o">.</span><span class="n">fromResource</span><span class="o">(</span><span class="s">"words.txt"</span><span class="o">).</span><span class="n">getLines</span><span class="o">.</span><span class="n">toList</span>
<span class="k">def</span> <span class="n">run</span><span class="o">(</span><span class="n">args</span><span class="k">:</span> <span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">])</span> <span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">Nothing</span>, <span class="kt">ExitStatus</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="n">hangman</span><span class="o">.</span><span class="n">redeemPure</span><span class="o">(</span>
<span class="k">_</span> <span class="k">=></span> <span class="nc">ExitStatus</span><span class="o">.</span><span class="nc">ExitNow</span><span class="o">(</span><span class="mi">1</span><span class="o">),</span>
<span class="k">_</span> <span class="k">=></span> <span class="nc">ExitStatus</span><span class="o">.</span><span class="nc">ExitNow</span><span class="o">(</span><span class="mi">0</span><span class="o">)</span>
<span class="o">)</span>
<span class="o">}</span>
<span class="k">val</span> <span class="n">hangman</span> <span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">IOException</span>, <span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span> <span class="o">???</span>
<span class="o">}</span>
</code></pre></div></div>
<p>We need a function to ask the name of the user. We do so by</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">getName</span> <span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">IOException</span>, <span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="k">for</span> <span class="o">{</span>
<span class="k">_</span> <span class="k"><-</span> <span class="n">putStrLn</span><span class="o">(</span><span class="s">"What is your name: "</span><span class="o">)</span>
<span class="n">name</span> <span class="k"><-</span> <span class="n">getStrLn</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">name</span>
</code></pre></div></div>
<p>Why to write code this way rather than simply doing a <code class="highlighter-rouge">StdLn.readLine</code>? With this approach we are only expressing the intent to read a line. We are not actually reading a line. The code to read the line is inside of IO and will be lazily evaluated at the edge of our application. And that’s the whole point of this application that we don’t any side effects inside our code we wrap them in the IO data structure and then evaluate them outside of the code. Our code just returns a value of type <code class="highlighter-rouge">IO[IOException, String]</code> it doesn’t actually read anything from the console.</p>
<p>The above code is a little verbose. We can shorten it by re-writing it as</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">getName</span> <span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">IOException</span>, <span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="n">putStrLn</span><span class="o">(</span><span class="s">"What is your name: "</span><span class="o">)</span> <span class="o">*></span> <span class="n">getStrLn</span>
</code></pre></div></div>
<p>We are telling the runtime that we don’t care about the output of PutStrLn. We evaluate the expressions in a sequence but discard the value of the first expression and keep only the output of the second.</p>
<p>We need a function to read the character being guessed by the player. This is very similar to getName. The only difference is that we have to do validation on the input data and if the validation fails we need to show an error and then ask for input again.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">getChoice</span> <span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">IOException</span>, <span class="kt">Char</span><span class="o">]</span> <span class="k">=</span> <span class="k">for</span> <span class="o">{</span>
<span class="n">line</span> <span class="k"><-</span> <span class="n">putStrLn</span><span class="o">(</span><span class="n">s</span><span class="s">"Please enter a letter"</span><span class="o">)</span> <span class="o">*></span> <span class="n">getStrLn</span>
<span class="n">char</span> <span class="k"><-</span> <span class="n">line</span><span class="o">.</span><span class="n">toLowerCase</span><span class="o">.</span><span class="n">trim</span><span class="o">.</span><span class="n">headOption</span> <span class="k">match</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">None</span> <span class="k">=></span> <span class="n">putStrLn</span><span class="o">(</span><span class="n">s</span><span class="s">"You did not enter a character"</span><span class="o">)</span> <span class="o">*></span> <span class="n">getChoice</span>
<span class="k">case</span> <span class="nc">Some</span><span class="o">(</span><span class="n">x</span><span class="o">)</span> <span class="k">=></span> <span class="nc">IO</span><span class="o">.</span><span class="n">now</span><span class="o">(</span><span class="n">x</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">char</span>
</code></pre></div></div>
<p>We also need a functional random number generator. This is also an interesting problem. Functional programs are required to return the same output value for a specific input. Like <code class="highlighter-rouge">add(2, 2)</code> will always return a 4. However, what to do about the random number generator? by design it’s supposed to return a different value each time it’s called with the same max value.</p>
<p>Once again we use the IO. We are using the <code class="highlighter-rouge">sync</code> method to wrap the scala.util.Random into the IO. Now our program doesn’t return a random number. it returns a value of type IO which lazily generates random numbers at the time of evaluation. No matter how many times you call the function above it will always return a value of type IO[Nothing, Int]. Thus, this code is referentially transparent because it returns our intent of generating a random number. Not the random number itself.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">nextInt</span><span class="o">(</span><span class="n">max</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span> <span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">Nothing</span>, <span class="kt">Int</span><span class="o">]</span> <span class="k">=</span> <span class="nc">IO</span><span class="o">.</span><span class="n">sync</span><span class="o">(</span><span class="n">scala</span><span class="o">.</span><span class="n">util</span><span class="o">.</span><span class="nc">Random</span><span class="o">.</span><span class="n">nextInt</span><span class="o">(</span><span class="n">max</span><span class="o">))</span>
</code></pre></div></div>
<p>Now that we have our random number generator, we need logic to randomly pick a word from out Dictionary</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">chooseWord</span><span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">IOException</span>, <span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="k">for</span> <span class="o">{</span>
<span class="n">rand</span> <span class="k"><-</span> <span class="n">nextInt</span><span class="o">(</span><span class="nc">Dictionary</span><span class="o">.</span><span class="n">length</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span> <span class="nc">Dictionary</span><span class="o">.</span><span class="n">lift</span><span class="o">(</span><span class="n">rand</span><span class="o">).</span><span class="n">getOrElse</span><span class="o">(</span><span class="s">"Bug in the program!"</span><span class="o">)</span>
</code></pre></div></div>
<p>We finally need one last helper method before we start writing the logic of the game. We need a method to render the State of the game. When the game starts we show “ - “ for each character. the player can count these and determine the total number of characters in the word. Now each time the player guesses the character correctly we display all occurrences of that character. Others are still rendered as “ - “. We also show all characters the player has guessed so far.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">renderState</span><span class="o">(</span><span class="n">state</span><span class="k">:</span> <span class="kt">State</span><span class="o">)</span> <span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">IOException</span>, <span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">word</span> <span class="k">=</span> <span class="n">state</span><span class="o">.</span><span class="n">word</span><span class="o">.</span><span class="n">toList</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">c</span> <span class="k">=></span>
<span class="k">if</span> <span class="o">(</span><span class="n">state</span><span class="o">.</span><span class="n">guesses</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">c</span><span class="o">))</span> <span class="n">s</span><span class="s">" $c "</span> <span class="k">else</span> <span class="s">" "</span>
<span class="o">).</span><span class="n">mkString</span><span class="o">(</span><span class="s">""</span><span class="o">)</span>
<span class="k">val</span> <span class="n">line</span> <span class="k">=</span> <span class="nc">List</span><span class="o">.</span><span class="n">fill</span><span class="o">(</span><span class="n">state</span><span class="o">.</span><span class="n">word</span><span class="o">.</span><span class="n">length</span><span class="o">)(</span><span class="s">" - "</span><span class="o">).</span><span class="n">mkString</span><span class="o">(</span><span class="s">""</span><span class="o">)</span>
<span class="k">val</span> <span class="n">guesses</span> <span class="k">=</span> <span class="s">" Guesses: "</span> <span class="o">+</span> <span class="n">state</span><span class="o">.</span><span class="n">guesses</span><span class="o">.</span><span class="n">toList</span><span class="o">.</span><span class="n">sorted</span><span class="o">.</span><span class="n">mkString</span><span class="o">(</span><span class="s">""</span><span class="o">)</span>
<span class="k">val</span> <span class="n">text</span> <span class="k">=</span> <span class="n">word</span> <span class="o">+</span> <span class="s">"\n"</span> <span class="o">+</span> <span class="n">line</span> <span class="o">+</span> <span class="s">"\n\n"</span> <span class="o">+</span> <span class="n">guesses</span> <span class="o">+</span> <span class="s">"\n"</span>
<span class="n">putStrLn</span><span class="o">(</span><span class="n">text</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>OK. So, now we have all our helper methods in place. So let’s write the main logic of the game. We start the game by greeting the player and then asking the player his/her name. Once we have that we randomly pick the word which the player has to guess and we render the create and render the initial state of the game. We finally start the game loop. The game loop asks the player to guess a new character.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">hangman</span> <span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">IOException</span>, <span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span> <span class="k">for</span> <span class="o">{</span>
<span class="k">_</span> <span class="k"><-</span> <span class="n">putStrLn</span><span class="o">(</span><span class="s">"Welcome to purely functional hangman"</span><span class="o">)</span>
<span class="n">name</span> <span class="k"><-</span> <span class="n">getName</span>
<span class="k">_</span> <span class="k"><-</span> <span class="n">putStrLn</span><span class="o">(</span><span class="n">s</span><span class="s">"Welcome $name. Let's begin!"</span><span class="o">)</span>
<span class="n">word</span> <span class="k"><-</span> <span class="n">chooseWord</span>
<span class="n">state</span> <span class="k">=</span> <span class="nc">State</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="nc">Set</span><span class="o">(),</span> <span class="n">word</span><span class="o">)</span>
<span class="k">_</span> <span class="k"><-</span> <span class="n">renderState</span><span class="o">(</span><span class="n">state</span><span class="o">)</span>
<span class="k">_</span> <span class="k"><-</span> <span class="n">gameLoop</span><span class="o">(</span><span class="n">state</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span><span class="o">()</span>
<span class="k">def</span> <span class="n">gameLoop</span><span class="o">(</span><span class="n">state</span><span class="k">:</span> <span class="kt">State</span><span class="o">)</span> <span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">IOException</span>, <span class="kt">State</span><span class="o">]</span> <span class="k">=</span> <span class="o">???</span>
</code></pre></div></div>
<p>In the gameLoop method we ask the player to make his/her guess. We update the game state with the guess the player made. We render the updated state of the game. Now we have to make a decision on whether to terminate the game or to keep playing. If the decision is to keep playing then we recursively call the game loop with the updated state. To check if we have to terminate the loop we look at the playerWon and playerLost properties.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">gameLoop</span><span class="o">(</span><span class="n">state</span><span class="k">:</span> <span class="kt">State</span><span class="o">)</span> <span class="k">:</span> <span class="kt">IO</span><span class="o">[</span><span class="kt">IOException</span>, <span class="kt">State</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">{</span>
<span class="n">guess</span> <span class="k"><-</span> <span class="n">getChoice</span>
<span class="n">state</span> <span class="k"><-</span> <span class="nc">IO</span><span class="o">.</span><span class="n">now</span><span class="o">(</span><span class="n">state</span><span class="o">.</span><span class="n">copy</span><span class="o">(</span><span class="n">guesses</span> <span class="k">=</span> <span class="n">state</span><span class="o">.</span><span class="n">guesses</span> <span class="o">+</span> <span class="n">guess</span><span class="o">))</span>
<span class="k">_</span> <span class="k"><-</span> <span class="n">renderState</span><span class="o">(</span><span class="n">state</span><span class="o">)</span>
<span class="n">loop</span> <span class="k"><-</span> <span class="k">if</span> <span class="o">(</span><span class="n">state</span><span class="o">.</span><span class="n">playerWon</span><span class="o">)</span> <span class="n">putStrLn</span><span class="o">(</span><span class="n">s</span><span class="s">"Congratulations ${state.name} you won the game!"</span><span class="o">).</span><span class="n">const</span><span class="o">(</span><span class="kc">false</span><span class="o">)</span>
<span class="k">else</span> <span class="k">if</span> <span class="o">(</span><span class="n">state</span><span class="o">.</span><span class="n">playerLost</span><span class="o">)</span> <span class="n">putStrLn</span><span class="o">(</span><span class="n">s</span><span class="s">"Sorry ${state.name} you lost the game. The word was ${state.word}"</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="k">_</span> <span class="k">=></span> <span class="kc">false</span><span class="o">).</span><span class="n">const</span><span class="o">(</span><span class="kc">false</span><span class="o">)</span>
<span class="k">else</span> <span class="k">if</span> <span class="o">(</span><span class="n">state</span><span class="o">.</span><span class="n">word</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">guess</span><span class="o">))</span> <span class="n">putStrLn</span><span class="o">(</span><span class="n">s</span><span class="s">"You guessed correctly!"</span><span class="o">).</span><span class="n">const</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span>
<span class="k">else</span> <span class="n">putStrLn</span><span class="o">(</span><span class="n">s</span><span class="s">"That's wrong. but keep trying!"</span><span class="o">).</span><span class="n">const</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span>
<span class="n">state</span> <span class="k"><-</span> <span class="k">if</span> <span class="o">(</span><span class="n">loop</span><span class="o">)</span> <span class="n">gameLoop</span><span class="o">(</span><span class="n">state</span><span class="o">)</span> <span class="k">else</span> <span class="nc">IO</span><span class="o">.</span><span class="n">now</span><span class="o">(</span><span class="n">state</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">state</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The game is pretty cute and I had a fun time playing it with my children. I hope to teach them the IO Monad :)
Thanks to <a href="https://twitter.com/jdegoes">John De Goes</a> and <a href="https://www.meetup.com/meetup-group-kyiv-scala-group/">Scala Kyiv Meetup</a> for spending time and effort in doing this.</p>
<p>The complete code of this game is located in my <a href="https://github.com/abhsrivastava/hangman">github</a> repo.</p>
<p>If you want to discover more such awesome technical talks for the Scala language, please subscribe to the <a href="https://old.reddit.com/r/ScalaConferenceVideos/">/r/ScalaConferenceVideos</a> channel.</p>Abhishek SrivastavaI came across this fantastic talk delivered by John De Goes for Scala Kyiv Meetup. In this talk he implements a small fun game called Hangman and in the process he teaches us the ZIO Library.Dependency injection of Akka Actors with Google Guice2017-11-03T12:00:00+00:002017-11-03T12:00:00+00:00//2017/11/03/Actors-With-Guice<p>I use Google Guice for dependency injection in most of my projects. I also use Akka Actors a lot for solving concurrency related tasks in my projects. I often write code like</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">a</span> <span class="k">=</span> <span class="n">injector</span><span class="o">.</span><span class="n">getInstance</span><span class="o">(</span><span class="n">classOf</span><span class="o">[</span><span class="kt">A</span><span class="o">])</span>
<span class="k">val</span> <span class="n">b</span> <span class="k">=</span> <span class="n">injector</span><span class="o">.</span><span class="n">getInstance</span><span class="o">(</span><span class="n">classOf</span><span class="o">[</span><span class="kt">B</span><span class="o">])</span>
<span class="k">val</span> <span class="n">c</span> <span class="k">=</span> <span class="n">injector</span><span class="o">.</span><span class="n">getInstance</span><span class="o">(</span><span class="n">classOf</span><span class="o">[</span><span class="kt">C</span><span class="o">])</span>
<span class="k">val</span> <span class="n">actorSystem</span> <span class="k">=</span> <span class="n">injector</span><span class="o">.</span><span class="n">getInstance</span><span class="o">(</span><span class="n">classOf</span><span class="o">[</span><span class="kt">ActorSystem</span><span class="o">])</span>
<span class="k">val</span> <span class="n">myActorRef</span> <span class="k">:</span> <span class="kt">ActorRef</span> <span class="o">=</span> <span class="n">actorSystem</span><span class="o">.</span><span class="n">actorOf</span><span class="o">(</span><span class="nc">Props</span><span class="o">(</span><span class="k">new</span> <span class="nc">MyActor</span><span class="o">(</span><span class="n">a</span><span class="o">,</span> <span class="n">b</span><span class="o">,</span> <span class="n">c</span><span class="o">)))</span>
</code></pre></div></div>
<p>And while this works. I still resent that now I have two ways of creating objects in my project. One via Guice approach of <code class="highlighter-rouge">injector.getInstance</code> and then a second approach which is specific for actors <code class="highlighter-rouge">actorOf(Props(...))</code> mechanism listed above. I always wished that I could use Guice consistently to get instances of actors and classes alike.</p>
<p>In this blog, we’ll try to do just that. Lets write up a minimal actor which will be used in this example.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="nn">com.abhi.logic</span>
<span class="k">class</span> <span class="nc">Logic</span> <span class="o">{</span>
<span class="k">def</span> <span class="n">add</span><span class="o">(</span><span class="n">i</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">j</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="n">i</span> <span class="o">+</span> <span class="n">j</span>
<span class="o">}</span>
<span class="k">package</span> <span class="nn">com.abhi.actor</span>
<span class="k">class</span> <span class="nc">MyActor</span><span class="o">(</span><span class="n">logic</span><span class="k">:</span> <span class="kt">Logic</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">Actor</span> <span class="o">{</span>
<span class="k">def</span> <span class="n">receive</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">case</span> <span class="n">msg</span><span class="k">:</span> <span class="kt">Msg</span> <span class="o">=></span> <span class="n">sender</span><span class="o">()</span> <span class="o">!</span> <span class="n">logic</span><span class="o">.</span><span class="n">add</span><span class="o">(</span><span class="n">msg</span><span class="o">.</span><span class="n">i</span><span class="o">,</span> <span class="n">msg</span><span class="o">.</span><span class="n">j</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">Msg</span><span class="o">(</span><span class="n">i</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">j</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span>
</code></pre></div></div>
<p>So I have written all my business logic in a simple class (which I can test easily) and now I am using my logic class as a dependency inside of the actor.</p>
<p>BTW, I am using the <a href="https://github.com/codingwell/scala-guice">codingwell/scala-guice</a> library to integrate Guice with my Scala project. Let us code up the Module required to perform all dependency injection.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">MyModule</span> <span class="k">extends</span> <span class="nc">AbstractModule</span> <span class="k">with</span> <span class="nc">ScalaModule</span><span class="o">{</span>
<span class="k">def</span> <span class="n">configure</span><span class="o">()</span> <span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="o">{</span>
<span class="n">bind</span><span class="o">[</span><span class="kt">ActorSystem</span><span class="o">].</span><span class="n">toInstance</span><span class="o">(</span><span class="nc">ActorSystem</span><span class="o">())</span>
<span class="n">bind</span><span class="o">[</span><span class="kt">Logic</span><span class="o">]</span>
<span class="o">}</span>
<span class="nd">@Provides</span>
<span class="nd">@Singleton</span>
<span class="nd">@Named</span><span class="o">(</span><span class="s">"MyActor"</span><span class="o">)</span>
<span class="k">def</span> <span class="n">getMyActor</span><span class="o">(</span><span class="n">actorSystem</span><span class="k">:</span> <span class="kt">ActorSystem</span><span class="o">,</span> <span class="n">logic</span><span class="k">:</span> <span class="kt">Logic</span><span class="o">)</span> <span class="k">=</span> <span class="o">{</span>
<span class="n">actorSystem</span><span class="o">.</span><span class="n">actorOf</span><span class="o">(</span><span class="nc">Props</span><span class="o">(</span><span class="k">new</span> <span class="nc">MyActor</span><span class="o">(</span><span class="n">logic</span><span class="o">)))</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>You can see that I am using the <code class="highlighter-rouge">@Named</code> approach to get the right instance. This is required because all actors in the end are created as <code class="highlighter-rouge">ActorRef</code> so we can’t use the <code class="highlighter-rouge">getInstance[T]</code> approach because the T is same for all actors. So we use names.</p>
<p>If our project had 100 actors. We will have one <code class="highlighter-rouge">@provides</code> method for each actor with a unique name. We can easily specify all our dependencies as parameters of this method. Since they have been bound earlier in the configure method, we get them transparently in the <code class="highlighter-rouge">@provides</code> method.</p>
<p>This also means that if tomorrow my actor starts depending on Logic1, Logic2, and Logic3 classes, I have just one place to make the change. the users of the actor don’t see the dependencies.</p>
<p>Now we need a small utility class which will make it easy for the clients to looking the named instances.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">ActorUtil</span> <span class="o">{</span>
<span class="k">implicit</span> <span class="k">class</span> <span class="nc">ActorInjector</span><span class="o">(</span><span class="n">injector</span><span class="k">:</span> <span class="kt">Injector</span><span class="o">)</span> <span class="k">=</span> <span class="o">{</span>
<span class="n">getActor</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span> <span class="k">:</span> <span class="kt">ActorRef</span> <span class="o">=</span> <span class="o">{</span>
<span class="n">injector</span><span class="o">.</span><span class="n">getInstance</span><span class="o">(</span><span class="nc">Key</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">classOf</span><span class="o">[</span><span class="kt">ActorRef</span><span class="o">],</span> <span class="nc">Names</span><span class="o">.</span><span class="n">named</span><span class="o">(</span><span class="n">name</span><span class="o">)))</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>We are creating this utility class so that our client code can easily lookup actors by their names. These names are what we have configured in the <code class="highlighter-rouge">@Provides</code> method for each actor.</p>
<p>Now let us write a client for our actors</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nn">ActorUtil._</span>
<span class="k">val</span> <span class="n">injector</span> <span class="k">=</span> <span class="nc">Guice</span><span class="o">.</span><span class="n">createInjector</span><span class="o">(</span><span class="k">new</span> <span class="nc">MyModule</span><span class="o">)</span>
<span class="k">val</span> <span class="n">myActor</span> <span class="k">=</span> <span class="n">injector</span><span class="o">.</span><span class="n">getActor</span><span class="o">(</span><span class="s">"MyActor"</span><span class="o">)</span>
<span class="k">val</span> <span class="n">msg</span> <span class="k">=</span> <span class="o">(</span><span class="n">myActor</span> <span class="o">?</span> <span class="nc">MyMsg</span><span class="o">(</span><span class="mi">10</span><span class="o">,</span> <span class="mi">20</span><span class="o">)).</span><span class="n">mapTo</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span>
</code></pre></div></div>
<p>If I was writing a class which needed the MyActor as a dependency. I could write</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Singleton</span>
<span class="k">class</span> <span class="nc">MyClass</span> <span class="nd">@Inject</span><span class="o">()(</span><span class="nd">@Named</span><span class="o">(</span><span class="s">"MyActor"</span><span class="o">)</span> <span class="n">myactor</span><span class="k">:</span> <span class="kt">ActorRef</span><span class="o">)</span> <span class="o">{</span>
<span class="n">myactor</span> <span class="o">!</span> <span class="nc">Msg</span><span class="o">(...)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>That’s pretty clean because now I can get instances of my actors using Guice mechanisms. This leads to more consistent coding style.</p>
<p>The full code of this example is located <a href="https://github.com/abhsrivastava/ActorGuice">here</a></p>Abhishek SrivastavaI use Google Guice for dependency injection in most of my projects. I also use Akka Actors a lot for solving concurrency related tasks in my projects. I often write code likeReplace SprayTestKit with a simple Scala DSL2017-10-27T12:00:00+00:002017-10-27T12:00:00+00:00//2017/10/27/Replace-SprayTestKit-With-Custom-DSL<p>I work on a Scala application which has several hundred test cases written in <a href="https://github.com/spray/spray/tree/master/spray-testkit/src">SprayTestKit</a>. One of the challenges in migrating this application to a new technology is that the moment we replace spray all our test cases break. (because test cases depend on spray). This makes the migration project more complex because we need to work with broken test cases during the migration. I wanted a way to decouple our test cases from SprayTestKit without having to write these all the test cases. Once the test cases are decoupled, we can use any technology on the server side and our test cases will immediately catch bugs if any contract is broken.</p>
<p>Let us look at the definition of a typical SprayTestKit test case which uses the DSL specified by the kit</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Get</span><span class="o">(</span><span class="s">"/foo/bar"</span><span class="o">,</span> <span class="s">"{'name': 'test', 'age': 20}"</span><span class="o">)</span> <span class="o">~></span> <span class="n">cookie</span> <span class="o">~></span> <span class="n">apiRoute</span> <span class="o">~></span> <span class="n">check</span> <span class="o">{</span>
<span class="n">responseAs</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="n">must</span> <span class="n">contain</span><span class="o">(</span><span class="s">"Say hello"</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>I really like this a lot because this makes it so easy to write test cases for web services. We will develop a DSL which looks similar to this, but doesn’t need spray framework. I will also make certain improvements to this DSL. The second parameter to the HTTP Verb methods takes in a string which must be the JSON representation of the request body. this forces me to call <code class="highlighter-rouge">asJson</code> methods again and again on my request objects. I will let my library handle the json conversion. So I will just pass my case classes as the second parameter to the HTTP Verb method. Second change is that the apiRoute parameter is a little redundant because the URL to the HTTP Verb method is the one which decides where the call will be made. So why to specify the same information twice?</p>
<p>So our DSL will look like this</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">case</span> <span class="k">class</span> <span class="nc">Input</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">age</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">Output</span><span class="o">(</span><span class="n">msg</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
<span class="nc">Get</span><span class="o">(</span><span class="s">"/foo/bar"</span><span class="o">,</span> <span class="nc">Input</span><span class="o">(</span><span class="s">"test"</span><span class="o">,</span> <span class="mi">20</span><span class="o">))</span> <span class="o">~></span> <span class="n">addCookie</span><span class="o">(</span><span class="s">"name"</span><span class="o">,</span> <span class="s">"value"</span><span class="o">)</span> <span class="o">~></span> <span class="n">check</span> <span class="o">{</span> <span class="n">resp</span> <span class="k">=></span>
<span class="k">val</span> <span class="n">output</span> <span class="k">=</span> <span class="n">responseAs</span><span class="o">[</span><span class="kt">Output</span><span class="o">](</span><span class="n">resp</span><span class="o">.</span><span class="n">body</span><span class="o">)</span>
<span class="n">assert</span><span class="o">(</span><span class="n">output</span> <span class="o">===</span> <span class="nc">Output</span><span class="o">(</span><span class="s">"Hello World foo"</span><span class="o">))</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Here you can assume that we are calling a web service written in any programming language which takes json representation of Input as a input parameter and returns json representation of Output as response. The web service need you to provide an authentication token via a cookie.</p>
<p>Another change I am made to the DSL is that the test cases doesn’t need to <code class="highlighter-rouge">Cookie</code> object because this will directly tie the test case to the Cookie object provided by my HTTP Library. Instead I use a function which takes two strings, and I will build the cookie internally. This means that my test cases don’t need to directly touch the Http library objects. This will enable me to easily switch my HTTP Library without changing my test cases again. Currently I will use <a href="http://http4s.org">Http4s</a>, but tomorrow I can use anything else without changing the test case.</p>
<p>So let’s get the easy part out. We need an enumeration which contains all the HTTP Status codes. I simply searched the web and created a simple scala enum which contains all the codes. this enum called StatusCodes can be found <a href="https://github.com/abhsrivastava/WebServiceTestKit/blob/master/src/main/scala/com/abhi/webservice/testkit/StatusCodes.scala">here</a>.</p>
<p>The first part of our DSL is the HTTP Verb function which creates the Request object.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">WebServiceTestKit</span> <span class="o">{</span>
<span class="k">private</span> <span class="k">def</span> <span class="n">toUri</span><span class="o">(</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">Uri</span> <span class="o">=</span> <span class="nc">Uri</span><span class="o">.</span><span class="n">unsafeFromString</span><span class="o">(</span><span class="n">url</span><span class="o">)</span>
<span class="k">private</span> <span class="k">def</span> <span class="n">toRequest</span><span class="o">(</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">method</span><span class="k">:</span> <span class="kt">Method</span><span class="o">)</span> <span class="k">=</span> <span class="nc">Request</span><span class="o">(</span><span class="n">uri</span> <span class="k">=</span> <span class="n">toUri</span><span class="o">(</span><span class="n">url</span><span class="o">),</span> <span class="n">method</span> <span class="k">=</span> <span class="n">method</span><span class="o">)</span>
<span class="c1">// HTTP Verbs
</span> <span class="k">def</span> <span class="nc">Post</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">t</span><span class="k">:</span> <span class="kt">T</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Request</span> <span class="o">=</span> <span class="n">toRequest</span><span class="o">(</span><span class="n">url</span><span class="o">,</span> <span class="nc">Method</span><span class="o">.</span><span class="nc">POST</span><span class="o">)</span>
<span class="k">def</span> <span class="nc">Get</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">t</span><span class="k">:</span> <span class="kt">T</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Request</span> <span class="o">=</span> <span class="n">toRequest</span><span class="o">(</span><span class="n">url</span><span class="o">,</span> <span class="nc">Method</span><span class="o">.</span><span class="nc">GET</span><span class="o">)</span>
<span class="k">def</span> <span class="nc">Put</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">t</span><span class="k">:</span> <span class="kt">T</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Request</span> <span class="o">=</span> <span class="n">toRequest</span><span class="o">(</span><span class="n">url</span><span class="o">,</span> <span class="nc">Method</span><span class="o">.</span><span class="nc">PUT</span><span class="o">)</span>
<span class="k">def</span> <span class="nc">Delete</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">t</span><span class="k">:</span> <span class="kt">T</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Request</span> <span class="o">=</span> <span class="n">toRequest</span><span class="o">(</span><span class="n">url</span><span class="o">,</span> <span class="nc">Method</span><span class="o">.</span><span class="nc">DELETE</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Great to with this, we created the first part of the DSL. We can create the Request object. However we need a way to convert our parameter of type T to JSON.</p>
<p>We will import Circe dependencies and then add the following code</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">WebServiceTestKit</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">client</span> <span class="k">=</span> <span class="nc">PooledHttp1Client</span><span class="o">()</span>
<span class="k">private</span> <span class="k">def</span> <span class="n">toUri</span><span class="o">(</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span><span class="k">:</span> <span class="kt">Uri</span> <span class="o">=</span> <span class="nc">Uri</span><span class="o">.</span><span class="n">unsafeFromString</span><span class="o">(</span><span class="n">url</span><span class="o">)</span>
<span class="k">private</span> <span class="k">def</span> <span class="n">toRequest</span><span class="o">(</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">method</span><span class="k">:</span> <span class="kt">Method</span><span class="o">)</span> <span class="k">=</span> <span class="nc">Request</span><span class="o">(</span><span class="n">uri</span> <span class="k">=</span> <span class="n">toUri</span><span class="o">(</span><span class="n">url</span><span class="o">),</span> <span class="n">method</span> <span class="k">=</span> <span class="n">method</span><span class="o">)</span>
<span class="c1">// HTTP Verbs
</span> <span class="k">def</span> <span class="nc">Post</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">t</span><span class="k">:</span> <span class="kt">T</span><span class="o">)(</span><span class="k">implicit</span> <span class="n">e</span><span class="k">:</span> <span class="kt">Encoder</span><span class="o">[</span><span class="kt">T</span><span class="o">])</span> <span class="k">:</span> <span class="kt">Request</span> <span class="o">=</span> <span class="n">toRequest</span><span class="o">(</span><span class="n">url</span><span class="o">,</span> <span class="nc">Method</span><span class="o">.</span><span class="nc">POST</span><span class="o">).</span><span class="n">withBody</span><span class="o">(</span><span class="n">t</span><span class="o">.</span><span class="n">asJson</span><span class="o">.</span><span class="n">noSpaces</span><span class="o">).</span><span class="n">unsafeRun</span><span class="o">()</span>
<span class="k">def</span> <span class="nc">Get</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">t</span><span class="k">:</span> <span class="kt">T</span><span class="o">)(</span><span class="k">implicit</span> <span class="n">e</span><span class="k">:</span> <span class="kt">Encoder</span><span class="o">[</span><span class="kt">T</span><span class="o">])</span> <span class="k">:</span> <span class="kt">Request</span> <span class="o">=</span> <span class="n">toRequest</span><span class="o">(</span><span class="n">url</span><span class="o">,</span> <span class="nc">Method</span><span class="o">.</span><span class="nc">GET</span><span class="o">).</span><span class="n">withBody</span><span class="o">(</span><span class="n">t</span><span class="o">.</span><span class="n">asJson</span><span class="o">.</span><span class="n">noSpaces</span><span class="o">).</span><span class="n">unsafeRun</span><span class="o">()</span>
<span class="k">def</span> <span class="nc">Put</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">t</span><span class="k">:</span> <span class="kt">T</span><span class="o">)(</span><span class="k">implicit</span> <span class="n">e</span><span class="k">:</span> <span class="kt">Encoder</span><span class="o">[</span><span class="kt">T</span><span class="o">])</span> <span class="k">:</span> <span class="kt">Request</span> <span class="o">=</span> <span class="n">toRequest</span><span class="o">(</span><span class="n">url</span><span class="o">,</span> <span class="nc">Method</span><span class="o">.</span><span class="nc">PUT</span><span class="o">).</span><span class="n">withBody</span><span class="o">(</span><span class="n">t</span><span class="o">.</span><span class="n">asJson</span><span class="o">.</span><span class="n">noSpaces</span><span class="o">).</span><span class="n">unsafeRun</span><span class="o">()</span>
<span class="k">def</span> <span class="nc">Delete</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">url</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">t</span><span class="k">:</span> <span class="kt">T</span><span class="o">)(</span><span class="k">implicit</span> <span class="n">e</span><span class="k">:</span> <span class="kt">Encoder</span><span class="o">[</span><span class="kt">T</span><span class="o">])</span> <span class="k">:</span> <span class="kt">Request</span> <span class="o">=</span> <span class="n">toRequest</span><span class="o">(</span><span class="n">url</span><span class="o">,</span> <span class="nc">Method</span><span class="o">.</span><span class="nc">DELETE</span><span class="o">).</span><span class="n">withBody</span><span class="o">(</span><span class="n">t</span><span class="o">.</span><span class="n">asJson</span><span class="o">.</span><span class="n">noSpaces</span><span class="o">).</span><span class="n">unsafeRun</span><span class="o">()</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This code compiles because its our job to pass the ecoder for type T as an implicit to the HTTP Verb methods. We will use Circe automatic type derivation to pass this ecoder without writing one.</p>
<p>Now let’s go for the second part of the DSL. We need a way to attach headers and cookies to this request object. We will use some functional programming so that we can build our DSL.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">addHeader</span> <span class="k">:</span> <span class="o">(</span><span class="kt">String</span><span class="o">,</span> <span class="kt">String</span><span class="o">)</span> <span class="k">=></span> <span class="nc">Request</span> <span class="k">=></span> <span class="nc">Request</span> <span class="k">=</span> <span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">value</span><span class="o">)</span> <span class="k">=></span> <span class="o">(</span><span class="n">req</span><span class="k">:</span> <span class="kt">Request</span><span class="o">)</span> <span class="k">=></span> <span class="n">req</span><span class="o">.</span><span class="n">putHeaders</span><span class="o">(</span><span class="nc">Header</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">value</span><span class="o">))</span>
<span class="k">val</span> <span class="n">addCookie</span><span class="k">:</span> <span class="o">(</span><span class="kt">String</span><span class="o">,</span> <span class="kt">String</span><span class="o">)</span> <span class="k">=></span> <span class="nc">Request</span> <span class="k">=></span> <span class="nc">Request</span> <span class="k">=</span> <span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">value</span><span class="o">)</span> <span class="k">=></span> <span class="o">(</span><span class="n">req</span><span class="k">:</span> <span class="kt">Request</span><span class="o">)</span> <span class="k">=></span> <span class="n">req</span><span class="o">.</span><span class="n">putHeaders</span><span class="o">(</span><span class="n">org</span><span class="o">.</span><span class="n">http4s</span><span class="o">.</span><span class="n">headers</span><span class="o">.</span><span class="nc">Cookie</span><span class="o">(</span><span class="nc">Cookie</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">value</span><span class="o">)))</span>
</code></pre></div></div>
<p>OK looks cryptic. Here we have a addHeader function. Which takes two strings as a parameter and returns another function as an output. the output function takes a http request object as input and returns a http request object as output (the output has the headers and cookies attached)</p>
<p>Now we need to build our <code class="highlighter-rouge">~></code> method which chains the output of the HTTP Verb method to the add Header method. We also need to apply our operation in infix style because our code is like <code class="highlighter-rouge">Post(...) ~> addHeader("foo", "bar")</code>. To meet this requirement we write an implicit class inside our WebServiceTestKit</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">implicit</span> <span class="k">class</span> <span class="nc">RequestOps</span><span class="o">(</span><span class="n">request</span><span class="k">:</span> <span class="kt">Request</span><span class="o">)</span> <span class="o">{</span>
<span class="k">def</span> <span class="o">~>(</span><span class="n">f</span><span class="k">:</span> <span class="kt">Request</span> <span class="o">=></span> <span class="nc">Request</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Request</span> <span class="o">=</span> <span class="n">f</span><span class="o">(</span><span class="n">request</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>When we apply the <code class="highlighter-rouge">~></code> operator on the Request object the compilation will fail. this will cause the compiler to look for an implicit conversion. it will convert our Request object to the RequestOps type. Luckily our functions like ‘addHeader’ and ‘addCookie’ are of type Request => Request. So we can easily do <code class="highlighter-rouge">Post(...) ~> addheader(...)</code>.</p>
<p>So the last piece now is the check method. We will apply the same strategy as we did for adding headers. We will have a function called check. Which takes in a function as a input parameter. This input parameter function accepts a Response object and then returns another function as output. the output function is of type Request => Unit.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">case</span> <span class="k">class</span> <span class="nc">WebTestKitResponse</span><span class="o">(</span><span class="n">status</span><span class="k">:</span> <span class="kt">StatusCodes.Value</span><span class="o">,</span> <span class="n">body</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
<span class="k">val</span> <span class="n">check</span> <span class="k">:</span> <span class="o">(</span><span class="kt">WebTestKitResponse</span> <span class="o">=></span> <span class="kt">Unit</span><span class="o">)</span> <span class="k">=></span> <span class="nc">Request</span> <span class="k">=></span> <span class="nc">Unit</span> <span class="k">=</span> <span class="o">(</span><span class="n">f</span><span class="o">)</span> <span class="k">=></span> <span class="o">(</span><span class="n">req</span><span class="k">:</span> <span class="kt">Request</span><span class="o">)</span> <span class="k">=></span> <span class="o">{</span>
<span class="k">val</span> <span class="n">response</span> <span class="k">=</span> <span class="n">client</span><span class="o">.</span><span class="n">fetch</span><span class="o">[</span><span class="kt">WebTestKitResponse</span><span class="o">](</span><span class="n">req</span><span class="o">)</span> <span class="o">{</span>
<span class="k">case</span> <span class="nc">Successful</span><span class="o">(</span><span class="n">resp</span><span class="o">)</span> <span class="k">=></span> <span class="n">resp</span><span class="o">.</span><span class="n">as</span><span class="o">[</span><span class="kt">String</span><span class="o">].</span><span class="n">map</span><span class="o">(</span><span class="n">b</span> <span class="k">=></span> <span class="nc">WebTestKitResponse</span><span class="o">(</span><span class="nc">StatusCodes</span><span class="o">.</span><span class="n">fromInt</span><span class="o">(</span><span class="mi">200</span><span class="o">),</span> <span class="n">b</span><span class="o">))</span>
<span class="k">case</span> <span class="n">fail</span> <span class="k">=></span> <span class="n">fail</span><span class="o">.</span><span class="n">as</span><span class="o">[</span><span class="kt">String</span><span class="o">].</span><span class="n">map</span><span class="o">(</span><span class="n">f</span> <span class="k">=></span> <span class="nc">WebTestKitResponse</span><span class="o">(</span><span class="nc">StatusCodes</span><span class="o">.</span><span class="n">fromInt</span><span class="o">(</span><span class="n">fail</span><span class="o">.</span><span class="n">status</span><span class="o">.</span><span class="n">code</span><span class="o">),</span> <span class="n">f</span><span class="o">))</span>
<span class="o">}.</span><span class="n">unsafeRun</span><span class="o">()</span>
<span class="n">f</span><span class="o">(</span><span class="n">response</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Our current <code class="highlighter-rouge">~></code> operator cannot be used for check function. because check function produces a function of type <code class="highlighter-rouge">Request => Unit</code>. but ~> expects <code class="highlighter-rouge">Request => Request</code>. So we will overload the <code class="highlighter-rouge">~></code> method as</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">implicit</span> <span class="k">class</span> <span class="nc">RequestOps</span><span class="o">(</span><span class="n">request</span><span class="k">:</span> <span class="kt">Request</span><span class="o">)</span> <span class="o">{</span>
<span class="k">def</span> <span class="o">~>(</span><span class="n">f</span><span class="k">:</span> <span class="kt">Request</span> <span class="o">=></span> <span class="nc">Request</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Request</span> <span class="o">=</span> <span class="n">f</span><span class="o">(</span><span class="n">request</span><span class="o">)</span>
<span class="k">def</span> <span class="o">~>(</span><span class="n">f</span><span class="k">:</span> <span class="kt">Request</span> <span class="o">=></span> <span class="nc">Unit</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">f</span><span class="o">(</span><span class="n">request</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>That’s it. Our <a href="https://github.com/abhsrivastava/WebServiceTestKit/blob/master/src/main/scala/com/abhi/webservice/testkit/WebServiceTestKit.scala">whole DSL</a> is less than 50 lines of code. Now we can write a simple client program that uses our DSL</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nn">com.abhi.webservice.testkit.WebServiceTestKit._</span>
<span class="k">import</span> <span class="nn">io.circe.generic.auto._</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">SayHello</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">SayHelloResponse</span><span class="o">(</span><span class="n">msg</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
<span class="k">object</span> <span class="nc">Example</span> <span class="k">extends</span> <span class="nc">App</span> <span class="o">{</span>
<span class="nc">Post</span><span class="o">(</span><span class="s">"/foo/bar"</span><span class="o">,</span> <span class="nc">SayHello</span><span class="o">(</span><span class="s">"abhishek"</span><span class="o">))</span> <span class="o">~></span> <span class="n">addHeader</span><span class="o">(</span><span class="s">"auth"</span><span class="o">,</span> <span class="s">"xxxyyyy"</span><span class="o">)</span> <span class="o">~></span> <span class="n">check</span> <span class="o">{</span> <span class="n">resp</span> <span class="k">=></span>
<span class="n">assert</span><span class="o">(</span><span class="n">resp</span><span class="o">.</span><span class="n">status</span> <span class="o">==</span> <span class="nc">StatusCodes</span><span class="o">.</span><span class="nc">OK</span><span class="o">)</span>
<span class="k">val</span> <span class="n">response</span> <span class="k">=</span> <span class="n">responseAs</span><span class="o">[</span><span class="kt">SayHelloResponse</span><span class="o">](</span><span class="n">resp</span><span class="o">.</span><span class="n">body</span><span class="o">)</span>
<span class="n">assert</span><span class="o">(</span><span class="n">response</span><span class="o">.</span><span class="n">msg</span> <span class="o">==</span> <span class="s">"Hello World abhishek"</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>It’s important not to forget to import <code class="highlighter-rouge">import io.circe.generic.auto._</code> because that gives us the Encoder of the cases classes for free. Otherwise we will have to manually write our encoders and that won’t be fun at all.</p>
<p>This DSL is not a drop in replacement for the SprayTestKit because we made some changes, so I still had to spend 1 day doing search and replace in my code to make minor code changes. But in the end all my 400+ test cases ran and i removed the dependency on spray client and spray test kit from my code.</p>
<p>I am uploading my code <a href="https://github.com/abhsrivastava/WebServiceTestKit">here</a>.</p>Abhishek SrivastavaI work on a Scala application which has several hundred test cases written in SprayTestKit. One of the challenges in migrating this application to a new technology is that the moment we replace spray all our test cases break. (because test cases depend on spray). This makes the migration project more complex because we need to work with broken test cases during the migration. I wanted a way to decouple our test cases from SprayTestKit without having to write these all the test cases. Once the test cases are decoupled, we can use any technology on the server side and our test cases will immediately catch bugs if any contract is broken.Reading and Writing data into RabbitMQ using Alpakka2017-10-03T12:00:00+00:002017-10-03T12:00:00+00:00//2017/10/03/Reading-Writing-Data-into-RabbitMQ-using-Alpakka<p>In <a href="https://abhsrivastava.github.io/2017/09/29/Scan-Cassandra-with-Alpakka/">Part 1</a> of this series we learnt how to use Alpakka Cassandra connector to scan entire cassandra table. In <a href="https://abhsrivastava.github.io/2017/10/02/Alpkka-File-CSV-Elastic/">part 2</a> of the series we saw how to use Elastic Connector of Alpakka to load data into ElasticSearch indexes. In part 3 we will look at how to read and write data into RabbitMQ using Alpakka.</p>
<p>We need to setup RabbitMQ for our quick prototype. Here are the steps which I used to stand up a RabbitMQ Test environment in minutes.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> brew install rabbitmq
vi /usr/local/etc/rabbitmq/rabbitmq-env.conf
</code></pre></div></div>
<p>I changed the contents of this file to</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CONFIG_FILE=/usr/local/etc/rabbitmq/rabbitmq
NODE_IP_ADDRESS=abhisheks-mini
NODENAME=rabbit@abhisheks-mini
</code></pre></div></div>
<p>Basically I removed the references to <code class="highlighter-rouge">127.0.0.1</code> and <code class="highlighter-rouge">localhost</code> and replaced with the machine name. I think this required if you’ll connect to the RabbitMQ Remotely.</p>
<p>now let us start the rabbitmq by <code class="highlighter-rouge">brew services start rabbitmq</code>. We can check whether the service started successfully by <code class="highlighter-rouge">rabbitmqctl status</code></p>
<p>BTW: most command line utilities are not in the PATH by default. So I had to edit my .bash_profile and add <code class="highlighter-rouge">/usr/local/sbin</code> directory to PATH variable.</p>
<p>Now, we need to setup our exchange, queue and its bindings.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rabbitmqctl add_user abhi abhi
rabbitmqctl set_user_tags abhi administrator
rabbitmqctl set_permissions -p abhi ".*" ".*" ".*"
rabbitmqctl add_vhost myvhost
rabbitmqctl set_permissions -p myvhost abhi ".*" ".*" ".*"
rabbitmqadmin declare exchange --vhost=myvhost name=exchange type=direct --user=abhi --password=abhi
rabbitmqadmin declare queue --vhost=myvhost name=queue durable=true --user=abhi --password=abhi
rabbitmqadmin declare binding --vhost=myvhost source=exchange destination=queue destination_type=queue routing_key="foobar" --user=abhi --password=abhi
</code></pre></div></div>
<p>Now if all the above steps go through successfully, then we can test our queue by publishing a test message</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rabbitmqadmin publish exchange=exchange --vhost=myvhost routing_key="foobar" --user=abhi --password=abhi payload="Hello World"
</code></pre></div></div>
<p>If this says “message published” then we are ready to roll.</p>
<p>So now let us pull in all the Alpakka connectors we need for this example. We need the file and CSV connectors to load our countrycapital.csv file and we need the AMQP connector for reading/writing data from/to RabbitMQ.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">name</span> <span class="o">:=</span> <span class="s">"AlpakkaAMQP"</span>
<span class="n">version</span> <span class="o">:=</span> <span class="s">"1.0"</span>
<span class="n">scalaVersion</span> <span class="o">:=</span> <span class="s">"2.12.3"</span>
<span class="n">libraryDependencies</span> <span class="o">++=</span> <span class="nc">Seq</span><span class="o">(</span>
<span class="s">"com.lightbend.akka"</span> <span class="o">%%</span> <span class="s">"akka-stream-alpakka-amqp"</span> <span class="o">%</span> <span class="s">"0.13"</span><span class="o">,</span>
<span class="s">"com.lightbend.akka"</span> <span class="o">%%</span> <span class="s">"akka-stream-alpakka-csv"</span> <span class="o">%</span> <span class="s">"0.13"</span><span class="o">,</span>
<span class="s">"com.lightbend.akka"</span> <span class="o">%%</span> <span class="s">"akka-stream-alpakka-file"</span> <span class="o">%</span> <span class="s">"0.13"</span>
<span class="o">)</span>
</code></pre></div></div>
<p>So first we will write some data into RabbitMQ. Jumping straight to the meat, let us crate the Akka Streams sink.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">queueName</span> <span class="k">=</span> <span class="s">"myqueue"</span>
<span class="k">val</span> <span class="n">queueDeclaration</span> <span class="k">=</span> <span class="nc">QueueDeclaration</span><span class="o">(</span><span class="n">queueName</span><span class="o">,</span> <span class="n">durable</span> <span class="k">=</span> <span class="kc">true</span><span class="o">)</span>
<span class="k">val</span> <span class="n">uri</span> <span class="k">=</span> <span class="s">"amqp://abhi:abhi@abhisheks-mini:5672/myvhost"</span>
<span class="k">val</span> <span class="n">settings</span> <span class="k">=</span> <span class="nc">AmqpSinkSettings</span><span class="o">(</span><span class="nc">AmqpConnectionUri</span><span class="o">(</span><span class="n">uri</span><span class="o">))</span>
<span class="o">.</span><span class="n">withRoutingKey</span><span class="o">(</span><span class="s">"foobar"</span><span class="o">)</span>
<span class="o">.</span><span class="n">withExchange</span><span class="o">(</span><span class="s">"exchange"</span><span class="o">)</span>
<span class="o">.</span><span class="n">withDeclarations</span><span class="o">(</span><span class="n">queueDeclaration</span><span class="o">)</span>
<span class="k">val</span> <span class="n">amqpSink</span> <span class="k">=</span> <span class="nc">AmqpSink</span><span class="o">.</span><span class="n">simple</span><span class="o">(</span><span class="n">settings</span><span class="o">)</span>
</code></pre></div></div>
<p>Quite straightfoward. We declare our queue, we create a settings object which contains the URI of the location where we want to connect. Ofcourse, we need to specify our routing key and our exchange.</p>
<p>This gives us a Akka Streams Sink which accepts a stream of objects of type <code class="highlighter-rouge">akka.util.ByteString</code></p>
<p>This is really convenient because we have already seen how to read our file as a stream of Bytestream in previous two blog entries.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">resource</span> <span class="k">=</span> <span class="n">getClass</span><span class="o">.</span><span class="n">getResource</span><span class="o">(</span><span class="s">"/countrycapital.csv"</span><span class="o">)</span>
<span class="k">val</span> <span class="n">path</span> <span class="k">=</span> <span class="nc">Paths</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">resource</span><span class="o">.</span><span class="n">toURI</span><span class="o">)</span>
<span class="k">val</span> <span class="n">source</span> <span class="k">=</span> <span class="nc">FileIO</span><span class="o">.</span><span class="n">fromPath</span><span class="o">(</span><span class="n">path</span><span class="o">)</span>
</code></pre></div></div>
<p>So now we have a source of type ByteSream and we have a sink which accepts a type of ByteStream. Time to build our graph and execute it.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">graph</span> <span class="k">=</span> <span class="nc">RunnableGraph</span><span class="o">.</span><span class="n">fromGraph</span><span class="o">(</span><span class="nc">GraphDSL</span><span class="o">.</span><span class="n">create</span><span class="o">(</span><span class="n">amqpSink</span><span class="o">){</span><span class="k">implicit</span> <span class="n">builder</span> <span class="k">=></span>
<span class="n">s</span> <span class="k">=></span>
<span class="k">import</span> <span class="nn">GraphDSL.Implicits._</span>
<span class="n">source</span> <span class="o">~></span> <span class="n">s</span><span class="o">.</span><span class="n">in</span>
<span class="nc">ClosedShape</span>
<span class="o">})</span>
<span class="k">val</span> <span class="n">future</span> <span class="k">=</span> <span class="n">graph</span><span class="o">.</span><span class="n">run</span><span class="o">()</span>
<span class="n">future</span><span class="o">.</span><span class="n">onComplete</span> <span class="o">{</span> <span class="k">_</span> <span class="k">=></span>
<span class="n">actorSystem</span><span class="o">.</span><span class="n">terminate</span><span class="o">()</span>
<span class="o">}</span>
<span class="nc">Await</span><span class="o">.</span><span class="n">result</span><span class="o">(</span><span class="n">actorSystem</span><span class="o">.</span><span class="n">whenTerminated</span><span class="o">,</span> <span class="nc">Duration</span><span class="o">.</span><span class="nc">Inf</span><span class="o">)</span>
</code></pre></div></div>
<p>Now in order to read the data we just wrote into our queue. we need to build a Akka Streams source for RabbitMQ.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">queueName</span> <span class="k">=</span> <span class="s">"queue"</span>
<span class="k">val</span> <span class="n">queueDeclaration</span> <span class="k">=</span> <span class="nc">QueueDeclaration</span><span class="o">(</span><span class="n">queueName</span><span class="o">,</span> <span class="n">durable</span> <span class="k">=</span> <span class="kc">true</span><span class="o">)</span>
<span class="k">val</span> <span class="n">uri</span> <span class="k">=</span> <span class="s">"amqp://abhi:abhi@abhisheks-mini:5672/myvhost"</span>
<span class="k">val</span> <span class="n">amqpUri</span> <span class="k">=</span> <span class="nc">AmqpConnectionUri</span><span class="o">(</span><span class="n">uri</span><span class="o">)</span>
<span class="k">val</span> <span class="n">namedQueueSourceSettings</span> <span class="k">=</span> <span class="nc">NamedQueueSourceSettings</span><span class="o">(</span><span class="n">amqpUri</span><span class="o">,</span> <span class="n">queueName</span><span class="o">).</span><span class="n">withDeclarations</span><span class="o">(</span><span class="n">queueDeclaration</span><span class="o">)</span>
<span class="k">val</span> <span class="n">source</span> <span class="k">=</span> <span class="nc">AmqpSource</span><span class="o">.</span><span class="n">atMostOnceSource</span><span class="o">(</span><span class="n">namedQueueSourceSettings</span><span class="o">,</span> <span class="n">bufferSize</span> <span class="k">=</span> <span class="mi">10</span><span class="o">)</span>
</code></pre></div></div>
<p>The code above is again straight forward. we provide our queue name, erver name, port and vhost. This gives us a source which reads data of type <code class="highlighter-rouge">IncommingMessage</code> we need to convert it into String. This can easily be done by our 3 flows (we have already used these in our previous blog entries)</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">flow1</span> <span class="k">=</span> <span class="nc">Flow</span><span class="o">[</span><span class="kt">IncomingMessage</span><span class="o">].</span><span class="n">map</span><span class="o">(</span><span class="n">msg</span> <span class="k">=></span> <span class="n">msg</span><span class="o">.</span><span class="n">bytes</span><span class="o">)</span>
<span class="k">val</span> <span class="n">flow2</span> <span class="k">=</span> <span class="nc">Flow</span><span class="o">[</span><span class="kt">ByteString</span><span class="o">].</span><span class="n">map</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">utf8String</span><span class="o">)</span>
<span class="k">val</span> <span class="n">sink</span> <span class="k">=</span> <span class="nc">Sink</span><span class="o">.</span><span class="n">foreach</span><span class="o">[</span><span class="kt">String</span><span class="o">](</span><span class="n">println</span><span class="o">)</span>
</code></pre></div></div>
<p>Once again. connect all the flows in a graph and execute it and you’ll see all the records being read from the RabbitMQ queue.</p>
<p>The entire example is available at my <a href="https://github.com/abhsrivastava/AlpakkaAMQP">github</a></p>Abhishek SrivastavaIn Part 1 of this series we learnt how to use Alpakka Cassandra connector to scan entire cassandra table. In part 2 of the series we saw how to use Elastic Connector of Alpakka to load data into ElasticSearch indexes. In part 3 we will look at how to read and write data into RabbitMQ using Alpakka.Alpakka File CSV and ElasticSearch Connectors2017-10-02T12:00:00+00:002017-10-02T12:00:00+00:00//2017/10/02/Alpkka-File-CSV-Elastic<p>In <a href="https://abhsrivastava.github.io/2017/09/29/Scan-Cassandra-with-Alpakka/">part 1</a> of this series I had shown how to use the Alpakka Cassandra Connector to scan the entire cassandra table.</p>
<p>Today we will look at 3 connectors. The File Connector, the CSV Connector and finally the ElasticSearch connector.</p>
<p>The scenario we will use is that you have a list of countries and capitals available as a <a href="https://github.com/icyrockcom/country-capitals/blob/master/data/country-list.csv">csv file</a> and we have to load these into ElasticSearch.</p>
<p>We will read the files using the File and CSV Connectors and then we will use ElasticSearch connector to create a Sink into ElasticSearch.</p>
<p>The build.sbt file is as follows</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">name</span> <span class="o">:=</span> <span class="s">"ElasticAkkaStreams"</span>
<span class="n">version</span> <span class="o">:=</span> <span class="s">"1.0"</span>
<span class="n">scalaVersion</span> <span class="o">:=</span> <span class="s">"2.12.3"</span>
<span class="n">libraryDependencies</span> <span class="o">++=</span> <span class="nc">Seq</span><span class="o">(</span>
<span class="s">"com.lightbend.akka"</span> <span class="o">%%</span> <span class="s">"akka-stream-alpakka-elasticsearch"</span> <span class="o">%</span> <span class="s">"0.13"</span><span class="o">,</span>
<span class="s">"com.lightbend.akka"</span> <span class="o">%%</span> <span class="s">"akka-stream-alpakka-csv"</span> <span class="o">%</span> <span class="s">"0.13"</span><span class="o">,</span>
<span class="s">"com.lightbend.akka"</span> <span class="o">%%</span> <span class="s">"akka-stream-alpakka-file"</span> <span class="o">%</span> <span class="s">"0.13"</span>
<span class="o">)</span>
<span class="n">mainClass</span> <span class="n">in</span> <span class="n">run</span> <span class="o">:=</span> <span class="nc">Some</span><span class="o">(</span><span class="s">"com.abhi.ElasticAkkaStreams"</span><span class="o">)</span>
</code></pre></div></div>
<p>Nothing surprising here. We are just importing the connectors we need.</p>
<p>Now Let us first read the files as a stream of strings.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">val</span> <span class="n">resource</span> <span class="k">=</span> <span class="n">getClass</span><span class="o">.</span><span class="n">getResource</span><span class="o">(</span><span class="s">"/countrycapital.csv"</span><span class="o">)</span>
<span class="k">val</span> <span class="n">path</span> <span class="k">=</span> <span class="nc">Paths</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">resource</span><span class="o">.</span><span class="n">toURI</span><span class="o">)</span>
<span class="k">val</span> <span class="n">source</span> <span class="k">=</span> <span class="nc">FileIO</span><span class="o">.</span><span class="n">fromPath</span><span class="o">(</span><span class="n">path</span><span class="o">)</span>
</code></pre></div></div>
<p>I have copied the country capitial csv in the resources folder. In order to read it we need to build the path to the resource and finally feed that path to the FileTailSource. The FileTailSource gives us a stream of Strings (one for each line).</p>
<p>Now We need to feed these lines to the CSV Connector for tokenization. the CSV Flow only accepts <code class="highlighter-rouge">akka.util.ByteString</code> so we need to convert our line to a ByteString</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">val</span> <span class="n">flow1</span> <span class="k">=</span> <span class="nc">Flow</span><span class="o">[</span><span class="kt">String</span><span class="o">].</span><span class="n">map</span><span class="o">(</span><span class="nc">ByteString</span><span class="o">(</span><span class="k">_</span><span class="o">))</span>
<span class="k">val</span> <span class="n">flow2</span> <span class="k">=</span> <span class="nc">CsvParsing</span><span class="o">.</span><span class="n">lineScanner</span>
<span class="k">val</span> <span class="n">flow3</span> <span class="k">=</span> <span class="nc">Flow</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">ByteString</span><span class="o">]].</span><span class="n">map</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">utf8String</span><span class="o">)</span>
</code></pre></div></div>
<p>In the code above, we first converted our String (containing a line) into a ByteString, then we gave that ByteString to the CSVParser. Which split the line into tokens (of type List[ByteString]) then we converted each token back to a string. At this point we can load the data into a domain object of <code class="highlighter-rouge">CountryCapital</code>.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">case</span> <span class="k">class</span> <span class="nc">CountryCapital</span><span class="o">(</span><span class="n">country</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">capital</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
<span class="k">val</span> <span class="n">flow4</span> <span class="k">=</span> <span class="nc">Flow</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]].</span><span class="n">map</span><span class="o">(</span><span class="n">list</span> <span class="k">=></span> <span class="nc">CountryCapital</span><span class="o">(</span><span class="n">list</span><span class="o">(</span><span class="mi">0</span><span class="o">),</span> <span class="n">list</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span>
</code></pre></div></div>
<p>The ElasticSearch Connector gives us a Sink to ElasticSearch. but that Sink accepts a stream of objects of type <code class="highlighter-rouge">IncommingMessage[JsObject]</code>. Here the type <code class="highlighter-rouge">JsObject</code> is coming from the spray-json library.</p>
<p>So we need to convert our domain object into the type <code class="highlighter-rouge">IncomingMessage[JsObject]</code> In order to facilitate this conversion we need to import a few classes which make it seamless to convert our CountryCaptial case class into spray-json JsObject.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">import</span> <span class="nn">DefaultJsonProtocol._</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">format</span> <span class="k">=</span> <span class="n">jsonFormat2</span><span class="o">(</span><span class="nc">CountryCapital</span><span class="o">)</span>
<span class="k">val</span> <span class="n">flow5</span> <span class="k">=</span> <span class="nc">Flow</span><span class="o">[</span><span class="kt">CountryCapital</span><span class="o">].</span><span class="n">map</span><span class="o">{</span><span class="n">cc</span> <span class="k">=></span> <span class="nc">IncommingMessage</span><span class="o">[</span><span class="kt">JsObject</span><span class="o">](</span><span class="nc">Some</span><span class="o">(</span><span class="nc">UUID</span><span class="o">.</span><span class="n">randomUUID</span><span class="o">.</span><span class="n">toString</span><span class="o">),</span> <span class="n">cc</span><span class="o">.</span><span class="n">asJson</span><span class="o">.</span><span class="n">asJsObject</span><span class="o">)}</span>
</code></pre></div></div>
<p>Now that wasn’t bad. Finally we can use Alpakka Elastic and build our sink</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">implicit</span> <span class="k">val</span> <span class="n">client</span> <span class="k">=</span> <span class="nc">RestClient</span><span class="o">.</span><span class="n">builder</span><span class="o">(</span><span class="k">new</span> <span class="nc">HttpHost</span><span class="o">(</span><span class="s">"abhisheks-mini"</span><span class="o">,</span> <span class="mi">9200</span><span class="o">)).</span><span class="n">build</span><span class="o">()</span>
<span class="k">val</span> <span class="n">sinkSettings</span> <span class="k">=</span> <span class="nc">ElasticsearchSinkSettings</span><span class="o">(</span><span class="n">bufferSize</span> <span class="k">=</span> <span class="mi">100000</span><span class="o">,</span> <span class="n">retryInterval</span> <span class="k">=</span> <span class="mi">5000</span><span class="o">,</span> <span class="n">maxRetry</span> <span class="k">=</span> <span class="mi">100</span><span class="o">)</span>
<span class="k">val</span> <span class="n">sink</span> <span class="k">=</span> <span class="nc">ElasticsearchSink</span><span class="o">(</span><span class="s">"myindex"</span><span class="o">,</span> <span class="s">"mytype"</span><span class="o">,</span> <span class="n">sinkSettings</span><span class="o">)</span>
</code></pre></div></div>
<p>The client object provides the server name and the port of ElasticSearch. Apart from that, the sink obviously needs the index and the type where ther data will be loaded.</p>
<p>Finally we need to build a Runnable Graph</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">implicit</span> <span class="k">val</span> <span class="n">actorSystem</span> <span class="k">=</span> <span class="nc">ActorSystem</span><span class="o">()</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">actorMaterializer</span> <span class="k">=</span> <span class="nc">ActorMaterializer</span><span class="o">()</span>
<span class="k">val</span> <span class="n">graph</span> <span class="k">=</span> <span class="nc">RunnableGraph</span><span class="o">.</span><span class="n">fromGraph</span><span class="o">(</span><span class="nc">GraphDSL</span><span class="o">.</span><span class="n">create</span><span class="o">(</span><span class="n">sink</span><span class="o">){</span><span class="k">implicit</span> <span class="n">builder</span> <span class="k">=></span>
<span class="n">s</span> <span class="k">=></span>
<span class="k">import</span> <span class="nn">GraphDSL.Implicits._</span>
<span class="n">source</span> <span class="o">~></span> <span class="n">flow1</span> <span class="o">~></span> <span class="n">flow2</span> <span class="o">~></span> <span class="n">flow3</span> <span class="o">~></span> <span class="n">flow4</span> <span class="o">~></span> <span class="n">flow5</span> <span class="o">~></span> <span class="n">s</span><span class="o">.</span><span class="n">in</span>
<span class="nc">ClosedShape</span>
<span class="o">})</span>
</code></pre></div></div>
<p>and then execute the graph</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">future</span> <span class="k">=</span> <span class="n">graph</span><span class="o">.</span><span class="n">run</span><span class="o">()</span>
<span class="n">future</span><span class="o">.</span><span class="n">onComplete</span> <span class="o">{</span> <span class="k">_</span> <span class="k">=></span> <span class="n">actorSystem</span><span class="o">.</span><span class="n">terminate</span><span class="o">()}</span>
<span class="nc">Await</span><span class="o">.</span><span class="n">result</span><span class="o">(</span><span class="n">actorSystem</span><span class="o">.</span><span class="n">whenTerminated</span><span class="o">,</span> <span class="nc">Duration</span><span class="o">.</span><span class="nc">Inf</span><span class="o">)</span>
</code></pre></div></div>
<p>Now in order to check whether the data is loaded or not. Issue the following GET request to your elasticsearch REST endpoint</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http://abhisheks-mini:9200/myindex/_search?pretty=true&q=*:*
</code></pre></div></div>
<p>In my case it showed me 231 rows.</p>
<p>One big Gotcha was that when you inject IncomingMessgaes into Elastic make sure that the ID is a GUID. For me the data load operaiton just loaded 51 rows when I was trying to use Country as a key.</p>
<p>The whole application can be found at my <a href="https://github.com/abhsrivastava/ElasticAlpakkaStreams">github</a>.</p>Abhishek SrivastavaIn part 1 of this series I had shown how to use the Alpakka Cassandra Connector to scan the entire cassandra table.Stream Avro Records into Kafka using Avro4s and Akka Streams Kafka2017-10-02T12:00:00+00:002017-10-02T12:00:00+00:00//2017/10/02/Stream-Avro-Records-into-Kafka<p>Continuing our quest to learn Akka Streams, we’ll take our same old <a href="https://github.com/icyrockcom/country-capitals/blob/master/data/country-list.csv">countrycapital.csv</a> and then we’ll convert each line into an AVRO Record and then write into a Kafka topic using Akka streams.</p>
<p>Why <a href="https://avro.apache.org/">Avro</a>. Avro is binary data serialization format. It is more concise than Json and is very interoperable. Almost all popular languages have Avro bindings and therfore can easily and efficiently read the data from Kafka topic.</p>
<p>Without any further ado. Here is our build.sbt</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">name</span> <span class="o">:=</span> <span class="s">"KafkaAvro"</span>
<span class="n">version</span> <span class="o">:=</span> <span class="s">"1.0"</span>
<span class="n">scalaVersion</span> <span class="o">:=</span> <span class="s">"2.12.3"</span>
<span class="n">libraryDependencies</span> <span class="o">++=</span> <span class="nc">Seq</span><span class="o">(</span>
<span class="s">"com.sksamuel.avro4s"</span> <span class="o">%%</span> <span class="s">"avro4s-core"</span> <span class="o">%</span> <span class="s">"1.8.0"</span><span class="o">,</span>
<span class="s">"com.typesafe.akka"</span> <span class="o">%%</span> <span class="s">"akka-stream"</span> <span class="o">%</span> <span class="s">"2.5.6"</span><span class="o">,</span>
<span class="s">"com.lightbend.akka"</span> <span class="o">%%</span> <span class="s">"akka-stream-alpakka-csv"</span> <span class="o">%</span> <span class="s">"0.13"</span><span class="o">,</span>
<span class="s">"com.lightbend.akka"</span> <span class="o">%%</span> <span class="s">"akka-stream-alpakka-file"</span> <span class="o">%</span> <span class="s">"0.13"</span><span class="o">,</span>
<span class="s">"com.typesafe.akka"</span> <span class="o">%%</span> <span class="s">"akka-stream-kafka"</span> <span class="o">%</span> <span class="s">"0.17"</span>
<span class="o">)</span>
</code></pre></div></div>
<p>Now let’s get some fundamentals out of the way. Basically we need to parse our CSV and then convert it into a stream of CountryCapital objects. I won’t explain the code here because I already explained this <a href="https://abhsrivastava.github.io/2017/10/02/Alpkka-File-CSV-Elastic/">here</a></p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">implicit</span> <span class="k">val</span> <span class="n">actorSystem</span> <span class="k">=</span> <span class="nc">ActorSystem</span><span class="o">()</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">actorMaterializer</span> <span class="k">=</span> <span class="nc">ActorMaterializer</span><span class="o">()</span>
<span class="k">val</span> <span class="n">resource</span> <span class="k">=</span> <span class="n">getClass</span><span class="o">.</span><span class="n">getResource</span><span class="o">(</span><span class="s">"/countrycapital.csv"</span><span class="o">)</span>
<span class="k">val</span> <span class="n">path</span> <span class="k">=</span> <span class="nc">Paths</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">resource</span><span class="o">.</span><span class="n">toURI</span><span class="o">)</span>
<span class="k">val</span> <span class="n">source</span> <span class="k">=</span> <span class="nc">FileIO</span><span class="o">.</span><span class="n">fromPath</span><span class="o">(</span><span class="n">path</span><span class="o">)</span>
<span class="k">val</span> <span class="n">flow1</span> <span class="k">=</span> <span class="nc">CsvParsing</span><span class="o">.</span><span class="n">lineScanner</span><span class="o">()</span>
<span class="k">val</span> <span class="n">flow2</span> <span class="k">=</span> <span class="nc">Flow</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">ByteString</span><span class="o">]].</span><span class="n">map</span><span class="o">(</span><span class="n">list</span> <span class="k">=></span> <span class="n">list</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">utf8String</span><span class="o">))</span>
<span class="k">val</span> <span class="n">flow3</span> <span class="k">=</span> <span class="nc">Flow</span><span class="o">[</span><span class="kt">List</span><span class="o">[</span><span class="kt">String</span><span class="o">]].</span><span class="n">map</span><span class="o">(</span><span class="n">list</span> <span class="k">=></span> <span class="nc">CountryCapital</span><span class="o">(</span><span class="n">list</span><span class="o">(</span><span class="mi">0</span><span class="o">),</span> <span class="n">list</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span>
</code></pre></div></div>
<p>So now let us convert our CountryCapital stream of objects into Avro Records</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">val</span> <span class="n">flow4</span> <span class="k">=</span> <span class="nc">Flow</span><span class="o">[</span><span class="kt">CountryCapital</span><span class="o">].</span><span class="n">map</span> <span class="o">{</span> <span class="n">cc</span> <span class="k">=></span>
<span class="k">val</span> <span class="n">baos</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">ByteArrayOutputStream</span><span class="o">()</span>
<span class="k">val</span> <span class="n">output</span> <span class="k">=</span> <span class="nc">AvroOutputStream</span><span class="o">.</span><span class="n">binary</span><span class="o">[</span><span class="kt">CountryCapital</span><span class="o">](</span><span class="n">baos</span><span class="o">)</span>
<span class="n">output</span><span class="o">.</span><span class="n">write</span><span class="o">(</span><span class="n">cc</span><span class="o">)</span>
<span class="n">output</span><span class="o">.</span><span class="n">close</span><span class="o">()</span>
<span class="k">val</span> <span class="n">result</span> <span class="k">=</span> <span class="n">baos</span><span class="o">.</span><span class="n">toByteArray</span>
<span class="n">baos</span><span class="o">.</span><span class="n">close</span><span class="o">()</span>
<span class="n">result</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In the code above i used Avro4s Serializer to convert the CountryCapital object into a strem of bytes containing the Avro Record.</p>
<p>Now that we have the Stream of Bytes, its easy to write it into a Kafka topic</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">producerSettings</span> <span class="k">=</span> <span class="nc">ProducerSettings</span><span class="o">(</span><span class="n">actorSystem</span><span class="o">,</span> <span class="k">new</span> <span class="nc">ByteArraySerializer</span><span class="o">,</span> <span class="k">new</span> <span class="nc">ByteArraySerializer</span><span class="o">)</span>
<span class="o">.</span><span class="n">withBootstrapServers</span><span class="o">(</span><span class="s">"abhisheks-mini:9093"</span><span class="o">)</span>
<span class="k">val</span> <span class="n">flow5</span> <span class="k">=</span> <span class="nc">Flow</span><span class="o">[</span><span class="kt">Array</span><span class="o">[</span><span class="kt">Byte</span><span class="o">]].</span><span class="n">map</span><span class="o">{</span><span class="n">array</span> <span class="k">=></span>
<span class="k">new</span> <span class="nc">ProducerRecord</span><span class="o">[</span><span class="kt">Array</span><span class="o">[</span><span class="kt">Byte</span><span class="o">]</span>, <span class="kt">Array</span><span class="o">[</span><span class="kt">Byte</span><span class="o">]](</span><span class="s">"test"</span><span class="o">,</span> <span class="n">array</span><span class="o">)</span>
<span class="o">}</span>
<span class="k">val</span> <span class="n">sink</span> <span class="k">=</span> <span class="nc">Producer</span><span class="o">.</span><span class="n">plainSink</span><span class="o">(</span><span class="n">producerSettings</span><span class="o">)</span>
</code></pre></div></div>
<p>Here we are using the <a href="https://doc.akka.io/docs/akka-stream-kafka/current/home.html">Akka Streams Kafka</a> library to create a akka stream Sink which acts as a Kafka Producer. We need to convert our stream of Array of bytes into a Producer Record. the producer record also contains the name of our topic which is “test” in my case.</p>
<p>Usually Kafka runs on port <code class="highlighter-rouge">9092</code>. I am running it on <code class="highlighter-rouge">9093</code> in order to avoid port conflict with ElasticSearch (which I also run on the same server).</p>
<p>Now that we have our Sink. All we need to do is to build our Runnable Graph and execute it</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">graph</span> <span class="k">=</span> <span class="nc">RunnableGraph</span><span class="o">.</span><span class="n">fromGraph</span><span class="o">(</span><span class="nc">GraphDSL</span><span class="o">.</span><span class="n">create</span><span class="o">(</span><span class="n">sink</span><span class="o">){</span><span class="k">implicit</span> <span class="n">builder</span> <span class="k">=></span>
<span class="n">s</span> <span class="k">=></span>
<span class="k">import</span> <span class="nn">GraphDSL.Implicits._</span>
<span class="n">source</span> <span class="o">~></span> <span class="n">flow1</span> <span class="o">~></span> <span class="n">flow2</span> <span class="o">~></span> <span class="n">flow3</span> <span class="o">~></span> <span class="n">flow4</span> <span class="o">~></span> <span class="n">flow5</span> <span class="o">~></span> <span class="n">s</span><span class="o">.</span><span class="n">in</span>
<span class="nc">ClosedShape</span>
<span class="o">})</span>
<span class="k">val</span> <span class="n">future</span> <span class="k">=</span> <span class="n">graph</span><span class="o">.</span><span class="n">run</span><span class="o">()</span>
<span class="n">future</span><span class="o">.</span><span class="n">onComplete</span> <span class="o">{</span> <span class="k">_</span> <span class="k">=></span>
<span class="n">actorSystem</span><span class="o">.</span><span class="n">terminate</span><span class="o">()</span>
<span class="o">}</span>
<span class="nc">Await</span><span class="o">.</span><span class="n">result</span><span class="o">(</span><span class="n">actorSystem</span><span class="o">.</span><span class="n">whenTerminated</span><span class="o">,</span> <span class="nc">Duration</span><span class="o">.</span><span class="nc">Inf</span><span class="o">)</span>
</code></pre></div></div>
<p>Reading Avro records from Kafka topic is also very straigtforward. The code is exactly the same as above. The only difference is that instead of a Producer, we are creating a Consumer</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">val</span> <span class="n">consumerSettings</span> <span class="k">=</span> <span class="nc">ConsumerSettings</span><span class="o">(</span><span class="n">actorSystem</span><span class="o">,</span> <span class="k">new</span> <span class="nc">ByteArrayDeserializer</span><span class="o">(),</span> <span class="k">new</span> <span class="nc">ByteArrayDeserializer</span><span class="o">)</span>
<span class="o">.</span><span class="n">withBootstrapServers</span><span class="o">(</span><span class="s">"abhisheks-mini:9093"</span><span class="o">)</span>
<span class="o">.</span><span class="n">withGroupId</span><span class="o">(</span><span class="s">"abhi"</span><span class="o">)</span>
<span class="o">.</span><span class="n">withProperty</span><span class="o">(</span><span class="nc">ConsumerConfig</span><span class="o">.</span><span class="nc">AUTO_OFFSET_RESET_CONFIG</span><span class="o">,</span> <span class="s">"earliest"</span><span class="o">)</span>
<span class="k">val</span> <span class="n">source</span> <span class="k">=</span> <span class="nc">Consumer</span><span class="o">.</span><span class="n">committableSource</span><span class="o">(</span><span class="n">consumerSettings</span><span class="o">,</span> <span class="nc">Subscriptions</span><span class="o">.</span><span class="n">topics</span><span class="o">(</span><span class="s">"test"</span><span class="o">))</span>
<span class="k">val</span> <span class="n">flow1</span> <span class="k">=</span> <span class="nc">Flow</span><span class="o">[</span><span class="kt">ConsumerMessage.CommittableMessage</span><span class="o">[</span><span class="kt">Array</span><span class="o">[</span><span class="kt">Byte</span><span class="o">]</span>, <span class="kt">Array</span><span class="o">[</span><span class="kt">Byte</span><span class="o">]]].</span><span class="n">map</span><span class="o">{</span> <span class="n">msg</span> <span class="k">=></span> <span class="n">msg</span><span class="o">.</span><span class="n">record</span><span class="o">.</span><span class="n">value</span><span class="o">()}</span>
<span class="k">val</span> <span class="n">flow2</span> <span class="k">=</span> <span class="nc">Flow</span><span class="o">[</span><span class="kt">Array</span><span class="o">[</span><span class="kt">Byte</span><span class="o">]].</span><span class="n">map</span><span class="o">{</span> <span class="n">array</span> <span class="k">=></span>
<span class="k">val</span> <span class="n">bais</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">ByteArrayInputStream</span><span class="o">(</span><span class="n">array</span><span class="o">)</span>
<span class="k">val</span> <span class="n">input</span> <span class="k">=</span> <span class="nc">AvroInputStream</span><span class="o">.</span><span class="n">binary</span><span class="o">[</span><span class="kt">CountryCapital</span><span class="o">](</span><span class="n">bais</span><span class="o">)</span>
<span class="n">input</span><span class="o">.</span><span class="n">iterator</span><span class="o">.</span><span class="n">toSeq</span><span class="o">.</span><span class="n">head</span>
<span class="o">}</span>
<span class="k">val</span> <span class="n">sink</span> <span class="k">=</span> <span class="nc">Sink</span><span class="o">.</span><span class="n">foreach</span><span class="o">[</span><span class="kt">CountryCapital</span><span class="o">](</span><span class="n">println</span><span class="o">)</span>
<span class="k">val</span> <span class="n">graph</span> <span class="k">=</span> <span class="nc">RunnableGraph</span><span class="o">.</span><span class="n">fromGraph</span><span class="o">(</span><span class="nc">GraphDSL</span><span class="o">.</span><span class="n">create</span><span class="o">(</span><span class="n">sink</span><span class="o">){</span> <span class="k">implicit</span> <span class="n">builder</span> <span class="k">=></span>
<span class="n">s</span> <span class="k">=></span>
<span class="k">import</span> <span class="nn">GraphDSL.Implicits._</span>
<span class="n">source</span> <span class="o">~></span> <span class="n">flow1</span> <span class="o">~></span> <span class="n">flow2</span> <span class="o">~></span> <span class="n">s</span><span class="o">.</span><span class="n">in</span>
<span class="nc">ClosedShape</span>
<span class="o">})</span>
<span class="k">val</span> <span class="n">future</span> <span class="k">=</span> <span class="n">graph</span><span class="o">.</span><span class="n">run</span><span class="o">()</span>
<span class="n">future</span><span class="o">.</span><span class="n">onComplete</span> <span class="o">{</span> <span class="k">_</span> <span class="k">=></span>
<span class="n">actorSystem</span><span class="o">.</span><span class="n">terminate</span><span class="o">()</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Once the items are read from the kafka topic, they are read as Array of bytes. We use the Avro4s Deserializer this time, to convert array of bytes into CountryCapital object.</p>
<p>The whole example can be found at my <a href="https://github.com/abhsrivastava/KafkaAvro">github</a>.</p>Abhishek SrivastavaContinuing our quest to learn Akka Streams, we’ll take our same old countrycapital.csv and then we’ll convert each line into an AVRO Record and then write into a Kafka topic using Akka streams.Getting Started with ElasticSearch using Scala and Elastic4s2017-09-30T12:00:00+00:002017-09-30T12:00:00+00:00//2017/09/30/Elastic4s<p>I wanted to learn ElasticSearch using the Scala library <a href="https://github.com/sksamuel/elastic4s">Elastic4s</a>.</p>
<p>First let us look at the SBT imports</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">name</span> <span class="o">:=</span> <span class="s">"Elastic4sLearn"</span>
<span class="n">version</span> <span class="o">:=</span> <span class="s">"1.0"</span>
<span class="n">scalaVersion</span> <span class="o">:=</span> <span class="s">"2.12.3"</span>
<span class="k">val</span> <span class="n">elastic4sVersion</span> <span class="k">=</span> <span class="s">"0.90.2.8"</span>
<span class="n">libraryDependencies</span> <span class="o">++=</span> <span class="nc">Seq</span><span class="o">(</span>
<span class="s">"com.sksamuel.elastic4s"</span> <span class="o">%%</span> <span class="s">"elastic4s-core"</span> <span class="o">%</span> <span class="s">"5.5.3"</span><span class="o">,</span>
<span class="s">"com.sksamuel.elastic4s"</span> <span class="o">%%</span> <span class="s">"elastic4s-tcp"</span> <span class="o">%</span> <span class="s">"5.5.3"</span><span class="o">,</span>
<span class="s">"com.sksamuel.elastic4s"</span> <span class="o">%%</span> <span class="s">"elastic4s-http"</span> <span class="o">%</span> <span class="s">"5.5.3"</span><span class="o">,</span>
<span class="s">"com.sksamuel.elastic4s"</span> <span class="o">%%</span> <span class="s">"elastic4s-streams"</span> <span class="o">%</span> <span class="s">"5.5.3"</span><span class="o">,</span>
<span class="s">"com.sksamuel.elastic4s"</span> <span class="o">%%</span> <span class="s">"elastic4s-circe"</span> <span class="o">%</span> <span class="s">"5.5.3"</span><span class="o">,</span>
<span class="s">"org.apache.logging.log4j"</span> <span class="o">%</span> <span class="s">"log4j-api"</span> <span class="o">%</span> <span class="s">"2.9.1"</span><span class="o">,</span>
<span class="s">"org.apache.logging.log4j"</span> <span class="o">%</span> <span class="s">"log4j-core"</span> <span class="o">%</span> <span class="s">"2.9.1"</span><span class="o">,</span>
<span class="s">"com.sksamuel.elastic4s"</span> <span class="o">%%</span> <span class="s">"elastic4s-testkit"</span> <span class="o">%</span> <span class="s">"5.5.3"</span> <span class="o">%</span> <span class="s">"test"</span>
<span class="o">)</span>
</code></pre></div></div>
<p>Here the entry log4j entries are a real life safer. i would have missed many issues with my applicaiton had I not been prodent enough to import them. Next we need to configure the log4j logger so that we can troubleshoot our appliction easily. Create a file called log4j.xml in <code class="highlighter-rouge">src/main/resources</code></p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?xml version="1.0" encoding="UTF-8"?></span>
<span class="nt"><Configuration</span> <span class="na">status=</span><span class="s">"WARN"</span><span class="nt">></span>
<span class="c"><!-- Author: Crunchify.com --></span>
<span class="nt"><Appenders></span>
<span class="nt"><Console</span> <span class="na">name=</span><span class="s">"Console"</span> <span class="na">target=</span><span class="s">"SYSTEM_OUT"</span><span class="nt">></span>
<span class="nt"><PatternLayout</span> <span class="na">pattern=</span><span class="s">"%d{YYYY-MM-dd HH:mm:ss} [%t] %-5p %c{1}:%L - %msg%n"</span> <span class="nt">/></span>
<span class="nt"></Console></span>
<span class="nt"><RollingFile</span> <span class="na">name=</span><span class="s">"RollingFile"</span> <span class="na">filename=</span><span class="s">"log/output.log"</span>
<span class="na">filepattern=</span><span class="s">"${logPath}/%d{YYYYMMddHHmmss}-fargo.log"</span><span class="nt">></span>
<span class="nt"><PatternLayout</span> <span class="na">pattern=</span><span class="s">"%d{YYYY-MM-dd HH:mm:ss} [%t] %-5p %c{1}:%L - %msg%n"</span> <span class="nt">/></span>
<span class="nt"><Policies></span>
<span class="nt"><SizeBasedTriggeringPolicy</span> <span class="na">size=</span><span class="s">"100 MB"</span> <span class="nt">/></span>
<span class="nt"></Policies></span>
<span class="nt"><DefaultRolloverStrategy</span> <span class="na">max=</span><span class="s">"20"</span> <span class="nt">/></span>
<span class="nt"></RollingFile></span>
<span class="nt"></Appenders></span>
<span class="nt"><Loggers></span>
<span class="nt"><Root</span> <span class="na">level=</span><span class="s">"info"</span><span class="nt">></span>
<span class="nt"><AppenderRef</span> <span class="na">ref=</span><span class="s">"Console"</span> <span class="nt">/></span>
<span class="nt"><AppenderRef</span> <span class="na">ref=</span><span class="s">"RollingFile"</span> <span class="nt">/></span>
<span class="nt"></Root></span>
<span class="nt"></Loggers></span>
<span class="nt"></Configuration></span>
</code></pre></div></div>
<p>Installing elastic search on my remote server was pretty easy</p>
<ol>
<li>brew install elasticsearch</li>
<li>search for a file called elasticsearch.yml and add the following line to it
network.host: myremote-server</li>
<li>brew services start elasticsearch</li>
</ol>
<p>Its import to set the bind address otherwise you will not be able to connect remotely to elastic search. I always install all server products remotely so that they don’t slow my MBP down.</p>
<p>In order to ensure that our product is up and running point your browser to http://myremote-server:9200. You should see the following output</p>
<pre><code class="language-txt"> {
"name" : "NYrp1l-",
"cluster_name" : "elasticsearch_abhisheksrivastava",
"cluster_uuid" : "rkNONwmrTq-6tsyvxN5Zig",
"version" : {
"number" : "5.6.2",
"build_hash" : "57e20f3",
"build_date" : "2017-09-23T13:16:45.703Z",
"build_snapshot" : false,
"lucene_version" : "6.6.1"
},
"tagline" : "You Know, for Search"
}
</code></pre>
<p>Make a note your the cluster name because this is needed. A <strong>lot</strong> of documentation on the internet assumes that your cluster name is <strong>elasticsearch</strong> and that is why programs fail to connect to elastic search.</p>
<p>Now we will write two applications. One using TcpClient and other using HttpClient to interact with the ElasticSearch server.</p>
<h2 id="tcpclient">TcpClient.</h2>
<p>In order to connect to elasticsearch using the TcpClient, we must be aware of our cluster name.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="nn">com.abhi</span>
<span class="k">import</span> <span class="nn">com.sksamuel.elastic4s.</span><span class="o">{</span><span class="nc">ElasticsearchClientUri</span><span class="o">,</span> <span class="nc">TcpClient</span><span class="o">}</span>
<span class="k">import</span> <span class="nn">org.elasticsearch.common.settings.Settings</span>
<span class="k">object</span> <span class="nc">Scala4s</span> <span class="k">extends</span> <span class="nc">App</span> <span class="o">{</span>
<span class="k">val</span> <span class="n">settings</span> <span class="k">=</span> <span class="nc">Settings</span><span class="o">.</span><span class="n">builder</span><span class="o">().</span><span class="n">put</span><span class="o">(</span><span class="s">"cluster.name"</span><span class="o">,</span> <span class="s">"elasticsearch_abhisheksrivastava"</span><span class="o">).</span><span class="n">build</span><span class="o">()</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">client</span> <span class="k">=</span> <span class="nc">TcpClient</span><span class="o">.</span><span class="n">transport</span><span class="o">(</span><span class="n">settings</span><span class="o">,</span> <span class="nc">ElasticsearchClientUri</span><span class="o">(</span><span class="s">"elasticsearch://abhisheks-mini:9300"</span><span class="o">))</span>
<span class="n">client</span><span class="o">.</span><span class="n">close</span><span class="o">()</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Note that I had to specify the name of the cluster in the properties. without this the application would not have worked.</p>
<p>Now we need to complete 3 tasks. We need to create an Index, Insert a document in that index and finally query the document. To complete each task you will need to familiarize yourself with the Elastic4s DSL. The pattern to use the DSL is pretty consistent. you first create the DSL command then execute the command using the client.</p>
<ul>
<li>Create Index DSL.
<ul>
<li><code class="highlighter-rouge">createIndex("bands").mappings(mapping("artist") as(textField("name")))</code></li>
</ul>
</li>
<li>Insert a document
<ul>
<li><code class="highlighter-rouge">indexInto("bands" / "artists") doc Artist("nirvana") refresh(RefreshPolicy.IMMEDIATE)</code></li>
</ul>
</li>
<li>Query the document
<ul>
<li><code class="highlighter-rouge">search("bands" / "artists") query "nirvana"</code></li>
</ul>
</li>
</ul>
<p>Now all we need to do is to tie in these 3 DSL commands into our application and execute them.</p>
<p>We will wrap these in Functions and then invoke those functions in a monadic way. Function to create the index</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">createElasticIndex</span><span class="o">(</span><span class="k">implicit</span> <span class="n">client</span><span class="k">:</span> <span class="kt">TcpClient</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="kt">CreateIndexResponse</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">import</span> <span class="nn">com.sksamuel.elastic4s.ElasticDsl._</span>
<span class="n">client</span><span class="o">.</span><span class="n">execute</span> <span class="o">{</span>
<span class="n">createIndex</span><span class="o">(</span><span class="s">"bands"</span><span class="o">).</span><span class="n">mappings</span><span class="o">(</span>
<span class="n">mapping</span><span class="o">(</span><span class="s">"artist"</span><span class="o">)</span> <span class="n">as</span><span class="o">(</span>
<span class="n">textField</span><span class="o">(</span><span class="s">"name"</span><span class="o">)</span>
<span class="o">)</span>
<span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Function to Insert a document in the Index</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">insertDocument</span><span class="o">(</span><span class="k">implicit</span> <span class="n">client</span><span class="k">:</span> <span class="kt">TcpClient</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="kt">RichIndexResponse</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">import</span> <span class="nn">com.sksamuel.elastic4s.ElasticDsl._</span>
<span class="n">client</span><span class="o">.</span><span class="n">execute</span> <span class="o">{</span>
<span class="n">indexInto</span><span class="o">(</span><span class="s">"bands"</span> <span class="o">/</span> <span class="s">"artists"</span><span class="o">)</span> <span class="n">doc</span> <span class="nc">Artist</span><span class="o">(</span><span class="s">"nirvana"</span><span class="o">)</span> <span class="n">refresh</span><span class="o">(</span><span class="nc">RefreshPolicy</span><span class="o">.</span><span class="nc">IMMEDIATE</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>And the Function to query the document</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">queryDocument</span><span class="o">(</span><span class="n">artist</span><span class="k">:</span> <span class="kt">String</span><span class="o">)(</span><span class="k">implicit</span> <span class="n">client</span><span class="k">:</span> <span class="kt">TcpClient</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="kt">RichSearchResponse</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">import</span> <span class="nn">com.sksamuel.elastic4s.ElasticDsl._</span>
<span class="n">client</span><span class="o">.</span><span class="n">execute</span> <span class="o">{</span>
<span class="n">search</span><span class="o">(</span><span class="s">"bands"</span> <span class="o">/</span> <span class="s">"artists"</span><span class="o">)</span> <span class="n">query</span> <span class="n">artist</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Now we tie these together as</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">futureArtist</span> <span class="k">=</span> <span class="k">for</span> <span class="o">{</span>
<span class="k">_</span> <span class="k"><-</span> <span class="n">createElasticIndex</span>
<span class="k">_</span> <span class="k"><-</span> <span class="n">insertDocument</span>
<span class="n">resp</span> <span class="k"><-</span> <span class="n">queryDocument</span><span class="o">(</span><span class="s">"nirvana"</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">resp</span><span class="o">.</span><span class="n">to</span><span class="o">[</span><span class="kt">Artist</span><span class="o">]</span>
<span class="n">futureArtist</span><span class="o">.</span><span class="n">onComplete</span><span class="o">(</span><span class="k">_</span> <span class="k">=></span> <span class="n">client</span><span class="o">.</span><span class="n">close</span><span class="o">())</span>
<span class="k">val</span> <span class="n">artistList</span> <span class="k">=</span> <span class="nc">Await</span><span class="o">.</span><span class="n">result</span><span class="o">(</span><span class="n">futureArtist</span><span class="o">,</span> <span class="nc">Duration</span><span class="o">.</span><span class="nc">Inf</span><span class="o">)</span>
<span class="n">artistList</span><span class="o">.</span><span class="n">foreach</span><span class="o">(</span><span class="n">println</span><span class="o">)</span>
</code></pre></div></div>
<h2 id="httpclient">HttpClient.</h2>
<p>Establishing a connection via the HttpClient is a little easier because we can connect just with the server name and port (without knowing the cluster name)</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">client</span> <span class="k">=</span> <span class="nc">HttpClient</span><span class="o">(</span><span class="nc">ElasticsearchClientUri</span><span class="o">(</span><span class="s">"abhisheks-mini"</span><span class="o">,</span> <span class="mi">9200</span><span class="o">))</span>
<span class="n">client</span><span class="o">.</span><span class="n">close</span><span class="o">()</span>
</code></pre></div></div>
<p>The process of using the DSL is the same. We just have to be careful that we import the Http DSL <code class="highlighter-rouge">com.sksamuel.elastic4s.http.ElasticDsl._</code> I just imported the previous TcpClient DSL at first and my application had tons of compiler errors.</p>
<p>Once again we write our 3 functions to create the index, insert documents in the index and then query.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="k">def</span> <span class="n">createMyIndex</span><span class="o">(</span><span class="k">implicit</span> <span class="n">client</span><span class="k">:</span> <span class="kt">HttpClient</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="kt">CreateIndexResponse</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">import</span> <span class="nn">com.sksamuel.elastic4s.http.ElasticDsl._</span>
<span class="n">client</span><span class="o">.</span><span class="n">execute</span><span class="o">{</span>
<span class="n">createIndex</span><span class="o">(</span><span class="s">"myindex"</span><span class="o">).</span><span class="n">mappings</span><span class="o">(</span><span class="n">mapping</span><span class="o">(</span><span class="s">"mytype"</span><span class="o">)</span> <span class="n">as</span><span class="o">(</span><span class="n">textField</span><span class="o">(</span><span class="s">"country"</span><span class="o">),</span> <span class="n">textField</span><span class="o">(</span><span class="s">"capital"</span><span class="o">)))</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">def</span> <span class="n">insertDocument</span><span class="o">(</span><span class="k">implicit</span> <span class="n">client</span><span class="k">:</span> <span class="kt">HttpClient</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="kt">BulkResponse</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">import</span> <span class="nn">com.sksamuel.elastic4s.http.ElasticDsl._</span>
<span class="n">client</span><span class="o">.</span><span class="n">execute</span> <span class="o">{</span>
<span class="n">bulk</span><span class="o">(</span>
<span class="n">indexInto</span><span class="o">(</span><span class="s">"myindex"</span> <span class="o">/</span> <span class="s">"mytype"</span><span class="o">).</span><span class="n">fields</span><span class="o">(</span><span class="s">"country"</span> <span class="o">-></span> <span class="s">"Mongolia"</span><span class="o">,</span> <span class="s">"capital"</span> <span class="o">-></span> <span class="s">"Ulaanbaatar"</span><span class="o">),</span>
<span class="n">indexInto</span><span class="o">(</span><span class="s">"myindex"</span> <span class="o">/</span> <span class="s">"mytype"</span><span class="o">).</span><span class="n">fields</span><span class="o">(</span><span class="s">"country"</span> <span class="o">-></span> <span class="s">"Nambia"</span><span class="o">,</span> <span class="s">"capital"</span> <span class="o">-></span> <span class="s">"Windhoek"</span><span class="o">)</span>
<span class="o">).</span><span class="n">refresh</span><span class="o">(</span><span class="nc">RefreshPolicy</span><span class="o">.</span><span class="nc">IMMEDIATE</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">def</span> <span class="n">queryCapital</span><span class="o">(</span><span class="n">capital</span><span class="k">:</span> <span class="kt">String</span><span class="o">)(</span><span class="k">implicit</span> <span class="n">client</span><span class="k">:</span> <span class="kt">HttpClient</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="kt">SearchResponse</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">import</span> <span class="nn">com.sksamuel.elastic4s.http.ElasticDsl._</span>
<span class="n">client</span><span class="o">.</span><span class="n">execute</span> <span class="o">{</span>
<span class="n">search</span><span class="o">(</span><span class="s">"myindex"</span><span class="o">).</span><span class="n">matchQuery</span><span class="o">(</span><span class="s">"capital"</span><span class="o">,</span> <span class="n">capital</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>and finally we can connect all the 3 functions by means of a simple for statement</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">val</span> <span class="n">futureResult</span> <span class="k">=</span> <span class="k">for</span> <span class="o">{</span>
<span class="k">_</span> <span class="k"><-</span> <span class="n">createMyIndex</span>
<span class="k">_</span> <span class="k"><-</span> <span class="n">insertDocument</span>
<span class="n">response</span> <span class="k"><-</span> <span class="n">queryCapital</span><span class="o">(</span><span class="s">"Ulaanbaatar"</span><span class="o">)</span>
<span class="o">}</span> <span class="k">yield</span> <span class="n">response</span><span class="o">.</span><span class="n">to</span><span class="o">[</span><span class="kt">Result</span><span class="o">]</span>
<span class="n">futureResult</span><span class="o">.</span><span class="n">onComplete</span><span class="o">{</span><span class="k">_</span> <span class="k">=></span> <span class="n">client</span><span class="o">.</span><span class="n">close</span><span class="o">()}</span>
<span class="k">val</span> <span class="n">results</span> <span class="k">=</span> <span class="nc">Await</span><span class="o">.</span><span class="n">result</span><span class="o">(</span><span class="n">futureResult</span><span class="o">,</span> <span class="nc">Duration</span><span class="o">.</span><span class="nc">Inf</span><span class="o">)</span>
<span class="n">results</span><span class="o">.</span><span class="n">foreach</span><span class="o">(</span><span class="n">println</span><span class="o">)</span>
</code></pre></div></div>
<p>The whole application can be found at my <a href="https://github.com/abhsrivastava/Elastic4sTest">github</a></p>Abhishek SrivastavaI wanted to learn ElasticSearch using the Scala library Elastic4s.Implicit Resolution Gotcha2017-09-29T12:00:00+00:002017-09-29T12:00:00+00:00//2017/09/29/Implicits-Resolution-Gotcha<p>While writing some slick code today an interesting problem poped up. Here is the code I was working on</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">MyApp</span> <span class="k">extends</span> <span class="nc">App</span> <span class="o">{</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">result</span> <span class="k">=</span> <span class="nc">GetResult</span><span class="o">(</span><span class="n">r</span> <span class="k">=></span> <span class="nc">Foo</span><span class="o">(</span><span class="n">r</span><span class="o">.<<,</span> <span class="n">r</span><span class="o">.<<))</span>
<span class="k">val</span> <span class="n">query</span> <span class="k">=</span> <span class="s">"""select id, name from foo"""</span><span class="o">.</span><span class="n">as</span><span class="o">[</span><span class="kt">Foo</span><span class="o">]</span>
<span class="o">...</span>
<span class="o">}</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">Foo</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
</code></pre></div></div>
<p>This compiles and works fine, but I never write my implicits inline becuase they just add noise to the code.</p>
<p>So I copy and pasted my implicit definition into the companion object</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">MyApp</span> <span class="k">extends</span> <span class="nc">App</span> <span class="o">{</span>
<span class="k">import</span> <span class="nn">Foo.result</span>
<span class="k">val</span> <span class="n">query</span> <span class="k">=</span> <span class="s">"""select id, name from foo"""</span><span class="o">.</span><span class="n">as</span><span class="o">[</span><span class="kt">Foo</span><span class="o">]</span>
<span class="o">...</span>
<span class="o">}</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">Foo</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
<span class="k">object</span> <span class="nc">Foo</span> <span class="o">{</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">result</span> <span class="k">=</span> <span class="nc">GetResult</span><span class="o">(</span><span class="n">r</span> <span class="k">=></span> <span class="nc">Foo</span><span class="o">(</span><span class="n">r</span><span class="o">.<<,</span> <span class="n">r</span><span class="o">.<<))</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Oddly, I get an error saying</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> could not find implicit value for parameter rconv: slick.jdbc.GetResult[Foo]
</code></pre></div></div>
<p>Now that’s pretty surprising because I am importing the variable directly. It turns out that the implicit resolution rules state that the return type of implicits must be specified explictly when importing them from an external place.</p>
<p>So the correct code was</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">MyApp</span> <span class="k">extends</span> <span class="nc">App</span> <span class="o">{</span>
<span class="k">import</span> <span class="nn">Foo.result</span>
<span class="k">val</span> <span class="n">query</span> <span class="k">=</span> <span class="s">"""select id, name from foo"""</span><span class="o">.</span><span class="n">as</span><span class="o">[</span><span class="kt">Foo</span><span class="o">]</span>
<span class="o">...</span>
<span class="o">}</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">Foo</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span>
<span class="k">object</span> <span class="nc">Foo</span> <span class="o">{</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">result</span> <span class="k">:</span> <span class="kt">GetResult</span><span class="o">[</span><span class="kt">Foo</span><span class="o">]</span> <span class="k">=</span> <span class="nc">GetResult</span><span class="o">(</span><span class="n">r</span> <span class="k">=></span> <span class="nc">Foo</span><span class="o">(</span><span class="n">r</span><span class="o">.<<,</span> <span class="n">r</span><span class="o">.<<))</span>
<span class="o">}</span>
</code></pre></div></div>Abhishek SrivastavaWhile writing some slick code today an interesting problem poped up. Here is the code I was working onScanning Cassandra Tables with Alpakka2017-09-29T12:00:00+00:002017-09-29T12:00:00+00:00//2017/09/29/Scan-Cassandra-with-Alpakka<p>I often have to write code to scan entire cassandra tables. The table in question here is more than 100 million rows where each row contains a hashmap with 300 keys on average.</p>
<p>We have a legacy code which uses raw jdbc code using Cassandra jdbc driver and while it works the code is exteremely unwieldy because we have to write mutable data structures inside of a for loop which gets executed by means of a callback. (Yuck!)</p>
<p>There are plenty of folks out there who use Spark for such purposes, but I like to work without the need for bulky things like Hadoop clusters.</p>
<p>I came accross a technology called Alpakka today and it made scanning Cassandra tables a breeze.</p>
<p>Here is my build.sbt</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">libraryDependencies</span> <span class="o">++=</span> <span class="nc">Seq</span><span class="o">(</span>
<span class="s">"com.lightbend.akka"</span> <span class="o">%%</span> <span class="s">"akka-stream-alpakka-cassandra"</span> <span class="o">%</span> <span class="s">"0.11"</span>
<span class="o">)</span>
</code></pre></div></div>
<p>And now let us scan the table</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">val</span> <span class="n">stmt</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">SimpleStatement</span><span class="o">(</span><span class="s">"select id, name from foo"</span><span class="o">).</span><span class="n">setFetchSize</span><span class="o">(</span><span class="mi">100</span><span class="o">)</span>
<span class="k">val</span> <span class="n">source</span> <span class="k">=</span> <span class="nc">CassandraSource</span><span class="o">(</span><span class="n">stmt</span><span class="o">)</span>
<span class="k">val</span> <span class="n">flow</span> <span class="k">=</span> <span class="nc">Flow</span><span class="o">[</span><span class="kt">Row</span>, <span class="kt">NotUsed</span><span class="o">].</span><span class="n">map</span><span class="o">{</span><span class="n">row</span> <span class="k">=></span> <span class="nc">Foo</span><span class="o">(</span><span class="n">row</span><span class="o">.</span><span class="n">getLong</span><span class="o">(</span><span class="mi">0</span><span class="o">),</span> <span class="n">row</span><span class="o">.</span><span class="n">getString</span><span class="o">(</span><span class="mi">1</span><span class="o">))}</span>
<span class="k">val</span> <span class="n">sink</span> <span class="k">=</span> <span class="nc">Sink</span><span class="o">.</span><span class="n">foreach</span><span class="o">[</span><span class="kt">Foo</span><span class="o">](</span><span class="n">println</span><span class="o">)</span>
<span class="k">val</span> <span class="n">graph</span> <span class="k">=</span> <span class="nc">RunnableGraph</span><span class="o">.</span><span class="n">fromGraph</span><span class="o">(</span><span class="nc">GraphDSL</span><span class="o">.</span><span class="n">create</span><span class="o">(</span><span class="n">sink</span><span class="o">){</span><span class="k">implicit</span> <span class="n">builder</span> <span class="k">=></span>
<span class="n">s</span> <span class="k">=></span>
<span class="n">souce</span> <span class="o">~></span> <span class="n">flow</span> <span class="o">~></span> <span class="n">s</span><span class="o">.</span><span class="n">in</span>
<span class="nc">ClosedShape</span>
<span class="o">})</span>
<span class="k">val</span> <span class="n">future</span> <span class="k">=</span> <span class="n">graph</span><span class="o">.</span><span class="n">run</span><span class="o">()</span>
<span class="nc">Await</span><span class="o">(</span><span class="n">future</span><span class="o">,</span> <span class="nc">Duration</span><span class="o">.</span><span class="nc">Inf</span><span class="o">)</span>
</code></pre></div></div>
<p>That’s it. Akka Steams API also gives you nice operations like fold on the flow so that you can neatly aggregate data.</p>
<p>Writing back into cassandra is already pretty easy .. thanks to the Akka Streams Sink.</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">cassandraSink</span><span class="o">(</span><span class="n">session</span><span class="k">:</span> <span class="kt">Session</span><span class="o">)</span> <span class="k">:</span> <span class="kt">Sink</span><span class="o">[</span><span class="kt">Foo</span>, <span class="kt">Future</span><span class="o">[</span><span class="kt">Done</span><span class="o">]]</span> <span class="k">=</span> <span class="o">{</span>
<span class="k">implicit</span> <span class="k">val</span> <span class="n">s</span> <span class="k">=</span> <span class="n">session</span>
<span class="k">import</span> <span class="nn">scala.concurrent.ExecutionContext.Implicits.global</span>
<span class="k">val</span> <span class="n">stmt</span> <span class="k">=</span> <span class="n">session</span><span class="o">.</span><span class="n">prepare</span><span class="o">(</span><span class="s">"update foo set name = ? where id = ?"</span><span class="o">)</span>
<span class="k">val</span> <span class="n">binder</span> <span class="k">=</span> <span class="o">(</span><span class="n">foo</span><span class="k">:</span> <span class="kt">Foo</span><span class="o">,</span> <span class="n">statement</span><span class="k">:</span> <span class="kt">PreparedStatement</span><span class="o">)</span> <span class="k">=></span> <span class="n">statement</span><span class="o">.</span><span class="n">bind</span><span class="o">(</span><span class="n">foo</span><span class="o">.</span><span class="n">name</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="n">lang</span><span class="o">.</span><span class="nc">Long</span><span class="o">.</span><span class="n">valueOf</span><span class="o">(</span><span class="n">foo</span><span class="o">.</span><span class="n">id</span><span class="o">))</span>
<span class="nc">CassandraSink</span><span class="o">[</span><span class="kt">Foo</span><span class="o">](</span><span class="n">parallelism</span> <span class="k">=</span> <span class="mi">10</span><span class="o">,</span> <span class="n">stmt</span><span class="o">,</span> <span class="n">binder</span><span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Now think sink can be attached to any Flow which emits a Foo.</p>
<p>Alpakka project can be found at <a href="https://developer.lightbend.com/docs/alpakka/current/">Alpakka</a> and at <a href="https://github.com/akka/alpakka">Github</a></p>Abhishek SrivastavaI often have to write code to scan entire cassandra tables. The table in question here is more than 100 million rows where each row contains a hashmap with 300 keys on average.