Previous related post: Hacker News API part 2
Github project related to this post
In part 2 of this series I presented an interative command line app that used the Fetch library to pull data from an online data source (the Hacker News API). In the related github repo I’ve developed that project in the following ways:
- Converted the code from a command line JVM app to a Scala.js app that runs in the browser
- Created a custom cache so we can query its size and clear it on demand
- Used Udash to create an interactive frontend to operate the fetches
- Visualize each round of the data fetch using reftree
You can check out the project live here
or watch a short video demonstration
What is hnfetchhs and how do you use it?
The idea of this blog post and accompanying github repo is to provide an interactive demo of the Fetch library. The
Stories per page input field lets you specify how many stories to fetch from the API, whilst the
Page input chooses the offset. In the background I use the top stories endpoint to get the IDs of all the top stories. The number of top stories known is shown in the title bar. At anytime you can reload the top story IDs using
Refresh Top Stories.
Fetch Page button will start a fetch job for the Ids with the current
Page. So if you are are on page 1 and you have 30 stories per page you will get the top 30 stories as shown on Hacker News right now.
Once a fetch has been run you can view a diagram of the fetch operation in detail using the
Last Fetch tab. If you play around with different page numbers and stories per page you can see the cache filling up, and you will see that repeated fetches of stories will result in a Nil fetch with no data to show.
You can also clear the current cache using the
All these tools together make it easy to experiment with the fetch data source and different kinds of caches etc. The rest of the post explains some of the process involved in building this app.
What is Fetch?
Fetch, an open source project by 47 Degs, is inspired by the Haskell library Haxl developed at Facebook to simplify the concurrent retrieval of data from multiple sources. You can read more about that There is no Fork: an Abstraction for Efficient, Concurrent, and Concise Data Access
You can read more about Fetch at their documentation page https://47deg.github.io/fetch/docs.html
Converting code to scala.js
This is the first time I’ve ported code written for the JVM to Scala.js so it was a learning experience but also the shortest part of the project. I created a Udash frontend using the built in project generator and started moving my code into the new project a piece at a time. It involved doing the following:
Change library import paths
"com.47deg" %%% "fetch" % "0.6.3",
"com.lihaoyi" %%% "upickle" % "0.4.4",
"org.typelevel" %%% "cats" % "0.9.0"
This part was easy since these libraries are all available for scala.js. You just change the first
@@@ and the scala.js library will be imported instead.
In my original code I used the PrettyTime library which is written in Java
"org.ocpsoft.prettytime" % "prettytime" % "3.2.7.Final"
"ru.pavkin" %%% "scala-js-momentjs" % "0.9.0"
Since fetching HTTP pages in my original code uses the ScalaJ HTTP library
"org.scalaj" %% "scalaj-http" % "2.3.0"
You can see the implementation for this in the source file
def hnRequest[T](url: String)(implicit r: Reader[T]) : Future[Either[String, T]]
The DataCache interface in Fetch allows you to update the cache with new elements and to get a particular element from the cache if it is there. In order to write an interactive tool to explore Fetch I wanted to be able to show the number of elements in the cache. I also wanted to be able to empty the cache, but that’s straightforward, you just replace the existing cache with a new empty one.
Take a look at the code in
Cache.scala for a simple DataCache implementation with the size function exposed.
The Udash guide tells you everything you need to know to make great interactive frontends (and two kinds of backends) in a typesafe manner and entirely in Scala. I’ve used the Bootstrap add-on to utilize features such as tabbed panes to switch between the stories and the fetch diagram.
Visualizing the Fetch rounds
Probably the most interesting part of this project, although it was also fairly easy to implement, was the visualization of each fetch round.
For this I used reftree, and you can find out more about it in this video https://www.youtube.com/watch?v=6mWaqGHeg3g
RefTree by writing an implicit conversion:
Here’s a sample diagram of the Fetch rounds
Here you can see that each round grabbed at least 8 items (that’s the number of items per round my data source specifies) and you can see the time in ms for each one. One thing to note is that the rounds are actually a queue not a list, but I found the list view was a lot easier to understand visually.
Some ideas I have for extending the project:
- Do fetch and display of comments for the stories
- Interactive queries around users, stories and comments
- Periodic update of top stories animating the movers on the current page
Feel free to fork the code on github and expand it your needs, and as always feel free to contact me at the links above with any questions or comments.
Copyright (C) 2017 Justin-Heyes-Jones - All Rights Reserved