You can create your own time lapse videos from log messages. It is not rocket science and is possible using a purely open source tool chain. In my previous blog, I explained how you can create a heat map from IP addresses in your log messages using syslog-ng, GeoIP, Elasticsearch and Kibana. Here we do a few more steps by configuring Kibana, taking regular screenshots and turning them into a video using OpenShot.
Unless you are only interested in learning how to create a time lapse using OpenShot, you should start by reading my previous blog about creating a heat map. It explains everything up to the point where you have your first heat map compiled from the geolocations of IP addresses on your screen: that is, you have a log source, you have parsed logs, sent them to Elasticsearch and are now displaying them in Kibana.
Creating a time lapse video requires installing a few more software on your machine:
You will also need to turn off any screensavers on the machine where you capture the screenshots, otherwise you will be taking screenshots of a black screen. Trust me, it is not that funny if you discover it only after a few hours…
I did everything in a virtual machine so my laptop was not blocked from use while the screenshots were being created.
There are many ways you can show information in a time lapse video:
Another question is how often you take a screenshot. For my time lapses, I configured one screenshot a minute. For a busy network, 1 minute might be too long, for a quiet network, it could be too short. Make sure that when you are taking screenshots, you consistently use the same interval.
Both of these settings (Time Range and Auto-refresh) can be configured when you click the clock icon in the upper right-hand corner of Kibana:
Once you configured Kibana, it is time to start creating screenshots. Before doing that, however, you have one more step to do: maximize the browser window and make the browser full screen. This is not strictly necessary, but this way you don’t have to post-process the images to remove possibly sensitive data like your bookmarks.
My desktop environment, GNOME, has a bundled screenshot application: gnome-screenshot. If you use another desktop, you might need to install another application, like “screenshot-tool”. The only important feature is that the application should work from the command line.
Use this command line from a terminal window running on the same desktop:
while true ; do sleep 60 ; gnome-screenshot -B ; done
All you need now is patience. I was collecting screenshots for a bit more than half a day. For a first experiment, an hour is enough (but be aware that this results in a 2-second time lapse video if you leave the default “30 frames a second” setting untouched). Once you have enough screenshots, switch to the terminal window and terminate the while loop using Ctrl-C.
Depending on your screenshot application, image files are saved to different locations with different names. In the case of gnome-screenshot, files are saved under the “Pictures” directory in your home directory with names that include the date and time the image was created. For example, “Pictures/Screenshot from 2017-08-26 09-24-08.png”. Unfortunately OpenShot does not recognize image sequences this way. You need to rename image files so they have a sequence number in their name, starting with zero.
There are many tools available if you want to mass rename files. My choice was “pyRenamer”. Using “pyRenamer”, you can use a GUI to rename the files:
The final step is to convert the screenshot files into a time lapse video. The use of OpenShot might be an overkill for this task as there are many command line tools which can do the job. On the other hand, OpenShot can hide away their complexities and has many additional features, like creating title screens (not covered here), which can come in handy.
When you start OpenShot, it will start up with an “Untitled Project” with no files and an empty time line. Here I describe only the minimal steps required to create a time lapse video. Check the OpenShot documentation if you want to add some sound or a title / end screen.
Heat maps and time lapse videos are extremely powerful tools when it comes to visualizing large amounts of raw data. They can be especially useful when you wish to highlight trends and potential focus areas that merit further attention.
As usual, I omitted many details to keep my blog at a reasonable length. Here I list a few resources worth reading if you want to learn more or if you get stuck along the way:
While my blogs focus on the open source edition (OSE) of syslog-ng, you can use the latest release of syslog-ng Premium Editon as well to parse log messages and add geographical information.