 |
| Titel |
Monitoring operational data production applying Big Data tooling |
| VerfasserIn |
Wim Som de Cerff, Hotze de Jong, Roy van den Berg, Jeroen Bos, Rijk Oosterhoff, Henk Jan Klein Ikkink, Femke Haga, Tom Elsten, Hans Verhoef, Michal Koutek, John van de Vegte |
| Konferenz |
EGU General Assembly 2015
|
| Medientyp |
Artikel
|
| Sprache |
Englisch
|
| Digitales Dokument |
PDF |
| Erschienen |
In: GRA - Volume 17 (2015) |
| Datensatznummer |
250109547
|
| Publikation (Nr.) |
 EGU/EGU2015-14843.pdf |
|
|
|
|
|
| Zusammenfassung |
| Within the KNMI Deltaplan programme for improving the KNMI operational infrastructure an new fully automated system for monitoring the KNMI operational data production systems is being developed: PRISMA (PRocessflow Infrastructure Surveillance and Monitoring Application).
Currently the KNMI operational (24/7) production systems consist of over 60 applications, running on different hardware systems and platforms. They are interlinked for the production of numerous data products, which are delivered to internal and external customers. All applications are individually monitored by different applications, complicating root cause and impact analysis. Also, the underlying hardware and network is monitored separately using Zabbix.
Goal of the new system is to enable production chain monitoring, which enables root cause analysis (what is the root cause of the disruption) and impact analysis (what other products will be effected). The PRISMA system will make it possible to dispose all the existing monitoring applications, providing one interface for monitoring the data production.
For modeling the production chain, the Neo4j Graph database is used to store and query the model. The model can be edited through the PRISMA web interface, but is mainly automatically provided by the applications and systems which are to be monitored. The graph enables us to do root case and impact analysis. The graph can be visualized in the PRISMA web interface on different levels.
Each ‘monitored object’ in the model will have a status (OK, error, warning, unknown). This status is derived by combing all log information available. For collecting and querying the log information Splunk is used.
The system is developed using Scrum, by a multi-disciplinary team consisting of analysts, developers, a tester and interaction designer.
In the presentation we will focus on the lessons learned working with the ‘Big data’ tooling Splunk and Neo4J. |
|
|
|
|
|
|