Just Enough Angular For Uchiwa

Uchiwa is a simple dashboard for Sensu, version 0.4.0 is especially great because of two new features:
  - bulk actions on events,
  - allows stashed checks to be hidden.

We can see pending events for all sources, stash/acknowledge them, filter out less important items.

Uchiwa is young, and therefore it's pages are very light, and provide little information, not enough to start analyzing it immediately.
Some people are experimenting with custom attributes (any sensu check may contain an arbitrary properties in payload),  http://roobert.github.io/2014/11/03/Sensu-with-Embedded-Graphite-Graphs/ , which I think is very nice idea.

But still, as I've said, not flexible enough.
In most cases, we still have to:

• Ctrl+C a piece of the alert text (sometimes),
• navigate to another browser tab, 
• find the appropriate link (relevant for this type of alert),
• Ctrl+V a piece of text,   
• get to next panel,
• start to analyze.

It would be great if we could make Uchiwa to build provide necessary links automatically when it's necessary.

Use case:

• If some server runs out of disk space or starts to use swap aggressively - I'd like to see a link to a Grafana dashboard that shows all system metrics for this particular server.
• If some service is running bad (avg response time increased, msg rate drops) - I'd like to see a link to a dashboard with all the details for this particular service and a list of dependent processes.
• If process ships logs to Logstash server - I'd like to see a link to specific Kibana dashboard page, already filtered by this particular process name and/or host name.
• Optional button to 'restart' the process (to be shown only when process is running bad)

Something like this:

What new information can we see here ?

At first,
We see a link to a dashboard for host 'i-abcd1234' (which is Grafana scripted dashboard).

We also see this host is running two back-end processes -  'Collector' and 'Aggregator'. For each of them, a direct link to process-specific dashboard (Grafana templated dashboard) and to Kibana logs filtered by process/host name is provided.

If host was running Collector process only, then Aggregator links would not be shown.

Finally,
'disk_usage' check status is 2 (Critical), and we see additional link to detailed 'DiskIO' dashboard.

Nice, isn't it ?
What would you say if I tell you it requires to change just 2 lines of original Uchiwa code?




From Angular JS documentation:

"In Angular, templates are written with HTML that contains Angular-specific elements and attributes. Angular combines the template with information from the model and controller to render the dynamic view that a user sees in the browser"

That's right, the key is to modify Uchiwa template responsible for client presentation in Uchiwa.
Given that Uchiwa is installed into /opt/uchiwa, template we need to change is : /opt/uchiwa/src/public/partials/client/index.html


A better way is to put all custom code into different template /opt/uchiwa/src/public/partials/client/custom.html, and  'ngInclude' it into index.html at place we like:

<div class="container-fluid client">
  <div class="row-fluid">
    <div class="col-lg-12">
      <ol class="breadcrumb" ng-if="!selectedCheck">
        <li><a href="#/clients">Clients</a></li>
        <li class="active">{{client.name}}</li>
      </ol>
      <ol class="breadcrumb" ng-if="selectedCheck">
        <li><a href="#/events">Events</a></li>
        <li><a ng-href="#/client/{{client.dc | encodeURIComponent}}/{{client.name | encodeURIComponent}}">{{client.name}}</a></li>
        <li class="active">{{selectedCheck.check}}</li>
      </ol>
      <div ng-include src="'partials/client/custom.html'"></div>
    </div>
  </div>
  ... 
</div> 

This way we bring minimal changes to the original template (and become less-dependent of it), and also get full freedom to modify our custom template.

 

 

 

Creating custom Uchiwa template

As you don't need to be an experienced Ruby developer to write Chef cookbooks, so you don't need to be Angular JS expert to customize Uchiwa templates.

At first,
we use Angular 'ngInit' directive to define variables dashboard_base and kibana_base, for later use:

<div ng-init="dashboard_base='<%= @grafana_url %>/#/dashboard'; kibana_base='<%= @kibana_url %>/#/dashboard/file';"> 

Next,
we add link to Grafana scripted dashboard and pass client name (hostname) as the argument. Link to Kibana is constructed in the same way. 'client.name' is Uchiwa expression, enclose it with double curly braces {{client.name}}  to render its value.
'fa-bar-chart-o' and 'fa-tachometer' belong to Font-Awesome

 <!-- unconditional link -->
 <span class="small-nav"> Instance 
   <a ng-href="{{dashboard_base}}/script/instance.js?i={{client.name}}" target="_blank">
     <i class="fa fa-tachometer"></i></a>
   <a ng-href="{{kibana_base}}/logstash.json?query=host:\&quot;{{client.name}}\&quot;" target="_blank">
      <i class="fa fa-bar-chart-o"></i></a>
 </span>

Each client has a list of subscriptions (a list of assigned Chef roles in our case). We can iterate ('ngRepeat') through the list, and check ('ngIf') subscription names:

   <span ng-repeat="subscr in client.subscriptions">
     <!-- Backend data processes -->
     <span ng-if="subscr.match('(^backend_app_).*')">
      <span>
       | {{subscr|extractName:'_app_'}}
       <a ng-href="{{dashboard_base}}/script/process.js?p={{subscr|extractName:'_app_'|lowercase}}" target="_blank">
         <i class="fa fa-tachometer"></i></a>
       <a ng-href="{{kibana_base}}/logstash.json?query=process:\&quot;{{subscr|extractName:'_app_'|lowercase}}\&quot;" target="_blank">
          <i class="fa fa-bar-chart-o"></i></a>
      </span>
     </span>

Client 'i-abcd1234' is subscribed for 'backend_app_collector' and 'backend_app_aggregator'. If next month we decide to develop and deploy new 'Vaporizer' process, we will deploy it with Chef role 'backend_app_vaporizer'  to match the regular expression.

Construction '{{subscr|extractName:'_app_'}}' is a custom  Filter applied to 'subscr' value. It returns a sub-string after '_app_'.
Examples:

{{'backend_app_collector'|extractName:'_app_'}} returns 'collector'
{{'Mongo_cluster_Identities'|extractName:'_cluster_'}} returns 'Identities'
{{'ES_cluster_name_production'|extractName} returns 'production'
{{'Putin_huylo_la_la_la'|extractName:'Putin_'|uppercase}} returns 'HUYLO_LA_LA_LA'


This is our custom filter, and, normally, it's code should be added to /public/js/filter.js file where all Uchiwa filters are defined:

filterModule.filter('extractName', function() {
  return function(input,item) {
    var re = new RegExp('(.*)' + ((item)?item:'_name_') + '(.+)$', 'i');
    var res = re.exec(input);
    return (res)?res[2]:input;
  };
});

We can, for example, create another filter to parse event output , or to build full '<img>' object for specific Graphite metric provided in client data or/and in alert payload.

As we don't want to modify original Uchiwa .js files (and loose all custom filters on next Uchiwa upgrade), we define it in a separate file  '/opt/uchiwa/src/public/js/filters-custom.js' instead, and include it in original '/opt/uchiwa/src/public/index.html'.

<!-- Uchiwa JS -->
<script src="js/app.js"></script>
<script src="js/constants.js"></script>
<script src="js/controllers.js"></script>
<script src="js/directives.js"></script>
<script src="js/factories.js"></script>
<script src="js/filters.js"></script>
<script src="js/filters-custom.js"></script>
<script src="js/providers.js"></script>
<script src="js/services.js"></script>
</body>
</html>

This was the second and the last change to original Uchiwa code.


Another example:
add ElasticSearch dashboard link if subscription matches 'Elasticsearch_cluster_*'  but not 'Elasticsearch_cluster_checks*' or 'Elasticsearch_cluster_momitoring*' :

     <span ng-if="subscr.match('(^Elasticsearch_cluster_(?!checks|monitoring)).*')">
      <span>
       | ES cluster: {{subscr|extractName:'_cluster_'}}
        <a ng-href="{{dashboard_base}}/db/es-cluster?var-name={{subscr|extractName:'_cluster_'|lowercase}}" target="_blank">
         <i class="fa fa-tachometer"></i></a>
      </span>
     </span>
  </span>
 </span>

If we have many more unique subscriptions, it is not good to create  'ngIf' block for each of them. Instead, we can list them all in one hash table and iterate:

 <!-- define regular processes, snowflakes, legacy processes, etc
      TODO: store it in Chef configuration and generate hash table during the chef-client run-->
 <span ng-init=" processes=[
   {role:'Collector_process',  dashboard:'db/old_collector',   arg:''},
   {role:'Aggregator_old',     dashboard:'db/old_aggregator',  arg:''},
   {role:'Aggregator2',        dashboard:'db/old_aggregator',  arg:''},
   {role:'Inferno_legacy_app', dashboard:'db/old_crawler',     arg:'var-queue=All'}
   ]" >
   <span ng-repeat="subscr in processes">
     <span ng-if="client.subscriptions.indexOf(subscr.role) > -1">
       | {{subscr.role}}
        <a ng-href="{{dashboard_base}}/{{subscr.dashboard}}&{{subscr.arg}}" target="_blank">
               <i class="fa fa-tachometer"></i></a>
     </span>
   </span>
 </span>

To the moment, we have been adding custom dashboard objects based on client data (name,subscriptions) only.
We may want to add more objects only when specific check is selected in Uchiwa:

 <span ng-if="selectedCheck">
   <!-- check_name is one of the below -->
     <span ng-if="['disk_usage','swapping','disk_iostat'].indexOf(selectedCheck.check) > -1">
       | <a ng-href="{{dashboard_base}}/script/detailed_io.js?i={{client.name}}" target="_blank">disk IO</a>
     </span>
 </span>

Few words about checks and events.

As you probably know, if Sensu check returns the status 0 (success), the output is discarded by Sensu so there's no way to access it in Uchiwa (there is no event data).

If check status is 1(warning) or 2(critical) then event data are available via  Uchiwa 'selectedEvent'.

This way we can distinguish between successful and failing checks, for example,
 show button only if particular check fails (Button onClick() can send a parameterized API request to your Rundeck server).

Another example:
For any failing Jenkins_* check show link to documentation page (if available) and render nested template. Documentation address is obtained via custom attribute 'playbook' from Sensu event payload. Nested template contains additional logic inside (like display build tests page in iframe) and helps us to keep our custom.html clean.

<div ng-if="selectedEvent">
  <div ng-if="selectedCheck.check.match('^jenkins_.*')">
    <span ng-if="selectedEvent.check.playbook">
      <a ng-href="{{selectedEvent.check.playbook}}">Help</a>
    </span>    
    <div ng-include src="'partials/client/jenkins.html'"></div>
  </div>
</div>

Full Gist is available.



If you ever looked for a tool to glue your Ops tools together - Uchiwa is a good candidate to become such a 'master dashboard'. You see how powerful Angular templates are and how easy it is to add a custom template.
You don't need to be Angular JS expert, you just need to know few basic things -  how to define variables (ngInit), check conditions (ngIf ) and iterate (ngRepeat).
And it is only 2 lines of code different from original Uchiwa. yet.

If Uchiwa could provide a way to define custom filters and custom templates - it would be great. A possible way to define custom filters has been explained, perhaps, template path could be made customizable too (in 'app.js'): 

var templateDir = 'partials';

angular.module('uchiwa').config(['$routeProvider', 'notificationProvider', '$tooltipProvider',
  function ($routeProvider, notificationProvider, $tooltipProvider) {
    $routeProvider
      .when('/', {redirectTo: function () { return '/events'; }})
      .when('/events', {templateUrl: templateDir + '/events/index.html', reloadOnSearch: false, controller: 'events'})
      .when('/client/:dcId/:clientId', {templateUrl: templateDir + '/client/index.html', reloadOnSearch: false, controller: 'client'})
      .when('/clients', {templateUrl: templateDir + '/clients/index.html', reloadOnSearch: false, controller: 'clients'})
      .when('/checks', {templateUrl: templateDir + '/checks/index.html', reloadOnSearch: false, controller: 'checks'})
      .when('/info', {templateUrl: templateDir + '/info/index.html', controller: 'info'})
      .when('/stashes', {templateUrl: templateDir + '/stashes/index.html', reloadOnSearch: false, controller: 'stashes'})
      .when('/settings', {templateUrl: templateDir + '/settings/edit.html', controller: 'settings'})