BugzillaMetrics Guide

The selections in the upper part of the page enable to filter the cases that are considered for the evaluation of a metric. This filter is also referred to as base filter since it determines the base set of cases for the evaluation. All listboxes allow multiple selection. If nothing is selected in a listbox there will be no filtering for the aspect represented in the listbox. Additionally cases can be filtered for the values of flags, and it can be filtered using regular expressions.

The widgets displayed in the Base Filter can be configured by the administrator (see Configuration of the Base Filter Widget).

The filter is applied at the end of each time interval (e.g. each week). Only the cases that match the filter will be considered for the calculation of a metric value for this time interval.

The grouping parameter can be used to show distinct time series for different partitions of the evaluated cases. Grouping by product for example will result in a chart with separate plots for each product. Selecting multiple grouping parameters will result in a chart with a plot for each combination of the possible values of the grouping parameters.

The evaluation period determines the time span for which values are calculated. By selecting "always till today" and saving the query, the end date will be adjusted each time the query is reloaded.

The time granularity determines for which time intervals the data points will be calculated. Possible values are day, week, month and year. If the time granularity is week, month or year the calculation will include the whole time interval to which the start and end date belong.

Data points will always be drawn in the chart at the first day of a week, month or year. Monday is considered to be the first day of a week.

Queries can be saved from each page. When saving a query the user must enter a name and a description of the query. The query name must be unique. The description of the query as well as the private/public visibility can be changed later. Public queries can only be changed by the user who initially saved them. Be aware that changing public queries might confuse other users.

Saved queries can be evaluated directly as well as reloaded in different ways. A reloaded query can then be modified.

BugzillaMetrics provides static links for metric results and saved queries.

To generate a static link for one or several saved queries, select these queries and click the button "Static Link". This will display a dialog with a static link. The given link will automatically trigger the metric calculation.

Metric results will also be available by a static link. To get the static link to metric results, use the button "Static Link" at the bottom of the result page. This simplifies to access metrics results again at a later point in time. Metric results are stored for a time specified by the administrator (default is 31 days).

This section will explain some concepts that are used in all of the category pages.

The evaluation of a metric in BugzillaMetrics is done in the following steps:

The following sections will explain some of the mentioned terms in more detail.

Here you can specify metrics based on selecting cases on which a certain event had occurred in a time interval.

To define a metric you have to choose an event filter and you have to choose a weight. The event filter specifies which case have to be considered in each time interval. Then a number will be assigned to each case. This number is called case value . The case value is determined by the weight (see Section 2.5.1.2, “Weight” ). In each time interval the end result for the metric will then be determined by the given calculation, e.g. the sum of the case values or the average of all case values.

An example metric specified from the page is the number of incoming cases (incoming rate). In order to count the incoming cases you have to consider the following events:

You have to select the corresponding items in the "Event filter" list. The items are implicitly connected by OR.

Since the incoming cases should just be counted, you can use the default weight. This will assign 1 to each case value. Then the case values are summed up by selecting the calculation "sum".

Reload the metric from the "My Queries" page to see the definition in the "Count Events" page.

Here you can specify metrics based on counting how often some event occurs for a certain case until a another specific event takes place.

To define a metric you have to choose two event filters denoted as "Event Count" and "Until". The first one specifies the events that are to be counted in the evaluation. The second event filter specifies the event on which the counting will stop.

A case will be considered in the evaluation at the point of time when the "Until" event occurs on this case for the first time. Then the number of occurrences of the event to be counted will be assigned as the case value. In each time interval the end result for the metric will then be determined by the given calculation, e.g. the maximum of the case values or the average of all case values.

An example metric specified from the page is the number of assignee changes before resolution. The first event filter will say that you want to count the number of assignee changes (a transition occurs on the field assignee). The "Until" filter will say that counting ends when the status changes to "Resolved", specified as a status transition from the set {"New","Assigned"} into one of the states {"Resolved", "Verified", "Closed"}.

Reload the metric from the "My Queries" page to see the definition in the "Count Until" page.

Here you can specify metrics based on the length of a time interval between the occurrence of two specified events.

To define a metric you have to choose two event filters denoted as "From" and "To". The first one specifies the beginning of the time interval. The second event filter specifies the end of the time interval.

A case will be considered in the evaluation each time when the "To" event occurs on a case. Then the time in days between the occurrences of the events will be assigned as the case value. In each time interval the end result for the metric will then be determined by the given calculation, e.g. the maximum of the case values or the average of all case values.

More advanced options for the Interval Length Calculation can be configured in the XML metric specification (see Section 3.5.3, “Interval length between two events”).

An example metric that can be specified from this page is "Age at Resolution". The "From" event filter will describe the creation of a case. The "To" event filter will say that the interval ends when the status changes to "Resolved", specified as a status transition from the set {"New", "Assigned"} into one of the states {"Resolved", "Verified", "Closed"}. Then the calculation will define that we are interested in the average age of the resolved cases. Reload the metric from the "My Queries" page to see the definition in the "Interval Length" page.

This evaluation will calculate the total time in days a case stayed in a certain state before an event occurred. This total time can include more than one time span. It sums up all time spans in which the case was in the specified state during its life time until the event takes place.

To define a metric you have to define the state of interest by specifying one or multiple status values. Then you have to define the event after which a case will no longer be considered.

A case will be considered in the evaluation each time the specified event occurs on a case. Then the total time in days the case had resided in that state will be assigned as the case value. In each time interval the end result for the metric will then be determined by the given calculation, e.g. the maximum of the case values or the average of all case values.

An example metric that can be specified from this page is "Residence Time in Status New". The selection of "state" will describe which status values should be considered when summing up the residence time. For this metric only the status "New" needs to be selected. The event filter will define when a case will no longer be considered, specified as a status transition from the set {"New","Assigned"} into one of the states {"Resolved", "Verified", "Closed"}. Reload the metric from the "My Queries" page to see the definition in the "Residence Time" page.

The details evaluation shows all case values for a certain case value calculation. In the next example, a details evaluation for the case value calculator "incoming rate" is shown. The evaluation has the name "incoming rate", too.

<details name="incoming rate"
         caseValueCalculator="incoming rate" />

Calculations use mathematical operations on the results of the case value calculation to calculate a number for the metric in a flexible way.

Operations based on sets of case values allow accessing the results from the case value calculation. The following case value set based operations are available:

These operations need to be parameterized with the identifier of a case value calculation. The following example sums the values from the case value calculator with the ID "incoming rate".

<sum caseValueCalculator="incoming rate" />

The constant operation is available for using constants in the calculation:

<constant>100.0</constant>

Binary operations can be used to combine results from other operations. The two child elements are the subcalculations that provide the results that are used as input for the binary operation. The available binary operations are:

The following example shows the calculation of defect rate, which is the percentage of bugs in bugs and feature requests. It uses all three different kinds of operations.

<calculation name="defect rate">
  <divide>
    <multiply>
      <constant>100.0</constant>
      <sum caseValueCalculator="bugs" />
    </multiply>
    <sum caseValueCalculator="bugs and enhancements" />
  </divide>
</calculation>

This calculator calculates a case value on an event specified by an event filter . The calculation is based on the weight the calculator is parameterized with. The following snippet shows the XML element for the count events calculator:

<countEvents id="id">
  <event>
    an event filter...
  </event>
  <weight>
    a weight...
  </weight>
</countEvents>

This calculator calculates counts the number of times an event has occurred for a case until another event happened. Both events can be specified by an event filter . The following snippet shows the XML element for the calculator:

<countEventsUntil id="id">
  <event>
    an event filter...
  </event>
  <until>
    an event filter...
  </until>
</countEventsUntil>

This calculator calculates the length of the time interval in days between two events that happen on a case. Both events can be specified by an event filter . The following snippet shows the XML element for the calculator:

<intervalLength id="id">
  <from>
    an event filter...
  </from>
  <to>
    an event filter...
  </to>
  <threshold thresholdInDays="7" useThresholdWeight="true" />
  <considerToEvent>eachTime<considerToEvent>
</intervalLength>

The "threshold" tag is optional. When this tag is used, the counting the length of the time interval will be stopped if the specified "to" event did not happen within this period. If the attribute "useThresholdWeight" is false, the number of days since the "to" event will be returned, but at most the days given in the threshold. If "useThresholdWeight" is true, the value 1 will be returned for cases that exceed the threshold, otherwise 0.

The "considerToEvent" tag is optional. Using this tag enables to specify when case values are created. Possible values are:

This calculator calculates the time in days a case was in a certain state before the time point of a certain event. At such an event, a case value is calculated. The event can be specified by an event filter and the state can be specified by a state filter. The following snippet shows the XML element for the calculator:

<stateResidenceTime id="id">
  <state>
    an state filter...
  </state>
  <event>
    an event filter...
  </event>
  <considerEvent>eachTime</considerEvent>
</stateResidenceTime>

The "considerEvent" tag is optional. Using this tag enables to specify when case values are created. Possible values are:

  <timePeriodGranularity>
    <month />
  </timePeriodGranularity>

Time periods can also be defined by the user to adjust metrics to release dates or business years.

  <timePeriodGranularity>
    <customGranularity>
       <aggregateAt>2006-01-15</aggregateAt>
       <aggregateAt>2006-08-16</aggregateAt>
       <aggregateAt>2006-12-03</aggregateAt>
       <aggregateAt>2007-10-03</aggregateAt>   
    </customGranularity>
  </timePeriodGranularity>

BugzillaMetrics works on a Bugzilla database from version 2.19.3 up to a least 4.0

BugzillaMetrics uses three custom tables to store common metrics, user-defined metric queries, and metric results. The tables can be placed in the Bugzilla database or in a separate database.

The following description shows how to deploy BugzillaMetricsFrontend from the file release into a tomcat installation. It requires to run tomcat at least with JAVA JRE 1.5.0_07 .

The following description shows how to deploy BugzillaMetricsFrontend from the source code into a tomcat installation.

It requires at least Eclipse 3.5 with WebTools Platform 2.0 and the Google plugin installed. You need to separately install the Google Web Toolkit. Please use GWT version 2.2 or later.

The Base Filter Widget displayed on the frontend can be configured by the administrator. The configuration is given in the tag baseFilter contained in the file settings.xml. The following picture shows the base filter widget with markers for the different types of elements that can appear in the widget.

The corresponding parts of the settings file are given in the following:

<baseFilterConfiguration>
	<frontendPresentationSettings>
	   <baseFilter>
		<enumList title="Classification" 
		          field="classification" 
		          width="90px" />
		...
		<space width="20px" />
		<groupingParametersListBox title="Grouping by:" 
		                           width="100px">
			<groupingParameter>product</groupingParameter>
			...
			<groupingParameter>assignee</groupingParameter>
		</groupingParametersListBox>
	</baseFilterRow>
	<baseFilterRow>
		<enumList title="Status" field="status" width="130px" />
		...
		<verticalPanel width="160px">
			<checkBox title="Only beyond deadline" 
			          field="beyondDeadline" />
			...
			<flagDropDown title="Flag:" width="100px" />
			
			<regExpDropDown title="" width="100px">
				<fields width="100px">
					<field>assignee</field>
					...
			        <field>targetMilestone</field>
				</fields>
			</regExpDropDown>
		</verticalPanel>
	</baseFilterRow> 
  </baseFilter>
  ...	
</frontEndPresentationSettings>

The tags of this file are explained in the following:

Furtheron the events displayed in the Count Events, Count Until, Interval Length, and Residence Time category, and weights displayed on the Count Events page can be configured. These settings are typically needed if BugzillaMetrics is used with the software configuration management extension, and related events and weight should also be visible on the frontend.

The metrics displayed on the "Common Metrics" tab page of the BugzillaMetrics frontend can be edited by changing the entries in the metrics database table. The typical way to do this would be to save your query from the frontend, then use an SQL statement like the following to copy this query into the metrics table.

INSERT INTO metrics SELECT name, description, query AS metric, 
chart from queries where id=1234

BugzillaMetrics supports the evaluation of custom fields defined in your Bugzilla installation. The following types of custom fields can be evaluated: FreeText, SingleSelect, MultiSelect, LargeTextbox, and DateTime.

To enable the evaluation of a custom field, an entry must be added to the file metric.xsd. The file is contained in \BugzillaMetricsFrontend\WEB-INF\lib\BugzillaMetrics.jar in the folder \org\bugzillametrics\core. (The jar-file can be opened as zip to edit the file.) The application does not need to be recompiled.

For a custom field called "cf_myCustomField" the following line must be added for the simpleType "field":

<xs:enumeration value="cf_myCustomField"/>

Make sure that the max_allowed_packet parameter of MySQL is at least 16MB. This setting can be changed in the configuration file /etc/my.cnf (See http://dev.mysql.com/doc/refman/6.0/en/program-variables.html).

Increase the java heap space of the servlet container. For example add the following option to your Tomcat startup script.

CATALINA_OPTS=="-Xmx2048m"

On a large Bugzilla database, the session timeout may be too low for some metric calculations. To increase the timeout value in Tomcat, please locate the following section in [Tomcat_home]/conf/web.xml:

<session-config>
     <session-timeout>30</session-timeout>
</session-config>

The timeout value is specified in minutes. Restart Tomcat after modifying the file, so the changes can take effect.

The ProxyPass directive allows remote servers to be mapped into the space of the local server; the local server does not act as a proxy in the conventional sense, but appears to be a mirror of the remote server.

If BugzillaMetrics is accessed using the ProxyPass directive, the timeout for the proxy request must be redefined. Otherwise a metric calculation may be timed out before the result is available, since some BugzillaMetrics calculation can take several minutes dependent on the size of the Bugzilla database. An example configuration is given in the following:

RewriteEngine on
RewriteRule ^/BugzillaMetrics$ /BugzillaMetrics/ [R]

#Timeout for proxy requests in seconds. 
ProxyTimeout 3600
Timeout 3600

ProxyPass /BugzillaMetrics/ 
          http://localhost:8080/BugzillaMetricsFrontend/ 
          keepalive=on
ProxyPassReverse /BugzillaMetrics/ 
                 http://localhost:8080/BugzillaMetricsFrontend/

# Adjusts the domain string / path of the session cookie                 
ProxyPassReverseCookieDomain 
                 http://localhost:8080/BugzillaMetricsFrontend/ 
                 public.example.com
ProxyPassReverseCookiePath / /BugzillaMetrics/

BugzillaMetrics can be configured to run on a database other than MySQL. You have to find a JDBC driver for your database. The JDBC driver for PostgreSQL can for example be downloaded from here. Copy the jar file of the JDBC driver into the directory /WEBINF/lib. Then add the attributes "jdbcdriver" (the class name of the jdbc driver class) and "subprotocol" (e.g. postgres or oracle) to the connection settings of the database given in the file settings.xml. Example:

<bugzillaDB 
		database="bugzilla" 
		user="bugmetrics" 
		password="password"
		host="localhost" 
		jdbcdriver="org.postgresql.Driver"
		subprotocol="postgres"
	/>
	     

This exception can be thrown during a metric evaluation and will be reported in the bugzilla_metrics.log file. It is caused by inconsistencies of the Bugzilla database. Bugzilla holds the change history in a table "bugs_activity". This table does not use IDs to refer to entities like products, assignees, etc. Instead the names of these entities are used. So the entries in the "bugs_activity" table must be resolved to the ids. Renaming an entity e.g. a product cause an inconsistency, since historic changes are no more associated with this entity.

The SCM integration works on a separate database that can be created in the following three steps:

Make sure that you have installed the following tools:

A file named "looging.properties" is used to configure logging during the import process. Download a template of this file from here. Usually you only have to adjust the path to the logging file (log4j.appender.A1.file).

A file named "settings.xml" is used to configure the import from CVS or Subversion". Download a template of this file from here. Configure the following items in your settings.xml file:

The content of the database is specified in a XML file. It consists of a root-element <db /> and two child elements <bugzilladb /> and <scmdb />. Both include a child element for each table entry in the Bugzilla- or SCM-database. The name of an element corresponds to the target table. For each column of the table there is an attribute named by the corresponding column.

The test framework uses metric specifications of the normal syntax. It offers the full range of functions.

The result of the metric calculation, which is a time series, is an XML file. The root element is <metricResult />. Its children are the specified groups: <group name="...">. For each of the calculated values of the time series, there is an element <timePeriod scope="..."> in which the calculations are stored: <calculation name="sum"> "value" </ calculation>.

Database content:

<db><bugzilladb>
  <user userid="1" />
  <user userid="2" />
  <product id="1" />
  <component id="1" />
  <bug bug_id="1" creation_ts="2006-08-07 12:00:00"
    priority="P4" assigned_to="2" />
  <bug bug_id="2" creation_ts="2006-08-14 12:00:00"
    priority="P1" />
  <bug bug_id="3" creation_ts="2006-08-21 12:00:00"
    priority="P2" assigned_to="2" />
  <bug bug_id="4" creation_ts="2006-08-28 12:00:00"
    priority="P3" />
</bugzilladb></db>    
    

Metric specification:

<?xml version="1.0" encoding="UTF-8"?>
<metric>
  <baseFilter>
    <value field="assignee">1</value>
  </baseFilter>
  <groupingParameters>
    <none />
  </groupingParameters>
  <groupEvaluations>
    <calculation name="sum" >
      <sum caseValueCalculator="default" />
    </calculation>
  </groupEvaluations>
  <caseValueCalculators>
    <countEvents id="default" >
      <event>
        <endOfTimeInterval/>
      </event>
      <weight>
        <mapping field="priority">
          <map from="P1" to="4" />
          <map from="P2" to="3" />
          <map from="P3" to="2" />
          <map from="P4" to="1" />
        </mapping>
      </weight>
    </countEvents>
  </caseValueCalculators>
  <evaluationTimePeriod>
    <timePeriod>
      <start>2006-08-14</start>
      <end>2006-08-27</end>
    </timePeriod>
  </evaluationTimePeriod>
  <timePeriodGranularity>
    <week />
  </timePeriodGranularity>
  <fixedFields />
</metric>
    
    

Expected result:

<metricResult>
  <group name="none">
    <timePeriod scope="week 33/2006">
      <calculation name="sum">4</calculation>
    </timePeriod>
    <timePeriod scope="week 34/2006">
      <calculation name="sum">4</calculation>
    </timePeriod>
  </group>
</metricResult>
    

The information for the SCM initialization includes the following main elements that are integrated in a XML schema:

The following operations can be used for the SCM setup:

The Import test launches an import process based on a SCM system with the above defined scenario. The result will be compared at database level. This is done using a database export from mysqldump as XML file. Because of differences depending on time and executing developer, the following entries are ignored:

The comparison is more flexible than a simple file-diff, but it has disadvantages:

Example: SCM test case for committing TestFile1.java in root folder of trunk (SVN repository):

Setup:

      <scm>
        <bugzilladb>
        </bugzilladb>
        <svn>
          <add sourceLocation=""
               sourceFileName="TestFile1.java"
               targetLocation="trunk"
               targetFileName="TestFile1.java" />
          <commit comment="bug1: Kommentar fuer den commit" />
        </svn>
      </scm>
    

Result comparison:

      <?xml version="1.0"?>
        <mysqldump xmlns:xsi=
          "http://www.w3.org/2001/XMLSchema-instance">
        <database name="scm">
	      <table_structure name="Developer">
            (...)
	      </table_structure>
	      <table_data name="Developer">
	        <row>
		      <field name="ID">1</field>
		      <field name="Bugzilla_Name">Error</field>
		      <field name="Scm_Name">root</field>
	        </row>
	      </table_data>
          (...)
        </database>
      </mysqldump>