JMeter: Using property function to fetch data from user input

If our objective is to run our test scripts from the command line with variable number of users and ramp up period without editing our test plan every now and then, we could do this using JMeter’s simplified property function(“__P”).

First we need to set our Thread Group property as follows:

Thread group thread and ramp-up properties

We could also use the property function when saving our test results.

Save graph result to a file

From the above settings, Jmeter will automatically generate a graph.jtl file inside result_dir folder.

Notice that number of threads, ramp-up period, result filename are all set as parameters.

After adding samplers and listeners to our testplan. We run our test from the command line by invoking:

jmeter -n -t test.jmx -Jgroup1.threads=10 -Jgroup1.ramp=5

From here onwards we could set different number of threads and ramp up period depending on the scenarios we’d like to test. Cool huh!

JMeter: Run scripts from the console

Running Jmeter scripts from the console is one way of reducing the resources needed to execute the load as compared to a GUI mode.

From the console, we say:
jmeter -n -t <testplan filename> -l <listener filename>
(e.g. jmeter -n -t testplan.jmx -l listener.jtl)

The first time I tried to run my scripts from the command line, this is what I exactly saw:

D:\Jmeter\bin>jmeter -n -t testplan.jmx -l listener.jtl
 Created the tree successfully using testplan.jmx
 Starting the test @ Fri Oct 28 07:50:29 CST 2009 (1256255429500)
 Waiting for possible shutdown message on port 4445
 Tidying up ...    @ Fri Oct 28 07:50:34 CST 2009 (1256255434468)
 ... end of run

This information tells us that execution was successful. But if I wanted to see more results of the test, Jmeter provides a listener that displays the summary results also from the console where we type our commands.

We do this by adding the “Generate Summary Results” listener in our testplan.

Again we type the same command in the console:

D:\Jmeter\bin>jmeter -n -t testplan.jmx -l listener.jtl
Created the tree successfully using testplan.jmx
Starting the test @ Thu Oct 29 11:23:49 CST 2009 (1256786629265)
Waiting for possible shutdown message on port 4445
Generate Summary Results +     5 in   8.5s =    0.6/s Avg:   981 Min:   296 Max:  2344 Err:     0 (0.00%)
Generate Summary Results +     1 in   0.2s =    4.3/s Avg:   232 Min:   232 Max:   232 Err:     0 (0.00%)
Generate Summary Results =     6 in   9.6s =    0.6/s Avg:   856 Min:   232 Max:  2344 Err:     0 (0.00%)
Tidying up ...    @ Thu Oct 29 11:24:02 CST 2009 (1256786642453)
... end of run

From here we could clearly see the results happening per thread which I prefer more than the first.

Here are some commands that we could use when running test in nonGUI mode:

-h, –help -> prints usage information and exit
-n, –nongui -> run JMeter in nongui mode
-t, –testfile <argument> -> the jmeter test(.jmx) file to run
-l, –logfile <argument> -> the file to log samples to
-r, –runremote -> Start remote servers (as defined in remote_hosts)
-H, –proxyHost <argument> -> Set a proxy server for JMeter to use
-P, –proxyPort <argument> -> Set proxy server port for JMeter to use

example:

jmeter -n -t test1.jmx -l logfile1.jtl -H 192.168.1.1 -P 8080

could clearly see the results per thread which I prefer more than the first.

JMeter: Simple Data Writer + Current Date

My first time to use Simple Data Writer, I have no idea how it works. So I figured it’s time to learn a new JMeter report (wink)

1. Add Simple Data Writer listener to my Google search script

2. Define its filename(where the generated result file will be saved) to:

D:\QA-Training\jakarta-jmeter-2.3.4\bin\reports\google.log.txt

3. Under “Log/Display” tick “Successes”

4. Run the test

I then opened google.log.txt in Open Office which is a comma delimited file and got this data:

Looks helpful right? Except for the timeStamp which I can’t decipher the format… yet

Anyway, rerunning my test scripts updates the log file and appends the new results at the end of the file. But what if you’d like to parameterized your report and include the current time in your log file everytime you execute your test?

JMeter can absolutely do this… since it supports certain Functions and Variables

One of which is the __time function which returns the current time in various formats:

• YMD = yyyyMMdd
• HMS = HHmmss
• YMDHMS = yyyyMMdd-HHmmss
• USER1 = whatever is in the Jmeter property time.USER1
• USER2 = whatever is in the Jmeter property time.USER2

From my current simple data writer listener, I edited its filename to:

D:\QA-Training\jakarta-jmeter-2.3.4\bin\reports\google.log.${__time(YMDHMS)}.txt

With this configuration, Jmeter generates a new log file with the current date and time in yyyyMMdd-HHmmss format, everytime I run the test. With regards to the timeStamp details, I’m still on my way to find it.

Bumper Stickers for Software QA

Here’s my personal favorite:

* Software Testing: Where failure is always an option.
* Improving the world one bug at a time.
* Software Testing: You make it, we break it.
* Software Testers don’t break software; it’s broken when we get it.
* Software Testers: We break it because we care.
* If developers are so smart, why do testers have such job security?
* Life is too short for manual testing.
* Trust, But Verify.
* The Definition of an Upgrade: Take old bugs out, put new ones in.
* We break software so you don’t have to.
* I used to build software…now I break it! Its a lot more fun!!
* All code is guilty, until proven innocent.
* It’s Automation, Not Automagic!
* Quality Assurance, we take the blame so you don’t have to.
* In God we trust, and for everything else we test.

Pick yours in the list. =)

Bumper Stickers for Software QA

Run your RC server via Rakefile

Gone tired of starting your selenium server by typing,

“java -jar selenium-server.jar"

from your console or if you’re luckier create a batch file then run it to start your server...


Here’s a more powerful tool to do it. Utilizing the selenium-client ruby gem, first you need to create a rakefile and define the tasks that you want to do. For instance we want to tell our rake to do the following tasks:

1. Start selenium server,

2. Stop selenium server, and

3. Restart selenium server

require 'rubygems'
require 'selenium/rake/tasks' # use selenium rc rake tasks that are bundled with the selenium-client gem

SELENIUM_RC_JAR = Dir[File.dirname(__FILE__) + "/selenium-server*.jar"].first

# Start selenium server task
Selenium::Rake::RemoteControlStartTask.new(:'rc:start') do |rc|
 rc.port = 4444
 rc.timeout_in_seconds = 3 * 60
 rc.background = true
 rc.wait_until_up_and_running = true
 rc.jar_file = SELENIUM_RC_JAR
 rc.additional_args << " -multiWindow -firefoxProfileTemplate FirefoxProfile"

end

# Stop selenium server task
Selenium::Rake::RemoteControlStopTask.new(:'rc:stop') do |rc|
 rc.host = "localhost"
 rc.port = 4444
 rc.timeout_in_seconds = 3 * 60
end

# Restart selenium server task
desc "Restart Selenium Remote Control"
task :'rc:restart' do
 Rake::Task[:"rc:stop"].execute [] rescue nil
 Rake::Task[:"rc:start"].execute []
end

In your console go to the root folder where your rake file is saved, type
“rake -T” or “rake –task”

D:\Projects>rake -T

Console will list all available tasks defined in the rake:

(in D:/Projects)
rake rc:restart  # Restart Selenium Remote Control
rake rc:start    # Launch Selenium Remote Control
rake rc:stop     # Stop Selenium Remote Control running

From this list you can restart/start/close the server by simply invoking the commands  e.g. rake rc:start

Selenium and Ruby setup

Newbie in Ruby? Never heard Selenium? Had a reformat?
No sweat! Here’s a list of requirements to set up Selenium and Ruby in your Windows machine.

SELENIUM Remote-Control (RC )

–> is a test tool that allows you to write automated web application UI tests in any programming language against any HTTP website using any mainstream JavaScript-enabled browser.

• get it here

• extract the file anywhere in you local machine

JAVA

–> Requirement to run the selenium server, should be 1.5 or later version

• get it here

• install Java and configure your PATH environment variable correctly

• from the console, you can verify the installation by typing:  java -version

SELENIUM RC RUBY CLIENT DRIVER

• requires Ruby

• recommended way to install: Ruby 1.8.6 One-Click Installer

• upgrade to latest version by getting the latest binary files of Ruby 1.8.6 from Ruby Source Code

Rubygems

• get it as a tgz or zip

• extract the archive

• 2 ways to install ruby:

from the console, go to the extracted directory then type:  ruby setup.rb

from your explorer, go to the extracted directory and double click setup.rb file

• install the ruby client driver as a rubygem by typing:  gem install <ruby gem>

gem install selenium-client

• install other gems that will be useful in your testing:

gem install rspec -v=1.2.6

gem install syntax

gem install faker

BDoc documentation

–> guide for all your rubygems documentations

• from the console type: gem install aptinio-bdoc

• then save the doc to your local, from the console  type:  bdoc //[path]

To get scripts or file from a repository through Git

1. from the console, install github, bash or gui OpenInGitGui-2.0.zip
2. generate an SSH rsa, type: $ ssh-keygen -t rsa
3. copy the generated id_rsa pub
4. paste it on the personal settings of ur public_key
5. then clone

Get all hyperlinks within a page using Nokogiri

Task: Create a selenium script using Ruby that will collect all the available links within a page.

In essence we will try to create a method that will parse the html source of the current page and get all the elements with css(‘a’) or xpath ‘//a’ which indicates an anchor element. First let’s try to do it in IRB.

Steps:

1. Start your server and fire up your irb

2. In your console, type

require 'nokogiri'

3. Initialize the page we want to test, say we want to get all the hyperlinks within a google home page.

page = "http://www.google.com.ph"

4. Type the following commands

doc = Nokogiri::HTML(open(page))
links = doc.css('a')
hrefs = links.map {|link| link.attribute('href').to_s}.uniq.sort.delete_if
{|href| href.empty?}

Of course we would not like to do the procedure every time in our console, thus we could save it as a method in our class like the following:

# method that will get all links using Nokogiri
 def get_all_hrefs_nokogiri
   page = self.get_location()
   doc = Nokogiri::HTML(open(page))
   links = doc.css('a')

   hrefs = links.map {|link| link.attribute('href').to_s}.uniq.sort.delete_if {|href| href.empty?}
   return hrefs
 end

# get all links without using Nokogiri
 def get_all_hrefs
   hrefs = []
   self.get_xpath_count('//a').to_i.times do |i|
     if self.is_element_present("document.links[#{i}]") {hrefs << self.get_attribute("document.links[#{i}]@href")}
     end
     return hrefs
   end
 end

Convert XML to CSV with Nokogiri Ruby gem

Once upon a time, in an exciting world of software testing… Exist QA team had been using Testlink 1.8.3 as an open-source tool for test management. They were happy and it serves them well not until their client request for a copy of the testcases with complete details in EXCEL format. Doomed! Testlink only offers generation of test specification in HTML, OpenOffice Writer and MS Word but unfortunately not in EXCEL.

But just like a princess with a prince charming… then came Nokogiri(saw in Japanese) gem from Ruby which is an HTML, XML, SAX  and Reader parser. It supports document searching via XPATH and CSS3 Selectors. Not to mention FasterCSV also a Ruby gem which provides a complete interface to CSV files and data.  It offers tools to enable you to read and write to and from Strings or IO objects, as needed.

First they install these precious gems in their Windows machine by executing the following commands:

gem install nokogiri
gem install fastercsv

With these tools Exist QA carefully plans a plot to solve their problem. Since Testlink has the ability to export testsuite together with its testcases in XML format, they use this advantage to pass it as an input file in their Testlink parser code in Ruby. Here’s their gameplan:

require 'rubygems'
require 'nokogiri'
require 'fastercsv'

FIELDS = %w{Testsuite ID Name Summary Steps Expected_Result }

def new_testcase(csv, suite, id, name, summary, steps, expectedresult)

  testcases = []
  testcases << suite
  testcases << "GPC - #{id}"
  testcases << name
  testcases << summary
  testcases << steps
  testcases << expectedresult

  csv << FasterCSV::Row.new(FIELDS, testcases)
end  

csv = FasterCSV.open(ARGV[1],"w")
csv << FIELDS

doc = Nokogiri::XML(open(ARGV[0]))

doc.xpath('//testsuite').each do |tsuite|
  puts "#{tsuite.attribute('name')}\n"

  doc.xpath('//testcase').each do |tcase|
    new_testcase(csv, tsuite.attribute('name'), tcase.css('externalid').inner_text,
      tcase.attribute('name'), tcase.css('summary').inner_text,
      tcase.css('steps').inner_text, tsuite.css('expectedresults').inner_text)
  end
end

All they need to do is run the program in their console following this format:

ruby <filename> “<input>” “<output>”

Where filename is the name of the Testlink parser code; input is the xml filename(generated XML file from Testlink) and output is the csv filename(file where the parsed xml data will be saved).

ruby tlparser.rb “test.xml” “test.csv”

Nokogiri and FasterCSV saves the day! Now they can provide the testcase report in no time, every time their client request for it. And Exist QA lives happily ever after…

Testlink Database SQL query

Objective # 1: Filter all testcases under Globe Portal Capabilities testplan (testplan_id = ‘9007′) and include summary, steps and expected result in the query

testplan_tcversions			tcversions
field	description		field	description
id	internal id for testcase versioning		id	unique id per testcase
testplan_id	id that determines the testplan		tc_external_id	external id
tcversion_id	unique id per testcase		version	latest version
node_order	order of the node		summary	summary of testcase
urgency	degree of urgency		steps	steps of testcase
			expected_results	expected result
			importance	degree of importance
			author_id	author
			creation_ts	date of creation
			updater_id	updater
			modification_ts	date of modification
			active	is active?
			is_open	is open?
			execution_type	type of execution 1-manual;2-automated

SQL query:

SELECT * FROM testplan_tcversions LEFT JOIN tcversions ON tcversions.id = testplan_tcversions.tcversion_id

WHERE testplan_tcversions.testplan_id = '9007'

Problem: Testcases have does not indicate its component/testsuite (Contacts, Search, Widgets etc.)

Objective # 2: Include Component(from nodes_hierarchy table) for each testcase in the query

nodes_hierarchy			node_types
field	description		field	description
id	unique id per testcase		id	id of node
name	description / name		description	description of node
parent_id	id of the parent node
node_type	type of node		node_types table
			id	description
			1	testproject
			2	testsuite=Component
			3	testcase=Testcase Summary
			4	testcase_version=Testcase Steps
			5	testplan
			6	requirement_spec
			7	requirement

SQL query:

SELECT a.*,  b.*, c.parent_id, c.node_type FROM testplan_tcversions a LEFT JOIN tcversions b ON b.id = a.tcversion_id LEFT JOIN nodes_hierarchy c ON a.tcversion_id = c.id

WHERE testplan_tcversions.testplan_id = '9007'

Problem: parent_id for testcases with node_type = ‘4′ is not sufficient to get the testcase component/testsuite

e.g.

nodes_hierarchy
id	parent_id	node_type	description	value
9068	9067	4	testcase steps	< steps >
9067	9010	3	testcase summary	User can view his Contacts
9010	9006	2	testsuite/component	Personalizations: Contacts
9006	-	1	testproject	Gportal Capabilities

In our result query for id = ‘9068′, ‘9067′ will only give us the summary/title of the testcase but not its component/testsuite , our next objective is to get the parent_id of ‘9067′

Objective # 3: Include the parent_id of  each parent_id for all the testcases in the query. Just like the following

id	parent_id	parent_id2	name
9068	9067	9010	Personalizations: Contacts

To get the parent_id for each parent_id from the nodes_hierarchy table, we create a subquery:

SELECT x.id, x.name, x.node_type_id, x.parent_id, y.parent_id as parent_id2

FROM nodes_hierarchy x left join nodes_hierarchy y ON x.parent_id = y.id

Then add our subquery to our major query in Objective # 1:

SELECT a.*, b.*, d.*
FROM testplan_tcversions a
LEFT JOIN tcversions b on a.tcversion_id = b.id
LEFT JOIN (<em>select x.id, x.name, x.node_type_id, x.parent_id, y.parent_id as parent_id2 from nodes_hierarchy x left join nodes_hierarchy y on x.parent_id = y.id</em>) d on a.tcversion_id = d.id
WHERE a.testplan_id = '9007'

Hats off Kuya Ed!

Understanding Performance Test and its Metrics

There are a lot of definition that you could draw out of the concept “Performance testing,” one of these, which I found brief and simple:

Performance testing is the process by which software is tested and tuned with the intent of realizing the required performance.

Regardless of the many terms that you could relate to “Performance,” like load, stress, spike, soak etc. There are three major categories that you should focus on when you do performance test:

Speed — Does the application respond quickly enough for the intended users?

Scalability — Will the application handle the expected user load and beyond?

Stability — Is the application stable under expected and unexpected user loads?

And in order for you to objectively measure the above categories, you need to carefully identify the suitable performance metrics to be used. To give you an overview of performance metrics, here are some of useful information from RadView Software’s White Paper:  Test Metrics – Which Are Most Valuable?

During a test session, virtual clients generate result data (metrics) as they run scenarios against an application. These metrics determine the application’s performance, and provide specific information on system errors and individual functions. Understanding these different metrics will enable you to match them to the application function and build a more streamlined test plan.

Scalability and Performance

1. Hits per Second

- a Hit is a request of any kind made from the virtual client to the application being tested. The higher the Hits Per Second, the more requests the application is handling per second. A virtual client can request an HTML page, image, file, etc.

2. Pages per Second

- measures the number of pages requested from the application per second. The higher the Page Per Second the more work the application is doing per second.

3. Throughput

- this is an important baseline metric and is often used to check that the application and its server connection is working.  Throughput measures the average number of bytes per second transmitted from the application being tested to the virtual clients running the test agenda during a specific reporting interval.  This metric is the response data size (sum) divided by the number of seconds in the reporting interval.

4. Rounds

- tells you the total number of times the test agenda was executed versus the total number of times the virtual clients attempted to execute the Agenda. The more times the agenda is executed, the more work is done by the test and the application.

Responses and Availability

1. Hit Time

- hit time is the average time in seconds it took to successfully retrieve an element of any kind (image, HTML, etc).  The time of a hit is the sum of the Connect Time, Send Time, Response Time and Process Time. It represents the responsiveness or performance of the application to the end user.

2. Time to First Byte

- this measurement is important because end users often consider a site malfunctioning if it does not respond fast enough.  Time to First Byte measures the number of seconds it takes a request to return its first byte of data to the test software’s Load Generator.

3. Page Time

- page time calculates the average time in seconds it takes to successfully retrieve a page with all of its content.  This statistic is similar to Hit Time but relates only to pages. In most cases this is a better statistic to work with because it deals with the true dynamics of the application.

With regards to choosing the performance metric, you should always consider the type of application that you’re testing. Say for an open and public web application where you expect that many concurrent users will hit the system at the same time, HIT PER SECOND would be a valuable metric to use, compared to an in-house application like accounting system where you could explicitly tell how many client will be using the application, HITS PER SECOND would be irrelevant.