Friday, April 13, 2012

Creating a delimited text file (test fixture) for Cucumber tests using Ruby.

This post is intended for anyone who needs to create a delimited text file in Ruby.  I talk a lot of specifics using my real world example, but you can can skip the Cucumber parts if that doesn't interest you.  I will briefly discuss other delimiters such as tabs, commas (for CSV files), etc.

A test fixture in software automation is data you have to have in place before you can run your test.  The Cucumber Book suggests using FactoryGirl, and I may very well move this code into that structure in the future when I get around to understanding it.  In the short term, I needed to generate a text file called a Ledes file.  I needed to create these files with data filled in from my Cucumber tests.  After I create the file, I upload it and manipulate it using the web pages on our system.

A Ledes file is used to transfer information between systems in the Legal world.  The standard was developed in 1998, and uses pipe | delimited text files with closed bracket line terminators [].  It's very simple.  A Ledes file is just headers and a bunch of line items.  Below is the file my code created for this example.

Ledes File


LEDES1998B[]
INVOICE_DATE|INVOICE_NUMBER|CLIENT_ID|LAW_FIRM_MATTER_ID|INVOICE_TOTAL|BILLING_START_DATE|BILLING_END_DATE|INVOICE_DESCRIPTION|LINE_ITEM_NUMBER|EXP/FEE/INV_ADJ_TYPE|LINE_ITEM_NUMBER_OF_UNITS|LINE_ITEM_ADJUSTMENT_AMOUNT|LINE_ITEM_TOTAL|LINE_ITEM_DATE|LINE_ITEM_TASK_CODE|LINE_ITEM_EXPENSE_CODE|LINE_ITEM_ACTIVITY_CODE|TIMEKEEPER_ID|LINE_ITEM_DESCRIPTION|LAW_FIRM_ID|LINE_ITEM_UNIT_COST|TIMEKEEPER_NAME|TIMEKEEPER_CLASSIFICATION|CLIENT_MATTER_ID[]
20120412|1334244177||||20120412|20120412|Automated Test Invoice||E|1|||20120412||E115|||Automated Test Line Item|12-3456789|33.42|||ABC.1D2345E[]
20120412|1334244177||||20120412|20120412|Automated Test Invoice||E|1|||20120412||E101|||Automated Test Line Item|12-3456789|127.94|||ABC.1D2345E[]


The first line is just a header for the file.  The second line lists all the headers for each line item.  The third and fourth lines list line items.  You will notice that I left many of the values blank.  This is because I did not need them for my testing.  It was still important to have the pipes there however.  As the Cucumber book suggests, we're going to develop outside in.  So here is my test:

features/create_invoice.feature


  Scenario: Upload Ledes File
    Given that I am logged in as a Firm User
    When I create an invoice with the following values:
      | LAW_FIRM_ID         | 12-3456789  |
      | CLIENT_MATTER_ID[]  | ABC.1D2345E |
    And I upload the invoice with the following line items:
      | EXP/FEE/INV_ADJ_TYPE  | LINE_ITEM_UNIT_COST | LINE_ITEM_EXPENSE_CODE  |
      | E                     |               33.42 | E115                    |
      | E                     |              127.94 | E101                    |
    Then the invoice should be created


As you can see, it's a very simple test.  First I make sure I am logged in as the correct kind of user.  The When/And steps are actually both needed to create an invoice.  The first table stores information that doesn't change between rows.  The second table holds the information that is different for each line item.

You will notice that in the first table, CLIENT_MATTER_ID is followed by brackets [].  This is because I'm directly passing the values through to my object that will create the file.  This means that if I make a typo in the first column of that table, I will get errors in my file.  Similarly, if I have typos in the first row of my second table, I will have errors in my file.  This was a conscious decision on my part when creating a Ubiquitous Language.  (Here is a more in-depth discussion on Ubiquitous Language and Agile Development.)

Everyone in my company is used to looking at and editing Ledes files.  When I use the exact headers in the steps, it is familiar to them.  They are used to seeing the line terminator with CLIENT_MATTER_ID, so I kept it in.  You may have noticed that the ID itself is also followed by the same line terminator in the file.  I left it out in in my table however, because everyone knows that it is not part of the ID, and it would be confusing if it was there and annoying to update the value in tests.

So now lets take a look at the step definitions.  I will skip the login code, as it is not germane to this discussion.  Let's look at the implementation of my When step:

features/step_definitions/create_invoice.rb - When Step


When /^I create an invoice with the following values:$/ do |table|
  @file_values = table.rows_hash
  @invoice_number = @file_values['INVOICE_NUMBER']
end


All this line does is pass information to the next step.  It's a bit of a kluge.  I'm creating two instance variables, and storing the data from the table in them.  First I'm turning my table into a hash using rows_hash which gives me a key/value pair for each row.  This method only works if you have two columns in your table.

Then I'm assigning the @invoice_number instance variable with whatever value is stored with the INVOICE_NUMBER key.  If I don't pass that value (which I didn't in this case), then @invoice_number will have a value of nil.  Why am I doing this?  Let's look at the definition of the And step:

features/step_definitions/create_invoice.rb - And Step


And /^I upload the invoice with the following line items:$/ do |table|
  if @invoice_number.nil?
    @invoice_number = Time.now.to_i.to_s
    @file_values['INVOICE_NUMBER'] = @invoice_number
  end
  filename = Dir.pwd.to_s + "/features/test_data/Automated Ledes File #{@invoice_number}.ledes"
 
  values = table.hashes

  my_file = LedesFile.new(filename)
  values.each do |line|
    line.merge!(@file_values)
    temp_line = LedesLineItem.new(line)
    my_file.write(temp_line)
  end


Meaty goodness.  The first thing I'm doing is checking to see if @invoice_number got a value in the last step.  If not, I'm going to make one up using the current timestamp.  I translate it into an integer to get a string of numbers, and then a string so that it's just a string value.  If I don't turn it into a string, I will have conflicts later on.  Then I'm going to pass my new invoice number into the @file_values hash, because I need it to be there too.

Then I'm going to create a file name.  Actually, it's a file path and file name.  Using Dir.pwd.to_s, I'm going to get my current directory and start my path off with that.  Whenever I run Cucumber, my directory changes to wherever the features/ folder resides.  I'm then creating the filename with the invoice number.  As you can see, if I pass in the invoice number from the script, it will get overwritten.  This is because you can only have one invoice with a given invoice number in the system at a time.

After I set filename, I'm going to create and fill a hash called values using the hashes method.  This turns the table into an array of hashes with the first row as the keys for all the subsequent rows.  Then I create a new LedesFile object.  I use a block to iterate through the values in values (creative n'est-ce pas?).  With each line, I merge the values from @file_values, because they need to be in each line.  I'm using the merge!() method, which is called a bang method.  It forces an overwrite.  This means any values in @file_values overwrite values in line.

Then we create a temp_line LedesLineItem object to hold our line item, and write it to the file.

So let's dig a little deeper and see how LedesLineItem and LedesFile work.  First, let us take a look at features/support/ledes_line_item.rb:

features/support/ledes_line_item.rb


class LedesLineItem

  attr_accessor :line

  def initialize(values)
    units = 1
    @line = Hash.new( "" )
    @line = {
        'INVOICE_DATE' => Date.today.strftime("%Y%m%d"),
        'BILLING_START_DATE' => Date.today.strftime("%Y%m%d"),
        'BILLING_END_DATE' => Date.today.strftime("%Y%m%d"),
        'INVOICE_DESCRIPTION' => "Automated Test Invoice",
        'LINE_ITEM_NUMBER_OF_UNITS' => units,
        'LINE_ITEM_DATE' => Date.today.strftime("%Y%m%d"),
        'LINE_ITEM_DESCRIPTION' => "Automated Test Line Item",
    }

    line.merge!(values)

  end

end

The member line is a hash that can be accessed externally.  I am simply populating it with some default values, and then anything passed through from our step is overwritten with merge!.  So for example if I want to change the INVOICE_DATE, I could just pass that in from my Cucumber step.

And here's the LedesFile object that puts the lines together:

features/support/ledes_file.rb


require_relative "ledes_line_item.rb"

class LedesFile

  attr_accessor :filename

  def initialize(filename)
    @filename = filename
    File.open(@filename, 'w') do |ledes_file|
      # File header
      ledes_file.puts "LEDES1998B[]"
      # Column headers.  I'm told these headers do not change, so I'm hard-coding them here.  I'm also told that the
      # order doesn't matter.  So I put them in and then read them out to make sure the lines get put in the right
      # order.  This means I don't care how the hash is created or passed in.
      ledes_file.puts "INVOICE_DATE|INVOICE_NUMBER|CLIENT_ID|LAW_FIRM_MATTER_ID|INVOICE_TOTAL|BILLING_START_DATE|BILLING_END_DATE|INVOICE_DESCRIPTION|LINE_ITEM_NUMBER|EXP/FEE/INV_ADJ_TYPE|LINE_ITEM_NUMBER_OF_UNITS|LINE_ITEM_ADJUSTMENT_AMOUNT|LINE_ITEM_TOTAL|LINE_ITEM_DATE|LINE_ITEM_TASK_CODE|LINE_ITEM_EXPENSE_CODE|LINE_ITEM_ACTIVITY_CODE|TIMEKEEPER_ID|LINE_ITEM_DESCRIPTION|LAW_FIRM_ID|LINE_ITEM_UNIT_COST|TIMEKEEPER_NAME|TIMEKEEPER_CLASSIFICATION|CLIENT_MATTER_ID[]"
    end
  end

  def write(line)
    File.open(@filename, 'a+') do |ledes_file|
      file = ledes_file.readlines # Read the headers in so we can place values in the appropriate order.
      headers = file[1].split("|")
      headers.each_with_index { |key, i| headers[i] = key.strip }
      headers.each do |key|
        ledes_file.print line.line[key]
        if key == 'CLIENT_MATTER_ID[]'
          ledes_file.print "[]\n"
        else
          ledes_file.print "|"
        end
      end
    end
  end

end

When a LedesFile object is created, a file is actually created on the system.  It is populated with a file headers and then column headers.  When a line is written, I open the file back up for writing, read out the headers, and then use that to make sure that each value from the passed in line goes into the correct column.  Then each column is terminated with a pipe, and if we are putting the CLIENT_MATTER_ID, I know we are at the end of the line and I drop brackets instead.

I never explicitly close the file, as that is handled when the block exits.  If I were worried about performance, I would make sure I could handle an array of lines so I only have to open the file for writing once.  Also, note the require_relative line.  This makes sure that when the LedesLineItem object is reference, we can find it in our current directory.

CSV and Other Delimiters

If I wanted to use a different delimiter, I would edit my headers in line 15 so that it appeared there.  Then in line 22, I would split on that character instead (for tabs use \t).  Line 26 searches for my last header, so you would need to change it to look for a different value, and in line 27 if you just need a new line, remove the brackets.  Finally in line 29, you would change the pipe to your delimiter.  You could of course put the delimiter in a variable, but I leave that as an exercise for the reader.

Conclusion

In the end, there were a number of things I needed to learn, such as how to write to files, and the difference between single and double quote strings in Ruby.  Once I got those ironed out, this was a very easy process.  I write this for anyone coming after me who finds it more of a struggle.

Wednesday, April 11, 2012

Stupid <acronym> tags, and how to find them (or any other deprecated tag).

I ran into a problem where the site I am testing uses the <acronym> tag.  The problem arose because watir-webdriver derives its HTML element objects from the HTML5 spec.  Guess what isn't in that spec?  The <acronym> tag.  I went through a long process to find a fast, reliable solution.  This was the HTML I was trying to deal with:

<acronym class="editable select fillwith:exp_codes default:E100"
title="Expense Code: Expenses" id="expense_code114_582_10777">
E100    </acronym>

This is the code I ended up with:

@browser.element(:tag_name => "acronym", 
                    :class => "default:#{expense_code}").when_present.text

I'm using the generic element tag, passing it the tag name of "acronym", and then a class identifier.  I had to do both the name and class, because neither on its own was sufficient.  I also found out that when passing a class, the values are space-delimited, and that watir-webdriver understands this.  The upshot was that I only needed to include the default:E100 bit.  expense_code is a variable being passed in which represents the E100 portion.  I added .when_present becauseI am running this right after I save changes, and Javascript is updating the page.  Without it, my test consistently fails because it is losing a race condition every time.

My first solution used a regex, and it killed my performance.  So, avoid those whenever possible, but don't be afraid to use them to get your watir-webdriver code working.  I also learned that while xpath and css selectors are options, ultimately there is usually a better (read: cleaner) way - you just have to find it.

If you're interested in my journey through using a regex, trying XPath, and settling on my final solution, you can find it documented on StackOverflow: http://stackoverflow.com/questions/10093114/how-can-i-find-an-acronym-tag-with-watir-webdriver-without-taking-a-huge-perfo

Thursday, April 5, 2012

SyntaxHighlight Using the New Blogger Redesigned Editor

I really wanted syntax highlighting for this blog. Almost as much as I want my own Dwarven Mugblade. (I just thought of that as I walked back from the kitchen with my coffee mug.) Swordchucks yo!

Anyway, I finally found SyntaxHighlighter.  It's a tool that works with Blogger.  Yay!  I found about 5 tutorials on using it, but none of them told me where to put the code with this new improved version of Blogger.  (Which I really like.)  So here's some instructions.

1.) Go to your Blogger blog and click Design in the upper right-hand corner of your blog.
2.) On the left navigation menu, click Template.
3.) Under the snapshot of your blog, click the Edit HTML button.
4.) When the warning pops up, click Proceed.
5.) Directly after the <head> tag, paste the following code:

<!-- CSS Styles -->
<link href='http://alexgorbatchev.com/pub/sh/current/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href='http://alexgorbatchev.com/pub/sh/current/styles/shCoreDefault.css' rel='stylesheet' type='text/css'/>
<!-- Choose a new theme if you want from http://alexgorbatchev.com/SyntaxHighlighter/manual/themes/ -->
<link href='http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css' rel='stylesheet' type='text/css'/>

<!-- JS Scripts -->
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shAutoloader.js' type='text/javascript'/>

<!-- Brushes -->
<!-- Get additional brushes by selecting them at http://alexgorbatchev.com/SyntaxHighlighter/manual/brushes/ -->
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushBash.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCpp.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCSharp.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCss.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJava.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJScript.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPerl.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPlain.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPhp.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPython.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushRuby.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushSql.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushXml.js' type='text/javascript'/>

<!-- Call SyntaxHighlighter and include the switch for using on Blogger -->
<script language='javascript' type='text/javascript'>
    SyntaxHighlighter.config.bloggerMode = true;
    SyntaxHighlighter.all();
</script>


6.) If you wish to add languages, you can just copy and paste a new line and replace the *.js file with one from the list here: http://alexgorbatchev.com/SyntaxHighlighter/manual/brushes/
7.) If you wish to change the color scheme by using a different scheme,you can test them out and then replace the call to the default scheme, choosing from the ones on this page:
http://alexgorbatchev.com/SyntaxHighlighter/manual/themes/. Note that this line isn't necessary, I just added it so changing the theme was easy.
8.) Click the Save button, then the Close button.
9.) When editing a post, select the Compose button and paste your code. I usually type some placeholder text after the code, so that new things I type later don't get accidentally sucked into the code block. Ex:
  def method_missing(sym, *args, &block)
    @browser.send sym, *args, &block
  end
10.) Click the HTML button and add these <pre> tags around your code:

<pre class="brush: ruby">
&nbsp; def method_missing(sym, *args, &amp;block)<br />
&nbsp;&nbsp;&nbsp; @browser.send sym, *args, &amp;block<br />
&nbsp; end<br />
</pre>

When you preview, you will end up with something like this:
  def method_missing(sym, *args, &block)

    @browser.send sym, *args, &block

  end
A few caveats.

1.) When previewing, you cannot scroll your code to the right.  It will work when you view the page.
2.) If you can't get a brush name to work, look at this page for brush aliases. http://alexgorbatchev.com/SyntaxHighlighter/manual/brushes/
3.) If you improperly close your opening <pre> tag by forgetting a second double quote, when you go back to Compose, everything after it will be deleted.  I suggest saving before you attempt to add code blocks.

The perfidy of AJAX and surprise attacks using when_present

If you are testing an asynchronous website which uses Javascript to create widgets, hide and show elements, etc, watir-webdriver can have trouble seeing them.  Watir does great waiting for pages to load without telling it specifically to do so.  However, when your AJAX calls change a website's Javascript, Watir doesn't know it needs to wait.

For example, my company has something called the Dashboard.  The Dashboard is a little tab that you can click and shoot out of the left-hand side of the page.  It's nifty, but my scripts were occasionally failing due to the AJAX not loading the page quickly enough.  An annoying, intermittent problem that took me a while to figure out and fix.

The solution is a little method called when_present.  I have a little method called Dashboard::switch_client.  It ensures I'm on the correct client site after I log in based on an Environment variable.

  def switch_client
    self.dashboard_button.click
    self.company_list.select @client
    self.dashboard_button.click
  end

My problem was in using the select box.  Sometimes it wasn't yet available.  Here's my solution:

  def switch_client
    self.dashboard_button.click
    self.company_list.when_present.select @client
    self.dashboard_button.click
  end

I now do this for all my menus and sub menus, calendar widgets, and any other widgets I have on the page.  when_present gives an element 30 seconds to appear by default before failing the test.  Over laggy connections, that's very helpful.

You can find more information and a couple similar methods for waiting for elements, including one that waits for an element to disappear in the RDoc: http://jarib.github.com/watir-webdriver/doc/Watir/Element.html#when_present-instance_method

Tuesday, April 3, 2012

Using Ruby to create linebreaks in Windows

I am creating a file generator for taking data in a cucumber test and dumping into into a proprietary pipe "|" delimited file format for upload into our system.  I plan on posting about that process later, but I wanted to highlight something I found a bit frustrating to understand.  This has nothing really to do with Cucumber or Watir.  It's a very basic Ruby thing, but I spent a while looking around for an answer.

I was having trouble getting newlines to work.  Most of my trouble ended up being with white space not being trimmed from a value I was comparing.  But after I resolved that issue (rather inelegantly, but that's another discussion), I still couldn't get it to work.  I was passing '\n', '\r\n', '\n\r', and finally I tried something.  I passed "\n".  It worked like a charm.  In the back of my mind was this nagging feeling.  Don't single and double quote characters work differently?  Yes they do.  Single quote strings output everything literally.  If you want to use escape characters such as \n, you need to use double quotes.

You can find more information here:  http://www.rubyist.net/~slagell/ruby/strings.html

It's important to know, this just works on Windows.  You don't need to pass a carriage return \r as well.  That will be added for you  Now, if you're outputting files that need to be used on a Linux box or Mac, you'll need to do something a little different.  You need to open the file for writing as Binary.  There's a good discussion of this on Stack Overflow: http://stackoverflow.com/questions/8064175/writing-unix-line-breaks-on-windows-with-jruby

File.open(filename, 'wb') do |file|
  file.print "My text\n"
end