Admin Script

1.       Welcome and Introduction

a.       Welcome participant to usability study.

b.      Introduce yourself and observer.

Hello, I’m Pallavi (Grace) and I’ll be walking you through today’s session. Let me also introduce my colleague. She will be taking some notes.

You’ll notice that I’ll be reading from this paper most of the time. We do this to ensure that we give the same information to all the people who are participating in this study.

We are working on improving ease of use of an open source personal finance management software. By taking part in this usability study, you will be helping us to improve this product.

c.       Tour the room, show the control console, cameras, offer water..

This is a usability lab facility here at University. This room (main room) is for discussion. This room (control room) is for audio and video control as we record the studies. And this room (study room) is the study room, as participants sit here and do the task, this camera records the activity from here and the screen interactions are also recorded and fed to the control unit.

d.      Have participant sit at their station. Join them at the table and

e.      Explain format of the study, following is the outline:

Thank you again for coming. Let me explain a little about what we are going to do here today, so you know what to expect. This is a usability study, the purpose of which is to evaluate the “ease of use” of a product.  We have asked you to come here to observe how you interact with and use the product. Since we are evaluating a product I want to assure you that we are in no way evaluating or testing you. Every action you take or question you ask is THE RIGHT ONE and merely highlights a feature of the product that is not clear to a potential customer such as yourself.

I will begin by having you fill out a brief questionnaire. Then I will explain how the study will be conducted using something called a Think-aloud protocol. After the study, I will have some follow-up questions for you and then another brief questionnaire. Then we will be done. Do you have any questions so far?

2.       Pre-study Questionnaire

a.       Give participant Pre-study Questionnaire and a pen. Pre-fill out participant number on the questionnaire.

b.      When finished, take questionnaire and pen from participant.

3.       Demonstrate Think-Aloud protocol

a.       Explain Think-Aloud protocol

As I mentioned, this study will be done with something called a think-aloud protocol. It simply means that I will give you a task to do and you will say out loud everything that you are thinking as you go. This helps me to understand your experience as you use the product to do some tasks. Note that this is not the same as offering how the product might be improved. These comments are important but we will address them after all the tasks have been completed.

Let me give you an example of thinking aloud while trying to accomplish the task of changing the time on my watch…

b.      Give example of think aloud protocol using setting the time on your watch.

c.       Have participant practice using think aloud protocol to load staples into a stapler.

d.      If first attempt is not sufficient, explain how to improve then repeat c.

4.       Introduce tasks

This study consists of you performing several tasks with a personal finance management software. I will give you a packet with one task per page. Please read through the entire page and then attempt the task that is asked. Do not move onto the next page until you feel you have fully completed the task. When you feel you have completed the task, say “I’m done” and I will instruct you to turn the page. Remember to say aloud anything that you are thinking as you do the task. If you feel stuck or have questions, try to work them through – I will not be available to answer questions. Also remember, there are no right or wrong answers here – our goal is to understand how this product is used by you. Do you have any questions before we begin?

5.       Give Tasks

a.       Give participant the packet of tasks and the box of supplies.

6.       Post-Study Questionnaire

a.       If there are any actions to get further detail (look for follow-up questions in the Data collection form), ask the participant to explain what he or she was thinking or trying to do at this time. Try: “I noticed you did x during task y, can you explain in more detail your thought process at that time?”

b.      Write responses on the data collection form next to the question.

7.       Post-Study questionnaire

a.       Give participant Post-Study Questionnaire and a pen. Pre-fill out participant number on the questionnaire.

b.      When finished, take questionnaire and pen from participant.

8.       Conclusion

a.       Thank participant for coming.

b.      Give participant gratuity.

Test tasks planning matrix

List of the issues looked at, the concrete questions to be tested, necessary task, setup material, possible evidence to indicate test failure or success and finally how that evidence will be collected are listed in detail in the table below:

Category, Issue	Concrete question	Material to be set up, Task	Evidence	How to collect data?
Terminology C,R,NR labels are  used without explanation as to what they stand for Concept of Reconciliation may not be understood as it is not a very common task in current online banking scene. Basic terminology issue of not understanding concept of ‘clear’ state	Can the user understand the difference between clear, reconcile and non-reconcile states?	·   Pre-entries of different type C,R,NR ·   Marketing material of the product stating the principles of double entry method specific to KMyMoney. ·   Wrong entry that needs to be corrected 1.     “You deposited a cheque yesterday and it got cleared today, please change the entry appropriately.” 2.     “You got your bank statement of your account abc please confirm/correct the entries for that account.”	·   For part 1: Change NR to C Be able to edit an entry ·   Part 2: Reconcile entire account Rectifying the wrong entry Edit entry	Think aloud, Screen video, Note points where the user gets stuck
Task flow After creating a scheduled transaction, the way to enter it is to go to home page, or by right clicking it, which may not be intuitive. Also, the right click option calls current month transaction as next transaction.	Can users create a scheduled transaction and enter that transaction for the current bill?	“You have paid your monthly apartment rent today by cheque, now please create a reminder to enter it every month, and enter this month’s rent payment and verify it.”	·   Create a schedule transaction. ·   Find it on home page or by right clicking. ·   Find entry in the ledger	Think aloud, Screen video, Note points where the user gets stuck
Task flow Adding a new category needs user to get out of context of Budget. Secondary: Navigation Option to change the year of the Budget is hidden under right click. Budgets icon is below the fold. Calculator and new budget icon are same and close to each other Green check mark to enter the budget is not logically placed enter layout.	Can the user create, view and make changes to the budget?	·   Set up budget categories ·   Prepare required data for 2010 budget, one of the category in the budget should not be in the categories that are initially set up. “You planned your next year’s budget as given, please enter that budget plan into your system.”	·   Entering 2010 year could be difficult as the options to edit year information is hidden ·   Choosing budget categories can be confusing to do inconsistent terminology, here they are called “account” ·   ‘Clear field’ button may be confusing ·   The switching between months, year and individual could be confusing.	Think aloud, Screen video, Note points where the user gets stuck
Navigation Budget visualization to view allocated values to all categories can be done only through reports and is not in context of Budget. Option to view in pie chart is not well exposed and hidden under right click. Pie chart report is hidden under very deep navigation.	Can users view the budget plan as a pie chart?	·   Enter 2009 budget ·   Make sure the report view option is not pie chart already “You want to view the budget you planned in pie chart format”	·   Be able to identify ‘report’ as ‘budget report’ ·   Figuring out use of right click for more options	Think aloud, Screen video, Note points where the user gets stuck

Patterns and Obstacles to Financial Management

Elizabeth P. Davis and Judith A. Weber, Patterns and Obstacles to Financial Management from Financial Counseling and Planning, Volume 1, 1990

This paper reports a questionnaire based study that reports on use of four recommended financial management practices and possible obstacles in following them. The user group studied was non-metropolitan households’ living in Kansas.  The four practices studied were: budgeting, keeping records of expenditures, comparing records to budget and estimating net worth. Respondents who reported having used a procedure were further surveyed to find level of implementation and those who responded to be not following a procedure were asked the reason for not doing so, both as multiple choice questions with facility to explain other factors. 

It was found that record keeping was the most practiced principle and second (80%) was budgeting. But follow up question revealed most (54%) people did budgeting both mentally and on paper. The reason for not being able to practice a budget was lack of a steady income.  Budgeting was planned for time slots of a month, several months and a year. About two third people compared their spending to their planned budget, and they did it on a monthly basis. Only 36% followed all the four practices. Some of the obstacles in following these practices were:  they take time, effort, sometimes cooperation from other family members.

The study is quite old, data were collected in 1984. This may make most results not be very relevant to this usability study.  Also the study didn’t particularly focus on use of any tools. Budgeting was discussed to be mostly “written” and expenditure tracked through saved receipts and checks but did not explain more on the specifics of how that information was handled or managed, which are limitations of a questionnaire kind of study. 

However, the paper still helps us identify the critical practices of personal finance management: record keeping, budgeting, comparing records to planned budget and finally estimating net financial worth. All these are essential basic features of personal finance software even today.  Other critical factors the paper brings to focus are:

·         How different kinds of income and expenditure patterns effect financial planning? This could be huge area to study in itself. As for this study it would mean to use same financial data as test material and try and make sure the actual income profiles of the study participant to be similar (steady or flexible).

·         Finance management may not always be a single person task but rather collaborative as the paper mentions the failure to keep up with record keeping was due to lack of cooperation from other family members. This is something even the current latest personal finance software should try to help.

Audience and issues

KMyMoney – Audience and Issues Statement

Product

KMyMoney is a personal finance manager for KDE (a desktop environment mainly for Unix-like systems). It enables users of open source operating systems to keep careful track of their personal finances by providing a broad array of financial features and tools.  Key features are:

1.       It uses double entry accounting principles to ensure accounts are kept in correct order

2.       Strives to be easy to use

3.       Provides all important features commercial systems provide.

Audience

KMyMoney strives to be the easiest open source personal finance manager to use, especially for the non-technical user. The audiences for this usability test are novices both in terms of domain knowledge (personal finance management) and in tool skills (Linux environment). They will satisfy the following requirements:

·         Should have used Linux:

o   Used it in past,

o   Could at least browse, and navigate through file structures in Linux.

o   Using it currently but currently use Windows or MAC more often.

·         But are primarily Windows or MAC users

·         We are looking at novices interested in using of personal finance management software. So, we are looking at user who are interested but have not extensively used any personal finance management software like MS Money, iMint etc. So the user should:

o   Should have an active bank account

o   Should be familiar with online banking

o   Check online statements of their bank account, pay bills through online banking and log in to it at least twice a month.

o   May have tried but not used personal finance management software for more than a month or recently (in last 2 months)

o   Attempted to create a budget for their finances in any form

o   Attempted to track income and expenses in any form

Issues

For this usability study we are focusing on 2 of the main features of the product: Transactions and Budget. 

The main issues we would like to focus in this study are Terminology, Task flow and Navigation.

Terminology:  We would like to test if the software gives sufficient assistance (with the tool tips) or is intuitive to a novice user in dealing with various terminology and icons that are used like: ‘Transfers’, ‘Reconciliation’, ‘Ledger’, ‘Clear’, ‘Records’.

·         KMyMoney uses double entry accounting principles. Some of the terminology like Reconcile and Non-Reconcile that comes with it are not common terminology in current online banking scenarios and could be confusing to the user.

·         In the software the states Clear, Non-Reconciled and Reconciled are labeled as C, NR and R without any assistance.

Task flows: We would like to test if the task flows are easy for the user to follow or not. Particularly the tasks like entering a scheduled transactions and creating a budget seems to be confusing for the following reasons:

Scheduled transactions:

·         After creating a scheduled transaction, the way to enter it is to go to home page, or by right clicking it, which may not be intuitive. Also, the right click option calls current month transaction as next transaction.

Budget:

·         Creating a budget brings in the categories for the budget as already chosen during the initial KMyMoney setup that could be confusing as the user might want to change this depending on his current budget plan.

·         Any changes to the categories at this point require the user to jump out of the context of budget creation and set up categories separately.

·         Also in this case the categories are wrongly referred as accounts.

Navigation:  There are a number of problems that affect smooth navigation in KMyMoney application.

·         Some of the elements might be bit hard to find. Some of the options are hidden under right click. For example, the option to change year for creating budget for a different year other than the current, is hidden under right click.

·         Also there are issues across the UI with tab and button interaction as the user tries to navigate through different types of transactions or budget time periods.

o   Switching between tabs doesn’t give sufficient feedback to indicate the exact function it supports. For example, to creating a new transaction if an older transaction is already selected and user clicks on the tab to choose a different transaction type and then tries to click the ‘New’ button to create the new transaction, the software does not support this task flow. The software upon clicking the tab already enters the “new” transaction creation mode and disables the new button. This happens while the user is not explicitly made aware of the already created new entry that is ready to be added with more details. This kind of interaction where the state is entered and ‘new’ button is disabled also happens in budget creation task flow.

o   In Budget creation for monthly basis, the first month’s entry is also the ‘monthly value’. This could be confusing as January’s value field is used for 2 different purposes which may not always be holding same value.

o   Viewing the created budget is split away from the context of budget creation. It is inside ‘reports’ a label which may not clearly spell out to the users ‘budget view’ hence breaking the navigation. 

o   Budgets icon is below the fold.

o   Calculator and new budget icon are same and close to each other

o   Green check mark to enter the budget is not logically placed enter layout.