Ask any UX designer or Human Factors researcher about their most memorable moments planning or running usability tests, and you’ll hear some real eye-openers. From challenging client relations, to logistical nightmares, to balky participants, planning and executing a successful usability study requires attention to detail, a deft personal touch, and a fair bit of improvisation in the face of the unexpected.
As a mid-career UX professional who has worked in the financial, healthcare, and medical device fields, I’m constantly thinking about ways to improve not only the quality and delivery of data from studies, but the overall process of planning and executing studies. While there is no single “right” way, there are certainly practices that can better the chances of a successful study.
Here are some useful pre- and post-study activities gathered from experience in the field. Some suggestions might be old hat to seasoned professionals, but if even one of them helps someone avoid stress, lost data or squandered time, sharing them will have been worth it.
Analyze thyself
It can be difficult for ever-busy UXers to step back and make time for the work necessary to interrogate how studies are planned, executed and reported. Yet, planning a short retrospective at the end of each project can ferret out potential areas for improvement that apply not only to the individual study, but to your UX practice as a whole. The devil is in the details, of course; capturing those findings and translating them into action is the real challenge. Subject your own processes to the analytic eye you would turn to a client’s product. Where can they be streamlined or improved?
Do you feel lucky? Well, do you?
Beyond the obvious calamities like no-show participants, minimizing potential problems in study execution comes from experience and the ability to recognize where problems are most likely to occur. If [BAD THING] happened, this session would be totally hosed. Particularly if you are a freelancer or working solo (hence no backup team), assessing the likelihood and impact of those BAD THINGS drives the planning that reduces the risk they will occur and reduces the severity when they do.
Repeat: Redundancy is good for you
Avoid depending solely on digital copies of critical documents. Make sure those digital copies are already downloaded to your laptop because WiFi is ubiquitous until it isn’t. Don’t assume you’ll have time to find a printer at a new study location. Bring paper copies – in particular, several copies of the protocol and discussion guide – organized and ready for quick retrieval.
Similarly, if you are testing material products and are responsible for transporting them to the site, have a teammate bring extra in case of lost or stolen luggage, or a fatigue-induced failure to pack. Consider whether mailing them in advance, with tracking, is an option.
Setting it up
Market research facilities
Know the site coordinator.
Make it crystal clear what front desk staff are responsible for, and assume there will be several shift changes over the day, and that your requirements may not survive the inevitable game of telephone.
For anything more complex than registering and compensating participants, provide the front desk with a clear step-by-step description of who gets what and when. This is particularly helpful if you have multiple Informed Consent Forms for different participant groups, or situations where some participants might return for follow-up sessions while others won’t.
In the field
Know your field site. If at all possible, take a walkthrough before your session to familiarize yourself with where things are, and any potential distractions or complications (discovering you’re on a bus route or in a WiFi deadzone, etc.)
If you’re going to be outside, have a fallback plan for inclement weather or an overly noisy environment.
Overall
Ensure all devices are fully charged before study start. Always have a backup battery or SD card for any devices you are using.
Check your recording devices. If you’re at a market research facility, check whether some camera adjustment is necessary for the best possible view of the participant.
If you are the one with the recording devices or software, ensure you and your team (if you have one) know how they work. Nothing is more frustrating than discovering after the fact that your session audio wasn’t recorded because someone forgot to press a button. Use sticky note prompts if necessary (you laugh, but it works.)
If you’re a solo practitioner running a study, a Livescribe pen that records audio is a great addition to your note-taking arsenal, because it’s much easier to find a particular sound bite in your notes.
Mise en place. A French culinary term that means “put in place,” it’s also an extremely useful tactic to minimize scrambling during study execution. Establish specific locations for critical assets and communicate where they are found. At a facility, lay materials out on a table. If you’re on the go, organize your bag so you know where everything is without fumbling around.
For printed or physical assets, sticky notes make great place labels so your team isn’t forced to rely on memory over a long day. Protocols are here, study guides are there, moderator checklists are over there.
Separate blank materials from anything participants have filled out.
Breaking it down
Tips for when the study is over:
If any assets are to be shipped back to clients or the office, have necessary contact and address information ready.
If you have more than a few items in a box, create a shipping manifesto, and crosscheck before sealing.
Double-check that any forms or physical media have been collected from site staff, and that any digital media has been transferred to portable form, where applicable.
While some of these activities are more relevant to studies involving physical products, most are broadly applicable to usability studies in general. Whether standardized in checklists or simply incorporated into routine practice, these simple organizational activities can help ensure a resilient and successful study.
With any scenario there will be unforeseen changes that will impact a project. A key to success in project management lies in planning and mitigation of risks. In international projects, these are five techniques in order to ensure a successful project.
Understand the Cultural Norms of the Countries
Before you begin a new project, it is important to understand the cultural norms of the region and city. Across global markets, there are certain idiosyncrasies that are critical to project success. A moderator from Spain may likely have a different vocabulary from one in Mexico. If you ask someone in Australia where the nearest parade is, they would direct you to a street, not necessarily a social event. I recommend spending a few moments before kickoff getting acquainted to the political climate, socio-economic considerations, geographic concerns, and holidays.
Strengthen Your Partnerships in that Region
The best asset to ensure a successful project in a specific region is your partnerships. Having more than one trusted partner in each region will help provide the insight into specific cultural considerations, neighborhoods, pricing, and best practices. For example, for a study in Mexico a partner would provide insight into concerns about mobile data usage, and purchase of SIM cards in that area. Excellent communication, collaboration, and feedback are the key to a successful and strong partnership, both internationally and locally.
Prepare Ahead for Technical Challenges
Each region has its own technological considerations, which may affect the success of a project. Certain countries have firewalls to prevent transmission of data or specific websites. Others may have a focus on mobile phone usage, and you may find it difficult to find a user with a PC. In this process, it’s important to review any potential roadblocks in technology, and address those in advance, or set appropriate expectations of the tech availability. Knowing ahead of time what things you have to adjust for, whether it be internet connectivity, use of specific tools, or local devices, are critical in order to have an international study go off without a hitch.
Set Realistic Expectations with Stakeholders
I’d say this is one of the more challenging aspects of international projects. It’s both exciting and challenging to work with teams across the world. In our technical environment, the remote stakeholders love the idea of being able to be in the moment 10,000 miles away. As we strive to deliver this level of service, it can be a challenge in some areas where the world of high-speed Wi-Fi is not a reality. One key component of an international study is being transparent with the stakeholders about what is realistic in different markets.
Focus on Successful Communication
If you have seen one theme around all of the tips above, they all involve successful communication. Communicating clearly and professionally with your colleagues, partners and stakeholders is essential. In international projects, there are so many moving pieces. Allowing information to slip, or failing to provide an important detail can lead to disaster if not addressed immediately. As an agilest, I personally like to check in with my team on a daily basis to ensure that any updates are communicated clearly to all parties.
Looking for a cutting-edge, quick and easy way to get your designs from concept to interactive prototype? Working in an agile design environment and need insight into the usability of your concept prior to wireframing? If you answered yes to either of these questions, the POP (Prototyping on Paper) is a design methodology and app you’ll want to check out.
Paper prototypes are not a new or novel concept to the IxD Designer; however, adding interactivity to these paper prototypes is rather new. Enter an amazing app POP. I discovered POP when I was designing a niche market yellow pages app for iOS with a fellow designer at a design bootcamp. We needed a quick, down and dirty way, to test the interactions of our interface. We wanted to validate our design decisions quickly prior to designing the wireframes. A fellow student introduced us to POP.
Using POP is simple. You download the app on your iOS, Android, or Windows mobile phone and start snapping photos of your drawing comps. You can then either work on a desktop/laptop or directly on your mobile phone adding ‘hotspots’ by drawing and dragging active areas and defining what action those hotspots should execute. Link pages together by assigning actions to these hotspots and build a functioning paper prototype of your app in minutes.
Take your concept one step further by using the built-in sharing function of POP. We decided to do some guerilla testing of our initial navigation structure and used the sharing function of POP to share our prototype with friends. We also used Key Lime’s mobile testing tool KLUE to record the interactions and video of the user while interacting with our product. These initial findings helped us tweak our design prior to building and testing wireframes.
Another great feature of POP is the ability to sync with Dropbox and integrate easily into your workflow. Drop photos of your pencil sketches into Dropbox and they immediately appear in POP’s app. Have photos from others working on the same app, share your Dropbox folder and have everyone drop their photos into Dropbox for immediate access to screens from all your team members.
POP is a great app for anyone who needs to quickly gain insight into their designs- heck they even have free downloadable sketch templates for all kinds of mobile devices! Have questions about prototyping on paper, or UX & IxD design in general? Reach out to me- troy@keylimeinteractive.com (UX Research Manager at Key Lime Interactive).
by Rick Damaso
Let’s face it. We have become a “Google it” society. We can answer almost any question in .00012 seconds (according to Google’s search results page) and get a pretty accurate response. So, before attending the LeanUX15 conference in Brooklyn, NY, I wondered what would Google’s response be for UX’s impact on modern day Systems Development Life Cycles (SDLCs)? Was it even possible to wrap terms like DevOps, Kanban, Lean or Agile into a neat package with a UX bow on top?
More importantly though, does this matter? Or were these terms just the buzzwords from the “The Valley” that larger companies all around were simply talking about emulating? Do new philosophies really only announce themselves when blue chip companies start adapting them?
To say it depends is a boring answer. But, of course, it depends. As a researcher, however, I wanted a straight and narrow answer.
I thought it would be prudent to first set the stage. Here are some quick snippets of what a quick Google Search will give you relating to these new SDLCs and UX. Agile- Developers focus on sustainable development. Sustainability is about good estimation, effective branching strategies for managing code, automated testing to protect quality, and continuous deployment to get fast feedback from users. DevOps- DevOps is a software development method that emphasizes communication, collaboration (information sharing and web service usage), integration, automation, and measurement of cooperation between software developers and other IT professionals. Kanban- Technique for managing a software development process in a highly efficient way. Producing software is a creative activity and therefore different to mass-production (Kanbans’s roots are in auto manufacturing) allowing us to apply the underlying mechanism for managing “production lines”. Lean UX- Lean UX is a set of principles that may be used to guide you to better, more desirable solutions for users. It’s not a process in which each tool is rigidly applied. Instead a group of ideals and principles to guide you in the design process.
So, that makes sense. If I was a designer sitting in a room with other designers, communication and putting this philosophy in practice shouldn’t be that difficult in theory. But, when you take these practices out of Silicon Valley and introduce it to the landscape of companies like Microsoft or Spotify with teams of designers on separate continents, it can make your head spin. How could you package what looks and feels like a startup mentality and scale it up effectively?
LeanUX15 took this challenge head on during a four day event in Brooklyn, NY. This conference was not all that different from any other, unless you consider flamingo colored windbreakers and Paul Bunion beards different. But, all “hipster” jokes aside, the usual laid-back vibes you find in the heart of Brooklyn’s Williamsburg neighborhood was noticeably different. Product managers, Lead Designers, Software Engineers and representatives from some of the world’s most iconic brands were buzzing with excitement on bringing these results driven practices to companies everywhere.
As UX researchers, we encounter organizations along all stages of the UX maturity cycle and work on projects from formative to summative stages, so I was pleased to hear that UX research is taking a prominent role in these SDLCs. As opposed to traditional validation testing, we were now seeing rapid production of software married with UX researchers, architects and designers alike.
So what does this force us to do? To borrow from one of the themes of the conference, it means we are now in the age of Designing for Service, Not Just Software.
Here are some quick hits as to how UX is impacting development and strategic vision:
Visualize your work, in knowledge clusters. Ideas are then disseminated to users as solutions in terms of their problems.
With a UX lens, setting realistic Work In Progress limits for each stage of production is critical. Accounting for time slots within stages for user testing as opposed to piecing it together at the end.
Manage flow to clearly identify bottlenecks and accurate metrics. When infusing UX research into your design process, you are hedging against expensive “revamps” at the tail end of your SDLC.
Make Policies Explicit. Stick to your design, research and implementation policies! However, the #1 policy should always be, “If one of your policies does not work, change it”. By first following your process and analyzing what is wrong, you will be in a much better position to fix it.
Implement Feedback Loops. Communicating accurate measurement with your target market is key. Measurements need to be relevant to the timing of your project, not “at the end of each quarter” or when “you have time for it”.
Empowering yourself and your team to think- you are allowed to think and change processes. These SDLCs are not recipes, instead they are thought of as disciplines. Every question you ask yourself must be phrased as, “is this a driving force to consider design for servicing users or just designing software?”
The message for us as researchers when entering a new frontier of rapid development and testing can be wrapped up with a quote by Prof. Barbara Adam: “The message for research is unambiguously clear: learning is a process with a history and a future; it is thus not containable within observable moments. It entails a joining of life-worlds, a drawing on collective and individual past knowledge as well as projected vision, all of which are brought to bear on the interactive present.”
One of the big themes of nearly every SXSW event we attended was personalization. Even events about the Future of TV had panelists talking about supplemental apps or making sure people could watch on the devices they chose. A news discussion with Dan Rather and Dan Pfeiffer also discussed how people consume news on the platforms of their choice, like Facebook and Twitter. Customers are looking for a more personal, customized experience in the place of their choice.
Predictive technology is making big strides in making these more curated experiences accurate. Facebook’s facial recognition technology is making use of their extensive data on user tagging so they can auto-tag your photographs when you post them. This technology may be more accurate than that of law enforcement. Netflix’s House of Cards was famously made by using data to understand that a political drama starring Kevin Spacey directed by David Fincher would be popular. An important consideration with using predictive data for customer recommendations is providing said data to customers.
Personalization is changing the landscape all over. I went to a talk by Karlie Kloss and Sara Wilson about technology and its role in Fashion Week. Models and editors can now deliver a more personal experience to a massive audience using Instagram and Twitter. They can let people into their lives remotely and enable fan interaction. Vogue recently had a cover featuring nine models with large Instagram followings.
Companies are integrating data into their operations in a variety of ways. Capital One is experimenting with personalized financial recommendations AND personalized offers / rewards recommendations in new apps Ideas and Level Money. Ideas provides recommendations for different types of activities in 4 beta markets: NYC, LA, Richmond, and DC. Level Money lets customers link accounts and program in budgets and receive alerts and content depending on their spending.
For television, companies are looking to make the experience more intimate for viewers. Some companies are experimenting with companion apps, especially in the UK. These might let viewers answer quizzes or play related games while watching. Other companies are trying to make promotions / advertisements more personal. For Game of Thrones Season 5, HBO ran a promotion called The Sight in which people would get text messages with video links that would disappear. The videos would be different for different users but communicate small snippets of information about the upcoming season in the guise of visions / dreams. In Spain, Canal + ran a promotion called 19 Reinos that turned all of Spain into an interactive Game of Thrones-themed game played via multiple different channels: Twitter, Facebook, brand websites, and physical stores. Customers all over the world are looking for targeted, personal experiences. User experience research is one way companies try to identify what kinds of experiences are most valuable to customers. Airbnb mentioned that when they redesigned their website, they made sure to keep the hosts involved in the process since their feedback was critical to its success. There was a fantastic talk by Etsy about how user experience feedback, both from users and from their clickstream data, was extremely valuable to their design process and their feature prioritization. Part of Capital One Labs’ approach to every project is a pilot study with 5-10k end users to understand how they’re using the product.
Stay tuned for more SXSW recaps in the coming months!
by Kelly Nercess Produce like Picasso
We all know Pablo Picasso and we all know he was a genius. Primarily known for his role establishing Cubism, he was also an efficient artist. He brought this discipline to each of his artistic periods including Traditional, the Blue Period, the Rose Period, his African-Influenced work, Neo-Classicalism and Surrealism. His innate talent and unstoppable drive meant that he produced an average seven new pieces of art every day. Today we have over 147,800 completed works of art from this amazing artist. At this point you may be comparing your own productivity to Picasso’s his jaw dropping feat; try not to feel too bad about yourself. There was a method to his madness.
The presentation was not a Picasso art history lesson, but rather a lesson on how to apply this work ethic to your daily tasks. How can we apply Picasso to our work? Brian Sullivan and J. Schuh presented their findings on how the average worker can apply these Picasso techniques to achieve success.
It all starts with the five P’s of Productivity: Passion, Purpose, Proficiency, Persistence and Partnership. These five components will pave the road to ‘producing like Picasso’.
Passion
The first words that came out of Picasso’s mouth was a form of the word ‘pencil’. Jose Ruiz, Picasso’s father, taught brush technique and was popular for his painting doves. When he noticed that his son loves to draw, he began giving him lessons. The start of his passion for art began at a young age and eventually led to his first oil painting at the ripe young age of 9. The name is the brand. Noticeably, Picasso has a different last name than his father. Could you imagine the iconic name being Pablo Ruiz instead of Pablo Picasso?
Picasso began art school and found himself daydreaming in class rather than focusing on what was being taught. “For being a bad student, I was sent to detention. I liked it there, because I took along a sketchpad and drew incessantly. I could have stayed there drawing forever.” Passion Points of Picasso:
Find your passion (drawing for Picasso).
Family support for focus and education.
Find a mentor, early on (Picasso’s father).
Get an education (for the sake of learning).
Know life events will fuel your passion.
Purpose
“Good artists copy. Great artists steal.” Stealing allows you to make something yours.
“Copying is doing exactly like someone else does. Stealing is when you take something, change it so much, the innovation is so disguised, so changed, that it looks like it belongs to you.” Steve Jobs can be known to use those words to his advantage. He took the branding that Picasso created and made it his own. Jobs considered Picasso his mentor and built the Apple brand using the artwork that Picasso created. If you notice the famous Apple ‘finder’ symbol, you will find a very close relation to the artwork of Picasso. Steve Jobs was shameless to steal ideas and build his brand off the work of this extraordinary artist.
There was no stopping the Apple empire, Jobs also wanted to be considered the Ritz-Carlton of retail. Someone is always there to great you at the door and the genius bar is a place to get advice on your products, rather than drinking a gin and tonic. Again, this goes back to the idea that Apple was shameless bout stealing great ideas. In order to be the best, you had to follow the footsteps of the best.
Looking back at the work of two brilliant innovators, they both continued to reinvent their brand and give their work a purpose. Picasso established his work in the Blue Period, Cubism and Neo-Classical, while Apple continued to push the boundaries with new technology including smartphones, tablets and computers.
Picasso held a great influence to Steve Jobs. Without his impact on the Apple brand, I would imagine some of the products we use today would not be the same.
“Steve Jobs admired Picasso because he could have taken a conventional approach and done it well for the rest of his life, but Picasso (like Jobs) tried to change things.” – Dr. Enrique Mallen, Forbes 2013 Purpose Points of Picasso:
Have an open mind to new experiences.
Be original. Challenge the status quo.
Look beyond your own design discipline.
Steal great ideas, but make them your own.
Take risks. Do not copy other people.
In part two of this article, I will deep dive into findings that Brian Sullivan and J. Schuh shared on the remaining three P’s: Proficiency, Persistence and Partnership. Stay tuned in our March newsletter for the final article!
