In a team meeting, a question was raised as to whether the marketing website for MeetEdgar was effectively communicating the inherent value of the web app and therefore whether it was effective in both persuading visitors to sign up for the app and providing them with appropriate expectations of what the app will be able to do for them.

MeetEdgar is a social media marketing automation web app targeted at small business owners to help alleviate the burden of building a strong social media presence for their business, which can be very time consuming. Upon joining MeetEdgar, users connect the social media accounts for their business. They then upload their social media posts into MeetEdgar’s content library and organize it into categories, for example “My Blog Posts” or “Spring Sale,” which allows them to curate and control the type of content that is published to their social media accounts without individually scheduling every single post.

Instead, to schedule content, the user creates a weekly recurring schedule, placing time slots throughout the week for their categories depending on which days and times they want to post about each category topic. The app then automatically pulls posts from their categories to publish to social media at the indicated times. 

The user can edit the schedule at any time to add or remove categories depending on whether the category is relevant to their current promotions. For example, a Spring Sale category might be added to the schedule on April 1st and removed on the 30th when the sale ends, allowing the user to prepare content in advance, keep their content organized, and reuse content when relevant throughout the year without needing to recreate or re-upload that content.

In comparison to competitor social media scheduling products, the weekly recurring schedule and content library save users significant time in repetitive scheduling and content uploading in contrast to most products that require each post to be scheduled individually and do not include content libraries to store posts.

The marketing website attempts to describe these differences and the various features that MeetEdgar offers. However, given that Edgar operates in a fundamentally different way than most competitor products, it may not be providing users with a clear understanding of MeetEdgar’s functionality and how that functionality can be used to improve the efficiency and increase the efficacy of a business’s social media presence.

Objectives

The User Research Team proposes to perform inherent value usability testing to answer the following research questions: 

• What aspects of the product provide the most value to long time users?
• What do new visitors to the marketing website understand the inherent value of the product to be before they sign up?

By comparing the results from long time users and new visitors, the team will identify any differences in the understanding of new visitors in contrast to the aspects of the product that long time users find most valuable. Additionally, as suggested by Spool (2004), the team may also be able to determine what features new visitors are excited by when visiting the website that long term users may be unaware of, either because they were introduced after they signed up or because they have simply not explored that feature.

By comparing and contrasting the results of the two participant groups, the User Research Team will make recommendations for changes to the marketing website to improve the communication about the most high value features of the product with the goal of increasing visitor sign up rates. The team will also propose recommendations for increasing utilization of existing features by long time users.

This research will not propose new features for the product, but rather will propose how to improve communication around existing features for both new visitors and existing users.

Methods

All participants in the study will be asked to sign the Statement of Informed Consent prior to participation (See Appendix A). Each participant will receive a coupon code for 1 month of free service with MeetEdgar as an incentive for their participation, a $49 value.

There are two options for completing this study using either live usability interviews or surveys. Each option is described below.

Option A: Live Usability Interviews

Usability interviews would be broken into two phases: Phase 1 in which long time users of MeetEdgar will be interviewed, and Phase 2 in which new visitors to the marketing website will be interviewed. Interviews will be conducted remotely via video meetings and utilizing screen sharing, lasting approximately 30-60 minutes. Each test will include the participant, the test moderator, and at least one observer from the team. To collect qualitative data, the moderator and observers will take notes throughout the tests on what the participant views, speaks out loud about, interacts with, or demonstrates confusion with. Tests will also be recorded allowing for further review and transcription.

At the conclusion of each test, the moderator and observers will meet to combine their notes, identify key takeaways from the interview, and add the data to a data collection spreadsheet for further analysis at the conclusion of all tests.

Phase 1

In Phase 1, the team will conduct live usability tests with long time users of MeetEdgar to determine a baseline for the most valuable aspects of the product as understood by these users. Users will be asked to walk the research team through their MeetEdgar account, thinking out loud, demonstrating their typical workflow, and describing the features they use most often. See Appendix B for the test protocol script that will be used to guide these interviews.

Participants and Recruitment for Phase 1: Approximately 7-10 participants will be recruited for this phase. Participants must be active MeetEdgar customers that have used the app for 12 consecutive months or more and who have not participated in a previous MeetEdgar research study. These users will be identified using the customer management system. Users who meet the criteria will be emailed an invitation to participate including the consent form and a link to sign up for a video meeting appointment using the calendar app, Calendly.

Phase 2

Phase 2 of the study will consist of live usability interviews with new visitors to the MeetEdgar marketing website who have never been a MeetEdgar customer or logged into the app. Participants will be asked to walk the research team through the marketing website, thinking out loud as they go to describe what they understand the app can do, how they are making their purchasing decision, and any aspects that are unclear to them. See Appendix B for the testing protocol that will guide these interviews.

Participants and Recruitment for Phase 1: Approximately 7-10 participants will be recruited for this phase. The team will utilize a chat pop up window on the marketing website to invite new visitors to the home page to participate in the interview by signing up for an interview time using the Calendly app. The invite will expressly ask the participant not to sign up for MeetEdgar until the interview is complete.

Option B: Surveys

Surveys would also be broken into two phases, with the first phase being conducted with long time users and the second phase being conducted with new visitors to the marketing website. Similar to the live interviews, the two surveys would ask participants to describe their experiences with either the product or the marketing website. The goal of this study is to produce qualitative data regarding what users perceive to be the most valuable aspects of Edgar. As such, the surveys will include a mix of open ended and multiple choice questions allowing users to describe their experiences. See Appendix C for the proposed survey protocols.

Participants and Recruitment: 

Participants for both phases will be recruited in the same manner as described in Option A, except that they will be sent a survey link instead of a link to sign up for an interview on Calendly. 30 participants will be recruited in each phase.

Comparison of Options A and B:

Both live usability interviews and surveys can result in qualitative data regarding which aspects of the product provide the most value to long time users and which aspects the marketing website is communicating to new visitors as providing the most potential value for them. However, each type of study has both strengths and drawbacks.

Live usability interviews provide the research team with first hand observation of the participant interacting with the product and the marketing site. Think aloud techniques and direct observation may offer additional insights that might not be self-reported in a survey format. However, this form of study requires a significantly higher time commitment for the team to be present on multiple 30-60 minute video meetings. Recordings of the meetings must also be transcribed. This means, fewer participants may be included for the study to be concluded in a timely manner.

Surveys, on the other hand, allow for more participants to be included in the study as they may be completed by the participant on their own time without moderation from the team, making this method of study cheaper in terms of paid team member time. However, given that the goal of this study is to gather qualitative data about the value proposition of the product and marketing website, survey questions must include open ended responses. The team has seen these types of open ended surveys provide valuable data for MeetEdgar studies in the past. However, the time saved in allowing for asynchronous responses may be lost again in standardizing and analyzing open ended survey responses from a larger number of participants. 

At this time, the research team recommends utilizing option A, live usability interviews, as the additional insight that may be gleaned through the think aloud techniques described in the interview protocols and observation may result in more usable data than surveys for this particular study. However, once any recommended changes resulting from the study have been implemented, the research team recommends completing a follow-up study using a survey with new visitors to gauge the effectiveness of the changes that have been implemented in communicating the value of the product on the marketing website.

Study Wrap Up and Data Analysis

Immediately following each interview, participants will be sent a followup email thanking them for their time and ensuring they have been successful in using the 1 month free coupon code.

Using the data collected, the team will first analyze the qualitative findings from Phase 1 to determine which aspects of the app are being most heavily used, which features provide the most value as demonstrated by the long term users, and which features might be under utilized by these users. 

Next, the team will analyze the data from Phase 2 to identify what new visitors determined to be the inherent value of MeetEdgar based on the information on the marketing website, which features they were most intrigued by, and any aspects of the website that may have caused confusion or hesitation in purchasing.

Lastly, the team will compare and contrast the results of the two phases to determine whether the marketing website is conveying the same inherent value to new visitors that long time users find in the app, where the website can be improved to convey this information better to increase conversion, and which features of the app can be better communicated to existing users to increase utilization.

Proposed Timeline

The User Research Team proposes to conduct the Inherent Value Test over the course of approximately 5 weeks. In Phases 1 and 2 combined, a total of approximately 14-20 participants will participate in 30-60 minute live tests, requiring a total time commitment from the team of anywhere from 10 hours to 20 hours depending on the total duration of the tests and the final number of participants. Additional time for test setup, data transposing, video transcription, participant followup, data analysis, and reporting is estimated at an additional 20 hours, for a total time of between 30 and 40 hours of work, spread out across three User Research Team members. 

Given schedule and time constraints of both team members and participants, interviews likely cannot be conducted back to back. Participants will be offered multiple time slots to choose from when signing up for a testing session, spreading the interviews out over a period of a few weeks. The following schedule is proposed: 

Tasks	Week 1	Week 2	Week 3	Week 4	Week 5
Calendar & Test Setup	X
Phase 1 Participant Identification & Recruitment	X
Phase 1 Testing		X	X
Phase 2 Participant Identification & Recruitment			X
Phase 2 Testing			X	X
Participant Followup					X
Analysis and Reporting					X

Summary

The User Research Team for MeetEdgar is proposing to conduct an Inherent Value Test in which we will compare the experience of long time users with the experience of new visitors to the marketing website to determine: 

• What aspects of MeetEdgar provide the most value to long time users; 
• What new visitors to the marketing website understand to be the value of Edgar; And,
• Where the MeetEdgar team might make improvements to the marketing website and to customer communication to increase conversion and feature adoption rates based on the value data.

The research team recommends conducting this study utilizing live usability interviews via video meetings over a period of 5 weeks, resulting in a final report to be distributed to the MeetEdgar team.

References

Spool, Jared M. (2004). Inherent Value Testing. User Interface Engineering. 

https://articles.uie.com/inherent_value_testing/Appendix A: Statement of Informed Consent

Statement of Informed Consent

You are invited to participate in a research study about MeetEdgar. The goal of this research study is to determine what aspects of MeetEdgar provide the most value to customers. 

This study is being conducted by the MeetEdgar User Research Team. To participate in the study, you must meet one of the following criteria:

• You are an active MeetEdgar customer who has used MeetEdgar for more than 12 consecutive months.
• You are a small business owner with a social media presence who has not used MeetEdgar previously.

Participation in this study is voluntary. If you agree to participate in this study, you should expect to spend 30-60 minutes speaking with the User Research Team on a Zoom meeting and be ready to share your screen with us so that we may follow along with you as you view the MeetEdgar website or app.

For your participation in this study, you will receive a coupon code for one month of free service with MeetEdgar. We do not expect that any of the questions or activities during the Zoom meeting will be upsetting for any participants. You may decline to answer any question the team poses to you and you may leave the Zoom meeting at any time.

The information you will share with us if you participate in this study will be kept completely confidential to the full extent of the law. Zoom meetings will be recorded for internal research purposes only and will not be shared with anyone outside the MeetEdgar team. At the conclusion of the study, recordings will be transcribed, any personally identifying data will be removed from the transcriptions, and the recordings will be destroyed.

Please note: You must be 18 or older to participate in this study. 

If you have any questions about this study, please contact User Research Team using the contact information below. 

usability@meetedgar.com

By adding your signature below, you are consenting to participate in this study. Please print or save a copy of this information for your records. 

Appendix B: Live Usability Interview Protocols

Script for Live Inherent Value Testing with Existing Long Term Users

Thanks for agreeing to speak with us! Our goal for today is to determine what you see as Edgar’s greatest value.  What you, specifically, gain from using Edgar for your business.  We want to know what works and what doesn’t work for you and why. So to begin, can you tell us a bit about yourself and your business, and what initially led you to joining Edgar? (allow for response, ask questions if needed)

Thanks so much for that info! Now, we’d like you to open your Edgar account in a new browser window and share your screen with us so that we can follow along with you as you navigate your Edgar account.

Next, we’d like you to pretend that we don’t know anything about Edgar.  As we go through things, talk to us as though we’re someone you’re recommending Edgar to that has never seen it before.  Walk us through your usual workflow when you log into Edgar.  Do whatever you would normally do when you log in or anything you would show someone if you were trying to explain how you use Edgar when they don’t know what it is.  

And as you do that, we’d like you to think out loud.  You can narrate your actions, for example I might say, “When I log into Edgar, I first go to the Library, and then I click the Pending content tab.  I usually filter now so I can look at content for this account only.” And so on. Just tell us what you’re doing as if describing it to someone who can’t see your screen.  Does that make sense? (wait for affirmative or clarify if they have questions)

And while you’re doing that feel free to share any thoughts or opinions on how the site works with us.  Things that you love about Edgar, things that you don’t love so much. We want to know what you usually think about while using Edgar, both the good and the not so good.

Along the way, we may ask you some questions about your workflow or the feedback you have for us.  And when you’re finished, just let us know that this is the point that you would normally log off.  We’ll ask you some follow up questions and then we’ll conclude the session.  Do you have any questions before we begin? (answer questions)

Let’s get started then.  Pretend you’ve just logged into Edgar and walk us through what you would normally do as if we’ve never seen the site before.

Wrap-up:

• Thank them for giving us a tour of their Edgar account.
• Follow-up question:
  • If they did not visit certain pages or features, prompt them to discuss those areas briefly (examples may include link shortening, bulk edit, imports, shuffle, skip, schedule, etc.). If they’ve never used something before, ask their impressions and whether they’re likely to use it in the future or how we could better have directed their attention to it. 
• Thank them again and provide them with the coupon code

Script for Live Inherent Value Testing with New Visitors

Thanks for agreeing to speak with us! Our goal for today is to determine what value you might find in Edgar after visiting our website.  You were chosen for this study because you are not an Edgar customer yet, and you’ve never actually logged into the app.  That’s just what we want.  First off, we’d love for you to tell us a little about yourself, your business, and what led you to go looking for something like Edgar. (allow for response, ask questions if needed)

Thanks so much for that info! We want to know how our website is doing at communicating to potential customers what Edgar can do for them.  To do that, we’d like you to share your screen with us and walk us through the site.  Let’s go ahead and get screen sharing started.  (help them screenshare)

Perfect!  We can see your screen.  Now, we’d like you to load the home page.  Start there and do whatever you would do when trying to decide whether to subscribe or not.  Scroll down, read the info, click links. Anything that helps you make your purchasing decision.

As you do that, we’d like you to think out loud so that we can follow your thought process.  For example, you might see a feature that you think would be useful to you, and you can tell us how you think you might use it for your business.  Or, you might read something you don’t understand or you wish there was more information about, and you can tell us what’s not clear to you.  

The key is that we want to know how our website communicates its value to you, specifically.  How can Edgar help you with your business? What things do you see on the site that make you excited to try Edgar and want to sign up?  What things are missing that you wish Edgar could do?  What parts of the site confuse you or turn you away from Edgar?  Any thoughts that come into your head about what you see, we want to hear them. Act as though I’m a coworker, and we are both viewing the site trying to decide whether it’s a good fit for your business.  Talk to me about what you see like you would talk to a coworker.

Let’s get started on the home page.  Just start by telling us your initial impressions of the site and then keep going through the information, clicking links and looking at things as you normally would if you were trying to decide whether to sign up for Edgar. When you feel like you’ve gotten all the information you need or like you’re at the point where you would usually make your decision and either sign up or move on to another option, just let us know.

Wrap Up:

• If they missed certain pages, direct them to and ask them to think out loud through them.
• If you took notes or need clarification on any of their comments, ask.
• Give them the coupon code
• Thank them for their time and sign off.Appendix C: Surveys

Appendix C: Surveys

Survey for Long Time Users

1. Tell us a bit about your business. What does the business do? (Open ended response)
2. What is your role within the business?
   1. Business Owner
   2. Employee of the business (Full or Part time)
   3. A contractor for the business
3. How does Edgar help you achieve the social media goals for your business? (Open ended response)
4. When you log into Edgar, what is your typical workflow? What page would you visit first, what actions would typically take, what parts of the app would you view or interact with, etc.? Describe a normal day using Edgar. (Open ended response)
5. What is your favorite feature of Edgar? (Multiple choice listing features)
6. Why is that feature your favorite? (Open ended response)
7. Are there any features of Edgar that you don’t use? Select all that apply. (Multiple choice listing features)
8. Why don’t you utilize the feature(s)? (Open ended response)
9. Is there anything you wish Edgar could do for you that he can’t do right now? (Open ended response)

Survey for New Visitors

1. Tell us a bit about your business. What does the business do? (Open ended response)
2. What is your role within the business?
   1. Business Owner
   2. Employee of the business (Full or Part time)
   3. A contractor for the business
3. What led you to look for a social media solution like Edgar? Have you used a social media automation app before? Which one(s)? (Open ended response)
4. Looking at the MeetEdgar.com home page, what do you think is the most important thing Edgar can do for your business? (Open ended response)
5. Looking at the MeetEdgar.com features page, what feature are you most excited about? (Multiple choice listing features)
6. Why are you excited about that feature? (Open ended response)
7. Are there any features you don’t understand? Select all that apply. (Multiple choice listing features)
8. What is it you don’t understand about the feature(s)? (Open ended response)
9. Based on what you have learned on the website, do you intend to try Edgar? Why or why not?