Singularity University moves from Moneyball to MVP Machine and sees a $30 return on every $1 invested

Intro

In 2002, Moneyball took the baseball world by storm. The General Manager of the Oakland Athletics, Billy Beane, a former top prospect himself, had begun to offer concrete proof that a data-driven approach to player evaluation could have a real impact in a world that was previously based on gut, experience, and tradition. Other Major League Baseball teams followed suit, populating their front offices with data scientists, who were able to evaluate talent efficiently and effectively.

In many ways, digital marketing has undergone a similar evolution. Analytical rigor and data science is no longer a niche pursuit, but the norm. Top companies can’t fill their analytical needs fast enough based on current talent pools.

Fast forward from 2002 to 2019, and Major League Baseball has taken the next step. MLB teams have tapped talent-development factory Driveline to not just improve how they evaluate players, but to bring the same rigor to player development. A book titled The MVP Machine chronicles Driveline’s use of detailed video analysis, unconventional devices, new technology, training and radar to track more detailed body movement and to develop proven approaches in-house rather than having to identify players who already fit the bill.

Teams are asking themselves:

Why pay for a great baseball player when you can create one? Similarly, why not create a great CRO professional?

Round Barn takes the same approach with our CRO clients, and our partnership with Singularity University is a great example. It’s one thing to deliver CRO service, it’s another to teach an organization to apply CRO on a day-to-day basis with a similar level of analytical rigor and impact.

Challenge

Singularity University’s VP of Marketing was looking for a partner to help his team establish a rigorous approach to A/B testing, so they could make better use of data to improve productivity and impact their bottom line.

After 11 years of steady expansion, the futurist educational institution founded by Peter Diamandis and Ray Kurzweil found itself facing growth, marketing and resource challenges right here in the present. In order to keep pace with the rapidly developing landscape, they would need to add significant growth expertise, and analytical A/B testing rigor to their team and company culture.

SU’s marketing leadership determined that they needed to streamline their processes and improve their ability to extract important lessons from their day-to-day business. More to the point, they needed to increase their conversion rate across both online and in-person programs.

Round Barn Labs helps improve conversion rates.

• 66% win percentage on tests
• $700,000 in increased profit
• 30:1 return on investment
• Note: ROI versus testing spend

“I have never gotten into a flow where we could execute so many tests so quickly. We were able to get into a cadence that allowed us to increase our batting average quickly over time achieving A/B test win after win after win.”    

- Peter Glenn, Director, Consumer Marketing

“I really enjoyed seeing the rigor...lots of impressive work. This level of CRO expertise has allowed us to rethink how we structure our growth team and A/B testing moving forward.”

- Cyndie Hoffman, Senior Growth Marketing Manager

• 18 wins from 27 completed tests, 6 TBD at the time of writing
• From live date of tests to August 2019
• Estimated return through end of year, based on results from January to August 2019.  

‍
Action

Enter Round Barn Labs. Just as baseball front offices enlisted Driveline to help them make better use of data in all aspects of player development, Singularity’s marketing leadership enlisted RBL to help them make rigorous testing and smart use of data a greater part of their company culture. It would be a challenge, since Singularity University had built a sizable business with $32 million in funding.

Working with SU’s newly minted Director of Growth, Round Barn started from scratch, installing a new A/B testing software, providing guidance on how to set up & analyze A/B tests, and changing the culture within their organization to support testing.

Beginning with SU’s online courses, RBL created a custom testing process, ramping up the testing schedule and and focusing on higher-ticket items as the program progressed – such as the annual Global Summit conference, the week-long Executive Programs, and the three-day Exponential Innovation Program. These all operate as independent businesses – with unique teams, tech stacks, programming, and target customers. It’s like working with 4 different customers under one roof.

‍

Results

The addition of Round Barn’s testing and growth expertise expanded the capacity of SU’s marketing team and helped prove the merit of the newly formed growth team. The first round of tests resulted in 6 wins out of 9, and yielded many lessons that would drive more effective and efficient marketing strategy.

Not only that, RBL had built trust within the organization amongst a variety of stakeholders in web development, design, and analytics. On weekly Zoom calls, RBL dove into the current live tests on Google Optimize, presenting both hypotheses and results, including impact on conversion rate (CVR) and revenue. This created more trust, allowing Round Barn Labs to integrate into the SU task management in Asana and allowing the SU growth team to increase their impact on their own organization.

As a group, we improved the testing rigor and culture within the company - on top of driving greater revenue. Within one quarter, Singularity had gone from crawling to sprinting, from running 9 tests across three months to 5 tests per week, and RBL was managing a rigorous testing program across SU’s 4 independent businesses.

Procedures created

1. A Team Approach: We helped inspire and facilitate greater collaboration between web development, analytics, and marketing.
2. Testing Rigor: We ramped up SU's testing program, moving from 1 test per week to a rapid cadence of 5 tests per week. Each client’s definition of ‘rapid’ depends on their culture, traffic, and capacity.
3. Seamless Integration: We worked as an extension of the SU team to improve their data-driven decision-making & relieve the burden of overtaxed team members.

Here are 2 of the 33 tests we’ve run for Singularity University.

FYI these were not the worst nor the best tests, but a good representation of the work

Test 1: Concise v Verbose

Move a critical customer notification Call To Action (CTA) button above the fold through simplification of the text greeting

Hypothesis: Will increase usability and drive more users to start the application.

Result: SU saw a consistent +41% increase through all steps of the application, and increased sales by $29,950  (test ran for 14 days, on half the traffic)

With 8 weeks until SU’s yearly event, Global Summit, implementation of this winning variation at a 41% increase would drive between $149,750 - $299,500.

Source: https://docs.google.com/document/d/1TjiTu1doA1rnpNF5D3cc69LWD6xvmQrdJ59vk9Clcjo/edit
(through Aug 19). Revenue is based on current SU traffic volume fluctuations and current SU ticket pricing.

Test 2: Apply Now Direct Link

Hypothesis: Changing the Call To Action (CTA) ‘Apply Now’ to point directly to the application will increase sign-ups.

Result: We saw a 76% increase in Global Summit (GS) applications completed at 99% statistical significance after three weeks.

Revenue Impact: $50,915 additional revenue during the test. $152,745 minimum impact over the 6 weeks preceding the Global Summit event (August 19).

This winning variation was implemented through a personalization campaign in Google Optimize.

Conclusion

Singularity University has made good use of Round Barn Labs’ CRO expertise to not only improve their results in the short- and medium-term, but to better incorporate growth marketing principles and processes into their day-to-day operations to inspire long-term, lasting growth.

Like Major League Baseball front offices and player development operations, SU is working to create a ‘Growth MVP’ pipeline of marketing professionals who have a deeper understanding of A/B testing and a data-driven approach to growth. Gary Marx is our strategic and tactical CRO partner leading the planning, rigor, and testing. He works in close collaboration with RBL on SU and other CRO projects. Gary was central to the success of this project and case study. Having run well over 10,000 A/B tests, across a range of clients from large enterprises to early-stage startups and everything in between. We know how to build growth through experimentation. To find out how Round Barn Labs can help you build this expertise within your marketing team, contact us today.

‍