A few weeks back I presented a webinar along with Brian Suthoff from Localytics. We delivered a mobile application measurement framework based on my SMART methodology and I wanted to follow up on the Q&A portion of the webinar. We unfortunately didn't have enough time to address each question during the webinar...
For those who did not attend, you can watch the session and/or download the slides here.
Q: How can I do A/B testing?
Q: If I instrument my iPad app with Google Analytics, and want to do A/B testing, is there a way to use Google Website Optimizer as my A/B testing platform?
Q: For analytics & optimization, A/B testing is essential. Please describe my options. Thank you.
A: Yes, you can build A/B testing logic into your app. For example, you can present different content or options to different users, and tag an event for the path (or option) they saw. Then in Localytics you create funnels that start with that event and end at your goal to easily compare the results of users sent in different directions. and you start the two funnels each with the different event. That way you can compare the results very easily.
Web-style A/B testing in apps is still in it’s infancy, testing and optimization is currently restricted to apps that leverage HTML, CSS, and JavaScript for the majority of their interface (Netflix, for example). By developing different UIs and assigning them to device IDs at time of app installation and activation (persisting them, of course) it is possible to measure user behaviour against specified KPIs by app UI to determine relative lift.
Q: If my app sends people to someone else's website - where the purchase conversion takes place - what are my options for measuring / tracking that conversion?
A: Apps are sandboxed from each other and from the web, so many of the conversion tracking techniques used online are not available. In this case, you could pass referring information (e.g., app ID, Localytics ID, campaign ID) to the website where code on the website would then record the referrer and capture the conversion. Essentially, it is up to the destination web site where the conversion takes place to pass on conversion metrics. Of course, if they will let you place your own tracking codes on their conversion pages - all the better...
Q: Device collects data in chunks and transmits when device becomes online to Localytics data collection servers. Is that a correct assumption?
A: Yes that's correct. During app usage the Localytics client library will cache all event data in local storage, even if the app is online. Then when scheduled by the app developer, but typically at the beginning and end of a session, or as part of a background service, the usage data will be compressed and uploaded to Localytics.
This is true for most vendor SDKs in that some data is sent upon app launch but it is up to the developer to instrument their app to send data at appropriate times during usage depending on current app processes. Best practice dictates that it is not prudent to send data to the collection servers upon each and every action but to batch requests to avoid overtaxing the devices data connection.
While not true batching of requests, Adobe has the ability to force the data collector ‘offline’ during processor intensive app routines and then return it to ‘online’ mode when complete. Additionally, using the offlineThrottleDelay function allows the developer to set the cadence (or delay) when sending requests to mitigate performance impacts once the application is online again.
Google SDKs allow for the batching of requests using the ‘dispatch’ method. It should be noted that with the Google SDK requests are set with a timestamp of when the app sends data to Google NOT when the action occurs.
Q: The fact that every big corporation is jumping into the app sea is not necessarily a measure of success. Are those apps really successful?
A: It certainly varies by company, just as not all websites are successful. How success is measured will also vary. But to offer an anecdotal example: Rue La La announced at Shop.org that 22% of their sales now come from mobile, which feels like a pretty great success. Also, eBay expects to see $5B in mobile revenue in 2011.
Q: How should Mobile Strategy be approached for applications that have already been launched with no or limited measurement
A: Fortunately, users upgrade the apps on their devices very quickly, so adding analytics to the next version of an already launched app is an easy first step. Beyond the basic integration and metrics, you'll also want to think about how your goals and questions translate into event tags for your app. There are many tips and suggestions on the Localytics blog and either Localytics or Semphonic can share additional best practices or consulting services to quickly get you started.
Q: So you are saying that the main difference with mobile analytics compared to traditional online analytics is the implementation?
Q: What is really different from mobile analytics compared to regular online analytics? Is there really a difference?
A: Implementation differences are more effect than cause. The biggest differences are that apps and websites are both built and used very differently. Web analytics can safely assume that users are online (or they couldn't reach the site) and that everything is received in chronological order at the time it happens. Web analytics are also more server-based and were not built to take advantage of PC/device APIs.
Apps may function completely offline, or suddenly lose connectivity, which dramatically changes how the implementation must be done and how data are cached, timestamped, uploaded, de-duplicated, etc. As mentioned on the call, Adobe SiteCatalyst report suites must be configured to support device timestamps, which then affects the rest of your SiteCatalyst deployment should you wish to rollup these report suites. Google analytics assumes that everything happens when it's reported, which is hardly true when 15% to 25% of app usage happens while offline.
Other differences include web time-outs vs app multitasking, location based on source IP or device API, power and network conservation, etc.
Q: Do you have any stats regarding the mobile Canadian market? How much different do you think it is compared to the US market?
A: Canada is experiencing rapid mobile growth, however current adoption is far less than that in the US due to carriers not supporting the broad extent of smartphones found here in the US. This is changing rapidly, however and it is predicted that more than 425 million smarphones and tablets will be sold in 2011. Deloitte has some predictions on the Canadian mobile market here - http://bit.ly/uimnia
Q: To the best practice of separate profiles per channel: what about an iPad user hitting our regular fixed web site, should that go to a dedicated iPad suite or is it OK if that mixes with our desktop/fixed web suite?
A: Regular websites tend to work well on iPads (non-Flash of course!). Depending on your business, you may still want to have a version of your site optimized for iPads or build a cross-platform site with dynamic layouts (the new Boston Globe site is a great example).
It is recommended that you create separate profiles for your various channels so as not to conflate differing device types and/or platforms. Having separate profiles for fixed web, mobile web, and mobile apps is the preferred method and again this is based on device type NOT site type. So in response to the question, it is perfectly acceptable to have iPad traffic to your fixed web site be included within your existing web site profile.
Q: If I instrument my iPad app with Google Analytics, and want to do A/B testing, is there a way to use Google Website Optimizer as my A/B testing platform?
Q: For analytics & optimization, A/B testing is essential. Please describe my options. Thank you.
A: Yes, you can build A/B testing logic into your app. For example, you can present different content or options to different users, and tag an event for the path (or option) they saw. Then in Localytics you create funnels that start with that event and end at your goal to easily compare the results of users sent in different directions. and you start the two funnels each with the different event. That way you can compare the results very easily.
Web-style A/B testing in apps is still in it’s infancy, testing and optimization is currently restricted to apps that leverage HTML, CSS, and JavaScript for the majority of their interface (Netflix, for example). By developing different UIs and assigning them to device IDs at time of app installation and activation (persisting them, of course) it is possible to measure user behaviour against specified KPIs by app UI to determine relative lift.
Q: If my app sends people to someone else's website - where the purchase conversion takes place - what are my options for measuring / tracking that conversion?
A: Apps are sandboxed from each other and from the web, so many of the conversion tracking techniques used online are not available. In this case, you could pass referring information (e.g., app ID, Localytics ID, campaign ID) to the website where code on the website would then record the referrer and capture the conversion. Essentially, it is up to the destination web site where the conversion takes place to pass on conversion metrics. Of course, if they will let you place your own tracking codes on their conversion pages - all the better...
Q: Device collects data in chunks and transmits when device becomes online to Localytics data collection servers. Is that a correct assumption?
A: Yes that's correct. During app usage the Localytics client library will cache all event data in local storage, even if the app is online. Then when scheduled by the app developer, but typically at the beginning and end of a session, or as part of a background service, the usage data will be compressed and uploaded to Localytics.
This is true for most vendor SDKs in that some data is sent upon app launch but it is up to the developer to instrument their app to send data at appropriate times during usage depending on current app processes. Best practice dictates that it is not prudent to send data to the collection servers upon each and every action but to batch requests to avoid overtaxing the devices data connection.
While not true batching of requests, Adobe has the ability to force the data collector ‘offline’ during processor intensive app routines and then return it to ‘online’ mode when complete. Additionally, using the offlineThrottleDelay function allows the developer to set the cadence (or delay) when sending requests to mitigate performance impacts once the application is online again.
Google SDKs allow for the batching of requests using the ‘dispatch’ method. It should be noted that with the Google SDK requests are set with a timestamp of when the app sends data to Google NOT when the action occurs.
Q: The fact that every big corporation is jumping into the app sea is not necessarily a measure of success. Are those apps really successful?
A: It certainly varies by company, just as not all websites are successful. How success is measured will also vary. But to offer an anecdotal example: Rue La La announced at Shop.org that 22% of their sales now come from mobile, which feels like a pretty great success. Also, eBay expects to see $5B in mobile revenue in 2011.
Q: How should Mobile Strategy be approached for applications that have already been launched with no or limited measurement
A: Fortunately, users upgrade the apps on their devices very quickly, so adding analytics to the next version of an already launched app is an easy first step. Beyond the basic integration and metrics, you'll also want to think about how your goals and questions translate into event tags for your app. There are many tips and suggestions on the Localytics blog and either Localytics or Semphonic can share additional best practices or consulting services to quickly get you started.
Q: So you are saying that the main difference with mobile analytics compared to traditional online analytics is the implementation?
Q: What is really different from mobile analytics compared to regular online analytics? Is there really a difference?
A: Implementation differences are more effect than cause. The biggest differences are that apps and websites are both built and used very differently. Web analytics can safely assume that users are online (or they couldn't reach the site) and that everything is received in chronological order at the time it happens. Web analytics are also more server-based and were not built to take advantage of PC/device APIs.
Apps may function completely offline, or suddenly lose connectivity, which dramatically changes how the implementation must be done and how data are cached, timestamped, uploaded, de-duplicated, etc. As mentioned on the call, Adobe SiteCatalyst report suites must be configured to support device timestamps, which then affects the rest of your SiteCatalyst deployment should you wish to rollup these report suites. Google analytics assumes that everything happens when it's reported, which is hardly true when 15% to 25% of app usage happens while offline.
Other differences include web time-outs vs app multitasking, location based on source IP or device API, power and network conservation, etc.
Q: Do you have any stats regarding the mobile Canadian market? How much different do you think it is compared to the US market?
A: Canada is experiencing rapid mobile growth, however current adoption is far less than that in the US due to carriers not supporting the broad extent of smartphones found here in the US. This is changing rapidly, however and it is predicted that more than 425 million smarphones and tablets will be sold in 2011. Deloitte has some predictions on the Canadian mobile market here - http://bit.ly/uimnia
Q: To the best practice of separate profiles per channel: what about an iPad user hitting our regular fixed web site, should that go to a dedicated iPad suite or is it OK if that mixes with our desktop/fixed web suite?
A: Regular websites tend to work well on iPads (non-Flash of course!). Depending on your business, you may still want to have a version of your site optimized for iPads or build a cross-platform site with dynamic layouts (the new Boston Globe site is a great example).
It is recommended that you create separate profiles for your various channels so as not to conflate differing device types and/or platforms. Having separate profiles for fixed web, mobile web, and mobile apps is the preferred method and again this is based on device type NOT site type. So in response to the question, it is perfectly acceptable to have iPad traffic to your fixed web site be included within your existing web site profile.
Recent Comments