Сравнительный анализ инструментов дистрибуции сборок приложений на iOS для мобильной разработки и тестирования
В статье анализируются существующие инструменты для дистрибуции сборок мобильных приложений на iOS, рассматриваются значительные преимущества и недостатки каждого инструмента, приводится обзор ключевой функциональности приложения для дистрибуции сборок приложений TestFlight. На основе практического опыта в ведущих IT-компаниях России автор делает вывод о ценности использования TestFlight в технологической индустрии для эффективной доставки сборок бета-тестировщикам и внутренней команде тестирования. Статья предназначена практикующим специалистам в сфере мобильного тестирования на платформе iOS.
One of the important stages in the process of developing and mobile application testing is delivery of the compiled build to the device of a testing engineer or participants of a beta testing program. Application code can only be tested as a build, and this build must be delivered to the testing department. In the field of testing, there has always been a problem of how to distribute builds as quickly, simply and cheaply as possible – this is the combination of benefits expected by all businesses in the IT industry. The issue of choosing a method of application distribution is especially relevant for small IT companies that are just beginning to build testing processes to ensure the quality of their products. In this article, I will analyze the available tools for distributing iOS mobile applications, present their advantages and disadvantages, and also give a detailed overview of the functionality of TestFlight by Apple Inc.
Currently there are four main and the most popular ways to distribute test builds on iOS:
- Compiling app in Xcode (IDE – Integrated development environment) on one's own device or simulator;
- Manual app distribution from CI/DI (continuous integration and continuous delivery), for example, TeamCity or Jenkins;
- Internal In-App Distribution System, which company can develop independently as a web-based platform for delivering applications to testers or beta testers;
- TestFlight – an online service for over-the-air installation and testing of mobile applications by Apple Inc.
Let us look at the summary table of advantages and disadvantages of the main solutions for distributing iOS mobile builds for IT business (Table 1):
App Distribution Tool or Method
Compiling in Xcode
1. No extra software is required
2. Code can be fixed on the go and built again instantly
1. Long building time
2. Build can be assembled only on devices with proper company's provisioning files, or on simulators
3. Access to repository code is required, so no outer testers and specialists are able to access it
4. MacOS computer is always required
1. Pipelines can be set in order to ensure all unit and UI tests are passing
2. Builds can be accessed right after the commit into repository was made
1. Smaller companies may not afford enterprise CI/CD plans
2. Special access role is required to access CI/CD
3. Can be installed with wires and computer only
Internal In-App Distribution System
1. Easy to distribute, install and switch between versions
2. Wires and computer are usually not required to install the build
1. It is complicated and costly to develop and maintain own distribution system, especially for smaller companies
2. Own services may experience long downtimes
1. Delivery process is automated
2. Every company that develops iOS app already has Apple Connect account, so it is easy to set up
3. Apps can be distributed to up to 100 internal testers and 10,000 beta testers; easy to manage invites and access for members
4. Tester can send feedback or bug reports right in TestFlight app
5. Easy to distribute, install and switch between versions
1. New app updates require Apple review
2. Longer processing times on Apple server (sometimes up to 0.5-2 hours)
As we can see, TestFlight has the most benefits in using, so let us go deeper into the above mentioned tool.
TestFlight service was founded in 2010 by IT investors Benjamin Satterfield and Trystan Kosmynka as a tool for delivering mobile application builds to iOS and Android, after which, quite soon in 2012, the startup was acquired by Burstly . Then Apple Inc. acquired Burstly in 2014, and one of the company's most important assets was taken over by the California-based IT industry giant. Within a month, Apple, as expected, quitted supporting Android and focused on integrating TestFlight into its own ecosystem.
Currently TestFlight is used to distribute mobile applications builds on iOS, iPadOS, macOS, tvOS, visionOS, watchOS, and iMessage platforms, on beta testers and internal test engineers (one application can be distributed to no more than 10,000 beta testers and 100 internal testers).
Let us proceed to an overview of the main functionality of TestFlight. It is important to note that we assume that we already have a ready-made build of the application for uploading to TestFlight . First of all, the developer needs to go to App Store Connect and enter credentials for authorization in the service . After successful authorization, we need to open the desired application and select the TestFlight section.
Fig. Overview of the main TestFlight screen of the application
Let us analyze the available actions on the main screen of the selected application in TestFlight. In the top panel, it is possible to switch between applications in the developer account by clicking on the application name. In the left panel, we can choose between several sections:
- Builds: in this section, we can switch between different platforms of the Apple ecosystem: iOS, macOS, tvOS, and so on. Switching to these categories will allow us to see the current build status of various operating systems in which the tested application is presented.
- Feedback: this section allows us to investigate crashes in tested builds, as well as see screenshots sent by testers inside TestFlight.
- Internal Testing: this section contains user groups on which test builds are distributed, with the ability to restrict available actions in accordance with the role in the application (such as Account Holder, Admin, App Manager, Developer, or Marketing). The higher the role, the more privileges are available to the user in internal testing. In internal testing, we can invite up to 100 users – in order to add additional testers, it will be necessary to remove existing ones to meet the limit.
- External Testing: this section contains groups of testers who receive beta versions of the application. In total, we can invite no more than 10,000 users – this can be done by using the invitation link to join the beta testing or by entering email addresses in App Store Connect .
- General Information: the section consists of three menu items:
- All Testers with information about the total current number of invited testers (up to 10,000 users).
- About TestFlight Data – informational pop-up that diagnostic data may appear within 72 hours since the event happened.
The main iOS Builds section contains current build statuses in each version of the application:
- Build – application build number within the selected version.
- Status – current test app availability status . The possible statuses are (Table 2):
Examples of statuses
There are problems that need to be fixed to start the reviewing process
Invalid Binary, Rejected. There is also Expired status which shows that this build is older than 90 days and no more available for installation
There is an ongoing processing on the side of the developer or Apple
Processing, Missing Compliance, Waiting for Export, In Compliance Review, Ready to Submit, Waiting for Review, In Beta Review
Application is available for testers
Ready to Test, Testing
- Groups – custom internal or external test groups that have access to a specific build.
- Invites – number of invitations sent to test a particular build.
- Installs – total build installs number.
- Sessions – total number of started sessions.
- Crashes – the number of build crashes. It should be noted that crashes may appear within up to 72 hours after the incident happened.
- Feedback – number of submitted build reviews.
Thus, we provided an overview of TestFlight functionality from the side of the application developer. We proved that the build management system provides wide abilities for administering test builds, and also provides a visual dashboard for version-by-version analysis of the test builds. Let us combine the information and review the pros and cons of using TestFlight for IT businesses:
- The ability to receive feedback from beta testers before the release of the application in the App Store, as well as to catch crashes at an early stage. The earlier a defect is found in the code, the cheaper the cost of the error for the company, which means using TestFlight allows the company to save resources.
- The ability to easily configure the distribution of internal builds of the application. Not every company has enough resources to develop and maintain its own distribution system for internal test builds, so TestFlight's out-of-the-box solution allows the company to set up the build distribution process as quickly and easily as possible .
- Ease of use for both sides – the company and the testers. The company sets up TestFlight from the already familiar App Store Connect system, and testers just need to accept the email invitation and install the build, as they used to do in the public App Store.
- The limit of 100 users for internal testing and 10,000 for external testing may be considered a disadvantage by some companies. However, Apple implemented this limitation on purpose – if a company has more than 100 test engineers, it should consider the option of its own system for delivering alpha versions of the application. This is a feasible task for medium and large businesses. The 10,000 user limit for external testing is set to ensure that TestFlight is not used as a replacement for the App Store – public builds for mass use must be distributed through App Store only, this rule is present in Terms of Service established by Apple Inc .
In conclusion we can state that compared to any other app distribution tool and method TestFlight offers a convenient way of app distribution which allows IT businesses to collect feedback from beta testers before the application is published on App Store and to avoid losses due to poorly tested applications, as well as to distribute internal builds on in-house testing engineers thus helping small and medium-sized businesses not to spend resources on developing their own app distributing system.
- Why Did TestFlight Sell To Burstly? “We Couldn’t Change The App Ecosystem Alone”. TechCrunch [Electronic resource]: Access mode: https://techcrunch.com/2012/03/05/democratize-the-data/ (date of access 19.08.2023).
- Ezugbaya, Aelitta. "Continuous Deployment For Cross-Platform Mobile Application." (2019).
- App Store Connect [Electronic resource]: Access mode: https://appstoreconnect.apple.com/ (date of access 19.08.2023).
- Fulcher, A., White, S. A. & others (2013). Testing, Promoting, and Launching a Mobile Application as an Extension Tool: A Case Study with IPMPro. HortTechnology hortte, 23(4), 407-410. Retrieved Aug 26, 2023, from https://doi.org/10.21273/HORTTECH.23.4.407
- App build statuses. Apple Developer Reference [Electronic resource]: Access mode: https://developer.apple.com/help/app-store-connect/reference/app-build-statuses (date of access 19.08.2023).
- Olaoye, A. (2022). Automated Build, Test, and Release with Fastlane. In: Beginning DevOps on AWS for iOS Development. Apress, Berkeley, CA. https://doi.org/10.1007/978-1-4842-8023-2_8
- TestFlight Terms of Service. Apple Legal [Electronic resource]: Access mode: https://www.apple.com/legal/internet-services/itunes/testflight/ (date of access 19.08.2023).