Сравнительный анализ инструментов дистрибуции сборок приложений на iOS для мобильной разработки и тестирования

One of the important stages in the process of developing and mobile application testing is delivery of the compiled build to the device of a testing engineer or participants of a beta testing program. Application code can only be tested as a build, and this build must be delivered to the testing department. In the field of testing, there has always been a problem of how to distribute builds as quickly, simply and cheaply as possible – this is the combination of benefits expected by all businesses in the IT industry. The issue of choosing a method of application distribution is especially relevant for small IT companies that are just beginning to build testing processes to ensure the quality of their products. In this article, I will analyze the available tools for distributing iOS mobile applications, present their advantages and disadvantages, and also give a detailed overview of the functionality of TestFlight by Apple Inc.

Currently there are four main and the most popular ways to distribute test builds on iOS:

1. Compiling app in Xcode (IDE – Integrated development environment) on one's own device or simulator;
2. Manual app distribution from CI/DI (continuous integration and continuous delivery), for example, TeamCity or Jenkins;
3. Internal In-App Distribution System, which company can develop independently as a web-based platform for delivering applications to testers or beta testers;
4. TestFlight – an online service for over-the-air installation and testing of mobile applications by Apple Inc.

Let us look at the summary table of advantages and disadvantages of the main solutions for distributing iOS mobile builds for IT business (Table 1):

Table 1

As we can see, TestFlight has the most benefits in using, so let us go deeper into the above mentioned tool.

TestFlight service was founded in 2010 by IT investors Benjamin Satterfield and Trystan Kosmynka as a tool for delivering mobile application builds to iOS and Android, after which, quite soon in 2012, the startup was acquired by Burstly [1]. Then Apple Inc. acquired Burstly in 2014, and one of the company's most important assets was taken over by the California-based IT industry giant. Within a month, Apple, as expected, quitted supporting Android and focused on integrating TestFlight into its own ecosystem.

Currently TestFlight is used to distribute mobile applications builds on iOS, iPadOS, macOS, tvOS, visionOS, watchOS, and iMessage platforms, on beta testers and internal test engineers (one application can be distributed to no more than 10,000 beta testers and 100 internal testers).

Let us proceed to an overview of the main functionality of TestFlight. It is important to note that we assume that we already have a ready-made build of the application for uploading to TestFlight [2]. First of all, the developer needs to go to App Store Connect and enter credentials for authorization in the service [3]. After successful authorization, we need to open the desired application and select the TestFlight section.

Fig. Overview of the main TestFlight screen of the application

Let us analyze the available actions on the main screen of the selected application in TestFlight. In the top panel, it is possible to switch between applications in the developer account by clicking on the application name. In the left panel, we can choose between several sections:

• Builds: in this section, we can switch between different platforms of the Apple ecosystem: iOS, macOS, tvOS, and so on. Switching to these categories will allow us to see the current build status of various operating systems in which the tested application is presented.
• Feedback: this section allows us to investigate crashes in tested builds, as well as see screenshots sent by testers inside TestFlight.
• Internal Testing: this section contains user groups on which test builds are distributed, with the ability to restrict available actions in accordance with the role in the application (such as Account Holder, Admin, App Manager, Developer, or Marketing). The higher the role, the more privileges are available to the user in internal testing. In internal testing, we can invite up to 100 users – in order to add additional testers, it will be necessary to remove existing ones to meet the limit.
• External Testing: this section contains groups of testers who receive beta versions of the application. In total, we can invite no more than 10,000 users – this can be done by using the invitation link to join the beta testing or by entering email addresses in App Store Connect [4].
• General Information: the section consists of three menu items:
  • All Testers with information about the total current number of invited testers (up to 10,000 users).
  • Test Information – in this section, we must fill in the information about the test application, feedback contacts and privacy policy. This information is required in order to start distributing test builds.
  • About TestFlight Data – informational pop-up that diagnostic data may appear within 72 hours since the event happened.

The main iOS Builds section contains current build statuses in each version of the application:

• Build – application build number within the selected version.
• Status – current test app availability status [5]. The possible statuses are (Table 2):

Table 2

• Groups – custom internal or external test groups that have access to a specific build.
• Invites – number of invitations sent to test a particular build.
• Installs – total build installs number.
• Sessions – total number of started sessions.
• Crashes – the number of build crashes. It should be noted that crashes may appear within up to 72 hours after the incident happened.
• Feedback – number of submitted build reviews.

Thus, we provided an overview of TestFlight functionality from the side of the application developer. We proved that the build management system provides wide abilities for administering test builds, and also provides a visual dashboard for version-by-version analysis of the test builds. Let us combine the information and review the pros and cons of using TestFlight for IT businesses:

1. The ability to receive feedback from beta testers before the release of the application in the App Store, as well as to catch crashes at an early stage. The earlier a defect is found in the code, the cheaper the cost of the error for the company, which means using TestFlight allows the company to save resources.
2. The ability to easily configure the distribution of internal builds of the application. Not every company has enough resources to develop and maintain its own distribution system for internal test builds, so TestFlight's out-of-the-box solution allows the company to set up the build distribution process as quickly and easily as possible [6].
3. Ease of use for both sides – the company and the testers. The company sets up TestFlight from the already familiar App Store Connect system, and testers just need to accept the email invitation and install the build, as they used to do in the public App Store.
4. The limit of 100 users for internal testing and 10,000 for external testing may be considered a disadvantage by some companies. However, Apple implemented this limitation on purpose – if a company has more than 100 test engineers, it should consider the option of its own system for delivering alpha versions of the application. This is a feasible task for medium and large businesses. The 10,000 user limit for external testing is set to ensure that TestFlight is not used as a replacement for the App Store – public builds for mass use must be distributed through App Store only, this rule is present in Terms of Service established by Apple Inc [7].

In conclusion we can state that compared to any other app distribution tool and method TestFlight offers a convenient way of app distribution which allows IT businesses to collect feedback from beta testers before the application is published on App Store and to avoid losses due to poorly tested applications, as well as to distribute internal builds on in-house testing engineers thus helping small and medium-sized businesses not to spend resources on developing their own app distributing system.