objective-c - KIF:如何自动运行/压力测试 iOS 应用程序以找出罕见的 UI 错误的原因？

Question

注意:我在标题中添加了 kif 只是为了搜索索引的目的，考虑到大部分答案都是讨论它

我正在寻找适用于 iOS 的类似 selenium 的东西，基本上是一个测试自动化/单元测试框架，可以多次运行某个 UI 场景直到它崩溃，这将帮助我缩小导致 UI 错误的原因很少和随机发生。

(顺便说一下，我已经对数据源/表交互的每一行代码进行了 NSLogged，并花了数小时分析潜在原因......但没有发现任何结论......同样，这个错误很少发生)。

我看了一些 unit testing frameworks in iOS ，但他们似乎有这么多。我不确定该选哪个。另外，我对 selenium 的引用是基于推测，因为我曾与过去在大型 Web 项目中使用过 Selenium 的 QA 人员合作(我假设 iOS 一定有类似的东西)。

既然我是一个 iOS 项目的单人团队，我将不得不戴上 QA 帽子并找出这个错误。

我遇到了一个典型的错误，当 UITableView 中插入的实际行数与数据源委托(delegate)返回的行数之间存在差异时，就会发生这种错误。这是错误信息:

*** Assertion failure in -[UITableView
 _endCellAnimationsWithContext:] Exception in insertRows: Invalid
 update: invalid number of rows in section 0.

The number of rows contained in an existing section after the update (2) must be equal to
 the number of rows contained in that section before the update (2),
 plus or minus the number of rows inserted or deleted from that section
 (1 inserted, 0 deleted) and plus or minus the number of rows moved
 into or out of that section (0 moved in, 0 moved out).

我单击一个 UITableViewCell，它会将我带到另一个 UITableView。有时它有效

有时(很少)它不会(出现上述错误):

Answer 1

更新:..我在分隔符后的底部添加了关于 KIF 2.0 的示例代码..对于那些对 KIF 比我面临的具体问题更感兴趣的人:

经过一些研究和试验..我将我的选择范围缩小到两个测试自动化库: Frank和 KIF .我最终决定在借用 cucumber's 时使用 KIF小 cucumber syntax描述我的单元测试。

我选择 KIF(而不是 Frank)的原因是 KIF 是 100% 基于 obj-c 的，而不是使用 ruby​​ 作为Frank 也是如此。所以设置更简单，更适用于我狭窄的测试用例要求。话虽如此，我承认如果我的应用程序更复杂(即使用来自多个服务器的输入等)，Frank 会更有用。你可以看到这个 excellent 的最后一个季度演示文稿以了解有关 KIF、Frank 和其他自动化测试框架(包括 Apple 自己的框架)的优缺点的更多信息 UI Automation .

在使用 KIF 后，我发现了导致上述错误的错误，并且我可以 100% 地使用 KIF 重现它!它很少发生的原因是因为它只发生在我非常快地点击屏幕时..而且因为 KIF 自动执行这些步骤..它以非常快的速度完成它们..这暴露了错误:).

下面是我用于测试的代码示例。这只是为了让您快速了解 KIF(和 Gherkin)可以为您做什么:

在一个文件中我指定了我想要运行的场景:

- (void)initializeScenarios;
{
    [self addScenario:[KIFTestScenario scenarioToCompleteSignInAndLoadInbox]];
    [self addScenario:[KIFTestScenario scenarioToFillAttachmentsWithData]];
    [self addScenario:[KIFTestScenario scenarioToViewAndLoadFileBucket]];
    [self addScenario:[KIFTestScenario scenarioToViewAndLoadFileBucketSubView]];
}

每个场景映射到步骤(要了解更多关于基于测试驱动程序开发的 gherkin 语法和行为驱动开发，我强烈建议阅读这本关于 cucumber 的优秀书籍):

/* @given the application is at a fresh state
   @and   the user already has an imap email account with a valid username/pwd

   @then  the user can successfully log in
   @and   the inbox view will be loaded
   @and   the inbox will get loaded with the latest batch of emails in the user inbox
 */
+ (id)scenarioToCompleteSignInAndLoadInbox
{
    KIFTestScenario *scenario = 
      [KIFTestScenario scenarioWithDescription:@"Test that a user 
                                                 can successfully log in."];
    [scenario addStepsFromArray:[KIFTestStep stepsCompleteSignInAndLoadInbox]];

    return scenario;
}


/* @given that the user is already signed in
   @and   the user has already downloaded their folders 

   @then  the user can click on the folders view
   @and   the user can click on the 'attachments' remote folder
   @and   the latest batch from the 'attachments' remote folder will download
 */
+ (id)scenarioToFillAttachmentsWithData {
    KIFTestScenario* scenario = 
      [KIFTestScenario scenarioWithDescription:@"Test that we can view the 
                                                 attachments folder and fill 
                                                 it with data."];
    [scenario addStepsFromArray:[KIFTestStep stepsToFillAttachmentsWithData]];
    return scenario;

}

/* @given that the user is already signed in
   @and   the user has already downloaded their folders
   @and   the user has already downloaded attachments

   @then  the user can click on inbox menu button
   @and   the user can click on folder list menu button
   @and   the user can click on the file bucket icon (on the account list view)
   @and   the data for the file bucket is fetched from the dbase
   @and   the file bucket view displayes the attachments
 */
+ (id)scenarioToViewAndLoadFileBucket {
    KIFTestScenario *scenario = 
       [KIFTestScenario scenarioWithDescription:@"Test that a user can successfully 
                                                  view and load 
                                                  file bucket parent view"];
    [scenario addStepsFromArray:[KIFTestStep stepsToViewAndLoadFileBucketPage]];

    return scenario;
}

/* @given that the user is already signed in
   @and   the user has already downloaded their folders
   @and   the user has already downloaded attachments
   @and   the user has already opened file bucket view 

   @then  the user can click on a random row in the file bucket view table
   @and   the subview will retrieve data from the dbase pertaining to that row
   @and   the subview will display the data in the uitableview
 */
+ (id)scenarioToViewAndLoadFileBucketSubView {
    KIFTestScenario *scenario = 
       [KIFTestScenario scenarioWithDescription:@"Test that a user can successfully
                                                  view and load filet
                                                  bucket sub view"];
    [scenario addStepsFromArray:[KIFTestStep stepsToViewAndLoadFileBucketSubPage]];
    return scenario;   
}

步骤是使用 KIF 的 UI 自动化方法定义的(这只是一个示例):

// this step assumes there is an attachment folder that contains emails with attachments
+ (NSArray *)stepsToFillAttachmentsWithData {

    NSMutableArray* steps = [@[] mutableCopy];

    [steps addObject:
        [KIFTestStep stepToTapViewWithAccessibilityLabel:@"InboxMenuButton"]];

    NSIndexPath* indexPath = 
        [NSIndexPath indexPathForRow:remoteAttachmentFolderNumber inSection:0];
    KIFTestStep* tapAttachmentRowStep = 
        [KIFTestStep stepToTapRowInTableViewWithAccessibilityLabel:
                                     @"attachments" atIndexPath:indexPath];

    [steps addObject:[KIFTestStep stepToWaitForNotificationName:
         (NSString *)kBeganSyncingOlderEmails object:nil           
                          whileExecutingStep:tapAttachmentRowStep]];

    [steps addObject:tapAttachmentRowStep];

    [steps addObject:
        [KIFTestStep stepToWaitForViewWithAccessibilityLabel:@"attachments"]];

    KIFTestStep *fillingInboxStep = 
        [KIFTestStep stepToWaitForNotificationName:
                                 (NSString *)kOldMailBatchDelivered object:nil];

    [fillingInboxStep setTimeout:kSpecialTimeoutForLongTests];
    [steps addObject:fillingInboxStep];

    return steps;
}

---

KIF 2.0 示例代码:KIF 2.0 使用 Xcode 5 的全新 test navigator .. 这比 KIF 1.0 所做的巨大改进..现在您的测试感觉比过去更加有机和自然..(即实时进行..而不是创建场景在未来运行等)。您甚至可以使用播放按钮等来测试每个。您应该尝试一下。

这里有一些例子(同样使用 gherkin 语法):

#import <KIF/KIF.h>
#import "KIFUITestActor+EXAdditions.h"
#import "KIFUITestActor+UserRegistration.h"

@interface LoginTests : KIFTestCase

@end
@implementation LoginTests

- (void)testReset {
    [tester flushDbase];
    [tester reset];
}

/* @given that the app is in a fresh clean state
 @and   that no one has ever registered with the server

 @then  the user can register their themselves with the server
 @and   immediately start with the rider's map
 @and   their location on the map shows
 */

- (void)testRegistration
{
    [tester flushDbase];
    [tester reset];
    [tester singleUserRegistration];
    [tester showUserCurrentLocationOnMap];
}

/* @given that the user has already registered with the server
   @and the user is not currently logged in

 @then  the user can login using their user name and password
 @and   immediately start with the rider's map
 @and   their location on the map shows
 */
- (void)testSuccessfulLogin
{
    [tester reset];
    [tester login];
    [tester showUserCurrentLocationOnMap];
}

/* @given that the user has already registered 
   @and that the user is already logged in before app launch

 @then the user starts on the map view with the location visible
 @and the button prompts them to set pick up location
 */
- (void)testStartOfApplication {
    [tester showUserCurrentLocationOnMap];
    [tester showsPickUpButton];
}
@end

下面是类别文件中一些测试用例的实现:

- (void)reset
{
    [self runBlock:^KIFTestStepResult(NSError **error) {
        BOOL successfulReset = YES;

        // Do the actual reset for your app. Set successfulReset = NO if it fails.
        AppDelegate* appDelegate = [[UIApplication sharedApplication] delegate];
        [appDelegate resetApp];

        KIFTestCondition(successfulReset, error, @"Failed to reset some part of the application.");

        return KIFTestStepResultSuccess;
    }];
}

- (void)flushDbase {
    [self runBlock:^KIFTestStepResult(NSError **error){
        NSURL *url = [NSURL URLWithString:@"http://randomdomain.com/flush_db"];
        NSURLRequest *request = [NSURLRequest requestWithURL:url];
        NSError *connectionError = nil;

        BOOL databaseFlushSucceeded = YES;

        NSURLResponse *response;
        NSData *resultData = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&connectionError];
        if (!resultData) {
            databaseFlushSucceeded = NO;
            KIFTestCondition(databaseFlushSucceeded, error, @"failed to connect to server!");
        }

        if (connectionError) {
            databaseFlushSucceeded = NO;
            KIFTestCondition(databaseFlushSucceeded, error, [NSString stringWithFormat:@"connection failed. Error: %@", [connectionError localizedDescription]]);
        }

        return KIFTestStepResultSuccess;
    }];
}


- (void)navigateToLoginPage
{
    [self tapViewWithAccessibilityLabel:@"login email"];
}

- (void)returnToLoggedOutHomeScreen
{
    [self tapViewWithAccessibilityLabel:@"Logout"];
    [self tapViewWithAccessibilityLabel:@"Logout"]; // Dismiss alert.
}