本文共 12651 字，大约阅读时间需要 42 分钟。

项目环境：

• python 3.6
• Django 1.11
• Coverage 5.0

减少测试代码运行时间的方法：

• 把单一的管道改为多管道并行，最后合并测试结果
• 减少数据库迁移的次数

具体做法：

stages:  - test  - resultPart_one:  stage: test  script:   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env" ]; then mkdir -p ~/$CI_RUNNER_DESCRIPTION/env; fi   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME" ]; then cd ~/$CI_RUNNER_DESCRIPTION/env/ && python3 -m venv $CI_PROJECT_NAME; fi   - source ~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME/bin/activate   - pip install --trusted-host mirrors.aliyun.com -i http://mirrors.aliyun.com/pypi/simple/ -r $CI_PROJECT_DIR/requirements.txt   - cd $CI_PROJECT_DIR   - if [ ! -d "$CI_PROJECT_DIR/logs" ]; then mkdir -p $CI_PROJECT_DIR/logs; fi   - touch $CI_PROJECT_DIR/logs/request.log   - touch $CI_PROJECT_DIR/logs/biotools.log   - echo $CI_JOB_ID >job_id.txt   - python manage.py runscript check_test_database_migrations --traceback  --settings=base_settings   - coverage run -p manage.py test app.tests.part_one --noinput --settings=base_settings --debug-mode  --keepdb   - python manage.py runscript rename_database --traceback  --settings=base_settings   - python manage.py runscript remove_testdb --traceback  --settings=base_settings  retry: 2  artifacts:    untracked: true    name: "$CI_JOB_NAME"    expire_in: 1 weekPart_two:  stage: test  script:   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env" ]; then mkdir -p ~/$CI_RUNNER_DESCRIPTION/env; fi   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME" ]; then cd ~/$CI_RUNNER_DESCRIPTION/env/ && python3 -m venv $CI_PROJECT_NAME; fi   - source ~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME/bin/activate   - pip install --trusted-host mirrors.aliyun.com -i http://mirrors.aliyun.com/pypi/simple/ -r $CI_PROJECT_DIR/requirements.txt   - cd $CI_PROJECT_DIR   - if [ ! -d "$CI_PROJECT_DIR/logs" ]; then mkdir -p $CI_PROJECT_DIR/logs; fi   - touch $CI_PROJECT_DIR/logs/request.log   - touch $CI_PROJECT_DIR/logs/biotools.log   - echo $CI_JOB_ID >job_id.txt   - python manage.py runscript check_test_database_migrations --traceback  --settings=base_settings   - coverage run -p manage.py test app.tests.part_two --noinput --settings=base_settings --debug-mode  --keepdb   - python manage.py runscript remove_testdb --traceback  --settings=base_settings  retry: 2  artifacts:    untracked: true    name: "$CI_JOB_NAME"    expire_in: 1 weekPart_three:  stage: test  script:   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env" ]; then mkdir -p ~/$CI_RUNNER_DESCRIPTION/env; fi   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME" ]; then cd ~/$CI_RUNNER_DESCRIPTION/env/ && python3 -m venv $CI_PROJECT_NAME; fi   - source ~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME/bin/activate   - pip install --trusted-host mirrors.aliyun.com -i http://mirrors.aliyun.com/pypi/simple/ -r $CI_PROJECT_DIR/requirements.txt   - cd $CI_PROJECT_DIR   - if [ ! -d "$CI_PROJECT_DIR/logs" ]; then mkdir -p $CI_PROJECT_DIR/logs; fi   - touch $CI_PROJECT_DIR/logs/request.log   - touch $CI_PROJECT_DIR/logs/biotools.log   - echo $CI_JOB_ID >job_id.txt   - python manage.py runscript check_test_database_migrations --traceback  --settings=base_settings   - coverage run -p manage.py test app.tests.part_three --noinput --settings=base_settings --debug-mode  --keepdb   - python manage.py runscript remove_testdb --traceback  --settings=base_settings  retry: 2  artifacts:    untracked: true    name: "$CI_JOB_NAME"    expire_in: 1 weekPart_four:  stage: test  script:   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env" ]; then mkdir -p ~/$CI_RUNNER_DESCRIPTION/env; fi   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME" ]; then cd ~/$CI_RUNNER_DESCRIPTION/env/ && python3 -m venv $CI_PROJECT_NAME; fi   - source ~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME/bin/activate   - pip install --trusted-host mirrors.aliyun.com -i http://mirrors.aliyun.com/pypi/simple/ -r $CI_PROJECT_DIR/requirements.txt   - cd $CI_PROJECT_DIR   - if [ ! -d "$CI_PROJECT_DIR/logs" ]; then mkdir -p $CI_PROJECT_DIR/logs; fi   - touch $CI_PROJECT_DIR/logs/request.log   - touch $CI_PROJECT_DIR/logs/biotools.log   - echo $CI_JOB_ID >job_id.txt   - python manage.py runscript check_test_database_migrations --traceback  --settings=base_settings   - coverage run -p manage.py test app.tests.part_four --noinput --settings=base_settings --debug-mode  --keepdb   - python manage.py runscript remove_testdb --traceback  --settings=base_settings  retry: 2  artifacts:    untracked: true    name: "$CI_JOB_NAME"    expire_in: 1 weekresult:  stage: result  script:   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env" ]; then mkdir -p ~/$CI_RUNNER_DESCRIPTION/env; fi   - if [ ! -d "~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME" ]; then cd ~/$CI_RUNNER_DESCRIPTION/env/ && python3 -m venv $CI_PROJECT_NAME; fi   - source ~/$CI_RUNNER_DESCRIPTION/env/$CI_PROJECT_NAME/bin/activate   - pip install --trusted-host mirrors.aliyun.com -i http://mirrors.aliyun.com/pypi/simple/ coverage   - cd $CI_PROJECT_DIR   - if [ ! -d "$CI_PROJECT_DIR/logs" ]; then mkdir -p $CI_PROJECT_DIR/logs; fi   - touch $CI_PROJECT_DIR/logs/request.log   - touch $CI_PROJECT_DIR/logs/biotools.log   - coverage combine --append   - coverage report  dependencies:   - Part_one   - Part_two   - Part_three   - Part_four

减少数据库迁移的次数的具体做法是把测试用到的sqlite数据库文件放在/dev/shm/目录下，每条管道运行测试代码之前，先在/dev/shm/目录下用job id创建一个文件夹，job id是每个job运行时唯一的id，可以查阅gitlab的官方文档了解更多，把sqlite数据库复制一份到新建的文件夹里面，再运行测试代码。运行测试代码的语句是：

coverage run -p manage.py test app.tests.part_one --noinput --settings=base_settings --debug-mode  --keepdb

注意一定要加上–keepdb

还有其他细节，例如检查数据库有没有更新，测试完成后删除测试用到的数据库等，我全都粘贴出来。

# -*- coding: utf-8 -*-import loggingimport osimport sqlite3import sysimport base_settingsimport shutilimport osimport statfrom django.conf import settings__author__ = 'JayChen'logger = logging.getLogger('scripts')def read_text():    """    读取文件中的job_id    :return:    """    with open(settings.PROJECT_PATH + '/job_id.txt', 'r') as f:        for line in f.readlines():            job_id = line.strip()    return job_iddef remove_db_files():    logger.info("model有更新，删除本地数据库")    for root, dirs, files in os.walk('/dev/shm', topdown=False):        for name in files:            os.remove(os.path.join(root, name))        for name in dirs:            os.rmdir(os.path.join(root, name))    job_id = read_text()    os.mkdir("/dev/shm/test_{}".format(job_id))    if os.path.exists("/dev/shm/test_{}".format(job_id)):        logger.info('文件夹-------' + "/dev/shm/test_{}".format(job_id) + '------创建成功')def coppy_and_rename():    job_id = read_text()    new_file_name = '/dev/shm/test_{}/vb_en.test_{}.db.sqlite3'.format(job_id, job_id)  # 文件新名字    origin_path = '/dev/shm/vb_en.test.db.sqlite3'  # 原始文件完整目录    # 数据库文件存在    if os.path.exists(origin_path):        if os.path.exists("/dev/shm/test_{}".format(job_id)):            logger.info('文件夹-------' + "/dev/shm/test_{}".format(job_id) + '------创建成功')            shutil.copy(origin_path, new_file_name)            os.chmod(new_file_name, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO)            logger.info('复制数据库并重新命名为:--------' + new_file_name.split('/')[4] + '------------')        else:            os.mkdir("/dev/shm/test_{}".format(job_id))            logger.info('文件夹-------' + "/dev/shm/test_{}".format(job_id) + '------创建成功')            shutil.copy(origin_path, new_file_name)            os.chmod(new_file_name, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO)            logger.info('复制数据库并重新命名为:--------' + new_file_name.split('/')[4] + '------------')    else:        logger.info('找不到数据库源文件')        if os.path.exists("/dev/shm/test_{}".format(job_id)):            for root, dirs, files in os.walk("/dev/shm/test_{}".format(job_id), topdown=False):                for name in files:                    os.remove(os.path.join(root, name))                for name in dirs:                    os.rmdir(os.path.join(root, name))        else:            os.mkdir("/dev/shm/test_{}".format(job_id))            logger.info('文件夹-------' + "/dev/shm/test_{}".format(job_id) + '------创建成功')    for i, j, k in os.walk('/dev/shm'):        print(i, j, k)def check_migrations():    base_dir = os.path.dirname(__file__)    migrations_data = []    if os.path.exists('/dev/shm/vb_en.test.db.sqlite3'):        try:            conn = sqlite3.connect('/dev/shm/vb_en.test.db.sqlite3')            migrations_data = conn.cursor().execute("select name from django_migrations where app='app'").fetchall()            conn.close()            data_length = len(migrations_data)            migrations_files = []            for f in os.listdir(base_settings.PROJECT_PATH + '/app/migrations'):                if f.endswith('.py') and f != '__init__.py':                    migrations_files.append(f)            files_length = len(migrations_files)            if data_length != files_length:                remove_db_files()            else:                for m in migrations_data:                    if m[0] + '.py' not in migrations_files:                        remove_db_files()        except sqlite3.OperationalError:            logger.info('读取本地数据库失败')        coppy_and_rename()    else:        coppy_and_rename()def run(*args):    logger.info('Start checking the test database migrations file.')    logger.info(args)    check_migrations()    logger.info('End checking.')'''./manage.py runscript check_test_database_migrations --traceback  --settings=base_settings  --script-args=one'''

remove_testdb.py

# -*- coding: utf-8 -*-import osimport loggingfrom django.conf import settings__author__ = 'JayChen'logger = logging.getLogger('scripts')def read_text():    """    读取文件中的job_id    :return:    """    with open(settings.PROJECT_PATH + '/job_id.txt', 'r') as f:        for line in f.readlines():            job_id = line.strip()    return job_iddef remove_db():    # 清空文件夹里面的文件，再删除文件夹    job_id = read_text()    for root, dirs, files in os.walk('/dev/shm/test_{}'.format(job_id), topdown=False):        for name in files:            os.remove(os.path.join(root, name))        for name in dirs:            os.rmdir(os.path.join(root, name))    os.rmdir("/dev/shm/test_{}".format(job_id))def remove_db_history():    """    job_id 是从小排到大的，只保留最近的20个文件夹    清理gitlab上运行失败后留下的数据库文件夹和文件    """    job_id = read_text()    job_id = int(round(float(job_id) - 20, 0))    filePath = '/dev/shm'    name_list = os.listdir(filePath)    for name in name_list:        if os.path.isfile('/dev/shm/' + name):            """            如果是文件而不是文件夹，则跳过，因为这个文件就是我用于复制到其他文件夹的数据库            """            continue        else:            data = int(round(float(name.split('_')[1]), 0))            if data < job_id:                for root, dirs, files in os.walk('/dev/shm/{}'.format(name), topdown=False):                    for file in files:                        os.remove(os.path.join(root, file))                    for dir in dirs:                        os.rmdir(os.path.join(root, dir))                os.rmdir("/dev/shm/{}".format(name))            else:                continuedef run():    logger.info('删除测试数据库')    remove_db()    remove_db_history()'''./manage.py runscript remove_testdb --traceback  --settings=base_settings'''

rename_database.py

# -*- coding: utf-8 -*-import osimport loggingimport shutilfrom django.conf import settings__author__ = 'JayChen'logger = logging.getLogger('scripts')def read_text():    """    读取文件中的job_id    :return:    """    with open(settings.PROJECT_PATH + '/job_id.txt', 'r') as f:        for line in f.readlines():            job_id = line.strip()    return job_iddef move_and_rename():    job_id = read_text()    origin_path = '/dev/shm/test_{}/vb_en.test_{}.db.sqlite3'.format(job_id, job_id)    new_file_name = '/dev/shm/vb_en.test.db.sqlite3'    if os.path.exists(origin_path):        shutil.copy(origin_path, new_file_name)        logger.info('更新数据库文件')    else:        passdef run():    move_and_rename()'''./manage.py runscript rename_database --traceback  --settings=base_settings'''

settings.py文件最后加上：

if os.path.exists(PROJECT_PATH + '/job_id.txt'):    # 读取文件中的job_id, gitlab上每条管道的job_id, 这个id是唯一的    with open(PROJECT_PATH + '/job_id.txt', 'r') as f:        for line in f.readlines():            job_id = line.strip()    TEST_SQLITE_PREFIX = '/dev/shm/test_{}/vb_en.test_{}.db'.format(job_id, job_id)else:    TEST_SQLITE_PREFIX = 'vb_en_testdb'DATABASES = {
       'default': {
           'ENGINE': 'django.db.backends.sqlite3',        'NAME': TEST_SQLITE_PREFIX + '.sqlite3',        'TEST': {
               'NAME': TEST_SQLITE_PREFIX + '.sqlite3'        },        'OPTIONS': {
               'timeout': 20,        }    }}

转载地址：http://jhomz.baihongyu.com/