--- metadata: - name: generator content: Diplodoc Platform v5.48.2 alternate: - https://admin.contest.yandex.ru/docs/en/examples/advanced-usage/default-tests-for-language.md - https://admin.contest.yandex.ru/docs/ru/examples/advanced-usage/default-tests-for-language.md --- > **Documentation Index:** Fetch the complete configuration index at https://admin.contest.yandex.ru/docs/en/llms.txt # Checking tasks with your own script If the programming task architecture doesn't suit you, you can configure the system's behavior in Yandex Contest when receiving solutions from users. Create a problem where the user has to implement a specific function. You can assign a specific number of points to the problem. To check an answer, use the unit testing system built into the language. > To set system behavior when receiving a submission, you can use the **make** compiler. In this case, behavior is determined by the configuration file **Makefile**. > > Using this behavior, you can run non-docker compilers that are available in the system. Here's an example using the Python3 interpreter. ## 1. Configure compilation {#settings} Compilation settings are used to prepare submissions for launch. Every programming language has a compilation stage in Yandex Contest, even interpreted languages. However, in this case, nothing is performed for most compilers. To prepare the user's file for running tests: 1. Create a **Makefile** in the root for the problem files: ```bash all: build build: /bin/sh ./build.sh run: /bin/sh ./run.sh ``` The script tells the system to execute the `./build.sh` script at the **build** compilation stage and execute the `./run.sh` script at the **run** stage when running a user solution and launching checkers. {% note alert %} Indents before and inside the **build** and **run** functions should be tabs rather than spaces. Otherwise, you'll get an error when running the compiled problem: `CE Makefile:4: *** missing separator. Stop.` {% endnote %} 2. Create the **build.sh** script and place it in the root for the problem files, just like for the **Makefile**: ```bash #!/bin/bash OUT=participantSolution.py TMP=tempNameForParticipantSolution.py cat $filename > $TMP || exit 1 rm $filename cat $TMP > $OUT || exit 1 ``` The participant's submission code is placed in a file on the server where it will be executed. The name of the file with the user code is contained in the variable `$filename`. In this script, you transfer the contents of the user solution to a file with a name you'll definitely know. This makes it easier to access this file in the following scripts. The contents are moved through a TMP file, so the files will not be lost if the participant sends one with the same name as the value in the variable **OUT**. {% note info %} The file names may differ. You can specify names that are better for you. {% endnote %} The result has the following file structure: ![](../../_images/examples/advanced-usage-examples/defTestRuner-step1-files.png) 3. Open the [Problems](https://admin.contest.yandex.ru/problems/) section. Under **Additional files and processing**, add the created files to **Files to compile**. {% note warning %} First add **Makefile**, then **build.sh**. {% endnote %} ![](../../_images/examples/advanced-usage-examples/defTestRuner-step1-filesSettings.png) ## 2. Configure the run time {#runtime} At the run time stage, the prepared user code is run according to specific rules. The checker is also run at this stage. The checker and user's solution are run as many times as there are tests in the test sets. Use this stage to run standard Python tests and estimate them with a prechecker. 1. Create a **run.sh** file in the root for the problem files. This script is called in **Makefile**: ```bash #!/bin/bash python3 run_tests.py ``` In this script, you run a Python script for running tests. 2. In the root for the problem files, create a **run_tests.py** file with a Python script. Specify the tests in it and run them: ```python #!/usr/bin/env python # coding: utf-8 import unittest import io from participantSolution import participantSolution # method import ? def get_basic_score(log): score = 0.0 test_scores = { 'test_first': 0.5, 'test_second': 1.0 } for line in log.strip().split('\n'): line = line.strip().split() if not line: continue if line[-1] == 'ok': score += test_scores[line[0]] return score class SampleTestSuite(unittest.TestCase): def test_first(self): result = participantSolution(123, 321) self.assertEqual(result, 444) def test_second(self): result = participantSolution(1, 5) self.assertEqual(result, 6) # Create an output stream. You will need it to receive and process information from the testrunner string_io = io.StringIO() # Create a suite and launch it. Output the result to the output stream from the last step suite = unittest.TestLoader().loadTestsFromTestCase(SampleTestSuite) runner = unittest.TextTestRunner(verbosity=2, stream=string_io).run(suite) # Run the standard output stream processing function score_basic = get_basic_score(string_io.getvalue()) # Output to stdout the full log of the user's submission print (string_io.getvalue()) # Output the points scored by the user to stdout. This value will be passed to the checker # If just passing the tests is enough, you can output, for example, OK/FAIL print(score_basic) ``` - The `get_basic_score` method can be used to process the console output of the `unittest` module. - The `SampleTestSuite` class contains the tests in the format required for the `unittest` module. The result has the following file structure: ![](../../_images/examples/advanced-usage-examples/defTestRuner-step2-files.png) 3. Open the [Problems](https://admin.contest.yandex.ru/problems/) section. In **Additional files and processing**, add the created files to **Runtime files**. {% note alert %} First add **run.sh**, then **run_tests.py**. {% endnote %} ![](../../_images/examples/advanced-usage-examples/defTestRuner-step2-filesSettings.png) ## 3. Configure the checker {#checker} The checkers are launched each time the user's solution file is executed and analyze the output in **stdin** at the runtime stage. Standard checkers work with data from test files and read text from the test's answer file. In this example, ignore what is specified in the test file and focus only on the code output in **stdin** at the solution launch stage. This is why you need to implement your own checkers. For more information about creating checkers, see [Settings](https://admin.contest.yandex.ru/docs/en/problem/main-problem.md). Let's look at a checker: 1. Create a **check_py** file in the root for the problem files: ```bash #!/bin/sh /usr/bin/python3 checker.py $1 $2 $3 ``` In this file, you call a checker in Python and pass three arguments. These are the paths to the files with **input** (test file), **output** (output of the user's solution at the run stage) and **answer** (expected answer of the test file). 2. Create a **checker.py** file in the root for the problem files: ```python import sys import datetime if __name__ == '__main__': stdout_file = sys.argv[2] with open(stdout_file) as f: lines = f.readlines() last_line = lines[-1].strip().split() basic_score = float(last_line[0]) if basic_score == 0.0: print ('Wrong answer!') sys.exit(1) else: print (basic_score) ``` The script reads the file passed in the second argument and finds its last line. It contains points awarded by the program with the tests. If the solution received 0 points, the script outputs text saying that the solution is incorrect and terminates the checker program with `exitcode 1`. The `exitcode` value determines the verdict that Yandex Contest assigns to the user's submission. For example, 0 is OK, 1 is WA, and 2 is PE. The verdict depends on the `exitcode` value as well as the checker type selected in the problem settings. Let's look at the **TESTLIB_EXITCODE_CHECKER** checker type. The result has the following file structure: ![](../../_images/examples/advanced-usage-examples/defTestRuner-step3-files.png) 3. Open the [Problems](https://admin.contest.yandex.ru/problems/) section. In **Checker settings**, specify: - **Type of checker**: TESTLIB_EXITCODE_CHECKER. - **Scoring checker**: YES. - **Checker files**: First add **check_py**, then **checker.py**. The resulting checker settings look like this: ![](../../_images/examples/advanced-usage-examples/defTestRuner-step3-filesSettings.png) ## 4. Create a test file {#test} Although test files are not used during launch, it is important to create a test file and make sure it is added to the test set. The system needs the test file and test set to know how many times it should run the user code. Test sets are also used in contest settings to see what data to display to participants on the submission page. For our example, you can create an empty test file `tests/01` and an empty answer file `tests/01.a`. ## 5. Check the result {#result} Create a file of the solution with the code: ```python def participantSolution(a, b): return a + b ``` If you did everything right, the submission page with this solution should look like this: ![](../../_images/examples/advanced-usage-examples/subm_res.png =400x) Write to Support