| commit | author | age | ||
| 2207d6 | 1 | # Code Coverage |
| W | 2 | |
| 3 | At some point you want to review which parts of your application are tested well and which are not. | |
| 4 | Just for this case the [CodeCoverage](http://en.wikipedia.org/wiki/Code_coverage) is used. When you execute your tests to collect coverage report, | |
| 5 | you will receive statistics of all classes, methods, and lines triggered by these tests. | |
| 6 | The ratio between all lines in script and all touched lines is a main coverage criterion. In the ideal world you should get 100% code coverage, but in reality 80% is really enough. Because even 100% code coverage rate doesn't save you from fatal errors and crashes. | |
| 7 | ||
| 8 | *To collect coverage information `xdebug` is required**. | |
| 9 | ||
| 10 |  | |
| 11 | ||
| 12 | Coverage data can be collected manually for both local and remote tests. Remote tests may be executed on different nodes, | |
| 13 | or locally but running through web server. It may look hard to collect code coverage for Selenium tests or PhpBrowser tests. But Codeception supports remote codecoverage as well as local. | |
| 14 | ||
| 15 | ### Configuration | |
| 16 | ||
| 17 | To enable code coverage put these lines in the global configuration file `codeception.yml`: | |
| 18 | ||
| 19 | ``` yaml | |
| 20 | coverage: | |
| 21 | enabled: true | |
| 22 | ``` | |
| 23 | ||
| 24 | That's ok for now. But what files should be present in final coverage report? | |
| 25 | Pass an array of files or directory to include/exclude sections. The path ending with '\*' matches the directory. | |
| 26 | Also you can use '\*' mask in a file name, i.e. `app/models/*Model.php` to match all models. | |
| 27 | ||
| 28 | There is a shortcut if you don't need that complex filters: | |
| 29 | ||
| 30 | ``` yaml | |
| 31 | coverage: | |
| 32 | enabled: true | |
| 33 | include: | |
| 34 | - app/* | |
| 35 | exclude: | |
| 36 | - app/cache/* | |
| 37 | ``` | |
| 38 | Include and exclude options can be redefined for each suite in corresponding config files. | |
| 39 | ||
| 40 | By default, if coverage is reported to be < 35% it is marked as low, and >70% is high coverage. | |
| 41 | You can also define high and low boundaries with `low_limit` and `high_limit` config options: | |
| 42 | ||
| 43 | ```yaml | |
| 44 | coverage: | |
| 45 | enabled: true | |
| 46 | low_limit: 30 | |
| 47 | high_limit: 60 | |
| 48 | ``` | |
| 49 | ||
| 50 | By default, show all whitelisted files in `--coverage-text` output not just the ones with coverage information is set to false, config option: | |
| 51 | ||
| 52 | ```yaml | |
| 53 | coverage: | |
| 54 | enabled: true | |
| 55 | show_uncovered: false | |
| 56 | ``` | |
| 57 | ||
| 58 | By default, show only the coverage report summary in `--coverage-text` output is set to false, config option: | |
| 59 | ||
| 60 | ```yaml | |
| 61 | coverage: | |
| 62 | enabled: true | |
| 63 | show_only_summary: false | |
| 64 | ``` | |
| 65 | ||
| 66 | For further information please refer to the [PHPUnit configuration docs](https://phpunit.readthedocs.io/en/latest/configuration.html) | |
| 67 | ||
| 68 | ## Local CodeCoverage | |
| 69 | ||
| 70 | The basic codecoverage can be collected for functional and unit tests. | |
| 71 | If you performed configuration steps from above, you are ready to go. | |
| 72 | All you need is to execute codeception with `--coverage` option. | |
| 73 | ||
| 74 | To generate a clover xml report or a tasty html report append also `--coverage-xml` and `--coverage-html` options. | |
| 75 | ||
| 76 | ``` yaml | |
| 77 | codecept run --coverage --coverage-xml --coverage-html | |
| 78 | ``` | |
| 79 | ||
| 80 | XML and HTML reports are stored to the `_output` directory. The best way to review report is to open `index.html` from `tests/_output/coverage` in your browser. | |
| 81 | XML clover reports are used by IDEs (like PHPStorm) or Continuous Integration servers (like Jenkins). | |
| 82 | ||
| 83 | ## Remote CodeCoverage | |
| 84 | ||
| 85 | ### Local Server | |
| 86 | ||
| 87 | If you run your application via Webserver (Apache, Nginx, PHP WebServer) you don't have a direct access to tested code, | |
| 88 | so collecting coverage becomes a non-trivial task. The same goes for scripts that are tested on different nodes. | |
| 89 | To get access to this code you need `xdebug` installed with `remote_enable` option turned on. | |
| 90 | Codeception also requires a little spy to interact with your application. As your application runs standalone, | |
| 91 | without even knowing it is being tested, a small file should be included in order to collect coverage info. | |
| 92 | ||
| 93 | This file is called `c3.php` and is [available on GitHub](https://github.com/Codeception/c3). | |
| 94 | `c3.php` should be downloaded and included in your application at the very first line from controller. | |
| 95 | By sending special headers Codeception will command your application when to start codecoverage collection and when to stop it. | |
| 96 | After the suite is finished, a report will be stored and Codeception will grab it from your application. | |
| 97 | ||
| 98 | Please, follow installation instructions described in a [readme file](https://github.com/Codeception/c3). | |
| 99 | ||
| 100 | To connect to `c3` Codeception uses url config from PhpBrowser or WebDriver module. | |
| 101 | But URL of index with `c3.php` included can be specified explicitly with `c3_url` parameter defined: | |
| 102 | ||
| 103 | ``` yaml | |
| 104 | coverage: | |
| 105 | # url of file which includes c3 router. | |
| 106 | c3_url: 'http://127.0.0.1:8000/index-test.php/' | |
| 107 | ``` | |
| 108 | > note: we can't have multiple `c3_url` on same host difference only by port. Please, use alias of domain | |
| 109 | (i.e. `frontend.dev:8000`,`backend.dev:8080`) instead. | |
| 110 | ||
| 111 | After the `c3.php` file is included in application you can start gather coverage. | |
| 112 | In case you execute your application locally there is nothing to be changed in config. | |
| 113 | All codecoverage reports will be collected as usual and merged afterwards. | |
| 114 | Think of it: Codeception runs remote coverage in the same way as local. | |
| 115 | ||
| 116 | ### Remote Server | |
| 117 | ||
| 118 | But if you run tests on different server (or your webserver doesn't use code from current directory) a single option `remote` should be added to config. | |
| 119 | For example, let's turn on remote coverage for acceptance suite in `acceptance.suite.yml`: | |
| 120 | ||
| 121 | ``` yaml | |
| 122 | coverage: | |
| 123 | remote: true | |
| 124 | ``` | |
| 125 | ||
| 126 | In this case remote Code Coverage results won't be merged with local ones, if this option is enabled. | |
| 127 | Merging is possible only in case a remote and local files have the same path. | |
| 128 | But in case of running tests on a remote server we are not sure of it. | |
| 129 | ||
| 130 | CodeCoverage results from remote server will be saved to `tests/_output` directory. Please note that remote codecoverage results won't be displayed in console by the reason mentioned above: local and remote results can't be merged, and console displays results for local codecoverage. | |
| 131 | ||
| 132 | ### Working Directory (Docker/Shared Mounts) | |
| 133 | ||
| 134 | If your remote server is accessed through a shared mount, or a mounted folder (IE: Docker Volumes), you can still get merged coverage details. | |
| 135 | Use the `work_dir` option to specify the work directory. When CodeCoverage runs, Codeception will update any path that matches the `work_dir` option to match the local current project directory. | |
| 136 | ||
| 137 | Given a docker command similar to: | |
| 138 | ```bash | |
| 139 | docker run -v $(pwd):/workdir -w /workdir... | |
| 140 | ``` | |
| 141 | ||
| 142 | Use the below configuration to allow coverage mergers. | |
| 143 | ```yaml | |
| 144 | coverage: | |
| 145 | remote: false | |
| 146 | work_dir: /workdir | |
| 147 | ||
| 148 | ``` | |
| 149 | ||
| 150 | ### Remote Context Options | |
| 151 | ||
| 152 | HTML report generation can at times take a little more time than the default 30 second timeout. Or maybe you want to alter SSL settings (verify_peer, for example) | |
| 153 | To alter the way c3 sends it's service requests to your webserver (be it a local or a remote one), you can use the `remote_context_options` key in `coverage` settings. | |
| 154 | ||
| 155 | ``` yaml | |
| 156 | coverage: | |
| 157 | remote_context_options: | |
| 158 | http: | |
| 159 | timeout: 60 | |
| 160 | ssl: | |
| 161 | verify_peer: false | |
| 162 | ``` | |
| 163 | ||
| 164 | Context stream options are [well documented at php.net](http://php.net/manual/en/context.php) | |
| 165 | ||
| 166 | ## Conclusion | |
| 167 | ||
| 168 | It's never been easier to setup local and remote code coverage. Just one config and one additional file to include! | |
| 169 | **With Codeception you can easily generate CodeCoverage reports for your Selenium tests** (or other acceptance or api tests). Mixing reports for `acceptance`, `functional`, and `unit` suites provides you with the most complete information on which parts of your applications are tested and which are not. | |
