 |
1 Layout Engine Visual Tests (reftest)
2 L. David Baron <dbaron@dbaron.org>, Mozilla Corporation
3 July 19, 2006
4
5 This code is designed to run tests of Mozilla's layout engine. These
6 tests consist of an HTML (or other format) file along with a reference
7 in the same format. The tests are run based on a manifest file, and for
8 each test, PASS or FAIL is reported, and UNEXPECTED is reported if the
9 result (PASS or FAIL) was not the expected result noted in the manifest.
10
11 Why this way?
12 =============
13
14 Writing HTML tests where the reference rendering is also in HTML is
15 harder than simply writing bits of HTML that can be regression-tested by
16 comparing the rendering of an older build to that of a newer build
17 (perhaps using stored reference images from the older build). However,
18 comparing across time has major disadvantages:
19
20 * Comparisons across time either require two runs for every test, or
21 they require stored reference images appropriate for the platform and
22 configuration (often limiting testing to a very specific
23 configuration).
24
25 * Comparisons across time may fail due to expected changes, for
26 example, changes in the default style sheet for HTML, changes in the
27 appearance of form controls, or changes in default preferences like
28 default font size or default colors.
29
30 Using tests for which the pass criteria were explicitly chosen allows
31 running tests at any time to see whether they still pass.
32
33 Manifest Format
34 ===============
35
36 The test manifest format is a plain text file. A line starting with a
37 "#" is a comment. Lines may be commented using whitespace followed by
38 a "#" and the comment. Each non-blank line (after removal of comments)
39 must be one of the following:
40
41 1. Inclusion of another manifest
42
43 include <relative_path>
44
45 2. A test item
46
47 <failure-type>* [<http>] <type> <url> <url_ref>
48
49 where
50
51 a. <failure-type> (optional) is one of the following:
52
53 fails The test passes if the images of the two renderings DO NOT
54 meet the conditions specified in the <type>.
55
56 fails-if(condition) If the condition is met, the test passes if the
57 images of the two renderings DO NOT meet the
58 conditions of <type>. If the condition is not met,
59 the test passes if the conditions of <type> are met.
60
61 random The results of the test are random and therefore not to be
62 considered in the output.
63
64 random-if(condition) The results of the test are random if a given
65 condition is met.
66
67 skip This test should not be run. This is useful when a test fails in a
68 catastrophic way, such as crashing or hanging the browser. Using
69 'skip' is preferred to simply commenting out the test because we
70 want to report the test failure at the end of the test run.
71
72 skip-if(condition) If the condition is met, the test is not run. This is
73 useful if, for example, the test crashes only on a
74 particular platform (i.e. it allows us to get test
75 coverage on the other platforms).
76
77 Examples of using conditions:
78 fails-if(MOZ_WIDGET_TOOLKIT=="windows") ...
79 fails-if(MOZ_WIDGET_TOOLKIT=="cocoa") ...
80 fails-if(MOZ_WIDGET_TOOLKIT=="gtk2") ...
81
82 b. <http>, if present, is the string "HTTP" (sans quotes), indicating that
83 the test should be run over an HTTP server because it requires certain
84 HTTP headers or a particular HTTP status. (Don't use this if your test
85 doesn't require this functionality, because it unnecessarily slows down
86 the test.)
87
88 HTTP tests have the restriction that any resource an HTTP test accesses
89 must be accessed using a relative URL, and the test and the resource must
90 be within the directory containing the reftest manifest that describes
91 the test (or within a descendant directory).
92
93 To modify the HTTP status or headers of a resource named FOO, create a
94 sibling file named FOO^headers^ with the following contents:
95
96 [<http-status>]
97 <http-header>*
98
99 <http-status> A line of the form "HTTP ###[ <description>]", where
100 ### indicates the desired HTTP status and <description>
101 indicates a desired HTTP status description, if any.
102 If this line is omitted, the default is "HTTP 200 OK".
103 <http-header> A line in standard HTTP header line format, i.e.
104 "Field-Name: field-value". You may not repeat the use
105 of a Field-Name and must coalesce such headers together,
106 and each header must be specified on a single line, but
107 otherwise the format exactly matches that from HTTP
108 itself.
109
110 HTTP tests may also incorporate SJS files. SJS files provide similar
111 functionality to CGI scripts, in that the response they produce can be
112 dependent on properties of the incoming request. Currently these
113 properties are restricted to method type and headers, but eventually
114 it should be possible to examine data in the body of the request as
115 well when computing the generated response. An SJS file is a JavaScript
116 file with a .sjs extension which defines a global |handleRequest|
117 function (called every time that file is loaded during reftests) in this
118 format:
119
120 function handleRequest(request, response)
121 {
122 response.setStatusLine(request.httpVersion, 200, "OK");
123
124 // You *probably* want this, or else you'll get bitten if you run
125 // reftest multiple times with the same profile.
126 response.setHeader("Cache-Control", "no-cache");
127
128 response.write("any ASCII data you want");
129
130 var outputStream = response.bodyOutputStream;
131 // ...anything else you want to do, synchronously...
132 }
133
134 For more details on exactly which functions and properties are available
135 on request/response in handleRequest, see the nsIHttpRe(quest|sponse)
136 definitions in <netwerk/test/httpserver/nsIHttpServer.idl>.
137
138 c. <type> is one of the following:
139
140 == The test passes if the images of the two renderings are the
141 SAME.
142 != The test passes if the images of the two renderings are
143 DIFFERENT.
144 load The test passes unconditionally if the page loads. url_ref
145 must be omitted, and the test cannot be marked as fails or
146 random. (Used to test for crashes, hangs, assertions, and
147 leaks.)
148
149 d. <url> is either a relative file path or an absolute URL for the
150 test page
151
152 e. <url_ref> is either a relative file path or an absolute URL for
153 the reference page
154
155 The only difference between <url> and <url_ref> is that results of
156 the test are reported using <url> only.
157
158 This test manifest format could be used by other harnesses, such as ones
159 that do not depend on XUL, or even ones testing other layout engines.
160
161 Running Tests
162 =============
163
164 At some point in the future there will hopefully be a cleaner way to do
165 this. For now, go to your object directory, and run (perhaps using
166 MOZ_NO_REMOTE=1 or the -profile <directory> option)
167
168 ./firefox -reftest /path/to/srcdir/mozilla/layout/reftests/reftest.list > reftest.out
169
170 and then search/grep reftest.out for "UNEXPECTED".
171
172 There are two scripts provided to convert the reftest.out to HTML.
173 clean-reftest-output.pl converts reftest.out into simple HTML, stripping
174 lines from the log that aren't relevant. reftest-to-html.pl converts
175 the output into html that makes it easier to visually check for
176 failures.
177
178 Testable Areas
179 ==============
180
181 This framework is capable of testing many areas of the layout engine.
182 It is particularly well-suited to testing dynamic change handling (by
183 comparison to the static end-result as a reference) and incremental
184 layout (comparison of a script-interrupted layout to one that was not).
185 However, it is also possible to write tests for many other things that
186 can be described in terms of equivalence, for example:
187
188 * CSS cascading could be tested by comparing the result of a
189 complicated set of style rules that makes a word green to <span
190 style="color:green">word</span>.
191
192 * <canvas> compositing operators could be tested by comparing the
193 result of drawing using canvas to a block-level element with the
194 desired color as a CSS background-color.
195
196 * CSS counters could be tested by comparing the text output by counters
197 with a page containing the text written out
198
199 * complex margin collapsing could be tested by comparing the complex
200 case to a case where the margin is written out, or where the margin
201 space is created by an element with 'height' and transparent
202 background
203
204 When it is not possible to test by equivalence, it may be possible to
205 test by non-equivalence. For example, testing justification in cases
206 with more than two words, or more than three different words, is
207 difficult. However, it is simple to test that justified text is at
208 least displayed differently from left-, center-, or right-aligned text.
209
210 Writing Tests
211 =============
212
213 When writing tests for this framework, it is important for the test to
214 depend only on behaviors that are known to be correct and permanent.
215 For example, tests should not depend on default font sizes, default
216 margins of the body element, the default style sheet used for HTML, the
217 default appearance of form controls, or anything else that can be
218 avoided.
219
220 In general, the best way to achieve this is to make the test and the
221 reference identical in as many aspects as possible. For example:
222
223 Good test markup:
224 <div style="color:green"><table><tr><td><span>green
225 </span></td></tr></table></div>
226
227 Good reference markup:
228 <div><table><tr><td><span style="color:green">green
229 </span></td></tr></table></div>
230
231 BAD reference markup:
232 <!-- 3px matches the default cellspacing and cellpadding -->
233 <div style="color:green; padding: 3px">green
234 </div>
235
236 BAD test markup:
237 <!-- span doesn't change the positioning, so skip it -->
238 <div style="color:green"><table><tr><td>green
239 </td></tr></table></div>
240
241 Asynchronous Tests
242 ==================
243
244 Normally reftest takes a snapshot of the given markup's rendering right
245 after the load event fires for content. If your test needs to postpone
246 the moment the snapshot is taken, it should make sure a class
247 'reftest-wait' is on the root element by the moment the load event
248 fires. The easiest way to do this is to put it in the markup, e.g.:
249 <html class="reftest-wait">
250
251 When your test is ready, you should remove this class from the root
252 element, for example using this code:
253 document.documentElement.className = "";
254
255
256 Note that in layout tests it is often enough to trigger layout using
257 document.body.offsetWidth // HTML example
258
259 When possible, you should use this technique instead of making your
260 test async.
261
262 Printing Tests
263 ==============
264 Now that the patch for bug 374050 has landed
265 (https://bugzilla.mozilla.org/show_bug.cgi?id=374050), it is possible to
266 create reftests that run in a paginated context.
267
268 The page size used is 5in wide and 3in tall (with the default half-inch
269 margins). This is to allow tests to have less text and to make the
270 entire test fit on the screen.
271
272 There is a layout/reftests/printing directory for printing reftests; however,
273 there is nothing special about this directory. You can put printing reftests
274 anywhere that is appropriate.
275
276 The suggested first lines for any printing test is
277 <!DOCTYPE html><html class="reftest-print">
278 <style>html{font-size:12pt}</style>
279
280 The reftest-print class on the root element triggers the reftest to
281 switch into page mode on load. Fixing the font size is suggested,
282 although not required, because the pages are a fixed size in inches.
283
284 The underlying layout support for this mode isn't really complete; it
285 doesn't use exactly the same codepath as real print preview/print. In
286 particular, scripting and frames are likely to cause problems; it is untested,
287 though. That said, it should be sufficient for testing layout issues related
288 to pagination.
