TaylorBench

What is it?

What isn't it?

Download

• Source
• MIDlet jar and MIDlet jad,   Size ~52Kb
• OTA (Accompli or compatibles): www.poqit.com
• Example servlet source, for Comms tests. The batch files are for Windows users. Compile the source first, and then use the batch files to generate an .ear and/or .war file.
• Example servlet .ear file (as tested on jBoss)

TaylorBench 1.
 

• Source

Major Changes from TaylorMark 1.0

• Added HTTP Comms tests, complete with accompanying noddy servlet to handle requests. I have not bothered to capture times for devices. There are too many unknowns, but the test should allow you compare your own devices against a known server.
• changed random number 'scatter' graph to be joined points, to make any anomalies  more visible.
• corrected POSE/MIDP comments, since I was accidently using the 'Early Access' version , and not '1.0 Beta' as I meant to do! Whoops...

Note on MIDlet size: the beast is getting rather big, so if you have problems loading it onto your device, consider breaking it down, in the source, to just contain the tests you want to run. This is pretty simple:

• remove the entry in the tests List
• remove the matching CommandAction check
• and of course, remove the unwanted test method itself.
• (Alternatively remove whole swathes of tests...)

Major Changes from BenchMark 0.9

• name changed to TaylorBench 1.0, to distinguish it from a myriad of other benchmarks in future...
• open source. Do what you will with it...
• options added to give user lots of flexibility as to what they can test
• RMS tests added
• CPU/VM tests added
• color (optionally) removed from low-level graphics loops
• changed all default iterations to get a balance between Palm dying with lack of memory, and Accompli running too slow...

Running

• Low-level graphics - tests the performance of lines, rectangles, ellipses, arcs, images, and text rendering

• RMS - tests record creation, reading (enumerated and random), and deletion

• CPU /VM - general CPU test, general Virtual Machine test, and a simple check to make sure random numbers really are randomly distributed
• Comms - tests the performance when uploading and downloading data to a remote HTTP server. Also allows testing of file loads from the MIDlet suite.

Your options are NOT saved. If you leave the category (or the MIDlet), you will have to set up your options again.
The majority of the options chosen (or defaulted) are reflected in the test's names. It makes them a little unwieldy, but it helps remind you what you've chosen.

Before many of the tests start, an array of random numbers is built, which can take a significant time (8 or 9 seconds on an Accompli 008, for example). Whilst this initialisation is taking place, 'Initializing...' is displayed. During the test  'Running...' will be displayed. Once the test is complete, the time taken is displayed. It is also displayed in the title bar of the previous menu when you press 'Back' in case the result was not visible on your device.

Here is more detail about each category of tests.

Low-Level Graphics

Here is the full set of tests:

• lines
• rectangles (outline and filled)
• ellipses (outline and filled)
• arcs (outline and filled)
• image (small, medium, large)
• fonts (small, medium, large)
• control. An empty loop to help estimate any non-graphical overheads

The 'control' test merely runs an empty loop, rather than drawing anything. This allows you to estimate the overhead in the looping, random number generation, color setting, and so on.

You have the option to change the following parameters (beware, the Option Form is more than a screenful. For example, Accompli 008 owners will need to jog down to see the rest of the Form):

• Repeat. The number of times to run the test, e.g. 1000 lines, 1000 rectangles, etc.
• Color choice.
• fixed grayscale means all shapes are drawn using the same grayscale (0x80), minimizing the overhead of color setup. You will not see a lot since there is no screen clearing between shape draws.
• fixed color means all shapes are drawn using the same color (0x00808080).
• random grayscale. Each different shape is drawn using a randomly chosen grayscale. You will see each shape drawing, and will believe the test is working :-)
• random color. Each different shape is drawn using a randomly chosen color.
• If you select random then the color (or grayscale) setting can have a serious effect on the time taken on some devices. Use the 'control' test to see what overhead is associated with random color/grayscale setting.
• Fill choice. Rectangles, ellipses and arcs can all be drawn as outlines or filled shapes.
• Font choice. A small, medium or large font (i.e. Font.Size_SMALL, etc) can be used for text rendering.
• Image choice. A small, medium or large image can be used when rendering images. The image creation is now performed before the test starts. What's measured here is just the drawing of the image onto the canvas:
• small = 20 * 15 pixels
• medium = 40 * 30 pixels
• large = 60 * 45 pixels
• Max co-ords. This option allows you to decide the maximum coords to draw with. Obviously, if you want to compare like with like, always choose the smallest screen of any of your devices, since drawing off-screen may be very quick (since nothing actually needs to be rendered!). The size you have chosen will be reflected in the options only when you leave and return to Options. (This is because the ItemStateListener functionality seems to only partially work on some devices. I couldn't get the coords in the Form to update dynamically, and the MIDP spec is really unclear whether I should expect this anyway...)
• MIDP minimum. 96 * 54 pixels
• full screen. The dimensions fo your specific device. This functionality assumes that all canvases are the same size. Works for most current devices, but may break in the future, if devices start 'windowing' MIDlets, rather than allowing them to have the maximum amount of screen space...
• custom. Takes the value of Maximum x and 'maximum y' fileds (below) as the maximum coordinates.
• Maximum x. Only valid if 'custom' co-ords are selected, above. Otherwise, it reflects the current choice.
• Maximum y. Only valid if 'custom' co-ords are selected, above. Otherwise, it reflects the current choice.

RMS

• clean store (not a test, but a function to make sure your record store is empty.)
• record creation
• record reading, using a enumeration
• record reading, randomly accessing records
• read deletion, randomly deleting
• control. An empty loop to help estimate the non-RMS overheads

The options are:

• Repeat. The number of times to run each test. (Accompli owners beware! 100 record creations takes around 5 minutes. Get it in that desk cradle now and make a coffee. No, make that two coffees...)
• Record size. The number of bytes in each record created/read/deleted.

Also, at the top of the options, you can see how many records you have in the record store at any one time.

Its worth trying tests like random reading/deleting with a large number of records. The easy/lazy way of doing this is to run create records several times, and build up a decent number of records in the store (say 500), and then randomly read 100 records.

CPU / VM

• system array copies. Gives an indication of CPU performance.
• VM test (multiply, divide, add). Gives an indication of VM performance.
• random ints. Draws a scattergraph of the random integers generated, to make sure they are actually randomly distributed! Since all our test rely on random numbers (especially the graphics ones), its vital that the random numbers truly are random.
• control. An empty loop to help estimate the non-CPU/VM overheads

The options are:

• Repeat. The number of times to run each test.
• Array size. For the array copying, the size of the integer array.
• Random range. The range of numbers to generate random numbers for, e.g. 0 to 95.

The random scattergraph leaves a lot to be desired - there is no axis labelling, but you get the idea.

Comms

• read file (or a dynamically generated number bytes) from an HTTP server.
• write file, or specific number of bytes to HTTP server.
• read local file, i.e. one that is in the TaylorBench MIDlet jar file. There are 3 image files available.

The options are:

• Repeat. The number of times to run each test. Each repeat re-opens the HttpConnection.
• URL. The URL to fetch the files/bytes from. The default URL points to my server, but I make absolutely no promises that its quick, or even up. If you're serious about performance testing, then run against your own server, so you can control a few of the myriad parameters that affect HTTP performance. :-(
• File choice. There are 3 image files already in the TaylorBench MIDlet suite which I have reused. Pick one using this option. The size of the file you've picked will be reflected afterwards, in the test's name.
• Length. If you select 'custom' as the File choice, specify the number of bytes to read/write here, instead of using an existing file. This requires the software (at both ends) to generate a byte array, and then send the contents of that byte array.

The required servlet/web server functionality is:
'GET':
- if the parameter 'filename' is specified in the requirest, then return that file
- if the parameter 'length' is specified in the request, then return that number of bytes.
- the servlet MUST set the 'Content-Length' when replying (simplifies the code at the MIDlet end, and possibly improves performance?)
'POST':
- just read in the bytes, and reply with 200 OK.

For more detail, 'Use the source, Luke'...

Future

Results (TaylorBench 1.1):

Notes:
Variations were around 10-20%, so take the accuracy of the times with a pinch of salt.
Palm timings are those generated by the Palm POSE emulator, and are believed accurate compared to a 16Mhz Palm such as the VII.
Palm POSE was v 3.2 running a Palm IIIc skin.

Low-level graphics:
The options used are all defaults, i.e.

• 2000 iterations
• fixed grayscale
• MIDP minimum coords (96*54)

RMS:
The options used are all defaults, i.e.

• 100 iterations
• 100-byte records

CPU/VM:
The options used are all defaults, i.e.

• 5000 iterations
• 1000 integer arrays

Similarly, had to decrease iterations on VMtest for SL45i. 1000 iterations were used, and then the resulting time was multiplied by 5.

Random ints graph shows quite a wide spread on all my devices/emulators - around 25-30%, but with no obvious dips, towards the extremities. Check yours, just in case...
 

DEVICE	array copy	VM test	control
Nokia 9210 (beta s/w)	0.8	failed - probably out of memory...	0.08
Siemen SL45i	99.6	3.68	0.24
Accompli 008 (beta s/w)	34.1	2.7	1.2
J2MEWTK DefaultGray on 550Mhz Pentium III	0.06	0.06	0.06
Palm POSE KVM 1.0 Beta	25.9	2.2   (i.e. 1.1 * 2 )	0.9
Palm Vx	25.2	2.4   (i.e. 1.2 * 2)	0.9
Motorola i85s	3.9	0.9 (i..e 0.44 * 2)	0.6