A friendly, invitation-only competition between web-based performance testing tools.
Wednesday March 15, 2017 at the Spring 2017 Software Test Professionals conference in Phoenix, AZ
In 2017, we are going to try something exciting and new by hosting a live performance testing event for performance testing tool creators. We are welcoming a friendly competition between the tools as a means of learning about our various strengths, innovations and techniques for overcoming the challenges of performance testing. By bringing together lead engineers with the industry's top tools we hope to share with the audience the leading practices for performance automation, scripting and analysis. And of course the fun of this event will be the live, 4-hour hosting by your somewhat crazy hosts James and Mark. Seriously, I think this might be the craziest thing we've done so far. More than the cheese plate episode.
The duration of the event is four hours and will include four distinct measured areas:
- Test Script Creation. With respect to the time allowed, a maximum of two business process scripts with a maximum of 25 steps will be included
- Test Design. This is for a representative load test with a ramp up and ramp down period. Tests will be for a maximum of 50 users, a duration of 25 minutes under full load. Tests shall include basic operating system monitoring of a LAMP stack instance. Admin level credentials will be provided to allow for integrated monitoring. Deep diagnostic integration (Dynatrace, Wily, NewRelic) is excluded from this competition
- Test Execution. All test execution targets will be in Amazon AWS. The data centers will be rotated relative to the participants in the test to allow for the closest AWS data center to the client to be the target of the test. All script development will occur on a dev API instance with matching configuration and data in the Amazon northern VA data center. A minimum of two Amazon data centers must be included. Test should be repeatable.
- Analysis and Reporting. A set of questions will be provided at the onset of the test which will drive a set of reports at the end.
The business process definitions, test definitions and reporting definitions will be provided at the onset of the four hour test window to allow for an even playing field for all vendors present.
The application for test in 2017 is the SugarCRM community edition. No third party tools will be included on top of the base SugarCRM installation This is a browser-based application. A custom AMI will be used for this event.
This is a live broadcast event, hosted by PerfBytes’ hosts James Pulley and Mark Tomlinson. Each hour of broadcast will have 50 minutes of airtime and 10 minutes of commercials. As a live event each participating vendor will have the following airtime events to highlight the company, the executive team and the skills of their professional service team (or exceptional customer).
- one profile of the participating team member ( two minutes – vendor provided resume of participant with live references by hosts)
- one profile of professional services/VAR organization ( two minutes – vendor provided Video )
- one profile of tool ( five minutes – Vendor provided video)
- one live executive interview ( ten Minutes )
Additionally the ten minutes of commercials each hour will be divided among all participants. The target is five participants, which allows for two (2) additional minutes of commercials per hour, eight (8) minutes over the broadcast window.. Commercials to be rotated in the order of sign up.
The event will be addressed similarly to an IronMan event, with points awarded in each of the four categories for a winner, plus an overall winner for the competition event for a total of five awards.
Scoring the Event
- 20 Points for each category against defined evaluation criteria.
- 5 additional points to be awarded for first to complete.
- 1 additional point will be awarded in execution for each Cloud provider included in the test execution beyond inclusion of two Amazon regions.
Evaluation criteria in each stage are to be objective, measurable and verifiable by a neutral industry professional from the hosts.
The progress of the event is to be reported via integration with log monitoring (Splunk, ELK Stack or other) dashboards on both the development and execution AMI instances. This represents a non intrusive method for pulling progress information
PerfBytes will be leveraging LiveStream Studio’s Web Remote Camera capability to integrate camera’s at each participating vendor location. More information on LiveStream’s Web Remote Camera capability can be found here. A dress rehearsal for video and audio integration will take place one week prior to the event.
Vendor Technical Requirements for remote participation:
- A web camera with a ¾ over the shoulder view for the participant will be required at each location along with integrated microphone.
- Live network connection to AWS
Additionally, PerfBytes will provide dedicated facilities in Software Test Professionals Conference venue for vendors who prefer to send executive or participant to a particular location for this event.
Trophies will be awarded in each category and for an overall winner.
Vendors invited (alphabetically):
- Apica LoadTest committed
- CA Blazemeter
- Flood.IO evaluating
- HP/Microfocus StormRunner Load
- Neotys Neoload evaluating
- OctoPerf committed
- SOASTA Cloudtest
This is a paid event. Fees are used to cover the time of the hosts, the setup and documentation for the event, the use of the production facilities, honorarium to judges for the event, infrastructure fees related to broadcast and food/beverage for any participants who elect to participate in-person at the Software Test Professionals conference broadcast location.
Fee for participation: $3,000 USD. $2,000 USD for those participating in the vendor showcase for the Spring 2017 Software Test Professionals Conference
All participating vendors will have access to footage from the event at the completion of the broadcast. This may be used for marketing purposes so long as the PerfBytes icon remains in the lower right hand corner to identify the source of the broadcast. This will be made available in MP4 downloadable form by PerfBytes.
PerfBytes will provide a unique sign up key for each vendor to use in marketing purposes. Information on viewers using the sign up key will be provided to the vendor.
The overall winner of the 2017 event will be highlighted for the next year on the PerfBytes website and will receive an additional read in highlight monthly for the next year. Category winners will be rotated for the same highlight on the PerfBytes News of the Damned broadcast.
A ten minute short video will be available 30 days after the conclusion of the event. All participants will have redistribution rights to the original broadcast and ten minute short version.
2017 Web Performance Tool Hackathon Critical Dates
|Feb 24th, 2017||Commitment Date for Vendor Participation|
|Feb 25th, 2017||Sign Up Keys Sent to Vendors
Event Sign up Page goes Live on Perfbytes.com
|March 7th, 2017||Network Dress Rehearsal for Video
Profile of Performance engineer due.
Vendor Commercials Due
|March 13th||Final Payment Due for event participation|
|March 15th, 2017||8am-noon Arizona Standard Time
|April 14th, 2017||Ten Minute Short Video of Event posted to PerfBytes Youtube Channel|