One of our primary undertakings for the SpreadsheetWEB 6.0 release was a major dedicated effort to improve the performance of our core spreadsheet calculation engine. The calculation engine was written in C++ and is optimized for server-side processing, focusing primarily on handling many requests across various workbooks. This component has been in continuous development since early 2003.
While the current version includes significant improvements to workbook loading times, we will focus this blog post on the enhancements geared towards calculation times (i.e. beyond the initial load). This refers to the time required for the core engine to perform a set of calculations, assuming the workbook has already been loaded into memory.
Testing Methodology
We used a complex spreadsheet (approximately 9.5 MB) that calculates the shipping costs of a package based on factors such as weight, size, origin, and destination. These variables constitute the inputs for the model. Once set, the engine performs its calculations and the system returns the defined output variables – in this case, the shipping cost.
The workbook was published as a Web Service application on a SpreadsheetWEB server (subsequently known as server application). A different application was running on a separate server (client application), performing calls to the API in order to set the corresponding inputs and request the required outputs from the model.
Four sets of tests were run to simulate various levels of concurrent usage. Each simulated user was sending 20 different sequential requests.
- 1 user
- 5 concurrent users
- 25 concurrent users
- 50 concurrent users
We took the average response time across all requests grouped by concurrency level in order to compare calculation performance under load.
Testing Setup
All tests were run out of the AWS US-West (Oregon) Data Center.
The client application was executed from an m4.xlarge machine running an instance of 64-bit Windows Server 2012 R2. The server application (SpreadsheetWEB server) was installed on a c5.2xlarge server, also running 64-bit Windows Server 2012 R2.
The versions of SpreadsheetWEB server that were tested include versions 5.2 and 6.0. The SpreadsheetWEB server configurations were identical across all instances, and each was contained to a maximum of 5 nodes per instance (i.e. worker processes, capable of handling requests).
Test Results
The chart below compares the average response times between versions 5.2 and 6.0 of SpreadsheetWEB under four different simulated concurrency scenarios.
The test results demonstrate that version 6.0 is approximately twice as fast as 5.2 in all concurrent usage scenarios. This is indicative of the effort that has been expended to improve our core engine over the past development cycle.
Based on the data, version 6.0 also features better optimization with regards to the distribution of concurrent requests across nodes. In version 5.2, the average response times increase threefold (0.5 to 1.5 seconds) when concurrent usage increases from 1 to 5 users. However, the response times in version 6.0 are largely unaffected at this concurrency level (from 0.3 to 0.4 seconds). In other words, 6.0 can distribute the load amongst 5 available nodes much more efficiently.
In summary, clients using version 6.0 will not only notice a major performance boost in the execution of their applications at runtime, but they will likely also be able to squeeze twice the workload from a 6.0 server.
