Back to Testing

SRI Testing Infrastructure

The SRI Testing Infrastructure includes all components dealing with the automated and manual testing of the Translator system.

Testing Components

• Translator Testing Model - schema relating to test resource specifications
• Tests Repository - contains all test assets/cases/suites
• Test Harness - automates running of queries and test runners
• Information Radiator - display test results
• ARAX GUI - display test results
• #translator-testing-alerts - Slack messages
• Feedback Repo - Translator issue reports

Test Runners

• Pass/Fail Test Runner - Quality Assurance Testing
• TRAPI Validation Test Runner - System Level Testing
• Benchmarks Test Runner - Quality Assurance Testing
• (Coming Soon) Performance Test Runner - Quality Assurance Testing

Test Expectations

• Pass/Fail Tests
• TopAnswer: The expected curie result is in the top 30 results.
• Acceptable: The expected curie result is in the top 50% of results.
• BadButForgivable: The expected curie result is either in the bottom 50% of results or not present.
• NeverShow: The expected curie result is not present in the results.
• TRAPI Validation
• The entire response message passes the current version of TRAPI.
• Benchmarks
• Given a set of “ground-truth” results for a given set of input queries, calculate the ratio of overlap between ground-truth and a response message results within a top-k range.
• Performance
• Run successively heavier load tests against the system to see how many concurrent users can send queries while returning consistent results.

Automated Pipeline

• A CronJob spins up a Docker Image of the Test Harness
• Test Harness downloads the Tests Repo and grabs the specified “Test Suite” based on the CronJob command
• Test Harness posts to Slack and creates a “Test Run” in the Information Radiator
• Test Harness loops over all of the “Test Cases” in the Test Suite
• For each Test Case, the Test Harness generates a TRAPI query, sends it to the System, and then passes along the responses
• Test Harness then loops over the “Test Assets” within the Test Case
  • Test Harness creates a single “Test” in the Information Radiator
  • Based on the Test Case metadata, the Test Harness passes the Test Asset and query responses along to a “Test Runner”
  • Test Runner performs its specific type of evaluation (pass/fail, TRAPI validation, MCQ, Pathfinder, etc.) and returns a test report
  • Test Harness uploads the test result to Information Radiator
• Once all tests have been run, the Test Harness posts to Slack with the results, sets the Test Run to complete in the Information Radiator, and the ARAX GUI then retrieves the results from the Information Radiator for display there

Manual Testing

Manual Tests are largely done by Translator consortium members who manually query the System and then evaluate the results they see. If issues are found, a GitHub Issue is then created in the Feedback Repo. Those issues are then triaged and assigned to the relevent/offending developer teams.