Baidu has been caught gaming the system on a recent test of its supercomputer capabilities. This was particularly significant because the Chinese search engine giant displayed results that outperformed those received by both Microsoft and Google.
The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) was set up to test the capabilities of supercomputers to recognise objects in images. It requires computers to classify images into a 1,000 different categories. This capability is a big deal for search engines as they work to improve the accuracy of their search results using purely automated means.
On average, trained humans have an error rate of 5-percent when taking the test; this stands in contrast to the 4.94-percent error rate of Microsoft, and the 4.8-percent rate earned by Google. Baidu, on the other hand, managed an impressive 4.58-percent error rate. While the percentages do not appear to differ by much, it represents a massive leap in computing capability.
Organisers of the ILSVRC noted that Baidu had greatly exceeded the number of weekly entries allowed. Each participant is allowed only two entries per week; however, Baidu had apparently set up 30 accounts and spammed the contest with more than 200 requests over a six month period – including 40 results during a five day period in March. The concern over the number of entries is because it allows Baidu to fine tune its results to the set of sample images. Essentially allowing the computer to take the test multiple times until it learns the correct answers to the questions.
Baidu itself has apologised for the incident, although it has no provided an explanation for what happened and instead called it a ‘mistake’. It is likely that Baidu, like other Asians, simply believed that it could have kept going until its supercomputer memorised the data set; much like how students sit for multiple mock exams until they know what kinds of questions will be asked. In any case, the Chinese company has been banned from submitting results to the ILSVRC for 12 months as punishment for attempting to game the system.
[Source: ILSVRC]
Follow us on Instagram, Facebook, Twitter or Telegram for more updates and breaking news.
