Phone : +91 95 8290 7788 | Email : sales@itmonteur.net

Register & Request Quote | Submit Support Ticket

Home » Cyber Security News » Apple researchers question AI’s reasoning ability in mathematics, ET CISO

Apple researchers question AI’s reasoning ability in mathematics, ET CISO

Apple researchers question AI’s reasoning ability in mathematics, ET CISO


A team of Apple researchers has questioned the formal reasoning capabilities of large language models (LLMs), particularly in mathematics.

They found that LLMs exhibit noticeable variance when responding to different instantiations of the same question.

Literature suggests that the reasoning process in LLMs is probabilistic pattern-matching rather than formal reasoning.

Although LLMs can match more abstract reasoning patterns, they fall short of true logical reasoning. Small changes in input tokens can drastically alter model outputs, indicating a strong token bias and suggesting that these models are highly sensitive and fragile.

“Additionally, in tasks requiring the correct selection of multiple tokens, the probability of arriving at an accurate answer decreases exponentially with the number of tokens or steps involved, underscoring their inherent unreliability in complex reasoning scenarios,” said Apple researchers in their paper titled “GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models.”

The ‘GSM8K’ benchmark is widely used to assess the mathematical reasoning of models on grade-school level questions.

While the performance of LLMs on GSM8K has significantly improved in recent years, it remains unclear whether their mathematical reasoning capabilities have genuinely advanced, raising questions about the reliability of the reported metrics.

To address these concerns, the researchers conducted a large-scale study on several state-of-the-art open and closed models.

“To overcome the limitations of existing evaluations, we introduce GSM-Symbolic, an improved benchmark created from symbolic templates that allow for the generation of a diverse set of questions,” the authors wrote.

GSM-Symbolic enables more controllable evaluations, providing key insights and more reliable metrics for measuring the reasoning capabilities of models.

“Our findings reveal that LLMs exhibit noticeable variance when responding to different instantiations of the same question,” said researchers, adding that overall, “our work provides a more nuanced understanding of LLMs’ capabilities and limitations in mathematical reasoning”.

  • Published On Oct 12, 2024 at 11:26 AM IST

Join the community of 2M+ industry professionals

Subscribe to our newsletter to get latest insights & analysis.

Download ETCISO App

  • Get Realtime updates
  • Save your favourite articles


Scan to download App

Information Security - InfoSec - Cyber Security - Firewall Providers Company in India

 

 

 

 

 

 

 

 

 

 

 

 

What is Firewall? A Firewall is a network security device that monitors and filters incoming and outgoing network traffic based on an organization's previously established security policies. At its most basic, a firewall is essentially the barrier that sits between a private internal network and the public Internet.

 

Secure your network at the gateway against threats such as intrusions, Viruses, Spyware, Worms, Trojans, Adware, Keyloggers, Malicious Mobile Code (MMC), and other dangerous applications for total protection in a convenient, affordable subscription-based service. Modern threats like web-based malware attacks, targeted attacks, application-layer attacks, and more have had a significantly negative effect on the threat landscape. In fact, more than 80% of all new malware and intrusion attempts are exploiting weaknesses in applications, as opposed to weaknesses in networking components and services. Stateful firewalls with simple packet filtering capabilities were efficient blocking unwanted applications as most applications met the port-protocol expectations. Administrators could promptly prevent an unsafe application from being accessed by users by blocking the associated ports and protocols.

 

Firewall Firm is an IT Monteur Firewall Company provides Managed Firewall Support, Firewall providers , Firewall Security Service Provider, Network Security Services, Firewall Solutions India , New Delhi - India's capital territory , Mumbai - Bombay , Kolkata - Calcutta , Chennai - Madras , Bangaluru - Bangalore , Bhubaneswar, Ahmedabad, Hyderabad, Pune, Surat, Jaipur, Firewall Service Providers in India

Sales Number : +91 95 8290 7788 | Support Number : +91 94 8585 7788
Sales Email : sales@itmonteur.net | Support Email : support@itmonteur.net

Register & Request Quote | Submit Support Ticket