About My Research

What is ARC?

Abstraction and Reasoning Corpus (ARC) is a benchmark dataset invented by François Chollet, to test the intelligence of artificial systems. Each task has averagely 2 - 5 example pairs and a test input. The solvers are to guess the rule that satisfies all the example pairs and apply test input to get the correct answer. Here, the picture on the right is an example ARC task. What grid would be in the “?” according to the rules found within the example pairs?

This task is simple. However, there are various types of tasks and it is not easy for a machine to solve it. It is known that human beings can solve it around 85% of the time, while machines can only solve it around 42%, according to the ARC-AGI Leaderboard.

Here is one additional note for you if you are interested in solving ARC tasks. Please visit o2arc.com. This is a web application created by our lab, where you can see and solve other ARC tasks. You can also create your own tasks!

System-1 and System-2 Thinking

Have you ever heard of System-1 and System-2 thinking?

They are two different ways of thinking. System-1 is fast and subconscious, while System-2 is slow and conscious. Due to the characteristics of the two systems, System-1 thinking produces more intuitive fast responses, while System-2 thinking produces more rational and logical responses. Neither is better or worse, they are just different. The point is humans are able to switch between them flexibly!

Currently, most of the AI systems provide pleasing performance on System-1 thinking. They are good at solving tasks that does not require logical processing. The limitation of current AI systems is well-known with the black box issue of the inference process, and lack of explainability. Moreover, considering LLMs as an AI system, they suffer from hallucination. We can see these issues when they are asked to solve tasks that require logical processing.

ARC is a task that requires System-2 thinking. By setting a short-term goal to solve ARC tasks, I am dedicated to develop a new type of AI systems that are able to do System-2 thinking. I am expecting so-called “a new module” to be plugged into the current AI systems and enable them to do System-2 thinking or further plays a role of left/right brain.

Research Motivation

To achieve the goal of System-2 thinking by solving ARC, I first pondered on how human beings solve ARC tasks. Not only in shallow and intuitive level, but also in deep and logical level by repeatedly asking “WHY” questions to my solution. Eventually, the problem solving process has divided into stages and it became the motivation of my research direction and the background of Abductive Symbolic Solver on Abstraction and Reasoning Corpus.

Steps of Human Solving ARC

I defined 6 steps of human solving ARC tasks.

1. Observation
2. Pattern, Change, Difference Recognition
3. Rule Formulation + Adjustment

4. Repeat

5. Observation (Test)
6. Application

The above process can be described in words as follows: We observe a single grid in the ARC task, then we recognize the patterns. Using the information, we formulate a rule. Then the process repeats until we observe all the grids provided in the example pairs. During the repetition, we may adjust the rule. Finally, we observe the test input and apply the rule appropriately to get the correct answer.

After formulating these steps, I focused on the knowledges we gain during the process and how we synthesize them to conclude the rule. Even though only a few example pairs are provided, humans can formulate the rule that satisfies all the example pairs. This brought me to the idea of abductive reasoning.

Abductive Reasoning

Abductive reasoning is a reasoning method to find the most plausible explanation within a limited number of observations. It is similar to inductive reasoning that draws general conclusions from specific instances. However, the difference is that abductive reasoning is under the constraint of limited observations while inductive reasoning is based on a large amount of observations that is enough to generalize a conclusion.

Within tasks that requires object selection, the abductive reasoning process can gain information of object’s property and relation which can be used as a constraint to find the most plausible object. Here, the accumulation of the property and relation information is done by so-called symbolic back-propagation.

By preserving the semantic meaning of the knowledges and applying abductive reasoning, this symbolic system appeared to narrow down the conditions for selecting the object in the test input. This architecture achieved a knowledge accumulation within the small number of examples which is the key to solve the task.

Research Interests

Above work is the inital idea of my research, so it may look raw. However, I believe the approach is reasonable and I am dedicated to develop it into a more robust system. The following contents are the research interests that I am currently focusing on and getting ready to be developed.