Intrinsic Goals for Autonomous Agents: Hierarchical Exploration Captures Future Volitional Action in Drosophila

Preprint

Declan Kutscher1,2,*, Reece Keller3,4,*, Alyn Kirsch4, Felix Pei3,2, Aran Nayebi2,3,4
1Department of Computer Science, University of Maryland, College Park
2Machine Learning Department, Carnegie Mellon University
3Neuroscience Institute, Carnegie Mellon University
4Robotics Institute, Carnegie Mellon University
*Equal contribution
Arxiv Code Cite

We asked whether intrinsic motivation alone can explain how flies generate spontaneous behavior and represent future actions. Across 20 intrinsic objectives in a biomechanical fly with no external reward, hierarchical skill discovery is the strongest joint match to fly behavior and neural signatures of future turning.

The fly appears to be maximizing state space coverage over the population not individuals via DIAYN, a form of curiosity!

Left: a real fly walking on a ball in the dark. Right: a trained DIAYN policy on the same kind of ball.

The training arena for our simulated fly agents has obstacles and multiple odor sources. The insets show each eye and the odor readout. The playback is at 0.24x speed.

Findings

01

Future actions become predictable

Similar to the fly's brain activity, neural activity in the intrinsically trained policies is predictive of future turn directions seconds in advance.

02

Skills reproduce behavioral diversity

Each point is a recording from a real fly or a model episode. The y-axis shows turning bias. DIAYN's repertoire of skills covers the distribution of the turning biases seen in the real fly population.

03

Internal geometry matches the IPS

In the real fly, the difference in activity between the left and right hemispheres of the IPS correlates with angular velocity, while the sum correlates with forward velocity. The same internal geometric relationship is present in the actor LSTM layer of the trained DIAYN agent.

Method

1

Train

We train PPO policies with reward from twenty intrinsic objectives in FlyGym.

2

Evaluate

Freeze the policy and place the fly on a ball in the dark, the virtual version of the real fly experiment.

3

Compare

Spontaneous turning, decoding of future turn direction, and low-dimensional internal structure, against real flies.

Fig. 1. The FlyGym simulator produces vision, odor and proprioception observations. They are encoded and then passed to the policy and IDM modules. The IDM produces a reward used for training PPO. The policy chooses one of seven motor primitives every 100 ms. A low-level controller turns that choice into joint commands.
Fig. 2. The agents are trained in an arena with obstacles and three odor sources. Evaluation is head-fixed walking on a ball treadmill. The observations are vision from the two compound eyes mapped to fly ommatidia, a bilateral odor signal, and proprioception.

Results

Future turns are readable before they happen

In the window from 15 to 8 seconds before a turn, the fly IPS predicts turn direction above chance. The same readout on frozen agents shows that only some intrinsic objectives develop this prospective signal. DIAYN is the objective that also keeps the fly's relationship between what the internal state predicts and what the behavior itself predicts.

Figure 3. Neural and behavioral prospective coding with future turn direction.

One objective covers the range of individual flies

DIAYN learns a set of skills. Held fixed, each skill is a different exploratory strategy. Plotted one rollout at a time, those strategies spread across left- and right-turning biases in the same way the flies do. Other objectives and random controls do not capture this variability.

Figure 4. Turning bias. Each point is one fly or one IDM episode from a tethered-ball recording. Positive values are right-turning, negative values left-turning. Panel b is the trajectories of the flies (gray) and the evaluated skills of DIAYN (blue)

Emergence of internal geometry of biological neural activity

In the fly IPS, the difference in activity between the left and right hemispheres correlates with angular velocity, while the sum correlates with forward velocity. The same relationship emerges in trained DIAYN and is absent in the random control.

Figure 5. Fly IPS, trained DIAYN, and random DIAYN. Color is angular velocity (top) and forward velocity (bottom). Last panel is the ranking of the mean Pearson r2 of each IDM's heatmaps with the Fly IPS.

BibTeX

@misc{kutscher2026intrinsic,
        title   = {Intrinsic Goals for Autonomous Agents: Hierarchical Exploration Captures Future Volitional Action in Drosophila},
        author  = {Declan Kutscher and Reece Keller and Alyn Kirsch and Felix Pei and Aran Nayebi},
        year    = {2026},
        eprint = {XXXX.XXXXX},
        archivePrefix = {arXiv},
        primaryClass={cs.XX},
          url = {xxxx}
      }