Logo image
Back
Utilizing language models in Markov decision processes to simulate performance gains in autonomous open systems
Thesis   Open access

Utilizing language models in Markov decision processes to simulate performance gains in autonomous open systems

Shahriyar Kabir
Master of Science (M.S.), Drexel University
Jun 2026
DOI:
https://doi.org/10.17918/00011400
pdf
Kabir_Shahriyar_20264.51 MBDownloadView
PDF Open Access Open Access (License Unspecified)

Abstract

Neurosciences Large language models Markov decision process Open systems Optimal foraging Phonological loops Working memory
This thesis presents a computational investigation into the functional role of biological memory constraints in autonomous agent behavior, using a Markov Decision Process (MDP) framework integrated with large language model (LLM) communication to simulate open-system organisms navigating a resource-uncertain grid environment. Across five progressively complex simulation versions, agents were equipped with incrementally realistic cognitive architectures -- from a basic Q-learning policy through biologically motivated long-term and short-term memory systems modeled after the phonological loop and serial position effect. The central hypothesis was that perfect and complete route memorization (Perfect Long Term Memory (LTM)) would outperform biologically constrained, memory-limited versions across all performance dimensions: organism survival cost, navigation path length, energy retention at goal-reaching, and food discovery per success. The hypothesis was tested using the most advanced version in three different implementations (Perfect LTM, Realistic LTM, Imperfect Short Term Memory(STM)), each sharing an identical environment and communication infrastructure but differing in how completely a successful route was inherited. Results partially supported the hypothesis: Perfect LTM achieved the lowest organism cost (mean 296.1 organisms for 10 successes; 4.16% success rate) and shortest navigation paths (mean 106 steps), confirming the efficiency advantage of complete memorization. However, the hypothesis was substantially refuted on the two dimensions most directly tied to biological viability. Memory-constrained versions arrived at the goal with 69-77% more energy than Perfect LTM (16.5 and 17.3 versus 9.8 units; both p < 0.001, Cohen's d > 1.2) and discovered more than twice as many food sources per success (2.27 and 2.25 versus 1.03 sources). The advantage of constrained versions grew stronger across successive successes, with Realistic LTM food discovery increasing 7.7-fold from Success 1 to Success 10 as inherited food-map knowledge compounded through exploratory navigation. The phonological loop capacity constraint in Imperfect STM, which truncated 44.1% of all agent communications, produced outcomes statistically indistinguishable from Realistic LTM at the aggregate level, indicating that the serial position effect on route memory is the primary driver of the explore-exploit tradeoff. We propose that biological memory constraints do not necessarily lead to performance deficits but can maintain productive and flexible environmental contact in open, resource-uncertain settings, with direct implications for the design of autonomous robots, swarm systems, and other AI agents deployed in dynamic real-world environments.

Metrics

1 Record Views

Details

Logo image