One of the research field’s key origin stories took place in the North American wilderness – and, be warned, it makes for some stomach-churning reading. The year was 1822 and a young trader named Alexis St Martin was loitering outside a trading post on what is now called Mackinac Island, in what is now Michigan, when a musket accidentally went off next to him, firing a shot into his side from less than a yard (91cm) away. His injuries were so bad that part of his lungs, part of his stomach and a good portion of his breakfast that day spilled out through the wound in his left side. Death seemed certain, but an army surgeon named William Beaumont rode to the rescue and saved St Martin’s life, although it took the best part of a year and multiple rounds of surgery.

What Beaumont couldn’t repair, however, was the hole in his patient’s stomach. This persistent fistula would remain a grim and lasting legacy of the accident, but Beaumont wasn’t one to pass up a good opportunity – however unpleasant. Realising that the hole provided a unique window into the human gut, he spent years investigating the intricacies of St Martin’s digestion. Exactly how willing a volunteer St Martin was is open to debate as Beaumont employed him as a servant while conducting research on him – the murky arrangement almost certainly wouldn’t be considered ethical today. Among the findings Beaumont uncovered during his studies of St Martin’s guts, however, included how they were affected by its owner’s emotions, such as anger.

Through this finding, Beaumont, who would go on to be lauded as the “father of gastric physiology“, had hit upon the idea of a “gut-brain axis” – that the gut and the brain aren’t entirely independent of one another but instead interact, with one influencing the other and vice versa. And now we know that the microorganisms within our gut make this process even more complex and remarkable.

“More and more research is revealing that the gut microbiome can influence the brain and behaviour across a variety of different animals,” says Elaine Hsiao, associate professor in integrative biology and physiology, at the University of California, Los Angeles (UCLA).

How exactly our microbiota might be influencing our mind is a growing, pioneering and still relatively novel field. But there have been advances over the last 20 years or so, particularly in animals. And, slowly, a case is being built to suggest that these microorganisms aren’t just a vital part of our physical selves, but also our mental and emotional selves, too.

“In medicine, we tend to compartmentalise the body,” says Cryan. “So, when we talk about issues with the brain, we tend to think about the neck upwards. But we need to frame things evolutionarily. It’s important to remember that the microbes were here before humans existed, so we have evolved with these ‘friends with benefits’. There has never been a time when the brain existed without the signals coming from the microbes.

“What if these signals are actually really important in determining how we feel, how we behave and how we act? And could we modulate these microbes therapeutically to improve thinking, behaviour and brain health?”

You might also like to read:

• Why your microbes like a good workout
• How language warps the way you perceive time
• The hidden way pollution affects us

Hsiao is one of the researchers leading the way in this field and her lab at UCLA has explored the part these microorganisms might play in everything from foetal brain development to cognition and neurological conditions such as epilepsy and depression. She has also investigated how these microbes might be influencing our brains and thinking.

“Specific gut microbes can modulate the immune system in ways that impact the brain and also produce molecules that signal directly to neurons to regulate their activity,” she says. “We find that gut microbes can regulate the early development of neurons in ways that lead to lasting impacts on brain circuits and behaviours. We also find that under shorter timescales, gut microbes can regulate the production of biochemicals, like serotonin, that actively stimulate neuronal activity.”

Indeed, research suggests our microbes may be communicating with our brains through numerous pathways, from immunity to biochemicals. Another candidate is the vagus nerve, which acts as the superfast “internet connection” between our brain and internal organs, including the gut. The bacteria Lactobacillus rhamnosus JB1, for example, appears to improve the mood of anxious and depressed mice. This beneficial effect is removed, however, when the signals travelling along the vagus nerve are blocked, suggesting it could be being used as a communication pathway by the bacteria.

Much of the research in this field is conducted in mice (and other small animals). And mice, of course, aren’t humans. But given the mindboggling complexities of establishing causality between microbial signals and changes in human thought and behaviour, animal studies have provided some intriguing insights into the strange interactions between bacteria and brain. Research, for example, shows that “germ-free” rats and mice (those without any microbiota after being reared in sterile conditions) are more prone to anxiousness, and less sociable than those with an intact microbiota. Germ-free mice, and those given antibiotics have also been found to be more hyperactive, prone to risky behaviour and less able to learn or remember. Antibiotics, which can reduce the microbiota in an animal, also reduce shoaling behaviour in zebrafish, while probiotics boost it.

Again, the human brain is vastly more complex than that of a rodent or fish, but they do share some similarities and can offer clues. It makes sense that bacteria, wherever they live, might benefit from helping their hosts to be more sociable and less anxious. By interacting with other people, for example, we help our bacteria spread. And whether or not they’re really pulling our strings, it’s in our microbes’ evolutionary interests to make their environment as conducive to survival as possible.

But do communicative microbes, congregating zebrafish or friendly mice really matter? Hopefully, yes, say the researchers. Ultimately, a better understanding of these processes could lead us towards ground-breaking new treatments for a range of mental health conditions.

“We’ve coined the term ‘psychobiotics‘ for [microbiota-based] interventions that have a beneficial effect on the human brain,” says Cryan. “And there are more and more of these psychobiotic approaches coming.”

There are caveats, of course. While some strains of bacteria appear to have a positive effect on the human mind, many others don’t and researchers have yet definitively to establish why – and how. Humans are also unfathomably complex, and when it comes to thinking and mental health, there are countless other factors at play, from genetics and personality to the environment around us.

“We need many more large-scale human studies to take into account these individual differences,” says Cryan. “And maybe not everyone will respond to a single bacteria in the same way because everyone will have a slightly different baseline microbiota anyway.”

Disclaimers aside, however, more research could bring fresh hope. “The good news is that you can change your microbiota, while there’s not a whole lot you can do to change your genetics – except blame your parents and your grandparents,” Cryan adds. “The fact that you can modify your microbiota potentially gives you agency over your own health outcomes.” Indeed, pro- and prebiotic supplements, simple dietary changes, such as eating more fermented foods and fibre – and even, perhaps, meditation – can help alter our microbiota in ways that benefit our minds.

Philip Burnet, an associate professor in the University of Oxford’s department of psychiatry, notes that many mental health conditions have been associated with changes in the microbiota. Often, this imbalance or “dysbiosis” is characterised by a reduced amount of certain bacteria, particularly those that produce short-chain fatty acids (such as butyrate, which is widely believed to improve brain function) when they break down fibre in the gut.

Indeed, a 2019 study by Mireia Valles-Colomer, a microbiologist at KU Leuven University of Leuven in Belgium at the time, and her colleagues found a correlation between the amount of these butyrate-producing bacteria and wellbeing. Specifically, the researchers noted in the study that: “Butyrate-producing Faecalibacterium and Coprococcus bacteria were consistently associated with higher quality of life indicators. Together with Dialister, Coprococcus spp. were also depleted in depression, even after correcting for the confounding effects of antidepressants.”

Human studies on the communication between the gut, the brain and the microbiota are still relatively few and far between. And Burnet urges caution: “It is not known whether these altered levels in gut bacteria cause low mood or whether microbial numbers change because people who are depressed might modify their eating habits or eat less.”

Nevertheless, he has been exploring how prebiotics (which encourage bacteria to grow) and probiotics (live bacteria) might one day be used as psychobiotics to nurture populations of “good” bacteria – and treat a variety of mental health conditions.

For example, one 2019 study by Burnet, Rita Baião, a psychologist also at the University of Oxford, and their colleagues uncovered some particularly interesting findings. Although the study was funded by a company that manufactures probiotic bacteria, it used a randomised, double-blind, placebo-controlled trial – considered to be a gold standard study design during which neither participants nor researchers are aware whether they are receiving the treatment or not. The researchers investigated the effect a multispecies probiotic might have on emotional processing and cognition in people with mild to moderate depression. But the study also monitored their mood before and after the experiment using the Patient Health Questionnaire-9 (PHQ-9), which measures depression severity.

The participants, who weren’t taking any other medication, were either given a placebo or a commercially available probiotic – which contained 14 species of bacteria, including Bacillus subtilis, Bifidobacterium bifidum, Bifidobacterium breve and Bifidobacterium infantis  for four weeks.

The results were fascinating, not least that the participants on the probiotic experienced a significant subjective improvement in mood compared with the group on placebo, essentially becoming less depressed according to the PHQ-9. Changes in the participants’ levels of anxiety, which were also measured, were not observed.

This was a small (71 participants), brief study and more research is needed to prove causality. But it’s an early indication that “psychobiotics” may one day be a helpful treatment for those with depression – particularly those who are reluctant to seek medical help or take traditional antidepressants, says Burnet. Indeed, psychobiotics won’t replace existing medications – but may eventually provide a helpful adjunct.

“They won’t make everyone happier,” says Burnet, but probiotics could one day complement more established mental health treatments. “Only time will tell whether we will have psychobiotics,” he adds. “But the field is really moving forward… This area of research is dominated by animal studies, however, so we do need more human studies using larger numbers of participants.”

But the potential of psychobiotics has captured imaginations.

“We also attracted a lot of interest from the public,” adds Burnet. “People are extremely interested in maintaining their health and wellbeing with natural supplements and encouraging the growth of good bacteria to support mental health has captured the imagination of the general public. Especially now, when people are more anxious and depressed as a result of the pandemic.”

With Amy Chia-Ching Kao and others, Burnet has also explored the role these microorganisms might play in psychosis – and whether prebiotics (which help to promote the growth of bacteria in the gut) might help people with the condition think more clearly.

Many people are aware that psychosis can cause hallucinations, delusions and a detachment from reality. But people with psychosis also often encounter difficulties with cognitive functions such as attention, memory and problem-solving, which can impact their ability to hold down jobs and relationships. While medication can be used to treat the hallucinations and delusions, improving sufferers’ cognitive impairments has proved more difficult.

A double-blind, placebo-controlled crossover study by Burnet and Chia-Ching Kao, however, suggests a possible way forward. “We found that giving a prebiotic to people with psychosis did improve cognitive function according to the clinical scales,” says Burnet.

At the start of the study, the participants were on medication and free of psychotic symptoms – but were still experiencing the cognitive impairment typical of psychosis. Over 12 weeks, they were given a prebiotic or a placebo while their metabolism, immunity and level of cognitive impairment was measured over time. At the end of the 12 weeks, they were then switched, so both groups had an equal amount of time on the prebiotic and the placebo.

And the effect was small but significant. The prebiotic improved overall cognitive function, particularly attention and problem solving, leading the researchers to conclude that the improvement was sufficient to boost social and mental wellbeing. There was no evidence of the participants’ immunity or metabolism changing so it wasn’t clear how the prebiotic may have triggered this effect. But it’s another small step towards understanding the relationship between our microbiota and our mental health and the potential development of new treatments for disorders that affect our thinking.

There are hints that the gut microbiota may affect cognitive skills more broadly, too. It is well known that antibiotics disrupt the gut microbiota, but do they affect our cognition?  One recent study, which monitored the health and wellbeing of 14,542 female nurses for several years while they worked for the NHS in the UK, found that those who had used antibiotics for long periods of time (more than two months) scored lower on cognitive tests such as learning, working memory and attention tasks than those who hadn’t taken such medication. Importantly, the cognition of the women who had taken antibiotics was slightly poorer when they were followed up seven years later. Although this is only a correlation, the researchers think it could be due to antibiotics-induced changes in the gut.

There’s still a very long way to go, though, to understand this properly. This is a fascinating but highly complex field, and research requires funding. The rewards, however, could be profound. “There are only a handful of specific microbes that have been studied so far,” says Hsiao. “Not necessarily because they are the most significant, but because we as scientists have much more to do to really understand the enormous diversity of microbes in the gut and how they function individually and as communities.

“I’m most excited about the opportunity to uncover new mechanistic understanding of how we and our microbial symbionts can work together to promote health and thwart disease.”

In the meantime, perhaps we should all pay a bit more attention to our microbiota. A Mediterranean diet that’s high in fibre, particularly from vegetables, is likely a good place to start. And fermented foods, such as kimchi and kefir (a fermented milk drink) may also be beneficial. In a small study with 45 participants, for example, Cryan and colleagues showed that those who were put on a diet that included a lot of fibre, prebiotics and fermented food (such as onions, yoghurt, kefir and sauerkraut), reported feeling less stressed than a control group which was on a different diet.

“What I like about fermented foods is that they democratise the science,” says Cryan. “They don’t really cost much and you don’t have to get them from some fancy store. You can do it yourself. In this field, we want to provide mental health solutions to people from all socioeconomic areas.”

The relationship we have with our microbiota is “a bit like a federation”, adds Cryan. “These microbes are our fellow travellers.” We’d do well to remember that – for the sake of both our physical and, quite possibly, mental health.

* Are You Thinking Clearly? 29 Reasons You Aren’t, And What To Do About It by Miriam Frankel and Matt Warren is published by Hodder Studio.

If you liked this story, sign up for the weekly bbc.com features newsletter, called “The Essential List” – a handpicked selection of stories from BBC Future, Culture, Worklife, Travel and Reel delivered to your inbox every Friday.