Drinking just a couple beers throughout the week may be hurting your health more than you think: Even moderate drinking can damage your brain, new research from the University of Oxford in the U.K. suggests.

In the study, researchers recruited 550 people, tracked their weekly alcohol intake, and measured their performance on brain tasks over a 30-year period. Then, they performed MRIs on them to analyze their brain structure.

They discovered that people who drank moderately—14 to 21 units of alcohol per week—were three times as likely to have deterioration in the right side of their hippocampus than those who abstained from alcohol. That’s important, since that area of the brain affects your memory and spatial navigational skills. Atrophy there is also a marker of Alzheimer’s disease, the researchers say.

Here’s what that means for your pub visits: Each unit contains eight grams of alcohol, and a standard drink is roughly 14 grams of alcohol. That’s 12 ounces of regular, 5 percent alcohol beer, or 5 ounces of 12 percent alcohol wine. So that means drinking between 8 and 12 beers a week would put you in that risk category.

People who drank more—at least 17 beers a week—showed even more deterioration in that brain region. They were nearly six times as likely to show atrophy in their hippocampus than those who didn’t drink at all.

Interestingly, there was no link between light drinkers and brain deterioration, showing that there doesn’t seem to be any brain benefit to drinking just a little bit.

Plus, the more alcohol people drank, the poorer their density of white matter—a part of the brain necessary for cognitive function—and the greater their decline in specific language skills.

Alcohol and its byproducts are toxic to nerve cells in your brain, and can cause structural damage in important areas of your brain like the hippocampus that are used in daily mental functions, the researchers say.

This suggests that even drinking within the recommended alcohol guidelines—no more than two drinks a day for men—may still contribute to some brain problems. In fact, here’s how drinking just a little booze each day may be killing you