Gdddk

If I remember correctly, there was never a male who could change the water and become the male version of the reverend mother. How did the Bene Gesserit know what they needed to do in order to make a male reverend mother, having never created one?

They didn't. They're actually pretty clear about that throughout the text, and it comes up again, in different ways, later in the series, when Leto II is enforcing his Golden Path: they are not engaging in specific actions expecting a guarantee of a specific outcome. "Any road followed precisely to its end leads precisely nowhere."

Instead, they're engaging in a long-term scientific project in which they experiment with mixing genes, and attempt to judge the potential of what has resulted, all without relying on computers or other sophisticated means, and while eschewing artificial insemination to attempt to produce the desired outcome, and then moving incrementally forward. They know that some of the traits they desire already exist in the human genome. Guild Navigators are almost always gendered masculine, for example, and had a measure of prescience, without having Other Memory. They reasoned it ought to be possible to create one male person who combined multiple powers in one genome, and then started breeding people who showed signs of those traits in hopes of eventually getting there.

In Dune, they believed, but did not know, that if they could match an Atreides daughter with a Harkonnen son (Feyd-Rautha), the results would either be what they hoped for, or very close to it. They were willing to risk an incestuously close pairing (since of course Lady Jessica is herself genetically Harkonnen, so pairing her daughter with her cousin carries other dangers) to achieve it, and then are of course angry when they're experiment is thwarted by Jessica's decision to bear a son (Paul) instead.

But even then, they didn't know it would work, and of course, in the end, they actually miscalculated by a generation, anyway. Paul proved able, and the ability proved to be heritable. I believe there is actually a line where one of the Bene Gesserit--I want to say Reverend Mother Gaius Helen Mohiam--bemoans the consequences of their miscalculation while also admitting philosophically that to miss the mark by one out of hundreds of generations is actually pretty good odds, and was always a possibility. Similarly, when she admits to Paul that he might be the One, she also shrugs at the possibility he might not be. After waiting so long, what was one more generation, or even two?

Leto II took up this same philosophy when he took over the Bene Gesserit breeding program for his own purposes. He hoped to eventually breed a human being who could escape him, and thus, by extension, any other prescient searcher, by being invisible to prescient senses, while not themselves being an oracle. He didn't know when it would come about, but he knew that the ability existed in the human gene pool--Count Hasimir Fenring had no ability to see the future, but could not himself be seen. But Fenring had also been sterile, which would not do for Leto's purposes--he needed someone who could also pass the trait along, so that humanity would be safe from a malevolent prescient power in the future.