First off, let me say that I think that situation (the players try something and the GM knows more than they do) can be a lot of fun, so I'm approaching it from that perspective.
Players try something that isn't a total shot in the dark, but isn't fully known or understood either. Let's say it's a magical ritual to summon a demon. They're trying to summon a demon with tunneling powers to get them under or through an enemy fortress wall.
Dice pool time!
For every part of this process that the players/characters absolutely do know, they place a die on the table. We know summonings require a magical power source, so we take out our magic crystal.
For every part of this process that the players/characters think they might know, but haven't proven, they hand a die to the GM. The demon summonings we've seen involved human sacrifice, so we're going to guess that's necessary too and kill one of our captives. The GM then secretly places that die in one of two hidden piles -- the "yes, that's how it works" pile, and the "no, it doesn't work like that" pile. If the GM hasn't yet established how it works, now's the time to decide or roll or look it up.
For every part of this process that the players/characters are fully aware that they're clueless about, the GM takes a die whenever it comes up. Well, we don't know how to indicate "tunneling", so I guess we'll dig a little under the ritual spot (GM takes a die) and draw a circle in dug-up earth (GM takes a die) and hope that sends some sort of signal. The players will expect these dice to go into the "no" pile, but still cross their fingers that they've guessed luckily or well and that some will end up in the "yes" pile.
If the GM notices anything else that isn't called out by the players but might impact their endeavor, the GM grabs another die. So, we stand in a circle, I guess, spread out evenly around the crystal, and- (GM grabs a die) oh, I guess that's a relevant part of the ritual too, huh? Or at least it might be.
Dice rolling time!
If doing it right means that it works, then no need to roll the players' table dice and the GM's "yes" dice. Just bring the "yes" dice out from behind the screen and add them to the players' dice, and total them up. The more the better. 12 dice of doing this spell correctly! That's enough to summon our demon, right? Yep!
If doing it right means only that it has a chance to work, then the GM rolls the "yes" dice in secret, pushing forward only the "successes" (rolls of 4-6 on a d6). The players roll their dice too and all successes are tallied. 6 successes on 12 dice of doing this spell correctly! That's a great effort, but only average luck. Enough to summon our demon?
Here's the nifty part: doing it wrong. The GM rolls the "no" dice in secret. For every success, the GM has to tell the players something that updates their understanding of this endeavor. For 4s and 5s on a d6, the level of clarity is flexible. GM: try to give them something they can work with, but don't just tell them "here's the right way to do it". If you're a little too obvious or a little too subtle, oh well, it's fine. For every 6 on a d6, though, the GM has to give the players something they can definitely use going forward. GM: narrate what happens, and if the players aren't clear on what it means, keep going until they are.
For every failure (1-3 on a d6) on a "no" die, the GM has to mess things up! 3s cause incidental damage to the characters and their stuff without ruining the endeavor itself; 2s partially ruin the endeavor itself; 1s do both. The more 1s and 3s, the greater the damage; the more 2s and 3s, the greater the ruining.
Note: "ruin the endeavor" does not mean "you didn't do enough right for it to work" -- that'd be the same as merely subtracting successes. "Ruin the endeavor" means that the task is completed just as successfully as it would have been otherwise, but the intent behind it meets opposition. For example: The tunneling demon appears and instantly begins tunneling through your target wall, while emitting a piercing shriek that can be heard for miles around. The demon then stands in the tunnel that it has created and continues to scream, blocking your path as guards rush in.
- If you stick to what you know, there are no surprises.
- If you branch out and take some very educated guesses, you probably improve the range of what you can accomplish, but you invite a little risk.
- If you start really speculating and experimenting, then you probably learn stuff for the future, and possibly increase the chance of a longshot working, but at massively increased risk of getting hurt or in trouble or losing your stuff.