HooooWeeee this is a doozy!
Tell you what, this would be a LOT easier if they let you score this type of question.
But where there's a will there's a way. Saddle up!
first, you need to present your card decks. I'm not exactly sure what your standards of presentation are. this is what I did for testing, I’m doing 2 decks 6 cards each:

If you want to upload an image of all the cards, you can do that in 'rich content editor'. If you need help with going the picture route let me know. Notice the page break between so they only see one deck at one time, if that’s what you want.
Next – add all your cards to the pick group and rank question. Like so:

All 40 of them.
Then click the gear to the left, and pick randomizer, and then advanced randomizer.


From here, use the arrow to move the options, all of them, to the ‘random subset’. Then, randomly insert however many you mean to make part of your quiz. I did 6 for testing.

Click save when done. You will need to copy this question and put it into a new block for the 2nd and 3rd memory trial. **This question gets its own block** you will see why in the survey flow, and no, you cannot duplicate the block in the survey flow as a shortcut. (it remembers the randomization, AND the previous answers are as they were left the first time.)
This is the most tedious part but you’re almost there.
In the survey flow, because pick rank group questions don’t have scoring, we have to make our own. ((**ProTip - The Duplicate button is your best friend here **))
Add a branch that checks if your first card, from deck one, went to deck one. Underneath, make an embedded data variable = 1. (if they got it right, score is 1)
Then, add a branch under that which checks if the second card from deck one went to the right group. Now you reassign your scoring variable using math syntax e.g. $e{ e://Field/firstTryScore + 1 }
It should look like this:

ALL.40.TIMES. sorry about that.
BUT, we have custom scored the question! Now we use that score to branch into another trial if their score was too low. Make a branch, choose embedded data, chose your scoring variable, and put the logic in. I chose 5 as my cutoff, and it should look like this:

Now it is a matter of rinse and repeat. Change the scoring variable name, that’ll help with your data collection. I did firstTryScore, and then secondTryScore and so on.
At the end of the third try, just put an end of survey tag. Like this:
There may be an easier way using javaScript if you can figure out how to grab the variables and test if they're in the right group. BUT this will do what you described.
I hope that helps!
-BUTCHHENRY

Wow! Thank you! Really creative thinking! Liked your idea.
As you said, a javaScript solution might be easier and parsimony would be a real plus. But this certainly solve my problem! Thank you very much
(and still, if somebody here know how to "grab the variables", that would be great!