I think you have a pretty good plan already. My advice would be to put a number of people through an 80% max rep test to help determine their dominant fiber type. i.e. if they do less than 8, put them in the fast twitch group, greater than 10 put them in the slow twitch group, or just weed out everyone but the fast twitches, and concentrate only on them. You could then do your test as planned, which is what I would suggest. By staying above 80% with your rep tests, you should see a linear correlation, where reps increase as %max decreases. You can just plug all your numbers in excel to get a best fit line, and it will give you a linear equation.
As I said, you could do this just for the fast twitch types, or you could come up with an equation for fast twitch type, slow twitch, and everybody in between.
If you wanted to estimate max based on a higher rep test, things would start to get difficult as it definitely would not be linear.
Be sure to keep in mind that you will get different results for different lifts.
Hope this all makes sense.