Nah, the current system is not fair, unless I make bets only for winning this competition.
I am not talking about getting rid of 5%, I am suggesting why not get rid of the entire bankroll thing?
My problem is I don't have a fixed bankroll. Nor is my bets have a reliable average. If I find a very good pick, my bet could be 100x as much as my normal one (don't go beyond 1 BTC, I can't afford that much, lol), so I can't.
Also, I was under the impression that betting twice on the same thing doesn't count,
but it is being counted. not sure.

Betting on the same thing does not count, if you are seeing that please send the example on message, or post it here and we will see if there is any problem with that.
As for your system, let me explain why we can't remove the bankroll.
This was your example:
This one:
Bet = 0.01 Payout = 0
Bet = 0.1 Payout = 0.12
ROI = 0.12/0.11 = 9.09%
I will set another example with your system, to try to explain the problem with it:Lets say you make 100 bets of 0.01 and lost all of them
Bet = 0.01 x (100 bets) Payout = 0
By this time you already lost 1 BTC, and you are in the final day of the competition, and you decide to do this:
Bet = 2 BTC @ 3 Payout = 6
ROI = 6/3 = 100%
You lost 100 bets, and you win one bet and get 100% ROI.
Do you think this is fair?
With the current system, and the same example, this is what you get:2BTC is 5% of your bankroll. So your bank is 40BTC
0.01 is 0.025% of your bankroll
So you lost 0.025% * 100 = 2.5% of your total bankroll with those 100 bets
You win 6 BTC with your 5% bet so you get 15% profit
So in the end: 15-2.5 = 12.5% ROI
That seems like a fair ammount and equal to everyone. If we don't have a bank, you can "delete" your entire progress with one bet, so there is no point in doing a competition, we could just award prizes for one single bet.