It is a lovely design. As a technical trainer for a Railcar builder I will show this around to my friends. You are probably breaking Federal Law but......
I see you have a double flange on the "rail wheels" which is probably causing someof your friction. Rail wheels are not flat or should not be flat they have a taper which allows them to make curves, the rail is not flat either.
Two words of caution, first there is not such thing as dead track so I would be really careful where I ride that thing, since you really cannot hear a train coming. Second, I would be very careful with my speed, wheels will tend to "hunt" which is a back and forth movement. This movement can become fairly violent quickly and on a vehicle like yours toss you off the bike.
I love this contraption and you can see some really nice scenery along right of ways.