am i playing a videogame or a math exam..?

Real calculus. Wasn't quite expecting that. There was one problem that I think had an error, it was something like this:

Lim x->6
(sqrt(x+3) + 3 )/(x-6)

I didn't run the calculations or graph it, but it seems like it must be infinity or DNE. Given answers were +/- 6^(+/- 1).

Non-math related feedback: Make the movement faster since it just wastes time having the characters walk slow. Actually, walking doesn't even matter, as far as I could tell. Just have a spacebar advance to the next area or a simple tap of the up arrow. With that change, I'd rate it 2.5/5. Getting higher would require some hints or explanations about how to solve problems that were missed. This is posted to the general public and right now it's not suited for that. Also, the pencil animations are cool and everything, but they can't be dodged or anything, so it's making the game unnecessarily complicated.

Edit: Should rename it to be "limit attack" or expand the game to other categories. I didn't get past the first boss, so if it is more comprehensive, then sorry about this comment. Although, if there are other bosses, it might be neat to let the player choose which boss to fight first.

And it looks like Rallyx agrees with my claim that one question has only incorrect answers. His link to wolfram alpha is easier to understand than my plaintext.

Uhh… two questions here:
a) Who is your target audience?
b) Is this a game, or an exam?

Because call me a masochist, but I finished the game. I was curious to see the ending, but it's not really worth it. It's just as underwhelming as the rest of the dialogues.
"You mastered calculus!" "You win!"

As for question a) – judging from my experience, you need to be in a senior year of high school or start attending university to be taught limits, derivatives and integrals and enrol in an advanced math course to know how to solve differential equations (not to mention you need to remember the process well enough to apply it here).
That's a very, very small target audience which means most people will either close the game straight away or just go with trial & error and memorize the correct answers.

For the above reasons, you're very unlikely to see any advanced math to be a requirement to complete a game. I don't even recall seeing a simple linear equation in a game before.

----

As for (b) – at first, I was quite intrigued by seeing limits. If you're fine with having a tiny target audience, you could make something unique, because as far as I'm aware there's not a game ‘for mathematicians’ like this. HOWEVER, my main gripe with the game is that solving math problems is really all there is. You could remove everything and keep the format just:
"[Math problem]. Pick solution: a) b) c) d)"
and the game would stay the same. At which point, it becomes an exam. You don't really "fight a boss", you "click on an answer".

You have to make it enjoyable. Why doesn't the player go to different rooms, collect items and solve ONE math problem for example to open a lock? Why doesn't he move around while fighting the boss?
Or solve a math problem to get extra lives.

Basically, it's an interesting concept, but the game needs content. It needs to keep going, let you do something enjoyable – not to make you stare at an equation for a few minutes.
As for the difficulty, I draw the line at being able to calculate the result in my head, except maybe some rare exceptions at the end of the game or during especially tough parts of it.
Derivatives are fun and quick (if you know how to calculate them to start with), limits need you to understand the principles but still can be done but the moment you ask me to calculate an integral of a fraction or differential equations (!), I'm going to phone it in with Wolfram. Or just guess and potentially restart the game a few times.

And there are a few bugs/mistakes, too:
– The play again bug which you seem to be aware of
– Using keyboard keys moves the page around. I think you need to "preventDefault" of the keyboard keys you're using. See here:
https://www.reddit.com/r/gamemaker/comments/3fnayp/html5_game_causes_webpage_to_scroll_when_arrow/
– Wolfram thinks the answer to one of the limits is wrong:
https://www.wolframalpha.com/input/?i=lim(x-%3E6)+(((x%2B3)%5E(1%2F2)+%2B+3)+%2F+(x-6))

Again, I like the concept and hope you'll build upon the idea. It just needs to be less of an exam and more of a game.