Where do I go from here...?

4 years is a long time. Be careful not to just throw it away because of forgotten dates, you'd probably regret it in the long run... The fact that he's forgotten about so many things lately doesn't necessarily mean that he doesn't care, he might just be going through a point in his life where he has a lot of other things to deal with on his mind. You could talk to him about how times like these are important to you, and exclaim that you don't forget his birthday etc. If you're falling out of love with him, there must be a bigger reason. Do what your heart tells you, but don't jump to conclusions - again, 4 years is a lot to throw away.