How Do I Tell My Real Estate Agent I No Longer Want To Work With Him?

I have recently started to work with a real estate agent who is also a neighbor and who I am going to use to sell my house once I buy a new one. But I found out that bringing him into my home search has not really helped me and in fact has hurt me. The houses I liked most so far are the ones that I found myself without his help. And the price that the builders ask for the houses change greatly if you have an agent, so I have realized that just by not having him, I can start out with prices that are at least 5% lower. Now how can I tell him that I no longer want him to help me find a new house but will still want to use him for selling my current home once I am ready for it? I never signed anything, can I get out of this without any legal problem? I like the guy and I am afraid he will hate me after this.