My khmer capacity

Sometimes, I am worried about my Khmer capacity.
I studied the basic of Khmer language already in Japan and now I stay in Cambodia.
Everyday I talk with peole in cambodian.
They look glad that I speak khmer.
I can hear people say, read and write some khmer language.
However of course my capacity is not enough at all.

For the development of my capacity, how should I do next?
For learning it more, should I memorize the unknown words in my note as before?
I feel this action will be an obstacle to the normal talk.

But If I don't do anything, will this poor condition be still ?
My some khmer friends know I study khmer language, and talk to me normally.
I feel happy if they think so because they put confidence in my capacity.
However, sometimes or more, I can't understand the meaning.
For hiding my poor capacity, sometimes I make the smile face for them baecause
I am afraid my attitude will take all the fun out of the party.
I know it is not good way for the advance, although I try to listen to the conversation and seek the plot.
I feel uneasy.
If we have the another purpose, it may be not necessary to learn khmer more and I may be also, but really I want to be good at using khmer.
I don't want to use English as possible.

Please tell me, if you have a good way to learning the foreign language.