You should also limit the number of plugins you are loading. People want content before features. Add the features as you grow. Members will want to see new features added, especially when the features are the ones they are asking for...this not only shows growth and development but also displays your willingness to listen to your users.
As @lloydpearsoniv stated, a better server will help immensely.