Srimad Bhagavatam Canto 11, Chapter 20, Text 19

SB 11.20.19

dharyamanam mano yarhi
 bhramyad asv anavasthitam
atandrito ’nurodhena
 margenatma-vasam nayet
Whenever the mind, being concentrated on the spiritual platform, is suddenly deviated from its spiritual position, one should carefully bring it under the control of the self by following the prescribed means.
Although one is seriously engaging the mind in Krsna consciousness, the mind is so flickering that it may suddenly be deviated from its spiritual position. One should then carefully bring the mind back under the control of the self. It is stated in Bhagavad-gita that if one is too austere or too sensuous one cannot control the mind. Sometimes one may bring the mind under control by allowing the material senses limited satisfaction. For example, although one may eat austerely, from time to time one may accept a reasonable amount of maha-prasadam, opulent foods offered to the temple Deities, so that the mind will not become disturbed. Similarly, one may occasionally relax with other transcendentalists through joking, swimming and so forth. But if such activities are performed excessively, they will lead to a setback in spiritual life. When the mind desires sinful gratification such as illicit sex or intoxication, one must simply tolerate the mind’s foolishness and by strenuous effort push on with Krsna consciousness. Then the waves of illusion will soon subside, and the path of advancement will again open wide.
Srimad Bhagavatam Canto 11, Chapter 20, Text 18
Srimad Bhagavatam Canto 11, Chapter 20, Text 20