It's okay to look away every 4, 5 seconds.
If you're talking to just one person, feel free to look away every 4 or 5 seconds, but make sure to not look down, as looking down may show disinterest. Look up, or to the sides. Looking up a little will appear as though you're thinking. Besides, continuously staring into a person's eyes will be uncomfortable to them as well.
If you're just the listener, occasional "aha", "I see", etc. will be appreciated. To maintain eye contact, change your focus every 4 or 5 seconds from one of their eyes to the other, and then briefly, to their mouth, and then back to the eyes. Keep rotating the focus this way.
If you're talking to more than one person, keep changing your focus from one person's eyes to another's every one or two sentences. If you keep eye contact with only one of your listeners, the others will lose interest.
Found an article from Michigan State University Extension, "Eye contact: Don’t make these mistakes", where they summarize points from The Conversation Aid as follows:
They suggest the following tips to help maintain good eye contact without staring:
Use the 50/70 rule. To maintain appropriate eye contact without staring, you should maintain eye contact for 50 percent of the time while speaking and 70% of the time while listening. This helps to display interest and confidence.
Maintain it for 4-5 seconds. Once you establish eye contact, maintain or hold it for 4-5 seconds. After this time passes, you can slowly glance to the side and then go back to establishing eye contact.
Think about where you’re looking. Maintaining eye contact is easy because you’re looking at the other person. However, when you look away, do it slowly without darting your eyes. This can make you look shy or nervous. And don’t look down; remember to look from side-to-side. Looking down can give the appearance that you lack confidence.
Establish eye contact right away. Before you begin talking, establish eye contact. Don’t look down or look at something before you begin speaking. Establish eye contact right away and then begin talking.
Listening with your eyes is important too: Remember the 70 percent rule (you should maintain eye contact for 70 percent of the time while listening)? Communication happens with your eyes while you’re listening just as much as when you’re talking. Remember that while you’re listening and maintaining eye contact, you should smile, open your face and look interested.
Practice. Eye contact will come easy to some, but if it doesn’t for you, it’s okay to practice until you become confident. You can look at an eyebrow or the space between the eyes and mouth. You can also practice with yourself in the mirror.
This article is the third and last in a series of articles that examined eye contact in communication. Remember that eye contact is a skill and it often takes time and practice to fine-tune our skills.