Before the Downfall of Númenor, Men fell swiftly under Morgoth, as they listened to his evil and cunning words. While they feared the darkness they believed, they also worshipped it.
Thus, some of them would sail west, as “they had heard a rumour that in the West there was a light which the Shadow could not dim.” (Silmarillion, “Akallabêth”)
Eärendil had sailed west just before then and never returned. After Manwë put forth Morgoth and shut him beyond the world for him never to return, the Edain set sail, following the Star of Eärendil.
The Eldar were summoned to the West, and a land was made for the Edain to dwell in which was neither in Middle-earth nor of Valinor. They are later called the Dúnedain – who would dwell under the protection of the Valar.
The Dúnedain learned the High Elven tongue, and became mighty in crafts, as well as nourished ship building and ship craft. However, the Lords of Valinor forbade them to sail westward – they had to be able to see Númenor. The Dúnedain couldn’t understand why they couldn’t go so far west, though the design of Manwë was that they couldn’t go there.
“…the Númenoreans should not be tempted to seek for the Blessed Realm, not desire to overpass the limits set to their bliss, becoming enamored of the immortality of the Valar and the Eldar and the lands where all things endure.” (Silmarillion, “Akallabeth”)
Valinor still remained visible to the world, and Ilúvatar permitted the Valar to go to the Earth. The Númenoreans were far sighted being able to see it from the Meneltarma. They didn’t dare to break the ban. However, the wise among them knew that it was the distant land, Avallónë, the havens the Eldar upon Eresseä. They brought many gifts to Númenor, and a seeding or a white tree of Celeborn which was planted in Eresseä – Galathilion the Tree of Túna, the image of Telperion that was given to the Eldar by Yavanna in the Blessed Realm. This was later called Nimloth as it grew and blossomed.
The Valar seldom went back to Middle-earth over the mountains, but stayed in the Blessed Realm. After the destruction of Númenor, Valinor, the Undying Lands, were no longer a physical part of Arda. This meant that for a time after that, only the Elves were allowed to sail there by the Straight Road.