- For many centuries, Japan did not have much of an incentive to go exploring farther, since it had two wealthy trading partners next door.
- During the period in which there was the greatest amount of world expansion and exploration and trade by European nations (17tht to the 19th centuries), Japan was closed to most of the outside world by choice by the rulers of the time.
The Very Verbose Version
Besides the obvious—that Japan is not at all close to the Americas (it's 5,145 miles by air from Tokyo to San Francisco), here's a bit of historical background as to why it's really unlikely they would have sailed that far.
Japan is quite close to China and Korea geographically. Whatever was lacking domestically, the Japanese could obtain via trade with those two nations. For much of Japan's recorded history, China was a very wealthy nation with tons of resources. Korea was not a poor nation either. By and large, Japan maintained cordial relations with both for centuries. There was a period in the 13th century when the Mongol Empire overran the continent; they made several attempts to invade Japan too, which all failed. (People at the time attributed it to some providential winds that didn't allow the Mongols to land—these were called kamikaze, or "god winds." Yes that term did come up in a very different context later in Japanese history.)
So, there used to be a lot of trade between Japan and the continent; there's even evidence that some goods from the Middle East reached Japan via China. Therefore there was very little incentive for Japan to reach out further. (As a sidenote, there was also a ton of piracy going on, too. The Chinese and Koreans called them "倭寇," meaning Japanese bandits/pirates basically, and blamed it on the Japanese, while the Japanese denied it, etc. Scholars today believe that the pirates were a mix of groups from all three nations, with Japanese pirates more active earlier on and Chinese ones more active later.)
Remember that the major objective of European explorers was to find alternate routes to South and East Asia, especially to obtain spices, which were a commodity in extremely high demand and very lucrative. It was a bonus that they found a whole continent, plus plenty of other things to plunder or trade for as they went along. Again, Japan lacked that kind of incentive to go exploring.
In the mid-16th century, Japan was "discovered" by the Portuguese, which started a little trade between those two nations. At the time, Japan's preindustrial, artisan-based society was more urbanized than many of the countries in Europe, so they exported a lot of finished goods, especially gold and silver wares (and silver ore) as well as paper. In turn, the Portuguese seem to have introduced firearms, among other things. The Portuguese, who were mostly Dominicans, seem to have gotten along fairly well with the Japanese initially. (There are still vestiges of this cordial relationship between the two nations in various words and foods in Japan that originate from the Portuguese language.) The Spanish who came afterward, especially the Jesuits, were a bit more aggressive. They actively sought to convert as many people as possible to Christianity, but more than that, they may have tried to meddle in internal political affairs, too. This was seen as a threat by the rulers of Japan, starting with Nobunaga Oda, continuing on to Hideyoshi Toyotomi, and finally Ieyasu Tokugawa, the founder of the Tokugawa dynasty who controlled Japan from the early 17th century to the latter half of the 19th century. Active persecution and execution of converted Christians started in Hideyoshi's time and was conducted in earnest when Ieyasu took over.
The Tokugawas (it was actually Ieyasu's son and successor that implemented the official decree, and the third Tokugawa shogun Iemitsu who finalized it) decided that the threat from the Europeans, especially the Jesuits, was too much for a nation that had endured decades of civil war, and closed the country to all trade and contact with the outside world in in the first quarter of the 17th century, starting in 1615. This policy was called "鎖国"—sakoku, or "chained nation." It was an extreme example of an isolationist policy. Only four ports in the entire country were allowed to conduct trade with outside entities. Most foreign trade was conducted via Dejima in Nagasaki, in southern Kyushu—well away from Edo (current day Tokyo). The only outside nations and entities with whom Japan had any trade at that time were China, Korea, Ryukyu (current day Okinawa, which became a tributary nation of first the powerful Satsuma-han in Kyusu, and later of the Edo government), the Ainu in the north, and the Dutch East India Company.
The Dutch, the only Europeans with whom Japan had any contact with for most of the Edo period, didn't have much interest in converting anyone to Christianity, so they were still allowed access to Japan, despite a few blips along the way. This was a pretty big advantage for them since besides the trade with Japan, they had a good harbor for restocking in Nagasaki on their trade voyages to Asia. All Western/European knowledge that filtered into Japan was called rangaku (蘭学), or Dutch science, during the Edo period. If not for that steady flow of knowledge about the West via the Dutch, Japan would have been in a much weaker position when they finally opened up the nation again.
The Edo bakufu (Tokugawa shogunate) was so absolute in adhering to its policy and enforcement of non-involvement with foreign affairs that in 1644, when the Ming dynasty in China was trying to fight off the Qings and asked Japan for assistance, the shogunate refused, even though they had a cordial relationship with the Ming. They also rejected the Portuguese, and at one point executed 61 Japanese who were found out to have had dealings with them in Macao. In addition, any Japanese nationals who were stranded at the time the sakoku policy went into effect were not allowed to return to their home country.
This state of affairs lasted from the early 17th century until the third quarter of the 19th century—more than 200 years. (The arrival of Commodore Matthew Perry in 1854 is generally thought of as the first attempted contact by the West with Japan since the early 17th century, but this isn't entirely correct. There were several overtures made before that, starting in the late 18th century, but the Tokugawa shogunate rejected them all.) During all that time, many of the major European countries were racing against each other to gain control of this or that land, to gain more resources and more land. In the meantime, Japan was perfectly content developing its own culture within its borders, with little interference from outside.
Whether one approach was better than the other depends on your perspective. The way the question is phrased seems to imply that exploration and expansion are good things, but consider this: When Japan did turn expansionist much later on, it didn't exactly turn out well, to put it mildly.
Additional Historical Notes (you're still reading this!?)
A general grasp of what was going on in Japan during the 15th through the early 17th centuries is pretty important in understanding Japanese history and how it resonates to the current day.
To start things off, from 1467 to 1573, the country was at civil war, with local lords beating each other up and the general populace getting caught between them. During the last part of that civil war (or warring states) period, Nobunaga Oda (or Oda Nobunaga if you use the traditional Japanese family name-first name order) was able to to unify most of the country. Nobunaga was killed, and one of his generals Hideyoshi Toyotomi was able to actually unite the country for the most part. Nobunaga Oda was on the whole fairly amenable towards the Spanish and the Portuguese, since he saw the power of the Buddhist monks as a far bigger threat to the stability of the nation. Nobunaga even encouraged Christian missionary activity to an extent in order the suppress the Buddhists. (There's no evidence that he himself was ready to convert though.)
Hideyoshi Toyotomi was cool with the Europeans for a time too, but eventually changed his mind about them. He started the persecution of Christian converts amongst the Japanese population—many died rather than give up their faith. (One of the most famous "Japanese martyrs" was a high-born lady of the warrior class called Tama Hosokawa (née Tama Akechi) who took the name Gracia after being baptized. She killed herself by having a servant run her through with a spear to avoid being taken prisoner by one of Hideyoshi's generals. Her legend is helped by the fact that she was supposedly breathtakingly beautiful.)
On the international front, Hideyoshi did have some expansionist ambitions; he planned to invade mainland Asia on several fronts, demanded that the king of Spain come to Japan to pay him homage, and even had designs on invading India one day. None of these came to fruition.
Ieyasu Tokugawa, who eventually succeeded Hideyoshi, saw Christianity as a threat to the stability of the nation, and by extension most of those pesky foreigners. You can even think of Ieyasu as an anti-Hideyoshi; many of his policies seem aimed directly at undoing much of what Hideyoshi did. Even the 4-caste system that was implemented during the Edo period when the Tokugawas reigned may have influenced even in a small way by the fact that Hideyoshi did not come from the warrior or samurai class (the shizoku), but started his career as a lowly farmer and foot soldier.
In turn, the Meiji Restoration leaders who came after the Tokugawa dynasty was finally brought down did everything they could to try undo everything they did, both domestically and overseas. To simply things a whole lot here, this eventually lead to Japan's aggressive expansionism and colonialism during the early part of the 20th century.
One more thing! I've always wondered a bit why the United States didn't try to take a much bigger stake in Japanese affairs after sakoku ended, since Matthew Perry was an American, after all. But of course, I hadn't realized until it was mentioned somewhere recently that when Japan was finally opened up to outside contact in the 1860s, the U.S. was mired in its Civil War. So the British got to have arguably the most influence in Japan initially, with the Germans (Prussians) not that far behind, plus the French to an extent. The United States did get in there eventually though. This multinational influence can be seen in various ways: the Japanese central government was modeled after the Prussians, as was the first constitution under the Meiji government. The railroads were based on the British system. And baseball trumped cricket as the new, popular sport!
Related reading on this later period:
More questions on Japan: