The decision, just weeks before the opening ceremony, ends months of speculation about whether spectators will be allowed at the pandemic-postponed Games. Overseas fans were banned in March.
"In light of the government's restrictions on public events, the spectator limit for the Olympic Games will be set at 50 percent of venue capacity, up to a maximum of 10,000 people in all venues," organisers said in a statement.
A decision on spectators at the Paralympics will be delayed until July 16, a week before the Olympics open.
And officials left open the possibility of a reversal if the virus rebounds.
"If there should be major dramatic change in the infection situation, we may need to revisit this matter amongst ourselves and we may need to consider the option of having no spectators in the venues," Tokyo Governor Yuriko Koike said.
Senior medical experts, including top advisors to the government, have said that holding the Games behind closed doors would be "ideal" from a health perspective.
They fear crowds of fans could fuel a new surge in infections in a country still racing to vaccinate its residents.
The decision was announced after five-way talks between Tokyo 2020 organisers and officials from Japan's government, the Tokyo government, the International Olympic Committee and the International Paralympic Committee.
Speaking before the meeting, IOC chief Thomas Bach said he was "absolutely sure that it will be a decision to best protect the Japanese people and all participants".
Tokyo 2020 had already reportedly scrapped plans to sell more tickets, and may now face the prospect of organising lotteries among existing holders for the right to attend events.
Before the Games was postponed last year, organisers had sold around 4.45 million Olympic tickets and nearly a million Paralympic tickets in Japan.
In December, organisers said they would be refunding 18 percent of Olympic tickets bought domestically and 21 percent for the Paralympics.
That is still likely to leave many events with more tickets sold than seats available.
Japan has seen a comparatively small virus outbreak, with nearly 14,500 deaths, despite avoiding the harsh lockdowns seen elsewhere.
But the vaccine rollout has been slower than in many developed countries, only picking up speed in recent days. Around 6.5 percent of the population is now fully vaccinated.
Organisers also face a sceptical public. Polls have regularly shown most Japanese would prefer to see the Games delayed further or cancelled altogether.
Recent surveys suggest a softening of opposition, with more in favour of holding the Games than cancelling it -- if postponement is not offered as an option.
A survey published Monday found around a third of respondents want the Games to happen, up from 14 percent last month, though a majority still prefer further delay or cancellation.
Organisers say strict rules will keep both athletes and the public safe, and Bach said Monday that "well over" 80 percent of people staying in the Olympic Village will be vaccinated.
Athletes will be barred from contact with the public and risk being kicked out of the Games if they violate rules including mask-wearing and daily virus tests.
In a taste of the complexities ahead, a member of Uganda's Olympic team tested positive on arrival in Japan on Saturday.
The team was reportedly all vaccinated and would have had to test negative before travel to Japan.
